First of all thanks We really appreciate the patience of our customers and can't thank you enough for the support you have given us so far. We're doing everything in our power to get on top of the network issues and deliver the awesome network that we all want!
What we've done so far The last couple of months have been quite frustrating, both for us and for our customers. Here's a quick update on what's been happening behind the scenes.
Manufacturer support - or lack of When we first started to see the performance issues appearing on the network a couple of months ago we started trying to get help from the manufacturers. Despite investing in excess of £25k with a single manufacturer we were unable to get any decent support from them.
Over a period of around 6 weeks we have exchanged dozens of emails with them explaining the network and the problems in great detail hoping that they would simply tell us what was wrong. Unfortunately this was not the case and we've recently given up with them entirely.
Plan B - a model or test lab Our network is quite complex in terms of the design and configuration, so to troubleshoot problems is quite tricky. We've decided to combat this by building a complete test lab that accurately represents the network our customers connect through. We are now testing various configuration changes with a view to finding the best possible setup and then deploying this globally in one change.
We are in talks with numerous professionals about the configuration and are working with them to double check everything across the entire network.
First steps towards resolution One of the challenges facing us is the way that the devices around the network store information or logs. We've been working on a way to centralise all of this information so that we can dig into the data easier and find what might be causing problems. This is being deployed to the network in the early hours of 27th October and will make it easier for us to respond to the needs of the network over the coming weeks.
What next? Over the coming weeks we will be monitoring the centralised logs and using this information to adjust and refine the settings in our test lab. This approach is believed to be the best and most reliable way to work through the current issues and we'll post new announcements with progress along the way.