Orange LLU Blackout
Five Points stand out for me:
1. Since the failure affected all of the UK LLU customers and therefore was not a DSLAM issue at a specific exchange then the only conclusion to be drawn is that currently Orange does not have redundancy in its’ network core.
2. TalkTalk is not the only ISP with severe reliability difficulties. Only this morning, OFCOM released an important report with one of the main conclusions “some business and residential consumers find comparing price and quality of communications providers difficult” I expect some regulatory action to force the operators to release quality of service information.
3. The Press are jumping on these negative stories about Broadband suppliers. It is very interesting to notice how many sources reported that Orange has gone down but how few reported Orange was back.
4. How pathetic the Orange network status information is. TalkTalk is even worse. The champion for providing detailed updated information is PlusNet and they should be commended.
5. How difficult it will be for the LLU suppliers to provide a decent quality of service with a small base of qualified engineers. The LLU suppliers are going to learn very quickly that there is a reason for the army of expensive engineers working for BT Wholesale.
As per normal, The Register has the exclusive on the cause of the problem - it wasn't a hardware problem after all - it was the fat fingers of an Alcatel engineer.
I work for Orange, in a role that has reasonable contact with the Alcatal guys who manage the Orange broadband infrastructure. The outage was caused by an "unauthorised" change to the network which one of the Alcatel guys performed.
As I understand it, the effect of this was a permissions problem with the hardware located out at the switches and exchanges, and now requires an engineer to visit each of the 4000+ sites to restart a few bits of kit. You will find that one by one the regions come back up, but at present approx 50 per cent of the LLU customers are still offline. I would imagine it will be another day or so before the rest of the customers are back online.