Categories
Blog Knowledge Base System Status

Telehouse outage 21st July 2016

Again today at 8am we started to see alarms coming in from customer sites. The effect seemed more widespread today with Zen inially posting they had issues and some ITSPs posting that they also had issues.

The outage appeared to have started at around ‘7am'(we saw first alarm at 8am , with at least one network at Telehouse North falling over due to a reported loss of power. The situation had mostly been resolved by around 11am, though are still seeing some flapping of services.

Like Wednesday’s outage, it appears that BT has a significant amount of networking equipment at Telehouse North. The has affected an unknown number of BT and Plusnet broadband subscribers, plus other smaller ISPs and services such as Zen Internet that make significant use of BT’s backhaul network.

We can confirm though we have slowly seen all customer alarms clearing. As many customers are aware that we operate a 24×7 monitoring platform so saw this issue start and checked that there was nothing we could do in most cases but also contacted key customers to warn them that they might be issues.
Therefore, any issues that Customers have experienced this morning when connecting to services using BT connectivity (including quality issues) should now be resolved. In the event that issues are still occurring, please reboot equipment on the BT line such as Firewalls or Routers and retest. Nagios monitor screen

If you have any questions whatsoever please do not hesitate to contact us, Also if you are a
Asterisk / Freepbx reseller or user and would like affordable monitoring please get in touch as we provide Asterisk Monitoring from £25 per year.

 

UPDATE ON 20th July issue

Tt looks like a power failure at Telecity Harbour Exchange, where BT and some other ISPs join the LINX peer exchange, is at least partially to blame for the major outage on Wednesday morning.

Service disruption as a result of the Telecity outage lasted for about an hour, from just before 8am until 9:15am, when full connectivity was restored.

LINX said in a statement:

This morning between 07:55 BS and 08:17 BST, one of the datacentres that houses equipment for The London Internet Exchange (LINX) experienced a partial power outage. This affected only one of a number of Internet peering nodes that LINX operates at the facility, and service was fully restored on the LINX network at 09:15 BST.

Several reports claim that this outage affected British Telecom and their services at LINX. While several networks connected to LINX were indeed affected, BT was not one of them. LINX provide two fully redundant platforms to offer better resilience to the UK’s Internet infrastructure, the second platform was not affected by this power outage.