left boarder
Network Notices

 
UTV INTERNET FAULT REPORT - 20100419-01
Date / Time :	19th April 2010 22.25
Subject 
[1st line synopsis] :	UTV Internet Belfast PoP outage
Reference :	20100419-01

Priority [1 - 5] :
(as per SLA)	-

Notification :

Who :	-

UTVi Network Engineers
When : 	20th April  2010 07:45
How :	-
Incident :	Belfast PoP network failure 

Cause :	A network loopback error was detected by the core network switches in the Belfast PoP @ 22:25. This is expected in a redundant switched core but instead of handling this event normally, the primary core switch suffered a software crash and entered an unstable state.  This switch continued to acknowledge heartbeat queries and so failover to the secondary core switch was not initiated. This loop condition was then able to propagate to access layer devices which reacted in disabling uplink ports as a security measure.

This caused multiple switching errors to the core and access layers – affecting individual hosts and network performance degraded substantially which affected all Belfast hosted services (co lo, dedicated, shared, authoritative DNS and mail).

Resolution / Actions :	UTVi engineers began working on this issue upon discovery, both remotely and onsite. After unsuccessful debug, core switches were physically restarted and connectivity briefly restored. Core redundancy was disabled to aid diagnosis and all access layer devices were disconnected from the core and brought back up in a controlled fashion. Several core configuration inconsistencies were discovered and repaired. These were determined to be symptomatic of the core switch failure rather than the root cause.

Upon restoration of the network, all services were health checked and restarted where required.

Next Action:  An update will be provided indicating any proposed or plans changes to the network to avoid reoccurrence of this fault.
Resolution Date / ETA :	The majority of services were restored on 20th April at 11:50.
Full fault resolution and investigation is still ongoing.
 
PLANNED ENGINEERING WORKS [PUBLIC]
 
Supplier Maintenance [TUE, 23-FEB-2010]
 
Planned Works Notification [THU, 18-FEB-2010]
 
NI EMERGENCY PLANNED ENGINEERING WORKS
 
EMERGENCY PLANNED ENGINEERING WORKS [PUBLIC
 
Emergency Planned Works Notification [TUE, 18-AUG-2009]
 
Emergency Planned Works Notification [FRI, 31-JUL-2009]
 
Planned Works Notification [TUE, 28-JUL-2009]
 
Planned Works Notification [TUE, 23-JUN-2009]
 
Planned Works Notification [TUE, 02-JUN-2009]
 
Planned Works Notification [TUE, 26-MAY-2009]
 
Emergency Works Notification [MON, 13-APR-2009]
right boarder