Tech details of yesterday's issue

Tech details of yesterday's issue

by handy giff-staffer nigel on ‎03-10-2013 18:45 - last edited on ‎20-08-2014 11:00 by handy giff-staffer zobia22

As Mike mentioned this morning in his blog, we want to share with you some further technical details of the outage that occurred yesterday.

 

Just after 4pm the volume of sigtran traffic between some of the core nodes on our network (the HLR/MSCs and charging nodes) spiked to unexpected levels, causing an outage for all calls and the majority of texts. Alarms were raised across the network nodes and routers, which we investigated, as well as checking the processors that failed on Friday


By diverting some of the signaling and taking some services off line the volume of signaling traffic dropped to levels that meant we could restore service, shortly after 8pm.  Sadly with the layers of complexity, it took us much longer than we would have wanted. 

 

Since implementing the fix, our suppliers and technical teams continued to work through the night investigating the reasons behind this issue.  As part of a planned support / operational change on Tuesday, we introduced a new MSC/HLR pair. The post implementation testing and checks all passed and it wasn’t until we hit the peak traffic time around 4pm that the signaling volumes resulted in the failure of the routes between the MSC/HLRs.

 
We backed out this change in the early hours of Wednesday morning and are now evaluating the best way to implement this in future. It is likely that we will have to redesign and retest our approach and we will do so before re-attempting this necessary work.

 

We are confident that the issues on Friday and yesterday, whilst being in the same area of the network, are not connected. We cannot however be complacent and so have already implemented extra monitoring and plans in case we start to see similar issues.

 

In addition to the problem that happened yesterday, some of you may have been impacted by an issue with the wider HLR network today, which has now been resolved for the majority of giffgaff members.

 

We are committed to providing a great service and understand that this has not been the case this week.

 

Nigel

Comments
by samwich on ‎10-10-2013 20:19
I haven't got the faintest idea what any of this means, but I get the impression that is how giffgaff want to keep it. Good luck. :smileyhappy:
by whatisgiffgaffdotcom on ‎13-10-2013 19:26

Thanks for the update

Looking for something?

Join giffgaff
  • For only £10 a month:

    • 500 minutes
    • Unlimited texts
    • 1 GB Internet
    • Free calls to giffgaff