Tech details of yesterday's issue

Tech details of yesterday's issue

by handy giff-staffer nigel on ‎03-10-2013 18:45 - last edited on ‎20-08-2014 11:00 by handy giff-staffer zobia22

As Mike mentioned this morning in his blog, we want to share with you some further technical details of the outage that occurred yesterday.

 

Just after 4pm the volume of sigtran traffic between some of the core nodes on our network (the HLR/MSCs and charging nodes) spiked to unexpected levels, causing an outage for all calls and the majority of texts. Alarms were raised across the network nodes and routers, which we investigated, as well as checking the processors that failed on Friday


By diverting some of the signaling and taking some services off line the volume of signaling traffic dropped to levels that meant we could restore service, shortly after 8pm.  Sadly with the layers of complexity, it took us much longer than we would have wanted. 

 

Since implementing the fix, our suppliers and technical teams continued to work through the night investigating the reasons behind this issue.  As part of a planned support / operational change on Tuesday, we introduced a new MSC/HLR pair. The post implementation testing and checks all passed and it wasn’t until we hit the peak traffic time around 4pm that the signaling volumes resulted in the failure of the routes between the MSC/HLRs.

 
We backed out this change in the early hours of Wednesday morning and are now evaluating the best way to implement this in future. It is likely that we will have to redesign and retest our approach and we will do so before re-attempting this necessary work.

 

We are confident that the issues on Friday and yesterday, whilst being in the same area of the network, are not connected. We cannot however be complacent and so have already implemented extra monitoring and plans in case we start to see similar issues.

 

In addition to the problem that happened yesterday, some of you may have been impacted by an issue with the wider HLR network today, which has now been resolved for the majority of giffgaff members.

 

We are committed to providing a great service and understand that this has not been the case this week.

 

Nigel

Comments
by suheluddin2 on ‎03-10-2013 18:48

Thanks for the update

by gazijazhar on ‎03-10-2013 18:49

Hello,

 

Yesterday I had issues with making/receiving calls. Now it seems to be ok but I have no internet access the whole day. Please advise is this the issue with the network or with my phone? Many thanks

 

Best regards,

 

Azhar

by 520415 on ‎03-10-2013 18:50
Thanks for the info
by hhaarrryyyy2020 on ‎03-10-2013 18:51
Ironic to post it now seeing as the network is down yet again.
by clairesexylegs on ‎03-10-2013 18:53
I'm having the same issues no ring tone carnt make calls or recieve and some txts arnt coming in this happened all day yesterday and resumed lst night but has started again today and still a issue at this time very inconvenient
by stales on ‎03-10-2013 18:57

No service all afternoon today and still no service.  You don't even know whether your service is working or not....some additional monitoring you have implemented.

by giffgaff Educator on ‎03-10-2013 19:00
Hey Claire and Stales, Sorry to hear you're currently affected - we do have a service issue here (http://giff.ly/1aO2iGz) where we can keep you both updated regularly on what's happening, Kind regards, Hazel.
by lotty15 on ‎03-10-2013 19:03

I wish i hadn't topped up my phone as i still can't receive calls or texts.

by beckie1994 on ‎03-10-2013 19:03

hello, since 10 this morning iv not bee able to recive calls or text but have been able to send them myself, not sure if its connected to this issue or not

by alisonjmboyd on ‎03-10-2013 19:04

I have still not got service!

 

Alison B

 

Looking for something?

Join giffgaff
  • For only £10 a month:

    • 500 minutes
    • Unlimited texts
    • 1 GB Internet
    • Free calls to giffgaff