Tech details of yesterday's issue

[#1]

As Mike mentioned this morning in his blog, we want to share with you some further technical details of the outage that occurred yesterday.

Just after 4pm the volume of sigtran traffic between some of the core nodes on our network (the HLR/MSCs and charging nodes) spiked to unexpected levels, causing an outage for all calls and the majority of texts. Alarms were raised across the network nodes and routers, which we investigated, as well as checking the processors that failed on Friday. 

By diverting some of the signaling and taking some services off line the volume of signaling traffic dropped to levels that meant we could restore service, shortly after 8pm.  Sadly with the layers of complexity, it took us much longer than we would have wanted. 

Since implementing the fix, our suppliers and technical teams continued to work through the night investigating the reasons behind this issue.  As part of a planned support / operational change on Tuesday, we introduced a new MSC/HLR pair. The post implementation testing and checks all passed and it wasn’t until we hit the peak traffic time around 4pm that the signaling volumes resulted in the failure of the routes between the MSC/HLRs.

 
We backed out this change in the early hours of Wednesday morning and are now evaluating the best way to implement this in future. It is likely that we will have to redesign and retest our approach and we will do so before re-attempting this necessary work.

We are confident that the issues on Friday and yesterday, whilst being in the same area of the network, are not connected. We cannot however be complacent and so have already implemented extra monitoring and plans in case we start to see similar issues.

In addition to the problem that happened yesterday, some of you may have been impacted by an issue with the wider HLR network today, which has now been resolved for the majority of giffgaff members.

We are committed to providing a great service and understand that this has not been the case this week.

Nigel

[#2]

Thanks for the update

[#3]

Hello,

Yesterday I had issues with making/receiving calls. Now it seems to be ok but I have no internet access the whole day. Please advise is this the issue with the network or with my phone? Many thanks

Best regards,

Azhar

[#4]

Thanks for the info

[#5]

No service all afternoon today and still no service.  You don't even know whether your service is working or not....some additional monitoring you have implemented.

[#6]

Hey Claire and Stales, Sorry to hear you're currently affected - we do have a service issue here (http://giff.ly/1aO2iGz) where we can keep you both updated regularly on what's happening, Kind regards, Hazel.

[#7]

I wish i hadn't topped up my phone as i still can't receive calls or texts.

[#8]

hello, since 10 this morning iv not bee able to recive calls or text but have been able to send them myself, not sure if its connected to this issue or not

[#9]

I have still not got service!

Alison B

[#10]

http://www.theregister.co.uk/2013/10/04/o2_customers_grateful_to_be_back_on_line/

BAckground to some issues affecting O2 networks (which will also have hit giffgaff users, as wella s Tesco Mobile).   

[#11]

I haven't got the faintest idea what any of this means, but I get the impression that is how giffgaff want to keep it. Good luck.