Today's Service Outage

gaffer · ‎16-03-2012

As you very likely to be already aware, today we've suffered a major service outage - it was caused by major water damage to the computer network in a our data centre and it took nearly 8 hours fix.

Firstly I'd like to offer my apologies for the interruption in service and thank all of our members for their patience whilst we fixed the issue.

The service is largely back up and running now - the exception is new activations which are currently being queued and our service teams are working on a fix.

Some of you have asked about compensation - we're going to focus on fixing the issues we've had and make sure we have a stable service first - and then we'll sit down think of ways to make it up to you.

We'll continue to post updates on the support community so click here for the latest news.

Thanks once again for your patience.

Mike Fairman

CEO giffgaff

Update 21:15

Thanks for everyone's comments - especially those appreciating our way of keeping you up to date!

In response to a few of the issues raised...

Porting - those that were scheduled for today may have been delayed - we're continuing to work on these as fast as we can.

Texts - we think that for most of the outage, inbound text messages should have been delivered. Whether those that failed initially will ultimately be delivered will be determined largely by the re-try policies of the network from which they were sent. In other words, you may or may not get them depending on which network sent them.

Root cause - we will of course investigate this and report on the conclusions. Our data centre does have an uninterruptable power supply and backup generators but its possible that even if these worked the water damage to the network gear was sufficient to bring the service down.

gospelsinging · ‎17-03-2012

I have read as many comments as i can and i got to say, the way people are being slated for complaining is really not on.

I have given up complaining about gg's poor service. It has been consistently bad for me and i am actively looking around for another deal.

What i do want to say is not only did this outage caused me inconvenience (nothing new there then) BUT most of all it caused me untold WORRY as i it dawned on me pretty ealry that if an EMERGENCY occurred with my KIDS, the school would not have been able to contact me whilst i was at work. (AND BEFORE some of you start talking about landlines, I do not have access to landlines where I work).

i could cope with not ringing out, but knowing that the SCHOOL wouldn't be able to contact me was more distressing.

There are many defending gg, and i'm not saying we should all gang up and drown them with complaints , but they are providing a paid for service and building a brand. They should have a back up plan in place and a kind gesture to recompensate. It shows they understand the customer's reliability and trust in it's service and brand.

pinkcalculator · ‎17-03-2012

Both my kids have asthma and thus I make absolutly sure they have a large number of contact numbers for myself my husband and two other contacts.

slyfox69 · ‎17-03-2012

Hi to all yes i have to agree it was an inconvinience yesterday but why are people moaning about it is beyond me ?? as every network has its problems from time to time i remember a few year ago where no one could put top ups on the o2 network & that was like that for a few days not just well it wasnt even a day that giff gaff was out of action & would like to thank all who quickly restored the service what the complainers forget is that giff gaff offer such a superb deal anyway & also ive found giff gaff to be a very reliable service provider so well done giff gaff keep up the excellent service that you've provided me !so all at giff gaff keep up this great network

jhunter · ‎17-03-2012

Thank god it back on 6pm it finally returned all day with my iPhone 4 s not working lets hope doesn't happen again for a while

dumper1966 · ‎17-03-2012

Thank you for the update and apology, I past the time by amusing myself with those people ranting how "rubbish and poor " GiffGaff are and how much compensation they should expect, my reply is if ya don't like move networks, pay more per month n stop moaning. GIFFGAFF IS NO1

hamandred · ‎17-03-2012

What anyone has to consider with any major unplanned outage (once service is restored) is what learning's can be applied. Failover infrastructure and business process needs to examined. Was it available? Was it sufficient? Did resources function as expected? Did 3rd parties and vendors react and perform within agreed service targets? Etc...

I am sure those accountable for business continuity are performing Post Incident Reviews and wanting to mitigate against the events that took place. I would also imagine, some are looking at some fundamental failings in their Disaster Recovery Plan. The CEO has stood up and apologised, now he has take action internally to restore user confidence.

If any of the members on the forum think they deserve more than an apology and some clear communication (i.e. Compensation), then perhaps their priority is wrong. Would we all rather the network utilise any monies to plug any gaps in infrastructure and drive value for customers? I think so, and far better than doling out 50p-£1 credit to those on already generous Goodybags!

I couldn't make or receive calls and texts or use mobile data all day. I survived. Other means still exist, free wifi continues to grow, Skype and Viber have not ceased to function and you can still buy a plethora of phone cards and use them in phone boxes. Oh, and the strange box of electronics sitting on your desk or at home is a landline telephone.

I am not saying everyone's circumstances could be solved in these ways (someone would have been in an emergency situation and been distressed at not having a service) but there are always workarounds and they need not cost the customer significantly more in most cases.

Just my opinion as a grumpy old man of well over 30...

paulwade · ‎17-03-2012

The way you've handled the situation has been brilliant. It's just another example of why everyone should use your network.

smdelahay · ‎17-03-2012

Hi nope happy with giffgaff.com

merment · ‎17-03-2012

Thank you for keeping us informed. Sh*t happens.

Perhaps you could think about how you might use any compensation you were thinking to build more resilience into your systems. This would be my preference.

Good luck, and thanks again for the updates. If only other companies were as open and prompt in their messages .... I'll stay with giffgaff while you continue this way.

ammanwales · ‎17-03-2012

Thanks for the update, love this network Giiiiiiff Gaff.... Hope they have the back up server this time as we been lost more than 8hrs without texting or call out............ So what ever happen again hope they switch to the back up server within 8mins reather than 8hrs!