Today's Service Outage

[gaffer]

As you very likely to be already aware, today we've suffered a major service outage - it was caused by major water damage to the computer network in a our data centre and it took nearly 8 hours fix.

Firstly I'd like to offer my apologies for the interruption in service and thank all of our members for their patience whilst we fixed the issue.

The service is largely back up and running now - the exception is new activations which are currently being queued and our service teams are working on a fix.

Some of you have asked about compensation - we're going to focus on fixing the issues we've had and make sure we have a stable service first - and then we'll sit down think of ways to make it up to you.

We'll continue to post updates on the support community so click here for the latest news.

Thanks once again for your patience.

Mike Fairman

CEO giffgaff

Update 21:15

Thanks for everyone's comments - especially those appreciating our way of keeping you up to date!

In response to a few of the issues raised...

Porting - those that were scheduled for today may have been delayed - we're continuing to work on these as fast as we can. 

Texts - we think that for most of the outage, inbound text messages should have been delivered. Whether those that failed initially will ultimately be delivered will be determined largely by the re-try policies of the network from which they were sent. In other words, you may or may not get them depending on which network sent them.

Root cause - we will of course investigate this and report on the conclusions. Our data centre does have an uninterruptable power supply and backup generators but its possible that even if these worked the water damage to the network gear was sufficient to bring the service down.

[wirdy]

The feedback on the problems has been superb, at a time when the staff will have been under a great deal of pressure to restore the service. What many won't appreciate is that even with a backup server the switchover can be time consuming, and is certainly not straightforward. Perhaps we could be told a bit more about the backup domain controller, whether there is a tertiary server, and whether the outage was due to physical co-location of PDC and BDC?

[chandlerp]

for those whinging that GiffGaff didn't have a backup plan and so cost them thousands of pounds, where, pray tell, was your backup plan?

[benjyyy]

For all those people who have said today's problems have cost them dearly, where were your backup plans?

[marthahotels]

What did we do before the mobile came out! Well the wife rung the pub you might of been in and tell you the T was on the table! Anything electrical can go down, Like my bank went down last Saturday so my debit card did not work!! So giffgaff is not the only one 2 go down!

[benjyyy]

Haha looks like chandlerp and I think very much alike!

[flateric]

Thanks for the updates today and the Blog post.

A chain is only as strong as its weakest link is true in many scenarios and hope this is resolved and a risk assessment of other potential outages.

Last time compensation was issued it went to a charity - Help for Heroes iirc. I was inconvenienced and had to use VOIP but nothing couldn't live with. There are others that had a real need and feel sorry for them (and the stress the giffgaff staff were under today).

Saw a post that said the following on a different thread in notices:

mrkylewalker wrote:
I can BUT my Nephew cant as just found out VIA Facebook he didnt make it through the car crash he was involved with @ 11am but NO one could inform me until i logged on to Facebook. So Thank you so much GiffGaff

[summy92]

 I am really disappointed that I couldn't use my phone all day due to this problem. I can understand and appreciate that the Giffgaff team must have been really busy trying to sort the problem out but as a giffgaffer I felt that this could have been avoided if you did not just rely on one supplier to power the whole network. However, putting what has been said to aside Thank you so much for keeping us updated  You definitely helped me control my anger and frustration. So Big Up Will

[kustominteriors]

thanks for the update, cant have been an easy day.  True we all rely on our phones and its been a nightmare not being able to communicate, especially when your located in a rural area with no access to the communication channels of a city but appreciate all the updates as and when they have happened.

ps, when the other networks go down it can be hours before they even bother to get in touch and even then they can be pretty useless and not update you, so if there is a positive in a negative situation it is the way everything was updated

[dj03djj]

Thanks for the updates, has been a pain not having a phone all day. Even had to use one of those Telephone Box things....first time in about ten years LOL. Rather the money be spent on back-up system than compensation....Still love GiffGaff.

[dugblare]

For my part I wish to thank giffgaff for keeping us informed in what must be a trying time for the Management Team. I judge a company by its performance when things go wrong and I can honestly say that I remain committed to GG for the foreseeable future. So GG I believe 99 per cent of members are behind you and look forward to participating in the full scope of the Services very soon