Get Help
Community

Today's Service Outage

handy giff-staffer

As you very likely to be already aware, today we've suffered a major service outage - it was caused by major water damage to the computer network in a our data centre and it took nearly 8 hours fix.

 

Firstly I'd like to offer my apologies for the interruption in service and thank all of our members for their patience whilst we fixed the issue.

 

The service is largely back up and running now - the exception is new activations which are currently being queued and our service teams are working on a fix.

 

Some of you have asked about compensation - we're going to focus on fixing the issues we've had and make sure we have a stable service first - and then we'll sit down think of ways to make it up to you.

 

We'll continue to post updates on the support community so click here for the latest news.

 

Thanks once again for your patience.

 

Mike Fairman

 

CEO giffgaff

 

 

Update 21:15

 

Thanks for everyone's comments - especially those appreciating our way of keeping you up to date!

 

In response to a few of the issues raised...

 

Porting - those that were scheduled for today may have been delayed - we're continuing to work on these as fast as we can. 

 

Texts - we think that for most of the outage, inbound text messages should have been delivered. Whether those that failed initially will ultimately be delivered will be determined largely by the re-try policies of the network from which they were sent. In other words, you may or may not get them depending on which network sent them.

 

Root cause - we will of course investigate this and report on the conclusions. Our data centre does have an uninterruptable power supply and backup generators but its possible that even if these worked the water damage to the network gear was sufficient to bring the service down.



853 Comments
newcomer

GG, You have apologised numerous times and that is enough for me :) The company I work for has back up servers, however they are in the same data centre ... so who said they don't have back up servers ? They could of just been in the same data centre .... In a country where natural disasters are uncommon .. who would even consider putting their backup servers in a different data centre ? If the slave has to copy the main server .. imagine hoe big the data connection would have to be !! I personally think to have backup servers in a different location does not make sense !! :) but thats my view !! Keep up the good work GG and dont worry about compensating me ... spend it on paying your staff lotssss of overtime ;)

beginner

I'd just made an offer on a house so it was a many thousands of pounds call I was waiting for, didn't really bother me when I figured out it was the network, not my phone. 

Compensation, get a life, why does everything have to be compensated, just give everyone an extra day on their goodie bags, job done.

Oh we got the house by the way, my wife has a phone, it's not hard to work round.

newcomer

Thank you Mike for being professional and providing your customers with an explanation, apology and acknowledgement of today's outage.

 

I too as some others here, work in IT for a global fortune 100 company and know only too well how an outage can have a major impact to the customer.  IT systems are not merely a single computer sat in some dusty room.  This is a major network infrastructure with complex equipment and services working to provide your calls, texts and data.

 

Disaster recovery and backup contingency is also very complex, time consuming and is also subject to the same chance of error (even when well planned and executed).

 

I agree with those that say, no compensation required, you all did your best to get us up and running with minimal fuss and at the same time provided constant updates and handheld those complaining.

 

To those complaining, if you rely so much on a mobile phone, where is your contingency (backup mobile, backup sim)?  Were you prepared for an outage?  What if your handset exploded into a ball of flames?  Don't complain to giffgaff when this was both unforeseen and beyond their control.

 

Instead of unnecessary compensation, plough any available resources into improving giffgaff's BCP and the already excellent low-cost service to it's members.

 

Thank you to all the giffgaff team and engineers for all your hard work.

newcomer

can a message be sent out to let people know when service is back please.:catvery-happy:

beginner
Thanks for the candid update. I appreciate that these forums are popular and well used, but surely in a crisis like this the first place to update should be http://status.giffgaff.com/ - still showing "No known issues" ten hours on!
sensei

If compensation is to be paid, then why don't giffgaff ask customers if they want the compensation or donate it to charity?

newcomer
I agree with the comments that a text could of been sent out (since incoming texts were working) to notify users that there was a problem. The systems that were affected; can anyone confirm if they were the systems that held information as to whether the user had credit/texts and therefore were allowed to send/make calls. If this check was true then the text/call passed to O2. This would also explain why balances and goodybags were not available.if this was the case, could a switch/temp rule have been added that for the duration of the outage, allowed all users to send texts/make calls, essentially waiving the'check.. Maybe talking rubbish but just thinking out loud really..
newcomer

What a load of crock, I came from vodaphone with also my daughter last month and I have never felt so angry with the service today because my daughter or myself couldn't contact each other when needed, I know that this wouldn't have happened if I had stayed on vodaphone because they have staff searching for problems before they happen, if for any reason it had happened I know they would have compensated us some how, well to top it off my minutes and BBM run out today and I can NOT put any goody bags on so what is the point of connecting to this network at all ? Yes the staff have been updating this site today because they probably had nothing else to do because the network was down while some poor bloke has been fixing the problem out. My partner only yesterday was on the phone to orange who he has a contract phone with for the past 5/6 years was asking them how much it would cost to buy his contract out so he could move over to giffgaff but I cant see that happening now can you ? Anyway rant over and please sort this out or I'm sure you will lose valid customers to keep your company afloat. 

by thebeard on 16-03-2012 19:57

A note to those who are criticising giffgaff for not having "backup". All services fail some time, if total communication is vital to you, have a (paygo) sim for another network at the ready.

 

Did you have another phone with you today? I bet not? How many people carry around 2 phones with them anyway, especially 'just in case' their main network goes down??? Hardly anyone and nor should they have to. Its up to giffgaff to have a backup - NOT their end users! How ridiculous.

 

In all my years of having a mobile phone with just about all providers, I have never had an outage anything like as long as this one has been today. 30 mins max. Never all day. Crazy!

 

 

dabbler

Thanks for all the updates guys. It's certainly not been GG's year so far, but at least you're getting to deal with issues relatively early in your lifetime...here's to a long problem-free life! I'm sure it'll start some time soon!