Today's Service Outage

gaffer · ‎16-03-2012

As you very likely to be already aware, today we've suffered a major service outage - it was caused by major water damage to the computer network in a our data centre and it took nearly 8 hours fix.

Firstly I'd like to offer my apologies for the interruption in service and thank all of our members for their patience whilst we fixed the issue.

The service is largely back up and running now - the exception is new activations which are currently being queued and our service teams are working on a fix.

Some of you have asked about compensation - we're going to focus on fixing the issues we've had and make sure we have a stable service first - and then we'll sit down think of ways to make it up to you.

We'll continue to post updates on the support community so click here for the latest news.

Thanks once again for your patience.

Mike Fairman

CEO giffgaff

Update 21:15

Thanks for everyone's comments - especially those appreciating our way of keeping you up to date!

In response to a few of the issues raised...

Porting - those that were scheduled for today may have been delayed - we're continuing to work on these as fast as we can.

Texts - we think that for most of the outage, inbound text messages should have been delivered. Whether those that failed initially will ultimately be delivered will be determined largely by the re-try policies of the network from which they were sent. In other words, you may or may not get them depending on which network sent them.

Root cause - we will of course investigate this and report on the conclusions. Our data centre does have an uninterruptable power supply and backup generators but its possible that even if these worked the water damage to the network gear was sufficient to bring the service down.

nexy · ‎16-03-2012

I appreciate the apology, that's now the only think that's persuaded me not to immediately begin porting my number elsewhere. Or at least, as soon as possible, as migration requests probably don't work at the moment. I understand your's is a relatively young company but I trust you appreciate that customers rely upon you. You have let us down, and at the risk of abusing a cliche, you've let yourselves down as well.

To have a single point of failure is entirely unacceptable. Damage to one site and the network goes down? What on earth were you thinking? 8 hours to resolution, largely during business hours? You are rapidly destroying the faith of your userbase as this is not the first major outage you've had recently. Word of mouth is a double-edged sword. Fool me once, Giffgaff...

To those comparing this issue to "my house flooded at Christmas" or "water leaks happen", that's utterly irrelevant unless your house is a major communications hub that thousands and thousands of people pay to use daily. In which case, you're an idiot for not having a working robust disaster recovery system that takes less than 8 HOURS to bring live.

thebeard · ‎16-03-2012

A note to those who are criticising giffgaff for not having "backup".

All services fail some time, if total communication is vital to you, have a (paygo) sim for another network at the ready.

There, now you'll have backup...

I'm also confident that giffgaff will learn from this outage, but no system can be made absolutely failure-proof.

Neil

coalfire · ‎16-03-2012

glad its sorted (if only temporary) , I thought it was my phone to begin with, not sure why I thought to check the site but glad I did. I had been waiting for a call about relatives hospital appointments so was able to call from another phone and give them a temporary number to contact me. Thanks again for the hard work and keeping us updated.

giffgafferman · ‎16-03-2012

We've lost nearly a whole days service here. I think its only right and fair that giffgaff add an extra day onto everyones goodybags. Its a gesture of goodwill and won't cost them that much.

You can't compare this to an electricity outgage at home, and they do pay compensation albeit small.

The network would have to go down on a day when I had to make lots of important phone calls and I was stuck at work where they don't let you use the company phone. Also lots of texts coming in and people wondering why you were not replying!

Why is there no backup duplicate equipment in another data centre? Cost issues? You would have thought there would be space in an 02 building somewhere for that?

nerakb · ‎16-03-2012

Thanks for the update Mike.Today can't have been easy,in many respects for all concerned at giffgaff.

Please make sure the service you provide is totally stable,maybe even a back-up system should anything like this ever happen again.

I haven't been with giffgaff for long but I'm happy being a giffgaffer.

imooates · ‎16-03-2012

Thanks for all the info today, it's been useful to have the constant updates. However I am interested to know why there isn't a backup server and if plans for an introduction of one in the future will be made?

oldyorkie · ‎16-03-2012

@pizadave .... it's PAYG ffs.... chuck your sim in the bin now!!... then take your phone and your little schoolboy posts anywhere you want .... no-ones keeping you here except you !!!???

D'oh !!!!

steviejay · ‎16-03-2012

My mobile phone is my only means of communication with the outside world, so today was a real pain.

dletley · ‎16-03-2012

Wow, didn't even notice the service was down...is that how unsociable I am?!

geordie69 · ‎16-03-2012

I agree with the small minority on here. Forget the compensation, maybe plolugh some cash into backup servers/systems.

Exactly the same thing happened about 14 months ago when there was a server outage and it took gaffers a whole day for a fix.....and even then the server was not stable for a number of weeks after this.

I cant say i missed my service today, but feel for the people that needed and depended on it.

Lets have some measures put in place to stop it happening again!