Today's Service Outage

gaffer · ‎16-03-2012

As you very likely to be already aware, today we've suffered a major service outage - it was caused by major water damage to the computer network in a our data centre and it took nearly 8 hours fix.

Firstly I'd like to offer my apologies for the interruption in service and thank all of our members for their patience whilst we fixed the issue.

The service is largely back up and running now - the exception is new activations which are currently being queued and our service teams are working on a fix.

Some of you have asked about compensation - we're going to focus on fixing the issues we've had and make sure we have a stable service first - and then we'll sit down think of ways to make it up to you.

We'll continue to post updates on the support community so click here for the latest news.

Thanks once again for your patience.

Mike Fairman

CEO giffgaff

Update 21:15

Thanks for everyone's comments - especially those appreciating our way of keeping you up to date!

In response to a few of the issues raised...

Porting - those that were scheduled for today may have been delayed - we're continuing to work on these as fast as we can.

Texts - we think that for most of the outage, inbound text messages should have been delivered. Whether those that failed initially will ultimately be delivered will be determined largely by the re-try policies of the network from which they were sent. In other words, you may or may not get them depending on which network sent them.

Root cause - we will of course investigate this and report on the conclusions. Our data centre does have an uninterruptable power supply and backup generators but its possible that even if these worked the water damage to the network gear was sufficient to bring the service down.

meepuk · ‎16-03-2012

Hello,

Appreciate the updates but ignoring the standard shouty comments but I think you guys need to address the following:

Were emergency calls (999 and 112) affected by this outage? I really hope you're going to say no and that you have processes in place with O2 to fail that traffic to or there's a secret industry solution in place.

Why did it take more than 2 minutes to realise your, apparently only, datacentre lost power? You may be skeleton crewed but surely you have some external and internal monitoring tools that'd flag this?

And finally the elephant in the rooom, why don't you have a DR site? Due to the impact this has had I think we at least deserve an explantion on the thought process behind this. I get it that no one wants to pay for something that they'll hopefully never use but we're not talking synchronus SAN replication and extending your L2 fabric with <1ms latency across 2 geographical diverse locations here. Simply enough to keep voice and text going would be enough to make any outages so much more platable.

I really hope GiffGaff learns from this but really, you're not some startup that are new to this, you're part of O2 and I would of thought you'd already know what can and does happen to a network.

Thanks

Meep

abx · ‎16-03-2012

I think all the "internet heroes" coming on here to defend giffgaff are wasting their time as much as I am.

It's almost like instead of a "standard" company's staff not caring, it's the members not caring instead.

I have a backup PAYG SIM from a reliable company as a way of ensuring I don't miss important text messages as I've got used to the fact the giffgaff is a weak network. Forget about not getting your texts from your husband/wife about dinner being in the oven; some people genuinely require the use of a mobile network as a way of contacting others during (for example) the last few days of pregnancy etc.

I don't really care what the reasons are for an outage (I would expect the company to inform it's customers of this which giffgaff to their credit did).

However, as a customer, paying for a service, a mobile phone service, I'd expect there to be some sort of backup plan to cover this. I know people are saying but "there was" and "there is"; IT DIDN'T WORK so it might as well not even exist. There should be a hot site ready to be ported in to action for restoring services to customers when something like this happens.

I'd also expect there to be a backup plan for the backup plan and then a backup plan for that. It's called disaster recovery with the aim of mitigating disaster scenarios or being able to continue providing a service when there is one. It's not a new concept but to giffgaff it obviously is not something they've developed well enough.

Right now I am still not sure full service has been restored and without going in to details it is currently a time when it is important that I receive all calls and texts from my better half so that has been a day of interrupted or non-existent service.

I've said it before and I'll say it again, it's not the customers that "moan" like me you should be worried about, it's the ones that don't moan but just leave, or realise that they shouldn't even join in the first place that you should worry about. Not that I think for a second that anyone in Giffgaff Ltd is the least bit worrying.

breakfromlife · ‎16-03-2012

When I woke up and saw that I couldn't reply to texts or make phone calls etc I had a bit of a panic but as soon as I saw the updates and how well all the staff kept everyone up date on what was going on what was fixed etc I felt reassured that you were all doing your best to solve the problem as fast as possible thank you all for the updates and getting everything fixed as fast as possible

zacharious · ‎16-03-2012

What action will be taken to stop

this happening again.

You give no compensation

you reputation has suffered.

I am very unhappy

like many other users.

Too slow and not enough

information.

Should it happen again.

I will leave.

Not happy.

janetjackson1945 · ‎16-03-2012

I think it would be sensible that suggestions of compensation were put to one side forever. As many have already said the compensation culture is truly more distasteful than having been without a functioning phone for 8 hours. If it was any of the other networks you would have had a generic message saying "We are aware there is a problem and our engineers are working round the clock to restore service" That is it!!! You certainly wouldnt have the CEO making a formal apology and specific information such as what the actual problem was I also doubt would be divulged.

I will agree that the bigger networks may have had better contingency plans but its a learning curve that Im sure GiffGaff will learn from.

Ultimately I would much rather GiffGaff concentrate its efforts on ensuring this doesnt happen again and put any available funds into UPS or other backup devices.

As a side issue to all whats happened I actually got a sizeable amount of work done today...... Quite refreshing to see there is life outside of the mobile phone world!

Finally I have to say to the CAPITAL LETTER and BIG FONT BRIGADE

GET A LIFE!!!

qasman · ‎16-03-2012

It has been a disappointing day for giffgaff customers. I am sure the staff did all they can, but i hope this does not happen again. I think giffgaff have had the most amount of network issues compressie to any other network ive been with. Oh well, the cheap offers do come at a price eh.

julie1967 · ‎16-03-2012

I'd like to say this to the person you goes by the name gypsymu...I dare you sit there and tell us gg members that we,re moaning cause the entire network went down today! You sound like an idiot...tell me..if you had a burst pipe in your house would you be sitting there grining and bearing it.. I doubt that you would be ..you'd soon be on the phone to the insurance company making a compliant and putting a claim in..gg members are allowed to complain..

williamuk · ‎16-03-2012

Mike,

I bet it has been a long day for you and your staff - THANKS for the updates.

I can fully understand why people are upset with the EGG Outage of Friday 16th March 2012 - I was one of us...

I personally would like to see any compensation put towards fixing the problem that caused this - but if there is a third party supplier responsible for this faulty service within their SLA (Service Level Agreements), and a penalty for compensation is contracted, then GG and its’ customers should go for it.

If there is not an SLA in place that covers this, then as one of the You of “The Mobile Network Run by You” I would be more than happy to come out of “corporate retirement” to help sort this oversight. My background is Strategy and Technical Services Manager for Cellnet, BT Cellnet, (pre-runner to O2) – of course my services would be my contribution to what I believe to be a Mobile Network of the future.

I believe that a power outage should not bring down a whole network. If designed properly and the correct due diligence done, a single point of failure should glow like a neon sign on a network design map.

GG is a great network and the people there must have worked their socks off today but learn to work smarter not harder.

THANKs again.

Bill Parkes

http://www.linkedin.com/in/billparkes

sorrowstouch · ‎16-03-2012

At least giffgaff kept us upto date, no other company would, it would just be that they are having issues

rainbow63 · ‎16-03-2012

The only compensation I need is the 20p you nicked fi my account before you covered your tracks with water.

Watergate all over again!!!!!!