Knowledge Base
Community

Update on Friday's (27th June) data issue

Started by: nigel
On: 30/06/2014 | 16:12
Replies: 12
Reply

by: nigel
former giff-staffer

on: 30/06/2014 | 16:12
I'd like to give you all a little more information about the issue that we had on Friday afternoon which impacted the giffgaff website, app and data services.
 
At around midday we started to received automated alerts from the web servers that support the giffgaff.com site and over the next hour or so we started to get cases raised for poor data experience. 
This meant a poor experience for members trying to use giffgaff.com and our giffgaff app, who got an intermittent service. Sometimes the servers would respond properly and sometimes they would time out. Many of you will have seen that you could load a page or two, or log in and then would get timeout errors so you couldn't finish what you wanted to do.
It also meant that many of you could not use data (3G or 4G) on your phones.

Our investigation focussed on two areas - individual IT elements (servers, firewalls etc) as well as tracing an increase in usage which was causing congestion on the servers.

After switching over to some of our back up IT elements we found no hardware issues and whilst there was an increase in traffic it was from a known IP range.

Around 3.30pm we identified the specific firewalls in the network that were causing the problems and we barred these connections. 
This brought back some of the service - meaning that you could access giffgaff.com over your 'home' internet or wifi, but data service on you phone was still very poor, including accessing the giffgaff app and giffgaff.com.
 
At around 6.30pm the network element that casued the issues on the firewalls was removed and this restored data and app traffic.
On Friday evening the Operations team worked to check the stability of the site and to clear transactions, attempted during the afternoon, that were stuck
Nigel

Message 1 of 13
by: carolshanahan
on: 30/06/2014 | 16:16
Thanks for an explanationSmiley Happy
Message 2 of 13
by: darrenpainter
on: 30/06/2014 | 16:18

Out of interest what had spontaneously caused those previously working specific firewalls to experience these problems.

 

Was it a result of planned maintenance, a lack of maintenance, an external source acting upon the firewalls ?

 

We now know the what but not the why, which would go some way to reassuring people of the likelyhood of a similar issue occuring.

Message 3 of 13
by: nigel
former giff-staffer

on: 30/06/2014 | 16:28

@Darren

 

a fair question ... the post incident report which we got this morning highlighted a planned change that was incorrectly deployed on Friday. So we're confident that it wont happen again, but we are reviewing all the processes around making changes of this type.

Nigel

Message 4 of 13
by: darrenpainter
on: 30/06/2014 | 16:31

thanks @nigel it's much better to know that it was simple human error and that more rigorous change control can be put in place to avoid a repeat of the issue.

Message 5 of 13
by: manser
on: 30/06/2014 | 19:37

Thanks for the information.

Get a free giffgaff Sim
Message 6 of 13
by: djmaccaofficial
on: 30/06/2014 | 19:46
This ain't Friday this is all the time I've got a giffgaff SIM with 2 goodbags going to waste as I can't get a signal to use it .. DONT TELL ME TO CHECK THE STATUS ya wanna try sorting ya network .. Or more people like me will also get a three SIM and top it up and realise how bad your service actually is
Message 7 of 13
by: gordie10
on: 30/06/2014 | 19:52

Not sure that last bit is correct. The giffgaff app was definitely not working at 6:30pm on Friday evening (and it wasn't just me that was experiencing that). Service was not restored to the app until some time during Saturday (daytime).

 

So what was that about? Was that a separate issue?

Message 8 of 13
by: ma2013
on: 30/06/2014 | 19:58

Thanks for the update, but my giffgaff app wasn't back to working order until Sunday night, which was annoying. Anyway, thankfully I was able to access it via a mobile browser (Chrome), so wasn't too bad. But hopefully this type of error can be avoided next time (i.e. as long as people can learn from mistakes, then minor glitches like this aren't too bad). Smiley Happy

If I've been helpful, feel free to award Kudos / Mark As Accepted Solution to posts. Get a free giffgaff Sim
Message 9 of 13
by: root2
on: 30/06/2014 | 23:54

Are you sure it was an error from giffgaff, or is there any connection to BT, as their broadband had issues for much of the weekend.

 

Who hosts the giffgaff site?

Message 10 of 13