Hi all,
Unplanned Outage
Hi all,
We are aware there are issues with the data traffic on our network. Some people are having a problems with Slow Data Speeds throughout the UK which we’re trying to fix now.
We are working together with O2 to try and fix this.
The impact of this is that giffgaff customers are experiencing slow data speeds when trying to connect to GPRS
We will keep you updated with any extra information we can get.
Update - 12:12
Unfortuantely nothing further to report. We're still investigating. Although some people are reporting their internet is back online, we still see plenty of people claiming the opposite. So hold tight while we continue to push for a resolution.
Update - 13:29
A fix has been applied which fixes the issue for most of the members on our base, but not everyone. To apply a proper fix that helps a 100% on the base they would likely have to pull the plug for a while. So rather than doing that right now at peak times, there is a debate on doing this tonight, rather right now.
We'll continue looking into other possible solutions to apply something that helps everybody (without having to pull the plug).
Update - 14:39
We''ve just had official confirmation that the maintenance to be carried out on the network to sort it for the full 100% has been scheduled for tonight at midnight.
The service appears stable and we’ll be monitoring over the course of the day and as said making some changes this evening.
Update - 9:40 (6 April)
Just to let you know, we have had the official confirmation that the data side of the network is now working as it should.
We'll look to see if we can get some more details later on, if possible, but for now, the network is fully functional and up and running.
Update - 15:10 (6 april)
After further analysis of the network data outage yesterday and the fix that has now been implemented we, together with O2, have found out that the issue was identified on the giffgaff network only. So not, as previously thought on both the O2 and the giffgaff network.
Our understanding of the issue was incorrect and our apologies for therefore giving incorrect information.
We had two configuration issues since starting charging for data and running the whole traffic through the service
On several occasions new devices connected very quickly after previous connections had been disconnected, but the previous session has not finished being closed down on all platforms. Given the number of users finding themselves without credit was probably very high the first day after enabling charging this condition was more prevalent and partially responsible for the issue yesterday. By not disconnecting users (leaving them connected but blocked) this will give us more stability.
We also saw that we were recovering IP addresses from the server more frequently than we should have been, this was set to an hour and should be longer, to allow the addresses to be cleared from the back end servers before they are re-allocated.
A change has been made to reduce the timing of this process and will be monitored today and tomorrow.
O2 has had constant monitoring in place since the fix has been put in place last night and the service is now running normal and stable with a 100% session success rate for data.
However, as I can see from some of the comments on this thread, some of you have let us know you are still having issues with data connections speeds and well as data connections altogether.
Should you have an issue, please raise a case with one of the agents, providing them with the details of where you are (postcode) and the speeds you are getting.
We will then pass these on to O2 to look at and see if we can find out why some of you are not getting the same speeds you previously got.
Sorry for the inconvenience
Kind Regards
Update - 17:08 (8 April)
Hi all,
Thank you all very much for your feedback on the slow data.
Using your feedback my team have just completed a comprehensive report and its our hope that this report will help with the ongoing investogations.
I'd like to personally thank you for your patience, support and invaluable information.
David
UPDATE – 11 April 2011
Hi Everyone,
We would like to thank you for sending in all the details regarding the data speed issues. We now have sufficient data to work with so for now there is no more need to submit further data or new cases.
Please note as part of trying to resolve these issues we have members running a Data Speed test and feedback may be tracked here
This is a priority for us however please note this isn’t something that can be resolved in a couple of days. Unfortunately due to the complexity of the data network and the many different aspects concerning this issue it may take some time to fully investigate and resolve any issues found with our partners and O2.
We thank you for your continued support and patience.
David
Update 12 april (12:42):
Hi all,
Just as an update, our tech team are at the moment at O2. Working with their techies to look at the different issues and what can be done to resolve some of these.
As far as we can understand now, implementing a data charging platform, no matter how much you test it, will always throw up some unexpected issues, and it will take a few weeks to iron out all the details during the period right after you've made it live.
So please be aware that this might unfortunately not be resolved completely within a couple of hours.
However, as far as we are aware there are 3 different issues that are being worked on.
The first is about how our data charging platform handles payments. Regardless of whether you are on a goodybag or not, you still work with that same platform. And part of this is that the platform takes a certain amount of credit from your account to start the data session with, and then refunds whatever is left over at the end of the session. At the moment that amount is set to a very low amount (not sure of the details here) which means the server continues to ping back and forth between the credit it has and what it should get. Which could cause slower speeds.
Secondly an issue with the APN settings is being investigated at the moment. Again, this is a priority and as such will have the necessary investigation attached to it and a team working on possible solutions.
And thirdly we are looking at all the different cases you guys have logged with us and are looking to see if we need to provide more details from a server side perspective or if we eventually need to ask for more from you.
I am hoping for more of an update this afternoon, with possibly some more details.
Cheers,
Vincent
Update 12 April - 13:58:
Hi all,
We have just had an update about the internet issues today so here is the latest update. The APN was the issue and although it wasn't down, it was not running at full capacity. This issue didn't affect everybody just some members, with slow browsing speeds and no internet access. The APN seems to have stabilised and should now be back up and running to 100% and it will continue to be monitored. If there is another failure the platform will be restarted, which will cause an data outage of 45min to 1hr but you will get a heads us if this needs to be done.
Cheers,
VIncent
Update 13 April - 09:09:
Following the outage yesterday, thank for the feedback. We have passed this on for further investigation and will keep you updated.
Cheers
David
Update 14 April - 10:57:
Hi all,
As an update this morning, we are continueing to work on this with the highest priority. We have the highest level people within O2 working on this as well as, of course, everyone here at giffgaff.
Although we have found there is definately an issue with the data speed itself and the connections we are setting up, it is still unclear where exactly in the infrastructure this issue is manifesting itself. Whether hardware, software or both is still unclear due to the complexity of the network.
The investigation is continueing and we are preparing ourselves for some diffecult discisions regarding the next steps we can make.
We will keep you updated and should have more information for you later this afternoon.
Once again, our apologies for the inconvenience caused.
Kind regards,
Vincent
Update 14th April - 17:25:
Hi all,
As an update I will try and explain where we are.
There are 2 separate issues, 1 being the slower data speeds and two being the connection difficulties.
As an update on the connection issues I can tell you we have found the root cause. And the way to explain this is possibly in an illustration of the M6 Toll road.
Imagine a steady stream of cars passing the toll booths. Now when one toll booth is out of order, the traffic needs to slow down a bit to let the rest through. Now imagine a couple of these booths not working and I guess you can see a traffic jam start appearing. And that means less people can get through (connect to the data network).
So although we have found the root cause which starts this traffic jam, we can't actually properly fix it. Obviously this was never intended to work this way, which means the way the software has been written can't actually cope with it.
So for now we have one of our suppliers working on a patch. And as the name applies, this is not a full blown solution, but rather a patch job.
On top of that we are implementing some changes to how the network itself deals with the traffic which should alleviate the problem as well as that we're implementing monitoring tools to keep a close eye on the queues that might be forming.
Should we see these queues forming and therefore impact the wider network (and so more people on giffgaff will experience slow speeds and connection issues, rather than a few which on retry will get in), we have made the decision to do an immediate reboot of the data network.
This means there will then be a data outage which last for about 30 minutes.
This also means that we here at giffgaff will get very little warning and in fact might be informed after the reboot has already started.
We have made this decision to ensure everyone is impacted as little as possible and the service stays running as optimal as possible for the most amount of people.
As for a full fix....
Well, here is the crux of the story I guess, as that will likely be a few weeks away.
Why?
Because we will need to redevelop the software almost from scratch as it will mean a complete rebuild of the platform.
As for the second issue, slower data speeds, we recognise that this is an issue and are working hard on finding a solution for that as well.
However, here we do have a problem locating, finding and replicating the issue.
As has been clear from the many replies here, the issue is not noticed across the board. We know that a certain amount of people are getting slower than usual speeds, but have not been able to exactly identify why. And when I say we, I mean giffgaff and O2 together as we are working very closely on this issue to find ways of improving our speeds.
Part of identifying the problem and finding a possible solution is happening right at this moment with the community and the O2 and giffgaff sims we have send out for people to test the variance in the network speeds people are getting.
We will continue these tests and we are looking to the community and those people who are testing this for us to possibly be directly contactable by the O2 engineers working on this, to ensure they get the exact information they need in order to solve the issue.
We will of course contact those individuals that we need additional information from to see if they are willing to participate in this direct contact and hope everyone is up for it.
That's it for now as an update. I hope it has been clear enough and we will of course continue to work hard on these issues and should there be additional information I can give you, you'll be the first to know.
Cheers,
Vincent
Update 15 April - 17:15:
Hi all,
I have another update for you today:
We are continuing to progress on three issues:
The core issue that caused the outage last week and for a few people this week was connections not being available (I explained this previously using the M6 toll roads as an analogy). Well, a work around has been implemented and additional monitoring is now in place so we hope this won’t happen again. Adding this as a permanent fix into the software is a few weeks away but for now we have addressed the issue.
We have identified a network element that may be giving giffgaff members a 30% lower speed than O2 prepay, we are now testing this in our test environment as data (radio) coverage and devices in the live environment makes this comparison harder. These results will be key in deciding next steps and we should have some feedback on these results on Monday.
We are still approaching individual members in relation to the very poor speed that members are experiencing, this information is going to the O2 engineers and they, in turn, may want to contact our members to get them to test whilst traces are running on the network. This may happen this afternoon or next week.
I can confirm
- There is no bandwidth restrictions that are applied to the giffgaff service from O2
- There are differences (APN and IN billing platform) to Tesco mobile and O2 and each of these are being looked at to determine whether these are the causes of differences between O2 / Tesco and giffgaff
A number of posts have suggested rolling back, and whilst this is a possible option, it’s not a likely one at the moment until we have a clear answer on the 2nd issue above, which we are expecting at the start of next week.
Once again, we are very sorry for the inconvience but please be aware we are working very hard on a solution.
Cheers,
Vincent
Update 19-04-2011 15:29
Hi all,
As an update I’d like to share with you our latest findings.
The changes made for the connection issues are providing stability and are looking good.
With thanks to our testers working on the Data speed test; we have been able to identify the cause of the slower data download speeds.
To get the same speeds as O2 Prepay (we are currently running at about 70%) we will be installing and configuring additional hardware. Allowing for validation and the Easter period our plan is to have this completed by the end of next week.
To ensure there are no more specific issues we will continue to look at individual cases where the data speeds are low or where there is no connection at all.
Once again, our apologies for the inconvenience caused
Cheers
David
Update 21-04-2011 16:20
This afternoon we received the results of the lab testing that compared data speeds in a controlled environment with differing architectures, giffgaff, o2 prepay and the previous set up, without any charging.
I also had an update on the connection problem (which Vincent made an analogy to the M6 toll) and when we could expect the additional hardware to provide a better download speed (the date I was given earlier in the week was Tuesday 26th).
Our aim was to provide much better speeds and a stable service next week, what I have seen in the last 24 hours indicates that we would not achieve that on Tuesday, as previously promised, but in a few weeks time.
So we have instructed O2 and their suppliers to roll back data charging and revert to the previous solution, they are looking to do this as soon as possible. But it needs to be tested and planned so it will not happen until after this bank holiday weekend.
We didnt choose to do this earlier as the workaround fixes were working and we expected the data speeds to be significantly improved on Tuesday
David or Vincent will give a further update on the actual date that we can meet on Tuesday afternoon.
Update 26-04-2011 15:33
Hi Guys.
Following our last Update 21-04-2011 16:20 where we announced to roll back data charging and revert to the previous solution.
Please note the roll back has been scheduled in for overnight Thursday 28th April at 12:30am and will last until 5:00am.
Cheers
David
Update 28-04-2011 09:21
Good morning all,
As some of already seen the roll back was successfully implemented and tested in the very early hours this morning.
For more details about free data and when the fix will be implemented please see our gaffer's blog post.
Cheers
David
Update on 12-05-2011 12:22
Hi all,
Following the rollback and reading members feedback it sounds like a small amount of members are still experiencing some Internet speed issues.
Where this is the case please contact an Agent and ask them to raise a new case for you. Please note the agents have a template that they will use to collate your individual data on a template and to send it onto O2 for further investigation.
Due to the length and complexity of this thread I'm going to create and new thread for this please see here and close this thread.
Thank you
David
fair enough
