For detailed notes on how these issues have developed and what has been done about them so far please read the updates below in this thread.
For a summary comment published by our CEO see here
FINAL UPDATE 31st August 11.12
We are finally at the end point of the issues which emerged last week.
Everything should be working normally (I am touching a large piece of wood as I say that).
There are a couple of hangover effects:
1) Activations are still taking longer on average than they did before these issues cropped up. New members activating today should expect that activation may take 1 - 2 hours. We are continuing to change the underlying system structure to improve this - and Nigel Suddell will be updating the community later this week with a detailed update on what went wrong and what are doing to prevent the same thing happening again
2) There is an intermittent fault that is affecting a handful of users (4 or 5) where they find themselves unable to buy a goodybag - the screen not progressing beyond 'continue'. These are being investigated
3) There are a similar number of members reporting disappearance of goodybags. These are also being looked at individually.
4) Ports - all of last week's port ins have now completed successfully - and we are watching today's file of 114 very closely.
5) Compensation. We have compensated all members affected by Ports. On the 1st September we are processing a file that will automatically compensate all members who were left out of pocket by the delayed goodybag issue. This credit will have two elements:
- a pro rata element that will compensate for any 'lost days'
- a refund element that will return the value of any giffgaff airtime credit spent during those lost days.
If you have any questions about compensation, and whether it applies to you please wait until 2nd September when the compensation file will have been applied - and at that point please contact an agent.
As we are now back to normal, this will be the final update on this thread and I will lock the thread but leave it up for reference.
Thanks once again to all those affected for your patience, and sorry this happened.
We have picked up a few issues with connected parts of our service today, namely activations, port ins and purchasing of goodybags. These all seem to be taking longer than normal, so we have escalated a technical investigation which has been given the highest priority. Normally these investigations resolve the issue within 4 hours.
Our technology and support teams are working flat out to make this happen. In the meantime please accept our apologies if you are affected.
At this point in time we don't have any detail on the nature of the underlying problem. When we do, I will update this thread.
Update 16.57 24th Aug:
We are getting hourly updates from the tech team - the latest is that it may be a firewall issue. As of now the issue remains in place and continues to mean that activations port ins and goodybag purchases are taking a long time to process.
If you are 'in progress' with one of these please be assured that your details will not be lost, however you may experience delays of several hours.
My advice at this stage is that if you can wait until this is resolved before trying to activate, port, or buy a goodybag then that is the simplest thing to do. However I do appreciate that this may be inconvenient.
The tech team now believe that they have isolated the underlying issue and resolved it. This is not yet confirmed as they are testing the fix as we speak. I have just personally tested purchasing a goodybag - which went through fine. I am also personally testing two SIMs for activation - both of which are showing 'in progress'. I will monitor these alongside the updates from the tech team.
There will be further updates once testing is confirmed. Once we are sure the fix is in, I will also update with detailed explanations of what (if anything) you need to do in each scenario... e.g. if your activation got hung up - whether this will go through automatically or whether you need to re-submit.
I will also request a detailed breakdown of what went wrong, though please be prepared that this may take some days to extract.
The fix has been applied. The top level analysis at the moment is that it was a problem with a firewall which has now been circumvented by using a different firewall. At this stage here are the scenarios we are looking at depending on your issue..please note that this is the best information I have available at this time, I cannot guarantee 100% that this won't change.
1) If your activation was delayed - you don't need to do anything, these will now start to 'unblock' and will go through the system one at a time. It will take some hours for this to complete, and I believe that the queue will start from the oldest first. It seems that the oldest time we have for this issue is around midday yesterday. There are approximately 350 activations in the queue right now.
These will continue to process until 10.00pm tonight when our activation back end goes offline as it always does. We cannot stop this being taken offline. However, if there are any activations from the 350 left in the queue at 10.00pm these will automatically start to be processed again when the system comes back online about 5.00am tomorrow morning.
2) If you were expecting to port in today - I will confirm this tomorrow, as we will need to talk to the central migration team about where these port requests got to before they got stuck. There are about 40 ports affected.
3) if you were expecting a queued goodybag to appear on your account - these scenarios can get complicated depending on whether the goodybag you queued is different to the one that was current, whether you tried to buy a new goodybag when the queued one didn't appear etc. For that reason we can't give generic advice, but will work through the different scenarios tomorrow. In the meantime some of you may see your queued goodybags appear overnight. There are 71 members affected by this issue. It may be that we need to deal one-to-one with members in these situations in order to properly fix the issue - including any refunds that would be applicable
I hope that is enough information for now. I realise that this has been a frustrating day for many of you, and I apologise for that. We will learn from this, and I will be organising a more detailed post on excatly what went wrong once this issue has been fully resolved.
Update 25th 09.14:
There are 30 activations remaining from the 350 that started to go through last night. These are being put through now manually.
We will update soon re. the situation with Port ins and queued goodybags
Port ins - the latest information is that these are starting to go through. At this stage it looks like most of the affected ports will process automatically and you will not need to re-submit.
Please note that the balance on your online account is the last part of the port process to complete - so once your phone is able to make and receive calls there will be a period of up to a few hours when the mygiffgaff details still show your old number and the balance is shown as NA or Not Available.
Here is what is happening with goodybag issues:
queued goodybag didn't begin when it should - these need to be fixed manually one by one on our back end system. We have a list of all affected members and are working through them one by one. It is a slow process as two separate fixes need to be applied to each account: 1) on the giffgaff website and 2) in the network. I will try to get an update of how many we have fixed so far. If this affects you you don't need to do anything, just keep checking your account and it will eventually update.
We will refund any spend you need to make from your credit during the time that the goodybag is not functioning when it should.
tried to buy a goodybag and couldn't - there shouldn't be very many, if any instances of this. But if this did happen then the thing to do is re-try the purchase and this time it should go through. You will nothave been charged for the previous attempt(s)
ports - update is that we continue to work through these and I will update at the end of the day if any remain. New port requests received today should not be affected.
We will be issuing a goodwill credit to everyone whose port was affected by our issue. Ammount TBC.
Delayed activations - as of this morning there were 30 activations remaining from the queue last night. These had falied to go through automatically for a variety of reasons and have been pushed through manually this morning.
What to do if your delayed activation still has not been processed by the end of today:
If, by 5.30pm today you have an activation from the last 2 days that has still not processed then please let me know by private message here on the forum. I can then follow these cases up directly with the technical team
Latest is that queued goodybags should now be active on your account - which means that the goodybag is working...even if the mygiffgaff section of our website does not show the latest goodybag. The site will catch up with the network later today.
We will be sending an SMS to all affected members informing them of this.
Responses from agents. If you have raised a question with an agent via the Ask an Agent link you may find that they are taking longer to get back to you than normal. More detail on why this is can be found here
For now, I have also taken the agents off the forum temporarily so that they can focus on processing the backlog of Ask an Agent questions.
In the meantime, while we apologise for not getting back to you personally in all cases, you can get up to date information here instead if you issue is one of those listed above.
Ports: All delayed ports should now have completed. If you were due to port in any day this week, including today, your port should now be working fine. All ports (59 in total) which were delayed due to our system fault have been compensated with an airtime credit of £15 which we feel reflects the seriousness of being without a mobile service.
Delayed activations: All delayed activations should now be working.
Queued goodybags: All queued goodybags which did not start on time should now be active (104 in total). We will compensate affected members but this will be worked out individually depending on usage and the duration of the goodybag not working.
update 26th Aug 7.45 am
Due to the events of the last two days there is a backlog of orders and activations now processing through the system. This means that 'new' activations i.e. those begun since yesterday evening will probably take longer than ususal to complete. I will work with the tech team to get a better estimate on exactly how long but my best guess at this stage is several hours.
In response to suggestions by members in this thread and elsewhere I am also getting a change made to the activation window so that new people activating will see a message warning them of delays before they enter the 6 digit code. Thanks for this suggestion.
We have now identified that one element of our IT network is processing our transactions very slowly and we have a back log in this area that we have been clearing since last night. 'Transactions' here means any of the back end processes that need to happen to complete a SIM order, an activation, a port in, or a purchase. And there is usually more than one transaction for each of these activities
Yesterday the total transaction queue was over 3,500 and at the moment it is around 1,800.
To minimise the amount going into the transaction queue we are implementing the following:
1) We are temporarily holding back all new SIM orders. This means that you can still order a SIM but we are quarantining the Order requests, and not progressing them - i.e. not sending them on for dispatch. We will do this for as little time as possible but at this stage this means that SIM orders made today are likely to take one day longer than normal to arrive
2) We are looking at ways to increase the throughput i.e. increase the speed with which the backlog of transactions gets processed
3) We are trying to prioritise the transactions and use manual intervention to speed up the most important transactions. In our eyes the single most important transactions are port in requests which have reached the stage where the member has lost service on their old network but has also has no service on our network.
4) We will temporarily stop taking new requests for Port Ins until we can guarantee a quicker and more reliable service. I am working to get the Port In form adapted so this is clear.
We are making progress with prioritising the urgent port cases, these are starting to complete and members are reporting services starting to work. The provision of service my be staggered i.e. you may find calls work, then text, then data.
If you are trying to buy a goodybag and the screen is freezing at 'continue' try this if you can: switch your browser to IE 8 or Firefox. We have reports from two members that this fixed the issue.
The backlog is now down to 1200 items
Update 27th August 9.45
- The backlog is now down to 728 items
- New orders are still being held back - but will be released later today - there are currently 498 of these SIM orders waiting to be despatched from our warehouse
- There are 127 Activations in the 'processing' state - this is down from 266 at 10.00pm last night
- I will do a separate update on Ports a little later
Overall things are getting better, today the tech teams will keep working on the immediate problem of the 127 activations and making sure that today's port ins are prioritised
Port in update 12.04:
Port ins (transferring your number to giffgaff) have 3 statuses:
1) requested - we have received your transfer request
2) confirmed - your other network has locked out your number (so you can't use it on their network) and we have begun the process of activating your number on our network (but it isn't active yet)
3) completed - your number is fully active on our network
Normally the whole process takes 2 working days, and the confirmed status stage - where you are without service on either network - is limited to a couple of hours.
This week our underlying problem with the firewall caused the 2nd stage, the confirmed status, to last much longer than it should. So that some members unfortunately were without service on any network for 30 hours or more.
Overall we have had 281 members with expected port in dates this week. So far 261 have entered the confirmed status, with the last 20 expected to enter that statsu today, in the next couple of hours.
There is no way to get a definitive confirmation of the completed statuses for those 261, i.e. we don't know for sure if all of them are now successfully ported in with a working giffgaff service.
However, from monitoring and responding on this thread, from the number of queries handled by agents, and the call backs that agents to to check newly ported in members we beleive that the vast majority of these ports are now, finally complete.
We are also, of course watching very closely through all parts of the system the 20 ports that are due to confirm and complete today - as the last thing we want is for these to get stuck in the confirmed status over the long bank holiday weekend.
The last issue connected to Port ins is this: if you requested a port in in the last few days we may have taken longer than normal to process your request due to the large volume of issues. That means that your port will not happen until Tuesday 31st August - however you won't be affected by the loss of service issue that affected ports this week. It is simply that we have taken a bit longer to start your process.
If you were due to port in this week any time up to yesterday - and your port has not completed, leaving you with no service, please reply on this thread or to an agent and your case will have the highest priority.
@kellybrowne, you appear to be in this bucket, I have escalated your case
If you are due to port in today - please watch this space, we are watching your port closely.
activations - there were 127 activations in the queue this morning, as of now there are 8 of these remaining. There are are further 50 activations which were started this afternoon which are still to go through.
ports - of the 281 ports due to complete this week 277 are now complete, only 4 remaining. These are being looked at individually
Whether you are activating or porting please remember that it may take a couple of hours for your mygiffgaff screen to fully update, and for you to have full service onyour phone.
We are picking up one or two isolated cases of goodybags reverting from 100 minutes to zero and these are also being investigated indivdually.
There are a handful of members now with remaining issues. A small team from giffgaff including myself and the CTO Nigel Suddell will continue to work on these tonight and into the weekend.
I will be offline for a few hours while I travel home. It is possible that these cases will be resolved during that time. I will update later this evening.
Thanks for your ongoing patience