Operational updates and ongoing infrastructure improvements

Operational updates and ongoing infrastructure improvements

by handy giff-staffer nigel on ‎02-11-2012 15:13 - last edited on ‎20-11-2012 21:25 by ayla_15

Hi,

 

For a while now, there has been a growing demand for the technical teams in giffgaff to share more about how the service is delivered and provide more details on the issues that members experience.

 

Overview
We’ve uploaded a schematic of the core architecture and a brief description of what each area does.
At a high level the architecture is split into two areas: Portal and Network.

The “Portal” is the giffgaff website and is represented by the orange / yellow group of servers. These are load balanced, web and application servers linked to a local database. The application here manages a number of functions
- log in and site registration
- end to end SIM orders and activations
- provisioning SIMs to the network
- managing single top ups and goodybag purchases
- integrating the Community and Help sections
- providing management information on members and financials
The “portal” is not involved with actual calls, texts or data, but the account will need to have credit and goodybags set up through the portal.

The “Network” provides the service from your handset – calls, texts, data. This is split across two different areas, the grey area which is the “Prepaid” architecture and the pink area which is the “Core” architecture.
The “Core” architecture is very similar for all prepay and pay monthly services, this manages your SIMs as you move around the country (and abroad) providing coverage.
The “prepaid” architecture is specific to your SIM and account, this is where your balance is held, where the details of what you have left in your goodybag is managed and where things such as auto top up and recurring goodybags are applied.

Overview.jpg

Issues
Looking at the issues that we have had over the last 6 months, I have picked out the top 4 (full service, data outage, SIMs not provisioned and goodybags) and highlighted where these issues were on the architecture.

I have shown the elements which caused the network service outages in the last 9 months, across both the core and prepaid areas.

 

network_issues.jpg

 

For SIM provisioning; during 2012 we have changed the way that we manage SIMs. At the moment once an order is placed and dispatched, giffgaff then update the network and service architecture to get the details set up ready for activation, this matches the orderer with the SIM so that we can track back to your order.

When there are file or connection issues or delays in updating all the systems then SIMs can get to members without being provisioned.

 

SIM_provisioning_issues.jpg 

For goodybags, the core information on each members account is held in the Prepaid service, but the application of these is mainly managed by the Portal (including taking money, managing goodybag changes and update, audit controls and reconciliation). However when recurring or queued goodybags are applied then this is managed by the Prepaid system and confirmation sent to the Portal.
If these systems are out of sync then the Portal may not show a goodybag, or may not allow another goodybag to be purchased.

 

goodybag_issues.jpg 

 

Improvements
There are two levels of improvement are well under way – capital investment and operational improvements
Capital investment is happening to manage the capacity growth and the resilience of the end to end service. The network is receiving £10M to address the total service outages in July and October, giffgaff are investing £9M to make the Prepaid solution more resilient, better supported and correctly sized.
giffgaff have also invested in additional hardware for the web servers in the hosting environment, additional servers, memory, disc etc.

Operational improvements have focussed on the end to end processes, specifically for SIM provisioning and goodybags, where multiple alerts and reconciliations are now generated and checked each day. This allows us to react more quickly and also identify the repeat causes that we can then fix. We already have some fixes to improve the code process to release in November and December and will be prioritising any more that we identify to be delivered in Q1.

 

blog_Investment.jpg

Future
Moving forward, we would like to share a monthly operational summary to cover
- major operational issues in the last month
- service levels (a traffic light or targeted system) against key services like web site available, voice / data / texts service, top up service.

 

Follow this link to see the full giffgaff Operations presentation.

 

Comments
by jazzyk on ‎02-11-2012 15:18

Thanks. Monthly summary sounds great

by giffgaff head-scratcher darrenpainter on ‎02-11-2012 15:22

Thanks Nigel, it really does help to see where the split of the issues are occurring, i.e core network (o2) versus MVNE side

Obviously you have more oportunity to influence the stability of the MVNE side rather than the parent network infrastructure

by jburnham96 on ‎02-11-2012 15:27

£1.5M a day to expand the data network? Wow!

by giffgaff head-scratcher darrenpainter on ‎02-11-2012 15:28
by jburnham96 on 02-11-2012 15:27

£1.5M a day to expand the data network? Wow!

--------------------------------------------------------------------------------

That isn't coming out of giffgaff's direct budget though....

Telefonica has deep pockets and would be funding that side of things to benefit o2, giffgaff and Tesco Mobile

by nosh on ‎02-11-2012 15:29
thanks :smileyhappy:
by pedromap ‎02-11-2012 15:40 - edited ‎02-11-2012 15:41

now that you know where it did all went pearshaped,i await a improvement eagerly

by khairul on ‎02-11-2012 15:42

I have a parody in mind for some of the graphics ... I'll make one and post it if I get the time :smileytongue:

 

Nice post, Nigel; good to know :smileyhappy:

 

Where you say "The network is receiving £10M to address the total service outages in July and October" do you mean O2 is giving you that money to spend, or O2 are addressing the issue themselves?

by khairul on ‎02-11-2012 15:43

Also, can we have something like this regarding the SIM delivery issues?

by kelvinlewis on ‎02-11-2012 15:45

The future's bright, the future's giffgaff :smileyhappy:

by giffgaff head-scratcher darrenpainter ‎02-11-2012 15:45 - edited ‎02-11-2012 15:46

khairul, it is o2 directly spending £10 million to replace the Ericsson supplied Customer User Database component that failed so spectaculaly in October and the previous outage

http://www.lightreading.com/document.asp?doc_id=226073

Looking for something?

Join giffgaff
  • For only £10 a month:

    • 500 minutes
    • Unlimited texts
    • 1 GB Internet
    • Free calls to giffgaff