Inside the pressure cooker at Safaricom’s Network Centre

Many Safaricom customers may not know that before the company moved to the well-known Safaricom house, it was housed in Westlands. The former headquarters remains the heart of Safaricom’s network, and it has not stopped beating even during the pandemic.

08 Jul 2020 . 14,094 Views

If you hop into a taxi in Nairobi’s city centre and ask the driver to drop you at Safaricom House, and then fall asleep, it is highly likely he will wake you up when you get to the entrance to the twin towers on Waiyaki Way known to insiders at Safaricom as HQ1 and HQ2.

That’s where most visitors end up because that’s where the Chief Executive Officer, the senior leadership and many of the staff sit.

Further down the road when you are in-bound to the city centre is the Safaricom Care Centre.

Set back from the road, and shaded by trees, the nine-floor building was the second home of the telecommunication company when it moved from Extelcoms House on Haile Selassie Avenue. It is known by the staff as SCC.

The fourth floor here has the air of a mission control room seen in Hollywood movies.

In one of the rooms with ‘Restricted Access’ and ‘Network Operations Centre’ on the door, there are four monitors to each desk in a crescent. On the wall, there are large TV screens.

Some have a live map of Kenya in green, with parts in yellow, and a key in the corner that tells you that things are getting bad if the yellow turns to orange and if the yellow turns to red. Some have lines on a graph. Some have figures that blink constantly, and others have large tables that show that things are good when they are green and there will be trouble if they turn red.

“I don’t know how many people have visited this place, but there are probably people in this company that don’t know that it exists. People who work for other divisions may never know that there is a command centre for the network,” says Meshack Kiminza, a Manager at the Network Operations Centre (NOC).

Meshack sits at the back of the room, with a clear view of all the big screens, and on his own computer, a dashboard on which he can see everything that is happening on the network.

As the team leader on one of the shifts, he needs to have a view of everything.

From all the systems that are visible at the NOC, Meshack is confident about the team’s capability: “By the time you notice that something is not working on the Safaricom network, there is already someone at NOC who has already picked it and it’s being worked on.”

Over the nine years he has worked at the NOC, Meshack has arrived at the perfect illustration of what the team that he leads does: the room in the White House where the President and senior members of his team monitor, live, operations by their military.

Photos from the Situation Room often feature the President and his team staring at screens on which they can see videos and data.

Like the team in the Situation Room, the team that Meshack oversees spends the day staring at screens, monitoring the entire Safaricom network.

They can see, on the giant screens and their individual monitors, everything that is happening on every service that the telecommunication company offers. There is also an ordinary TV screen showing what is on TV, which is often the updates on the news or the latest football action.

Technically, that room is known as the Front Office while the Back Office consists of the engineers behind the network, who sit in another room.

The Front Office team will make routine checks and calls, using an Incident Management Software to identify and get faults solved. Incidents are marked, in order of their severity, as major, critical, or crisis.

Major and critical incidents are handled by the team at NOC, but when it gets to crisis mode, everyone needs to know.

Once in a while, a crisis will inevitably strike and when it does, says Meshack, it gets very busy, and things get very hot.

Calls have to be made to the engineers and technicians who repair broken transmissions masts and cut cables, or the software engineers who have to figure out codes; calls have to be made to the senior colleagues up the road who run the departments and ultimately, the CEO will organise for the customers and the public to know that work is underway to restore services.

Inside the pressure cooker at Safaricom’s Network Centre

“It’s a pressure cooker environment,” said Meshack. A crisis requires military precision, and the core team managing one will usually go into a War Room – a virtual space where they can track and trace everything being done to restore services.

Recently, there was a routine power outage at one of the switching stations in Nairobi and the back-up generators failed to kick in automatically because of a fault in their fuel system.

The switching rooms and the data centre host voice, data and value-added services – NOC parlance for M-PESA and other services –  and there would have been an outage, and the accompanying outrage from customers.

“As it is in most of the cases,” Meshack explained, “NOC picked the alarms immediately it happened and activated all processes and resources to avert a crisis.” ‘

They scrambled a team to diagnose the root cause, physically provided resources to help get fuel to the generators and assisted in the transfer of services to other equipment to avert a large-scale disaster if the worst were to happen.

Because of the amount of coordination required, the team realised early on that face-to-face interaction is necessary.

When coronavirus reached our shores and brought with it the need for social distancing and working from home where possible to limit the spread of the novel virus, they tried working from home and found that it was no good.

For a start, it would require that they hold a 12-hour meeting on Microsoft Teams, which was not practical, and the usual vagaries of teleconferencing would often get in the way of work.

At the end, it was decided that only the Back Office could work from home.

The Front Office team settled on having four shifts of 16 people working 12 hours per shift.

With the Back Office team working from home, the 16 Front Office team per shift was able to spread out in the building and practice social distancing.

Each team must hand over to the next shift and they avoid being in close proximity by leaving 10 minutes to the end of their shift and handing over virtually and then following it up with a call instead of the physical meetings they were used to.

Collaboration is kept up by having meetings on Microsoft Teams.

It might sound all serious and ordered, and it is, because they are involved in serious work, but it’s not all work.

On normal days, what some have taken to calling ‘Before Covid’, the teams would take five minutes’ breaks to stretch led by the in-house champion trained by the Human Resource Department.

“Being a pressure cooker environment, if you don’t consider the mental wellbeing of the staff, then it can easily trip and become worse. Consciously we take care of that,” said Meshack.

The entire team at NOC has been together so long they have grown into a very social group. When he had a new-born, before Covid-19, Meshack recalled, 20 of his colleagues from SCC visited him at home to see the newest member of his family.

So keen are they on this side of their working life that they have set up a separate virtual meeting room dedicated entirely to catching up with each other.

“Out of that unofficial interaction, we build a lot of bonds. It may look like it’s not by design, but it is by design. We try to make it as unofficial as possible and that and that works to our advantage if there is a major incident and we need to collaborate more closely under intense pressure,” he said.

At a personal level, Meshack decompresses by cycling and swimming – to keep fit and sane, he says – as well as walking and going camping. The father of two daughters likes to connect with Mother Earth and appreciate what it offers us.

Working with the Safaricom Foundation, he has also helped organise for the repair of dilapidated classrooms in a needy area of Makueni County.

Covid-19 has given Meshack and the team impetus to fast-track a project they were working on to automate every function that can be done by machines.

“We have had to figure out how to be future-proof,” he added.

That will be done via the Next Generation NOC project, which he says will revolutionise the NOC and align it with the future needs of Safaricom and customers by refining the tools and processes through automation. NOC is also applying lessons learnt from best practice in the industry worldwide.

Nobody really knows whether we will ever go back to living and working like we did before Covid-19, but Meshack can tell that face-to-face interactions will reduce, and the team will need to have a better way to working away from the premises.

Still, there will always be a team with its eyes on the network, and the NOC will be that blinking beacon that never loses its bright light.

 

 

Podcasts

Social Feed

  • Safaricom Newsroom @SafaricomNews
    RT @SafaricomPLC: @PeterNdegwa_ Over the last 20 years, we've leveraged the purpose-driven business model & mobile technology's power to Tr…
    #SafaricomNews
  • Safaricom Newsroom @SafaricomNews
    RT @SafaricomPLC: Celebrating Safaricom at 20 #TwendeTukiuke https://t.co/q6jbdivVso
    #SafaricomNews
  • Safaricom Newsroom @SafaricomNews
    RT @SafaricomPLC: Our senior leadership team at the launch of our 20th Anniversary. We're excited to be marking this special milestone toda…
    #SafaricomNews