Affected Service: All connectivity Description: Network Outage Johannesburg and Cape Town Data Centres Impact: No connectivity Start Time: 18 Feb 2021 07H45 End Time: 18 Feb 2021 08H33 At 07H45 our network became unresponsive. Our engineers could not access our datacentre environments remotely.
Engineers were immediately dispatched to the datacentres and arrived at 8H20.
After analysing the issue at hand, we picked up that there was a traffic storm coming from a partner network. This was amplified and flooding our network and causing it to be unresponsive.
We then shut down the interconnections between our networks in Durban, Cape Town and Johannesburg at 08H30. At this stage, our network started to stabilise and respond as normal.
The Cisco broadcast and multicast storm mitigation measures implemented last month when a similar incident occurred involving the same partner failed to prevent the outage. Our engineers are investigating why the measures failed.
Our engineers have moved the interconnection to the partner from the core switches at the data centres to router ports to ensure that if we experience such traffic from the partner network again this traffic is blocked off from affecting the core switching environment.
We have not yet received an incident report from the partner as to the root cause.
We apologise for the inconvenience caused and reiterate we are committed to offering the best service possible.
Thank you for your understanding.