QLD nbn Internet Outage

Incident Summary

At approximately 8 AM on 25/05/23, the main backhaul fibre providing nbn internet services to Queensland was disconnected due to physical damage caused to the cable. Additionally, 4G Mobile Broadband services experienced packet loss due to congestion caused by the outage of Queensland nbn Internet services.

Incident Details

The issue was first detected by URL Networks monitoring on 25/05/23 at 8 AM. Upon receiving this alert the issue was investigated and it was discovered that all QLD NBN services were impacted.

The issue was then raised with the upstream carrier, who was able to confirm that the outage was occurring in the upstream network and was impacting all Queensland-based nbn Internet services.

At approximately 8:30 AM the upstream provider advised of a suspected break in their primary fibre backhaul link via the Optus Fibre network and engineers were being deployed.

At 9:30 AM we detected that 4G mobile broadband services were experiencing packet loss. This was quickly investigated and the issue was found to be external to our network. Our upstream provider was notified and confirmed that the network was suffering from congestion due to the huge increase in traffic caused by the outage of Queensland nbn Internet connections.

At 11:43 AM the upstream carrier advised that a section of their primary aerial fibre link experienced a break across a major road and that their civil crew were working to run temporary cabling to allow splicing to repair the cable.

At 1:52 PM the upstream carrier advised that the onsite civil crew had encountered roadblocks making repairs due to the location of the break and the length of cable (100M) that needed to be routed across a major road. Further traffic control was required before they were able to proceed.

At 2:14 PM we saw the majority of Queensland nbn Internet connections restored and customers back online. At this time the packet loss being experienced on 4G mobile broadband services stopped and the connections stabilised and were functioning normally.

At 5:06 PM the upstream carrier confirmed that onsite splicing of replacement aerial fibre cable had been completed and service had been restored.

Resolution and Preventative Steps

Once the issue was located corrective steps were taken by the fibre team and services were restored at approximately 2:14 PM AEST on 25/05/23.

Unfortunately with internet services, outages such as this can happen occasionally as fibre cables can become severed due to outside construction works or even vehicles colliding with aerial cabling. Therefore to minimise future disruption we will be doing the following:

1. Working with the upstream carrier to determine whether redundant links are in place (or the possibility of them being installed) to provide redundancy so that in this situation, traffic could be routed via the redundant fibre, minimising any impact to services.
2. Working with the upstream carrier to determine whether additional capacity can be provisioned/allocated as well as traffic shaping/policing can be deployed on the 4G mobile broadband network to minimise congestion of services in the case of a widespread outage.
3. Reviewing our policies and procedures to determine whether the situation could have been handled better and communicated by our team.