The screw that took out the Internet

The internet seemed a bit slow this evening.

There was no sudden loss of connectivity and nothing obviously broken. Pages were loading, things were still working, but the connection just didn't feel quite right.

I opened the UniFi app to see what was going on and immediately spotted something odd. Vodafone, my primary internet connection, was nowhere to be seen. The UDM Pro was running entirely on the secondary WAN connection, which in my case is Starlink.

That was slightly surprising, because I hadn't actually noticed the Vodafone connection going down.

Finding the fault

My Vodafone connection reaches the network rack in the garage via Ethernet from the house. I recently had some work done in one of the rooms along that route, so my first suspicion was the physical cable path.

After a bit of cable swapping and rerouting between the house and garage, Vodafone came straight back up.

That narrowed things down considerably.

The original Cat5 Ethernet cable had been routed beneath the skirting board 25 years ago by me in the room where carpet had recently been fitted. While the carpet fitters were working, they had asked me about using some fairly long screws.

At the time I didn't think much of it.

In hindsight, there is a very good chance one of those screws found the network cable.

I haven't pulled the skirting apart to prove it and probably won't bother. The replacement route works and running another cable is easier than investigating a cable that I already know I no longer trust.

C'est la vie.

Starlink had been doing all the work

Once Vodafone was working again, I opened the Starlink app.

That explained why I had finally noticed something was wrong.

Starlink was still online and working normally as far as the hardware was concerned, but the account had used all of its high-speed data allowance. The service had therefore been throttled.

That is perfectly fine for this setup. Starlink is there primarily as a backup connection rather than as my normal broadband service.

What was more interesting was that the backup had apparently been carrying the entire network for long enough to exhaust its high-speed allowance without anyone in the house noticing.

That made me curious about exactly how long Vodafone had actually been down.

Checking the UniFi logs

The UniFi event history provided the answer.

At 12:13:09 on 9 November, UniFi recorded packet loss on the Vodafone WAN connection.

At 12:14:38, WAN1 was declared down and the UDM Pro switched over to WAN2, my Starlink connection.

Vodafone did not return until 20:36 on 10 November.

That means the network had been running on Starlink for roughly 32 hours and 22 minutes.

Interestingly, UniFi did not record a clean Ethernet link-down event on the WAN port. It simply saw increasing packet loss and eventually loss of usable internet connectivity.

That actually fits reasonably well with the suspected cable damage. A cable doesn't necessarily have to be completely severed to cause trouble. Damage to one or more conductors could leave the Ethernet link appearing to exist while producing enough errors and packet loss to make the connection unusable.

The failover worked

The main takeaway from all of this is that the backup system worked rather better than I realised.

The UDM Pro detected the Vodafone failure and switched automatically to Starlink. Nobody had to intervene and the network stayed online.

More importantly, the failure went unnoticed for over a day.

It was only when Starlink had consumed its high-speed data allocation and dropped to a reduced speed that the difference became obvious enough for me to investigate.

That is a fairly good real-world test of the setup, although it wasn't one I had planned.

The Starlink app showed the dish as online and the router correctly operating in bypass mode. The only warning was that the high-speed data allowance had been exhausted. There was no indication of a dish fault, obstruction or Starlink hardware problem.

Once the Vodafone cable route was changed, the UDM Pro restored WAN1 and normal service resumed.

Something else to monitor

The experience has highlighted one weakness in the current setup.

UniFi handled the outage exactly as intended, but it handled it so well that I didn't know the primary connection had failed.

For a short outage that is ideal. For an outage lasting more than a day, I would prefer to know about it before the backup connection starts burning through its data allowance.

The next job will therefore be to add some monitoring and reporting around WAN failover.

At a minimum, I want an alert if Starlink becomes the active WAN connection for more than a short period. I will probably also log primary and backup WAN status into the existing Home Assistant, Node-RED, InfluxDB and Grafana setup so that failovers can be graphed and reviewed afterwards.

It would also be useful to report how long the network has been running on backup and perhaps generate a notification if the primary connection has remained unavailable for 15 or 30 minutes.

That should mean the next time Vodafone disappears, UniFi can still keep everything running while making sure I actually know it has happened.