Between 12:02 and 12:25 on the 5th October the core network at the Datanet Farnborough Data Center suffered from a stability issue causing packet loss and interruption to connectivity for some customers.
The source of these issues was isolated to specific ports which were disabled allowing the network to return to stability.
We continue to investigate the cause but do not expect to see any further issues. Datanet apologises for the inconvenience suffered during this incident.
A significant part of the issue was that a customer generated (reflected back to us) a massive number of spurious VRRP packets on the network (over 3 million per second). This caused the Juniper DDoS protection filter to drop VRRP packets across all interfaces on the router which broke the default gateways for our customers.
Having investigated in detail we identified an additional feature within the Juniper DDoS protection which allows detection, logging and automated blocking at a Flow level. This means that rather than breaking VRRP at the router level, we now drop packets at a customer (ie VLAN) level.
We have tested and enabled this feature on both routers in Ark.
