We had reachability issues affecting our HEXONET systems on Dec 06, 2023 between 2023-12-06 11:28 UTC until 11:54 UTC.
As a result our API gateways were unavailable during this timeframe. As a consequence, the HEXONET Control Panel was partially not working, as it relies on API connectivity.
11:28 – Availability of HEXONET API gateways was interrupted. Monitoring immediately showed services going offline and investigation started.
11:35 – First clients and internal users report unavailability.
11:42 – Investigations showed that isues were caused by a firewall failure. Mitigation already ongoing.
11:54 – Mitigations finished, all services went up again.
During a routine datacenter operation a firewall was powered off by mistake. Full redundancy has been restored within the next few hours after all services went up again at 11:54 UTC. Single items issues won't be able to impact operations going forward.
No domain management was affected.
No data has been corrupted.
No impact on Security.
Network configuration will be changed to reduce dependencies of different systems. Also, full redundancy of our firewall setup has been established to make sure singular failures will not have impacts on operation.