HEXONET reachability issues
Incident Report for HEXONET
Postmortem

Statement for HEXONET reachability issues on Dec 06, 2023

Issue Summary

We had reachability issues affecting our HEXONET systems on Dec 06, 2023 between 2023-12-06 11:28 UTC until 11:54 UTC.

As a result our API gateways were unavailable during this timeframe. As a consequence, the HEXONET Control Panel was partially not working, as it relies on API connectivity.

Timeline and Detailed Description of Impact

11:28 – Availability of HEXONET API gateways was interrupted. Monitoring immediately showed services going offline and investigation started.

11:35 – First clients and internal users report unavailability.

11:42 – Investigations showed that isues were caused by a firewall failure. Mitigation already ongoing.

11:54 – Mitigations finished, all services went up again.

Root Cause

During a routine datacenter operation a firewall was powered off by mistake. Full redundancy has been restored within the next few hours after all services went up again at 11:54 UTC. Single items issues won't be able to impact operations going forward.

Impact on Security

No domain management was affected.
No data has been corrupted.
No impact on Security.

Follow Up Actions

Network configuration will be changed to reduce dependencies of different systems. Also, full redundancy of our firewall setup has been established to make sure singular failures will not have impacts on operation.

Posted Dec 07, 2023 - 13:16 UTC

Resolved
All systems are up and running again and we haven't monitored any issues since 11:54 UTC.

We will provide a Post Mortem shortly.
Posted Dec 06, 2023 - 16:55 UTC
Monitoring
Unfortunately we were experiencing reachability issues of our HEXONET API gateways on 2023-12-06 between 11:28 and 11:54 UTC.

Cause was a failed firewall which has now been replaced and all services are recovering and we are monitoring the situation.

We apologize for any inconvenience this may have caused.
Posted Dec 06, 2023 - 12:14 UTC
This incident affected: API interfaces (HTTPS API, EPP API, XRRP API, SOAP API, XMLRPC API).