Incident Report
Summary
The details of this report are pertaining to an incident affecting HEXONET customers that caused issues with domain contact management for specific TLD domains. For a subset of these domains, general domain management actions were affected also. A copy of this report has been sent, via email, to the affected customers.
Impact
The initial impact was on a subset of domains that were not manageable in HEXONET customer accounts. Following this, a further small subset of those received outbound transfer event notifications (note: at no point were domains transferred out of HEXONET).
During resolution steps, customers with specific TLD domains experienced errors when updating domain name contact handles before a final resolution of this incident was reached.
List of impacted TLDs can be found on this incident status page below.
During this incident, there has been no risk to the resolution, ownership, or lifecycle management (including renewals) of affected domain names.
Root Cause
During a routine configuration change of an internal system account, a human-driven step was not followed entirely according to process, causing a break between the HEXONET and registry system synchronisation.
During a re-synchronisation process, the HEXONET system created read-only AUTO contacts for select domains, which in turn caused customers to be unable to update their domain associated contacts.
Remediation & Prevention
Remediation steps taken during this incident:
Prevention steps will be actioned to increase level of detail in this process documentation. Additionally, future such changes will require added review steps to sign off. Future scope to automate this process will be investigated.
Timeline
31 Jan 13:00 UTC
As part of an internal system account configuration change, domains were moved between two internal system accounts.
A subset of domains were not moved to the new account. Domains that were not moved could no longer be managed through customer HEXONET accounts.
31 Jan
First customer contact notification of issue received by HEXONET Support.
1 Feb
Following validation of the customer report, internal investigations into the issue began.
5 Feb
Investigations escalated to Engineering and issue identified as caused by steps to change internal account configuration.
6 Feb
Synchronisation of domains initiated between HEXONET and Registry system to resolve management issue. Synchronisation completed but led to the creation of AUTO contact handles on all specific TLD domains as a result. This sent out the events ‘UPDATE=OK’ to affected customers.
These AUTO handles caused the HEXONET system to no longer be able to manage domain contacts within the Registry System.
Initial mechanisms to reassign original contacts failed.
7 Feb
Identified original domain management issue as being linked to specific statuses of domains.
8 Feb
Management issue domains transfer completed to updated internal system account. Management capability reinstated for these domains.
Contact data issue escalated internally – status page created for incident. https://status.hexonet.net/incidents/gx124dcdbm2n
9 Feb
Customer incident notification sent to all affected Resellers. Hourly status updates through status page.
9 Feb
Identified database tables needed for restore of contact data. Started restore process.
9 Feb
Prepared script to restore original contact handles from the database backup. Executed script for all affected domains.
9 Feb
Validated contacts updated on domains and AUTO handles removed. Contact management capability restored at HEXONET.
9 Feb 17:19 UTC
Incident confirmed resolved via email and status page.
As always, we remain dedicated to your success and hope this background information supports your understanding for all that transpired. If you have any questions or would like to connect with our team, please always contact us at help@hexonet.support .
Your HEXONET Team