Service Issue
Incident Report for NetDocuments US
Postmortem

On April 13th at approximately 11:18 pm MDT, NetDocuments experienced a disruption with the US Service.   During the event customers would have been unable to authenticate to the Service or unable to reach the login page altogether.

We were alerted to the start of the issue via our automated monitoring infrastructure and were able to begin investigating immediately.  By approximately 3:20 am MDT on April 14th the Service had been stabilized and operations had returned to normal.

An investigation found the root cause was related to a software defect in the recently deployed 23.7 version of the Service affecting the retrieval of cached information.   Once the cause was identified, steps were taken to roll the Service back to the previous version of 23.6 in order to stabilize the Service and allow for the defect to be corrected before the new version is deployed again.

As a part of ongoing and continuous improvement, NetDocuments is evaluating and adding additional test cases to enhance the Beta and Deployment processes to prevent a similar defect from being able to affect the Service in this way again.

We apologize for any inconvenience this disruption may have caused.

Thank you,

The NetDocuments Team

Posted Apr 17, 2023 - 19:37 EDT

Resolved
This incident is now resolved. We will conduct a postmortem of this event, which will be posted within 2 business days.

If you continue to experience these issues, please open a ticket with our Support team for further assistance.
Log into support.netdocuments.com to submit a support request for your issue.
However, if you need to urgently contact Technical Support via the phone use the following information:
US customers call 801-226-6882 or 866-NETDOCS (638-3627)
EU customers call +44(0) 2034.556770
Posted Apr 14, 2023 - 05:59 EDT
Monitoring
Our engineering team has implemented a fix to resolve the issue with logging into the Netdocuments service and are monitoring. If you continue to experience these issues, please open a ticket with our Support team for further assistance. We will conduct a postmortem of this event, which will be posted within 2 business days.

We prefer that you go to support.netdocuments.com to submit a support request for your issue, so we can manage it more efficiently and provide you with access to your cases.
However, if you need to contact Technical Support via the phone use the following information:
US customers call 801-226-6882 or 866-NETDOCS (638-3627)
EU customers call +44(0) 2034.556770
Posted Apr 14, 2023 - 05:13 EDT
Update
We are continuing to work on a fix for this issue. We will provide a further update in 15 mins.
Posted Apr 14, 2023 - 04:38 EDT
Identified
Our engineering team has identified the issue causing users to be unable to login to the service. We are currently doing work to help mitigate the issue. We will provide a further update in 15 mins.
Posted Apr 14, 2023 - 04:18 EDT
Update
Our engineering team continue to investigate this service outage. We are actively working to mitigate this issue and will provide a further update in 15 minutes.
Posted Apr 14, 2023 - 03:55 EDT
Update
Our engineering team continue to investigate this service outage. We are actively working to mitigate this issue and will provide a further update in 15 minutes.
Posted Apr 14, 2023 - 03:36 EDT
Update
Our engineering team continue to investigate this service outage. We are actively working to mitigate this issue and will provide a further update in 15 minutes.
Posted Apr 14, 2023 - 03:20 EDT
Update
Our engineering team continue to investigate this service outage. We are actively working to mitigate this issue and will provide a further update in 15 minutes.
Posted Apr 14, 2023 - 03:02 EDT
Update
Our engineering team continue to investigate the service outage. We are actively working to mitigate this issue and will provide a further update in 15 minutes.
Posted Apr 14, 2023 - 02:39 EDT
Update
Our engineering team continue to investigate the service outage. We are actively working to mitigate this issue and will provide a further update in 15 minutes.
Posted Apr 14, 2023 - 02:16 EDT
Investigating
Our engineering team is investigating reports of inability to access the service. We apologize for the inconvenience, and will provide updates as we proceed when we have more information.
Posted Apr 14, 2023 - 01:59 EDT
This incident affected: Platform.