Service Issue - Unable to Save or Create Documents

Incident Report for NetDocuments US

Postmortem

At approximately 8:45 PM MDT, October 14, 2019, the US instance of NetDocuments incurred a hardware failure on a specific node responsible for entropic encryption. Under typical circumstances, the system would revert to a secondary node in the event of a failure, thus rendering the issue transparent to all users. In this instance, the secondary node incurred an issue of its own and failed to send the proper notifications to NetDocuments personnel. During the incident, customers would not have been able to upload new documents to the Service.

We are working to investigate and identify the root cause of the hardware and notification failures and will build measures to safeguard against this type of issue in the future. We apologize for any inconvenience this caused our Customers.

If you have any questions or are experiencing issues related to this incident, please contact NetDocuments Technical Support.

US customers call 801-226-6882 or 866-NETDOCS (638-3627)
EU customers call +44(0) 2034.556770
AU customers call +61 2 8310 4319

Posted Oct 15, 2019 - 18:24 EDT

Resolved

This incident is now resolved. A postmortem (Reason For Event) will follow within the next 24-48 hours. We apologize for any inconvenience this may have caused.

NetDocuments Team

Posted Oct 15, 2019 - 01:30 EDT

Monitoring

A fix has been implemented and the Service has been functional for the past 30 minutes. We will monitor the Service for the next 30 minutes. If no additional impact is observed, we will resolve this issue.

We will provide a final update in 30 minutes unless new information is available.

NetDocuments Team

Posted Oct 15, 2019 - 01:00 EDT

Update

Our engineering and network operations teams have identified the issue and are continuing efforts to restore full functionality to the Service.

We will provide an additional update in 30 minutes unless new information is available, at which time, we will provide another update.

NetDocuments Team

Posted Oct 15, 2019 - 00:50 EDT

Identified

Posted Oct 15, 2019 - 00:20 EDT

Update

Our engineering and network operations teams are continuing troubleshooting efforts to restore full functionality to the Service.

We will provide an additional update in 30 minutes unless new information is available, at which time, we will provide another update.

NetDocuments Team

Posted Oct 15, 2019 - 00:15 EDT

Update

Our engineering and network operations teams are continuing efforts to restore full functionality to the Service.

We will provide an additional update in 15 minutes.

NetDocuments Team

Posted Oct 15, 2019 - 00:01 EDT

Update

Our engineering and network operations teams are continuing efforts to restore full functionality to the Service.

We will provide an additional update in 15 minutes.

NetDocuments Team

Posted Oct 14, 2019 - 23:44 EDT

Update

We continue to investigate this issue. We will provide an additional update in approximately 15 minutes.

NetDocuments Team

Posted Oct 14, 2019 - 23:27 EDT

Investigating

We are aware of some customers who may be experiencing slowness or the inability to save documents. We are actively working to rectify this issue. Please be aware that users may experience some slowness while we work to mitigate any impact.

We will provide an update in approximately 15 min.

If you are concerned that you are affected by this please contact technical support via the support portal https://support.netdocuments.com or via the phone.

AU customers call +61 2 8310 4319
EU customers call +44 2034.556770
US customers call 801-226-6882 or 866-NETDOCS (638-3627)

We apologize for any inconvenience.

NetDocuments Team

Posted Oct 14, 2019 - 23:09 EDT

This incident affected: Platform.