Degraded Performance
Incident Report for NetDocuments US
Postmortem

On September 15th at approximately 8:40 am MDT, NetDocuments experienced a disruption with the US Service.   During the event customers would have experienced slowness in downloading and opening documents or in some instances would have received an error from ndOffice when attempting to perform this action.

We were initially alerted to the start of the issue via our automated monitoring infrastructure and were able to begin investigating immediately.  By approximately 3:09 pm MDT the Service had been stabilized and download speeds had returned to more normal levels.  Additional performance tuning continued until 6:00 pm MDT.

An investigation found the root cause was related to a change in the recently deployed 23.18 version of the Service dealing with the method used to connect to an analytics logging function.  The change introduced delays in the logging call that affected the download transactions.  This resulted in the slowness experienced for customers on the US Service.   Once the problem was fully identified the Service was rolled back to the previously stable 23.17 version and performance tuning was implemented on the logging function.  An updated fix has since been implemented in 23.18 and the new version has been redeployed.

As a part of ongoing and continuous improvement, NetDocuments is evaluating and adding additional test cases to enhance the Beta and Deployment processes to prevent a similar defect from being able to affect the Service in this way again.

 

We apologize for any inconvenience this disruption may have caused.

 

Thank you,

The NetDocuments Team

Posted Sep 21, 2023 - 14:41 EDT

Resolved
The incident in the US Service is now resolved. We are continuing to monitor the situation closely and will update this post again if any reoccurrences of the problem are found.

If you continue to experience these issues, please open a ticket with our Support team for further assistance.

Log into support.netdocuments.com to submit a support request for your issue.

However, if you need to urgently contact Technical Support via the phone use the number closest to your region:
US customers call 801-226-6882 or 866-NETDOCS (638-3627)
EU customers call +44(0) 2034.556770
AU customers call +61 2 8310 4319
Posted Sep 15, 2023 - 20:39 EDT
Update
We have identified the issue with performance in the US service region. We are implementing changes to correct the issue and the Service may be slower than normal and have minor connection issues over the next 2 hours.

We will post an update once the solution has been implemented and the Service is stable.
Posted Sep 15, 2023 - 15:43 EDT
Identified
NetDocuments has identified the cause of the performance issue within the US Service. We are working to implement a fix that should resolve this issue.

We will continue to provide updates to this post as the issue is diagnosed and resolved.

If you are experiencing difficulty, please contact support via the portal or by phone.

https://support.netdocuments.com/

AU customers call +61 2 8310 4319
EU customers call +44 2034.556770
US customers call 801-226-6882 or 866-NETDOCS (638-3627)
Posted Sep 15, 2023 - 15:19 EDT
Update
NetDocuments continues to investigate a potential latency issue in the US Service.
We will continue to provide updates to this post as the issue is diagnosed and resolved.

If you are experiencing difficulty, please contact support via the portal or by phone.

https://support.netdocuments.com/

US customers call 801-226-6882 or 866-NETDOCS (638-3627)
AU customers call +61 2 8310 4319
EU customers call +44 2034.556770
Posted Sep 15, 2023 - 14:50 EDT
Investigating
We have found a symptom of the potential latency issue persists, we are continuing with our investigation.

We will continue to provide updates to this post as the issue is diagnosed and resolved.

If you are experiencing difficulty, please contact support via the portal or by phone.

https://support.netdocuments.com/

US customers call 801-226-6882 or 866-NETDOCS (638-3627)
AU customers call +61 2 8310 4319
EU customers call +44 2034.556770
Posted Sep 15, 2023 - 13:57 EDT
Monitoring
The service has stabilized and operations have returned to normal performance. Our engineering team is monitoring.

If you continue to experience these issues, please open a ticket with our Support team for further assistance.

Log into support.netdocuments.com to submit a support request for your issue.

However, if you need to urgently contact Technical Support via the phone use the number closest to your region:
US customers call 801-226-6882 or 866-NETDOCS (638-3627)
EU customers call +44(0) 2034.556770
AU customers call +61 2 8310 4319
Posted Sep 15, 2023 - 12:53 EDT
Update
We are still working to diagnose and resolve the latency issue some customers are currently experiencing with the US Service.

We will continue to provide updates to this post as the issue is diagnosed and resolved.

If you are experiencing difficulty, please contact support via the portal or by phone.

https://support.netdocuments.com/

US customers call 801-226-6882 or 866-NETDOCS (638-3627)
AU customers call +61 2 8310 4319
EU customers call +44 2034.556770
Posted Sep 15, 2023 - 12:08 EDT
Investigating
NetDocuments is investigating a potential latency issue in the US Service that could be affecting some customers.

We will continue to provide updates to this post as the issue is diagnosed and resolved.

If you are experiencing difficulty, please contact support via the portal or by phone.

https://support.netdocuments.com/

US customers call 801-226-6882 or 866-NETDOCS (638-3627)
AU customers call +61 2 8310 4319
EU customers call +44 2034.556770
Posted Sep 15, 2023 - 11:25 EDT
This incident affected: Platform and Subsystem (Search).