Intermittent Performance Issue

Incident Report for NetDocuments US

Postmortem

At approximately 09:30 EDT NetDocuments platform monitoring began reporting congestion with a database sub-system. According to operating procedures, the impacted databases were failed over to alternate servers. This process took about 10 minutes to complete and by 09:45 EDT the NetDocuments platform returned to normal operating performance.

At 11:16 EDT the same database sub-system again reported congestion and mal-performance. Again the failover procedure was executed. This time however, the mal-performance did not cease. The issue was then escalated to our Tier II Platform Engineering and Software Engineering teams. Triage teams assembled and began to evaluate the situation. Various troubleshooting events occurred until the root cause was identified.

The root cause was found to be tied to an application software change in the document indexing pipeline. This change included additional load placed upon the database infrastructure that resulted the mal-performance. Once the software change was identified, the software roll-back process was executed and system performance returned to normal at around 12:10 EDT.

Posted Apr 12, 2019 - 11:11 EDT

Resolved

Our engineering team has implemented a fix to resolve the performance issue. If you continue to experience slowness, please open a ticket with our Support team for further assistance.

We prefer that you go to support.netdocuments.com to submit a support request for your issue, so we can manage it more efficiently and provide you with access to your cases.

However, if you need to contact Technical Support via the phone use the following information:

US customers call 801-226-6882 or 866-NETDOCS (638-3627)
EU customers call +44(0) 2034.556770
AU customers call +61 2 8310 4319

Posted Apr 11, 2019 - 14:14 EDT

Monitoring

Performance on the Service has stabilized and we are working to implement a fix.

Posted Apr 11, 2019 - 12:39 EDT

Investigating

The Service is currently experiencing intermittent performance issues. We are working to resolve the situation and apologize for any inconvenience.

Posted Apr 11, 2019 - 11:45 EDT

This incident affected: Platform.