Users in the OnDemand environment are unable to connect

Summary:

On September 10th, 2024, users in the OnDemand environment were receiving errors when connecting to the OBeer application between 1:15 PM and 2:25 PM MST. The outage was caused by a malfunction in the Domain Controller. We successfully restored access by restarting the service. To prevent similar incidents in the future, we are implementing additional detection measures.

Timeline:

• 1:15 PM MST: OnDemand users begin reporting issues connecting to the application.
• 2:23 PM MST: The domain controller is identified as the source of the problem
• 2:25 PM MST: The domain controller is restarted. Users are confirmed to be able to connect.
• 2:45 PM MST: All reported issues are confirmed resolved.

Root Cause:

The outage was caused by an issue with the DNS resolution function of the Domain Controller, which prevented the domain from being contacted from external connections. A restart fixed the issue.

Additional Remediation:

• To help catch and prevent future issues, we added some monitoring to the Domain Controller:

  • DNS service monitoring
  • DNS response time monitoring.