Enterprise 1 customers unable to connect to OBeer

Summary:

On January 5th, 2024, users in the Enterprise 1 environment experienced intermittent trouble connecting to the OBeer application between 1:50 PM and 3:30 PM MST. The outage was caused by a malfunction in the Connection Broker component. We successfully restored access by restarting the service. To prevent similar incidents in the future, we are implementing additional redundancy measures.

Timeline:

• 1:50 PM MST: Enterprise 1 users begin reporting issues connecting to the application.
• 2:12 PM MST: Engineering team begins investigating the issue.
• 3:02 PM MST: Investigation identifies the Connection Broker as the source of the problem.
• 3:15 PM MST: The Connection Broker is restarted.
• 3:30 PM MST: Users regained full access to the application.

Root Cause:

The outage was caused by an internal malfunction within the Connection Broker component. The specific issue was due to an issue with a windows component, and was

Corrective Actions:

• Restart of the Connection Broker: This resolved the immediate issue and restored access to the application.

• Increased Redundancy: To prevent similar incidents in the future, we are implementing the following measures:

  • Load Balancing: We are enabling load balancing for the Connection Broker to distribute traffic across multiple instances and avoid a single point of failure.
  • Second Connection Broker: We are adding a second Connection Broker to further increase redundancy and provide failover capabilities.