Overview:
Between 3/12/2021 9:47:32 PM MST to 3/13/2021 8:24:32 AM (10 hours 37 minutes), all distributor portal pages for customer in the OnDemand environment were unavailable.
Cause of the issue:
The Citrix Netsuite (load balancer) software responsible for routing the URLs to the correct customer's SAP databases had an issue with the purchased license.
The license was an 'eternal' license, so it should never have expired.
Resolution:
While on phone with Citrix support to try to resolve the license issue,
the Orchestra DevOps was also simultaneously beginning to create an alternate routing method, in the event Citrix was not able to find the issue.
After several hours working with Citrix support,
it was determined that it would be faster to switch the routing over to an internal Load Balancer.
This took several hours to copy over, but was completed by 8:24 AM MST