New user registration
No notices reported this month
The issue has resolved. The team will be looking into the errors during this maintenance work to find out the root cause.
Routine maintenance impacted the infrastructure more then expected, causing some 5xx errors for API requests and user logins.
A service degradation was detected for our US region, where responses for authentication and APIs were very slow. A caching mechanism used to accelerate the retrieval of refresh tokens failed and caused excessive 5xx errors for client connections. This was remediated during the incident with additional recovery methods put in place to prevent this behaviour.
The service degradation has been resolved and the root cause identified. All systems in the US region are now performing as expected.
A service degradation was detected for our US region, where responses are very slow.
The US region experienced degradation and slower than normal loading of pages and responses from the API. A service designed to help manage connections into the backend infrastructure experienced problems, which then caused 5xx errors on our frontend services. The services handling connections was reset to resolve the issue resulting in all all services returning to normal.
A service degradation was detected for our US region where responses from our API are very slowly or timing out.
All regions suffered very slow response times for a few minutes. A change was deployed that caused the production services to respond very slow or time out until the change was completely rolled out. No action was required for the production service to return to normal.
Services have been restored and operating normally.
A service degradation was detected across our data regions where responses were not connecting and were resolving in errors.
2023 Dec to 2024 Feb