As of 1:43 PM PDT, error rates and latencies for invoked on API Gateway endpoints in the US-WEST-2 Region are now at normal levels. The issue began at 9:20 AM PDT when error rates and latencies for API Gateway began to increase. Error rates began to improve at 10:38 AM PDT, when engineers took action to reduce contention within the subsystem that handles request processing for API Gateway. Error rates continued to improve until 1:10 PM PDT, when engineers applied a mitigation to resolve the contention within the affected subsystem. These actions accelerated recovery, and by 1:43 PM PDT, error rates and latencies had returned to normal levels. Affected AWS services have now recovered as well. The issue has been resolved and the service is operating normally.
Posted Sep 28, 2022 - 16:15 PDT
Update
Latest update from Amazon:
Increased Invoke Error Rates 11:33 AM PDT We continue to work on resolving the elevated error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region. We continue to see a significant improvement in error rates, starting at 10:40 AM PDT, but are not seeing full recovery yet. The issue is caused by contention within the subsystem that is responsible for request processing within the API Gateway service. Engineers are engaged and have applied traffic filters as a precautionary measure, while they work to identify the root cause and resolve the issue. Engineers continue to work to reduce contention within the affected subsystem, which we believe will resolve the elevated error rates and latencies. Customers with applications that use API Gateway, or customers invoking Lambda functions via API Gateway, will be experiencing elevated error rates and latencies as a result of this issue. The AWS services listed below are also experiencing elevated error rates as a result of this issue. While we have seen improvements in error rates since 10:40 AM PDT, recovery has stalled and we do not have a clear ETA on full recovery. For customers that have dependencies on API Gateway and are experiencing error rates, we do not have any mitigations to recommend to address the issue on the customer side. We do expect error rates to continue to improve as contention with the affected subsystem resides, and will provide further updates as recovery progresses.
Posted Sep 28, 2022 - 12:11 PDT
Investigating
The is a problem with the Amazon API Gateway in US-West-2 beginning at 9:40am and ongoing. Multiple services are impacted - EC2, Connect, Workspaces, and many others. Amazon is actively working to resolve the issue, and UCSB engineers are monitoring the situation.
Posted Sep 28, 2022 - 11:19 PDT
This incident affected: Campus Technical Services (AWS).