Elevated 5xx “context canceled” errors impacting serverless inference

Final update

All services are operating normally. We will continue to monitor the system to ensure ongoing reliability. Thank you for your patience while we worked to resolve this issue.

1. Resolved · Apr 28, 19:33 UTC
   All services are operating normally. We will continue to monitor the system to ensure ongoing reliability. Thank you for your patience while we worked to resolve this issue.
2. Monitoring · Apr 28, 19:00 UTC
   Service for Serverless Inference has been restored. We’ve implemented tighter rate limits to help prevent recurrence and are closely monitoring system performance. Some users may still experience intermittent latency as we complete final stabilization efforts. Our team remains actively engaged to ensure full recovery. We appreciate your patience and will provide further updates as needed.
3. Identified · Apr 28, 15:59 UTC
   We have identified an issue affecting our service and are currently working to implement a fix. Our team is actively investigating and taking the necessary steps to restore normal operations as quickly as possible. We appreciate your patience and will provide updates as soon as more information becomes available.
4. Investigating · Apr 28, 13:45 UTC
   Serverless inference customers are experiencing elevated 5xx errors, including “context canceled” responses. This may result in intermittent request failures. Our team is actively investigating and will provide updates as more information becomes available.