Part Order requests may be slow or incorrectly failing when generating builds
Incident Report for Carbon
Postmortem

Beginning at 1:30 PM PST on Wednesday, September 4 and lasting through 11:00 PM PST on Thursday, September 5 our print preparation platform was unable to consistently generate builds automatically. The system experienced the highest rate of failures between 5:40 AM PST and 11:00 PM PST on Thursday, September 5. During this time, customers experienced a high rate of failed packer jobs resulting in a large magnitude of failed Part Orders.

Customers should re-submit any failed printed parts that occurred during this time frame. 

Subsequently, the system experienced an additional outage between 7:00 AM PST and 12:45 PM PST on September 6, 2024  that stemmed from the prior outage. The issue has now been resolved and we are currently working through the backlog of parts waiting to be processed. 

To resolve this issue, there are a number of printed parts that were marked as failed. Customers should re-submit any failed printed parts that occurred during this time frame.

Posted Sep 06, 2024 - 13:30 PDT

Resolved
This incident has been resolved.
Posted Sep 06, 2024 - 13:30 PDT
Investigating
Carbon is currently experiencing a partial outage. Users may experience slow or failure to generate builds when submitting a Part Order request.
Posted Sep 06, 2024 - 10:09 PDT
This incident affected: Carbon API and Model Management and Processing.