We have prioritized the delayed workstreams that were impacted by this incident. They continue to catch up without issue, and we have not observed any additional degradation. This incident is now resolved.
Posted Oct 12, 2023 - 22:04 UTC
Monitoring
Between 1am and 12pm US Eastern Time on 10/12, Vitally's background jobs system experienced sporadic failures. These failures caused individual background job runners to degrade and have their work marked as failed so that another job runner would pick up its slack. Unfortunately, the original runner would eventually complete the work that was marked as failed, in addition to another runner starting the same work, causing duplicative downstream impacts.
This has surfaced to customers in a few ways, such as multiple emails to reconnect an integration being sent when only one was needed, some delayed and duplicated integration syncs, and potential duplicative pushes of updated data to integrations.
This would not have impacted execution of playbooks--conversations, tasks, and projects created by playbooks would remain singular and not impacted by this specific cluster's outage. So we do not expect your customers to have been impacted by this incident.
We've identified the workload that caused our background clusters instability and have mitigated it as of 12pm US Eastern Time today. We are catching up on our delayed workstreams and will dedicate resources to preventing this in the future.
Posted Oct 12, 2023 - 20:35 UTC
This incident affected: Product and Analytics Data Processing.