Temporary Data Ingestion Delay for US Projects

Summary

Temporary Data Ingestion Delay for US Projects

# Summary Between approximately **11:35 PM PT on July 10 and 7:19 AM PT on July 11, 2026**, data ingestion for Mixpanel projects in the US region ran behind by up to ~2 hours. During this window, reports and dashboards temporarily showed incomplete data — recent time ranges could appear as sharp, artificial drops in metrics such as active users or revenue. **No data was lost.** All events were queued durably and processed in full once the backlog cleared; metrics returned to accurate values on their own, with no customer action required. The cause originated in our ingestion controls: for a small number of very high-volume projects, the ingestion rates we allowed had drifted out of alignment with the capacity provisioned for those projects. Large historical imports — entirely legitimate use of the platform — were therefore admitted faster than their infrastructure could absorb, and two properties of our pipeline turned that localized overload into a region-wide delay. The fixes below realign those controls so that any import, of any size, is automatically kept within safe bounds. # What happened Mixpanel's ingestion pipeline is sharded and multi-tenant: each project's data is distributed across a set of partitions sized for its expected volume, and for throughput, events from many customers are processed together in batches. This design delivers high efficiency, but it depends on one invariant: the rate at which we admit a project's traffic must match the capacity provisioned for it. When that invariant holds, even very large imports are absorbed smoothly. Here, it did not hold. A small number of very large historical data imports ran on projects whose allowed ingestion rates had, over time, grown well beyond their provisioned partition capacity. The excess volume concentrated onto specific partitions as **hot spots**, saturating the portion of the streaming fleet serving them. Two factors then widened the impact: * **Batched, multi-tenant processing amplified the hot spots.** Because events from many customers travel together in batches, slowness and failures on the overloaded partitions delayed unrelated customers' events sharing those batches. * **Automatic scale-up was ineffective.** Adding capacity cannot dissolve a hot spot of this kind, because the overloaded partitions stay pinned to the same infrastructure, and a capacity limit in a component of our streaming infrastructure prevented the added capacity from taking effect. Together, these turned what should have been a brief, self-healing slowdown into a multi-hour delay requiring manual intervention. # Timeline \(Pacific Time, July 10–11\) * **11:35 PM** — Automated alerting detected the ingestion backlog; on-call engineer engaged immediately. * **12:49 AM** — Status page incident posted; impact scoped to the US region only. * **1:23 AM** — The largest contributing import was paused. * **1:55–6:48 AM** — Progressive mitigations: additional traffic sources throttled, certain backlogged data deferred with the owning customer's agreement, failure isolation enabled in the pipeline, and affected infrastructure nodes replaced. * **7:19 AM** — Backlog fully processed; all projects current. Status page moved to monitoring, then resolved after a stable observation period. # Root cause 1. **Ingestion rate limits misaligned with provisioned capacity.** For the driving projects, the rates our platform admitted had grown out of step with the infrastructure provisioned for them, so legitimate high-volume imports were let in faster than their partitions could absorb. This is the systemic root cause. The imports themselves were a legitimate use of the platform. 2. **Batched, multi-tenant processing amplified the overload.** Failures on the overloaded partitions delayed unrelated customers' events sharing the same processing batches, spreading a localized problem across the platform. 3. **Traffic to the overloaded partitions could not be redistributed.** Partition-to-server assignment in the streaming layer was not load-aware, so the hot spots stayed pinned to the same servers regardless of fleet size — total capacity was sufficient, but it could not be brought to bear. A scaling limit in a component of our streaming infrastructure compounded this by preventing scale-up from adding capacity, which delayed diagnosis until engineers intervened manually. # What we're changing The end state we are building toward: **every project's ingestion limits automatically match its provisioned capacity, so that imports of any size, including full historical backfills and data warehouse syncs, can run without advance coordination, and one project's volume is prevented from affecting another's data freshness.**. Already deployed — improving our ingestion handling: * **Improved hot-spot handling.** Traffic distribution in the streaming layer is now load-aware, spreading concentrated load more evenly across the fleet, and individual problem items are now retried separately instead of holding up the rest of their batch — reducing, though not eliminating, the impact a localized overload can have on unrelated traffic. * **Re-provisioned the highest-volume workloads** onto appropriately sized infrastructure, prioritized by risk. Already deployed — changing how we operate the third-party streaming service: * **Audited and re-tuned the fleet's capacity profile** so individual servers have substantially more headroom for concentrated load, and worked with the provider to resolve the scaling limitation encountered during the incident. In progress: * **Capacity-aware rate limiting** — closing the gaps between individually granted project rate limits and each project's real provisioned capacity, extending rate limiting to ingestion paths that previously lacked it, and coupling any future limit increase to a capacity increase, which is designed to prevent this class of misalignment from recurring.. * **Finer-grained volume monitoring and alerting** so capacity misalignment is detected and corrected before it can affect any customer. * Evaluating stronger workload isolation/backlog recovery prioritization for bulk/backfill traffic paths, so historical imports have reduced impact on live traffic # Common questions * **Was any data lost?** No. Events were durably queued throughout the incident and were fully processed once the backlog cleared. Any metric drops seen during the window were a display artifact of the delay and self-corrected. * **Do I need to coordinate large imports or backfills with Mixpanel?** No — Our goal is for the platform to keep any import within safe rates automatically, so that backfills and warehouse syncs can run without scheduling or notice. We are still rolling out the capacity-aware controls that deliver this. In the meantime, if you are planning an unusually large import or backfill, we recommend coordinating with your account team so we can confirm capacity in advance. * **How is this prevented going forward?** The systemic fix is tightening the gaps between individually granted project rate limits and each project's real provisioned capacity, and extending rate limiting to the ingestion paths that previously lacked it — so overload of this kind is stopped at admission. In addition, the scaling limitation that prolonged the incident is fixed, the pipeline now retries individual problem items separately so a localized overload has far less impact on unrelated traffic, and bulk traffic is being further isolated from live traffic. * **Can individual projects be prioritized during recovery?** This capability did not exist during the incident — all projects recovered at the same rate. We are evaluating prioritization mechanisms for backlog recovery as part of our follow-up work. We apologize for the disruption and for the concern the temporarily depressed metrics caused. Please reach out through your account team or support with any questions.

• Started

  Jul 11, 2026 · 12:53 AM

• Resolved

  Jul 11, 2026 · 8:54 AM

• Duration

  8h

• Severity

  Minor

• ◐

  Investigating

  Jul 11 · 12:53 AM Mixpanel

  We are currently experiencing delays in our data ingestion pipeline, which is affecting real-time data for projects enrolled in the US projects, resulting in delays for a subset of projects. While no data is being lost, our engineering team is actively investigating the matter and working to restore real-time functionality as quickly as possible. We appreciate your patience during this time. If you have any questions, please <a href="https://mixpanel.com/get-support">contact support.</a>

• ◐

  Investigating

  Jul 11 · 2:17 AM Mixpanel

  We are continuing to investigate this issue.

• ◆

  Identified

  Jul 11 · 3:15 AM Mixpanel

  The issue has been identified and a fix is being implemented.

• ◆

  Identified

  Jul 11 · 5:54 AM Mixpanel

  We are continuing to work on a fix for this issue.

• ◉

  Monitoring

  Jul 11 · 7:25 AM Mixpanel

  A fix has been implemented and we are monitoring the results.

• ✓

  Resolved

  Jul 11 · 8:54 AM Mixpanel

  This incident has been resolved.

• ✓

  Postmortem

  Jul 16 · 4:47 PM Mixpanel

  # Summary Between approximately **11:35 PM PT on July 10 and 7:19 AM PT on July 11, 2026**, data ingestion for Mixpanel projects in the US region ran behind by up to ~2 hours. During this window, reports and dashboards temporarily showed incomplete data — recent time ranges could appear as sharp, artificial drops in metrics such as active users or revenue. **No data was lost.** All events were queued durably and processed in full once the backlog cleared; metrics returned to accurate values on their own, with no customer action required. The cause originated in our ingestion controls: for a small number of very high-volume projects, the ingestion rates we allowed had drifted out of alignment with the capacity provisioned for those projects. Large historical imports — entirely legitimate use of the platform — were therefore admitted faster than their infrastructure could absorb, and two properties of our pipeline turned that localized overload into a region-wide delay. The fixes below realign those controls so that any import, of any size, is automatically kept within safe bounds. # What happened Mixpanel's ingestion pipeline is sharded and multi-tenant: each project's data is distributed across a set of partitions sized for its expected volume, and for throughput, events from many customers are processed together in batches. This design delivers high efficiency, but it depends on one invariant: the rate at which we admit a project's traffic must match the capacity provisioned for it. When that invariant holds, even very large imports are absorbed smoothly. Here, it did not hold. A small number of very large historical data imports ran on projects whose allowed ingestion rates had, over time, grown well beyond their provisioned partition capacity. The excess volume concentrated onto specific partitions as **hot spots**, saturating the portion of the streaming fleet serving them. Two factors then widened the impact: * **Batched, multi-tenant processing amplified the hot spots.** Because events from many customers travel together in batches, slowness and failures on the overloaded partitions delayed unrelated customers' events sharing those batches. * **Automatic scale-up was ineffective.** Adding capacity cannot dissolve a hot spot of this kind, because the overloaded partitions stay pinned to the same infrastructure, and a capacity limit in a component of our streaming infrastructure prevented the added capacity from taking effect. Together, these turned what should have been a brief, self-healing slowdown into a multi-hour delay requiring manual intervention. # Timeline \(Pacific Time, July 10–11\) * **11:35 PM** — Automated alerting detected the ingestion backlog; on-call engineer engaged immediately. * **12:49 AM** — Status page incident posted; impact scoped to the US region only. * **1:23 AM** — The largest contributing import was paused. * **1:55–6:48 AM** — Progressive mitigations: additional traffic sources throttled, certain backlogged data deferred with the owning customer's agreement, failure isolation enabled in the pipeline, and affected infrastructure nodes replaced. * **7:19 AM** — Backlog fully processed; all projects current. Status page moved to monitoring, then resolved after a stable observation period. # Root cause 1. **Ingestion rate limits misaligned with provisioned capacity.** For the driving projects, the rates our platform admitted had grown out of step with the infrastructure provisioned for them, so legitimate high-volume imports were let in faster than their partitions could absorb. This is the systemic root cause. The imports themselves were a legitimate use of the platform. 2. **Batched, multi-tenant processing amplified the overload.** Failures on the overloaded partitions delayed unrelated customers' events sharing the same processing batches, spreading a localized problem across the platform. 3. **Traffic to the overloaded partitions could not be redistributed.** Partition-to-server assignment in the streaming layer was not load-aware, so the hot spots stayed pinned to the same servers regardless of fleet size — total capacity was sufficient, but it could not be brought to bear. A scaling limit in a component of our streaming infrastructure compounded this by preventing scale-up from adding capacity, which delayed diagnosis until engineers intervened manually. # What we're changing The end state we are building toward: **every project's ingestion limits automatically match its provisioned capacity, so that imports of any size, including full historical backfills and data warehouse syncs, can run without advance coordination, and one project's volume is prevented from affecting another's data freshness.**. Already deployed — improving our ingestion handling: * **Improved hot-spot handling.** Traffic distribution in the streaming layer is now load-aware, spreading concentrated load more evenly across the fleet, and individual problem items are now retried separately instead of holding up the rest of their batch — reducing, though not eliminating, the impact a localized overload can have on unrelated traffic. * **Re-provisioned the highest-volume workloads** onto appropriately sized infrastructure, prioritized by risk. Already deployed — changing how we operate the third-party streaming service: * **Audited and re-tuned the fleet's capacity profile** so individual servers have substantially more headroom for concentrated load, and worked with the provider to resolve the scaling limitation encountered during the incident. In progress: * **Capacity-aware rate limiting** — closing the gaps between individually granted project rate limits and each project's real provisioned capacity, extending rate limiting to ingestion paths that previously lacked it, and coupling any future limit increase to a capacity increase, which is designed to prevent this class of misalignment from recurring.. * **Finer-grained volume monitoring and alerting** so capacity misalignment is detected and corrected before it can affect any customer. * Evaluating stronger workload isolation/backlog recovery prioritization for bulk/backfill traffic paths, so historical imports have reduced impact on live traffic # Common questions * **Was any data lost?** No. Events were durably queued throughout the incident and were fully processed once the backlog cleared. Any metric drops seen during the window were a display artifact of the delay and self-corrected. * **Do I need to coordinate large imports or backfills with Mixpanel?** No — Our goal is for the platform to keep any import within safe rates automatically, so that backfills and warehouse syncs can run without scheduling or notice. We are still rolling out the capacity-aware controls that deliver this. In the meantime, if you are planning an unusually large import or backfill, we recommend coordinating with your account team so we can confirm capacity in advance. * **How is this prevented going forward?** The systemic fix is tightening the gaps between individually granted project rate limits and each project's real provisioned capacity, and extending rate limiting to the ingestion paths that previously lacked it — so overload of this kind is stopped at admission. In addition, the scaling limitation that prolonged the incident is fixed, the pipeline now retries individual problem items separately so a localized overload has far less impact on unrelated traffic, and bulk traffic is being further isolated from live traffic. * **Can individual projects be prioritized during recovery?** This capability did not exist during the incident — all projects recovered at the same rate. We are evaluating prioritization mechanisms for backlog recovery as part of our follow-up work. We apologize for the disruption and for the concern the temporarily depressed metrics caused. Please reach out through your account team or support with any questions.