Understanding event backfills

What is an event backfill?

Backfill in Orb allows you to add historical data for past events that were missed or delayed, ensuring proper billing accuracy. This is particularly useful when certain usage events aren’t captured in real time. By backfilling, you can ensure that invoices align with actual customer usage, preventing discrepancies in billing cycles. This process is asynchronous and will update relevant draft invoices with updated usage values.

Backfilling is commonly used when system downtime, integration issues, or delayed reporting cause missing data past the grace period. For example, if a customer’s usage wasn’t captured due to an outage, backfill lets you add this data retroactively. Orb’s API offers endpoints for creating backfills and managing event updates efficiently.

Creating a backfill

1. Create the backfill, specifying its parameters.

2. Ingest usage events, referencing the backfill (query parameter backfill_id).

3. Close the backfill, propagating the update in past usage throughout Orb.

What happens during a backfill?

When a backfill is closed, the system:

1. Invalidates the cache for all affected customers

2. Re-evaluate usage for the backfill timeframe

3. Update invoices depending on their state:

   1. For DRAFT invoices: Usage totals will be updated on the draft invoice.

   2. For ISSUED invoices: Usage totals will not be updated as these invoices are finalized, and their values are frozen.

Important considerations

• Creating a new backfill before an ongoing one is reflected is not allowed

• Reverting a backfill requires a similar process and may take a comparable amount of time

• Customer Scope: If you don't specify a customer_id or external_customer_id, the backfill will apply to all customers.

• Event Replacement: The replace_existing_events parameter determines how backfill events interact with existing events:

  • If true: The backfill events will replace all existing events in the time range. If no events are provided in the backfill, all events for that time range will be effectively removed.

  • If false: Backfill events will be added to the existing set of events. Note: that event idempotency does not apply in this case, potentially resulting in duplicate events.

• The time it takes for a backfill to complete depends on:

  • Number of events ingested

  • The time range covered by the backfill

  • Total number of customers in your account

  • Whether the backfill is at the account level or for specific customers