# Job synchronization design worksheet

This is a design outline, not an executable SDK. Adapt it to your database, API plan and audience. See https://thejobsapi.com/articles/prevent-expired-jobs-in-your-app and the current API documentation.

## Define your scope

- Country and title/location/employer inclusion rules:
- Fields required for display:
- Source job ID and local record key:
- Copies to update: database, search index, page cache, sitemap, recommendations, email queue:
- Maximum tolerated lag for each copy:
- Monthly record/request budget and reserve:

## Choose a bootstrap

`/v1/jobs` supports filtered search. `/v1/changes` is country-wide and has no title/province/employer filters. Returned change events consume job allowance, even when your application discards them. `since=0` reads retained history; it is not a latest-cursor operation.

1. Estimate the whole-country change volume before selecting that strategy for a niche.
2. For a full event replay: start at an explicit cursor, process all pages, save the server's `next_since` and continue while `more` is true. Test time and cost first.
3. For a snapshot plus events: establish a documented checkpoint boundary before taking the snapshot, capture intervening events durably, import, then replay. Do not invent a latest-cursor endpoint. If you cannot establish the boundary, the snapshot is not proven gap-free.
4. A bounded recurring niche search plus reconciliation may be more practical. Missing from one page is not expired. Verify status/current inclusion before removing; document consistency limits.
5. The 6m active feed has a first-seen lookback of 183 days; do not call it an all-time active snapshot.

Chosen approach and why:
Known bootstrap gap or limit:
Expected allowance usage:

## Commit local state and cursor together

The following steps are conceptual, not literal SQL or library calls:

1. Fetch a change batch outside the local transaction.
2. Begin a local database transaction; lock or compare-and-swap the expected country checkpoint.
3. Validate the response and coalesce repeated job IDs if appropriate.
4. Reconcile current job state. The event has a historical kind/time, but its attached job is a current projection, not the historical snapshot. A reopened job can appear in older expiry events. Fetch current full detail when event and current state disagree or required content is absent; budget those calls.
5. Upsert/delete local display state by stable ID and current inclusion policy.
6. Enqueue durable outbox operations with a local monotonic version.
7. Persist the returned `next_since` in the same transaction.
8. Commit. On any error roll back state, outbox and cursor together.
9. Process outbox entries with retries and version checks; complete only after the sink accepts the operation.

Keep one ordered processor per country/checkpoint or enforce a database lock. A transaction in your database does not cover a remote search index.

## Failure tests

| Fault | Required result |
| --- | --- |
| Crash before commit | State and checkpoint both roll back; replay succeeds |
| Crash after commit, before indexing | Durable outbox remains pending |
| Same batch delivered twice | No duplicate local rows or duplicate user alerts |
| Search index unavailable | Retry without losing the queued removal |
| Old index update arrives after removal | Older local version cannot resurrect the job |
| Job leaves niche while still active | Remove from niche without asserting source expiry |
| Expired role later reopens | Current state wins; stale historical event cannot close it again |
| Failed/partial search page | Do not expire unobserved jobs |
| Quota exhausted | Stop safely, alert, retain checkpoint and pending work |
| Several countries | Independent checkpoints; do not assume contiguous sequence numbers |

## Operations checklist

Record last successful API fetch, committed checkpoint, local content revision, oldest pending outbox item and last successful sink update. Alert on backlog age and failed removals, not just worker uptime.

After a controlled test closure, verify every public copy and pending alert. Record elapsed time separately for source-to-provider, provider-to-local database and local database-to-display. Do not label import time as the employer's posting time.

Owner:
Recovery procedure:
Last successful replay test: