zero gravitySOLUTIONS
← All insightsSAAS ENGINEERING / CHECKPOINT DESIGN

Building a resumable SaaS import

QUICK ANSWER

A resumable import needs a stable input snapshot, durable progress, and repeat-safe application of each logical item. Save checkpoints only after their covered writes are durable, preserve row-level outcomes, and distinguish resuming the same import from submitting a corrected revision.

Inside this article 5 sections

A resumable import is not simply a loop that remembers the last row number. It needs to know which input version it is reading, which logical records have been applied, and whether the saved checkpoint describes durable work. Otherwise, a worker restart can skip uncommitted rows or apply committed rows twice.

Consider a customer uploading a supplier catalog. The file contains identifiers, descriptions, and prices. Some rows are invalid, some update existing suppliers, and the customer may upload a corrected file while the first import is still running. The import model must make these situations explicit.

Freeze the input and its interpretation

Store the uploaded object as an immutable input for one import identity. Record its checksum or equivalent integrity evidence, tenant, schema version, mapping choices, and parsing configuration. Replacing a file at the same storage path must not silently change the meaning of an in-progress import.

Separate uploading from validation and application. A preview can identify malformed rows, ambiguous identifiers, and incompatible columns before durable writes begin. Record whether invalid rows block the whole import or are skipped with a downloadable outcome report. The product should explain that policy before the user commits the import.

Do not assume byte offsets are safe checkpoints for every file format. Quoted CSV fields can contain line breaks, compressed streams may not support arbitrary seeking, and spreadsheet parsing may require more context. A staging phase can normalize the source into stable logical rows or chunks with their own identities, making later resumption independent of parser internals.

Commit progress with the work it describes

ILLUSTRATIVE CHUNK PROTOCOL
  1. Read stable staged rowsSelect a bounded chunk from the immutable import revision.
  2. Apply guarded writesUse tenant and logical item identity to recognize repeated application.
  3. Persist outcomes and checkpointCommit business changes, row outcomes, and covered progress together where one database owns them.
  4. Acknowledge worker completionA restart can inspect committed progress even if this acknowledgement was lost.
The checkpoint follows durable work. A crash before commit repeats the chunk; a crash after commit discovers that its effects already exist.

For a single-database import, write the chunk's changes and checkpoint in one transaction where practical. Keep chunks small enough to bound lock duration, memory, and rollback cost. One transaction for an enormous file may preserve atomicity while making the application unusable for other customers.

If a chunk spans external systems, a local checkpoint cannot establish remote atomicity. Track each external operation separately, persist intent, and reconcile uncertain results before marking the item complete. An import that provisions remote accounts needs a richer outcome model than one that updates local descriptions.

Parallel chunks require more than a single “last completed row.” If chunk three finishes before chunk two, advancing the checkpoint past both would skip unfinished work after a restart. Store per-chunk completion or maintain a contiguous completed frontier plus explicit outstanding chunks. Define how retries claim chunks without allowing two workers to overwrite each other's progress.

This self-contained Python fixture uses an in-memory SQLite database to demonstrate one atomic chunk boundary. The deliberately raised error occurs after the row and checkpoint writes but before commit. It verifies transaction behavior without requiring a production import environment.

import sqlite3

connection = sqlite3.connect(":memory:")
connection.executescript("""
CREATE TABLE applied_rows (row_id INTEGER PRIMARY KEY, value TEXT NOT NULL);
CREATE TABLE progress (import_id TEXT PRIMARY KEY, next_row INTEGER NOT NULL);
INSERT INTO progress VALUES ('demo-import', 0);
""")
try:
    with connection:
        connection.execute("INSERT INTO applied_rows VALUES (?, ?)", (0, "supplier-a"))
        connection.execute("UPDATE progress SET next_row = ? WHERE import_id = ?", (1, "demo-import"))
        raise RuntimeError("simulated interruption before commit")
except RuntimeError:
    pass
assert connection.execute("SELECT count(*) FROM applied_rows").fetchone()[0] == 0
assert connection.execute("SELECT next_row FROM progress").fetchone()[0] == 0
connection.close()

The example covers database writes in one transaction, not file storage, provider calls, or parallel chunk ordering. Run equivalent interruption cases against the actual database driver and isolation level. External effects need their own durable intent and reconciliation protocol because rolling back this database transaction cannot undo a completed remote action.

Separate duplicate prevention from update policy

A stable import-item identity can prevent repeated application of the same staged row. It does not by itself decide whether two different rows refer to the same supplier. Define the business matching rule separately, using a tenant-scoped supplier identifier where possible. Names alone often produce ambiguous matches.

Decide how duplicate source identifiers behave: reject the duplicate set, choose a documented winner, or merge according to a specific rule. Avoid allowing worker scheduling order to decide which price survives. If parallel execution makes the final result nondeterministic, resumption will be difficult to explain even when every row is processed exactly as coded.

Also define conflicts with interactive edits. A user may correct a supplier while the import is running. Version checks can prevent the import from silently overwriting a newer change. The conflict should appear as a row outcome with enough context for resolution, rather than triggering repeated retries that can never satisfy the stale assumption.

Report progress as outcomes, not a spinner

Expose counts for validated, applied, rejected, conflicted, and unresolved rows. A percentage based on bytes read can be misleading if applying the final rows dominates runtime. Explain whether the displayed progress measures parsing or durable application, and show a state transition when the import moves between those phases.

Cancellation should stop new admissions and let the current bounded transaction reach a known outcome. It does not automatically undo rows already committed. If rollback is a product requirement, design a reversible change record or compensating import explicitly. A button labeled Cancel should not imply that all previous writes will disappear.

Submitting corrected data should create a new input revision linked to the original import. Resuming should continue the same immutable revision. Keeping these actions distinct helps operators answer whether an apparent duplicate is a retry, a correction, or a genuinely new customer request.

Interrupt every checkpoint boundary

Stop a worker before writes, during the transaction, after commit but before acknowledgement, and while parallel chunks finish out of order. Resume and compare the final supplier state and row outcomes with an uninterrupted run. Include a changed source object, duplicate identifiers, and a concurrent user edit. Verify both completeness and absence of repeated effects.

The background job guide supplies durable execution principles, and tenant-aware idempotency supplies request identity. SaaS implementation brings those pieces together in an import experience that can explain what happened to every customer row.

FROM INSIGHT TO ACTIONCustom software development

Make customer imports recoverable.

We can implement staged validation, checkpointed writes, and row-level outcomes that support operators and users through interruptions.

Explore software developmentDiscuss your build

Start with your goals.
We’ll work through the next step together.