A kill switch for running integrations
An integration kill switch must define behavior at each execution stage. Stop admitting new effects, hold eligible queued work, reconcile calls already in flight, and continue the callbacks needed to learn outcomes. Configuration distribution and re-enable policy are part of the design, not details hidden behind a boolean.
Inside this article 5 sections +
A flag that hides a button does not stop an integration. Requests may already be accepted, jobs may be queued, and workers may be waiting for a provider response. A useful kill switch specifies what happens to each of those states and how operators verify that new external effects have stopped.
Consider a shipping integration that starts producing invalid labels after a provider change. The desired response is to stop new label purchases while preserving evidence about purchases already submitted. Disabling every code path indiscriminately can also stop the webhooks and reconciliation calls needed to discover what happened.
Define the action being disabled
Name the controlled capability precisely: buying new labels, generating previews, receiving provider callbacks, or reconciling previous requests. A single shipping_enabled boolean can conceal several unrelated decisions. Use a small explicit operating mode where the workflow needs more than on and off, such as enabled, hold-new-effects, and recovery-only.
Scope the control to the affected provider, tenant set, or operation type. An account-specific credential problem should not necessarily stop all customers. Conversely, a global provider defect should not rely on updating thousands of account flags individually. Define precedence when global and scoped controls disagree, and make the evaluated result visible in diagnostics.
| Execution stage | Disable behavior | State to preserve |
|---|---|---|
| New request | Reject clearly or accept into an explicit hold | User intent and expected next action |
| Queued operation | Do not start a new provider effect | Operation identity, deadline, and queue position |
| Call in flight | Stop admitting replacements; reconcile outcome | Provider key and last known attempt |
| Callback received | Continue verified outcome processing | Confirmed provider result |
| Re-enable | Release eligible work at a bounded rate | Cancellation, expiry, and original identity |
Evaluate at admission and immediately before effects
Check the control when accepting an operation so the product can give an honest response. Check again at the worker's effect boundary because a job may have waited while the configuration changed. A flag value copied into the job at creation time describes the original admission decision, not necessarily current permission to execute.
Persist the configuration generation or revision observed by important transitions. An operator can then explain that operation X was admitted under generation seven and held under generation eight. This is diagnostic evidence, not a substitute for checking the current policy. Keep the operation's business identity stable across those transitions.
A check immediately before a network call still leaves a small race with a later flag change. Distributed configuration cannot retroactively prevent a request already sent. Define the acceptable propagation and in-flight window, and use stronger controls where necessary, such as provider credential revocation or a centralized admission boundary. Those controls have their own availability and recovery consequences.
Choose behavior when configuration is stale
Configuration may be cached by an SDK or agent, and processes may observe updates at different times. Set an expected refresh policy and measure the age of the value each worker uses. A dashboard showing the desired flag state is not proof that every replica has applied it.
Define startup behavior when configuration cannot be retrieved. For a consequential purchase operation, refusing new effects until a sufficiently fresh policy is available may be appropriate. A read-only preview may tolerate a last-known value longer. Choose per capability rather than imposing one blanket fallback that either stops harmless work or permits unwanted effects.
Keep the emergency control path independent enough to remain usable during the incident it addresses. If changing the flag requires the failing integration's database or API, operators may be unable to contain it. Control access carefully and record who changed the mode, why, and which configuration revision was deployed.
Illustrative operation evidence:
operation_id: label-req-204
business_state: held
hold_reason: provider_disabled
admitted_config_revision: 7
last_evaluated_config_revision: 8
provider_attempt_state: not_started
expires_at: explicit business deadline
Resolve work already outside the process
For an in-flight provider request, cancellation may stop waiting locally without stopping the provider. Record the operation as uncertain until an authoritative result arrives or reconciliation resolves it. Do not mark it canceled merely because the flag changed, and do not generate a new operation when the integration is re-enabled.
Continue verifying and processing callbacks that establish outcomes. If the provider itself is sending malformed callbacks, quarantine them with preserved evidence rather than applying them blindly. The disable policy should distinguish accepting trustworthy outcome information from initiating new purchases.
Separate business cancellation from operational holding. A held label request may still be wanted later; a canceled order must not resume automatically. Recheck the current order state and deadline before release. The queue's waiting state alone cannot tell the worker which interpretation applies.
Expose both desired mode and observed mode in the operator view. Include configuration age, the number of workers reporting each revision, held-operation counts, and the most recent external attempt. This lets the incident owner distinguish a successful configuration change from a deployment that still has stale executors issuing requests.
Re-enable through a controlled drain
Before re-enabling, resolve the defect and identify the backlog classes: never attempted, confirmed failed, uncertain, expired, and canceled. Only eligible classes should enter automatic recovery. Keep uncertain operations on a reconciliation path so the first recovery burst does not buy labels twice.
Release a small representative set, observe provider responses and business outcomes, then increase admission within the provider budget. Count new work and held work together. A sudden return to full throughput can overload a recovering dependency and recreate the incident under a different name.
Run a drill with a flag change while a request is accepted, while a job waits, immediately before a call, and after provider acceptance. Include one stale replica and a restart without configuration access. The reconciliation guide explains uncertain effects, and retry-storm prevention explains recovery pacing. Integration implementation should make the off switch a tested workflow, not a comforting label.