Argo CD was found to treat the last revision it compared during a level‑1 refresh as if it were already synced, which triggers a self‑heal that runs a partial sync and omits any Sync‑hook jobs. When a deny window closes, the first refresh that arrives decides whether a full sync (with hooks) or a self‑heal runs, meaning a migration job can be skipped even though the UI reports a successful sync.
How the race condition manifests
Argo CD performs two kinds of refreshes:
- Level‑2 (timer‑driven): runs every two to three minutes, compares the live state against the target revision from Git.
- Level‑1 (object‑driven): triggered by a change to a managed object and reuses the controller’s cached state.
When a deny window ends, no event announces the change. The next refresh—whether level‑1 or level‑2—finds that syncing is now allowed. If a level‑1 refresh arrives first, it uses the revision recorded during the window (the one it last compared) as the “last synced” revision. Seeing no diff, it skips a full sync and instead runs self‑heal, which only reconciles out‑of‑sync resources and deliberately skips all hook types. Because self‑heal writes no operation history, the UI still shows Synced and Healthy for the application.
If a level‑2 refresh wins the race, it compares the live state against the newer target revision, detects a change, and performs a full sync that includes the Sync hook, executing the migration job as expected.
Why hooks are not a guarantee
Sync hooks run only when the operation that triggers them is a full sync. Any other path that can bring a new commit to the cluster—self‑heal, automated pruning, or manual kubectl apply—does not invoke hooks. The discovered bug shows that a partial sync, which is the default behavior of self‑heal, will never run a Sync hook, regardless of configuration flags such as RunHooksOnPartialSync or the hook’s type (PreSync, Sync, PostSync). Changing the hook type or enabling the newer flag does not alter the outcome because the underlying refresh path still skips hooks.
Recommended architectural fix
To make migrations reliable, treat them as ordinary managed resources rather than side‑effects of a sync operation. Remove the hook annotations and assign a negative sync wave so the job is created before the rest of the application:
apiVersion: batch/v1
kind: Job
metadata:
name: db-migration-v11 # change per release
annotations:
argocd.argoproj.io/sync-wave: "-1"
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: busybox:1.36
command: ["sh", "-c", "echo migrating schema to v11; sleep 3"]
Because the job now appears in the desired state, any sync—full or partial—detects it as missing and creates it, guaranteeing execution before the application pods become ready. This also ensures the operation is recorded in Argo CD’s history, making failures visible in the UI.
Related CloudNinjas coverage: DevOps.
What This Means For Practitioners
Teams using Argo CD should audit any critical one‑off tasks (database migrations, certificate rotations, etc.) that are currently implemented as Sync hooks. Verify that those tasks are either part of the declarative state with appropriate sync waves or are protected by an external orchestration step that does not rely on the timing of refreshes. Monitor self‑heal attempt counts and watch for partial syncs that report success without a corresponding hook execution. Adjust deny‑window durations or refresh intervals if the race window cannot be eliminated, but the safest path is to remove the hook dependency altogether.
