Live
GPU‑Accelerated Inference on AWS Lambda via Managed InstancesClaude Code’s “Don’t Ask Again” local file leaks credentials to public reposPost‑training RL with PivotRL lifts Nemotron 3 Ultra coding benchmarkIntegrating AI Coding Agents into Development Workflows: Protocols, Practices, and Security ImplicationsAWS Quick’s new Meta ads AI connector: what engineers need to knowAI router data leakage forces engineers to rethink prompt pipelinesRust’s Agent Client Protocol simplifies AI‑agent integration for production workloadsUsing Gaussian Sharpening with Cloudflare Images: Practical Guidance for EngineersGPU‑Accelerated Inference on AWS Lambda via Managed InstancesClaude Code’s “Don’t Ask Again” local file leaks credentials to public reposPost‑training RL with PivotRL lifts Nemotron 3 Ultra coding benchmarkIntegrating AI Coding Agents into Development Workflows: Protocols, Practices, and Security ImplicationsAWS Quick’s new Meta ads AI connector: what engineers need to knowAI router data leakage forces engineers to rethink prompt pipelinesRust’s Agent Client Protocol simplifies AI‑agent integration for production workloadsUsing Gaussian Sharpening with Cloudflare Images: Practical Guidance for Engineers

Argo CD sync race can silently skip migration hooks after a deny window

AI SummaryPowered by AI

A race between level‑1 and timer‑driven refreshes after a deny window closes can cause Argo CD to perform a self‑heal that skips Sync hooks, leaving migrations unexecuted. Practitioners must treat migrations as part of the desired state to avoid hidden schema mismatches.

Argo CD was found to treat the last revision it compared during a level‑1 refresh as if it were already synced, which triggers a self‑heal that runs a partial sync and omits any Sync‑hook jobs. When a deny window closes, the first refresh that arrives decides whether a full sync (with hooks) or a self‑heal runs, meaning a migration job can be skipped even though the UI reports a successful sync.

How the race condition manifests

Argo CD performs two kinds of refreshes:

  • Level‑2 (timer‑driven): runs every two to three minutes, compares the live state against the target revision from Git.
  • Level‑1 (object‑driven): triggered by a change to a managed object and reuses the controller’s cached state.

When a deny window ends, no event announces the change. The next refresh—whether level‑1 or level‑2—finds that syncing is now allowed. If a level‑1 refresh arrives first, it uses the revision recorded during the window (the one it last compared) as the “last synced” revision. Seeing no diff, it skips a full sync and instead runs self‑heal, which only reconciles out‑of‑sync resources and deliberately skips all hook types. Because self‑heal writes no operation history, the UI still shows Synced and Healthy for the application.

If a level‑2 refresh wins the race, it compares the live state against the newer target revision, detects a change, and performs a full sync that includes the Sync hook, executing the migration job as expected.

Why hooks are not a guarantee

Sync hooks run only when the operation that triggers them is a full sync. Any other path that can bring a new commit to the cluster—self‑heal, automated pruning, or manual kubectl apply—does not invoke hooks. The discovered bug shows that a partial sync, which is the default behavior of self‑heal, will never run a Sync hook, regardless of configuration flags such as RunHooksOnPartialSync or the hook’s type (PreSync, Sync, PostSync). Changing the hook type or enabling the newer flag does not alter the outcome because the underlying refresh path still skips hooks.

Recommended architectural fix

To make migrations reliable, treat them as ordinary managed resources rather than side‑effects of a sync operation. Remove the hook annotations and assign a negative sync wave so the job is created before the rest of the application:

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration-v11  # change per release
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: migrate
        image: busybox:1.36
        command: ["sh", "-c", "echo migrating schema to v11; sleep 3"]

Because the job now appears in the desired state, any sync—full or partial—detects it as missing and creates it, guaranteeing execution before the application pods become ready. This also ensures the operation is recorded in Argo CD’s history, making failures visible in the UI.

Related CloudNinjas coverage: DevOps.

What This Means For Practitioners

Teams using Argo CD should audit any critical one‑off tasks (database migrations, certificate rotations, etc.) that are currently implemented as Sync hooks. Verify that those tasks are either part of the declarative state with appropriate sync waves or are protected by an external orchestration step that does not rely on the timing of refreshes. Monitor self‑heal attempt counts and watch for partial syncs that report success without a corresponding hook execution. Adjust deny‑window durations or refresh intervals if the race window cannot be eliminated, but the safest path is to remove the hook dependency altogether.

Originally published atCNCF