Design GitOps Workflow

Medium40 min
1 / 30
understanding6 min read

Problem Statement: GitOps as Cluster Source of Truth

How Problem Statement: GitOps as Cluster Source of Truth (understanding) informs GitOps Workflow architecture and interviewer depth.

Problem Statement: GitOps as Cluster Source of Truth

GitOps treats Git as the single source of truth for Kubernetes (and often VM) desired state. Argo CD, Flux, and GitLab Agent continuously reconcile live clusters to manifests in Git—PRs become change requests, merges trigger automated sync, and rollbacks are often git revert.

Core interview story
  1. Declarative manifests in Git (Helm, Kustomize, plain YAML, OCI artifacts).
  2. Continuous reconciliation — operators detect drift and repair or alert.
  3. Environment promotion via branches or directories (env/staging, env/prod).
  4. Progressive delivery with sync waves, health checks, and optional Argo Rollouts.
Scale anchors (state aloud)
  • 400 microservices across 12 EKS/GKE/AKS clusters, ~2,000 Applications in Argo CD.
  • ~800 manifest commits/day; webhook bursts ~30 events/sec after large merges.
  • Reconciliation: poll + webhook driven; target p95 sync latency < 3 min for non-prod, manual/auto gates for prod.
javaOne Dark Pro
1public final class SyncPolicyGate {
2 public boolean allowAutoSync(boolean prod, boolean prMerged, boolean policyPass) {
3 if (prod) return prMerged && policyPass;
4 return policyPass;
5 }
6}
pythonOne Dark Pro
1from dataclasses import dataclass
2
3@dataclass
4class DesiredRevision:
5 repo_url: str
6 target_revision: str
7 path: str
8
9def out_of_sync(live_hash: str, desired_hash: str) -> bool:
10 return live_hash != desired_hash
typescriptOne Dark Pro
1interface ApplicationSpec {
2 project: string;
3 destinationCluster: string;
4 path: string;
5 autoSync: boolean;
6}
7
8export function shouldReconcile(app: ApplicationSpec, driftDetected: boolean): boolean {
9 return driftDetected || app.autoSync;
10}

How to open this one

The framing that lands for GitOps is Git as the single source of truth with a controller continuously reconciling the cluster toward the committed manifests — so a deploy is a merge and a rollback is a git revert. Lead with the reconcile loop and drift detection (the controller reverts manual kubectl changes back to what Git declares), and the failure story that proves it: someone hot-patches prod by hand, the controller notices the drift and restores the declared state. That shows you understand GitOps makes the repo, not the cluster, authoritative.

Key Highlights

  • GitOps workflow limits blast radius by measuring candidate traffic before full promotion.
  • Baseline comparison beats global averages when traffic is diurnal.
  • Fail closed on stale metrics or failed mesh ACKs.
  • Expand-contract migrations are mandatory under split traffic.
Say this aloud
I gate promotion on sync health and diff severity with minimum sample sizes, and rollback faster than users notice regression.
Production tip
Pre-compute recording rules for baseline and candidate error rates to cut analysis query cost 40%.

Section Rescue Kit

Buzzwords to use:

progressive syncError Budget

Safe statements:

  • "I will gate promotion on sync health and diff severity against a stable last-known-good revision, not gut feel."
  • "If metrics are inconclusive, I hold sync health and widen observation window before promoting."
  • "Rollback must be one-click and faster than mean time to detect regression."
Design GitOps Workflow - System Design | WinJob | WinJob