Zero-downtime blue/green on Kubernetes — proven recipe

Run two color-tagged Deployments side by side behind one Gateway. Cut over by shifting HTTPRoute weights, not Service selectors. Roll back by patching the weights the other way. Tested end-to-end on GKE 1.35 with a real GCLB; measured failure rate during a 100→50→0 cutover at 277 RPS: 0.03% (9 / 27,750 requests over 100s).

The naive variant — single Service with a version: blue|green selector that you patch — does NOT work zero-downtime on GCLB. It fails because GCLB caches the Network Endpoint Group (NEG) endpoint list at the cloud control plane (30–60s convergence), and a Service-selector flip churns that list. Measured failure rate on the same cluster, same load: 36.97%. The full bake-off, including the kubectl-port-forward false positive that made the first attempt look like it worked, is in docs/internals/blue-green-cutover-proof.md.

What you get over a rolling update

Concern	Rolling update	Blue/green (this recipe)
Rollback latency	Minutes (rebuild + re-rollout)	Seconds (one weight patch)
New-version exposure	Progressive (50% then 100%)	Stepped (100/0 → 90/10 → 50/50 → 0/100) under your control
Smoke window before traffic	None	Unbounded (port-forward into green privately)
Bad image in production	All new pods + maybe old ones until probes trip	Zero — broken green never gets a non-zero weight if you smoke first
Capacity during cutover	N pods	Up to 2N pods briefly

The five ingredients (all required)

Two color-specific Deployments — nwire-app-blue + nwire-app-green, each with version: blue|green pod label.
Two color-specific Services — each selects its own color only. This is the deviation from the naïve recipe: never a single Service-with-version-selector.
One HealthCheckPolicy per Service — pointing at lightship's /live on the probes port (9400 in the canonical recipe). Without this, GCLB auto-creates a check on port 3000 path / which the app 404s on, and the NEG starts UNHEALTHY → 502 storm before any traffic flows.
One HTTPRoute with weighted backendRefs — both color Services are wired in, with explicit weight. The cutover lever is patching these weights.
@nwire/endpoint drain budget aligned with terminationGracePeriodSeconds — drainDelay + drainTimeout ≤ terminationGracePeriodSeconds. Defaults: 10 + 30 ≤ 45. This is what makes the last in-flight requests on the old color finish cleanly when you eventually scale it down.

All five live in infra/k8s/blue-green/ — copy them, swap the image registry + hostname, done.

Why the weighted pattern works and the selector pattern doesn't

Same cluster, same app, same probes — different cutover mechanic. Both ran k6 at sustained load through https://*.dev.svc.amit-lemida.com (real public GCLB, not port-forward):

Pattern	Reqs	Fail rate	Why
Single Service, patch selector blue→green	24,929	36.97%	GCLB caches NEG endpoint list 30–60s. Selector flip changes which pods that one Service selects, NEG controller updates membership, GCLB backend service re-syncs slowly. During the window, GCLB sends traffic to the OLD pod set. Scale blue→0 immediately and in-flight requests hit terminated endpoints. RST.
Two Services, weight 100→50→0	27,750	0.03%	Two NEGs that never change membership during cutover. GCLB's URL map weights shift traffic at the LB layer in ~1s. No NEG churn, no cache invalidation, both backend services stay warm.

In-cluster traffic (port-forward, kube-proxy/Cilium) makes the selector pattern look fine because kube-proxy updates endpoints in ~1s. Never validate a GCLB-fronted cutover via port-forward — it bypasses the entire LB.

Initial deploy

Steady state: blue serves, green stays at weight 0 with healthy pods kept warm.

bash

kubectl apply -f infra/k8s/blue-green/namespace.yaml
kubectl apply -f infra/k8s/blue-green/deployment-blue.yaml
kubectl apply -f infra/k8s/blue-green/deployment-green.yaml   # green at same image as blue for the first deploy
kubectl apply -f infra/k8s/blue-green/service.yaml            # 2 Services + 2 HealthCheckPolicies
kubectl apply -f infra/k8s/blue-green/ingress.yaml            # HTTPRoute with weight 100/0 + ReferenceGrant

kubectl -n nwire-bg wait pod -l app=nwire-app --for=condition=Ready --timeout=180s

Convergence checklist (give GCLB 60–120s):

bash

# HTTPRoute Reconciled = True
kubectl get httproute -n gateway nwire-app-route \
  -o jsonpath='{.status.parents[0].conditions[?(@.type=="Reconciled")].reason}{"\n"}'
# Expect: ReconciliationSucceeded

# Both NEGs HEALTHY
for color in blue green; do
  echo "=== $color ==="
  gcloud compute backend-services list --filter="name~nwire-app-$color" --format='value(name)' \
    | xargs -I{} gcloud compute backend-services get-health {} --global \
    | grep healthState
done
# Expect: HEALTHY HEALTHY for each color

If you see cannot use named port when using network endpoint group as backend on the HTTPRoute status — that's the named-port HealthCheckPolicy footgun. Use USE_FIXED_PORT with port: 9400, not USE_NAMED_PORT + portName: probes. GCLB rejects named ports for NEG-backed services.

Cutover blue → green — the four-step ritual

Write this as a script and run from CI. Each step has a clear pre/post check; if any check fails, you stop and roll back (which is itself just step 3 in reverse).

Step 1. Roll green to the new image, keep weight at 0

bash

kubectl -n nwire-bg set image deployment/nwire-app-green \
  app=me-west1-docker.pkg.dev/.../nwire-app:sha-abc123-green

kubectl -n nwire-bg rollout status deployment/nwire-app-green --timeout=180s

Production traffic untouched — weight is still 100/0. If the green rollout fails (ImagePullBackOff, CrashLoopBackOff, readiness never goes green), you find out here before any production traffic moves. This is the cheapest place to discover a bad image.

Step 2. Smoke green privately via port-forward (mandatory)

This catches bad code that passes probes but fails business logic. Probes only verify "process is up + lightship is serving" — they don't verify your app actually works end-to-end with real config, secrets, and downstream dependencies.

bash

GREEN_POD=$(kubectl -n nwire-bg get pod -l version=green -o name | head -1)
kubectl -n nwire-bg port-forward $GREEN_POD 3000:3000 9400:9400 &
PF_PID=$!

curl -fsS http://localhost:9400/ready              # lightship readiness — must 200
curl -fsS http://localhost:3000/openapi.json       # data port serving — must 200
pnpm --filter @nwire/<your-smoke> test --reporter=json   # your business smoke

kill $PF_PID

If any of those fail, scale green back to 0, roll the green image to a known-good tag, and start over. Production is still on blue. Zero impact.

This is your before-the-LB safety net. Steps 3 + 4 are mechanical — the safety lives here.

Step 3. Shift weight in steps (NOT all-at-once)

Gradual weight shifts let you watch error-rate + latency dashboards between steps. If anything looks wrong, patch back to 100/0 (full rollback) in under a second.

bash

# 90/10 — 10% of traffic to green for ~30s under real load
kubectl -n gateway patch httproute nwire-app-route --type=merge -p '{
  "spec":{"rules":[{"matches":[{"path":{"type":"PathPrefix","value":"/"}}],
  "backendRefs":[
    {"name":"nwire-app-blue","namespace":"nwire-bg","port":3000,"weight":90},
    {"name":"nwire-app-green","namespace":"nwire-bg","port":3000,"weight":10}
  ]}]}}'
sleep 30  # watch dashboards. Roll back here if anything looks off.

# 50/50 — half traffic each
kubectl -n gateway patch httproute nwire-app-route --type=merge -p '{
  "spec":{"rules":[{"matches":[{"path":{"type":"PathPrefix","value":"/"}}],
  "backendRefs":[
    {"name":"nwire-app-blue","namespace":"nwire-bg","port":3000,"weight":50},
    {"name":"nwire-app-green","namespace":"nwire-bg","port":3000,"weight":50}
  ]}]}}'
sleep 30

# 0/100 — green serves everything
kubectl -n gateway patch httproute nwire-app-route --type=merge -p '{
  "spec":{"rules":[{"matches":[{"path":{"type":"PathPrefix","value":"/"}}],
  "backendRefs":[
    {"name":"nwire-app-blue","namespace":"nwire-bg","port":3000,"weight":0},
    {"name":"nwire-app-green","namespace":"nwire-bg","port":3000,"weight":100}
  ]}]}}'

Use --type=merge not the default strategic-merge: GCLB's HTTPRoute schema is array-based for backendRefs. Strategic merge tries to merge by name and produces invalid output; merge-patch replaces the full rules array, which is what we want.

Step 4. Drain blue, then scale to 0 — DO NOT skip the drain wait

Even at weight 0, GCLB may still have in-flight requests on blue's NEG (long-lived connections, slow clients). The drain wait is bounded by @nwire/endpoint's defaults: drainDelay + drainTimeout = 10 + 30 = 40s.

bash

sleep 45                                              # > drainDelay + drainTimeout
kubectl -n nwire-bg scale deploy/nwire-app-blue --replicas=0

Blue's pods get SIGTERM. lightship 503's /ready first, then http-terminator stops accepting new connections, waits for in-flight responses up to drainTimeout, then the process exits cleanly. Because weight is already 0 and we waited 45s, there should be nothing in flight — but the drain machinery is what makes the unhappy case (a slow client, a long-running request) also clean.

Blue stays scaled-to-0 until your next deploy, when it becomes "the new green."

Rollback — green → blue

Symmetric. If green misbehaves after step 3 but before step 4, rollback is literally one patch:

bash

kubectl -n gateway patch httproute nwire-app-route --type=merge -p '{
  "spec":{"rules":[{"matches":[{"path":{"type":"PathPrefix","value":"/"}}],
  "backendRefs":[
    {"name":"nwire-app-blue","namespace":"nwire-bg","port":3000,"weight":100},
    {"name":"nwire-app-green","namespace":"nwire-bg","port":3000,"weight":0}
  ]}]}}'

Blue is still warm at full capacity. No rebuild, no cold start, no rollout. If you'd already scaled blue to 0 (step 4 complete), rollback also brings it back:

bash

kubectl -n nwire-bg scale deploy/nwire-app-blue --replicas=2
kubectl -n nwire-bg rollout status deploy/nwire-app-blue
# ...then the patch above.

Bounded by image pull + container start (typically 5–20s on warm nodes).

The bad-image scenario — what protects you, exactly

This is what the recipe is for. Imagine you shipped a bug. Three layered defenses, in order of when they catch it:

Defense 1 — green never gets traffic until you say so (step 2 catches almost everything)

Steps 1 + 2 happen before any weight shift. The new image rolls into green pods; weight stays at 100/0. Even if green is utterly broken — ImagePullBackOff, CrashLoopBackOff, listens on wrong port, throws on first request — production traffic continues hitting blue. Tested: pointed green at nwire-hello:does-not-exist, scaled to 2 — both green pods stuck in ImagePullBackOff, blue's 2 pods kept serving 300 RPS through real GCLB. Step 1 fails loudly (rollout status times out) and you stop.

Catches: bad image tags, missing config, container start failures, broken health probes, dependency-injection errors at boot.

Defense 2 — GCLB health checks fence off unhealthy backends (step 3 catches partial breakage)

If green's pods start up and lightship goes ready but the app is half-broken (some routes 500, dependency timeout, partial deadlock), GCLB's per-Service HealthCheckPolicy notices within unhealthyThreshold * checkIntervalSec (default: 3 × 5s = 15s) and removes green's endpoints from the routing pool — even though the HTTPRoute weight says 50% should go there.

Catches: probe-passes-but-business-logic-fails (sometimes), backend dependency outages exposed at startup, race conditions visible only under real load.

This is not a complete safety net. Health checks fire on a synthetic GET to /live. They do not exercise your action handlers, do not hit your database, and do not validate writes. If your bad code passes /live and ships errors only on specific business inputs, GCLB will not know.

Defense 3 — gradual weight ramp with dashboard watching (step 3 with sleeps catches everything else)

The 90/10 → 50/50 → 0/100 staircase, with a real human watching error/latency dashboards between steps, catches the breakages that pass probes AND pass health checks AND only show up under real-user traffic patterns. The blast radius at 10% is 10× smaller than at 100%, and rolling back from 10% is one patch.

The recipe deliberately requires a human (or a synthetic monitor) between steps. The framework can automate the patches, but the decision to advance comes from the people who own the business outcome of the change.

What this recipe explicitly does NOT defend against

Stateful migrations that aren't backward-compatible. If green requires a DB schema change blue doesn't understand, this recipe is necessary but not sufficient. Standard answer: backward-compatible migration → green deploy → forward-compatible migration.
Long-running workflows that span the cutover. A workflow started on blue that fires a timer after blue is scaled to 0 has nowhere to land. Use a durable timer store (@nwire/mongo or @nwire/drizzle with a Postgres-backed scheduler) and the timer fires on green.
In-memory caches. Won't transfer to green. Either warm green's caches during step 2 or use a shared cache (Redis).
Bus / queue consumers. Both colors connected to the same NATS / SQS will round-robin. If green is broken, half your queue work fails. Add a per-color subscription gate or pause green's consumers during smoke.

Reference: the proof data

Test	Path	RPS sustained	Reqs	Failures	Fail rate
Selector cutover, port-forward	kube-proxy / Cilium only	30	1,801	0	0.000%
Selector cutover, port-forward heavy	kube-proxy / Cilium only	775	46,542	0	0.000%
Selector cutover, real GCLB	laptop → GCLB → NEG → pod	415	24,929	9,218	36.97% ❌
Weighted cutover, real GCLB (this recipe)	laptop → GCLB → NEG → pod	277	27,750	9	0.03% ✅

Full numbers, methodology, k6 scripts, and a "what was wrong with my first conclusion" post-mortem live in docs/internals/blue-green-cutover-proof.md.

Verifying the manifests

infra/k8s/blue-green/__tests__/cutover.test.ts runs in pnpm test. It asserts:

All manifests have valid apiVersion / kind / metadata.name.
Probe paths match what lightship actually serves (/live, /ready).
terminationGracePeriodSeconds ≥ drainDelay + drainTimeout.
Blue and green Deployments differ only in name, version label, image tag.
Both color Services exist with matching color selectors.
HTTPRoute references both color Services with explicit weights summing to 100.
HealthCheckPolicy uses USE_FIXED_PORT (not USE_NAMED_PORT) for NEG compatibility.

Set RUN_K8S_E2E=1 to additionally run kubectl apply --dry-run=server against a live API server.

Production hardening checklist

[ ] Replace :blue / :green tags with SHA-pinned tags (:sha-abc123-blue, etc.) — never re-use a tag.
[ ] Add a PodDisruptionBudget per color: minAvailable: 1.
[ ] Add HorizontalPodAutoscaler per color (idle green at min replicas is cheap insurance).
[ ] Wire OTel (recipe) — watch the cutover in traces. The runtime.dispatch span tree should show envelopes finishing on the old color and starting on the new color, no orphans.
[ ] Automate the cutover script (a shell wrapper around the four steps, with wait_for_dashboard_green between weight steps).
[ ] Synthetic monitor on the public hostname so the 10% ramp window has a real client driving it.

Zero-downtime blue/green on Kubernetes — proven recipe ​

What you get over a rolling update ​

The five ingredients (all required) ​

Why the weighted pattern works and the selector pattern doesn't ​

Initial deploy ​

Cutover blue → green — the four-step ritual ​

Step 1. Roll green to the new image, keep weight at 0 ​

Step 2. Smoke green privately via port-forward (mandatory) ​

Step 3. Shift weight in steps (NOT all-at-once) ​

Step 4. Drain blue, then scale to 0 — DO NOT skip the drain wait ​

Rollback — green → blue ​

The bad-image scenario — what protects you, exactly ​

Defense 1 — green never gets traffic until you say so (step 2 catches almost everything) ​

Defense 2 — GCLB health checks fence off unhealthy backends (step 3 catches partial breakage) ​

Defense 3 — gradual weight ramp with dashboard watching (step 3 with sleeps catches everything else) ​

What this recipe explicitly does NOT defend against ​

Reference: the proof data ​

Verifying the manifests ​

Production hardening checklist ​

See also ​

Zero-downtime blue/green on Kubernetes — proven recipe

What you get over a rolling update

The five ingredients (all required)

Why the weighted pattern works and the selector pattern doesn't

Initial deploy

Cutover blue → green — the four-step ritual

Step 1. Roll green to the new image, keep weight at 0

Step 2. Smoke green privately via port-forward (mandatory)

Step 3. Shift weight in steps (NOT all-at-once)

Step 4. Drain blue, then scale to 0 — DO NOT skip the drain wait

Rollback — green → blue

The bad-image scenario — what protects you, exactly

Defense 1 — green never gets traffic until you say so (step 2 catches almost everything)

Defense 2 — GCLB health checks fence off unhealthy backends (step 3 catches partial breakage)

Defense 3 — gradual weight ramp with dashboard watching (step 3 with sleeps catches everything else)

What this recipe explicitly does NOT defend against

Reference: the proof data

Verifying the manifests

Production hardening checklist

See also