From Zero to Canary: Engineering a Real CI/CD Pipeline

I recently spent two hours in front of a room of engineers, live in VS Code, deliberately breaking a test, just to watch a pipeline catch it and stop a bad deploy in its tracks. That one moment, more than any slide, is usually what makes CI/CD click for people. Not the definition. The consequence.

I've built and taught this stack enough times, as a full-stack engineer shipping real products, and as a mentor walking newer developers through it from scratch, that I want to write down how I actually think about it. Not the marketing version ("ship faster with confidence"). The engineering version: what each layer of a pipeline is actually protecting against, and where the real decisions live.

The Three Letters People Mix Up

CI, Continuous Delivery, and Continuous Deployment get treated as synonyms. They're not, they're three different bets on how much you trust your own automation.

• Continuous Integration only answers one question: is this code correct? It checks out the code, installs dependencies, lints, tests, and builds, on every push. It never touches production. If your pipeline stops here, you have verification, not release.
• Continuous Delivery goes further: the app is always in a deployable state, but a human still presses the button. This is where most regulated or enterprise systems sit, QA, a lead, a product owner all sign off before code reaches users.
• Continuous Deployment removes that button entirely. Tests pass → code ships. No approval step. This only works if your test suite is good enough that you trust it more than you trust a human double-check, which is a harder bar than most teams admit. The distinction matters because picking the wrong one for your context is a real failure mode. A fintech compliance system probably shouldn't skip the human gate. An internal tool with a solid test suite probably shouldn't have one.

What's Actually Running Under the Hood

A GitHub Actions pipeline is just four ideas stacked on top of each other:

• A trigger decides when it runs, push, pull_request, a cron schedule, or a manual workflow_dispatch.
• A job is a unit of work that gets its own disposable virtual machine.
• Steps run in order inside a job, checkout, install, test, build.
• A matrix lets you run the same job multiple times with different inputs, in parallel, for free. That last one is where a lot of engineers stop too early. Here's a matrix that tests against three Node versions simultaneously, not three separate workflow files, one:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [18, 20, 22]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm install
      - run: npm test

If Node 18 fails and 20/22 pass, you get a precise, isolated failure, not a vague "something broke." That precision is the entire point of CI: turning "it's broken" into "line 42 of utils.js breaks under Node 18."

Testing in Layers, on Purpose

The order of test execution isn't arbitrary, it's a cost curve. I run unit tests first because they're cheap and catch the most common failures in seconds. Integration tests run next, checking that modules actually talk to each other correctly (a passing unit test tells you nothing about whether your API can reach your database). End-to-end tests run last, because they're slow and expensive, they simulate an actual user, browser included, and I want to fail fast on the cheap checks before paying for the expensive ones.

git push → unit tests → integration tests → e2e tests → build → deploy

If a stage fails, everything after it stops. That's deliberate, there's no reason to spin up a full browser session to test a login flow if a unit test already proved the password hashing function is broken.

Secrets Don't Belong in Code, Ever

This one is non-negotiable, and I've seen it get violated more than any other practice on this list. An API key in a .env file that gets committed is not "safe because I'll remove it later". Git history doesn't forget. The fix is boring and effective: pull secrets from the platform's encrypted store at runtime.

env:
  API_KEY: ${{ secrets.API_KEY }}

GitHub encrypts it, masks it in logs automatically, and scopes it to the workflow run. No custom tooling required , just discipline about never typing a real credential into a source file.

Where the Real Engineering Judgment Lives: Deployment Strategy

Anyone can write a workflow that runs npm test. The part that separates a pipeline that looks good from one that survives a bad release is how you push new code to real users. I think about it as picking how much blast radius you're willing to expose at once.

Rolling deployment replaces instances gradually, one server updates while others keep serving traffic. It's Kubernetes' default for a reason: no extra infrastructure, no downtime, and if you're already running multiple replicas, it's nearly free.

Blue-green deployment keeps two full environmentsone live, one idle. You deploy to the idle one, test it fully in isolation, then flip traffic instantly:

upstream app {
  server green-server:3000;
}

nginx -s reload

The appeal is the rollback story: if green misbehaves, you switch back to blue in seconds. The cost is that you're paying for double the infrastructure, most of the time, for that safety net.

Canary deployment is the one I reach for when the stakes are highest. New code goes to a small slice of real traffic 5%, sometimes less, while I watch error rate, latency, and logs closely. If it's healthy, the percentage climbs. If it's not, I roll back before most users ever touch the new version. This is what Netflix, Google, and Amazon default to, and it's not caution for its own sake it's the correct response to the fact that at their scale, even a 0.1% bug affects real people.

None of these is "the best" strategy in the abstract. Rolling is the pragmatic default. Blue-green is the right call when you need an instant, guaranteed rollback and can afford the infrastructure. Canary is the right call when the cost of a bad release, even briefly, is genuinely high. Knowing which one a system needs, not just how to configure any of them is the actual skill.

The Repo, If You Want to Run It Yourself

I don't think pipelines are something you fully understand from reading about them, I didn't, until I broke a few in front of an audience and had to explain why, live. So I open-sourced the three example pipelines I built for a recent CI/CD mentoring session: a minimal single-job CI setup, a three-environment pipeline with matrix builds and manual approval gates, and a full unit → integration → e2e testing pyramid wired as sequential jobs.

github.com/umar-aziz-dev/DW-Session-CI-CD-Pipeline-Engineering

Clone it, break a test on purpose, watch the pipeline stop it. That exercise teaches more in five minutes than most articles do in five pages including, probably, this one.