Ship infrastructure before services, with environment approvals

Roll out cluster and queue changes first, wait for the required reviewers on GitHub, and deploy the services that depend on them only once the change has been approved and has succeeded.

2 min readInfrastructureEnvironment approvalsDeploy window

Order matters most for infrastructure

A queue migration, a new cluster setting, a changed secret: the services that depend on them must not deploy first. When infrastructure and services live in different repositories, “first” is easy to get wrong, especially when someone has to approve the infrastructure change before it runs.

Make the dependency explicit

In a flow, the infrastructure box comes first and the worker that depends on it is connected on success. The worker only starts once the infrastructure deploy has finished and succeeded. If it fails, the worker is skipped, and nothing half-migrated goes out.

The cluster change waits for its reviewers; the worker waits for the cluster change.

Keep GitHub’s protection rules

Kodmos is a GitHub App: runs are started by the app, and your environment protection rules still apply. When a job targets a protected environment with required reviewers, the run waits on GitHub, and Kodmos shows the deployment as Needs approval with the rest of the chain waiting behind it. Approving on GitHub lets it carry on.

.github/workflows/apply.yml
name: Apply
on:
  workflow_dispatch:

jobs:
  apply:
    runs-on: ubuntu-latest
    # Required reviewers are set on this environment, in the repository's settings.
    environment: Production
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/apply.sh

Plan the change for a quiet window

Plan the change as an event with a deploy time, say 07:30, when traffic is low. The Platform team owns infra and worker, so they are the ones asked for branches and reminded before the deadline. Once the list locks, whoever is on at 07:30 only has to press Deploy; the environment’s reviewers approve the run on GitHub.

Home: the branches you owe, and what is deploying right now.Home: the branches you owe, and what is deploying right now.
Home: the branches you owe, and what is deploying right now.

Who hears what

Everyone involved gets an email when the deploy starts and another when it has finished, with each project’s result. DevOps also hears about failures, and every step stays in the event’s activity for the post-mortem nobody wants to write.

Try it on your next release.

Connect your GitHub organization, invite your team, and put the next release day on the calendar.