Migrating from Kaniko to Docker in Cloud Build (for multi-stage caching)

Introduction

I use Cloud Build as GitOps to CI/CD my Rails application into Cloud Run.

The latest Rails generated a multi-stage Dockerfile, so I thought it was a good time to modernize my existing Docker builds to use multi-stage builds to reduce image size.

In my existing cloudbuild.yaml configuration, I was using kaniko for build caching. kaniko was previously mentioned in Google Cloud documentation as a way to speed up Cloud Build builds, but this mention is already gone from the current documentation.

Somehow kaniko doesn’t play nice with multi-stage builds, and it looks like an unsupported repo from Google Cloud, lacking future support (to be fair, it declares that “kaniko is not an officially supported Google product”). Specifically, I could not get kaniko to cache multi-stage builds properly to reduce build times to what I had before.

In this blog post, I show how we can revert back to the native and modern docker build command and remove kaniko as a build dependency to achieve build caching for multi-stage Dockerfiles.

Exploring Docker Build

Docker Build is Docker’s native command for building images. By default, the docker build command utilizes a default builder underneath that acts as the backend for builds. This default builder utilizes the docker build driver.

There are 4 types of build drivers:

• docker: uses the BuildKit library bundled into the Docker daemon.
• docker-container: creates a dedicated BuildKit container using Docker.
• kubernetes: creates BuildKit pods in a Kubernetes cluster.
• remote: connects directly to a manually managed BuildKit daemon.

In my case, I want to cache my images using Artifact Registry, so I need a build driver that plays nice with the registry cache type.

According to this page:

The default docker driver supports the inline, local, registry, and gha cache backends, but only if you have enabled the containerd image store.

According to the containerd link, it is used under the hood by Docker Desktop, but I think I need something else to build in a Cloud Build environment? (not sure what it uses)

According to this other page:

The docker driver only supports cache exports using the inline and local cache backends.

So it looks like docker-container is the driver to use instead, and it supports the registry cache. I could be wrong and maybe the docker driver can actually be used. But hey, I got it working.

Create a custom builder

Let’s then create a custom builder with name mybuilder (renamed to make it obvious) based on the docker-container driver:

docker buildx create --name mybuilder --driver docker-container

or the equivalent in cloudbuild.yaml:

# Create builder
- name: "docker"
  args:
    ["buildx", "create", "--name", "mybuilder", "--driver", "docker-container"]

You’ll likely need to use the docker image directly from Docker Hub, because the usual gcr.io/cloud-builders/docker is a few versions behind.

Build the image with the custom builder

Then, let’s use this custom builder to create our Docker image while utilizing caching:

# Build image
- name: "docker"
  args:
    [
      "build",
      "-t",
      "$_IMAGE_NAME:latest",
      "--cache-from",
      "type=registry,ref=${_IMAGE_NAME}/cache",
      "--cache-to",
      "type=registry,ref=${_IMAGE_NAME}/cache,mode=max",
      "--build-arg",
      "RAILS_ENV=${_RAILS_ENV}",
      "--builder",
      "mybuilder",
      "--load",
      ".",
    ]

Note using the builder created in the previous step: --builder mybuilder

The --load command is a shorthand for --output=type=docker which basically stores the built image locally. There is an --image option to make this command push the image directly to the registry but I found this to be a very slow operation. In my testing, it was faster to generate the image locally and then do a single manual push.

_IMAGE_NAME is set on the Cloud Build trigger to something like <REGION>-docker.pkg.dev/<PROJECT NAME>/<REPOSITORY NAME>/<PACKAGE NAME>, which is the Artifact Registry endpoint.

The mode=max exports layers for all stages, while mode=min on only exports layers already in the final build stage. So max it is for maximum build speedup.

Push the image

Then the usual docker push:

# Push image
- name: "docker"
  args: ["push", "$_IMAGE_NAME:latest"]

Summary

With this caching setup, I am able to bring my multi-stage build times back down to match those of my earlier kaniko-based builds.

Modern docker CLI supports multi-stage build caching, kaniko is no longer required (in my use case).