Speeding up Ubuntu Docker builds with podman

27 Feb 2020

Hacking around with Docker images on a laptop is a productive way to do experimental development but if your building Ubuntu images locally and have a lot of packages to apt install your going to be stuck waiting for downloads if you need to change something within a single layer.

If your living the dream and hanging out on cafe WIFI this can be a big deal.

A really simple fix for this would be to mount /var/cache/apt as a volume so that files can be cached on the host. Unfortunately support for docker build -v ... doesn’t exist yet:

Introducing `podman`

podman is a drop-in replacement for the docker command which brings many new features and Kubernetes integration. One of these features is… podman build support for volumes :)

There’s a great blog post from Redhat that walks docker users through how podman and its companion buildah fit together.

Follow the podman installation guide when your ready to try it out.

Speeding up Ubuntu image building

With podman installed, lets take a look at how we can plug-in a cache directory for Docker builds.

apt uses /var/cache/apt as its cache directory, so if we wanted to cache *.debs at ~/podman_apt_cache our podman build command becomes something like:

mkdir -p ~/podman_apt_cache
podman build -v ~/podman_apt_cache:/var/cache/apt .

Real world performance

With around 40 packages to install over installed

docker build

$ time docker build .  > /dev/null

real	5m52.357s
user	0m0.338s
sys	0m0.189s

podman build

$ time podman build -v ~/podman_apt_cache:/var/cache/apt . > /dev/null

real	2m8.366s
user	2m5.982s
sys	0m27.275s

Summary

Build time reduced by around 63%!

Caveats/troubleshooting

No speed increase

The stock ubuntu image cleans up /var/cache/apt automatically. To fix this we just need to tell apt not to clean the cache by adding:

RUN rm /etc/apt/apt.conf.d/docker-clean

To our Dockerfile.

Could not get lock /var/lib/apt/lists/lock

If a build was interrupted the lock file will still be present. Since we’re running apt inside a container it safe to just remove the file:

rm ~/podman_apt_cache/archives/lock

The package cache file is corrupted

Another error that can be caused by interrupted builds is:

E: Problem renaming the file /var/cache/apt/srcpkgcache.bin.iKB2BL to /var/cache/apt/srcpkgcache.bin - rename (2: No such file or directory)
W: You may want to run apt-get update to correct these problems
E: The package cache file is corrupted

The fix for this is to just cleanup the files:

rm ~/podman_apt_cache/pkgcache.bin
rm ~/podman_apt_cache/srcpkgcache.bin

Shared builds

Sharing the apt cache between concurrent builds won’t work properly because apt expects exclusive access. Since this article is geared around solo developers working on occasional builds on a laptop, this can be a problem for another day.

Putting it all together

Dockerfile

Our Dockerfile should look something like this:

FROM ubuntu:eoan-20200114

# stop package cleanup on exit
RUN rm /etc/apt/apt.conf.d/docker-clean

ENV DEBIAN_FRONTEND 'noninteractive'
RUN apt-get update && apt-get install -y --no-install-recommends \
  your package list here...
  && rm -rf /var/lib/apt/lists/*

Having to type out all the commands and remember the workarounds for corrupt files isn’t fun, so lets create a Makefile to simply things:

Makefile

build:
	mkdir -p ~/podman_apt_cache
	podman build -v ~/podman_apt_cache:/var/cache/apt .
clean:
	rm -f ~/podman_apt_cache/pkgcache.bin
	rm -f ~/podman_apt_cache/srcpkgcache.bin
	rm -f ~/podman_apt_cache/archives/lock

Now we can build our image by typing:

make

And if we encounter corruption errors we can fix them by typing:

make clean

Conclusion

Caching external downloads between builds saves both time and data so podman is a great way to speed up builds and prepare your systems for better Kubernetes integration.

Need help optimising something else? Contact us!

geoffwilliams@home:~$