Self-contained multi-stage Dockerfiles are rad

dockerscalascala.jsfly.iodeployopsnginx

In the last few years I pivoted my personal projects from mostly bad libraries to mostly bad full stack applications. That profound transformation was in no small part due to discovering Fly.io, its excellent CLI fly, and its excellent Docker support. You see, when you run fly launch or fly deploy in a folder that contains a Dockerfile, fly CLI will automatically build the container and deploy it directly on Fly.io infrastructure. This had all the delicious signs of the cloud computing nirvana that was promised to me 20 years ago when I had to phone an actual, meat and bone person merely to reboot my collocated server in an actual data centre.

Docker containers as first class packaging mechanism have great appeal – they erase the need for various tailored solutions for the hundreds of combinations with different stacks and languages, exposing very little to the world, save for network ports. Fly.io promises that if you can package your application as a docker container, then they can deploy it to the world in a microVM. And for the most part, they hold up their end of the bargain. And even though I moved my favourite apps away from Fly back to a dedicated server (for cost and performance reasons) with a Kubernetes cluster, I still think it's the correct way of thinking about deploying applications.

To gain entry to this deployment paradise, we need to make our Dockerfiles self-contained. To reduce the ironic burden of self-containment, our Dockerfiles should be multi-stage (where it makes sense).

Let's go.

TLDR

  • Self-contained multi-stage dockerfiles are great
  • We test them with a full stack Scala application
  • Beginner to intermediate level assumed, and/or people less familiar with Docker
  • Github repo

Self-contained Dockerfiles

To avoid any doubts, when I say self-contained, I mean the following:

On a clean system with nothing but the dependencies of the docker runtime installed, a single run of the command docker build . -t my-app, ran in the folder where the application code is checked out, produces a runnable Docker image my-app which fully satisfies the functional requirements of the application, allowing for configuration to happen only through the means of environment variables.

This definition means that all of the necessary runtime dependencies should be already packaged into the docker container. Configuration via environment variables allows for communication with service-like dependencies that make no sense inside the application container – for example, it makes no sense to package our app with its own Postgres process, but instead we can make it read the DATABASE_URL environment variable and use that to connect to a Postgres instance or cluster running somewhere else.

Let's see a couple of examples. All of the application code here is written using Scala 3, which we build and package using Scala CLI.

Backend service

Let's start simple – we want to build a docker container that exposes a single HTTP endpoint in a Scala web service.

The code for the service is quite simple – I'm using http4s here, as that's one of the several main HTTP libraries you will find across Scala companies, and the one I have the most muscle memory associated with:

//> using dep org.http4s::http4s-ember-server::0.23.32
//> using dep org.http4s::http4s-dsl::0.23.32
//> using scala 3.7

import cats.effect.*
import org.http4s.*, dsl.io.*, org.http4s.implicits.*
import com.comcast.ip4s.*
import org.http4s.ember.server.EmberServerBuilder
import concurrent.duration.*

object Hello extends IOApp:
  override def run(args: List[String]): IO[ExitCode] =
    val host =
      args.headOption.flatMap(Host.fromString(_)).getOrElse(host"0.0.0.0")

    val port =
      args.tail.headOption.flatMap(Port.fromString(_)).getOrElse(port"8080")

    val routes = HttpApp[IO]:
      case GET -> Root / "api" / "hello" =>
        Ok("Hello, World!")
      case req =>
        NotFound(s"${req.uri} not found")

    EmberServerBuilder
      .default[IO]
      .withHost(host)
      .withPort(port)
      .withHttpApp(routes)
      .withShutdownTimeout(0.seconds)
      .build
      .evalTap(server =>
        IO.consoleForIO.errorln(s"Running server on ${server.baseUri}")
      )
      .useForever

The container building process should be self-contained, so our Dockerfile starts from the most basic JVM image and installing Scala CLI

FROM eclipse-temurin:24

RUN apt update && apt install -y curl
RUN curl https://raw.githubusercontent.com/VirtusLab/scala-cli/refs/heads/main/scala-cli.sh > /bin/scala && \
    chmod +x /bin/scala && \
    scala config power true && \
    scala run -e 'println("hello")' -S 3.7 --server=false

So far nothing special – just install the Scala CLI launcher script and put it in /bin as scala. One important line here is

scala run -e 'println("hello")' -S 3.7 --server=false

Why do we run something here instead of just calling scala --version to check that it works? Well, because the commands we ran so far don't depend on any files in our project (e.g. we don't use ADD/COPY commands), Docker builder will cache the filesystem results produced by those commands, and then not run them again until we change the commands themselves.

So it's a good opportunity to force Scala CLI to do something useful and download the compiler and various other things it needs to run our app. Doing it now means not doing it later – which is a secret, of course, all successful people know. For the rest of us, procrastination is one of the few human things we haven't outsourced to the machines.

Note that I'm passing --server=false to Scala CLI – this is because docker build does not have a notion of long lived processes, so there is no point wasting time downloading dependencies Scala CLI needs for Bloop (the build server)

At this point we have everything we need to introduce our code to the equation:

WORKDIR /workdir
COPY backend.scala .
RUN mkdir -p /app && scala package . -f -o /app/backend --assembly --server=false

Nothing special here either, just basic packaging with Scala CLI, at the end of which /app/backend will be an executable file that will launch our service. Note that I'm using --assembly – to make sure that all the jars necessary for the application are downloaded locally and during packaging, instead of being downloaded on applications' first start up, which is the default mode for package subcommand.

All that is left is to let Docker know which command to run when the container starts:

EXPOSE 8080
CMD [ "/app/backend", "0.0.0.0", "8080" ]

That's it! Packaging our application is now just docker build . -t blog-dockerfiles-1. Try it yourself in the repository.

The resulting image is 750MB, with eclipse-temurin:24 being a whopping 425MB.

One thing you might notice immediately is that scala command (and all its dependencies) is still installed in the container. This is due to the build process being linear and additive, where subsequent commands append to the filesystem state produced by the previous ones. We have no need for it – the packaged application is an über jar, which only requires JVM runtime and JDK libraries, and nothing else.

This clearly separates the environment in which we build our application into something runnable, and the environment in which the application runs. Usually the runtime environment is much slimmer than the build one.

Let's see how we can express this with Dockerfiles.

Multi-stage Dockerfiles

For quite some time now, Docker supported separated build steps into several different stages, that can copy files from each other, and continue where the previous one left off, with the engine figuring out the graph of dependencies between stages, and running them in parallel.

The official docs do a great job demonstrating some of the basics, so let's adapt our stack of choice to the multi-stage docker format.

Backend service

We start our Dockerfile in a similar way, but this time we give the build stage a name, one we can refer to later:

FROM eclipse-temurin:24 as build

Scala CLI is installed in the same way, so we'll skip that.

The application will be built using a slightly different command – I want to use a reduced size Alpine image, which doesn't even contain bash. You might be thinking – "but we don't use bash anywhere?". It's the correct first reaction – until you realise that the nice runnable files Scala CLI produces for us are actually specially crafted bash scripts with a preamble and the compiled bytecode appended to the end.

Let's ask Scala CLI to just produce a JAR file, by passing --preamble=false:

WORKDIR /workdir
COPY backend.scala .
RUN mkdir -p /app && scala package . -f -o ./backend \
    --preamble=false \
    --assembly --server=false

Now, the rest of the commands contain yet more changes. First of all, let's start a new build stage – one based on a different image:

FROM eclipse-temurin:24-alpine

Then, we copy our build JAR from the build stage:

COPY --from=build /workdir/backend /app/backend

And finally, we can run it just using java command:

EXPOSE 8080
CMD [ "java", "-jar", "/app/backend", "0.0.0.0", "8080" ]

That's it! Yet again, packaging our application is just docker build . -t blog-dockerfiles-2. Try it yourself in the repository.

This time the docker image is only 346MB, but the added bonus is that if our runtime stage required some more dependencies (apart from built application binary), Docker engine could've run those build steps in parallel with other stages, saving quite a bit of time.

Fullstack application

Let's take things a bit further now. A good sign of a serious application is gratuitous usage of NGINX – which is exactly what we are going to do.

Our goal will be a full stack Scala application, frontend by NGINX responsible for proxying API requests and serving static files.

Let's modify the backend ever so slightly, to output the current timestamp on each request:

val routes = HttpApp[IO]:
  case GET -> Root / "api" / "hello" =>
    IO.realTimeInstant.flatMap: time =>
      Ok(s"Hello, World! The time is $time")

Our frontend will be very simple (if you are familiar with the excellent Laminar library), invoking this endpoint every 500ms, and rendering the response raw:

//> using scala 3.7
//> using platform js
//> using dep com.raquo::laminar::17.2.1

import com.raquo.laminar.api.L.*

@main def frontend =
  renderOnDomContentLoaded(
    org.scalajs.dom.document.getElementById("content"),
    div(
      child.text <--
        EventStream
          .periodic(500)
          .flatMapSwitch: _ =>
            FetchStream.get("/api/hello")
    )
  )

Notice that unlike most other full stack Scala projects, we are not using shared code here, as our interaction is trivial. For really anything more complex, I'd recommend expressing your backend endpoints in some shared code, e.g. via Tapir directly, or through Smithy4s code generation.

For our application we also need a HTML entry point, which defines both the container to mount the app into, and references the generated JS file:

<!doctype html>
<html>
    <head>
        <meta charset="UTF-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <title>Scala.js Fullstack App</title>
    </head>
    <body>
        <div id="content"></div>
        <script src="/frontend.js"></script>
    </body>
</html>

Let's move on to the build itself.

Parallel stage builds

This time, we will start by installing Node.js first. Our application doesn't really need it, but once you start adding JS dependencies and using a bundler (such as Vite), you will be happy to have seen this snippet for installing Node.js I spent hours on (seriously, I tried so many other solutions):

FROM eclipse-temurin:24 AS scala-cli

WORKDIR /workdir
RUN apt update && apt install -y curl xz-utils
RUN if [ "$(uname -m)" = "x86_64" ]; then \
    curl -Lo node-install.tar.xz https://nodejs.org/dist/v22.18.0/node-v22.18.0-linux-x64.tar.xz; \
    else \
    curl -Lo node-install.tar.xz https://nodejs.org/dist/v22.18.0/node-v22.18.0-linux-arm64.tar.xz; \
    fi && \
    tar -xf node-install.tar.xz && rm *.tar.xz && mv node-v22* node-install
ENV PATH=/workdir/node-install/bin:$PATH

With Node installed, we install Scala CLI just like before, but this time we will also run println("hello") using Scala.js (which will require Node.js) – this will ensure that all the Scala.js dependencies are in the disk cache and working properly.

RUN curl https://raw.githubusercontent.com/VirtusLab/scala-cli/refs/heads/main/scala-cli.sh > /bin/scala && \
    chmod +x /bin/scala && \
    scala config power true && \
    scala run -e 'println("hello")' -S 3.7 --server=false && \
    scala run -e 'println("hello")' -S 3.7 --js --server=false # sic!

We are now ready to build the application – and we will do so in two stages running in parallel, one for backend and one for frontend. To do so, we just use separate named FROM directives for each of the stages.

One thing to note is by introducing named stages we're complicating things somewhat as we will need to explicitly copy the files of interest into the runtime stage – but I think this inconvenience is nothing in comparison to the wins you get in parallel building.

Apart from the stage names, backend and frontend builds look the same:

FROM scala-cli AS backend-build
WORKDIR /workdir
COPY backend/ backend/
RUN mkdir -p /app && scala package backend/ -f -o /app/backend \
    --preamble=false \
    --assembly --server=false


FROM scala-cli AS frontend-build
WORKDIR /workdir
COPY frontend/ frontend/
COPY index.html /app/assets/index.html
RUN scala package frontend/ -f -o /app/assets/frontend.js --js-mode release --server=false

FROM scala-cli will ensure that both stages will share the disk cache we set up during the Scala CLI installation stage earlier, but any artifacts they produce will be kept in their respective stages, and not shared implicitly (though you can copy them explicitly, which comes soon).

Setting up NGINX

One thing we haven't agreed on yet is the actual NGINX configuration for running our project. Let's go over the requirements:

  1. NGINX should serve some static files from the location we give it
  2. All the requests to /api/* should be proxied to our backend service
  3. It should serve the index.html page at /, or any other path tht doesn't match the previous two rules

The rule for serving a limited range of static file is simple:

location ~* \.(js|css|html)$ {
    root /app/assets;
    try_files $uri =404;
}

Proxying /api/ requests (note that the port is hardcoded to 8080, which we should align with the server startup command):

location /api/ {
    proxy_pass http://localhost:8080;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

And finally, the last rule to serve index.html at all other requests:

location / {
    root /app/assets;
    try_files $uri /index.html;
}

Notice how we refer to localhost:8080, assuming that the web service is already running at that address.

This poses an interesting question – how do we run both the backend and NGINX as part of the same container, when containers generally contain a single command as the entrypoint?

After googling around, I found the solution that works for me – create a new entrypoint (in the for of a bash script) that starts both nginx and our backend service, and uses the built in shell mechanism to wait for all background processes to terminate before exiting.

The bash script is very simple, and there are variants of it all over the Internet, which makes me think it's not a crazy approach:

#!/usr/bin/env bash

set -xeuo pipefail

java -jar /app/backend localhost 8080 &
# capture the PID of the backend process
BACKEND_PID=$!

# Start nginx
nginx

# Trap to kill backend when this process exits
trap "kill $BACKEND_PID; nginx -s stop" EXIT

# Wait
wait

Building runtime container

Okay, nginx is configured, time to build the actual container we'll use for deployment. The plan is simple:

Start with a JDK base,

FROM eclipse-temurin:24
WORKDIR /workdir

install nginx and copy all configuration files for it,

RUN apt update && apt install -y nginx
COPY nginx.conf /etc/nginx/sites-available/default
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh

then copy backend and frontend files to locations that our nginx.conf expects,

COPY --from=backend-build /app/backend /app/backend
COPY --from=frontend-build /app/assets/ /app/assets/

and finally, set up the entrypoint to be the bash script we wrote earlier

EXPOSE 80

CMD [ "docker-entrypoint.sh" ]

And that's it, with all the added complexity, the container is only a single docker build . -t blog-dockerfiles-3 command away. Try it in repository.

Docker context hygiene

One frequent issue I encounter with this approach is accidental cache invalidation. Using COPY commands requires careful crafting of the docker context, to ensure that we only invalidate cache if the sources have changed, instead of some random files we forgot to exclude.

It happens too often that the .dockerignore file is not set up correctly and various files produced by build tools end up triggering cache invalidation.

To make sure that docker context is setup correctly, I've been using this magic command:

printf 'FROM busybox\nCOPY . /tmp/build\nRUN find ./tmp/build' | DOCKER_BUILDKIT=1 docker build -f- . --progress=plain --no-cache 2>&1

All it does is uses COPY . /tmp/build to copy what Docker sees as entire build context into /tmp/build, then list files there. Here's sample output from:

#7 [3/3] RUN find ./tmp/build
#7 0.148 ./tmp/build
#7 0.148 ./tmp/build/sample_docker_context.log
#7 0.148 ./tmp/build/backend.scala
#7 0.148 ./tmp/build/Dockerfile
#7 0.148 ./tmp/build/.metals
#7 0.148 ./tmp/build/.metals/.tmp
#7 0.148 ./tmp/build/.metals/metals.log
#7 0.148 ./tmp/build/.metals/metals.mv.db
#7 0.148 ./tmp/build/.bsp
#7 0.148 ./tmp/build/.bsp/scala-cli.json
#7 0.148 ./tmp/build/.scala-build
#7 0.148 ./tmp/build/.scala-build/ide-options-v2.json
#7 0.148 ./tmp/build/.scala-build/ide-envs.json
#7 0.148 ./tmp/build/.scala-build/ide-inputs.json
#7 0.148 ./tmp/build/.scala-build/ide-launcher-options.json
#7 0.148 ./tmp/build/.scala-build/.bloop
#7 0.148 ./tmp/build/.scala-build/.bloop/2-self-contained-multi-stage-backend_89225bb011.json

Oh-oh! The Scala CLI build folder and Metals state folder are both added to the build context. This means that we're just one careless COPY . . command away from the container being rebuilt any time anything changes in those folders (and they do change all the time). Let's pop them into .dockerignore:

**/.scala-build
**/.metals
**/.bsp
Dockerfile
.dockerignore
**/*.log

And check the context again:

#7 [3/3] RUN find ./tmp/build
#7 0.174 ./tmp/build
#7 0.174 ./tmp/build/backend.scala
#7 DONE 0.2s

Much better!

Good docker context hygiene is critical for faster builds.

Multi-platform Docker images on Github Actions

One nice consequence of the dockerfiles being self-contained is that the workflows that publish them are basically identical no matter what repository you have – you can basically copy one of my workflows into your project (with a self-contained Dockerfile), change the name of the repository and get

  1. Docker images built for both AMD64 and ARM64 platforms
  2. Pushed to https://ghcr.io registry
  3. Different platform images merged into a single manifest, making sure that you pull the correct image no matter what platform you are on
  4. A separate :buildcache image published to registry and used to dramatically speed up subsequent builds

Conclusion

What I have shown here is not novel by any means, but it's one of those subjects I think are worth re-iterating, as they make deployment and dependency setup much easier.

It's not without drawbacks, as we are delibrately throwing away any build tool caches to try and build the project in a somewhat isolated and reproducible environment – but for that price we get a lot of nice things and uniform build instructions, which I think is worth it.