Series TL;DR
Introduction
Web server - NGINX Unit
- SNUnit - interface to NGINX Unit
Cloud deployment on Fly.io
Conclusion

Series TL;DR

We are deploying a Scala Native backend (with NGINX Unit) and Scala.js frontend (with Laminar) on Fly.io
We are using my SN binding generator
We are using Scala 3 heavily
Code on Github
Deployed app
Roach - postgres bindings and interface
Navigation:
- Previous: Part II - Postgres and OpenSSL
- Next: Part IV - Building the backend
- Full series

Introduction

While it's fun to play with so much new (to me) technology, the end goal of this project is to build a usable service with an automated deployment procedure and reasonable attention to reliability and API design.

If I were building a backend on the JVM, I would immediately reach out for Http4s because it's the stack I'm most familiar with and it has never let me down in a non-repairable way.

Alas, http4s is not available (yet) for Scala Native, and neither is Cats Effect. Part of what makes both of them really powerful is their ability to utilise system resources by using JVM's multi-threading. And multi-threading is not available on Scala Native at all.

So to run our APIs in a somewhat scalable fashion, we have to look elsewhere - process-based concurrency and an external web server. Enter NGINX Unit.

Web server - NGINX Unit

Advertised as a polyglot application web server, Unit is designed to run applications written in any number of languages, serve static files, handle instant responses and redirects, etc.

At the core, Unit is a single stateful process, to which you can submit the configuration for various listeners that will be handling TCP requests on particular ports.

Listeners can pass the requests to routers of varying complexity, and you can match on the various request attributes to redirect it to correct handler.

The way this configuration is updated is demonstrated here:

A simple configuration for a single-file app would look like this:

local_config.json

{
  "listeners": {
    "*:8080": {
      "pass": "routes"
    }
  },
  "routes": [
    {
      "match": {
        "uri": "/api/*"
      },
      "action": {
        "pass": "applications/app"
      }
    }
  ],
  "applications": {
    "app": {
      "processes": {
        "max": 5,
        "spare": 2,
        "idle_timeout": 180
      },
      "type": "external",
      "executable": "/usr/bin/twotm8",
    }
  }
}

Let's break it down:

We instruct Unit to listen on TCP port 8080, and pass all the traffic to the routes handler
routes themselves match on the uri, and if it's a /api/*, pass the traffic to app handled under applications
app itself is an external handler, which is specified as a path to executable, and optional configuration about the number of processes. In this example we want maximum of 5 processes (under load), with 2 processes being kept around for at most idle_timeout (in seconds) amount of time without any requests.

If you have Unit installed and unit process running, then you should be able to submit this configuration to it with a simple cURL command:

curl -X PUT --data-binary @local_config.json \
     --unix-socket /opt/homebrew/var/run/unit/control.sock \
     http://localhost/config/

Note the --unix-socket argument - Unit doesn't have its own exposed HTTP-over-TCP endpoint, the communication happens over a unix socket.

Thankfully, containerised version of Unit allows you to put configuration JSON files in a special location (/docker-entrypoint.d/) from which they will be read and applied on startup.

I felt like the container-focused nature of Fly.io as a cloud provider, and NGINX Unit shipping pre-made containers was a good combination to explore.

When deployed, our infrastructure will look pretty simple:

For Unit to correctly pass requests to our app and process the responses from it, we could integrate with Unit's C API, but thankfully someone has already done this.

SNUnit - interface to NGINX Unit

This project contains both the utilities necessary to handle Unit requests and responses, but also integration with trail library for basic matching on URLs - something we will use to make the definitions of our routes more readable.

The README covers the full setup for a minimalistic SNUnit application, so we won't be replicating it here in full, but instead let's extend it with a few helper methods, to make our endpoint definitions more uniform once we start writing the app logic itself.

At its core, the raw SNUnit API is very simple - request handlers implement this interface:

trait Handler:
  def handleRequest(req: Request): Unit

And snunit-routes module adds another interface:

trait ArgsHandler[Args]:
  def handleRequest(req: Request, args: Args): Unit

And it's intended to be used with the route constructors from the Trail library, e.g. a route like this:

import trail.*
Root / "api" / Arg[String]

Represents a value of type Route[String], which can parse an incoming URL and extract the correct arguments in correct positions.

Unfortunately, the library itself is not published for Scala 3 (I've added a PR to remedy that, but it might take a while for many reasons), but thankfully we can use the 2.13 version of it with just a little bit of build setup.

Scala 2.13 libraries in Scala 3

The possibility of using Scala 2.13 libraries with Scala 3 is a controversial feature. Among the library maintainers, it's frowned upon and avoided - it introduces conflicting transitive dependencies in the library dependency tree.

On the other hand, for applications (that are not used as dependency of any other projects, i.e. they're the leaves in the tree) it's a good proposition - as long as the desired library doesn't use Scala 2 macros - they will not work in a Scala 3 project. This is one of the reasons why it's impossible to use a 2.13 version of ScalaTest. Needless to say, the tapestry of Scala versions is as rich as ever, but that shouldn't stop us.

To add both SNunit and the optional snunit-routes module to our project, we only need to give the routes module special treatment, because snunit itself is published for all Scala versions:

build.sbt

libraryDependencies += "com.github.lolgab" %%% "snunit" % Versions.SNUnit,
libraryDependencies += (
  "com.github.lolgab" %%% "snunit-routes" % Versions.SNUnit cross CrossVersion.for3Use2_13
  ).excludeAll(
    ExclusionRule("org.scala-native"),
    ExclusionRule("com.github.lolgab", "snunit_native0.4_2.13")
  )

The cross CrossVersion.for3Use2_13 is what tells SBT to use the 2.13 artifact and not look for Scala 3 artifact at all.

The excludeAll part is necessary to avoid bringing in snunit itself transitively, but for Scala 2.13. Otherwise SNUnit will righteously complain:

[error] Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("file:/..."), "demoApp"):
[error]    com.github.lolgab:snunit_native0.4 _3, _2.13

It also excludes all of the modules from Scala Native's standard library, for exactly the same reason.

With this in mind, we should have no issues using the snunit-routes module.

Response helpers

When it comes to responses, the API provided is very simple:

trait Request:

  def send(statusCode: StatusCode, content: Array[Byte], headers: Seq[(String, String)]): Unit

  def send(statusCode: StatusCode, content: String, headers: Seq[(String, String)]): Unit

Let's extend this with a few helper methods:

extension (r: Request)
  inline def badRequest(
      content: String,
      headers: Map[String, String] = Map.empty
  ) =
    r.send(
      statusCode = StatusCode.BadRequest,
      content = content,
      headers = Seq.empty
    )
  inline def redirect(location: String) =
    r.send(
      StatusCode.TemporaryRedirect,
      content = "",
      headers = Seq("Location" -> location)
    )

  inline def noContent() =
    r.send(StatusCode.NoContent, "", Seq.empty)

  inline def unauthorized(inline msg: String = "Unathorized") =
    r.send(StatusCode.Unauthorized, msg, Seq.empty)

  inline def notFound(inline msg: String = "Not found") =
    r.send(StatusCode.NotFound, msg, Seq.empty)

  inline def serverError(inline msg: String = "Something broke yo") =
    r.send(StatusCode.InternalServerError, msg, Seq.empty)
end extension

Combining Trail routes

For my use, I want to have all the routes defined in a single place, in a readable, easy to understand way.

Because the SyncServerBuilder from SNUnit accepts a single Handler for all the possible requests, the pattern that is used in the snunit-routes is to define a RouteHandler, which takes a route, a handler for the route, and a fallback handler, in case the route doesn't match.

Instead of chaining the request handlers in a nested pattern, let's define a simple function to chain any number of handlers:

trait ApiHelpers:
  inline def builder(routes: (Route[?], ArgsHandler[?])*): Handler =
    routes.foldRight[Handler](_.notFound()) { case ((r, a), acc) =>
      RouteHandler(
        r.asInstanceOf[Route[Any]],
        a.asInstanceOf[ArgsHandler[Any]],
        acc
      )
    }

What this function will produce is a single Handler, that for each request will try all the routes in given order, and if none of them match, it will respond with a NotFound response.

Note that the routes themselves don't specify the HTTP method the handler will respond to. This means we need to add another helper, like this:

trait ApiHelpers:
  // ...
  inline def api(methods: (Method, Handler)*): Handler =
    val mp = methods.toMap

    request =>
      mp.get(request.method) match
        case None => request.notFound()
        case Some(h) =>
          h.handleRequest(request)
  end api

// ...
object ApiHelpers extends ApiHelpers

Put together, we can define full APIs like this:

import snunit.*
import snunit.routes.*
import trail.*

import ApiHelpers.*

@main def hello =
  // actual request handlers
  val handle_string: ArgsHandler[String] =
    (req, id) => req.send(StatusCode.OK, "hello", Seq.empty)

  val handle_index: ArgsHandler[Unit] =
    (req, id) => req.noContent()

  val handle_create: ArgsHandler[Unit] =
    (req, id) => req.send(StatusCode.OK, "idx", Seq.empty)

  // full API, combined into a single Handler
  val routes: Handler =
    api(
      Method.GET ->
        builder(
          Root / "api" / Arg[String] -> handle_string,
          Root / "api" / "index" -> handle_index
        ),
      Method.POST ->
        builder(
          Root / "api" / "create" -> handle_create
        )
    )
end hello

Note that because both ArgsHandler and Handler have only a single method interface, we can use the lambda syntax to define instances of these types.

Handling JSON payloads with Upickle

Our API will be predominantly JSON-based, and while I entertained the idea of binding to a library like cjson, the thought of writing another wrapper was depressing, mainly because the project itself turned out to already take 10 times more time that I planned.

So instead, we will be using uPickle which is thankfully published for all Scala versions and all the platforms. Let's add it to our build:

build.sbt

libraryDependencies += "com.lihaoyi" %%% "upickle" % "1.5.0",

There are two main operations we will need upickle for -

sending a JSON response and
conditionally accepting a JSON payload, as long as it's the correct shape

Sending JSON response is easy:

import upickle.default.{Writer, writeJs}

extension (r: Request)
  inline def sendJson[T: Writer](
      status: StatusCode,
      content: T,
      headers: Map[String, String] = Map.empty
  ) =
    r.send(
      statusCode = status,
      content = writeJs(content).render(),
      headers = headers.updated("Content-type", "application/json").toSeq
    )

The only restriction we put on the type T of response is that there's a upickle Writer instance for it.

Accepting JSON payloads is also simple, we just need to handle the exceptions that uPickle will throw if the payload is not valid json/not matching the shape we require.

import upickle.default.{Reader, read}

inline def json[T: upickle.default.Reader](inline request: Request)(
    inline f: T => Unit
) =
  val payload = Try(upickle.default.read[T](request.contentRaw))

  payload match
    case Success(p) => f(p)
    case Failure(ex) =>
      scribe.error(
        s"Error handling JSON request at <${request.method.name}> ${request.path}",
        ex
      )
      request.badRequest("Invalid payload")
end json

Note that we log error using the excellent Scribe logging library which was conveniently released for Scala 3 Native right at the time when I needed it most ;-)

It can be added to our build like this:

libraryDependencies += "com.outr" %%% "scribe" % "3.8.1"

This function will allow us to define handlers like this:

private val sum_numbers: ArgsHandler[Unit] = (req, i) =>
  json[Vector[Int]](req) { numbers => 
    val sum = numbers.sum

    req.sendJson(StatusCode.OK, Map("result" -> sum))
  }

I've chosen Vector[Int] as input and Map[String, Int] because uPickle will contain built-in JSON codecs for these types.

For our actual app, we will be defining payloads as typesafe case classes and will use built-in codec derivation mechanisms.

Global error handling

One last issue we will deal with is for exception handling. Currently, if there's an exception thrown in any of our handlers, it will be bubbled up until the application itself quits abnormally.

This is not what we want - transient errors shouldn't shut down the application. Even though NGINX Unit will re-route the request if the application crashed, we shouldn't take out instances willy-nilly - they are generally a scarce resource, there's much fewer instances in comparison to the number of requests they're supposed to be handling.

With that said, there's one exception we identified that should terminate the application (and slate it for a restart by Unit), and that would be an exception caused by Postgres connection being terminated by the server - we've decided in the previous installment is that reconstructing the exact state of our session is costly and error-prone, so instead we will take advantage of application fast startup and reconnect to postgres then.

With that in mind, let's define a special handler combinator:

inline def handleException(inline handler: Handler): Handler = req =>
  try handler.handleRequest(req)
  catch
    case exc: RoachFatalException =>
      scribe.error(
        s"Failed request at <${req.method.name} ${req.path}> " + 
        "because of postgres, killing the app",
        exc
      )
      req.send(StatusCode.ServiceUnavailable, "", Seq.empty)
      throw exc
    case exc =>
      scribe.error(s"Failed request at <${req.method.name} ${req.path}>", exc)
      req.serverError("Something broke yo")

For fatal exceptions coming from out Postgres layer, we re-throw the exception after logging it. It's important for us to send the response still, otherwise the connection will be left hanging.

For all other exceptions, we log the exception itself and minimal request information for future debugging.

This allows us to wrap our routes in this logic:

val routes: Handler =
  handleException(
    api(
      Method.GET ->
        builder(
          Root / "api" / Arg[String] -> handle_string,
          Root / "api" / "index" -> handle_index
        ),
      Method.POST ->
        builder(
          Root / "api" / "create" -> handle_create
        )
    )
  )

And with this we have enough API helpers to actually focus on application logic, in the next installment.

But before that, let's see how our Unit application can actually be deployed on our cloud provider of choice - Fly.io

Cloud deployment on Fly.io

The particular deployment strategy that we will use is Deploying via Dockerfile.

The promise of this service is simple:

Install the flyctl CLI
Have your app's Dockerfile ready
Run flyctl launch and answer some of the questions, such as app name

And if you're correctly authenticated, then the Dockerfile and the local folder contents will be sent to one of Fly's remote builder servers, where the app will be built from the Dockerfile, and deployed to Fly's infrastructure.

And indeed, after mucking about just a little with exposed ports in Fly configuration, the experience amounted pretty much to what was promised, which I'm very impressed by.

Docker build

The runtime container for our application will be based on NGINX Unit's official docker image. But to build the application from sources we will need many other things (namely Libpq, OpenSSL, Unit's development libraries, LLVM for Scala Native, JDK for SBT, etc.), which will balloon our docker image with things we don't need at runtime.

Instead, we will use Docker's multi-staged build functionality. First stage (builder) will be the container compiling and linking our code, producing a single binary. The second stage will be the runtime container, and it will copy the linked binary from the builder stage.

App build container

Let's iterate the tools we will need in the builder container:

We need JDK for the Scala compiler and SBT
We need SBT itself
We need LLVM 13 for both the binding generator and Scala Native
- Binding generator will also need libclang to generate bindings
We need a few development libraries for binary to be linked:
- libpq-dev for postgres
- libssl-dev for OpenSSL
- unit-dev for NGINX Unit

As the base of our build image, we will use the official OpenJDK build, based on Debian 11.

Here's the commands for the builder stage:

FROM openjdk:17-bullseye as builder

RUN apt update && \
    apt install -y curl && \
    # install SBT
    curl -Lo /usr/bin/sbt https://raw.githubusercontent.com/sbt/sbt/v1.6.2/sbt && \
    chmod +x /usr/bin/sbt &&\
    # install LLVM installer dependencies
    apt install -y lsb-release wget software-properties-common && \
    wget https://apt.llvm.org/llvm.sh && \
    chmod +x llvm.sh && \
    # install LLVM 13
    ./llvm.sh 13 && \
    apt install -y libclang-13-dev &&\
    # install libpq for postgres
    apt install -y libpq-dev && \
    # install Unit, OpenSSL and Unit development headers
    curl --output /usr/share/keyrings/nginx-keyring.gpg  \
      https://unit.nginx.org/keys/nginx-keyring.gpg && \
    echo "deb [signed-by=/usr/share/keyrings/nginx-keyring.gpg] https://packages.nginx.org/unit/debian/ bullseye unit" >> /etc/apt/sources.list.d/unit.list && \
    echo "deb-src [signed-by=/usr/share/keyrings/nginx-keyring.gpg] https://packages.nginx.org/unit/debian/ bullseye unit" >> /etc/apt/sources.list.d/unit.list && \
    apt update && \
    apt install -y unit-dev libssl-dev

# 1
ENV LLVM_BIN "/usr/lib/llvm-13/bin" 

# 2
ENV SN_RELEASE "fast"
ENV CI "true"

# 3 
COPY . /sources

# 4
RUN cd /sources && sbt clean buildApp

We're setting the special LLVM_BIN variable to be picked up by Scala Native - it will choose the correct version of LLVM toolchain
We're setting the SN_RELEASE variable in the docker build, which we will use to build the optimised version of the application during deployment - it's much slower than the defaults, so we use the variable to only do it during deployment.

To make it work in SBT, we need this configured on the app module:

build.sbt
```
 .settings(nativeConfig := {
   val conf = nativeConfig.value
   if (sys.env.get("SN_RELEASE").contains("fast"))
     conf.withOptimize(true).withLTO(LTO.thin).withMode(Mode.releaseFast)
   else conf
 })
```
We are copying the build context (sources and all) into the /sources folder inside the container

We run the buildApp SBT task, which is defined as follows:

build.sbt

 val buildApp = taskKey[Unit]("")
 build := {
   buildBackend.value
 }

 val buildBackend = taskKey[Unit]("")
 buildBackend := {
   val target = (app / Compile / nativeLink).value

   val destination = (ThisBuild / baseDirectory).value / "build"

   IO.copyFile(
     target,
     destination / "twotm8",
     preserveExecutable = true,
     preserveLastModified = true
   )
 }

All this task is doing is building our backend (nativeLink), and then copying the resulting binary into the build folder under the current workspace root.

Once we build the docker image, for example, locally: docker build . -t build-pq, the result will contain a /sources/build/twotm8, which is the single binary produced by our application's backend.

Runtime container

To produce the runtime Docker image that will actually be deployed, we need a lot less. The only runtime dependencies of our app are:

NGINX Unit's runtime library
openssl (we don't need the -dev package anymore)
libpq dynamic library (we don't need the libpq-dev package anymore)

The docker image provided by Unit appears to be Debian based, and as such already has OpenSSL installed, as well as, of course, Unit's own libraries.

Therefore the entire build of our image is much smaller:

FROM nginx/unit:1.26.1-minimal 

# 1
COPY --from=builder /sources/build/twotm8 /usr/bin/twotm8

# 2
COPY config.json /docker-entrypoint.d/config.json

# 3 
RUN apt update && apt install libpq5 && chmod +x /usr/bin/twotm8

EXPOSE 8080

# 4
CMD ["unitd", "--no-daemon", "--control", "unix:/var/run/control.unit.sock", "--log", "/dev/stderr"]

Because it's a multistage build, we can copy the built binary from the builder stage, into the /usr/bin/twotm8 location of the runtime image
We copy the NGINX Unit config (see below) into the special location that will be recognised by Unit's startup scripts.
We install the runtime libpq library and make sure the application binary is executable.
We override the default Unit's command to make sure it logs into /dev/stderr - I've found this was necessary to make both Scribe working, and Fly.io correctly showing the logs.

The exact config.json will change when we talk about resiliency and when we add frontend and static files, but here's the version we can reference for now:

{
  "listeners": {
    "*:8080": {
      "pass": "routes"
    }
  },
  "routes": [
    {
      "match": {
        "uri": "/api/*"
      },
      "action": {
        "pass": "applications/app"
      }
    }
  ],
  "applications": {
    "app": {
      "type": "external",
      "executable": "/usr/bin/twotm8"
    }
  }
}

Both these definitions can go into a single Dockerfile, and you can use flyctl deploy to trigger the build and deployment! Nifty.

Github Actions deployment

Thankfully, Fly already has a Github Action to make deployment easy.

All we need to do is first get the API token:

$ flyctl auth token

And set up a GH action with the token put into the FLY_API_TOKEN secret on the repository:

.github/workflows/ci.yml

name: Deploy to Fly
on:
  push:
    branches: ["main"]
jobs:
  deploy:
    name: Deploy proxy
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: superfly/flyctl-actions@master
        env:
          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
        with:
          args: "deploy"

And that's it! Whenever we push, the docker image will be rebuilt and app redeployed.

The layers in the builder image get cached on the Fly's builders, and the only stage that runs most of the time is the app building one. With the full app, including frontend and fully finished backend, I've been getting 2-3 minute deploys which is not bad, considering most of time is spend during the compilation and linking, and not waiting for the actions.

Application configuration

After the first flyctl launch, a fly.toml file will be generated at the root of the project.

By default it will just contain the app's name, i.e.

fly.toml

app = "twotm8-web"

After scavenging for configuration and sewing together several found snippets online, here's the configuration that I found working well:

fly.toml

[[services]]
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []

[services.concurrency]
  hard_limit = 500
  soft_limit = 250
  type = "requests"

[[services.ports]]
  handlers = ["http"]
  port = 80

[[services.ports]]
  handlers = ["tls", "http"]
  port = 443

[[services.tcp_checks]]
  grace_period = "1s"
  interval = "15s"
  restart_limit = 6

We will check this file in, to make sure deployed app is configured the same every time.

Conclusion

NGINX Unit is a polyglot application web server we can use with Scala Native (throught SNUnit)
Fly.io allows for easy deployment of Docker-based application
Multi-stage docker builds let us deploy containers with minimal runtime dependencies

Twotm8 (p.3): NGINX Unit and Fly.io