Smithy4s full stack (p.4): Frontend and closing thoughts

smithysmithy4sscalaweavertestcontainersseries:smithy4s

Series TL;DR

If you are tired of the long-ass posts in this and Twotm8 series - I have good news for you.

The general principles I've learnt and developed have been thoroughly covered in the Twotm8 frontend building. Application's structure, principles behind SPAs, styling - all of these will remain the same.

Here we will just focus on some of the new things I've used (or had to create) to match the differences between the two apps. One of those (and probably the biggest one) is the approach to authentication - we no longer persist the long lived tokens in local storage, for security reasons, and instead expect the frontend to gracefully handle token expiration and continuous refresh.

The other is a different approach to local development set up, using Vite and new things available in Scala.js. Let's start with that.

Build configuration

The build starts simply enough:

build.sbt

lazy val frontend = projectMatrix
  .in(file("modules/frontend"))
  .jsPlatform(Seq(Versions.Scala))
  .defaultAxes(defaults*)
  .dependsOn(shared)
  .enablePlugins(ScalaJSPlugin, BundleMonPlugin)
  .settings(
    scalaJSUseMainModuleInitializer := true,
    // ...

Note that we depend on the shared module, which gives us access to two things:

  1. Generated Smithy4s code, cross-compiled for Scala.js
  2. Validation methods

We won't use the validation methods in this post, even though they are quite handy - the validation logic can be implemented on the client side as well, with a much faster feedback loop and lesser load on the server. Note that we should still repeat the validation on the server side, because if anyone wanted to bypass the rules - they'd invoke the API methods directly.

The smithy4s generated definitions are the real deal - we will use them to implement access to the entire backend API without writing a single HTTP request by hand - this last part is what I dreamed of my entire life ever since I was a wee1 lad.

Additionally, they contain a lot of the classes and newtypes we can use to increase the typesafety of our frontend - after all, it manipulates the same data as the backend, and deserves the same amount of care.

Dev configuration with Vite

There wasn't anything wrong with how we developed the frontend for Twotm8 - entire app being rendered as a single huge JS file, which was quite fast to rebuild with incremental linking.

"If it ain't broke - don't fix it" is a saying used by weak people who never had to justify a week-long refactor that ended up making everything universally worse. So let's make things difficult.

Scala.js 1.10.0 added an exciting new method for module splitting - SmallModulesFor(packages: List[String]). This method of module splitting instructs Scala.js to fragment specified packages into modules as small as possible, while all other classes will generate as few modules as possible.

This means that during incremental development the linker won't have to re-link the entire app and its dependencies, but can actually only relink the set of modules that actually changed.

To take advantage of this during local development I've decided to also try Vite.js - it's a Javascript bundler and build tool with lots of features. By increasing the granularity, browser will reload only a very small subset of JS files, avoiding re-parsing the entire (huge) multi-megabyte bundle.

JS tooling and JVM tooling go together like Montecchi and Capuleti, which is to say "not at all" and it results in multiple deaths and a Baz Luhrmann movie. Regardless, if it's meant to be - nothing will stop us.

First, let's reconfigure the frontend module:

build.sbt

    scalaJSLinkerConfig ~= { config =>
      if (isRelease) config
      else
        config
          .withModuleSplitStyle(
            ModuleSplitStyle.SmallModulesFor(List("frontend"))
          )
          .withModuleKind(ModuleKind.ESModule)
          .withOutputPatterns(OutputPatterns.fromJSFile("%s.mjs"))
    },

We're asking Scala.js to produce ES modules, and to split classes in frontend package as small as possible. Additionally, we're renaming the files to have a .mjs extension - to indicate those are ES modules, not standalone JS scripts.

What is this isRelease: Boolean variable? This module splitting is only useful for us during local development - when the app is bundled for release, we still want to have the fully optimised single file. So we define isRelease like this:

build.sbt

lazy val isRelease = sys.env.get("RELEASE").contains("yesh")

Remember it, for we will use it a couple more times.

Next, we need to add some configuration for Vite. I don't want to pollute the root folder of the project, so let's write all configuration in a new folder - ./frontend-build.

First - package.json for NPM:

{
  "name": "jobby",
  "version": "0.1.0",
  "scripts": {
    "dev": "vite",
    "build": "vite build",
    "serve": "vite preview"
  },
  "devDependencies": {
    "vite": "^3.0.0"
  }
}

Our only dependency is vite itself, so go ahead and

cd frontend-build && npm i

Continuous development

Now, before we configure Vite itself, there's imporant caveats to go over:

  1. Vite positions itself as a dev server first, bundler second - which means that the entry point to your app (index.html) is very important to it - you will get errors if it's missing,

  2. There is no way to configure where Vite will look for index.html - you can configure the root folder, where all of your JS files are, and index.html will be expected to be there as well,

  3. When Scala.js outputs all of the JS files it created, it will remove any other files from its target folder

(3) and (2) don't play together nice, as our index.html will need to constantly be recreated if we put it in Scala.js' targe folder. And every time we recreate it, its modification will re-trigger Vite's reload.

The solution will be threefold:

  1. Point Vite at the folder just above the Scala.js' target folder
  2. Use a relative path in index.html
  3. Copy index.html into folder from (1) only if it doesn't exist

For (2), here's what our index.html looks like:

frontend-build/index.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Jobby</title>
  </head>
  <body>
    <div id="appContainer"></div>
    <script type="module" src="/frontend-fastopt/main.mjs"></script>
  </body>
</html>

Note the relative path to main.mjs.

For (1), we need to write a Vite config as follows:

frontend-build/vite.config.js

/**
 * @type {import('vite').UserConfig}
 */

import path from 'path'

export default {
  root: path.resolve(__dirname, './../modules/frontend/target/js-3/'),
}

Note that we're pointing at the js-3 folder in frontend's target output - under that folder Scala.js will create frontend-fastopt/, where all of our JS modules will be located.

Now, for (3) we need to write a custom SBT task. Well, two actually. First off, frontendOutput which triggers the JS compilation and returns the folder where the files were generated:

build.sbt

lazy val frontendOutput = taskKey[(Report, File)]("")

lazy val frontendJS = frontend.js(Versions.Scala)

ThisBuild / frontendOutput := {
  if (isRelease)
    (frontendJS / Compile / fullLinkJS).value.data ->
      (frontendJS / Compile / fullLinkJS / scalaJSLinkerOutputDirectory).value
  else
    (frontendJS / Compile / fastLinkJS).value.data ->
      (frontendJS / Compile / fastLinkJS / scalaJSLinkerOutputDirectory).value
}

Note that we're using isRelease again.

And using it will be our task buildFrontend that triggers JS compilation and copies the index.html:

lazy val buildFrontend = taskKey[Unit]("")

buildFrontend := {
  val (_, folder) = frontendOutput.value
  val buildDir    = (ThisBuild / baseDirectory).value / "frontend-build"

  val indexHtml = buildDir / "index.html"
  val out       = folder.getParentFile() / "index.html"

  import java.nio.file.Files

  if (!Files.exists(out.toPath) || IO.read(indexHtml) != IO.read(out)) {
    IO.copyFile(indexHtml, out)
  }
}

Now we can run ~buildFrontend and it will continuously generate the JS files and copy index.html.

Embedding frontend in backend's resources

With frontendOutput task it's very simple to put our frontend into a place where the server can serve static files from:

lazy val app = 
    // ...
  .settings(
    Compile / resourceGenerators += {
      if (isRelease)
        Def.task[Seq[File]] {
          val (_, location) = (ThisBuild / frontendOutput).value

          val outDir = (Compile / resourceManaged).value / "assets"
          IO.listFiles(location).toList.map { file =>
            val (name, ext) = file.baseAndExt
            val out         = outDir / (name + "." + ext)

            IO.copyFile(file, out)

            out
          }
        }
      else Def.task { Seq.empty[File] }
  })

Here we are taking all the JavaScript files produced and put them into backend's resources location, under the assets/ subfolder.

When the app is packaged, those resources will be put on the classpath.

Setting up backend proxy

If we want to iterate on both backend and frontend independently, it would be cool if the two didn't depend on each other. For deployment purposes they will, as we will are embedding the frontend into the backend resources.

Locally though, I want to run the following independent terminal sessions:

  1. sbt '~app/reStart 9000' - it will launch the backend on port 9000, restarting it if code changes
  2. sbt '~buildFrontend' - continuously rebuild frontend
  3. cd frontend-build && npm run dev - which will launch Vite's development server

We can configure Vite (which is a dev server after all) to proxy all HTTP requests on a particular path to be sent over to a remote server.

Doing so is very simple, we just need to add this bit of config:

vite.config.js

// ...
export default {
  // ...
  server: {
    proxy: {
      '/api': {
        target: 'http://localhost:9000',
        changeOrigin: true,
      }
    }
  }
}

This way the server and frontend will be reloaded individually and our local set up will mimic the way it's going to be deployed - with backend being served at /api/ path, without any proxies.

API client

In Twotm8, we wrote all of our HTTP requests by hand, which not only was annoying, but also quite brittle, as the methods, endpoint names and response models were not shared with the backend and could go out of sync at any moment.

This time we have all these generated classes and API definitions, and as we found out when writing tests, all we need to turn these API traits into fully fledged HTTP clients is an instance of Client[IO].

On the browser, such instance will be backed by the Fetch API, and the Http4s wrapper for it will come from http4s-dom project.

The container for our APIs is very simple:

import jobby.spec.*
import com.raquo.airstream.core.EventStream as LaminarStream

class Api private (
    val companies: CompaniesService[IO],
    val jobs: JobService[IO],
    val users: UserService[IO]
):
  def future[A](a: Api => IO[A]): Future[A] =
    a(this).unsafeToFuture()

  def stream[A](a: Api => IO[A]): LaminarStream[A] =
    LaminarStream.fromFuture(future(a))

Because those clients return IO[..] values, which Laminar doesn't understand, we provide a couple of helpful methods to convert those results to Future or EventStream. A more correct approach would be to use Dispatcher, approach omitted for laziness reasons.

We can use SimpleRestJsonBuilder to create it using Fetch-based client:

object Api:
  def create(location: String = dom.window.location.origin.get) =
    val uri = Uri.unsafeFromString(location)

    val client = FetchClientBuilder[IO].create

    val companies =
      SimpleRestJsonBuilder(CompaniesService)
        .client(client, uri)
        .fold(throw _, identity)

    val jobs =
      SimpleRestJsonBuilder(JobService)
        .client(client, uri)
        .fold(throw _, identity)

    val users =
      SimpleRestJsonBuilder(UserService)
        .client(client, uri)
        .fold(throw _, identity)

    Api(companies, jobs, users)
  end create
end Api

Believe it or not, but that's it! Now calls like api.future(_.users.refresh(refreshToken = None)) will send a correctly formed HTTP request to our server, which it is guaranteed2 to understand, as they're formed from exactly the same model.

Authentication refresh loop

Most of the API calls require a valid access token - which can only be obtained by calling our refresh endpoint while having the valid refresh token available in the hardened cookies.

These cookies are not accessible from JavaScript, they're only added to HTTP calls, and let's just assume that the server will do its part and correctly set the cookies in response to valid login request.

When the app first loads, we don't know anything about the state of user's authentication, which we quickly rectify by calling the refresh endpoint. The result of that operation can be described with the following enum:

import jobby.spec.AuthHeader
import scala.scalajs.js.Date

enum AuthState:
  case Unauthorised
  case Token(value: AuthHeader, renewed: Date, length: Int)

Where either the server confirmed that the user doesn't have valid refresh token to authenticate itself, or it returned an access token with a set expiration time (usually quite short).

The fact that at page load time we don't have that knowledge can be represented by wrapping it in Option. Therefore the most important state in the entire app is Option[AuthState], which various operations will need to process.

We can encapsulate the state in a class like this:

class AppState private (
    _authToken: Var[Option[AuthState]],
    val events: EventBus[AuthEvent]
):
  val $token: StrictSignal[Option[AuthState]] = _authToken.signal

  val $authHeader: Signal[Option[AuthHeader]] = _authToken.signal.map {
    case Some(tok: AuthState.Token) => Some(tok.value)
    case _                          => None

  }

  def authHeader: Option[AuthHeader] = _authToken.now() match
    case Some(tok: AuthState.Token) => Some(tok.value)
    case _                          => None

  val tokenWriter: Observer[Option[AuthState]] = _authToken.writer
end AppState

object AppState:
  def init: AppState =
    AppState(
      _authToken = Var(None),
      events = EventBus[AuthEvent]
    )

end AppState

Where we expose the writer and reader for the token, but not the Var itself.

That state will need to be checked regularly, because token can expire, and for the most part those checks will return immediately, because the token is still valid.

Additionally, user actions, such as logout, can force the change of state - we'd like to handle that as well.

These actions can all be boiled down to this specification of events:

enum AuthEvent:
  case Check, Reset
  case Force(value: AuthState)

To account for these various ways the state can change, we will use the EventBus from Laminar, where various flows will push instances of AuthEvent, with this sketch diagram explaining the arrangement:

Before we write the loop implementation itself, let's write smaller, composable parts of it.

First of all, our entire auth refresh component will be written as a class:

class AuthRefresh(bus: EventBus[AuthEvent], period: FiniteDuration)(using
    state: AppState,
    api: Api
):
  //...

Simplest action is logout:

  private def logout: EventStream[Some[AuthState]] =
    api
      .stream(
        _.users
          .refresh(None, logout = Some(true))
          .as(Some(AuthState.Unauthenticated))
      )

Where we just invoke the relevant API and expect to write Unauthenticated into the global app state.

Next is the token refresh itself, a single exchange operation:

  private def refresh: EventStream[Some[AuthState]] =
    api
      .stream(
        _.users
          .refresh(None)
          .map { out =>
            out.access_token
              .map(tok => AuthHeader("Bearer " + tok.value))
              .map(AuthState.Token(_, new Date, out.expires_in.value.toInt))
          }
          .recoverWith { case _: UnauthorizedError =>
            api.users
              .refresh(None, logout = Some(true))
              .as(Some(AuthState.Unauthenticated))
          }
      )

Where we handle both the success and a particular error from the server.

Next, let's codify the various sources of events:

  private val eventSources: EventStream[AuthEvent] = EventStream
    .merge(
      bus.events,
      EventStream.periodic(period.toMillis.toInt).mapTo(AuthEvent.Check),
      state.$token.changes.collect { case None => AuthEvent.Reset }
    )

which are:

  1. Events from the EventBus
  2. Periodic Check events
  3. Events triggered by particular change of the token state

With those components, our token refresh loop looks like this:

  def loop =
    eventSources
      .withCurrentValueOf(state.$token) // sample current state
      .flatMap {
        // if state has not been computed yet,
        // attempt to do a token refresh
        case (_, None) => refresh
        
        // if we are asked to reset the state, log user out
        case (AuthEvent.Reset, _) => logout
        
        // if an operation like Login forces the new state 
        // just return it 
        case (AuthEvent.Force(value), _)

        // if the user is not authenticated, do nothing
        case (_, Some(AuthState.Unauthenticated)) =>
          EventStream.empty
        
        // if user is authenticated, check the token expiration time
        case (_, Some(AuthState.Token(t, d, maxAge))) =>
          val secondsSinceIssue = (Date.now - d.valueOf()) / 1000

          val remaining = maxAge - secondsSinceIssue
          
          // if there's only a minute of token left, force 
          // a refresh
          if remaining <= 60 then refresh
          else EventStream.empty // otherwise do nothing
      } --> state.tokenWriter

The final bit --> state.tokenWriter makes sure that the result of that refresh loop is written into the global authentication state

The .loop part can then be used in your application like this:

given state: AppState = AppState.init
given Api             = Api.create()

val tokenRefresh = AuthRefresh(state.events, Config.tokenRefreshPeriod)

val app = div(
  tokenRefresh.loop
)

Where we are aligning the lifetime of the div in app with the time during which events in the loop will be processed.

For something like the app's entry point, the lifetime is going to be equal to the app's entire lifetime.

Thoughts on Smithy and Smithy4s

I'm happy to say that this is all I wanted to cover here! The frontend code is really very similar to what we've made for Twotm8, just cleaner in parts.

One other thing I'd call out is using Monocle to deal with repetitive form input logic, here for example, but it's not really revolutionary.

This project has been a long time in making - not because it was particularly hard, but because I was particularly lazy.

The sun has risen and set on many other side projects during the literal months I was writing and rewriting the app and the posts.

One thing was clear during all this time - within the constraints of this application (use whatever I want), I haven't ever felt a pang of desire to write HTTP logic, both server side and client side.

Purposefully restricting my access to low level details of the protocol felt limiting at first (and you can clearly see it in a way we're handling the Set-Cookie headers), but in the end removing the cruft around the routes, JSON codecs, and other HTTP things, made it easier to follow just the business logic - especially visible in the code we wrote for integration tests.

Smithy as a language is evolving, but it already has enough flexibility to work in both Scala and Scala.js.

Smithy4s is also evolving as we speak, rolling out breaking changes in full accordance to the early-semver philosophy.

Newtypes existing on IDL level and handled beautifully in the code generator are a great way to delegate even more tedious work to Smithy4s.

It wouldn't be a fair summary if I didn't list any problems I've encountered:

  1. Rapidly prototyping on the Smithy specs with cross-platform modules led to a lot very strange undercompilation issues with Zinc.

    I frequently resorted to cleaning generated files just to get the changes recompiled.

    My attempt at fixing those seemed wildly successful on my project, but was reverted because of some other uninteded consequences.

    Obviously it will need to be revisited.

  2. Github doesn't support Smithy for syntax highlighting

    For a single person project this is not as big of an issue - I browse all of this code using locally set up syntax highlighting and LSP - which makes the experience excellent.

    But if you are relying on Github file viewer for working with those files it can become annoying quickly.

  3. IDE experience with generated files is a hit and miss

    Even using SBT as the BSP server was causing strange issues with things like Go To Definition.

    With newest versions of SBT this seems to have been improved, but a lot of this code has been written without any IDE support and it really shows it's perfect.

The positives far outweighed the negatives, and I would highly recommend giving Smithy4s a go - even if it's just for a solitary backend.

Thanks for reading up to this point (or even scrolling through), good luck and have fun!

  1. "wee" of course meaning "little", in the Scottish dialect of English

    ↩

  2. assuming there are no bugs in Smithy4s implementation, of course

    ↩