TLDR:
- We are calling Java and Scala JVM code from Scala Native using JNI
- Github repo for sample code
- Github repo for a more batteries-include SBT template
For as long as I remember, I knew there was a way to interact with JVM and Java applications from lower level languages, such as C and C++. The thought of it being possible made me uneasy, but I always dismissed it as yet another tragic practice, another glimpse of humanity's violent nature, like mass murders and burgers drenched in cheese for TikTok views. Perhaps I never worked in an old enough bank or insurance company, where desire to never change the appraisal logic written in Java 1.2 is at odds with desire to move a slow backend to blazing fast C++.
Thing is, JNI is very old. And people use it, calling into JVM from C and C++, and probably for good reasons as well! To its credit, JNI interface is written in fairly approachable dialect of C, which opens the opportunity to lots of languages that can pass the minimal bar for interop – C binary interface.
One such language is Scala Native. Buckle up, let's go for a short but memorable ride.
What are you doing, Scala already has a JVM implementation
This is a natural reaction to what I am proposing here – Scala, for decades now, lived and breathed whatever fumes JVM exudes. It's the main target, and interoperability with Java is very high on priority list of language designers. Language design often focuses on the ability to make abstractions fast within Scala's main target – JVM bytecode. And people are rightfully excited about the possibility of using modern JDKs advanced features in Scala's backend.
Still! Scala Native is a very exciting project, one I support wholeheartedly and blog profusely about. Due to its nature, Scala Native cannot work with Java libraries. And Scala libraries need to be recompiled with Scala Native compiler plugin to be usable. And while the world of Scala libraries that physically cannot support Scala Native is shrinking (with the notable exception of reflection heavy libraries such as Akka ecosystem), the much larger world of Java libraries is as unapproachable as it was when Scala Native first started.
So let's imagine ourselves in a situation where our heart yearns for Scala Native, but the harsh reality forces us to use some Java library that has no chance of being ported to Scala, let alone Scala Native, for either reasons of effort, or practicality.
What on earth even prompted all this
On August 10, 2022, an issue appeared in one of my repositories that hosts various examples of using my sn-bindgen project to work with C libraries. The author attempted to use sn-bindgen (at the point a very unstable project) to generate bindings to JNI. Bindgen failed, spectacularly.
But it did give me enough of a jolt to look into JNI yet again, and to read the actual header files with my own eyeballs. Those header files are on your computer to, if you have JDK installed! Check out $JAVA_HOME/include/jni.h file, where $JAVA_HOME is the location of installed JDK. On my MacOS machine, for example, the file is in /Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/include/jni.h.
At the time, I couldn't fix bindgen to make it work, and promptly gave up. Then 2 more years have passed and during that time I gave up on great many wonderful things. Until one faithful Saturday morning where I had nearly 3 whole hours of interrupted me time (this phrase should come with a trigger warning for fellow parents of small kids that have no family in town and no regular weekend childcare option). It was during that time that I started thinking about JNI again, and this time I stumbled on an excellent blog post which I didn't see in 2022 somehow.
That post truly has everything – the why, the how, the what. I recommend you read it as well. Most importantly, it had the necessary linking flags and the sample C code that would work on MacOS which is what I use.
So I immediately jumped in to translate it to Scala Native.
How does it work
The process is very simple
- First, we need to generate the Scala 3 Native bindings to JNI
- Then, we need to write a program using those bindings, by relying heavily on the blog post that has a sample program written in C
- Finally, we need to find the magical Clang flags to build the app
That's it! At this point it becomes just a regular Scala Native application and we can do whatever.
Let's look at the steps in a bit more detail
Bindings
This part turned out to be uneventful – the header is in a well known location, and the only platform-specific thing it requires is the jni_md.h header which was easy to find.
See an example from my Makefile.
After this part all we have is libjni.scala file with raw bindings to JNI.
Initialising the JVM from Scala Native
To start doing meaningful things in our application, we need to initialise the JVM correctly, and obtain an initialised value of JNIEnv, which becomes the context parameter in every single method we invoke in JNI interface, and is just an alias of JNINativeInterface_ which contains a lot of C function pointers that do the actual useful things.
The initialisation hits on several annoyances in Scala Native's C interop, which become clear when you are forced to work with double pointers, which JNI uses a lot in initialisation.
Thankfully, you only need to do it once, so no matter how horrible it is, you just copy paste it:
import libjni.all.*, libjni.constants.*
import scalanative.unsafe.*
// Part 1: initialising basic JNI interface
val iface = libjni.structs.JNIInvokeInterface_()
val args = JavaVMInitArgs()
(!args).version = jint(JNI_VERSION_1_8)
(!args).nOptions = jint(0)
val vm = doublePointer(JavaVM(iface))
val env = doublePointer[JNIEnv](JNIEnv(null))
val res = JNI_CreateJavaVM(
vm,
env.asInstanceOf[Ptr[Ptr[Byte]]],
args.asInstanceOf[Ptr[Byte]],
)
if res.value != JNI_OK then sys.error("Failed to create Java VMn")
// just look at this... horrors beyond human comprehension
val jvm: JNINativeInterface_ = (!(!(!env)).value)
As I said, double pointers are a pain, so I'm using a little helper:
inline def doublePointer[A: Tag](value: A): Ptr[Ptr[A]] =
val ptr1 = stackalloc[A]()
val ptr2 = stackalloc[Ptr[A]]()
ptr2.update(0, ptr1)(using Tag.materializePtrTag[A])
!ptr1 = value
ptr2
end doublePointer
it is inline to make sure stackalloc is not affected by this method at all.
Once we have this jvm value on hand, we can actually use it to work with the JVM machinery.
Let's print something
Now that we have access to JNINativeInterface_, let me show you what it takes to just print a single string using Java's trusty System.out.println:
val system = jvm.FindClass(!env, c"java/lang/System")
assert(system.value != null, "Failed to find java.lang.System class")
val outField =
jvm.GetStaticFieldID(!env, system, c"out", c"Ljava/io/PrintStream;");
assert(outField.value != null, "Failed to find .out field on System")
val out = jvm.GetStaticObjectField(!env, system, outField)
assert(out.value != null)
val printStream = jvm.GetObjectClass(!env, out)
assert(printStream.value != null)
val printlnMethod =
jvm.GetMethodID(!env, printStream, c"println", c"(Ljava/lang/String;)V")
assert(printlnMethod.value != null)
val str =
jvm.NewStringUTF(!env, c"Hello world from Java from... Scala Native?..")
jvm.CallVoidMethodV(!env, out, printlnMethod, toCVarArgList(str))
We are using the CallVoidMethodV which is a JNI method that supports varargs, that Scala Native also supports.
Looks horrible, doesn't it? Get a class, get the field ID, get the field, get the class, get the method, create java.lang.String, and only then you can call the method. A lot of work for simple System.out.println!
The good news is that it's all mechanical. Just inspecting the byte code of a given class can give enough information to automatically generate chains of those calls. The hardest is figuring out the method signatures, but thankfully the logic
I can just about imagine a code generator that recovers the System.out.println UX by generating code that uses JNI.
Adding libraries to classpath
All you need to do is add -Djava.class.path=... to the VM initialisation options. Both the sample code and the template already have this.
What can you do with it?
The JNI interface is extensive, and you can not only do basic things, like finding/creating classes and invoing their methods, but also reflection.
Once you pay the price of setting up the JNI once, you can use a very straightforward API to do a lot of things that JVM can do.
Well damn, now I want to try it
Let me help you!
I've put together a powerful SBT template that handles everything:
- Bindgen configuration in case you want to generate and tweak bindings yourself
- Correct Native flags to build your application
- A sample application doing JNI calls
- A fully setup build to test your application with any JVM dependencies you wish to play with
All in glorious Scala 3, all wired in so you can just do
git clone https://github.com/keynmol/scala-native-jni-template.git && cd scala-native-jni-template && sbt binary/run
With no extra settings! As long as you have JDK installed on your machine.
Please, please give it a go, and see for yourself that it's not super scary and you can totally play with it from the comfort of your own home.
I want to go crazy with this
SO DO I!
The mechanical nature of all the code required on the native side makes it a perfect target for a code generator.
Generating such code for Java libraries should be very easy, because there is no name mangling going on.
Generating it for Scala will be harder, but not by a lot, as long as you restrict the interface to look more Java-like. A small price to pay to be able to use some Scala libraries that have no chance of being ported to Scala Native.
You can even make your own, simplified facades to your Scala code, and only generate bindings for that!
As an experiment, take the template and hand write the bindings you think would be pleasant to use – take for example the HttpClient class from JDK and try to make it work with JNI. This will be worthwhile.
Good luck and have fun!