Java AOT compilation and challenges

Before delving into AOT and how we used it to compile our Java application into machine instructions ahead of time, let’s take a moment to understand the system we’re working on and the reasons why we chose to use AOT technology.

The system at a glance

Before talking about resilience, let’s first see what kind of issue we are trying to solve.

The system we are working on is a Digital Rights and Rules Manager. Digital Rights Management (DRM) is an approach to copyright protection for digital media. A DRM system prevents unauthorized redistribution of digital media and restricts how consumers copy content they’ve purchased.

Our system generates a key for encryption and stores the encrypted content in the operator’s content delivery network (CDN). When a user wants to play this content, they need a license that contains a key for the decryption of the content and policies. These policies define the quality of the content (HD, UHD), set geo-restrictions and time limits, etc. To obtain the license, the user goes back to our service.

During the FIFA World Cup, we encountered an issue with our Java applications. When a large number of users tuned in simultaneously and requested keys from our service, we scaled our application servers horizontally to handle the anticipated load. However, even with this scaling, we noticed that response times were slow and 99th percentile times were increasing for the initial X-number of minutes. Upon investigation, we discovered that the issue was due to a well-known problem with the HotSpot VM’s slow startup time.

In this article, we will cover several topics related to slow startup times in Java Virtual Machines (JVMs), including the reasons why this issue occurs, common approaches used to address it, how we resolved it using AOT compilation, and the challenges we faced during the AOT compilation process.

What is a slow startup and why it happens in JVM’s

Before we dive into the topic of slow startup times in Java Virtual Machines (JVMs), let’s first understand how Java code is compiled and executed. When Java code is compiled, it is transformed into an intermediate, platform-independent format called bytecode. This bytecode is later executed on a platform-dependent JVM.

Bytecode refers to the intermediate representation of Java source code that is generated by the Java compiler. Rather than producing machine code or executable files directly, the compiler generates bytecode that can be executed by the Java Virtual Machine (JVM).

So, when we want to execute a piece of Java code in JVM below steps happen.

1. Loading: The classloader loads the class file from the file system or network and transforms it into an internal format that can be used by the JVM. The loaded classes are stored in the method area of the JVM.
2. Verification: The bytecode of the loaded class is verified to ensure that it is well-formed and does not violate any security restrictions.
3. Preparation: The JVM allocates memory for the class variables and initializes them with their default values.
4. Resolution: The symbolic references used in the bytecode are resolved to actual references to methods and fields.
5. Initialization: The static variables of the class are initialized, and the static initializer block is executed.
6. Interpretation: The JVM interprets the bytecode and executes the instructions one by one. The interpreter converts each bytecode instruction into native machine code and executes it.
7. Profiling: The JVM collects information about the execution of the program, such as the frequency of method calls and the types of objects created.
8. Compilation: The JIT compiler analyzes the profiling information and selectively compiles parts of the bytecode to native machine code, optimizing the code for the specific hardware and operating system. The compiled code is stored in the code cache.
9. Execution: The JVM executes the compiled code directly, bypassing the interpreter and resulting in faster execution times.

JVM interprets bytecode to generate machine instructions and execute them. However, interpreting the same code block every time it’s executed can become a performance overhead. To avoid this, Java provides a Just-In-Time (JIT) compiler that converts the bytecode to hardware-specific machine instructions and stores them in the ‘Instruction Cache’ of the JVM memory.

However, the JIT compilation occurs only after the method block has been executed a certain number of times. Until then, the JVM continues to interpret the bytecode every time it is executed, leading to slow startup times for the HotSpot JVM.

Java offers a configuration setting called -XX:CompileThreshold that controls the number of method executions before JIT compilation kicks in.

When we scale our application, the newly added application instances may serve requests with higher latency due to this slow startup issue. As a result, users may experience a poor user experience and bad 99th percentile performance.

Because of this slow startup issues, we do not use JAVA in Lambdas of cloud computing.

The current application when we scale the response times looks like

Image1: Scaling application

hotspot-jvm-response-times

Memory and CPU consumption while handling requests

The above diagram shows that when we scaled up our application server at 11:03:00, the overall application response times increased. However, after a few minutes, at 11:04:00, the application’s throughput almost doubled, and response times decreased.

During the period between 11:03:00 and 11:04:00, the application did not respond correctly, taking almost a minute to warm up and serve requests faster. This slow startup time impacted the 99th percentile response time, which increased to 84 milliseconds.

Furthermore, the third diagram shows that when the application is handling requests at full capacity, it consumes almost 1 GB of memory.

What are the approaches people use to fix them

Approach-1: Warmup the application before it can serve the actual load

This is done by horizontally scaling the service and warming up all critical paths using dummy requests before adding them to the load-balancer/service. To minimize the time taken for warming up, the -XX:CompileThreshold=1000 is set, so that JIT compilation happens after 1000 times execution and the instructions will be cached for later usage.

Advantages

• Reduce latency of live traffic
• Better 99 percentiles as the application do not have slow startups because it was already warmed up before adding it to live service.

Disadvantages

• More time to add the server to handle live traffic, because before adding to the service we warm it, which takes extra time.
• If the critical path we warm up has side effects like adding data to a database of publishing events to Kafka in these cases it creates unnecessary data.
• Extra code to maintain in your application.

Approach 2: Ahead-of-time compilation(AOT)

By using AOT, we can generate machine instructions directly during the compilation of the Java code instead of generating bytecode, thus eliminating the need for interpreting bytecode during runtime. This means that we no longer need the JIT compiler during runtime, and our executable will be a machine instruction binary.

Advantages

• Lower Latency
• Better 99 percentiles
• Less runtime memory as JIT compile, native instruction cache, and Metaspace where bytecode is stored will be removed from the runtime of the JVM.

Disadvantages

• More time is taken to compile to code as we generate machine instructions.
• Machine instructions generated by AOT can be less optimized compared to those generated by JIT. JIT has the advantage of being able to gather profiling information at runtime, which can lead to more optimized machine instructions that take into account the specific characteristics of the running application and the underlying hardware. In contrast, AOT compilation happens before runtime and cannot take into account such dynamic factors.

Image 2: Scaling using AOT.

aot-response-times

Memory and CPU usage when handling requests

The image above shows that when we scale the application, it quickly responds with better latency, and the 99 percentiles are significantly improved, measuring at 22 milliseconds, which is much better than the 84 milliseconds shown by Hotspot VM in the previous section. Additionally, the memory consumption is much lower compared to HotSpot VM.

How to configure native compilation in maven

Steps to convert the bytecode java application to AOT.

1. Add Maven native plugin

 <plugin>
    <groupId>org.graalvm.buildtools</groupId>
    <artifactId>native-maven-plugin</artifactId>
    <version>${graal.plugin.version}</version>
    <executions>
      <execution>
        <goals>
          <goal>build</goal>
        </goals>
        <phase>package</phase>
      </execution>
    </executions>
    <configuration>
      <imageName>${project.name}</imageName>
      <mainClass>{main-application-loader-class}</mainClass>
    </configuration>
</plugin>

2. Create a folder under the resources folder to add conversion options and other configs. the folder name should be META-INF/native-image/{group-id}/{artifact-id}

3. Add native-image.properties file which contains your options to native-image builder.

Args =\
-H:EnableURLProtocols=http,https \
-H:+PrintClassInitialization \
-H:+ReportExceptionStackTraces \
-H:+ReportExceptionStackTraces \
-H:ReflectionConfigurationResources=${.}/reflect-config.json \
-H:JNIConfigurationResources=${.}/jni-config.json \
-H:ResourceConfigurationResources=${.}/resource-config.json \
-H:ConfigurationFileDirectories=${.} \
--no-fallback \
--enable-http \
--enable-https \
--gc=serial \
-R:MaximumHeapSizePercent=70 \
-R:MinHeapSize=300m \
--features=our-own-feature-class

More details about options can be found in Native Image Options (graalvm.org)

Issues we faced during the AOT compilation of our JAVA application

Now we know why we need AOT, let’s see what issues we faced while converting a JAVA application to AOT compiled Java application.

Log4j reflection issues

Log4j and other logging libraries use reflection to load logging providers at runtime. However, AOT does not support reflection because there is no bytecode to use reflection during runtime. If we compile our service, which uses log4j, to AOT machine instructions, it will build without any issues. However, when we run the native code, it will fail because it cannot use reflection to load the logging code. This limitation needs to be considered when using AOT for Java applications.

Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.apache.logging.slf4j.Log4jLoggerFactory.<clinit>(Log4jLoggerFactory.java:35)
        at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:53)
        at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:41)
        at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
        at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
        at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
Caused by: java.lang.IllegalStateException: java.lang.InstantiationException: org.apache.logging.log4j.message.DefaultFlowMessageFactory
        at org.apache.logging.log4j.spi.AbstractLogger.createDefaultFlowMessageFactory(AbstractLogger.java:246)
        at org.apache.logging.log4j.spi.AbstractLogger.<init>(AbstractLogger.java:144)
        at org.apache.logging.log4j.status.StatusLogger.<init>(StatusLogger.java:105)
        at org.apache.logging.log4j.status.StatusLogger.<clinit>(StatusLogger.java:85)
        ... 9 more
Caused by: java.lang.InstantiationException: org.apache.logging.log4j.message.DefaultFlowMessageFactory
        at java.lang.Class.newInstance(DynamicHub.java:639)
        at org.apache.logging.log4j.spi.AbstractLogger.createDefaultFlowMessageFactory(AbstractLogger.java:244)
        ... 12 more
Caused by: java.lang.NoSuchMethodException: org.apache.logging.log4j.message.DefaultFlowMessageFactory.<init>()
        at java.lang.Class.getConstructor0(DynamicHub.java:3585)
        at java.lang.Class.newInstance(DynamicHub.java:626)
        ... 13 more

To ensure that our application can still use log4j and run properly with AOT, we need to provide the AOT compiler with reflection information. This allows the AOT compiler to include the necessary logging code in the machine instructions it generates when building the image.

[
    {
    "name":"org.apache.logging.log4j.message.DefaultFlowMessageFactory",
    "methods":[{"name":"<init>","parameterTypes":[] }]
  }
]

To ensure that our application can still use log4j and run smoothly with AOT, we must provide the AOT compiler with reflection information for all reflections used in our service or by the libraries we use, including log4j. However, manually adding all these reflections can be challenging, so the GraalVM team has provided a way to generate them automatically.

To automatically generate these files, we first need to run the application using the JAR, but we will attach an agent that examines the code our application runs for user requests and generates these files automatically. This eliminates the need for manual effort and ensures that all necessary reflections are included in the AOT machine instructions.

java -agentlib:native-image-agent=config-output-dir=src/main/resources/META-INF/native-image/org.example/Java-AOT-Example -jar target/Java-AOT-Example-1.0-SNAPSHOT-runnable.jar

Below are the files generated

More details: https://www.graalvm.org/22.0/reference-manual/native-image/Agent/

Bouncy castle provider issues.

To use Bouncy Castle for encryption with higher key-sizes in our Java applications, we need to register it as a new security provider in our JVM security providers. Here’s how to do it:

Security.addProvider(new BouncyCastleProvider());

Because of this, we were getting below error.

2023-04-26 10:49:21,308 ERROR [org.example.Main] - Error occurred while deploying.
com.oracle.svm.core.jdk.UnsupportedFeatureError: Trying to verify a provider that was not registered at build time: BC version 1.7. All providers must be registered and verified in the Native Image builder. 
        at com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:89)

To resolve the error message, we need to register all the providers during build time. GraalVM provides a way to do this using the concept called “Feature.” With a Feature, we can register all the providers at build time and tell the native compiler to use this information provided by the Feature.

Here’s how we can use the Feature of GraalVM to register providers at build time:

import org.bouncycastle.jce.provider.BouncyCastleProvider;
import org.graalvm.nativeimage.ImageSingletons;
import org.graalvm.nativeimage.hosted.Feature;
import org.graalvm.nativeimage.hosted.RuntimeClassInitialization;
import org.graalvm.nativeimage.impl.RuntimeClassInitializationSupport;

import java.security.Security;

public class BouncyCastleFeature implements Feature {
  /**
   * @param access The supported operations that the feature can perform at this time
   *               Description:
   *               1. Register bouncy castle provider at build time.
   *               2. Re-initializing the DRGB default and NonceAndIV at run time as we need seed for random generators.
   *               We can not use compile time static seed for random generation.
   *               3. We will provide this feature to native-image builder, so that it can use and register bouncy castle(BC) along with SunJCE crypto providers.
   */
  @Override
  public void afterRegistration(AfterRegistrationAccess access) {
    RuntimeClassInitialization.initializeAtBuildTime("org.bouncycastle");
    final var rci = ImageSingletons.lookup(RuntimeClassInitializationSupport.class);
    rci.rerunInitialization("org.bouncycastle.jcajce.provider.drbg.DRBG$Default", "");
    rci.rerunInitialization("org.bouncycastle.jcajce.provider.drbg.DRBG$NonceAndIV", "");
    Security.addProvider(new BouncyCastleProvider());
  }
}

Now we can tell native-builder to use this feature

--features=org.example.features.BouncyCastleFeature

Kotlin reflection issues

Although we ran our application with the agent attached, we found that some of the Kotlin Types reflection information was not added to the reflection-config.json file automatically. To ensure that all Kotlin types work with reflection, we had to manually add these missing types to the reflection-config.json file.

{
  "name": "kotlin.reflect.jvm.internal.ReflectionFactoryImpl",
  "allDeclaredConstructors": true
},
{
  "name": "kotlin.KotlinVersion",
  "allPublicMethods": true,
  "allDeclaredFields": true,
  "allDeclaredMethods": true,
  "allDeclaredConstructors": true
},
{
  "name": "kotlin.KotlinVersion[]"
},
{
  "name": "kotlin.KotlinVersion$Companion"
},
{
  "name": "kotlin.KotlinVersion$Companion[]"
}

Initialize at run time and build time

Most of the time, you may encounter errors indicating that certain classes cannot be initialized at build time and must be initialized at run time or vice versa. In such cases, GraalVM provides options to specify which classes need to be registered at runtime and which ones are to be registered at build time.

--initialize-at-run-time=class/package
--initialize-at-build-time=class/package

Example code is available at https://github.com/iranna90/Java-AOT-Example

For more details about GraalVM please refer to their documentation https://www.graalvm.org/22.0/reference-manual/native-image/

I hope this article has helped you understand how Ahead-of-Time (AOT) compilation works in Java and how to address some of the issues that can arise when making your Java project AOT compatible.