4.12.2015

Spring Boot's fat jars vs. Docker

I love Spring Boot. I really do. But I'm also do love Docker. And combination of them makes me really happy panda developer. But one of the most coolest things about Spring Boot - fat jars - anti-pattern in Docker world, where images should be layered. Can we solve this? Heck yeah!

Preparations

First we need to create Spring Boot project for tests. I will use Spring Boot CLI and single-Groovy-file project for this. Let's "clone" application from this tweet:


Now if you will run

spring jar app.jar app.groovy
You will get fat app.jar with your entire project. What is the size for this jar? 17Mb. Why? Because it's fat and contains whole Spring Boot stack, including Spring Framework, Groovy and even embedded Tomcat! Now we can pack it with Docker:


What's wrong with it? Let's build an image:

$ docker build .
Sending build context to Docker daemon 34.87 MB
Sending build context to Docker daemon
Step 0 : FROM java:8-jre
 ---> 028f36974b77
Step 1 : ADD app.jar /app/
 ---> f39fac8b6c8d
Removing intermediate container d3c168bb5b09
Step 2 : CMD java -jar /app/app.jar
 ---> Running in 3fbb5ba0cf4b
 ---> 51d0def78e12
Removing intermediate container 3fbb5ba0cf4b
Successfully built 51d0def78e12
Now change message inside your app.groovy file to "Hello dockerized world!" and pack it again with "spring jar" command above. If you will build your Docker image one more time than it will create a new layer the same size - 17Mb. And every time you need to change something in your project you will create a layer with all dependencies.

Exploded Jar? WAT?

Did you know that you can explode your fat jar and still can run it? Just try yourself, it works with any Spring Boot-based fat jar:

$ unzip -q app.jar -d app
$ java -cp app org.springframework.boot.loader.JarLauncher --server.port=8081
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.2.3.RELEASE)


...

2015-04-12 13:22:44.484  INFO 6379 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8081 (http)
2015-04-12 13:22:44.486  INFO 6379 --- [           main] .b.c.j.PackagedSpringApplicationLauncher : Started PackagedSpringApplicationLauncher in 2.706 seconds (JVM running for 3.078)
How it can help us? Answer is inside an exploded directory:

$ ls -la app
total 24
drwxr-xr-x   7 bsideup  staff   238 Apr 12 13:21 .
drwxr-xr-x   6 bsideup  staff   204 Apr 12 13:21 ..
-rw-r--r--   1 bsideup  staff    75 Apr 12 12:38 Dockerfile
drwxr-xr-x   3 bsideup  staff   102 Apr 12 12:38 META-INF
-rw-r--r--   1 bsideup  staff  5136 Apr 12 12:38 ThisWillActuallyRun.class
drwxr-xr-x  37 bsideup  staff  1258 Apr 12 12:38 lib
drwxr-xr-x   3 bsideup  staff   102 Apr 12 12:38 org
Do you see this "lib" folder? It contains all your project dependencies. They are not changing often. So why not to cache them?

Docker caching

How to cache something in Docker? Just "ADD" it before the main files. Modify your Dockerfile with this changes:


And build your image:

$ rm -rf app/ && unzip -q app.jar -d app
$ docker build .
Sending build context to Docker daemon 34.87 MB
Sending build context to Docker daemon
Step 0 : FROM java:8-jre
 ---> 028f36974b77
Step 1 : ADD app/lib/ /app/lib/
 ---> 01fd4225441d
Removing intermediate container 77d555817fa8
Step 2 : ADD app /app/
 ---> 846a92bd6fc3
Removing intermediate container 86b26a48efde
Step 3 : CMD java -cp /app/ org.springframework.boot.loader.JarLauncher
 ---> Running in 2e677531c0e5
 ---> e90f469994c7
Removing intermediate container 2e677531c0e5
Step 4 : EXPOSE 8080
 ---> Running in f05d3868f6ed
 ---> 526ff5ad98ae
Removing intermediate container f05d3868f6ed
Successfully built 526ff5ad98ae
Now on Step 1 Docker will create a layer with your libraries and cache it until they will be changed.

Not so fast


UPDATE: since this fix https://github.com/spring-projects/spring-boot/issues/2807 future steps are not required anymore if you're using Spring Boot 1.3.0 or newer.


What will happen if you will run "spring jar" command again and then build an image? Depends on your Docker knowledge. Lets try:

$ spring jar app.jar app.groovy
$ rm -rf app/ && unzip -q app.jar -d app
$ docker build .
Sending build context to Docker daemon 34.87 MB
Sending build context to Docker daemon
Step 0 : FROM java:8-jre
 ---> 028f36974b77
Step 1 : ADD app/lib/ /app/lib/
 ---> 3270477dd01f
Removing intermediate container 08a0ab7ed4ef
Step 2 : ADD app /app/
 ---> f9372f874d47
Removing intermediate container 751de4852196
Step 3 : CMD java -cp /app/ org.springframework.boot.loader.JarLauncher
 ---> Running in 4954fb8d37d3
 ---> e2fc978f2740
Removing intermediate container 4954fb8d37d3
Step 4 : EXPOSE 8080
 ---> Running in 9f70e24d5bc0
 ---> 75999f39321b
Removing intermediate container 9f70e24d5bc0
Successfully built 75999f39321b

This was my first reaction on happened. But then I realized that timestamp for libraries was changed after packaging. Can we fix it? Yes we can!

$ find ./app/lib/ | xargs touch -t 0000000000.00
$ docker build .
Sending build context to Docker daemon 34.87 MB
Sending build context to Docker daemon
Step 0 : FROM java:8-jre
 ---> 028f36974b77
Step 1 : ADD app/lib/ /app/lib/
 ---> 8028a9ba2b7b
Removing intermediate container 52d21772824a
Step 2 : ADD app /app/
 ---> 7ed806b0089f
Removing intermediate container f070b31695bf
Step 3 : CMD java -cp /app/ org.springframework.boot.loader.JarLauncher
 ---> Running in 9a84ec5214ae
 ---> 8fda50a60f87
Removing intermediate container 9a84ec5214ae
Step 4 : EXPOSE 8080
 ---> Running in 8543d0799c68
 ---> 06614c96e115
Removing intermediate container 8543d0799c68
Successfully built 06614c96e115
Now every new build will use cache for libraries:

$ spring jar app.jar app.groovy
$ rm -rf app/ && unzip -q app.jar -d app
$ find ./app/lib/ | xargs touch -t 0000000000.00
$ docker build .
Sending build context to Docker daemon 34.87 MB
Sending build context to Docker daemon
Step 0 : FROM java:8-jre
 ---> 028f36974b77
Step 1 : ADD app/lib/ /app/lib/
 ---> Using cache
 ---> 8028a9ba2b7b
Step 2 : ADD app /app/
 ---> e0b404447dbe
Removing intermediate container 8952413133a9
Step 3 : CMD java -cp /app/ org.springframework.boot.loader.JarLauncher
 ---> Running in 8c2e55128dc0
 ---> 30f8fb7f4eba
Removing intermediate container 8c2e55128dc0
Step 4 : EXPOSE 8080
 ---> Running in b08967c7a454
 ---> 9e028bbfef90
Removing intermediate container b08967c7a454
Successfully built 9e028bbfef90

Now your services are real microservices.

15 comments:

  1. Doesnt ADD app/ also copy the content from ADD app/lib/ a second time and therefore duplicate the lib files ?

    ReplyDelete
    Replies
    1. It does. BUT since there are no changes in this file between layers this files will not be added to the layer.

      This technique was used in some official docker images like this:
      https://github.com/joyent/docker-node/blob/04e6f537ede555b2558abfab32a1b8d31e7c1500/0.12/onbuild/Dockerfile#L6-8

      Delete
  2. Hello

    thanks for the nicely written article. Why not instead of creating a container that includes the fat jar as a layer, add a simple volume to the original docker image (java-8) and execute from the shared volume?

    You can have a volume for each microservice - and because it is only read from, launch as many of the container with the volume. Is there something I'm missing?

    For versioning support, you could have a shared volume with /app/v1, /app/v2 ... etc and launch the container with the same volume, just different run command...

    ReplyDelete
    Replies
    1. Hey

      1) With volumes you will need to deliver 2 things (Docker image (java-8) and your app.jar file to mount it later as volume) instead of one (Docker image)
      2) With trick from article host machine will download only new layer with our app code without libs, otherwise your full app.jar INCLUDING libs will be shipped, which is bad.
      3) With volumes you increase complexity of deployments, because it will require to configure and create additional volumes compared to "just run this image"
      4) It's really hard to work with volumes on systems such Amazon Elastic Beanstalk and other PaaS Docker providers, much easier to "just run this image"

      Delete
  3. Hi Sergei:

    I am finding that using the method above still generate full size layer on my docker images:

    Docker build steps:
    ...
    Step 5 : COPY myproject/WEB-INF/lib /projects/myproject/WEB-INF/lib
    ---> Using cache
    ---> 1fc17d01e8bc
    Step 6 : COPY myproject /projects/myproject
    ---> 176cf85fddc6
    Removing intermediate container 04705debcb7f

    $ docker history my_docker_image_name
    IMAGE CREATED CREATED BY SIZE COMMENT
    176cf85fddc6 30 minutes ago /bin/sh -c #(nop) COPY dir:ace4d8f9b722302e38 87.82 MB
    1fc17d01e8bc 32 minutes ago /bin/sh -c #(nop) COPY dir:fb8aaa151a1698ed71 83.68 MB
    ...

    If we make small change. it will still try to sent out the last layer of the docker image(~80MB) out, did I miss something here?

    Regards
    Ken

    ReplyDelete
  4. Great post....!!
    Post is good and easy to understand.....IT Hub Online Training providing Spring Online Training Course.

    ReplyDelete
  5. Hi Sergei,

    Thanx a lot.

    I tried your method but it does not seem to work with docker 1.9.1. My app layer is about 40 MB, my lib layer around 37. Even if the lib layer is cached, the app layer will always be pushed as the full 40 MB, instead of the assumed 3 MB. It works, if I physically move the lib folder out of the exploded folder and then do something like

    COPY ./lib/ /app/lib/
    COPY ./app/ /app/

    in the docker file.

    Best Regards,

    Kai

    ReplyDelete
  6. If you want to download spring go to this website. You will get spring with dependencies: http://jar-download.com/download/spring-context-jar-4.1.6.RELEASE-download-with-dependencies.php

    ReplyDelete
  7. Do you think we need dockerize spring boot apps? since we can run spring boot executable jar files just with java -jar command..we currently have dockerize spring apps, but pushing images to registry and then download from them, taking high deployment time. what you do think about it?

    ReplyDelete
    Replies
    1. The issue with just relying on 'java -jar' is that the app deployment requires a certain version of java installed. This manual deploy and upgrade process can be a headache at scale. Docker automates this. Docker also allows for higher level orchestration with tools like docker swarm or kubernetes

      Delete
  8. This approach is not really useful and is introducing complexity and other bad practices for zero value - you should treat your JAR file as your application artefact and your Docker release image should place it in as a new layer each time you build the image. Such an operation is not normally considered "slow" or inefficient.

    The real challenge with caching comes if you use a Docker image as a build image, and avoiding downloading all dependencies each time you run the build.

    ReplyDelete
  9. Good admin.keep it up.your blog was informative. Thank you for sharing.Get more interesting details about Java Training in Marathahalli | Java Training in Bangalore

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. You might want to checkout a gradle plugin I created to automate all of this.

    https://github.com/gclayburg/dockerPreparePlugin

    Just add this to your build.gradle:


    plugins {
    id "com.garyclayburg.dockerprepare" version "1.1.0"
    }


    Works with spring boot jars or wars. What do you guys think?

    ReplyDelete
  12. Thanks for providing such a useful information. Trainers Desk providing best sap erp module online training.

    ReplyDelete