Tag: elasticbeanstalk

Remotely connect to JMX behind an AWS Loadbalancer

JMX is useful to gather various information about a running Java application. Mostly it is used to gain detailed data about CPU-consumption (where are the hotspots in my code?) or Memory-allocation (what is eating up my memory?). Especially when a system is under load and behaves ‘unexpectedly’ that information is valuable. Here are the steps you have to take in order to get remote access to JMX when you host your servers in the Amazon cloud behind a loadbalancer.

1. Associate Public IP Address

The servers that are created in your ElasticBeanstalk environment need to have a public IP-Address. To do this, make sure you’ve selected the following option in your Elastic Beanstalk environment’s configuration:

Elastic Beanstalk → Your Application → Your Environment → Configuration →  VPC → Associate Public IP Address

Be aware that exposing the instances behind a LB undermines it’s purpose: only having one public end-point. By making all your instances public, they are open for attacks of any kind!

2. Change server startup parameters

A quick search on stackoverflow reveals the required arguments you have to add to your Java application:

-Dcom.sun.management.jmxremote.port=PORT
-Dcom.sun.management.jmxremote.rmi.port=PORT
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=PUBLIC_IP

In a previous post I’ve described how to pass these arguments to the Java application (add them as JAVA_OPTS) and choosing a PORT is trivial. However you should use the same port for .port and .rmi.port to minimize the ports you have to open in the security group later on.

Now what about the PUBLIC_IP? If you leave out this parameter, JMX will most likely bind to the private IP-Address and you wont be able to access it from outside. My first attempt was checking the instance’s environment variables to see if the server’s public IP is added there. Unfortunately it was not. My second attempt was to expose an environment variable JMX_IP utilizing .ebextensions like this:

commands:
  01-setip:
    command: export JMX_IP=$(curl http://169.254.169.254/latest/meta-data/public-ipv4)

The idea was to export an environment variable JMX_IP and then reference it in the JAVA_OPTS like so -Djava.rmi.server.hostname=${JMX_IP} which did not work as well. I’m not sure why (maybe you know?), but I ended up changing the script that builds my deployment artifact:


#!/bin/bash
set -e
now=$(date +"%Y-%m-%d_%H-%M-%S")
tag="./target/app-$now.zip"
mvn clean install
echo "web: ./run.sh" > target/Procfile
echo "JMX_IP=\$(curl http://169.254.169.254/latest/meta-data/public-ipv4)" > target/run.sh
echo "exec java \$JAVA_OPTS -Djava.rmi.server.hostname=\$JMX_IP -jar app.jar \$JAVA_ARGS" >> target/run.sh
chmod +x target/run.sh
zip -j ${tag} target/Procfile target/run.sh target/app.jar

view raw

build.sh

hosted with ❤ by GitHub

This results into the following run.sh file which is executed during deployment:


JMX_IP=$(curl http://169.254.169.254/latest/meta-data/public-ipv4)
exec java $JAVA_OPTS -Djava.rmi.server.hostname=$JMX_IP -jar app.jar $JAVA_ARGS

view raw

run.sh

hosted with ❤ by GitHub

As you can see, the public ip address is fetched from the instance’s meta-data URL and added as an option to the Java process. If you added the startup parameters from above, the server should start JMX and expose it on the public IP address.

3. Open JMX port in security group

In the final step you have to open a port to be able to connect to your server. Before we proceed, be aware that we disabled any authentication! That means that as soon as the port is open, anyone that knows the IP of your server can access to it. Therefore, either close the port as soon as you’re finished your work and/or add SSL to your JMX configuration!

To open the port, I recommend creating a new security group under

EC2 → Security Groups → Create Security Group

in your VPC. Add an Inbound rule to open a TCP port you’ve chosen for JMX (if you decided to give a different port for rmi, it has to be opened as well) and add your IP address as the source. Note the security group’s name (e.g. sg-abcdefg) and add it to your Elastic Beanstalk configuration:

Elastic Beanstalk → Your Application → Your Environment → Configuration → Instances → EC2 security groups

Finally, you should be able to add and connect the EC2 instances using jvisualvm.

Configuring A Staged Java Application On AWS ElasticBeanstalk

Update

I’ve just blogged about a maven only solution.

In this blog post I’ll describe how to deploy and setup a maven driven Java application on AWS ElasticBeanstalk with different staging profiles. To understand what’s going on, you should have knowledge about AWS and how to setup an ElasticBeanstalk environment.

I have setup this github-repository which contains a very basic maven project. Let’s have a look at it’s pom:


<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0&quot;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance&quot;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"&gt;
<modelVersion>4.0.0</modelVersion>
<groupId>io.github.jklingsporn.eb</groupId>
<artifactId>example</artifactId>
<version>1.0-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<extensions>true</extensions>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.6</version>
<configuration>
<finalName>app</finalName>
<archive>
<manifest>
<mainClass>io.github.jklingsporn.eb.Main</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<appendAssemblyId>false</appendAssemblyId>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

view raw

pom.xml

hosted with ❤ by GitHub

No big surprises here: we set the compiler level to Java 8 and tell maven to build an executable fat-jar called app.jar during package-phase. The main itself just prints the passed arguments in the console as shown below:


public class Main {
public static void main(String[] args) {
System.out.println(String.format("Look at these wonderful arguments %s", Arrays.toString(args)));
}
}

view raw

Main.java

hosted with ❤ by GitHub

Executing mvn clean install should generate the class files and a runnable jar file in the target-folder. One could take this file and upload it to ElasticBeanstalk. We’re done.

Just kidding.

Two problems may arise:

  1. ElasticBeanstalk will reject the file the next time you upload it, because it already has a file called app.jar.
  2. The application wouldn’t print any arguments, because you haven’t passed any to it.

Solving the first problem is quite easy: you can just append the current timestamp (or project version) to the jar. Either manually, or by changing the finalName-tag to something like

app-${project.version}

or

app-${maven.build.timestamp}

However this doesn’t solve the second problem. To tackle this, one must understand how the application is started after it was deployed. On the EC2-instance, next to the uploaded jar-file there is a file called Procfile, which is used to start the jar. By default, it consists of a one-liner:

web: java -jar app.jar

The only thing you need to know regarding the first word web is:

The command that runs the main JAR in your application must be called web, and it must be the first command listed in your Procfile.

Good news is, that a user can upload it’s own Procfile which leads us to the question: where to put the arguments, especially if you have different arguments for each staging profile. At first sight, a good fit are maven-profiles, but that solution has two major drawbacks: secret informations like passwords should not be stored in your pom. And secondly, each time you change the arguments, you have to redeploy the whole application.

A much better approach is to store that kind of data on the instance itself using ElasticBeanstalk’s configuration. To do so, go to your environment, select the application you are using (or create a new one based on Java SE-Platform), click on Configuration, Software Configuration and create two new environment variables: JAVA_OPTS and JAVA_ARGS. Set the value of JAVA_OPTS to something like -Dfile.encoding=utf8 and JAVA_ARGS to my_staging_property. The values are now available on the instance – time to create our own Procfile. To do so, I’ve created a bash-script (Windows users look here) like this:


#!/bin/bash
set -e
now=$(date +"%Y-%m-%d_%H-%M-%S")
tag="./target/app-$now.zip"
mvn clean install
echo "web: ./run.sh" > target/Procfile
echo "exec java \$JAVA_OPTS -jar app.jar \$JAVA_ARGS" > target/run.sh
chmod +x target/run.sh
zip -j ${tag} target/Procfile target/run.sh target/app.jar

view raw

build.sh

hosted with ❤ by GitHub

  • Lines 3 and 4: I decided to append the timestamp to the build artifact to fix the redeploy issue.
  • Line 6, 7 and 8: Our Procfile just executes another shell script, called run.sh. That script uses the environment variables defined above to start the application.
  • Line 9: Zip everything into one file.

So instead of executing maven directly, run ./build.sh from the terminal. This will create a file with a name like app-2017-05-03_14-50-07.zip in the target/ – folder of your maven project which can then be deployed to any staging environment. Configuration is solely done by ElasticBeanstalk. E.g. set JAVA_OPTS to -Xmx256M on your dev-environment and -Xmx16G on production. Changing some parameters only requires to update your EB-configuration (e.g. tune some GC-settings during a load test) instead of redeploy everything.