JMX is useful to gather various information about a running Java application. Mostly it is used to gain detailed data about CPU-consumption (where are the hotspots in my code?) or Memory-allocation (what is eating up my memory?). Especially when a system is under load and behaves ‘unexpectedly’ that information is valuable. Here are the steps you have to take in order to get remote access to JMX when you host your servers in the Amazon cloud behind a loadbalancer.
1. Associate Public IP Address
The servers that are created in your ElasticBeanstalk environment need to have a public IP-Address. To do this, make sure you’ve selected the following option in your Elastic Beanstalk environment’s configuration:
Elastic Beanstalk → Your Application → Your Environment → Configuration → VPC → Associate Public IP Address
Be aware that exposing the instances behind a LB undermines it’s purpose: only having one public end-point. By making all your instances public, they are open for attacks of any kind!
2. Change server startup parameters
A quick search on stackoverflow reveals the required arguments you have to add to your Java application:
-Dcom.sun.management.jmxremote.port=PORT
-Dcom.sun.management.jmxremote.rmi.port=PORT
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=PUBLIC_IP
In a previous post I’ve described how to pass these arguments to the Java application (add them as JAVA_OPTS) and choosing a PORT is trivial. However you should use the same port for .port
and .rmi.port
to minimize the ports you have to open in the security group later on.
Now what about the PUBLIC_IP? If you leave out this parameter, JMX will most likely bind to the private IP-Address and you wont be able to access it from outside. My first attempt was checking the instance’s environment variables to see if the server’s public IP is added there. Unfortunately it was not. My second attempt was to expose an environment variable JMX_IP utilizing .ebextensions like this:
commands: 01-setip: command: export JMX_IP=$(curl http://169.254.169.254/latest/meta-data/public-ipv4)
The idea was to export an environment variable JMX_IP and then reference it in the JAVA_OPTS like so -Djava.rmi.server.hostname=${JMX_IP}
which did not work as well. I’m not sure why (maybe you know?), but I ended up changing the script that builds my deployment artifact:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
set -e | |
now=$(date +"%Y-%m-%d_%H-%M-%S") | |
tag="./target/app-$now.zip" | |
mvn clean install | |
echo "web: ./run.sh" > target/Procfile | |
echo "JMX_IP=\$(curl http://169.254.169.254/latest/meta-data/public-ipv4)" > target/run.sh | |
echo "exec java \$JAVA_OPTS -Djava.rmi.server.hostname=\$JMX_IP -jar app.jar \$JAVA_ARGS" >> target/run.sh | |
chmod +x target/run.sh | |
zip -j ${tag} target/Procfile target/run.sh target/app.jar |
This results into the following run.sh file which is executed during deployment:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JMX_IP=$(curl http://169.254.169.254/latest/meta-data/public-ipv4) | |
exec java $JAVA_OPTS -Djava.rmi.server.hostname=$JMX_IP -jar app.jar $JAVA_ARGS |
As you can see, the public ip address is fetched from the instance’s meta-data URL and added as an option to the Java process. If you added the startup parameters from above, the server should start JMX and expose it on the public IP address.
3. Open JMX port in security group
In the final step you have to open a port to be able to connect to your server. Before we proceed, be aware that we disabled any authentication! That means that as soon as the port is open, anyone that knows the IP of your server can access to it. Therefore, either close the port as soon as you’re finished your work and/or add SSL to your JMX configuration!
To open the port, I recommend creating a new security group under
EC2 → Security Groups → Create Security Group
in your VPC. Add an Inbound rule to open a TCP port you’ve chosen for JMX (if you decided to give a different port for rmi, it has to be opened as well) and add your IP address as the source. Note the security group’s name (e.g. sg-abcdefg) and add it to your Elastic Beanstalk configuration:
Elastic Beanstalk → Your Application → Your Environment → Configuration → Instances → EC2 security groups
Finally, you should be able to add and connect the EC2 instances using jvisualvm
.