I'm on site these last days and one of my tasks was to execute some load testing and monitor a Liferay instance hosted in a network accessed only via SSH port. No other port except 22 and 80 was open. So after almost 2 days struggling with several different configurations I was able to monitor a remote instance of Liferay through JMX+RMI over SSH tunneling. I had to put together several pieces of information to get this working, let's see the steps.
I will split this post in two configuration sections for JMX server (Tomcat) and JMX client (VisualVM).
JMX server
First of all you need to drop in the catalina-jmx-remote.jar to TOMCAT_HOME/lib directory of Tomcat. This will add to the classpath the listener
org.apache.catalina.mbeans.JmxRemoteLifecycleListener responsible for enable the JMX+RMI over SSH tunneling. More info about this listener can be found at
JMX Remote Lifecycle Listener. This jar may be found under extras directory of download binary area of Tomcat. Only versions newer or equal to 6.0.24 has it available. Your second step is configure the listener on
server.xml add the following snippet under the <Server> tag (around line 35).
<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002" useLocalPorts="true" />
The third and last step on the server side is set new parameters to JVM as following to enable JMX remote. Set them on the setenv.(sh|bat) of your Tomcat installation
JAVA_OPTS="$JAVA_OPTS -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
JMX client
Again your first step is drop in the jar catalina-jmx-remote.jar into the VISUALVM_HOME/platform/lib. This is a required step otherwise you gonna get a ClassNotFoundException for JmxRemoteLifecycleListener. After that you must start a SSH connection to your server with two tunnel enabled, you can use the command below
ssh -vN -L10001:localhost:10001 -L10002:localhost:10002 myuser@www.myliferayremoteserver.com
Then you can fire a VisualVM and add a reference to your remote server under the Remote section. Finally you should open a connection using this connection string service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi and voilà! We are done and ready to start the real work!
Other considerations
If you are trying to monitor a cluster of machines be aware that the property java.rmi.server.hostname=localhost will break the cache replication.
Maybe your are wondering about the reference of localhost in almost all commands instead of the server's names, right? Don't worry this is the right way beacuse we are using SSH tunnelling.
Some folks would like to work with jconsole. Server configuration are identical. On the client side you gonna fire jconsole with the following parameter -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:catalina-jmx-remote.jar to be able to connect.
If you have connection problems I would suggest you to use jconsole to debug. You can use the follwoing command line and logging.properties file to enable debugging
jconsole -J-Djava.util.logging.config.file=logging.properties -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:catalina-jmx-remote.jar
// logging.properties file to activate JMX traces.
// Use this file on the java command line:
// java -Djava.util.logging.conig.file=<logging.properties>
//
handlers= java.util.logging.ConsoleHandler
.level=INFO
java.util.logging.FileHandler.pattern = %h/java%u.log
java.util.logging.FileHandler.limit = 50000
java.util.logging.FileHandler.count = 1
java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter
java.util.logging.ConsoleHandler.level = FINEST
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
// Use FINER or FINEST for javax.management.remote.level - FINEST is
// very verbose...
//
javax.management.level=FINEST
javax.management.remote.level=FINER
Abraço!
