I use apache bench or http_load if non-ssl to simulate a bunch of users banging on the site while monitoring the memory usage in jvisualvm remotely and adjust accordingly til I get the behavior desired.
Given your hardware I'd assume you'll need some resources for other apps like backup etc. and recommend for a starting point a setenv.sh of;
1JAVA_OPTS="$JAVA_OPTS -XX:NewSize=700m -XX:MaxNewSize=700m -Xms2560m -Xmx2560m -XX:MaxPermSize=512m -Dfile.encoding=UTF8 -Duser.timezone=GMT -Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=20 -XX: ParallelGCThreads=8"
concatenate this to the above string to enable JMX;
1 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=33100
Please sign in to flag this as inappropriate.