Forums de discussion

Liferay Cache Replication.

thumbnail
Andew Jardine, modifié il y a 9 années.

Liferay Cache Replication.

Liferay Legend Publications: 2416 Date d'inscription: 22/12/10 Publications récentes
Hi Guys,

I think I have hit a wall. I have a 2 node cluster enabled. I am using a unicast configuration over tcp. In my portal-ext.properties I have the following settings.


        cluster.link.channel.properties.control=${catalina.base}/conf/unicast.xml
        cluster.link.channel.properties.transport.0=${catalina.base}/conf/unicast.xml
        ehcache.bootstrap.cache.loader.factory=com.liferay.portal.cache.ehcache.JGroupsBootstrapCacheLoaderFactory
        ehcache.cache.event.listener.factory=net.sf.ehcache.distribution.jgroups.JGroupsCacheReplicatorFactory
        ehcache.cache.manager.peer.provider.factory=net.sf.ehcache.distribution.jgroups.JGroupsCacheManagerPeerProviderFactory
        net.sf.ehcache.configurationResourceName.peerProviderProperties=file=/${catalina.base}/conf/unicast.xml
        ehcache.multi.vm.config.location.peerProviderProperties=file=/${catalina.base}/conf/unicast.xml
        lucene.replicate.write=true   


I have modified the liferay-multi-vm-clustered.xml configuration and added a cache to store my stuff. This is just a test -- but everything is the same, and at the end of that file I have added the following:


        <cache eternal="true" maxelementsinmemory="50000" name="com.yp.sample.liferay.portlet.SampleCachingPortlet" overflowtodisk="false" timetoidleseconds="600">
                <cacheeventlistenerfactory class="com.liferay.portal.cache.ehcache.LiferayCacheEventListenerFactory" properties="" propertySeparator="," />
        </cache>


I removed the properties from the string so that the defaults, replicatePuts etc are all set to true rather than false. I figured that this was the correct settings since I DO want to replication to occur. When I restart my server I use jVisualVM, the MBeans tab and am able to see my custom cache as defined above. The attributes for replication are not exposed, but I assume that they are the defaults since I gave it none.

I also turned on the ClusterBridgeMessageListener to INFO level logging so that I could see replication messages -- at least I assume that my write will be shown. When I start the server I can see INFO from this class --


02:49:16,714 INFO  [liferay/search_writer/SYSTEM_ENGINE-1][ClusterBridgeMessageListener:52] Bridging cluster link multicast message {destinationName=liferay/search_writer/SYSTEM_ENGINE, response=null, responseDestinationName=null, responseId=null, payload={arguments=[com.liferay.portal.kernel.search.SearchContext@257d9d68, {uid=[PluginPackageIndexer_PORTLET_liferay/notifications-portlet/6.2.0.3/war], entryClassName=[com.liferay.portal.kernel.plugin.PluginPackage], moduleId=[liferay/notifications-portlet/6.2.0.3/war]...


however I noticed that they always reference the lucene indexer thread. The individual caches all have replication of puts set to false so I am guessing that I see these messages only when lucene index writes occur. But, for my own CUSTOM cache I defined, I would expect to see messages when I put stuff in the cache.

In my test portlet I have --


...
String key = "key";
String value =  ParamUtil.getString(actionRequest, "value", null);

PortalCache cache = MultiVMPoolUtil.getCache(cacheName);
cache.put(key,value);
...


.. and I can see them going into the cache. And in a single node scenario I know it is working. But when I introduce the second node, my custom writes to my custom cache are not available on the other server.

Anyone have an idea what I am missing here?
thumbnail
David H Nebinger, modifié il y a 9 années.

RE: Liferay Cache Replication.

Liferay Legend Publications: 14914 Date d'inscription: 02/09/06 Publications récentes
Andew Jardine:
Anyone have an idea what I am missing here?


Yep. Solr.

Seriously, when you're in a cluster config you have two options for lucene. The first is that each node maintains lucene separately. This works while both nodes are up, but when you take a node out of the cluster that lucene index will fall behind. The second option is that they both update a shared lucene in a shared filesystem. This you have to worry about locking issues and the poor performance that comes with a networked filesystem.

When setting up a clustered environment, it's always best to choose a clustered indexing system. With Solr in place you wouldn't have to worry about individual nodes keeping their own indexes up to date, wouldn't have to worry about contention issues, etc.

It really is the only way to fly...
thumbnail
Andew Jardine, modifié il y a 9 années.

RE: Liferay Cache Replication.

Liferay Legend Publications: 2416 Date d'inscription: 22/12/10 Publications récentes
Hey David!

Thank God someone answered LOL. My issue isn;t actually with the indexer -- this is working for me. The issue is that when I put something in my custom cache, on Node A for example, and then I check on Node B, the same item is not there. It doesn't look like my "custom cache" is replicating and I can't figure out why.

Do you see anything (from my original message) that would be a problem? missing configuration maybe?
thumbnail
David H Nebinger, modifié il y a 9 années.

RE: Liferay Cache Replication.

Liferay Legend Publications: 14914 Date d'inscription: 02/09/06 Publications récentes
No, but ehcache isn't the best tool for cluster cache replication. Terracota, etc., tend to be a little more reliable. I haven't been through the documentation in awhile, but there may be something there that might help out: http://ehcache.org/documentation/2.6/replication/jgroups-replicated-caching.
thumbnail
Andew Jardine, modifié il y a 9 années.

RE: Liferay Cache Replication.

Liferay Legend Publications: 2416 Date d'inscription: 22/12/10 Publications récentes
Hey David,

Thanks for the link, I'll check it out. A few additional things I have noticed -- some of the cache configs contain


 <bootstrapcacheloaderfactory class="com.liferay.portal.cache.ehcache.LiferayBootstrapCacheLoaderFactory" />


I added that to my configuration and then checked the class out and found


		if (PropsValues.CLUSTER_LINK_ENABLED &amp;&amp;
			PropsValues.EHCACHE_CLUSTER_LINK_REPLICATION_ENABLED) {
...


The second property didn't ring a bell (for 6.2). The first I have set to true but when I checked the portal.properties for the second -- it was missing! The getter util I believe returns false for missing properties so that is a big problem! I've added it to my portal-ext.properties as true and am testing further. Fingers crossed that it the root of my problem.
thumbnail
David H Nebinger, modifié il y a 9 années.

RE: Liferay Cache Replication.

Liferay Legend Publications: 14914 Date d'inscription: 02/09/06 Publications récentes
Great! Good luck, Andrew, I'll keep my fingers crossed too emoticon
thumbnail
Jack Bakker, modifié il y a 9 années.

RE: Liferay Cache Replication.

Liferay Master Publications: 978 Date d'inscription: 03/01/10 Publications récentes
link I give here perhaps useful, though I haven't scaled into this territory myself ... yet

https://www.liferay.com/web/mika.koivisto/blog/-/blogs/how-do-i-cluster-liferay-with-terracotta-
thumbnail
Juan Gonzalez, modifié il y a 9 années.

RE: Liferay Cache Replication.

Liferay Legend Publications: 3089 Date d'inscription: 28/10/08 Publications récentes
Hi Andrew,

did it finally worked for you? I see some parts that you may check:

1-You said you modified liferay-multi-vm-clustered.xml. Where did you put that file back? You should specify the new location through this property:

ehcache.multi.vm.config.location=<your_new_file_path></your_new_file_path>


2-Did you put

singleton_name="<your_cluster_name>"</your_cluster_name>


in your unicast.xml so all caches shares the same transport?
thumbnail
Andew Jardine, modifié il y a 9 années.

RE: Liferay Cache Replication. (Réponse)

Liferay Legend Publications: 2416 Date d'inscription: 22/12/10 Publications récentes
Hey Juan,

I did manage to get it working -- and one of the key pieces was indeed that singleton_name setting as part of the TCP envelope in the config file. Just in case someone else comes across this post with the same problem -- here is the full setup I used.

NOTE: This is for Unicast clustering over TCP, and of course, it needs to be done for each node in your cluster.

1. Go to your TOMCAT_HOME/webapps/ROOT/WEB-INF/lib (assuming ROOT is your Liferay Portal application) and crack open the jgroup.jar.
2. Extract from that jar the tcp.xml file and place it in the TOMCAT_HOME/conf folder
3. Open the TOMCAT_HOME/conf/tcp.xml file and for the TCP envelope (<TCP) add as the first attribute sungleton_name=<my name>
4. Save and close the file.
5. Open the TOMCAT_HOME/bin/setenv.sh file and add the following CATALINA_OPTS settings --


CATALINA_OPTS="$CATALINA_OPTS -Djgroups.bind_addr=<current node ip>"
CATALINA_OPTS="$CATALINA_OPTS -Djgroups.tcpping.initial_hosts=<nodea_ip>[7800],<nodeb_ip[7800],...,<node[i]n[ i]_ip>[7800]"
</nodeb_ip[7800],...,<node[i]n[></nodea_ip></current>


6. Save and close the file.
7. Open your portal-ext.properties file and add the following lines


## ------------------------------------------------------------------------------------
## Cluster
##
## NOTE: Additional changes to the JVM startup options are also required.
## -Djava.net.preferIPv4=true
## -Djgroups.bind_addr=<local ip -- must be set for each server>
## -Djgroups.tcpping.initial_hosts=<node1>[7800],<node2>[7800],...,<noden>[7800]
##
        # enable liferay clustering
        cluster.link.enabled=true

        # enable debug mode to see some useful messages to validate clustering is working
        cluster.executor.debug.enabled=true

        # this value needs to be set to something that is reacheable at startup. There
        # could be situations where protected networks don't allow access, so the default
        # of google.com:80 will fail. Best option here is to set this to the same value
        # as the database server since we know at startup it will be reacheable.
        cluster.link.autodetect.address=<some reacheable ip>

        # Provide the configuration files that are required to establish the channels. Liferay uses
        # JGroups for its clustering capabilities. The file referenced here (tcp.xml) is pulled from the
        # jgroups.jar found in TOMCAT_HOME/webapps/ROOT/WEB-INF/lib. 
        cluster.link.channel.properties.control=${catalina.base}/conf/tcp.xml
        cluster.link.channel.properties.transport.0=${catalina.base}/conf/tcp.xml
 

## ------------------------------------------------------------------------------------
## Ehcache
##
# location of cache configuration file (relative to webapps/ROOT/WEB-INF/classes. The purpose
# of this file is to provide custom caches or to override default settings shipped with Liferay
# ehcache.multi.vm.config.location=/META-INF/liferay-multi-vm-clustered.xml
# location of configuration file used to establish the TCP/Unicast chanel for replication. See cluster link section
# for more details.
        net.sf.ehcache.configurationResourceName.peerProviderProperties=file=${catalina.base}/conf/tcp.xml
        ehcache.multi.vm.config.location.peerProviderProperties=file=${catalina.base}/tcp.xml

        # Settings for classes responsible for loading caches and  listening/handling cache events.
        ehcache.bootstrap.cache.loader.factory=com.liferay.portal.cache.ehcache.JGroupsBootstrapCacheLoaderFactory
        ehcache.cache.event.listener.factory=net.sf.ehcache.distribution.jgroups.JGroupsCacheReplicatorFactory
        ehcache.cache.manager.peer.provider.factory=net.sf.ehcache.distribution.jgroups.JGroupsCacheManagerPeerProviderFactory

## ------------------------------------------------------------------------------------
## Search
##
       # enable replication of search index across the cluster. This should be enabled in environments
       # where there is no central search server/cluster. By default Liferay will store the search index
       # on the local disk in LIFERAY_HOME/data/lucene -- this setting ensures that with this configuration,
       # changes to the index on one node are sent to the others. 
       lucene.replicate.write=true
</some></noden></node2></node1></local>


8. Save and close the file
9. Restart the server.

In my case I am also providing custom cache objects so I took the liferay-multi-vm-clustered.xml file from the source tree, added my custom definitions and then placed it into the webapps/ROOT/WEB-INF/classes/META-INF folder. If you do this, then you also have to provide this setting in your portal-ext.properties file --


ehcache.multi.vm.config.location=/META-INF/liferay-multi-vm-clustered.xml


Also, by default the caches are set not to replicate puts. You can reference the ehcache documentation, but to provide an example here, this is the cache config I used to make sure everything was working.


        <cache eternal="true" maxelementsinmemory="50000" name="com.yp.sample.liferay.portlet.SampleCachingPortlet" overflowtodisk="false" timetoidleseconds="600">
                <cacheeventlistenerfactory class="com.liferay.portal.cache.ehcache.LiferayCacheEventListenerFactory" properties="replicatePuts=true,replicateUpdates=true,replicateRemovals=true,replicateAsynchronously=true,replicateUpdatesViaCopy=true" propertySeparator="," />
                <bootstrapcacheloaderfactory class="com.liferay.portal.cache.ehcache.LiferayBootstrapCacheLoaderFactory" />
        </cache>


Notice the properties are set to replicate all actions. All this worked for me.
thumbnail
David H Nebinger, modifié il y a 9 années.

RE: Liferay Cache Replication.

Liferay Legend Publications: 14914 Date d'inscription: 02/09/06 Publications récentes
Wahoo, Andrew, nice work and thanks for sharing.
thumbnail
Jack Bakker, modifié il y a 9 années.

RE: Liferay Cache Replication.

Liferay Master Publications: 978 Date d'inscription: 03/01/10 Publications récentes
Awesome Andrew !
thumbnail
Daniel Tyger, modifié il y a 8 années.

RE: Liferay Cache Replication.

Regular Member Publications: 105 Date d'inscription: 06/02/13 Publications récentes
Andrew - thank you very much for sharing your setup / steps. This helped us, tremendously. I only want to share our related experience that makes this setup a little trimmer. I found that some of your properties are not used when implementing the ehcache EE plugin and ClusterLink:

Only these steps / props were needed to get our unicast cluster up / running fine (2-node cluster):

1) In {tomcat}/bin/setenv.sh add:
-Djgroups.tcpping.initial_hosts=10.current.node.ip[7800],10.other.node.ip[7800] \

2) As described, set up the tcp.xml file and place in: {tomcat}/conf/. singleton_name= property is critical.

3) Some of the properties you listed are actually overridden by the EhCache EE plugin (if you are using that) - Only these are needed in Properties (portal-ext) with that plugin in play:
## DEBUG:
cluster.executor.debug.enabled=true
##
## Indexing
##
cluster.link.enabled=true
lucene.replicate.write=true
##
## Need a gateway address here...e.g.:
cluster.link.autodetect.address=dbhost:3306
##
## Caching - Install the EhCache EE plugin from Mktplace
##
ehcache.cluster.link.replication.enabled=true
##
## Unicast settings
##
cluster.link.channel.properties.control=${catalina.base}/conf/xtivia-tcp.xml
cluster.link.channel.properties.transport.0=${catalina.base}/conf/xtivia-tcp.xml

Custom cache options remain after these steps only if you want to override the existing - which can also be done via hook plugin for hot deployable change options, as described in Liferay 6.2 Cluster Documentation, here.

As a side note - for the interested - we went round and round for a while with either multicast / unicast failing due to overly-restrictive firewall settings... opening all the addresses / ports that Liferay docs say are used for multicasting is not enough. Multicast send / receive tests may work fine, but the cluster may not work. Even allowing all broadcast / multicast traffic was not enough. It seems more communication channels / addresses are utilized by Liferay during cluster communications. You may want to add a rule on each node firewall that lets any traffic come from the companion node(s) -s addresses. This let us see successful tests then refine the fw post setup...
thumbnail
Pier Paolo Ramon, modifié il y a 9 années.

R: Liferay Cache Replication.

Junior Member Publications: 90 Date d'inscription: 25/05/10 Publications récentes
Glad you find an answer! Could you please set your reply as the answer to your question so it's marked as solved?

Sent from my iPhone with Liferay.com Forums