Foros de discusión

(Resolved) Clustering ehcache on 4.3.5

thumbnail
Alex Wallace, modificado hace 16 años.

(Resolved) Clustering ehcache on 4.3.5

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
I was really trying to do it on our customized 4.3.1 but decided to do it on 4.3.5 to see if I could get it to work...

I'm only doing EHCACHE right now. What does it take to make two liferay instances on two separate machines to update each other's db cache?

I placed this on both servers in portal-ext.properties


   net.sf.ehcache.configurationResourceName=/ehcache/hibernate-clustered.xml
   ehcache.multi.vm.config.location=/ehcache/liferay-multi-vm-clustered.xml


Ignoring if the following was necessary, i also placed it on each of the servers (adjusted the binding address for each)


comm.link.properties=UDP(bind_addr=192.168.1.35;mcast_addr=231.12.21.102;mcast_port=45566;ip_ttl=32;mcast_send_buf_size=150000;mcast_recv_buf_size=80000):PING(timeout=2000;num_initial_member\
s=3):MERGE2(min_interval=5000;max_interval=10000):FD_SOCK:VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(gc_lag=50;retransmit_timeout=300,600,1200,2400,4800;max_xmit_size=8192):UNICAST(timeout=300,6\
00,1200,2400):pbcast.STABLE(desired_avg_gossip=20000):FRAG(frag_size=8096;down_thread=false;up_thread=false):pbcast.GMS(join_timeout=5000;join_retry_timeout=2000;shun=false;print_local_addr=true\
)


Then i fired up both servers... Added blogs portlet to a page, change an entry on one, and the second server doesn't refresh the changes...

Nothing is displayed on the logs that indicates that the machines are talking to each other...

I'm sure I'm missing something!

I'll appreciate very much your help here...

Thanks!
thumbnail
Alex Wallace, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
I doubt it would make any difference, but i just stup also the <cluster section in tomcat's server.conf

It made no difference in regards to the ehcache replication as I suspecter... but thought it would not hurt...
thumbnail
Alex Wallace, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
I went back to trying to cluster oscache on my custom 4.3.1 using it's own hibernate file... And seem to have gotten it to work in one direction: Updates on server B affect server A, but not the other way around... That's pretty wierd, but sem that I've made some progress... I'm about to diff the entire appservers ...

Next i will try the liferay-multi-vm file ...
thumbnail
Alex Wallace, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
Actually it was only the multi-vm file that was working... I suspect the db cache is not working at all and it is only the content cache that is being replicated... This suspicion is because when i refresh a page on server A, i do see some changes made in server B, but if i use the portlet that displays the changes (blogs) to try to edit the some info, the info cached locally is displayed instead of the up to date one...
thumbnail
Alex Wallace, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
I think i got the whole thing to work! for some unkown reason, the debian servers i was using were not transmitting anything to each other... but i was able to use some Macs in the cluster and those worked fine... So i blame it on debian and don't really care right now since the actual OS on the cluster will not be debian... It would be nice to figure out why debian seems to be blocking all the multicast packets... multicast shows as enabled on the eth adaptors... anyways...

thanks!
thumbnail
Artur Linhart, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Expert Mensajes: 459 Fecha de incorporación: 13/09/07 Mensajes recientes
Hello,

I must solve very similar problem. I also have liferay 4.3.5 running under debian etch - but I have it bundled with tomcat 6. Do You have also tomcat 6?

I found something similar to the error You have described in the other forum:

Cause was: error unmarshalling return; nested exception is:        
java.io.EOFException


I have tested If the problem is really the debian and I do not think so - I connected with the sniffing tool to the network between the both servers and there are running the multicast packets to the address 230.0.0.2 port 4447 - like expected.

So I do not think the problem is in Debian itself, at least in my case. I found to this topic something on the page http://ehcache.sourceforge.net/documentation/tomcat.html to this topic.

It looks like this is related to the problems with RMI under Tomcat. The problem should be connected with the classloaders - like comes from the Tomcat 6 documentation on http://tomcat.apache.org/tomcat-6.0-doc/class-loader-howto.html, from the perspective of a web application, class or resource loading looks in the following repositories, in this order:
  • Bootstrap classes of your JVM
  • System class loader classses (described above)
  • /WEB-INF/classes of your web application
  • /WEB-INF/lib/*.jar of your web application
  • $CATALINA_HOME/lib
  • $CATALINA_HOME/lib/*.jar


So, they propose in ehcache documentation to move the ehcache.jar (and in the case of other problems also backport-concurrent.jar) into the common libs of Tomcat. I tried it but this was not enough. I think here have to be differentiated between tomcat 5.x and tomcat 6.0, because the libraries in commons used in tomcat 6 have been completely changed (they made their own stuff to avoid the conflicts with the previously used libraries like commons-* etc.). So, maybe this will then work in the case Tomcat 5.x. In the case of the Tomcat 6 I had to move following jars from webapps/ROOT/WEB-INF/lib into the lib/ext folder:
  • backport-concurrent.jar
  • commons-collections.jar
  • commons-logging.jar
  • dom4j.jar
  • ehcache.jar
  • hibernate3.jar
  • jaxen.jar

- otherwise the portal application did not even start or process the first request. But after I made it, it looks so the cache replication is trying to work "better" - how fast should be the replication between the cluster nodes? I tried to have two sessions,e very from the different node. The I tried to change the page - sometimes the change is on the second site within a second, sometimes it is there not after more than 5-10 seconds... What does it mean-does it work or not? :-)
thumbnail
Artur Linhart, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Expert Mensajes: 459 Fecha de incorporación: 13/09/07 Mensajes recientes
Unfortunatelly it did not really help and I have no idea what to do. But in my case I think the problem is not on Debian side... emoticon
thumbnail
Artur Linhart, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Expert Mensajes: 459 Fecha de incorporación: 13/09/07 Mensajes recientes
So, finally I got it.

There are not a lot of things to do, main point is - DO AVOID MYSTAKES :-)

Short summary of the configuration steps and possible problems and advices to get it to work with debian etch, tomcat 6.0.13, jdk 1.5.0_10, Liferay 4.3.5:

1. Be sure to have correctly configured load balancer with sticky sessions (do not forget to specify jvmRoute in the Engine tag of the server.xml on all tomcat nodes)

2. Add into the portal-ext.properties the two lines
net.sf.ehcache.configurationResourceName=/ehcache/hibernate-clustered.xml
ehcache.multi.vm.config.location=/ehcache/liferay-multi-vm-clustered.xml

- and be REALLY SURE You have written it correctly

3. Restart the whole tomcat machine (theoretically it is enough to restart tomcat only, but sometimes there hung some java processes after the tomcat has been shut down, so I think this is the best way how to avoid problems)

... and that's it.


What can be usefull in the case it does not work:

A. the advice with RMI Problems on Tomcat (copy the libaries into the common libs of tomcat) did not worked for me - or is not necessary, there had to be copied a lot of libraries... which could then cause some conflict with evt. own possible installed portlets

B. If debugging the behavior of the ehcache, it is enough to set the debug level in the Admin Portlet (I used DEBUG level by the net.sf.ehcache and net.sf.hibernate keys) - the default DEBUG option in log4j.properties is not necessary to be set to DEBUG.

C. If You have multiple clusters running on the same network segment, it is better - or maybe necessary - to change the multicast port on which the clusters are communicating. so all nodes in one cluster should have the same multicast port which should be other like have the nodes, communicating together with the second cluster. So, for example I had 2 virtual linux machines, so 2 IP addresses and on every machine has been defined 2 liferay (tomcat) instances. The also the configuration had to be changed according to this fact to avoid problems in the cache communication.
So, on the first installation there was enough to perform the configuration like described above, on the second instance of liferay (tomcat) I have taken the both files
/ehcache/liferay-multi-vm-clustered.xml
and
/ehcache/hibernate-clustered.xml
from the jar
{tomcat}/webapps/ROOT/WEB-INF/lib/portal-impl.jar
and have copied them into the new directory
{tomcat}/webapps/ROOT/WEB-INF/classes/ehcache
(this has priority before the default settings located in the jar) and performed following modifications:
  • In the file liferay-multi-vm-clustered.xml I changed the default configuration

    <cacheManagerPeerProviderFactory
    class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
    properties="peerDiscovery=automatic,multicastGroupAddress=230.0.0.2,multicastGroupPort=4447,timeToLive=1"
    propertySeparator=","
    />

    into the configuration

    <cacheManagerPeerProviderFactory
    class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
    properties="peerDiscovery=automatic,multicastGroupAddress=230.0.0.2,multicastGroupPort=5557,timeToLive=1"
    propertySeparator=","
    />
  • In the file hibernate-clustered.xml I changed the default configuration

    <cacheManagerPeerProviderFactory
    class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
    properties="peerDiscovery=automatic,multicastGroupAddress=230.0.0.1,multicastGroupPort=4446,timeToLive=1"
    propertySeparator=","
    />
    <cacheManagerPeerListenerFactory
    class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
    />

    into the configuration

    <cacheManagerPeerProviderFactory
    class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
    properties="peerDiscovery=automatic,multicastGroupAddress=230.0.0.1,multicastGroupPort=5556,timeToLive=1"
    propertySeparator=","
    />
    <cacheManagerPeerListenerFactory
    class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
    />

- both steps have to be performed on both (all) corresponding nodes of one cluster.
I think, because the cache is communicationg over multicast packets, it does not matter if the cluster nodes are located on the same machine or not, IMHO this has to be performed always if two different clusters have to be running on the same network segment (so, for example as 4 physical machines, etc).
thumbnail
Alex Wallace, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
Arthur. I'm sorry i did not chime in earlier... For some reason i was not notified of posts to this ticket...

I'm glad you got it to work tho...

Indeed it ended up working for me also, using the files you mention above...

Howerver i did encounter wierd things... It worked fine right away using Mac computers... I have not tested Debian, since production uses Gentoo, I tried Gentoo.... At first it would not work, but gentoo was missing multicast at the kernel level...

I also added a solaris box in the mix for testing....

I did not test using a load balancer, I was hitting the 2 different IPs on 2 browser windows and confirming the cache was being propagated...

I used portal-log4j-ext file for enabling debugging accross restarts...

What i saw happenning, and this was indeed an OS config issue, is that some machines add themselfs to the mcast as 127.0.0.1, which of course is wrong wrong... and causes attempts to update caches on all nodes at 127.0.0.1... Have you seen this? I'd like to know how to force each cluster to bind to a particular ip...

anyways... glad it worked for you!

thanks!
thumbnail
Artur Linhart, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Expert Mensajes: 459 Fecha de incorporación: 13/09/07 Mensajes recientes
Hm :-(
Unfortunatelly, I did not have this situation in my case...

According to the OS - I can only say what I use exactly - so I use the virtual servers, runing as Xen instances in paravirtual mode on x64 Intel processor box. I did not compile some specific kernel for it, but took the "xen vanilla" kernel coming with the binary tarball of open source xen 3.1.0 - so the kernel 2.6.18-xen for the linux domains, ruing over virtualization realized with xen-3.1.0. As far as I know, Gentoo can be also Xenified, so maybe it there are really some kernel problems, this kernel could help - or You could just install xen and run the gentoo like virtual servers :-) OK, this would be much more effort than necessary, I hope... But if all other ways fail... This could help at least find the differences between the both kernels...

If You test your clusters, do You have only "one cluster" separated from the other one? Because of my last remark in my previous post - I thin the cache got confused because using the same port like the other clusters did...

In the attachment is the config of the kernel which I use in virtual instances - 2.6.18-xen... So maybe You can compare it with Your kernels if it helps to figure out some options for the multicasting stuff... Unfortunatelly I never used multicasting for something, so I cannot help so much on the OS configuration level...

I hope this could help a little bit...

Do You have some log examples from the gentoo machines reporting the errors?

Good luck, Archie

P.S. How do You test if the cache is synchronized? I just tried to change something in the page - the portlet order, etc... Or just tried tomodify some data and perform a refresh in the second window.. Maybe it is not enough and something still does not work also in my case - but after I configured it properly, all the tests I made were working properly...
.
thumbnail
Alex Wallace, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
thanks a lot for all the info...

indeed after adding some modules to the kernel gentoo worked... the issue with 127.0.0.1 was dealt with changing /etc/hosts, but i don't know is that's the best way... i would still like to try to force each cluster to bind to a specific IP...

I'll try to generate some logging info... but most of it was connection refused on 127.0.0.1 ... that's pretty much cuz each machine took that IP as themselfs...

We only have one cluster on this test network, so i didn't have to change ports...

I also tested the same way you did, by changing a blog entry on one node and making sure the other node received a request to update the cache and it reflected it...

thanks for all!
thumbnail
Alex Wallace, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
did you enable tomcat clustering? are you replicating your sessions?

i wonder if we should disable that as i've heard it is not efficient... i'm not sure yet how to disqble it, but i belive there is a place in ROOT.xml to do it...
thumbnail
Artur Linhart, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Expert Mensajes: 459 Fecha de incorporación: 13/09/07 Mensajes recientes
Hello,

no, I did not use the fail-over sessions or session replication...

I have only specified the jvmRoute in load balancer and did noch change ROOT.xml in any way...

cheers, Archie

P.S. I posted before one new message for You how to configure the static distribution hosts for ehcache, You can try it :-)
thumbnail
Artur Linhart, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Expert Mensajes: 459 Fecha de incorporación: 13/09/07 Mensajes recientes
One more idera - but I think this is rather a wish - how do You start tomcat, with or without security manager? I start it without... I have seen in the internet some posts, where were also the problems with the connections to the address 127.0.0.1 in the similar situations and this was conected with the insufficient settings of the security manager, whcih did not allowed the RMI accesses - so something little bit similar to ehcache since it uses also RMI if I understand it correctly.


Did You also tried the ehcache debugger or some sniffing tool to monitor traffic going in the network with clusters? This could be some help - but in my case I could not get the ehcache debugger working...
thumbnail
Alex Wallace, modificado hace 16 años.

Re: [Liferay Forums][1. Installation / Deployment / Setup]RE: Clustering eh

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
that's very interesting info... will keep this in mind... It may be a
combination of things... I know for sure that some servers were
entering the cluster reporting themselfs as 127.0.0.1 but the
connection refused issue could very well be what you list here...

thanks!


On Jan 10, 2008, at 1:42 PM, Artur Linhart at Liferay's Community
Forums wrote:

> One more idera - but I think this is rather a wish - how do You
> start tomcat, with or without security manager? I start it
> without... I have seen in the internet some posts, where were also
> the problems with the connections to the address 127.0.0.1 in the
> similar situations and this was conected with the insufficient
> settings of the security manager, whcih did not allowed the RMI
> accesses - so something little bit similar to ehcache since it uses
> also RMI if I understand it correctly
> --
> Liferay Community Forum
> mb.239393.387086@events.liferay.com
> http://www.liferay.com/web/guest/community/forums/message_boards/
> message/387086
thumbnail
Alex Wallace, modificado hace 16 años.

Re: [Liferay Forums][1. Installation / Deployment / Setup]RE: Clustering eh

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
haven't really tried the debugger... i had not seen there was one....
I know it can be monitored via JMX but felt i had to write my own
module... I'll look for it and see if i can get it to work...

thanks!

On Jan 10, 2008, at 1:43 PM, Artur Linhart at Liferay's Community
Forums wrote:

> One more idera - but I think this is rather a wish - how do You
> start tomcat, with or without security manager? I start it
> without... I have seen in the internet some posts, where were also
> the problems with the connections to the address 127.0.0.1 in the
> similar situations and this was conected with the insufficient
> settings of the security manager, whcih did not allowed the RMI
> accesses - so something little bit similar to ehcache since it uses
> also RMI if I understand it correctly.
>
>
> Did You also tried the ehcache debugger or some sniffing tool to
> monitor traffic going in the network with clusters? This could be
> some help - but in my case I could not get the ehcache debugger
> working...
> --
> Liferay Community Forum
> mb.239393.387086@events.liferay.com
> http://www.liferay.com/web/guest/community/forums/message_boards/
> message/387086
thumbnail
Artur Linhart, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Expert Mensajes: 459 Fecha de incorporación: 13/09/07 Mensajes recientes
One more thing - the static distribution list should be also possible to specify, like they write in
http://ehcache.sourceforge.net/EhcacheUserGuide.html#id.s19.7.2...

Hey, in this manual there is also the example of the manual configuration:


    Configuring Manual Discovery:
    Manual discovery is configured as per the following example:
    <cachemanagerpeerproviderfactory class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory" properties="peerDiscovery=manual,
                          rmiUrls=//server1:40000/sampleCache1|//server2:40000/sampleCache1
                          | //server1:40000/sampleCache2|//server2:40000/sampleCache2" propertySeparator="," />

    Valid properties are:
    * peerDiscovery (mandatory) - specify "manual"
    * rmiUrls (mandatory) - specify a pipe separated list of rmiUrls, in the form
                            //hostname:port

    The hostname is the hostname of the remote CacheManager peer. The port is the listening
    port of the RMICacheManagerPeerListener of the remote CacheManager peer.


So You could test it with direct connections :-)
thumbnail
Alex Wallace, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
thanks for that info... I did see something like that, but in my understanding this doesn't use multicast, or does it? In all honesty i should reread the manual... my impression is that under certain configuration, the communication between servers would be about sending N-1 messages per update, where with multicast it is always only one...

I'm thinking more along the lines of making each server bind to a specific IP address... for instance, if you have 3 adaptors on one machine, forcing tomcat and ehcache to only use one particular one...

I know tomcat can be told what address to bind to, tomcat clustering can be told what address to listen to for multicast... I will search to see if ehcache can be told what address to bind to too...

as always, thank you very much!
thumbnail
Artur Linhart, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Expert Mensajes: 459 Fecha de incorporación: 13/09/07 Mensajes recientes
:-) You are welcome...

Will You have much more nodes in Your cluster than 2? Because if no, then maybe the N-1 is not so big problem....

I think, if You get it running - at least for test - with the static address specification, then You can be sure the problem is on the multicast level only and concentrate on this one only... but it is possible, there are also some other problems, whcih cause the multicasts are generated in a bad form - and if this happens by the static mapping, then at least You can localize the problem better without having to deal with the unclear (at least for me:-) ) behavior of multicasting packets in the IP subsystem...
thumbnail
Alex Wallace, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
thats a great idea... i'll give it a try in the next few days...

at the moment i have the setup working and the errors about the 127 gone by configuring stuff in the os.. mainly in /etc/hosts

but you make a very good point...
thumbnail
Alex Wallace, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
here is a sample of the errors in log i mentioned:


16:09:38,603 DEBUG [MulticastRMICacheManagerPeerProvider:118] Unable to lookup remote cache peer for //127.0.0.1:64435/com.liferay.portal.security.permission.PermissionCacheUtil. Removing from peer list. Cause was: Connection refused to host: 127.0.0.1; nested exception is: 
        java.net.ConnectException: Connection refused
16:09:38,603 DEBUG [MulticastKeepaliveHeartbeatReceiver:173] Aborting processing of rmiUrls since failed to add rmiUrl: //127.0.0.1:64435/com.liferay.portal.security.permission.PermissionCacheUtil
16:09:43,604 DEBUG [MulticastRMICacheManagerPeerProvider:118] Unable to lookup remote cache peer for //127.0.0.1:64435/com.liferay.portal.security.permission.PermissionCacheUtil. Removing from peer list. Cause was: Connection refused to host: 127.0.0.1; nested exception is: 
        java.net.ConnectException: Connection refused
16:09:43,604 DEBUG [MulticastKeepaliveHeartbeatReceiver:173] Aborting processing of rmiUrls since failed to add rmiUrl: //127.0.0.1:64435/com.liferay.portal.security.permission.PermissionCacheUtil
16:09:48,604 DEBUG [MulticastRMICacheManagerPeerProvider:118] Unable to lookup remote cache peer for //127.0.0.1:64435/com.liferay.portal.security.permission.PermissionCacheUtil. Removing from peer list. Cause was: Connection refused to host: 127.0.0.1; nested exception is: 
        java.net.ConnectException: Connection refused
16:09:48,604 DEBUG [MulticastKeepaliveHeartbeatReceiver:173] Aborting processing of rmiUrls since failed to add rmiUrl: //127.0.0.1:64435/com.liferay.portal.security.permission.PermissionCacheUtil
16:09:53,607 DEBUG [MulticastRMICacheManagerPeerProvider:118] Unable to lookup remote cache peer for //127.0.0.1:64435/com.liferay.portal.security.permission.PermissionCacheUtil. Removing from peer list. Cause was: Connection refused to host: 127.0.0.1; nested exception is: 
        java.net.ConnectException: Connection refused
16:09:53,607 DEBUG [MulticastKeepaliveHeartbeatReceiver:173] Aborting processing of rmiUrls since failed to add rmiUrl: //127.0.0.1:64435/com.liferay.portal.security.permission.PermissionCacheUtil
thumbnail
Alex Wallace, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
to make the thread more useful, and also for my own learning I'd like to add:

1 - I'm planning on also doing the clustering for jackrabbit... Althought I've been wondering if i should not use jackrabbit and use a filesystem for all the documents, mounted on nfs...

another concern here I have is that I already have documents loaded in jackrabbit and that the clustering configuration i've seen uses the database... I'm not sure if i can still cluster jackrabbit and use an nfs located datastore...

if not, i know that i can export all the stuff from jackrabbit, and reimport it to the database... but i'll habe to write code for that... I've only found a tool that can export from File datastores, but not import to jdbc stores...

2 - lucene: my plan is to have each node in the cluster have it's own index, and then make them reindex periodically to stay in sync... this will be done with cuartz cron jobs that will check for items being updated in the databes that have, say, only 10 minutes of age... A full reindex will happen at night...

at one point we'll do a Solr implementation and migrate our searches to that...

there is also the question of failover with solr, as normally there is only one server that indexes, and multiple slaves... one interesting setup for solr is to do updates to multiple servers and not replicate the indexes at all... that gives you fail over at the expense of sending multiple updates when indexing...
thumbnail
Artur Linhart, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Expert Mensajes: 459 Fecha de incorporación: 13/09/07 Mensajes recientes
Alex Wallace:
to make the thread more useful, and also for my own learning I'd like to add:

1 - I'm planning on also doing the clustering for jackrabbit... Althought I've been wondering if i should not use jackrabbit and use a filesystem for all the documents, mounted on nfs...

another concern here I have is that I already have documents loaded in jackrabbit and that the clustering configuration i've seen uses the database... I'm not sure if i can still cluster jackrabbit and use an nfs located datastore...

if not, i know that i can export all the stuff from jackrabbit, and reimport it to the database... but i'll habe to write code for that... I've only found a tool that can export from File datastores, but not import to jdbc stores...

2 - lucene: my plan is to have each node in the cluster have it's own index, and then make them reindex periodically to stay in sync... this will be done with cuartz cron jobs that will check for items being updated in the databes that have, say, only 10 minutes of age... A full reindex will happen at night...

at one point we'll do a Solr implementation and migrate our searches to that...

there is also the question of failover with solr, as normally there is only one server that indexes, and multiple slaves... one interesting setup for solr is to do updates to multiple servers and not replicate the indexes at all... that gives you fail over at the expense of sending multiple updates when indexing...


Ad jackrabbit
- this is the theme for me as well... I prefer to store everything in database... I hope this will have no performance problems... According to the nfs - I think this should be possible, the only question would be for me:
- if jackrabbit caches somehow the found documents and files - but then it is the same problem if it is on nfs or in database.
- how are there solved conflicts by modifications of the same documents at the same time - but this is possibly so rare case so one can be "optimistic" and say this will ot happen... But if it really happens, then I could imagine it is better to have it in the database...

For me there is the question if the documents should be stored in the same database like is the liferay database. Do You have an idea, where the relations to the Liferay entities are stored? I guess this should be in Liferay DB... So, from this point of view to assure 100% consitency it would be good to share the same DB for liferay and documents. This would also reduce the management effort. In this case the DB would be always consistent if executing the backup. If having 2 databases then there is always the problem with the backup of 2 databases at oe time - I can do the "live" backup of one database, but I think I canot do backup of two schemas at the same time if the liferay will be still running...
But with nfs the consistent backup would be much more problematic than with the database.

Is there any possibility to use jackrabbit for the image gallery too? As a different repository... But maybe this would have not so much advantage... At least now if I thransferred the Liferay Db from production to the integration test infrastructure, there were all images, so I hope this is inside of the liferay DB :-)

Ad Lucene:
This will be interesting for us too, but I do not have any idea, because we do not use the searching yet - so I have to study first this theme theoretically. I also have no experiences with Solr - what is the main advantage to have such a liferay-independent service? That You can use ot also for other purposes than only indexing liferay? If it is so liferay-independent is it then possible to configure it so, the searches affect also the permissions or areas/communities, etc.? I hope if it is directly in Liferay, it is taken into the consideration... And as written in this thread, the Solr integrations hould be present in Liferay 5.0 which should come in some months, I hope - because the 4.4 should come in next days... So I personally would rather wait what will be present in the next version
thumbnail
Alex Wallace, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Liferay Master Mensajes: 640 Fecha de incorporación: 5/11/07 Mensajes recientes
Artur Linhart:

Ad jackrabbit
- this is the theme for me as well... I prefer to store everything in database...


I would not have any issues with having it all on the db, except that we already have jackrabbit content in production, which i'll need to move to the db... and for this, i'll have to write some code...

Artur Linhart:

- if jackrabbit caches somehow the found documents and files - but then it is the same problem if it is on nfs or in database.
- how are there solved conflicts by modifications of the same documents at the same time - but this is possibly so rare case so one can be "optimistic" and say this will ot happen... But if it really happens, then I could imagine it is better to have it in the database...


Jackrabbit can be clustered... In theory is just a matter of changing the configuration in the repository.xml file... If you look at it, you'll see what in theory needs to be done to make it use the db and be clustered... The theoretical purpose of that is to solve the issues you pose above, amongst other things...

Artur Linhart:

For me there is the question if the documents should be stored in the same database like is the liferay database. Do You have an idea, where the relations to the Liferay entities are stored? I guess this should be in Liferay DB...


AFAIK the jackrabbit jdbc store creates its own filesystem on the db (based on config)... The db store would not normally be used as relational db tables to retrieve the documents... And yes, i'm pretty sure that the relatinship to that data is kept in liferay's tables... So i think that it would not hurt you if you use a separate db for jackrabbit... imho...

Artur Linhart:

Is there any possibility to use jackrabbit for the image gallery too? As a different repository... But maybe this would have not so much advantage... At least now if I thransferred the Liferay Db from production to the integration test infrastructure, there were all images, so I hope this is inside of the liferay DB :-)


the images (by default) are saved in the db... so i don't think thew can go with jackrabbit w/o code changes...


Artur Linhart:

Ad Lucene:


The advantage of having Solr vs Lucene is that it is a server... While lucene out of the box runs inline with your VM.... One could of course expose lucene via web services to achieve the same (which is partially what Solr does, althought it does add some improvements/enhancements in other areas as well) .... When your search engine can be in a different box, you can scale it better...

I would definitely advise to wait for Liferay to have Solr... we may do that as well... I depends on how bad we need it... If we get to it, i'll contact Liferay to see if we can develop in a way that liferay could reuse... we'll see...

thanks!
thumbnail
Artur Linhart, modificado hace 16 años.

RE: Clustering ehcache on 4.3.5

Expert Mensajes: 459 Fecha de incorporación: 13/09/07 Mensajes recientes
Alex Wallace:

Artur Linhart:

Ad Lucene:


The advantage of having Solr vs Lucene is that it is a server... While lucene out of the box runs inline with your VM.... One could of course expose lucene via web services to achieve the same (which is partially what Solr does, althought it does add some improvements/enhancements in other areas as well) .... When your search engine can be in a different box, you can scale it better...

I would definitely advise to wait for Liferay to have Solr... we may do that as well... I depends on how bad we need it... If we get to it, i'll contact Liferay to see if we can develop in a way that liferay could reuse... we'll see...

thanks!


Yes, it is clear - but I do not know how the integration of portal with Lucene works - Of course, it would be very nice to have some independent server which is able to perform indexing for various services/sites but I do not know, it then the integration according to the permissions etc. can be done very well... Of course in the case there should be indexed the public pages or possibly only the closed clearly specifiable community only this will be somehow possible, but in the case there should be some public pages and some visible with some specific permissions only, I have no idea how and if could be this solved in some smooth way...
thumbnail
zaki benz, modificado hace 15 años.

RE: Clustering ehcache on 4.3.5

New Member Mensajes: 15 Fecha de incorporación: 21/07/08 Mensajes recientes
HI all,

net.sf.ehcache.configurationResourceName=/ehcache/[b]hibernate-clustered.xml[/b]
ehcache.multi.vm.config.location=/ehcache/[b]liferay-multi-vm-clustered.xml[/b]


I have looked into the two files you mentioned above :
I've found that there are two caches on two different multicast addresses.
Can someone explaine that for me, why there is two caches ?

thanx.
thumbnail
Artur Linhart, modificado hace 15 años.

RE: Clustering ehcache on 4.3.5

Expert Mensajes: 459 Fecha de incorporación: 13/09/07 Mensajes recientes
Hello,

I think, the reason is:
1. hibernate uses in the Liferay configuration ehcache for the 2nd level caching - so this is the first multicast address
2. Liferay itself uses ehcache for the synchronization of the other cached data (not hibernate based) like layout caching, etc - so, this would be the second multicast address.

If there would be only one address, thenn the hibernate data would be probably mixed twith the other, what could cause problems - I could imagine, there could be the problem with the uniqueness of the identifications of the resources, etc...

It is also a question if there is a way how to come to the references to the ehcache-objects hidden in hibernate if the same ehcache "machine" should be used for other liferay data - in Liferay, as far as I think, actually two ehcache "machines" are instantiated, so they have different ports to communicate.

I did not looked into the source codes, but it could be the logical explanation for the usage of two independent communication pipes.

It is also true, If You would like to have two different Liferay clusters on one IP network segment, then You have to use 4 different ports for the communication - 2 in the first cluster 2 in the second cluster - otherwise the caches got confused by the dynchronization of the data from the other cluster. I tried it :-))) - if was pretty difficult to come to the conclusion, why the cache synchronization does not work - but if You understand more from it, then it is obvious.

with regards, Archie
thumbnail
zaki benz, modificado hace 15 años.

RE: Clustering ehcache on 4.3.5

New Member Mensajes: 15 Fecha de incorporación: 21/07/08 Mensajes recientes
Hi,

Thank you for your reply but I still have an other question
In your post above you didn't mention the configuration of jgroups and when I looked in portal.properties I've found this :


##
## Communication Link
##

#
# Set the JGroups properties used by the portal to communicate with other
# instances of the portal. This is only needed if the portal is running in a
# clustered environment. The JGroups settings provides a mechanism for the
# portal to broadcast messages to the other instances of the portal. The
# specified multicast address should be unique for internal portal messaging
# only. You will still need to set the Hibernate and Ehcache settings for
# database clustering.


Is it because its not necessary to configure jgoups that you omitted to mention it above or do I have to configure it too?

Regards.
thumbnail
Artur Linhart, modificado hace 15 años.

RE: Clustering ehcache on 4.3.5

Expert Mensajes: 459 Fecha de incorporación: 13/09/07 Mensajes recientes
Hello, Zaki,

Yes, You are right, this parameter should or could be set too. But it works also without it and in the original descriptions of the clustering this point has not been listed.

As says Michael Young in http://www.liferay.com/web/guest/community/forums/-/message_boards/message/257721:

2) JGroups is NOT used anymore. JGroups was used as part of the older OsCache implementation for caching. Ehcache uses its own mechanism and you configure it there.


- so this is also the reason, why it has not to be set up....

But...

I looked a little bit into the source code and found the JGroup is still used in 4.3.5 Version - but it is not importatnt and I do not understand why it is used - is used to reload the themes in the cluster after the new instance was started - look into the LayoutLocalServiceImpl.java, there are following lines in the method importThemes:

		themeLoader.loadThemes();

		CommLink commLink = CommLink.getInstance();

		MethodWrapper methodWrapper = new MethodWrapper(
			ThemeLoaderFactory.class.getName(), "loadThemes");

		commLink.send(methodWrapper);



but the given method ThemeLoaderFactory.loadThemes just only reloads the themes - so it is used for the synchronization in the case the new theme has been imported into the liferay to assure it will be reloaded/present on all cluster nodes... ?

Interesting point, I have to test it. Till now, we made the theme deployment on every cluster node again, what is sufficient for us. After setting this option, maybe the themes will be synchronized correctly within the cluster without necessity to redeploy them manually on every node.

By the way, in 5.1.2 this is also still used - but it has been moved from LayoutLocalServiceImpl into the LayoutImporter class, which performs the LAR import. In 4.3.5 The LAR Import/Export does not work well, so we never used it - and also never hat problems with it :-).

But it f the setting of this option would cause the theme has not to be deployed on every node, this would be a benefit.

With best regards, Archie
thumbnail
zaki benz, modificado hace 15 años.

RE: Clustering ehcache on 4.3.5

New Member Mensajes: 15 Fecha de incorporación: 21/07/08 Mensajes recientes
thanks a lot Archie ;)
thumbnail
Syed Mudassir Ali, modificado hace 13 años.

RE: Clustering ehcache on 4.3.5

New Member Mensajes: 10 Fecha de incorporación: 29/11/10 Mensajes recientes
Hi Guys,
Iam novice to liferay and trying caching(ehcache) of few java(serializable) objects in a multiple Tomcat 6.0.26 server environment(clusttered).
followed all the instructions you have mentioned here.

But i was not able to replicate cache.getting below exception.
I have Completely no idea on what iam doing wrong.

07:37:38,905 WARN [RMIAsynchronousCacheReplicator:328] Unable to send message to remote peer. Message was: RemoteException occurred in server thread; nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.lang.ClassNotFoundException: org.cts.sample.SampleFormBean (no security manager: RMI class loader disabled)
java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.lang.ClassNotFoundException: org.cts.sample.SampleFormBean (no security manager: RMI class loader disabled)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:336)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
at net.sf.ehcache.distribution.RMICachePeer_Stub.send(Unknown Source)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.flushReplicationQueue(RMIAsynchronousCacheReplicator.java:317)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.replicationThreadMain(RMIAsynchronousCacheReplicator.java:119)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.access$100(RMIAsynchronousCacheReplicator.java:56)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator$ReplicationThread.run(RMIAsynchronousCacheReplicator.java:376)
Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.lang.ClassNotFoundException: org.cts.sample.SampleFormBean (no security manager: RMI class loader disabled)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:296)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.ClassNotFoundException: org.cts.sample.SampleFormBean (no security manager: RMI class loader disabled)
at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:375)
at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:165)
at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:620)
at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:247)
at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:197)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:480)
at net.sf.ehcache.Element.readObject(Element.java:796)
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at net.sf.ehcache.distribution.EventMessage.readObject(EventMessage.java:139)
at sun.reflect.GeneratedMethodAccessor160.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at java.util.ArrayList.readObject(ArrayList.java:593)
at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:306)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:290)
... 9 more


any kind of help on the issue would be of great use for me.

Thanks in Advance.
Syed Mudassir Ali.
thumbnail
sushil patidar, modificado hace 10 años.

RE: (Resolved) Clustering ehcache on 4.3.5

Expert Mensajes: 467 Fecha de incorporación: 31/10/11 Mensajes recientes
Hi Artur ,
Do you have any idea how to make sure using JMX console that distributed caching is working.?
Any help regarding this will be appreciated.

Regards