High availability in Liferay

[Maciej Posluszny, modificado hace 8 años.]

Hello,

I'm currently using Liferay CE 6.2 + Tomcat 7. I successfully managed to get 2 Liferay instances to work in a cluster. My load balancing solution is Apache's mod_jk with sticky sessions enabled. Everything works just fine, but what I want to do (or get to know if it's worth it) is to replicate session so the following scenario would occur:

A portal user is filling the form in a JSF portlet on node 'A'. Let's say it's a bunch of personal info and it takes 3 steps to submit it. User just finished filling out info in step 2 and node 'A' goes down. He's been redirected to node 'B' with his session. What I want to achieve is to restore the form that he's been working on.

Is there any way to achieve it? What is the cost of such solution?

Thanks in advance!

[David H Nebinger, modificado hace 8 años.]

Don't use server side sessions for managing jsf state. It's too much data to replicate in a session cluster anyway.

Use your client side state management. That way, regardless of what happens on the server side, the JSF app will continue to work just fine.

[Juan Gonzalez, modificado hace 8 años.]

Hi Maciej,

What you really want is HA (High Availability) which can be achieved by enabling session replication. This feature depends heavily on each server implementation (Tomcat in this case), and depending on which one is used it will work fine, perform well or just the opposite.

Here you have more info for Tomcat:

https://tomcat.apache.org/tomcat-7.0-doc/cluster-howto.html

David H Nebinger:

Don't use server side sessions for managing jsf state. It's too much data to replicate in a session cluster anyway.

As always, "it depends", I've been working with JSF view state since JSF 1.2 with session replication enabled and everything worked like a charm. It just depends on how are you implementing those views (and JSF 2 improved a lot in this aspect).

David H Nebinger:

Use your client side state management. That way, regardless of what happens on the server side, the JSF app will continue to work just fine.

It won't work if having @SessionScoped beans for example, and these are a lot of cases.

[David H Nebinger, modificado hace 8 años.]

Ah, well, I've never been a fan of session storage period, regardless of portlet implementation technology

[Juan Gonzalez, modificado hace 8 años.]

David H Nebinger:

Ah, well, I've never been a fan of session storage period, regardless of portlet implementation technology

Yep, the trick here is to store as few information as possible. Some developers thinks session is like a personal HDD and is free...

It sounds equivalent when Python people blame on Java memory usage. The culprit usually relies on source code that hadn't been optimized enough to make good usage of available heap.

[David H Nebinger, modificado hace 8 años.]

And in general if you don't plan on doing said optimizations, you'll always be better off not using session storage all, especially for replication across a cluster.

[Maciej Posluszny, modificado hace 8 años.]

David and Juan,

Thank you for these tips, I really appreciate it.

Juan Gonzalez:

What you really want is HA (High Availability) which can be achieved by enabling session replication.

Actually I have my Tomcats configured just as in the tutorial you have just linked. After seeing Tomcat logs I can tell that session replication via DeltaManager is working. After one node dies, connection is being passed to second node, user is still signed in. But data in portlets as I described in example above are lost. I don't know if it's a problem with session replication, thinking it's working just fine. So once again is there a way to store such data on server side with session replication in Tomcat?

David H Nebinger:

Use your client side state management. That way, regardless of what happens on the server side, the JSF app will continue to work just fine.

I'll definetely try it, thanks for that

One more question. Would session replication work for @SessionScoped beans within portlets?

Again, thank you for your concern, greatly appreciated

[Juan Gonzalez, modificado hace 8 años.]

Hi Maciej,

Maciej Posluszny:

But data in portlets as I described in example above are lost. I don't know if it's a problem with session replication, thinking it's working just fine. So once again is there a way to store such data on server side with session replication in Tomcat?

Do you have some simple portlet example for us to test?

Maciej Posluszny:

One more question. Would session replication work for @SessionScoped beans within portlets?

Yes, it should work.

[Maciej Posluszny, modificado hace 8 años.]

I'm currently working on my main goal which is replicating @SessionScoped bean among all nodes (since my manager is DeltaManager).
I made a simple portlet which uses @SessionScoped backing bean. It has a list of String elements which are inserted by the user. I'm trying to make other nodes receive that bean with new values.

Juan Gonzalez:

Do you have some simple portlet example for us to test?

Sure, I'll gladly share it

Maybe the problem is that I should include some kind of method that updates the session attributes and sends it among nodes?

I really appreciate your help, means a lot ,
Maciej

[Juan Gonzalez, modificado hace 8 años.]

Hi again Maciej,

thanks for the sample portlet.

Maciej Posluszny:

Maybe the problem is that I should include some kind of method that updates the session attributes and sends it among nodes?

Nope, this should happen automatically.

Let me some days to check this. Will come back to you.

Thanks!

[Maciej Posluszny, modificado hace 8 años.]

Thank you Juan, looking forward to hearing for you

[Andrew Jardine, modificado hace 8 años.]

Hey Guys,

I had to do this pretty recently with Tomcat 7.0.42 and the session replication was not working for me as expected. I had all the usual animals in place -- tomcat recommended settings, network settings (multicast), <distributable/> tag etc but it just wouldn't take. I started digging into Tomcat and I found that, in my case, there seemed to be a problem with how tomcat was working. I opened a ticket with Tomcat and was told I was wrong -- lol -- despite the fact that I had evidence of it NOT working. All my settings (confirmed by the guy at Tomcat) were correct and he said it "works for me". The only difference I can think of might have been that he was probably not testing with a portal. At any rate, I did manage to get it working and you can see what I had to do on this thread:

https://www.liferay.com/community/forums/-/message_boards/message/50125775

For the me issue was less about getting the nodes to replicate the session though and more about how it behaved when a node in the cluster dissapeared. The link might still be useful though as it does contain the settings that I used to have replication working in the first place. I do see the occasional failed message, but in a two node cluster with A LOT of session usage (very close to the "personal HDD" comment above) it seems to be working properly.

[Juan Gonzalez, modificado hace 8 años.]

hi Andrew,

Andrew Jardine:

you can see what I had to do on this thread: https://www.liferay.com/community/forums/-/message_boards/message/50125775?_19_redirect=https%3A%2F%2Fwww.liferay.com%2Fcommunity%2Fforums%2F-%2Fmessage_boards%2Fsearch%3F_19_keywords%3Dsession%2Breplication%26_19_searchCategoryId%3D0%26_19_breadcrumbsCategoryId%3D0%26_19_redirect%3Dhttps%253A%252F%252Fwww.liferay.com%252Fcommunity%252Fforums%252F-%252Fmessage_boards%252Fsearch%253F_19_advancedSearch%253Dfalse%2526_19_keywords%253Dcluster%2526_19_threadId%253D0%2526_19_breadcrumbsMessageId%253D0%2526_19_searchCategoryId%253D0%2526_19_delta%253D20%2526_19_resetCur%253Dfalse%2526_19_breadcrumbsCategoryId%253D0%2526_19_cur%253D2%2526_19_redirect%253Dhttps%25253A%25252F%25252Fwww.liferay.com%25252Fcommunity%25252Fforums%25252F-%25252Fmessage_boards%25252Fmy-posts%2526_19_andOperator%253Dtrue%26_19_formDate%3D1442235837024

Thanks for that info, can you please update the link as it points to this thread?

[Andrew Jardine, modificado hace 8 años.]

Ooops -- sorry about that. I had to search for it to find that post and apparently provinding a link that includes the full search result URL was causing issues. Anyway, updated and tested. Should be good now.

[Juan Gonzalez, modificado hace 8 años.]

Maciej Posluszny:

I'm currently working on my main goal which is replicating @SessionScoped bean among all nodes (since my manager is DeltaManager).
I made a simple portlet which uses @SessionScoped backing bean. It has a list of String elements which are inserted by the user. I'm trying to make other nodes receive that bean with new values.

Juan Gonzalez:

Do you have some simple portlet example for us to test?

Sure, I'll gladly share it

Maybe the problem is that I should include some kind of method that updates the session attributes and sends it among nodes?

I really appreciate your help, means a lot ,
Maciej

Hi Maciej,

coming back to the actual thread issue , I will check that next week, we've opened this issue a long time ago for checking this :

https://issues.liferay.com/browse/FACES-1687

[Olaf Kock, modificado hace 8 años.]

Sorry for stepping in - I want to be nitpicking and add some vaguely related aspect, so that others that come across this thread later will also see a different opinion: There is an argument for session replication in HA scenarios. However, if you'd configure a cluster in order to handle more load, you shouldn't consider session replication as it imposes a significant overhead per request.

The typical advice is to get along with client-side state and optionally an SSO system that makes sure you don't have to log in again should one machine go down. Session Replication is a quick fix, but typically for a problem that you think you want to solve, not for the problem that you actually need to solve.

I don't want to go so far as to analyze if you're one of the 5% of people where I'd say session replication is the way to go - just leave a trail for everybody else to pay attention: For 90-95% of usecases I believe that session replication does not solve the underlying problem, even though it looks like it does on first sight.

You have been warned. And I'm quiet again now.

[Andrew Jardine, modificado hace 8 años.]

I'm with Olaf and Dave on this one. I always recommend avoiding the session as much as possible. I think the issue is that "using the session" is both an "old way" of doing things and too convinient for developers to resist. There was a time when it was one of, it not the only, options, but these days there are so many different ways to do the same thing. If you are building a modern web application then the options are seemingly limitless.

There are obviously a lot of different scenarios that need to be considered, but I think Olaf has it bang on. 5% of those cases actually warrant session replciations and really the goal should be to try to completely decouple your application from the session at the outset if you can. Using a database or a cache, or better still the HTML5 storage mechanism, to store details that a user needs across requests I think is a much better solution that relying on a big fat session that needs to be copied all over the place. I have turn the logging levels up in Liferay with session replication enabled and on a relatively light page (say 3 - 4 portlets) the log looks like the matrix. There is A LOT of chatter. The statistics that are output at the end of the day for an almost silent sever (lets say 20 users / hour) were showing upwards of 3GB in data transfer -- just in message replication. Imagine what it would be like to have 4,5 or 10 servers with even a moderate load of 5000 users per day.

Olaf's advice is sound and I would echo it. Avoid session usage and session replication at all costs if you can.

[Juan Gonzalez, modificado hace 8 años.]

Andrew Jardine:

I'm with Olaf and Dave on this one. I always recommend avoiding the session as much as possible. I think the issue is that "using the session" is both an "old way" of doing things and too convinient for developers to resist. There was a time when it was one of, it not the only, options, but these days there are so many different ways to do the same thing. If you are building a modern web application then the options are seemingly limitless.

Hmmm this sounds more like a "hype" instead of an empirical statement, let's talk below.

Andrew Jardine:

Using a database or a cache, or better still the HTML5 storage mechanism

Database? Seems another complexity layer that should be implemented/tested properly. HttpSession is working since more than 13 years now, and I thing that implementation will work much better that any cool solution you can implement in 2 weeks.
Cache? I have the same opinion as mentioned above. Why reinvent the wheel when it was inventend and properly tested since thousands of years?
HTML5? Hmmm many security issues there (https://www.owasp.org/index.php/HTML5_Security_Cheat_Sheet#Local_Storage).

Andrew Jardine:

I think is a much better solution that relying on a big fat session that needs to be copied all over the place.

THIS is the actual issue. Big fat sessions shouldn't be stored in user session. For other storage there are other possibilities for that purpose, like the ones you mentioned before.

Andrew Jardine:

I have turn the logging levels up in Liferay with session replication enabled and on a relatively light page (say 3 - 4 portlets) the log looks like the matrix. There is A LOT of chatter. The statistics that are output at the end of the day for an almost silent sever (lets say 20 users / hour) were showing upwards of 3GB in data transfer -- just in message replication. Imagine what it would be like to have 4,5 or 10 servers with even a moderate load of 5000 users per day.

This probably would depend in some "not optimized" session replication implementation, or a bad webapp implementation (if for each request a session attribute has to be changed without really doing anything maybe there is something wrong there). But again, HttpSession isn't guilty here IMHO.

Andrew Jardine:

Olaf's advice is sound and I would echo it. Avoid session usage and session replication at all costs if you can.

Don't agree at all. HttpSession is a great standard (yes, standard means you don't have to reinvent the wheel for each webapp server you would like to use) mechanism for sharing serializable data. If you don't have this, you won't have high availability, unless you spend weeks and months to implement it right.

Just my thoughts...

[Olaf Kock, modificado hace 8 años.]

Juan Gonzalez:

Andrew Jardine:

Olaf's advice is sound and I would echo it.....

Don't agree at all....

WHAT?

I love to put words in other people's mouth

Not to start a big argument here - just this one thing: I frequently see session replication being automatically configured where it shouldn't be - e.g. where the overhead of storing/replicating heavy session information draws on performance. When the cluster is not built for HA but for being able to cover a load that a single machine can't cover any more, then it's better to get along without it.

Think about the price that you pay and if it's worth the gain: How often does one machine shut down without upfront warning? This is what you're protecting against. If there's a scheduled maintenance window you typically take a machine out from the loadbalancer and just don't have any more new sessions. As soon as everybody on that machine is done you can take down the machine.

[Juan Gonzalez, modificado hace 8 años.]

Olaf Kock:

Not to start a big argument here - just this one thing: I frequently see session replication being automatically configured where it shouldn't be - e.g. where the overhead of storing/replicating heavy session information draws on performance. When the cluster is not built for HA but for being able to cover a load that a single machine can't cover any more, then it's better to get along without it.

I agree, as always "it depends" on the environment. For example, I can tell you how doctors from a 4000+ employees hospital will feel if a server goes down when they are doing their reviews each morning (and they have one patient each 15 min by average during 7 hours each day). They can't lose the content they are writing or even wait for a SSO CAS to log in automatically into other server. That's a crisis in the real world.

Olaf Kock:

Think about the price that you pay and if it's worth the gain: How often does one machine shut down without upfront warning? This is what you're protecting against. If there's a scheduled maintenance window you typically take a machine out from the loadbalancer and just don't have any more new sessions. As soon as everybody on that machine is done you can take down the machine.

It can happen, for sure. If you have a simple "read-only" site, then it doesnt matter if users are redirected to one server or another. If it's an application with intensive form fields editing (more updates than reads), then loosing the work being made even in 5 minutes can be very bad sometimes (and saving many drafts automatically sometimes is much worse talking about performance and HDD space saving).

[Olaf Kock, modificado hace 8 años.]

Let's settle this for the "it depends" and "don't use session replication as the default configuration without intensely thinking about it". I also agree that there are reasons to configure it.

A setup with 4000 users (probably not even concurrent) hopefully doesn't need a setup that *requires* two servers just to keep up operations under load - but purely for HA. So we're not even contradicting each other.

[David H Nebinger, modificado hace 8 años.]

To me session usage has a similar code smell as global variables.

Sure you can do it, but that doesn't make it right.

Any implementation you can build using session variables you can also build as stateless and not use session variables.

In one case you're worried about session replication, size of session objects and ability of cluster to sync session state changes in time before crashes.

In the other you're cluster node neutral but pay a penalty to fetch or rebuild an object (or graph) that you might had stored in the session.

We should never take a stance that "one size fits all", each architectural and design choice should be based on realistic assessment of the environment, the requirements, the experience of the team, room in the plan for tuning and optimization and prioritization of specification details.

There are cases where session usage may make sense, but session usage should not be left to the whim of the developer that wants to choose a shortcut, it should be a known characteristic of the design.

Unless there's a specific reason to use sessions, in general I think it's better not to.

[Andrew Jardine, modificado hace 8 años.]

There is one very important factor that has not been mentioned on this thread -- and while it may seem like its is an irrelevant factor, if you work in services, it most certainly is not.

... Because the client asked/told me to.

[David H Nebinger, modificado hace 8 años.]

If you work in services, though, you should have been brought in as a domain expert so you should be able to guide them to the right decision.

Explaining the issues as highlighted in this thread should demonstrate to any client the perils of using session storage just because they want to...

[Andrew Jardine, modificado hace 8 años.]

Come on David. Surely you have been playing this game long enough to know that expertise often falls on deaf ears. All the items listed in this thread are definitely part of the conversation, but if the person making the decision doesn't totally understand it all, they will go with their own experience (which may or may not be valid) or with a decision based on how they interpret these details.

I think the overall message is valid. There are multiple ways to solve these problems and decisions are probaby made based on a number of factors from situation to situation. With all that said, everyone goes into the conversation with a preferred approach. it sounds to me like your preferred aproach is the same as mine -- where possible, keep the application as (session) stateless as you can.

[James Falkner, modificado hace 8 años.]

Thread of the month!

-James "Just trying to get my post count up" Falkner

[David H Nebinger, modificado hace 8 años.]

James Falkner:

-James "Just trying to get my post count up" Falkner

Can't catch me...

- Dave "On his way to 10k" Nebinger

[David H Nebinger, modificado hace 8 años.]

Andrew Jardine:

Come on David. Surely you have been playing this game long enough to know that expertise often falls on deaf ears. All the items listed in this thread are definitely part of the conversation, but if the person making the decision doesn't totally understand it all, they will go with their own experience (which may or may not be valid) or with a decision based on how they interpret these details.

Yep, we can lead the horse to water but you can't make him take a bath.

[Andrew Jardine, modificado hace 8 años.]

Hahaha -- I use that expression religiously. Though the version I am familiar with is "You can lead a horse to water, but you can't make him drink."

... now we're REALLY off topic, which is REALLY bad for a couple of members with "Community Moderator" in their profile ;)

[Juan Gonzalez, modificado hace 8 años.]

Olaf Kock:

Let's settle this for the "it depends" and "don't use session replication as the default configuration without intensely thinking about it". I also agree that there are reasons to configure it.

Probably this is the best summary of this thread. The general rule is "don't add any complexity to your app without knowing what you're doing". At the same time, "don't avoid any feature just because someone doesn't like it, investigate by yourself" .

[Maciej Posluszny, modificado hace 8 años.]

That's the best conclusion in my opinion as well

I want to respond to everyone who joined this thread with a big thanks. That's a lot of knowledge and clues to me and people who come across this thread.
I'm still doing some research on what you guys suggested.
I really do appreciate your support, thanks a ton

Maciej