Jackrabbit w/ Postgres - Scalability Issues

[Dustin Kintanar, modifié il y a 11 années.]

Hello all,

We have a site implemented in Liferay that has a Document Library setup to use Jackrabbit pointed at a Postgres database - all images displayed on the site are stored within the Document Library. Under normal loads, things run smoothly, but under heavy loads, we begin to see failures. While running a load test of 500 concurrent users with a 10 minute ramp-up time, we were receiving a large number of 400 Bad Request errors. The two most common errors repeating in the logs are the following:

[indent]21:42:51,935 ERROR [liferay/buffered_increment_parallel-4][JDBCExceptionReporter:76] FATAL: sorry, too many clients already
21:42:51,938 ERROR [liferay/buffered_increment_parallel-4][BufferedIncrementMessageListener:52] Cannot write buffered increment value to the database
----------------------------------------------------------------------------------------------------------------------------
22:39:23,288 ERROR [http-bio-8080-exec-564][JDBCExceptionReporter:76] [http-bio-8080-exec-564] Timeout: Pool empty. Unable to fetch a connection in 0 seconds, none available[size:100; busy:35; idle:0; lastwait:20].
22:39:25,236 ERROR [http-bio-8080-exec-564][AdvancedPermissionChecker:933] com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.dao.orm.ORMException: org.hibernate.exception.GenericJDBCException: could not execute query
com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.dao.orm.ORMException: org.hibernate.exception.GenericJDBCException: could not execute query[/indent]

Has anyone come across a Jackrabbit setup that is as resilient as Liferay? From what we've seen, Liferay has been able to sustain itself under loads of several thousand concurrent users. Sure, performance slows down to a crawl, but the site does respond and when the load stops, the site's performance bounces back. Jackrabbit seems to be unable to keep up under several hundred concurrent users.

There is some Liferay documentation that mentions Liferay doesn't setup indexes for Jackrabbit, but on a closer inspection it looks like they do exists.

Thanks,
Dustin

[Hitoshi Ozawa, modifié il y a 11 années.]

Comments in the following issue may give you some insight. Better to use DBStore instead of JCRStore.

http://issues.liferay.com/browse/LPS-26332

[David H Nebinger, modifié il y a 11 années.]

Well, liferay 6.1.1 CE GA2 ships with jackrabbit 2.1.6, and according to the info at jackrabbit.apache.org, DB connection pooling was added in jackrabbit 2. Seeing as most of your exceptions seem to be related to db connection issues, I'd start by adding some connection pooling parameters and tune the parameters to accommodate the load you're expecting.

[Dustin Kintanar, modifié il y a 11 années.]

Thanks for the suggestions!

After posting, I actually did go ahead and upgrade the Jackrabbit jars to the 2.6 version that was released recently. After adding a connection pool that each element could refer to and migrating our entire Jackrabbit repository to the new configuration, the setup seems more resilient, but still not quite there. I'm still in the process of working with our DBA to determine what the Max Connections are set at, but I have updated the max pool sizes to values low enough where we no longer get the "sorry, too many clients already", but the "Timeout: Pool empty. Unable to fetch a connection" errors persist.

I will take a look into the DBStore to see if that will improve the situation.

[David H Nebinger, modifié il y a 11 années.]

So the "timeout pool empty" is due to all of the available connections being allocated. This means you want to increase the number of connections in the pool to accommodate the additional load, not decreasing the size.

[Dustin Kintanar, modifié il y a 11 années.]

Right - lowering the pool size was more a side effect of not knowing what the maximum_connections was set to on the Postgres side... I just wanted to make sure I was falling below that limit.

Thanks!
Dustin

[Dustin Kintanar, modifié il y a 11 années.]

Just for anyone who is looking into a similar issue - we were able to gain more stability by upping both the maxThreads on the connector in server.xml and the database pool size. We are still in the process of tweaking the settings to improve load times, but the environment is much more stable. Going through the Deployment checklist also went a long way to stabilizing our environment, the link to the document is below:

http://www.liferay.com/documents/14/8440800/LPEE%206.1%20Deployment%20Checklist/6f17fad4-e93e-4503-a027-eb2327ea818c?elq=

- Dustin

[Patrick Hisshion, modifié il y a 9 années.]

You mentioned you were able to sustain several thousand concurrent users. What setup did you use to do something like that?

[Dustin Kintanar, modifié il y a 9 années.]

Hi Patrick,

It was a while ago, but our general load test consists of 10 machines we run JMeter on that we ramp up anywhere from 50-250 threads. Looking back, saying 1000s is definitely a bit unclear, but in general, what we've found is that under heavy load, Liferay will begin to run slower, but has the ability to recover without the need of a service restart.

As far as achieving the performance on Liferay, it's definitely a case-by-case scenario, but what we've seen the most help is from a combination of JVM and page tuning.

- Dustin