« Back

Solving Performance Issue in Liferay 6.1 Staging

General Blogs May 30, 2013 By Kan Zhang

Issue

In Liferay 6.1, the staging site is extremely slow if versioning is enabled. It can take 30 seconds to 10 minutes to refresh a page on the staging site (depends on how many contents/portlets on your page).

Cause

If you enable hibernate show SQL by setting the log level to DEBUG for "org.hibernate.SQL" category in "Server Administration" or use JAVA profiler, you will find the following SQL is executed hundreds of times which slows down the staging site a lot:

The AOP class com.liferay.portal.service.impl.LayoutSetLocalServiceStagingAdvice has a pointcut on LayoutSetLocalService. The LayoutSetLocalService can be called hundreds of times in one page refresh. Each time the LayoutSetLocalService is getting called, the LayoutSetLocalServiceStagingAdvice.wrapLayoutSet() will be invoked.

If we look at the code: com.liferay.portal.service.impl.LayoutSetLocalServiceStagingAdvice.wrapLayoutSet()

It will first calls LayoutSetStagingHandler layoutSetStagingHandler = LayoutStagingUtil.getLayoutSetStagingHandler(layoutSet);

Check the code com.liferay.portal.staging.LayoutStagingImpl.getLayoutSetStagingHandler(LayoutSet layoutSet):

The above method will always return null because layoutSet is not a proxy class.

It will return null back to the LayoutSetLocalServiceStagingAdvice.wrapLayoutSet() method. Then the (LayoutSet)ProxyUtil.newProxyInstance( PACLClassLoaderUtil.getPortalClassLoader(), new Class[] {LayoutSet.class}, new LayoutSetStagingHandler(layoutSet)) will be to create a new proxy class to warp the LayoutSet.

Here comes important part: When creating a new LayoutSetStagingHandler(layoutSet), it will call the com.liferay.portal.service.persistence.LayoutSetBranchFinderImpl.findByMaster(long groupId, boolean privateLayout) method. But the LayoutSetBranchFinderImpl finder method does not have the Liferay finder cache enabled, so Liferay will query the database every time it is getting called.

So, each time LayoutSetLocalService is getting called, there will be at least one database query to the LayoutSetBranch table, which makes the staging site very slow.

Solution

The best solution in my mind is to change the way how Liferay handles the branching/versioning in Liferay staging or to reduce the call to LayoutSetLocalService class. But it requires a lot of work. I am still working on it and will send a pull request if I can finish it.

Another solution is to use ext plugin to overwrite the LayoutSetBranchFinderImpl class to add the Liferay finder cache and overwrite LayoutSetBranchPersistenceImpl class to clear the Liferay finder cache upon create/update/delete.

If you do not want to use ext environment, here is a workaround using hook plugin:

Override the com.liferay.portal.service.LayoutSetBranchLocalService in the hook plugin:

Implement the com.kzhang.liferay.portal.service.LayoutSetBranchLocalServiceWrapper:

The main idea is to leverage getLayoutSetBranches(groupId, privateLayout) method to get the master LayoutSetBranch instead of using the finder method, because the getLayoutSetBranches() method has Liferay entity cache enabled. But as you can see, it has a silly for loop in the code which is not cool.

After deployed the hook, the page loading time is reduced from 3 minutes to 2 seconds.

Or, what you can do is: you can use getLayoutSetBranch(groupId, privateLayout, "main-variation") instead of getLayoutSetBranches(groupId, privateLayout) to get better performance if you never change the name of the master branch (the default name of the master branch is "main-variation").

Threaded Replies Author Date
Very nice work Kan!!! How close are you to... Ray Augé May 31, 2013 11:47 AM
.. oh, and is there a ticket created for the... Ray Augé May 31, 2013 11:51 AM
See http://issues.liferay.com/browse/LPS-35856 ... Ray Augé May 31, 2013 2:17 PM
Cool :) The pull request is to add the cache... Kan Zhang June 3, 2013 3:43 PM
Hi Kan, Do you think same issue exist for... Mahesh Panchal June 1, 2013 11:03 AM
Hi Mahesh, I double checked the source code... Kan Zhang June 4, 2013 8:30 AM
Hi Kan, Thanks for your explanation. Slowness... Mahesh Panchal June 3, 2013 3:54 PM
Hi Mahesh, 10000 is max no of entries should... Kan Zhang June 4, 2013 8:29 AM
Is this issue solved in EE? This is a no-go for... Jens Meinecke June 5, 2013 5:52 AM
It would be really nice if the outstanding... Ray Augé June 5, 2013 6:29 AM
Hi Ray, It seems that LS has created ticket for... Mahesh Panchal June 7, 2013 6:09 PM
Hi Mahesh, I want to know if you were able to... Matías Javier Elola July 23, 2013 9:02 AM
Hi Matias, We received patch from support for... Mahesh Panchal July 30, 2013 8:38 AM
Thanks Mahesh for your response. My co-worker... Matías Javier Elola July 30, 2013 9:26 AM
Kan, thank you for this post and all your other... gordon daniels August 19, 2013 1:53 PM

Very nice work Kan!!!

How close are you to implementing a full solution? Can you send me the pull request? (https://github.com/rotty3000)
Posted on 5/31/13 11:47 AM.
.. oh, and is there a ticket created for the issue? If not, can you create one?
Posted on 5/31/13 11:51 AM.
See http://issues.liferay.com/browse/LPS-35856

I referenced your blog post emoticon

See https://github.com/brianchandotcom/liferay-portal/pull/11487
Posted on 5/31/13 2:17 PM.
Hi Kan,
Do you think same issue exist for com.liferay.portal.service.LayoutLocalService ?

We have 2000 private pages and clicking Private pages tab in control panel or clicking select pages on publish window invokes following query 2000 times.

select layoutimpl0_.plid as plid14_, layoutimpl0_.uuid_ as uuid2_14_, layoutimpl0_.groupId as groupId14_, layoutimpl0_.companyId as companyId14_, layoutimpl0_.createDate as createDate14_, layoutimpl0_.modifiedDate as modified6_14_, layoutimpl0_.privateLayout as privateL7_14_, layoutimpl0_.layoutId as layoutId14_, layoutimpl0_.parentLayoutId as parentLa9_14_, layoutimpl0_.name as name14_, layoutimpl0_.title as title14_, layoutimpl0_.description as descrip12_14_, layoutimpl0_.keywords as keywords14_, layoutimpl0_.robots as robots14_, layoutimpl0_.type_ as type15_14_, layoutimpl0_.typeSettings as typeSet16_14_, layoutimpl0_.hidden_ as hidden17_14_, layoutimpl0_.friendlyURL as friendl18_14_, layoutimpl0_.iconImage as iconImage14_, layoutimpl0_.iconImageId as iconIma20_14_, layoutimpl0_.themeId as themeId14_, layoutimpl0_.colorSchemeId as colorSc22_14_, layoutimpl0_.wapThemeId as wapThemeId14_, layoutimpl0_.wapColorSchemeId as wapColo24_14_, layoutimpl0_.css as css14_, layoutimpl0_.priority as priority14_, layoutimpl0_.layoutPrototypeUuid as layoutP27_14_, layoutimpl0_.layoutPrototypeLinkEnabled as layoutP28_14_, layoutimpl0_.sourcePrototypeLayoutUuid as sourceP29_14_ from Layout layoutimpl0_ where (layoutimpl0_.groupId=? )AND(layoutimpl0_.privateLayout=? )AND(layoutimpl0_.parentLayoutId=? ) order by layoutimpl0_.parentLayoutId ASC , layoutimpl0_.priority ASC

Not sure why service class is not retrieving all layouts in one query and firing query for each ;ayout.
Posted on 6/1/13 11:03 AM.
Cool emoticon
The pull request is to add the cache layer which works well.
I was also working on to reduce the following call
--------
(LayoutSet)ProxyUtil.newProxyInstance(
PACLClassLoaderUtil.getPortalC­lassLoader(),
new Class[] {LayoutSet.class},
new LayoutSetStagingHandler(layoutSet));
--------
in LayoutSetLocalServiceStagingAdvice.wrapLayoutSet() because I don't like the fact the LayoutSetStagingHandler is being newed hundreds to thousands times in one single request.

The LayoutStagingUtil.getLayoutSetStagingHandler(layoutSet) acts as a cache layer which works well when updating the layout because the layoutSet is already wrapped. But it does not work when reading the layoutSet from database.

I was thinking to cache the LayoutSetStagingHandler to prevent it to be newed again and again. But I think it is not critical right now.

Same thing happens in LayoutLocalServiceStagingAdvice for layouts too.

One more thing, the LayoutSetLocalService.createLayoutSet(long layoutSetId) and the LayoutLocalService.createLayout(long plid) does not work because LayoutSetLocalServiceStagingAdvice/LayoutLocalServiceStagingAdvice has pointcut on it -- it tries to fetch the group object from the newly created Layout/LayoutSet (with groupId=0) which throws the NoSuchGroupException.
Posted on 6/3/13 3:43 PM in reply to Ray Augé.
Hi Mahesh,

I double checked the source code and the layout has the cache enabled.

Regarding to your issues, I think there are two possible causes:
1. The max number of elements in cache is set too low in in your liferay ehcache xml. (By default the max number of elements in cache is 10000 for each entity ).
2. The cache is cleared upon the layout(page) update. If you changed anything on the page, the cache will be cleared. Liferay will invokes the query first time when you go to the "Manage pages". But why it has so many queries (2000 time in your case)? Here is the reason:
The "Manage pages" displays a tree view for the dropdown box for "copy to page" function and "link to page" page type. To generate the tree view, Liferay calls com.liferay.portal.util.LayoutLister.getLayoutView() to create the tree.
In LayoutLister.getLayoutView(), it has a recursive method called _createList(long parentLayoutId, int parentId, int depth) which iterates thru the each of the layout to get the child layouts of each layout. If you have 2000 pages, it will call _createList_createList() 2000 times and you will have 2000 queries in the database.
I am working to improve the above logic too, but recently I was overwhelmed by my work. Trying hard to find time...
Posted on 6/4/13 8:30 AM in reply to Mahesh Panchal.
Hi Kan,
Thanks for your explanation.
Slowness of Manage Page seems to be due to LayoutLister.getLayoutView() and now we noticed that even clicking page from site navigation is causing 2000 DB calls. These calls seems to be for LayoutRevision and noticed that multiple calls are made for same layoutId.
It looks like that LayoutRevision also need to be revisited emoticon

Regarding ehcache point, I see that 10000 is max no of entries then 2000 layout should be OK, isn't it ?
Posted on 6/3/13 3:54 PM in reply to Kan Zhang.
Hi Mahesh,

10000 is max no of entries should be OK for 2000 layouts. So it is not the cache issue.

Regarding the LayoutRevision concerns, it is caused by LayoutLocalServiceStagingAdvice.wrapReturnValue() - same reason as I mentioned in the blog post - but is for LayoutRevision insetad of LayoutSetBranch. But the LayoutRevision has the finder cache enabled, it only happens first time after you update a layout.
Posted on 6/4/13 8:29 AM in reply to Mahesh Panchal.
Is this issue solved in EE? This is a no-go for our production environment.
Posted on 6/5/13 5:52 AM.
It would be really nice if the outstanding issues had tickets created with all these details attached and we can make sure that they get fixed and patches are created for EE.

@Kan, Mahnesh, can you guys do that, then link them here so we can work on them please?
Posted on 6/5/13 6:29 AM in reply to Jens Meinecke.
Hi Ray,
It seems that LS has created ticket for this issue - http://issues.liferay.com/browse/LPS-36049.
Posted on 6/7/13 6:09 PM in reply to Ray Augé.
Hi Mahesh,
I want to know if you were able to solve the "Manage page" slowness on 6.1.1 CE GA2
I'm working with almost 3000 public pages and, as you know, it turns very very slow when it comes to load/update layouts in the manage page section.
I've read the ticket (LPS-36049) and it says that the problem is solved in 6.2.0 CE M6 but I'm looking for the solution for the 6.1.1 CE version.
Do you know anything else about this problem and how to solve it in the latest stable community edition?
Thank you very much Mahesh, and of course, thanks a lot to Kan who wrote this blog entry.
Posted on 7/23/13 9:02 AM in reply to Mahesh Panchal.
Hi Matias,
We received patch from support for 6.1.20 EE version.
Posted on 7/30/13 8:38 AM in reply to Matías Elola.
Thanks Mahesh for your response.
My co-worker found the solution last friday while I was out.
For those who want to know, we have replaced the LayoutLister class with an improvement made by Raymond Auge in http://issues.liferay.com/browse/LPS-36049 (the same issue I've mentioned before).
I think the next time I'll have to read more carefully.
Thanks Ray.
Posted on 7/30/13 9:26 AM in reply to Mahesh Panchal.
Kan, thank you for this post and all your other contributions. I really like the bootstrap-theme too. I am learning alot from you.

thanks again
Posted on 8/19/13 1:53 PM.