Foren

Liferay 6.0.4 Document Library Slow Performance

thumbnail
Ivan Fong, geändert vor 13 Jahren.

Liferay 6.0.4 Document Library Slow Performance

Junior Member Beiträge: 35 Beitrittsdatum: 11.06.10 Neueste Beiträge
I have installed LR 6.0.4 on our server and found the following strange behavior in Document Library Portlet.

I have imported 1000 folders and 6600 files into the LR through WebDAV yesterday.

I have then re-index all the search index on the server.

It takes about 15 seconds to load when Organization Administrator click the Document Tab to load the Document Library Portlet.

But it takes almost 3 minutes to load when Organization Member do the same action.

During the page load, the mySQL consumes all the CPU power on the server.

Then, I enable the query log on mySQL server to log down all SQL statement and found that the number of SQL statements the Organization Administrator will do is A LOT less than around 30 compare to Organization Member of 1500 SQL statements.

I wonder is there any way to improve this performance issue??

Thank you in advance.

Ivan Fong
thumbnail
Samuel Liu, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Expert Beiträge: 301 Beitrittsdatum: 27.05.10 Neueste Beiträge
Hi Ivan,

This certainly does sound like a curious issue. Do you know how previous Liferay versions compare at all, or are you only using Liferay 6.0.4?

If you need a doc library right now and this bug doesnt seem like it will be fixed soon, may I recommend that Liferay hooks into Documentum, Alfresco, and many other document Libraries emoticon

I'm going to look into this one some more. Thanks for the tip!

Warm regards,
Sam
thumbnail
Ivan Fong, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Junior Member Beiträge: 35 Beitrittsdatum: 11.06.10 Neueste Beiträge
Dear Samuel:

Actually, I have not tried with such a large dataset in LR 5 before.

The truth is the total documents we target to put on the LR server will be around 160GB and more than 20000 files.

I have attached the mysql query log of the Organization Administrator and Member for your kind review.

I guess I will try to use LR5 and see if the same thing happens again before try to use Alfresco. It is because I am worry about whether the search portlet will search the Alfresco documents.

Please kindly update me if there is any new findings.

Thank you for your kind help.

Ivan Fong
thumbnail
Samuel Liu, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Expert Beiträge: 301 Beitrittsdatum: 27.05.10 Neueste Beiträge
Thanks Ivan!

Yeah I'm going to look into this tomorrow when I get into work. It would be terrible if document library wasn't capable of handling a large number of files. I'll load up a fairly large number and see if anything goes wrong. If nothing happens I'll keep adding till I get the same error.

I don't know too much about Alfresco, Documentum or the alternatives either -- I just know that we are built for integration with them. I'll ask about the searching bit and get back to you on that tomorrow.

Thanks for finding these bugs btw, they're really helpful!

Warm regards,
Sam
thumbnail
Ivan Fong, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Junior Member Beiträge: 35 Beitrittsdatum: 11.06.10 Neueste Beiträge
Hi Samuel:

I have tried setup the LR 5.2.3 with similar setting of 6.0.4 including the users, organization, roles...etc.

I have upload similar files to it through WebDAV and their document library folder information are shown below respectively.

LR 5.2.3
Path: /opt/liferay-portal-5.2.3/data/document_library
Size: 6.52 GB
Files: 8926
Folders: 8847

LR 6.0.4
Path: /opt/liferay-portal-6.0.4/data/document_library
Size: 11.98 GB
Files: 7740
Folders: 7724

I have put the document library portlet on an individual page in the Organization's Private page.

I found that time it takes to load the page for both Organization Administrator and Organization User are REALLY fast, it takes less than 3 seconds and 4 seconds for the 5.2.3 to load the portlet respectively.

I have also upload the SQL log for you to do some analyze.
thumbnail
Samuel Liu, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Expert Beiträge: 301 Beitrittsdatum: 27.05.10 Neueste Beiträge
Hi Ivan,

Thank you for this detail. It makes my job so much easier! I'll run similar tests locally and file a bug report to get this fixed ASAP. Seems like a regression bug, and document library bugs are all serious.

Warm regards,
Sam
thumbnail
Ivan Fong, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Junior Member Beiträge: 35 Beitrittsdatum: 11.06.10 Neueste Beiträge
Hi Samuel:

Base on your experience, is it possible for you to give me a time frame on when it will be fix? In terms of weeks or in terms of months does affect how and when we launch our intranet portal.

It is because I need to decide whether we should launch LR 5 or LR 6 to our intranet and update our board of directors next Monday.

Thank you for kind help.

Ivan Fong
thumbnail
Samuel Liu, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Expert Beiträge: 301 Beitrittsdatum: 27.05.10 Neueste Beiträge
Hi Ivan,

Sorry I do not know when this will be fixed. However, I can tell you that I am in the middle of reproducing it right now, so that I can tell people to fix it asap. I'm not sure who will work on it, how long it will take, or which nightly build it will come out in though.

If you launch LR5 on your intranet but decide to upgrade when the bug is fixed though, that might be optimal for you. I am hoping this is something that will come out in a nightly build in a matter of weeks, not months. Document Library is very important to Liferay so I will do my best to get this out the door for you fast.

Sorry for the inconvenience! emoticon

Warm regards,
Sam
thumbnail
Samuel Liu, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Expert Beiträge: 301 Beitrittsdatum: 27.05.10 Neueste Beiträge
Hi Ivan,

Internally the Document Library is being worked on actively as we speak. Turns out our most recent nightly build has a broken webDAV somehow emoticon

I'm going to contact the guy working on it to make him aware of this issue you've discovered.

Hopefully we can have this resolved quickly.

Thanks for your help!
Sam
thumbnail
Samuel Liu, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Expert Beiträge: 301 Beitrittsdatum: 27.05.10 Neueste Beiträge
Hi Ivan,

I might have a temporary workaround for you in 6:

So the developer working on WebDAV told me that the bug seems to have to do with document library's permissioning. Your fix would be to set Liferay to use permission algorithm 5. You can set this in portal-ext.properties {../tomcat-6.0.26/webapps/ROOT/WEB-INF/classes/portal-ext.properties}

permissions.user.check.algorithm=5


Let me know how it goes if you try emoticon

Webdav is working again in the latest nightly build and hopefully 6.0.5 will get released today or within a few days.

Warm regards,
Sam
thumbnail
Ivan Fong, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Junior Member Beiträge: 35 Beitrittsdatum: 11.06.10 Neueste Beiträge
Hi Samuel:

Thank you very much for your replay.

I have tried it and now it takes about 15 seconds to open the document library.

It is a GREAT improvements but I think LR 6 can do a lot better than this when I compare to 5.2.3.

I will upload more files and folder to test its performance tomorrow.

Thank you once again and I will keep you posted.

Ivan Fong
thumbnail
Samuel Liu, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Expert Beiträge: 301 Beitrittsdatum: 27.05.10 Neueste Beiträge
Thanks so much Ivan for your help emoticon Our guys are working on it now, keep us posted!

Warm regards,
Sam
thumbnail
Ivan Fong, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Junior Member Beiträge: 35 Beitrittsdatum: 11.06.10 Neueste Beiträge
Hi Samuel:

I have found that after I implemented the workaround, the load time of the document library portlet is a lot faster but it introduce a new problem as well.

After I use the workaround, I found that all organization member cannot access the folder in document library except the organization administrator.

So I have reverted the workaround now.

For the time being, I think I will propose our team to use 5.2.3 until 6 is more stable and usable in our case.

Thank you for your kind help.

Ivan fong
thumbnail
Samuel Liu, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Expert Beiträge: 301 Beitrittsdatum: 27.05.10 Neueste Beiträge
Hey Ivan,

We will work hard to get 6.0's doclib working right emoticon

Thanks for all your help and support.

Warm regards,
Sam
thumbnail
Ivan Fong, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Junior Member Beiträge: 35 Beitrittsdatum: 11.06.10 Neueste Beiträge
Hi Samuel:

May I know whether the document library is working properly in 6.0.5?

Thank you.

Ivan Fong
Alan Schrock, geändert vor 13 Jahren.

RE: Liferay 6.0.4/6.0.5 Document Library Slow Performance

New Member Beiträge: 9 Beitrittsdatum: 16.09.10 Neueste Beiträge
Are there any updates to this? I have a similar situation from what Ivan is talking about. I have been using Liferay 5.2.3 successfully with around 1300 files, 800 folders, 180 MB. It works fairly quickly in Liferay 5.2.3 but is very slow on initial load for non-admins still on 6.0.5 with the nightly build from September 3rd. Using mysql as well.

We utilize this also fairly extensively and switched over to Liferay 6 already for production.

NOTE: it also doesn't matter whether it is the full document library on the page or the document library display. Any page with a large document library repository/document library display of that repository takes at least 20-40 seconds to initially load.

Please give an update.

Thanks!

Alan
thumbnail
Jack Bakker, geändert vor 13 Jahren.

RE: Liferay 6.0.4/6.0.5 Document Library Slow Performance

Liferay Master Beiträge: 978 Beitrittsdatum: 03.01.10 Neueste Beiträge
webdav is painfully slow in 6.0.5 compared to 5.2.3

I also noticed dlfileversion title is only 75 characters wide ; so many files I try to transfer over to new version don't get there till I widen that field (using postgresql)
thumbnail
Jack Bakker, geändert vor 13 Jahren.

RE: Liferay 6.0.4/6.0.5 Document Library Slow Performance

Liferay Master Beiträge: 978 Beitrittsdatum: 03.01.10 Neueste Beiträge
here is why: http://issues.liferay.com/browse/LPS-12238
Alan Schrock, geändert vor 13 Jahren.

RE: Liferay 6.0.4/6.0.5 Document Library Slow Performance

New Member Beiträge: 9 Beitrittsdatum: 16.09.10 Neueste Beiträge
That would do it!

Also, the nightly build from the 16th of September seems to have sped up the access significantly so they must have made this change.

Not sure that will help your webdav issue, but I also find webdav slow with the image gallery so that may be an overall issue with webdav and liferay still...
thumbnail
Jack Bakker, geändert vor 13 Jahren.

RE: Liferay 6.0.4/6.0.5 Document Library Slow Performance

Liferay Master Beiträge: 978 Beitrittsdatum: 03.01.10 Neueste Beiträge
thanks for the note on the nightly option

my postgresql cpu allocation is hitting the ceiling on webdav imports
Alan Schrock, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 9 Beitrittsdatum: 16.09.10 Neueste Beiträge
What is the current status of the document library for 6.0.x? I have just applied the nightly build from 10/10/10 and though, as my other post reports, the speed of the access is better, it is still really too slow for use with larger numbers of files. My overall set of files is large, but each folder isn't large.

I see that the issue refered to has been closed so I am not sure if there is anyone working on this anymore or not.

So, if a fix isn't coming soon, is there a way to backrev to the 5.2.3 version of the document library? If so, can this be done without destroying my existing documents or will I have to do a full copy in again?

Thanks

Alan
thumbnail
Matti Lindell, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beitrag: 1 Beitrittsdatum: 14.10.10 Neueste Beiträge
I'd also like to know the status of this. Even if the DocumentLibrary UI is fixed, WebDAV suffers from same/similar permission checks which start from DLFolderServiceImpl.getFolders

When the scenario is the following:

-PERMISSIONS_USER_CHECK_ALGORITHM=6
-Large DocumentLibrary content (> 1000 folders and with files)
-WebDAV -> Document library import

importing a file to DL, inline permission checks create very massive DB queries (posgresql) and cause big performance hit.

Can someone verify is this is caused by the new permission check algorithm?
Has the new algorithm been extensively tested with large (as in production) contens?


I took a thread dump when the server was under heavy load and the culprit is the permission checking done in DLFolderPersistenceImpl.filterFindByG_P:

java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135)
org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104)
org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
org.postgresql.core.PGStream.ReceiveChar(PGStream.java:259)
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1620)
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367)
org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:271)
com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
org.hibernate.loader.Loader.getResultSet(Loader.java:1849)
org.hibernate.loader.Loader.doQuery(Loader.java:718)
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:270)
org.hibernate.loader.Loader.doList(Loader.java:2294)
org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2172)
org.hibernate.loader.Loader.list(Loader.java:2167)
org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:316)
org.hibernate.impl.SessionImpl.listCustomQuery(SessionImpl.java:1832)
org.hibernate.impl.AbstractSessionImpl.list(AbstractSessionImpl.java:165)
org.hibernate.impl.SQLQueryImpl.list(SQLQueryImpl.java:179)
com.liferay.portal.dao.orm.hibernate.SQLQueryImpl.list(SQLQueryImpl.java:86)
com.liferay.portal.kernel.dao.orm.QueryUtil.list(QueryUtil.java:60)
com.liferay.portal.kernel.dao.orm.QueryUtil.list(QueryUtil.java:52)
com.liferay.portlet.documentlibrary.service.persistence.DLFolderPersistenceImpl.filterFindByG_P(DLFolderPersistenceImpl.java:2317)
com.liferay.portlet.documentlibrary.service.persistence.DLFolderPersistenceImpl.filterFindByG_P(DLFolderPersistenceImpl.java:2227)
com.liferay.portlet.documentlibrary.service.impl.DLFolderServiceImpl.getFolders(DLFolderServiceImpl.java:208)
thumbnail
Ray Augé, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Liferay Legend Beiträge: 1197 Beitrittsdatum: 08.02.05 Neueste Beiträge
You guys feel like testing the following patch?

It's against trunk, but it shouldn't be hard to adapt for any version (at least for 6.0+).

Basically this uses the index for listing files instead the DB.

Note that I do think that while trying this I think I uncovered a bug with the re-indexing logic. When you use this, the top level files disappear after a re-index. But, adding more files or re-adding any missing files results in showing all the files again. Might be something with a cache somewhere even.

I tried solving the slowness of the import, but I haven't spotted specific issue yet. I did notice a small speedup when using the NIO file transfer setting.

[tt]-Dcom.liferay.portal.kernel.util.StreamUtil.use.nio=true ## default=false[/tt]

There is also

[tt]-Dcom.liferay.portal.kernel.util.StreamUtil.buffer.size=8192 ## default=8192[/tt]

As for permissions during import, I'm not sure how this could affect the import but I'll try looking into it.
thumbnail
Ray Augé, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Liferay Legend Beiträge: 1197 Beitrittsdatum: 08.02.05 Neueste Beiträge
One more thing, my test was to set:

[tt]dl.file.max.size=30720000

dl.file.extensions=*[/tt]

Then take workspace/portal-trunk (eclipse working copy) and drop that on the Guest webdav root. That's 80k+ files, totaling ~1.6 GB.
Alan Schrock, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 9 Beitrittsdatum: 16.09.10 Neueste Beiträge
I will let the other guys who are dealing with the WEBDAV issues more consistently try this out.

You mentioned the indexing issue. Another issue I have found is file/folder counts. I believe these counts started showing incorrectly once the nightly build came out to help speed up the library some. The files are there, but at the top level it shows only a few files in the count. There are more than 2000 files in there and as you drill down, they are actually there, just not counted correctly. I did add most of these files initially via WEBDAV, so it may be something with that and the indexing you are talking about. Doing a reindex of the database doesn't matter. I am also using the file system, not database for storage of the files (can't remember if either is an option for doc library or if that is only an image gallary thing).

Library is still very slow for general users. It seems every role but administrator it is slow for (even community owners). Doesn't matter if I give portal permissions to Doclibrary either - all rights, it is still slow. I haven't tried the latest nightly builds for a week or so to see if anything has changed. It is to the point where I am thinking about going back to 5.2.3 or to another platform.

Thanks for your work on this.

Alan
thumbnail
Ray Augé, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Liferay Legend Beiträge: 1197 Beitrittsdatum: 08.02.05 Neueste Beiträge
The solution to solving the slowdown in later builds was to only show the count for files and folders one level bellow a give folder.

The query to calculate the viewable total counts for each folder in a page was deemed too costly (since it was cascading down the entire tree, which was resulting in the huge queries mentioned earlier).

So, you have a tradeoff:

- enabled permission checking so that you hide things the user can't see and show less information (gonna be a little slower since you can't do as much caching)
- disable permission checking which will revert back to where you show everything to everyone, and only give errors on access (gonna be fast since you're caching straight up queries since everyone is the same)

Since we had so many complains about the old way of doing it (no permission checking/give error on access) that we went with the permission + less detail option.

The current implementation shouldn't be to slow (only slightly slower than the old way, but the benefits should outweigh the slowdown.
Alan Schrock, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 9 Beitrittsdatum: 16.09.10 Neueste Beiträge
Thanks. I appreciate the difficulty of trying to address many competing desires and appreciate all of your work.

With the latest build, I am still getting between 20 and 40 seconds for a subfolder to appear. My one folder has 787 files, 657 folder, when I go down to one of the folders within this (that has 185 files/137 folders) it takes 40 seconds between the time I click on the link and the library displays. This is similar to the other folders. All together, I have 1401 files, 823 subfolders. This is logged in as a general user, not admin. Much faster as admin. I am running last night's build (10/28/10) with mysql, tomcat 5.

Also, to add to the computation issues and wish lists, it would really be nice to have folder inheritance as well. Right now, the permission's tag goes on the file and it gets permissions from the folder it is put in. If that folder changes, though, it does affect it's permissions. It is really nice to be able to add roles to an upper file and have the lower files accessible. Even if there would be a routine to propogate the persmissions down, that would be a start. Just a thought.

Thanks for all you do.

Alan
thumbnail
Ray Augé, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Liferay Legend Beiträge: 1197 Beitrittsdatum: 08.02.05 Neueste Beiträge
Ok Alan,

Let's try something slightly different.

Open up [tt]/html/portlet/document_library/folder_columns.jspf[/tt]

Comment out line 20:

[tt]//DLFolderServiceUtil.getSubfolderIds(subfolderIds, scopeGroupId, curFolder.getFolderId(), false);[/tt]

And one line 27 set the value to 0:

[tt]int curFileEntriesCount = 0;//DLFolderServiceUtil.getFileEntriesAndFileShortcutsCount(scopeGroupId, subfolderIds, status);[/tt]

And then see how the performance changes.

Maybe we still need to look at some alternatives here for the statistics.
Alan Schrock, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 9 Beitrittsdatum: 16.09.10 Neueste Beiträge
I implemented this the other night, but had too many other processes going on my machine yesterday to give it a good test. I just completed this.

For smaller folders, this seems to be doing better. We are down to maybe 10-15 seconds to display. Still slow, overall, but better.

For my larger, more top-level folders, I am still seeing between 20 and 30 seconds before the folder displays. That is still a bit long in web response.

Thanks

Alan
thumbnail
Ray Augé, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Liferay Legend Beiträge: 1197 Beitrittsdatum: 08.02.05 Neueste Beiträge
I have on last ditch suggestion for you to try. (this is just for evaluation purposes, not a final solution).

Inside [tt]/html/portlet/document_library[/tt], find all references to [tt]DLFolderServiceUtil[/tt] (in the jsps and jspfs only, there should be ~19) and rename them to [tt]DLFolderLocalServiceUtil[/tt] (except the import in init.jsp, leave that one).

This will effectively eliminate all inline permission checking.

If you test that and see significant improvement, then we may have some work to do yet on inline permissions on large data sets.

Please let us know.
Alan Schrock, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 9 Beitrittsdatum: 16.09.10 Neueste Beiträge
Ray,

I just tried this on my test system and it makes the document library work fast like in 5.2.3.

You say this isn't a permanent solution, just a test. What would be the negatives if I put this on my production server?

Thanks!

Alan
thumbnail
Ray Augé, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Liferay Legend Beiträge: 1197 Beitrittsdatum: 08.02.05 Neueste Beiträge
The only drawback is that you loose the features added in 6.0+ which was to have filtered content appear to users.

What you'll see now is what we had in <=5.2 which was to show everything, and just throw an error when someone tried to access something they didn't have permission to see.

There are no other drawbacks.
Alan Schrock, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 9 Beitrittsdatum: 16.09.10 Neueste Beiträge
That is clear.

Thanks Ray.

Alan
thumbnail
Johnny Z ., geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 19 Beitrittsdatum: 11.07.09 Neueste Beiträge
Ray Augé:
I have on last ditch suggestion for you to try. (this is just for evaluation purposes, not a final solution).

Inside [tt]/html/portlet/document_library[/tt], find all references to [tt]DLFolderServiceUtil[/tt] (in the jsps and jspfs only, there should be ~19) and rename them to [tt]DLFolderLocalServiceUtil[/tt] (except the import in init.jsp, leave that one).

This will effectively eliminate all inline permission checking.

If you test that and see significant improvement, then we may have some work to do yet on inline permissions on large data sets.

Please let us know.


Hi, Ray,

Checking in to see if a permanent solution has been found or released? Please let me know. Thanks in advance and Happy New Year.

Regards,

John
thumbnail
Johnny Z ., geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 19 Beitrittsdatum: 11.07.09 Neueste Beiträge
Hi, Ray - any help on a permanent solution? We would really like to know. Thanks.




Ray Augé:
I have on last ditch suggestion for you to try. (this is just for evaluation purposes, not a final solution).

Inside [tt]/html/portlet/document_library[/tt], find all references to [tt]DLFolderServiceUtil[/tt] (in the jsps and jspfs only, there should be ~19) and rename them to [tt]DLFolderLocalServiceUtil[/tt] (except the import in init.jsp, leave that one).

This will effectively eliminate all inline permission checking.

If you test that and see significant improvement, then we may have some work to do yet on inline permissions on large data sets.

Please let us know.
thumbnail
Ray Augé, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Liferay Legend Beiträge: 1197 Beitrittsdatum: 08.02.05 Neueste Beiträge
There is currently a patch that is being evaluated.

I'll try to get that done as soon as possible.

It does follow the general principle that the statistics could be off slightly whereby it would not account for permissions, that way it could use a cached query (as it did in 5.2). Thus the counts would be off. But for the actual listing of paged entries, that would always take permissions into account.

So this is what is being looked at.

Sorry for the delay.
thumbnail
Johnny Z ., geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 19 Beitrittsdatum: 11.07.09 Neueste Beiträge
Thank you for the update, Ray. Please post a note here when the patch has passed QA and is ready for release.

Thanks again,

Johnny Z.
thumbnail
Alexey Kakunin, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Liferay Master Beiträge: 621 Beitrittsdatum: 07.07.08 Neueste Beiträge
I've tested with Liferay 6.0.5 - disabling counting of subfolders and subfiles count helped to improve performance.

But it is better simple check folderColumns in folder_columns.jspf - does it need to calculate counts or not (since folderColumns may be configured):


int curFoldersCount = 0;
int curFileEntriesCount = 0;

if (ArrayUtil.contains(folderColumns, "num-of-folders") || ArrayUtil.contains(folderColumns, "num-of-documents")) {
	List subfolderIds = new ArrayList();
	
	subfolderIds.add(new Long(curFolder.getFolderId()));
	
	DLFolderServiceUtil.getSubfolderIds(subfolderIds, scopeGroupId, curFolder.getFolderId());
	
	curFoldersCount = subfolderIds.size() - 1;
	curFileEntriesCount = DLFolderServiceUtil.getFileEntriesAndFileShortcutsCount(scopeGroupId, subfolderIds, status);
}



==
Alexey Kakunin
EmForge: Liferay Based Project Hosting Service
thumbnail
Ray Augé, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Liferay Legend Beiträge: 1197 Beitrittsdatum: 08.02.05 Neueste Beiträge
I've pretty much resolved myself to the fact that statistics should be using the cached non-permissioned result and therefore should be as fast as it was in previous versions. (i.e. only the actual paged query itself should be permissioned)

All you need to do is replace the ServiceUtil with the LocalServiceUtil calls where the use is for collection of stats (num-of-folders, num-of-documents).

You should also change the call getSubfolderIds on search to the LocalService. I've actually found that having it as it is, is in fact duplicating some permission checking to begin with so changing to the LocalService is both faster due to cache, as well eliminating duplication of a check which is done later anyway.

I'm going to commit these changes soon (as I get the chance) and I'm sure that the outcry is worthy for this to reach back to next 6.0 and 5.2 EE SPs.
thumbnail
Alexey Kakunin, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Liferay Master Beiträge: 621 Beitrittsdatum: 07.07.08 Neueste Beiträge
Not sure - since it may confuse users: someone will see that folder has subfolders and files (in statistics) - but then will click into folder - will not see them (because has no permission to see them).
So, next question will be: "THere are the files????"

So, I'd like the fact, what Liferay used remote-services to get this statistic and it returned values taking into account current user permissions.
Of course - performance is a problem - we met problem even with 50 files and folders!

So, disabling displaying the statitics was a solution for us - it is better to do not display any statistic even displayb incorrect. Especially in case it is possible to avoid statistics columns from the portlet configuration - only problem was (in Liferay 6.0.5) - even statistics columns was avoided - statistics calculation was called - and my fix is only for this.
thumbnail
Ray Augé, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Liferay Legend Beiträge: 1197 Beitrittsdatum: 08.02.05 Neueste Beiträge
That sounds very reasonable.

Perhaps what we could do is:

1) make sure that disabling the columns disables the calculation of stats
2) offer 2 columns that can be used in the settings
a) stats with permission checks
b) stats without permission checks

Thoughts?
thumbnail
Alexander Chow, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Liferay Master Beiträge: 518 Beitrittsdatum: 20.07.05 Neueste Beiträge
Or add what many operating systems do -- asynchronously load the stats.
thumbnail
Rafael Vega, geändert vor 12 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 3 Beitrittsdatum: 05.05.11 Neueste Beiträge
Ray Augé:
That sounds very reasonable.

Perhaps what we could do is:

1) make sure that disabling the columns disables the calculation of stats
2) offer 2 columns that can be used in the settings
a) stats with permission checks
b) stats without permission checks

Thoughts?


Hi,
I have Installed LifeRay 6.0.5 CE + Tomcat + MySQL.
I have 300 roles, 200 folders/subfolders and 300 files in use.
I have poor performance when browse through the document libraries. I've reported this issue days ago (LPS-16599).
I've been reading this thread and I don't have clear idea about the steps to follow.
Initially, I changed the configuration to use algorithm ver 5, but disappear aprox. 90% of permissions in existing document libraries, folders and files emoticon

Ray Auge propose edit folder_columns.jspf in lines 20, 27 and it seems almost a right workaround but after this, he proposed a better workaround ( rename DLFolderServiceUtil by DLFolderLocalServiceUtil)

I have no clear what to do to in this moment:
What is the best workaround I can use?
Can I use the last nightly build? It works?
I know LR V6.1.X has resolved the LPS-12238. Do I have to wait for the 6.1.X release?

Any help would be greatly appreciated.
Thanks,
thumbnail
ilke Muhtaroglu, geändert vor 12 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Regular Member Beiträge: 226 Beitrittsdatum: 12.05.09 Neueste Beiträge
Hi,

I have applied what have been told at Forum post that I have started because of performance failure at document library:

http://www.liferay.com/community/forums/-/message_boards/view_message/8986666#_19_message_8986666

The solution proposed was to disable the inline permission check via returning false in the method

InlineSQLHelperImpl.java -> public boolean isEnabled(long groupId)

I have done what has been proposed but the solution yield to unaccessible document ?

Is there any solution to this issue till now ?

We are in production and we distributed the documents in several folders to balance this performance issue and the reuslt is 25 seconds per page. Can you please assist to a solution ?

We are really stucked.


Kind Regards.

ilke



ilke









******************************

Sir,

thanks I applied what you said.

It improved the performance that is right but It the documents unaccessible!!!
------

It says,

" the requested resource was not found ! "
http://xx.xx.xx.xx/documents/323/323/aaa.pdf

-----

But it works fine for test (administrator) user.

Any idea, or help please ?

thanks in advance.

ilke



*******************************
thumbnail
Srikanth Reddy Sanivarapu, geändert vor 12 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Regular Member Beiträge: 203 Beitrittsdatum: 15.11.08 Neueste Beiträge
Did you try to bounce the DB Server once? Thanks.

Regards,
Srik
thumbnail
ilke Muhtaroglu, geändert vor 12 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Regular Member Beiträge: 226 Beitrittsdatum: 12.05.09 Neueste Beiträge
Sir,

what is bouncing the DB server, I couldnt get the term ? Can you explain the operation supposed to do ?

Regards.

ilke
Zenobia L, geändert vor 12 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Junior Member Beiträge: 28 Beitrittsdatum: 08.04.11 Neueste Beiträge
Hello,

is this issue resolved in 6.0.6? I see JIRA has an issue reporting a fix in 6.1, but didn't find information regarding 6.0.6

thank you,
Z
Teet Raid, geändert vor 12 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 2 Beitrittsdatum: 26.12.11 Neueste Beiträge
Hi,

Ray Augé:
That sounds very reasonable.

...

Thoughts?


Just a sidenote. There is no performance degradation when logged in as a user who has administrator privileges (role).

t.
Carlos Vicente, geändert vor 12 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Junior Member Beiträge: 25 Beitrittsdatum: 08.06.10 Neueste Beiträge
I Have the same problem. But i think i found the real cause.

When y use Document Library (or forums), Liferay throws a query to the database with a join to the resourcemission table
The problem is that for some reason, at least in mysql, this query doesn't use any of the indexes defined in resourcepermission and do a full scan. In our installation resourcepermission has 6 million rows, and this query has a really poor performance creating temporary tables in disk.
I'm triyn to force in the query to use a index, but without success at the moment.
thumbnail
Johnny Z ., geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 19 Beitrittsdatum: 11.07.09 Neueste Beiträge
Alan Schrock:
Thanks. I appreciate the difficulty of trying to address many competing desires and appreciate all of your work.

With the latest build, I am still getting between 20 and 40 seconds for a subfolder to appear. My one folder has 787 files, 657 folder, when I go down to one of the folders within this (that has 185 files/137 folders) it takes 40 seconds between the time I click on the link and the library displays. This is similar to the other folders. All together, I have 1401 files, 823 subfolders. This is logged in as a general user, not admin. Much faster as admin. I am running last night's build (10/28/10) with mysql, tomcat 5.

Also, to add to the computation issues and wish lists, it would really be nice to have folder inheritance as well. Right now, the permission's tag goes on the file and it gets permissions from the folder it is put in. If that folder changes, though, it does affect it's permissions. It is really nice to be able to add roles to an upper file and have the lower files accessible. Even if there would be a routine to propogate the persmissions down, that would be a start. Just a thought.

Thanks for all you do.

Alan


Hi, Alan,

I thought I would check with you to see if you had any success in implementing any of the suggestions or if you had success with finding a work-around/resolution on your own?

Any help/suggestions would be greatly appreciated.

Regards,

JZ
Alan Schrock, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 9 Beitrittsdatum: 16.09.10 Neueste Beiträge
Johnny,

I just responded to the one test so you can see the results from that. Still slow, but a little bit better. For the webdav one, I haven't done any testing with that it.

I was curious on my response times with 5.2.3 so I fired up my old version. Nearly an identical repository yieled approx. 3 seconds between click and folder display. Obviously, changes have happened which were needed to get some better permission checking and this is the pain of the process of change. I do like, though the 3 better than 20! :-)

And, yes, as you mentioned, disabling permission checking does it portal-wide and that isn't an option.

Just got an email with another test from Ray, I'll try that and post a reply.

Alan
thumbnail
Johnny Z ., geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 19 Beitrittsdatum: 11.07.09 Neueste Beiträge
Thanks, Ray. And thanks Alan too! We will try and let you know what we find. Cheers, Johnny
thumbnail
Johnny Z ., geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 19 Beitrittsdatum: 11.07.09 Neueste Beiträge
Any Update on how to address this issue without having to use the LR5 permission? Thanks in advance.
thumbnail
Johnny Z ., geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 19 Beitrittsdatum: 11.07.09 Neueste Beiträge
Ray Augé:
...
- enabled permission checking so that you hide things the user can't see and show less information (gonna be a little slower since you can't do as much caching)
- disable permission checking which will revert back to where you show everything to everyone, and only give errors on access (gonna be fast since you're caching straight up queries since everyone is the same)


Ray - where do we disable permission checking? Is this something we can manage in the portal ext properties file? Any help would be greatly appreciated. Thanks!
thumbnail
Ray Augé, geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

Liferay Legend Beiträge: 1197 Beitrittsdatum: 08.02.05 Neueste Beiträge
Sigh!!! My vague explanations strike again!

I didn't mean "disable" as in "a configuration option". I should not have used the word "disable" at all. In it's place I should have written "not program it to use inline permission checking".

My apologies for sending you down a road that don't exist.
thumbnail
Johnny Z ., geändert vor 13 Jahren.

RE: Liferay 6.0.4 Document Library Slow Performance

New Member Beiträge: 19 Beitrittsdatum: 11.07.09 Neueste Beiträge
Ray Augé:
In it's place I should have written "not program it to use inline permission checking".


Ray - thank you for the clarification. I am on the business side and am not very technical. So, I don't exactly understand what you mean by "not program it to use inline permission checking". I shared your response with my developer and he also doesn't understand. Is your suggestion something that can be done at the portlet level? I ask because we have tried changing the permission structure for the Portal (e.g., use the permissions for LR5) and while it fixed the [slow] responsiveness of the Doc Library, it had huge negative effects on other aspects of our implementation. Thus we rolled back to the original permission structure. If we could remove the permission checking for just the Doc Library as a work-around, it would suffice for the time being.

Thanks for all of your help. We eagerly look forward to your response.

Cheers, Johnny Z.