Liferay Sync consume all server CPU

[Fabio Trezzi, modificado hace 9 años.]

Hi,
we have Liferay Sync (1.2.2-GA3) with that sync with Liferay version 6.2 (Liferay Portal Community Edition 6.2 CE GA2 (Newton / Build 6201 / March 20, 2014)) backed by SQL Server

The system worked fine (initially with an older version of the sync) for weeks but recently every time the sync starts the server start spawning new threads untill it completely saturate the server.

Server logs doesn't report anything useful and the sync log is attached to this post

I've tried rebuilding lucene indexes, remove all files from the index (max size 0) create a new clean installation of the sync into a new computer but without success.

Is it possible that we have to many files ? (44gb of data in sync)

Thanks for your time

[Dennis Ju, modificado hace 9 años.]

Hi Fabio,

How many Sync clients do you have connecting to the portal? Each Sync client should only make one connection to the portal (so one thread per client). If you're seeing more threads spawned then clients, then there's likely an issue with the Sync client. Can you send me the logs from a Sync client (logs are found in ~/.liferay-sync/logs). You can attach logs here or email me directly at [e-mail].

Did the issue start happening immediately after you upgraded to 1.2.2? What version were you running before upgrading?

[Fabio Trezzi, modificado hace 9 años.]

Hi Dennis,
thanks for your reply! You can find the sync log attached to my first post.

We currently use just one user and just one sync installation, we use it to sync out respository in a network shared disk with the portal. The problem is not the sync installation of the shared disk because to ensure it I've installed a new sync installation on a new computer (sync version 1.2.2) and I have the same problem here trying to sync the files locally.

Initially the system worked great withe the older version of the client 1.2.1 if I'm correct but then the sync started ignoring all the files. These are the old logs:

11 set 2014 17:13:59,233 DEBUG [FileEventHandler:36] Received event MODIFY : D:/LiferaySync/liferay-sync/Guest/certificati_ce
11 set 2014 17:13:59,264 DEBUG [FileEventHandler:36] Received event ADD : D:/LiferaySync/liferay-sync/Guest/certificati_ce/PIPPO2.pdf
11 set 2014 17:14:01,405 ERROR [FileModelBaseEvent:81] Ignored file
11 set 2014 17:14:01,420 DEBUG [FileModelManager:1070] Ignored file: D:/LiferaySync/liferay-sync/Guest/certificati_ce
11 set 2014 17:14:01,576 ERROR [FileModelBaseEvent:81] Ignored file
11 set 2014 17:14:01,576 DEBUG [FileModelManager:1070] Ignored file: D:/LiferaySync/liferay-sync/Guest/certificati_ce/PIPPO2.pdf

So I've suggested to upgrade to a new version of the sync but now everytime the sync start all the CPU go to 100% and the portal dies. Htop register a lot of threads and I can see the sync icon that tell that the repository is synchronized (not true). Sometimes it became grey (mac edition) and then it goes back to black.

I've thought that it was a timeout problem. The sync request a list, the server takes to much time to respond, the sync request it again (sync log reports lot of timeout errors) and this cause the cpu to saturate and a lot of threads generated. I've doubled the number of core but the result its still the same.

Do you have any idea ?

Attached to this post you can find the thread dump generated while the CPU were at 100%

Thanks again!

[Dennis Ju, modificado hace 9 años.]

I see the problem now. It is indeed because of the size of one of your repositories. When Sync first initializes, it makes a request to grab the entire file structure of a site. For very large repositories, this can take a long time (especially if you have large files). The default timeout for Sync connections is set to one minute, after which it disconnects and attempts to reconnect and tries to grab the file structure again. The thread dump shows several threads are stuck trying to download the list.

This issue is also more likely to occur with 1.2.2 than 1.2.1 because 1.2.1 makes recursive calls to grab each folder individually which caused problems if the server disconnected while downloading (see SYNC-1236). I imagine if you try syncing with repos with less files, this won't be an issue.

I've created a ticket to track this issue SYNC-1256. I can drastically improve the performance of fetching the file list due to behavior changes we made in preparation for Sync 2. Please note, our active development efforts are currently on Sync 2 which won't encounter this same issue.

[Dennis Ju, modificado hace 9 años.]

Do you have a test environment where you can duplicate the issue? I can send you a developmental build for the Sync plugin that should resolve the issue, but I wouldn't run it in a production environment.

[Fabio Trezzi, modificado hace 9 años.]

Hi Dennis,
thanks very much for your help! As you immagined if I request the system to download a smaller repository everything works fine.

I've seen that the Sync has an option to do a check every 30 minutes this alter the timeout or not ?

I can try to arrange a different system for testing purpose if you share a development build with me.

Thanks again!

[Dennis Ju, modificado hace 9 años.]

Hi Fabio,

Unfortunately, the 30 min check will not affect the timeout.

I have attached a development build of sync-web. Note, it's the latest build (commit c3b3364) of the sync-web portlet available here. The key changes are here.

[Dennis Ju, modificado hace 9 años.]

Have you been able to try the new development build of sync-web? Any feedback?

Thanks,
Dennis

[Fabio Trezzi, modificado hace 9 años.]

Hi Dennis,
thanks a lot for your help and support. Unfortunately we are in a hurry with this project and we can't wait to create a separate environment to test the sync. We were using the sync to sync a shared network disk with the portal. I've switched to the webdav protocol. We have attached the portal as a network disk on windows and we keep the portal aligned using rsync or some other tools.

Thanks again for your support if we have time we really like to test your build
Fabio