Liferay Sync 3 Beta Released

[Michael Young, módosítva 9 év-val korábban]

Read the announcement here.

[Chris Stavros, módosítva 8 év-val korábban]

Hi Michael,

Do you happen to know if the required "Sync Beta Connector" that is required to use the 3.x beta client is backward compatible with 1.2.x client?

Thank you,
Chris

[Dennis Ju, módosítva 8 év-val korábban]

The connector is indeed backwards compatible with Sync client 1.2.x.

[Chris Stavros, módosítva 8 év-val korábban]

Oh, shiny! Thanks, Dennis!

[Ankur Srivastava, módosítva 8 év-val korábban]

Hi,

Do you know if this release supports SSO ? We are using SiteMinder and want to use Sync.

Thanks,
Ankur

[Dennis Ju, módosítva 8 év-val korábban]

Hi Ankur,

When Sync 3 GA1 is released, SSO will be beta feature (available only for portal EE).

We are currently providing beta SSO builds to customers to test various environments. We have successfully tested with SiteMinder, Shibboleth, and OpenAM/OpenSSO, but we're still testing more SSO environments before making it an official supported feature. Please let me know if you would like to receive and test the SSO build.

[Ankur Srivastava, módosítva 8 év-val korábban]

Hi Dennis,

Thanks for your quick response and for sharing the details. This sounds really good.

It will be great if you can please share the SSO build. We are using Liferay EE 6.2 SP8 currently. That ways we can test and make plans to incorporate that.

Again thanks.

Ankur

[Vicki Lea Tsang, módosítva 8 év-val korábban]

Hi Ankur,

Thanks for your interest in the Sync SSO beta. Please can you contact me at [e-mail] and provide me the name of the company holding the LR EE license and the email address that you would like the beta information to be sent to.

Best Regards, Vicki

[Ankur Srivastava, módosítva 8 év-val korábban]

Hi Dennis,

Can you please help us understand a couple of things related to Sync -

1. We are concerned about the Portal performance getting impacted because of Sync usage. Is that something to be worried about ? Like when the document sync will happen then can it affect the performance.

2. We have documents stored in Liferay documents and media as well as in another folder location, to which Liferay has references. Will all the docs be synced up or only the ones which are maintained in the Liferay document library ?

Thanks,
Ankur

[Dennis Ju, módosítva 8 év-val korábban]

Hi Ankur,

1) Around the time Liferay Sync 3 GA1 is released, we will also publish a recommended architecture and performance white paper. This should give a good idea of how Sync might impact the performance of your portal.

2) Liferay Sync only syncs with Liferay's own document library.

[Ankur Srivastava, módosítva 8 év-val korábban]

Dennis Ju:

Hi Ankur,

1) Around the time Liferay Sync 3 GA1 is released, we will also publish a recommended architecture and performance white paper. This should give a good idea of how Sync might impact the performance of your portal.

2) Liferay Sync only syncs with Liferay's own document library.

Hi Dennis.

Thanks for your prompt response. Its good to hear that a whitepaper will be published for the same. Any idea by when can we expect it ?

Also the SYNC documentation outlines

"Additionally, administrators will need to grant unauthenticated access to the following
URL so Sync clients can read context information required for OAuth:
http(s)://<serveraddress>/
api/jsonws/syncweb.
syncdlobject/getsynccontext"

We have firewalls in place which won't allow unauthenticated access. Is there a workaround for this, so that our users can access it via SSO ?

Thanks,
Ankur

[Dennis Ju, módosítva 8 év-val korábban]

Hi Ankur,

The whitepaper should be released alongside with Sync 3 GA1 (or shortly thereafter) sometime in the next week (or two).

OAuth/SSO will not be officially supported in GA1. The feature will still be in beta and is planned for Sync 3.1. We're still testing the OAuth/SSO implementation with various client environments. (There's still at least one tested SSO environment that isn't currently working with the Sync SSO alpha builds).

Right now, there are no workarounds for needing unauthenticated access, so a firewall exception would need to be made (for that one URL).

[Sven Aggenbach, módosítva 8 év-val korábban]

Dennis Ju:

...1) Around the time Liferay Sync 3 GA1 is released, we will also publish a recommended architecture and performance white paper. This should give a good idea of how Sync might impact the performance of your portal....

Well I am very curious about this too, because I have mixed results using Liferay Sync. Let me share my experiences below.

I have been testing Liferay Sync 3.0.8 beta, on Mac OS 10.8.5 for a week now, connected to a Liferay Portal CE 6.2.3 GA4 (which is an upgrade from an CE 6.1). Our repository on disk is about 23GB, however this includes versioning. The repository size of only the most recent versions of all files is about 8GB.

I was able to sync this repo using my Mac. But then I started to drag files in Finder from my local HDD to the Synced folder (total file size of files to be transmitted about 3GB ). At first this seemed to go well, however during the process 'SyncUILauncher' takes 100% CPU, spans over 60 threads and consumes the incredible amount of 1.2GB of Memory (and rising).

I inspected the logs and found a lot of (broken pipes):
09:39:37.735 TRACE [pool-14-thread-3] [BaseEvent:156] Processing event UpdateFileEntriesEvent
09:39:48.011 DEBUG [pool-6-thread-1] [BaseHandler:72] Handling exception java.net.SocketException: Broken pipe
09:39:49.012 TRACE [pool-14-thread-2] [BaseEvent:156] Processing event RetryServerConnectionEvent
09:40:18.066 TRACE [pool-14-thread-2] [BaseJSONHandler:276] Handling response RetryServerConnectionHandler

The other strange thing is that Liferay Sync once in a while indicates that it is 'Synced' and then some minutes later starts sending files again even nothing has changed on my local drive (verified in the status dialog in the top menu that shows about 458 files going up together with the a data transfer rate they are transmitted at). Even when it now and then indicates the status as being 'Synced', I do not see all files showing up in Liferay. This kind of confuses me: how certain can I be that when Liferay Sync indicates 'Synced' everything actually is synced?

Because of these results, the following questions came to my mind:

• what does the number of files in the status bar indicate:
  
  • the number of files that still need to be synced,
  • the number of simultaneously transferred files,
  • some other thing?
• how can I identify on client side if an individual file has been synced or not (it would be very convenient to have a list/table of the files that are stored locally with an indication if they are already synced or not) - e.g. can I interpret messages in the log like 'Processing event AddFileEntryEvent file path' that a file has successfully been transmitted to the portal?
  
• it seems that filenames that start with a dot '.' are not being transmitted to the portal, I see '[Watcher:211] Ignored file path' messages for these files. Is that expected behavior (if so: why are these files exclude?)?
  
• how do re-transmissions work? when a broken-pipe or lost connection occurs during a file transmission, is the file re-send in whole or does the transmission continues where it left off?
  
• is there a way to indicate that sync does not transmit full file content, but only meta data (like 'name' and 'size) and then when a file is locally accessed it is being fetched at that time only (just-in-time loading)? I foresee (based on my results so far) that when I let all employees here start using Liferay Sync we will make our Liferay portal unresponsive for a long time

Potentially it is a great product but I am kind of hesitant to adopt it for the full organisation at this moment, so please prove me wrong

[Dennis Ju, módosítva 8 év-val korábban]

Hi Sven,

Thanks for the detailed feedback. Below are my responses to your various concerns/questions.

- Unfortunately, most Java apps do not return reclaimable memory to the operating system even if it's no longer being used (an issue with the JVM). The heap size in 3.0.8 was set to 1gb, so the process will use 1gb + extra memory for the JVM. Sync doesn't actually use this much memory, so in GA1, we will reduce this to half. CPU usage is quite heavy while syncing as file downloads/uploads tend to be CPU intensive. A quick test of downloading a single large file through Google Chrome shows a CPU usage of 70%, and uploading a couple large files through Dropbox shows a CPU usage of 150%.

- The icon should continue to show a "syncing" status until all file transfers are complete. If the icon is toggling between "synced" and "syncing" even while files are transferring, we'd need to investigate further. This may be related to the random disconnect errors you're encountering. Can you provide more details on your environment? What is your network setup (connected via LAN or WAN to the portal, network bandwidth speed, etc) and what kind of machine is the portal running on?

- The status bar shows the number of files left to be synced. How many files are actually simultaneously uploading/downloading depend on the "Max Connections" value set in the Sync Admin Control Panel portlet (default is 1).

- Files that begin with "." are ignored since they are considered hidden/system files used particularly for Mac/*nix environments. Since these files are not meant for collaborative sharing and are not used in Windows environments, they are ignored by default. However, if you can provide a use case for why you would want to sync these files, we can explore options to change the behavior.

- If there is a disconnect, the entire file is re-transmitted.

- There is no way for "just-in-time loading" as we cannot detect when a file is accessed locally 100% of the time, plus this reduces the value of using Sync to have immediate access to files, even when you're offline. Actually, browsing via WebDAV would essentially be a "just-in-time loading" solution. But it is true that having many Sync clients on a busy document library can be taxing on the portal. Our white paper will discuss our recommended architecture (setting up a separate server/cluster for Sync requests) and will show numbers of how the portal performs w/ x number of sessions and y number of documents.

[Sven Aggenbach, módosítva 8 év-val korábban]

Dennis Ju:

- Unfortunately, most Java apps do not return reclaimable memory to the operating system even if it's no longer being used (an issue with the JVM). The heap size in 3.0.8 was set to 1gb, so the process will use 1gb + extra memory for the JVM. Sync doesn't actually use this much memory, so in GA1, we will reduce this to half. CPU usage is quite heavy while syncing as file downloads/uploads tend to be CPU intensive. A quick test of downloading a single large file through Google Chrome shows a CPU usage of 70%, and uploading a couple large files through Dropbox shows a CPU usage of 150%.

Ah yes, you are absolutely right that file transmissions are CPU intensive. I have to admit I don't have any experience with using Dropbox, so probably the same problems apply there. I am just wondering about the (for us real-life) scenario that an organisation has adopted Liferay, has one way or another filled its Document & Media Library with thousands of files and then encourages all its employees to start using Liferay Sync. I expect every employee will encounter that the sync will take a (very) long time and during the process keeps using a lot of CPU (with degradation of the user experience in return).

Dennis Ju:

- The icon should continue to show a "syncing" status until all file transfers are complete. If the icon is toggling between "synced" and "syncing" even while files are transferring, we'd need to investigate further. This may be related to the random disconnect errors you're encountering. Can you provide more details on your environment? What is your network setup (connected via LAN or WAN to the portal, network bandwidth speed, etc) and what kind of machine is the portal running on?

Sure! I am running Liferay Sync v3.0.8 beta on Mac OS 10.8.5. The server is a XEN-based virtualised machine running Ubuntu 12.04LTS with 4 cores Intel XEON E5530 and 4GB of memory available to the JVM (which happens to be Tomcat 7). The server is located in one of our data centres and yes, that has some switches and routers and of course firewalling. I tested the uploads/downloads from several locations (that all have NAT) where the one with the highest bandwidth (from client to server with all interconnecting hops) is in theory 100Mbps, but separate measurements (outside Liferay Sync) showed it to be 40Mbps - steady. I keep suffering from dropping connections in Liferay Sync though. To rule out it is not my Macbook Pro late 2010 model, I will try some other devices here as well - didn't do that yet - so will get back to you on that later.

Dennis Ju:

- The status bar shows the number of files left to be synced. How many files are actually simultaneously uploading/downloading depend on the "Max Connections" value set in the Sync Admin Control Panel portlet (default is 1).

Thanks, I wasn't aware of that option yet. Will play around with it and see what happens.

Dennis Ju:

- Files that begin with "." are ignored since they are considered hidden/system files used particularly for Mac/*nix environments. Since these files are not meant for collaborative sharing and are not used in Windows environments, they are ignored by default. However, if you can provide a use case for why you would want to sync these files, we can explore options to change the behavior.

By default that sounds fine. I cannot think of files starting with a '.' that I would like to have synced, however I can think of more files that I want to have excluded of sync, e.g. I can imagine that I don't want movies to be synced to my laptop. So it would be very nice and convenient to have a user configurable list where people can indicate which type of files they want to be part of the sync. Thinking out loud even further, I can also imagine that it would be nice to have the ability to base the set of files to be synced on filesize, e.g. a filter that only allows files smaller than x[MB] to be synced. I do realize that this conflicts with the idea of having a local copy of your D&M library, because you might be missing critical files based on the filters set... well I guess it would be a trade off...

Dennis Ju:

- If there is a disconnect, the entire file is re-transmitted.

Ok, so how does that work with mobile devices, large files and unstable connections? By doing full re-transmits I can imagine that it is much harder to get every files synced than by being able to do incremental transmits.

Dennis Ju:

- There is no way for "just-in-time loading" as we cannot detect when a file is accessed locally 100% of the time, plus this reduces the value of using Sync to have immediate access to files, even when you're offline. Actually, browsing via WebDAV would essentially be a "just-in-time loading" solution.

Ok, then I am wondering even more how this is supposed to work with mobile devices and very large D&M library's. I don't have tens of gigabytes available on my phone.... ;) So what is the scenario for mobile phones and tablets that use the mobile sync app?

Dennis Ju:

But it is true that having many Sync clients on a busy document library can be taxing on the portal. Our white paper will discuss our recommended architecture (setting up a separate server/cluster for Sync requests) and will show numbers of how the portal performs w/ x number of sessions and y number of documents.

Sounds like a reasonable solution! Would that separate server/cluster then require additional Liferay EE licenses in that case?

[Dennis Ju, módosítva 8 év-val korábban]

Sven Aggenbach:

Ah yes, you are absolutely right that file transmissions are CPU intensive. I have to admit I don't have any experience with using Dropbox, so probably the same problems apply there. I am just wondering about the (for us real-life) scenario that an organisation has adopted Liferay, has one way or another filled its Document & Media Library with thousands of files and then encourages all its employees to start using Liferay Sync. I expect every employee will encounter that the sync will take a (very) long time and during the process keeps using a lot of CPU (with degradation of the user experience in return).

Yes, users should expect the initial sync to potentially take some time as the download can be on the order of gigabytes (depending on the sites). However, after the initial sync, file transfers should be relatively intermittent and mostly transparent to the user in regards to CPU usage.

Sven Aggenbach:

Sure! I am running Liferay Sync v3.0.8 beta on Mac OS 10.8.5. The server is a XEN-based virtualised machine running Ubuntu 12.04LTS with 4 cores Intel XEON E5530 and 4GB of memory available to the JVM (which happens to be Tomcat 7). The server is located in one of our data centres and yes, that has some switches and routers and of course firewalling. I tested the uploads/downloads from several locations (that all have NAT) where the one with the highest bandwidth (from client to server with all interconnecting hops) is in theory 100Mbps, but separate measurements (outside Liferay Sync) showed it to be 40Mbps - steady. I keep suffering from dropping connections in Liferay Sync though. To rule out it is not my Macbook Pro late 2010 model, I will try some other devices here as well - didn't do that yet - so will get back to you on that later.

Can you retry your tests with the latest Sync 3 GA1 that was just released? It's resolved a number of syncing issues since the beta. link

Sven Aggenbach:

By default that sounds fine. I cannot think of files starting with a '.' that I would like to have synced, however I can think of more files that I want to have excluded of sync, e.g. I can imagine that I don't want movies to be synced to my laptop. So it would be very nice and convenient to have a user configurable list where people can indicate which type of files they want to be part of the sync. Thinking out loud even further, I can also imagine that it would be nice to have the ability to base the set of files to be synced on filesize, e.g. a filter that only allows files smaller than x[MB] to be synced. I do realize that this conflicts with the idea of having a local copy of your D&M library, because you might be missing critical files based on the filters set... well I guess it would be a trade off...

Yes, we've thought of these same features and plan to add them in a future release for the Sync Admin control panel portlet.

Sven Aggenbach:

Ok, so how does that work with mobile devices, large files and unstable connections? By doing full re-transmits I can imagine that it is much harder to get every files synced than by being able to do incremental transmits.

This is also a valuable feature we hope to add in a future release.

Sven Aggenbach:

Ok, then I am wondering even more how this is supposed to work with mobile devices and very large D&M library's. I don't have tens of gigabytes available on my phone.... ;) So what is the scenario for mobile phones and tablets that use the mobile sync app?

The mobile apps do indeed load files "just-in-time", but the desktop app will immediately download all files for selected sites.

Sven Aggenbach:

Sounds like a reasonable solution! Would that separate server/cluster then require additional Liferay EE licenses in that case?

That's a good question! I'm honestly not sure about this. I would advise contacting your sales rep for details on this.

[Marlene Tare, módosítva 8 év-val korábban]

I have 3.0.6 installed. It is mapped to the site level. All folders, subfolders and files are still downloading over 3 days. Is there a way to only map to a particular folder? I have concern that if HR have secured documents they are downloaded to a local client they are no longer secure.

[Denis Signoretto, módosítva 8 év-val korábban]

Hi Marlene,

I've followed the webinar "Introducing the New Liferay Sync 3.0" I can say that the folder selection feature isn't currently available but it's in roadmap as reported in the slides of the presentation.

Regards,
Denis.

[Dennis Ju, módosítva 8 év-val korábban]

Selective folder syncing is indeed on the roadmap for a future release. (likely the next feature release after GA1).