Foren

Very large file upload in LifeRay Portal problem

Aingaran Somaskandarajah, geändert vor 16 Jahren.

Very large file upload in LifeRay Portal problem

New Member Beitrag: 1 Beitrittsdatum: 28.09.07 Neueste Beiträge
Hi,

We have created a Portal application for Call Centre workers based on LifeRay Portal (v4.2.1 build 3501). One function of the Portal is to enable the upload of certain csv files by call centre agents for overnight batch processing.

So far the csv file sizes have not been an issue as they have been less than 2MB. The files we are now working with are around 100MB in size and will increase significantly in the future (possibly upto 1GB ). We are uploading using a standard HTTP PUT which is pushed in to a server location or to the DB as a CLOB.

The problems we are facing are:

1) With files around 50MB/80MB the files start uploading but the Portal user (login) session times out because it takes too long.

2) With very large files around 100MB + the Portal issues a OutOfMemoryException with regards to Heap Space. The Heap space on our Test box was set to 512MB.

What would be the best way to address these issues? If there are any code snippets or pointers to projects where people have overcome similar issues it would be very useful to us.

It would also be nice to have some sort of "progress bar" to indicate how long it will take - is there a plugin/community written app for this?

I apologise if this exact subject has been covered before - I searched the forum but could only find info about standard file uploads (which we have already implemented)

Thank you in advance for any help you guys can provide,

Bob
thumbnail
Jorge Ferrer, geändert vor 16 Jahren.

RE: Very large file upload in LifeRay Portal problem

Liferay Legend Beiträge: 2871 Beitrittsdatum: 31.08.06 Neueste Beiträge
Hi Aingaran,

Aingaran Somaskandarajah:

1) With files around 50MB/80MB the files start uploading but the Portal user (login) session times out because it takes too long.


You can configure Liferay (portal.properties and web.xml) to have larger sessions.

Aingaran Somaskandarajah:

2) With very large files around 100MB + the Portal issues a OutOfMemoryException with regards to Heap Space. The Heap space on our Test box was set to 512MB.


Liferay 4.2.1 stored the uploaded files in memory during the upload process. Liferay 4.3.1 fixed this and now copies large files to disk as they are being uploaded. So you may want to upgrade to get this fix, or investigate the source code and backport the solution to 4.2.

Aingaran Somaskandarajah:

It would also be nice to have some sort of "progress bar" to indicate how long it will take - is there a plugin/community written app for this?


That's something that you can implement in your web application. Take a look at the document library for an example.

Having said all this, I would recommend a completely different approach for this functionality, specially since you expect these files to grow larger than a Gigabyte. I would upload those files to the server using an specific protocol, such as FTP and leave them in a directory accessible to Liferay and then, once it's uploaded, just issue a request to Liferay to read it from disk and start the batch processing.
Alessandro Perazzetta, geändert vor 14 Jahren.

RE: Very large file upload in LifeRay Portal problem

New Member Beiträge: 5 Beitrittsdatum: 18.06.09 Neueste Beiträge
Hi all,

Jorge Ferrer:
Hi Aingaran,

Aingaran Somaskandarajah:

1) With files around 50MB/80MB the files start uploading but the Portal user (login) session times out because it takes too long.


You can configure Liferay (portal.properties and web.xml) to have larger sessions.


From my experience i did some changes on portal-ext.properties, i applied these params for extending long time sessions:

# Extend session timeout
#session.timeout=30
session.timeout=5
session.timeout.auto.extend=true


Jorge Ferrer:

Aingaran Somaskandarajah:

2) With very large files around 100MB + the Portal issues a OutOfMemoryException with regards to Heap Space. The Heap space on our Test box was set to 512MB.


Liferay 4.2.1 stored the uploaded files in memory during the upload process. Liferay 4.3.1 fixed this and now copies large files to disk as they are being uploaded. So you may want to upgrade to get this fix, or investigate the source code and backport the solution to 4.2.


I read around here that from 4.1 release there's the possibility to select the storage hook implementation and also select some tweaks to overhead the standard limit, and these my settings:

# Increase Document Library upload filesize
# 0 Unlimited
#dl.file.max.size=0
dl.file.max.size=1048576000

##
## Upload Servlet Request
##

#
# Set the maximum file size. Default is 1024 * 1024 * 100.
#
#com.liferay.portal.upload.UploadServletRequestImpl.max.size=1048576000
com.liferay.portal.upload.UploadServletRequestImpl.max.size=800000000

#
# Set the temp directory for uploaded files.
#
#com.liferay.portal.upload.UploadServletRequestImpl.temp.dir=C:/Temp
com.liferay.portal.upload.UploadServletRequestImpl.temp.dir=/tmp

#
# Set the threshold size to prevent extraneous serialization of uploaded
# data.
#
com.liferay.portal.upload.LiferayFileItem.threshold.size=262144

#
# Set the threshold size to prevent out of memory exceptions caused by
# caching excessively large uploaded data. Default is 1024 * 1024 * 10.
#
com.liferay.portal.upload.LiferayInputStream.threshold.size=10485760



#
# Use our own StringMaker to extend StringBuffer and ByteArrayMaker to
# extend ByteArrayOutputStream. This will add analytical tracking of our
# char and byte usage so we can better optimize memory by minimizing
# unnecessary garbage collecting that comes from unspecified array
# growth.
#
com.liferay.portal.kernel.util.ByteArrayMaker.initial.size=8000
com.liferay.portal.kernel.util.StringMaker.initial.size=128


referring to this issue http://issues.liferay.com/browse/LEP-4277 seems that there's a ByteArrayMaker that shadow the upload for cache and even if i setup the threshold i ran however into this problem (OutOfMemoryError):

[#|2009-06-18T12:17:59.343+0200|SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=19;_ThreadName=httpSSLWorkerThread-8080-1;_RequestID=14027152-c9f3-47ef-b3c5-70ab6099b80e;|ApplicationDispatcher[] PWC1231: Servlet.service() for servlet Main Servlet threw exception
java.lang.OutOfMemoryError
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:247)
at com.liferay.portal.util.FileImpl.write(FileImpl.java:753)
at com.liferay.portal.util.FileImpl.write(FileImpl.java:763)
at com.liferay.portal.kernel.util.FileUtil.write(FileUtil.java:284)
at com.liferay.documentlibrary.util.FileSystemHook.addFile(FileSystemHook.java:94)
at com.liferay.documentlibrary.service.impl.DLLocalServiceImpl.addFile(DLLocalServiceImpl.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy227.addFile(Unknown Source)
at com.liferay.portlet.documentlibrary.service.impl.DLFileEntryLocalServiceImpl.addFileEntry(DLFileEntryLocalServiceImpl.java:278)
at com.liferay.portlet.documentlibrary.service.impl.DLFileEntryLocalServiceImpl.addFileEntry(DLFileEntryLocalServiceImpl.java:173)
at com.liferay.portlet.documentlibrary.service.impl.DLFileEntryLocalServiceImpl.addFileEntry(DLFileEntryLocalServiceImpl.java:135)



At the moment my JVM settings:
<jvm-options>-Xms2048m</jvm-options>
<jvm-options>-Xmx2650m</jvm-options>
<jvm-options>-XX:MaxPermSize=128m</jvm-options>

but none of these changes make the upload working for a file larger than 500M




Jorge Ferrer:

Aingaran Somaskandarajah:

It would also be nice to have some sort of "progress bar" to indicate how long it will take - is there a plugin/community written app for this?


That's something that you can implement in your web application. Take a look at the document library for an example.

Having said all this, I would recommend a completely different approach for this functionality, specially since you expect these files to grow larger than a Gigabyte. I would upload those files to the server using an specific protocol, such as FTP and leave them in a directory accessible to Liferay and then, once it's uploaded, just issue a request to Liferay to read it from disk and start the batch processing.



How did you implement the batch processing?
Aditi Guha, geändert vor 10 Jahren.

RE: Very large file upload in LifeRay Portal problem

New Member Beiträge: 4 Beitrittsdatum: 02.07.13 Neueste Beiträge
What is the maximum value for com.liferay.portal.upload.UploadServletRequestImpl.max.size ?
thumbnail
bharath dg dg, geändert vor 14 Jahren.

RE: Very large file upload in LifeRay Portal problem

New Member Beiträge: 19 Beitrittsdatum: 10.06.09 Neueste Beiträge
hi Aingaran..

I am trying to upload file with simple jsp + portlet. Its not working for me.

My code is here

DiskFileItemFactory factory = new DiskFileItemFactory();
PortletFileUpload pfu = new PortletFileUpload(factory);
pfu.setSizeMax(uploadMaxSize); // Maximum upload size

Its printing pfu object .

List fileItems = (List) pfu.getItemIterator(request);

When i tried to print fileItems object,its printing null.

If you have any idea please tell me..

Thanks,
Bharath