Migration to 6.1 fails while verifying document library

[Willem Vermeer, módosítva 12 év-val korábban]

We're upgrading from 6.0.5 CE to 6.1.1. In the 6.1.1 (=master) codebase the problem was fixed where the upgrade process would abort when a document (image) could not be found. However, further on in the process something similar still crashes the upgrade. In VerifyDocumentLibrary.checkMimeTypes() when one of the dlFileEntries cannot be found the upgrade aborts with a stacktrace. This is quite problematic because in our case the upgrade runs for 5 hours before reaching this error and then we have to start all over.

Is this a known problem/bug and can something be done about it for the 20 April release?

Stacktrace:
17:13:27,300 INFO [VerifyProcess:76] Verifying com.liferay.portal.verify.VerifyDocumentLibrary
17:13:47,701 ERROR [MainServlet:198] com.liferay.portal.kernel.events.ActionException: com.liferay.portal.verify.VerifyException: com.liferay.portal.verify.VerifyException: com.liferay.portlet.documentlibrary.NoSuchFileException: /localhome/wvermeer/upgradegitmaster/data/document_library/10131/2777753/26203/1.1
com.liferay.portal.kernel.events.ActionException: com.liferay.portal.verify.VerifyException: com.liferay.portal.verify.VerifyException: com.liferay.portlet.documentlibrary.NoSuchFileException: /localhome/wvermeer/upgradegitmaster/data/document_library/10131/2777753/26203/1.1
at com.liferay.portal.events.StartupAction.run(StartupAction.java:58)
at com.liferay.portal.servlet.MainServlet.processStartupEvents(MainServlet.java:1219)
at com.liferay.portal.servlet.MainServlet.init(MainServlet.java:195)
at javax.servlet.GenericServlet.init(GenericServlet.java:160)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1266)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1185)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1080)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5015)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5302)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:897)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:649)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1581)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.liferay.portal.verify.VerifyException: com.liferay.portal.verify.VerifyException: com.liferay.portlet.documentlibrary.NoSuchFileException: /localhome/wvermeer/upgradegitmaster/data/document_library/10131/2777753/26203/1.1
at com.liferay.portal.verify.VerifyProcess.verify(VerifyProcess.java:82)
at com.liferay.portal.verify.VerifyProcessUtil._verifyProcess(VerifyProcessUtil.java:117)
at com.liferay.portal.verify.VerifyProcessUtil._verifyProcess(VerifyProcessUtil.java:78)
at com.liferay.portal.verify.VerifyProcessUtil.verifyProcess(VerifyProcessUtil.java:46)
at com.liferay.portal.events.StartupHelper.verifyProcess(StartupHelper.java:137)
at com.liferay.portal.events.StartupHelperUtil.verifyProcess(StartupHelperUtil.java:52)
at com.liferay.portal.tools.DBUpgrader.verify(DBUpgrader.java:201)
at com.liferay.portal.events.StartupAction.doRun(StartupAction.java:182)
at com.liferay.portal.events.StartupAction.run(StartupAction.java:52)
... 20 more
Caused by: com.liferay.portal.verify.VerifyException: com.liferay.portlet.documentlibrary.NoSuchFileException: /localhome/wvermeer/upgradegitmaster/data/document_library/10131/2777753/26203/1.1
at com.liferay.portal.verify.VerifyProcess.verify(VerifyProcess.java:82)
at com.liferay.portal.verify.VerifyProcess.verify(VerifyProcess.java:87)
at com.liferay.portal.verify.VerifyProcessSuite.doVerify(VerifyProcessSuite.java:40)
at com.liferay.portal.verify.VerifyProcess.verify(VerifyProcess.java:79)
... 28 more
Caused by: com.liferay.portlet.documentlibrary.NoSuchFileException: /localhome/wvermeer/upgradegitmaster/data/document_library/10131/2777753/26203/1.1
at com.liferay.portlet.documentlibrary.store.FileSystemStore.getFileAsStream(FileSystemStore.java:215)
at com.liferay.portlet.documentlibrary.store.StoreProxyImpl.getFileAsStream(StoreProxyImpl.java:163)
at com.liferay.portlet.documentlibrary.store.SafeFileNameStoreWrapper.getFileAsStream(SafeFileNameStoreWrapper.java:249)
at com.liferay.portlet.documentlibrary.store.DLStoreImpl.getFileAsStream(DLStoreImpl.java:260)
at com.liferay.portlet.documentlibrary.store.DLStoreUtil.getFileAsStream(DLStoreUtil.java:425)
at com.liferay.portlet.documentlibrary.service.impl.DLFileEntryLocalServiceImpl.getFileAsStream(DLFileEntryLocalServiceImpl.java:656)
at com.liferay.portlet.documentlibrary.service.impl.DLFileEntryLocalServiceImpl.getFileAsStream(DLFileEntryLocalServiceImpl.java:639)
at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:112)
at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:71)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108)
at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:211)
at $Proxy181.getFileAsStream(Unknown Source)
at com.liferay.portlet.documentlibrary.service.DLFileEntryLocalServiceUtil.getFileAsStream(DLFileEntryLocalServiceUtil.java:461)
at com.liferay.portal.verify.VerifyDocumentLibrary.checkMimeTypes(VerifyDocumentLibrary.java:135)
at com.liferay.portal.verify.VerifyDocumentLibrary.doVerify(VerifyDocumentLibrary.java:230)
at com.liferay.portal.verify.VerifyProcess.verify(VerifyProcess.java:79)
... 31 more
Caused by: java.io.FileNotFoundException: /localhome/wvermeer/upgradegitmaster/data/document_library/10131/2777753/26203/1.1 (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at com.liferay.portlet.documentlibrary.store.FileSystemStore.getFileAsStream(FileSystemStore.java:212)
... 49 more
Stopping the server due to unexpected startup errors

[Kristjan Purre, módosítva 11 év-val korábban]

any leads yet ? we have same problem.

[Jitendra Rajput, módosítva 11 év-val korábban]

Are you using same hook for document library in 6.1.1 ? It should be same as you have used in 6.0.5.

[Kristjan Purre, módosítva 11 év-val korábban]

Thnx for reply ,

yes we are using right hook ,... and got around that problem allready by creating missing files in doc library

but , now as usual .. now we have next problem ..

17:30:41,027 INFO [pool-2-thread-1][VerifyProcess:81] Verifying com.liferay.portal.verify.VerifyGroup
17:30:41,783 INFO [pool-2-thread-1][VerifyProcess:81] Verifying com.liferay.portal.verify.VerifyJournal
17:32:46,341 WARN [pool-2-thread-1][VerifyJournal:127] Unable to update asset for article 1047168: No User exists with the primary key 708914
17:32:46,353 WARN [pool-2-thread-1][VerifyJournal:127] Unable to update asset for article 1047147: No User exists with the primary key 708914

Verify Journal freezed .. 60h+ .. on db side was Commit still running


is there a way to disable Verify in upgrade process ? ..

[Daniel Tyger, módosítva 10 év-val korábban]

Kristjan Purre:

is there a way to disable Verify in upgrade process ? ..

I realllllly want to know how to prevent VerifyProcesses as well - I have implemented 6 different workarounds to get close to the end of a 5.2.3 upgrade > 6.06, but keep dying after many hours of waiting for it to finish... 1000+ roles, 1000+ groups, 1500 docs, 1600 images, et etc...

I have used these to no avail - the upgrade process does not obey these props:
#verify.processes=com.liferay.portal.verify.VerifyProcessSuite
#verify.frequency=0

Anyone?

And, afterwards, I want the db to reflect a fully-upgraded db

I have hacked release_ and changed 5203 > 6006 but when I try to go further later (6.1.1 or 6.2) (after inexing and converting check algorithm 5 > 6) I get further failures that don't like the state of my release...

exhausted from expending so much effort on hacking around these processes...

daniel

[Sabrina Del Prete, módosítva 10 év-val korábban]

I found that the only way to disable Verify process during Upgrade process is to override the default property
"verify.processes=com.liferay.portal.verify.VerifyProcessSuite" with a class that does not exist.

I updated version 6.0.5 to 6.1.2 GA3 and it worked for me.
I hope it helps.

[Victor Zorin, módosítva 10 év-val korábban]

...with a class that does not exist...

Still it is better to provide a verification class, especially that fixes migration issues.
This is the download link to a verifier JAR that we built for use in a number of migration projects from 5.1.0+ to 6.1.2 GA3:

http://downloads.smartforms.myoffice24x7.com/latest/document-verifier.jar

Put the following into portal-ext.properties:

verify.processes=com.myoffice24x7.liferay.portal.verify.VerifyProcessSuite
verify.dl.file.name.normalization.enabled=true
verify.dl.file.name.normalization.max.list.page.size=50
layout.import.dl.file.name.normalization.enabled=true

[Daniel Tyger, módosítva 10 év-val korábban]

Thank you, Victor, for sharing this example verify override and Sabrina for sharing the workaround!

[Laurent Saboret, módosítva 10 év-val korábban]

Dear Victor,

I bumped into the same verification problem while upgrading to Liferay 6.1.2.

The URL http://downloads.smartforms.myoffice24x7.com/latest/document-verifier.jar does not work anymore. Would you be kind enough to re-post this file?

Best regards,
Laurent

[Lisa Simpson, módosítva 10 év-val korábban]

I've been documenting in this thread all of my upgrade issues. I notice that the thread is notoriously silent and that NO ONE from Liferay is commenting. I've managed to create some work around kludges for somethings but others do not. I have found, even in the earlier versions that upgrading the Document Library was where things all fell apart. Mostly it seemed to be looking for files that don't exist. *sigh*

[Laurent Saboret, módosítva 10 év-val korábban]

Dear Victor,

I had a problem on my computer. The URL http://downloads.smartforms.myoffice24x7.com/latest/document-verifier.jar works fine.

Best regards,
Laurent

[Laurent Saboret, módosítva 10 év-val korábban]

Victor Zorin:

...with a class that does not exist...

Still it is better to provide a verification class, especially that fixes migration issues.
This is the download link to a verifier JAR that we built for use in a number of migration projects from 5.1.0+ to 6.1.2 GA3:

http://downloads.smartforms.myoffice24x7.com/latest/document-verifier.jar

Put the following into portal-ext.properties:

verify.processes=com.myoffice24x7.liferay.portal.verify.VerifyProcessSuite
verify.dl.file.name.normalization.enabled=true
verify.dl.file.name.normalization.max.list.page.size=50
layout.import.dl.file.name.normalization.enabled=true

Dear Victor,

Thank you a lot. Your document-verifier.jar fixed the permissions of my Liferay 6.1.2 CE ga3 installation without the Verify Document Library bug (LPS-35280). This fixed the Control Panel permissions bug (LPS-40291) :-)

Would you be kind enough to share your source code? This would give a chance to Liferay developers and/or IcePatchers.org contributors to integrate it in the portal source code.

Best regards,
Laurent

[Juan Gonzalez, módosítva 10 év-val korábban]

Hi Laurent,

seems this is the LPS: https://issues.liferay.com/browse/LPS-37322