Combination View Flat View Tree View
Threads [ Previous | Next ]
toggle
K R
liferay upgrade from 5.2 to 6.0 sp2
March 18, 2012 9:51 AM
Answer

K R

Rank: New Member

Posts: 5

Join Date: June 2, 2009

Recent Posts

We are upgrading our LR EE instance 5.2 sp4 to 6.0 sp2. Our environment is oracle 11g and tomcat 6 on redhat. We have about 60,000 users or so synched from ldap. Our upgrade script has been running for almost 4 days now without finishing. It has been stuck in the com.liferay.portal.verify.VerifyPermission stage. In our test and dev where we dont have all users synched it finished within 14 hours or so, not normal either. We are working with their support on what the issue might be but from what we hear it is considered normal to take this long given the number of users we have. I wanted to reach out to find out what experiences some others have had with this. No enterprise product we have worked with takes this long to upgrade. Poor programming and architecture seems to be the issues here. Since it is considered normal we were also surprised there were no mentions of this in any of their guides. It should probably start with that disclaimer. Anyway, other than waiting in frustration not much we are doing at this point. Thanks in advance for any info you may have based on your experience.
David H Nebinger
RE: liferay upgrade from 5.2 to 6.0 sp2
March 18, 2012 11:13 AM
Answer

David H Nebinger

Community Moderator

Rank: Liferay Legend

Posts: 8426

Join Date: September 1, 2006

Recent Posts

Well, if you look at the code you'd be able to understand why it takes so long...

I'm guessing your permission user check algorithm is still set to 5, so I'll cover that here...

Basically it gets all of the unique model names from the ResourceAction table.

For each unique name, it gets the list of action ids.

From there it calls PermissionLocalServiceImpl.checkPermissions(), passing in the model name and the list of action ids tied to the model.

In checkPermissions(), it finds all resources for the model name at the individual scoped level (this is probably what is killing you).

For each resource found, it iterates through each of the action ids finding a permission for the action id and resource id. If a permission is found, it goes on to the next one. If one is not found, checkPermission() is called with the resource and action id.

checkPermission() creates a new Permission instance (storing in db), optionally adds the new permission to the community and guest role permissions, and also adds the permission to the owner role. Finally it will also update the lucene index used to cache permissions (more db access).

So you should be able to do some estimation along here to identify how many records you're hitting as a result of this process. Count how many unique resource actions you have, count how many action ids you have, check how many resources you have, check how many resource codes you have (with scope value 4, individual). Multiply those together and that represents the max number of permissions that will need to be created.

From that max, you can subtract the number of rows from the permission_ table that have already been created.

That's going to determine how many new permission records will be added. Optionally you're also doing inserts into the community and guest role permissions, but you are inserting into the owner role and updating the lucene index on top of that.

That's just for algorithm 5, algorithm 6 is different but will require similar iterations...
K R
RE: liferay upgrade from 5.2 to 6.0 sp2
March 19, 2012 10:34 AM
Answer

K R

Rank: New Member

Posts: 5

Join Date: June 2, 2009

Recent Posts

Hi David,
Thanks for taking the time to explain how the process works. However, since we are an EE customer I expect Liferay to do that math for me and provide me with a script or something that does what you state above to tell me how long the upgrade will take. We have a fairly simple install and the upgrade finally finished after 4+ days. I cannot imagine what a nightmare it would have been if we were heavily utilizing the communities and other features. There was nothing very "enterprise" about this whole upgrade. The process was utilizing only one core out of all the 16 available cores in the server and instead of utilizing database scripts to do db heavy tasks everything is done at the app node. I guess that helps in supporting multiple platforms but doesn't benefit any enterprise environments utilizing powerful databases where this work could have been done.
Anyway, glad that the upgrade finished but overall highly disappointed with the process. Hopefully LR can get their act together and understand that this "acceptable" upgrade time frame of many days is not realistic in any sense.
David H Nebinger
RE: liferay upgrade from 5.2 to 6.0 sp2
March 19, 2012 12:52 PM
Answer

David H Nebinger

Community Moderator

Rank: Liferay Legend

Posts: 8426

Join Date: September 1, 2006

Recent Posts

Sorry you had an unfavorable upgrade situation, but honestly I can tell you that it is not always easy to estimate how long the upgrades will take... Different users of Liferay have their own situations, some with higher user counts and some with fewer, some using roles assigned directly to users and others leveraging the organization/community roles.

When you're dealing w/ database inserts, updates, and deletes it often doesn't matter how many cores you have. In this particular instance the statements need to run sequentially (to satisfy all of the various testing cases) and multiple cores don't offer any benefit.

Long and short of it is that you don't do major version upgrades of Liferay every month (or even every year), and the major version upgrades provide major new functionality and typically require extensive under the hood changes. Although this upgrade took awhile to complete, in the end it's the last one that you'll need to worry about for awhile.

I, too, would prefer some sort of progress indicator, at least something dumped out to the console window so, in cases like this, you know it's doing something for those 4 days and not left wondering if it's just stuck in a loop or something...
Iker Garcia
RE: liferay upgrade from 5.2 to 6.0 sp2
December 4, 2012 7:59 AM
Answer

Iker Garcia

Rank: New Member

Posts: 3

Join Date: November 28, 2011

Recent Posts

We are trying to upgrade a portal from Liferay 5.2.3 to 6.0.
At the moment the database upgrade process is 'stuck' at Verifying com.liferay.portal.verify.VerifyPermission

We are using the permission algorithm 5 and that is what we have indicated in the 6.0 portal-ext.property file before starting the server. permissions.user.check.algorithm=5

In our 5.2.3 database we have the following data:

RESOURCE_: 173,154 entries.
RESOURCECODE: 116 entries (Scope = 4)
RESOURCEACTION: 0 entries
RESOURCEPERMISSION: 0 entries

Does this mean that the upgrade process will have to insert 20,000,000+ new entries in PERMISSION_ table ?
At current rate (22 inserts / minute) this would take almost 2 years !!! Which is absolutely infeasible.


This is the hibernate logs we are getting (22 per minute):

Hibernate: insert into Permission_ (companyId, actionId, resourceId, permissionId) values (?, ?, ?, ?)
Hibernate: select permission0_.permissionId as permissi1_28_, permission0_.companyId as companyId28_, permission0_.actionId as actionId28_, permission0_.resourceId as resourceId28_ from Permission_ pe
rmission0_ where (permission0_.actionId=? )AND(permission0_.resourceId=? )
Hibernate: select permission0_.permissionId as permissi1_28_, permission0_.companyId as companyId28_, permission0_.actionId as actionId28_, permission0_.resourceId as resourceId28_ from Permission_ pe
rmission0_ where (permission0_.actionId=? )AND(permission0_.resourceId=? )
Hibernate: select permission0_.permissionId as permissi1_28_, permission0_.companyId as companyId28_, permission0_.actionId as actionId28_, permission0_.resourceId as resourceId28_ from Permission_ pe
rmission0_ where (permission0_.actionId=? )AND(permission0_.resourceId=? )
Hibernate: select permission0_.permissionId as permissi1_28_, permission0_.companyId as companyId28_, permission0_.actionId as actionId28_, permission0_.resourceId as resourceId28_ from Permission_ pe
rmission0_ where (permission0_.actionId=? )AND(permission0_.resourceId=? )
Hibernate: select permission0_.permissionId as permissi1_28_, permission0_.companyId as companyId28_, permission0_.actionId as actionId28_, permission0_.resourceId as resourceId28_ from Permission_ pe
rmission0_ where (permission0_.actionId=? )AND(permission0_.resourceId=? )
Hibernate: select permission0_.permissionId as permissi1_28_, permission0_.companyId as companyId28_, permission0_.actionId as actionId28_, permission0_.resourceId as resourceId28_ from Permission_ pe
rmission0_ where (permission0_.actionId=? )AND(permission0_.resourceId=? )
Hibernate: select permission0_.permissionId as permissi1_28_, permission0_.companyId as companyId28_, permission0_.actionId as actionId28_, permission0_.resourceId as resourceId28_ from Permission_ pe
rmission0_ where (permission0_.actionId=? )AND(permission0_.resourceId=? )
Hibernate: select permission0_.permissionId as permissi1_28_, permission0_.companyId as companyId28_, permission0_.actionId as actionId28_, permission0_.resourceId as resourceId28_ from Permission_ pe
rmission0_ where (permission0_.actionId=? )AND(permission0_.resourceId=? )
Hibernate: select roleimpl0_.roleId as roleId41_, roleimpl0_.companyId as companyId41_, roleimpl0_.classNameId as classNam3_41_, roleimpl0_.classPK as classPK41_, roleimpl0_.name as name41_, roleimpl0
_.title as title41_, roleimpl0_.description as descript7_41_, roleimpl0_.type_ as type8_41_, roleimpl0_.subtype as subtype41_ from Role_ roleimpl0_ where (roleimpl0_.name=? ) order by roleimpl0_.name
ASC

Regards
Iker Garcia
RE: liferay upgrade from 5.2 to 6.0 sp2
December 18, 2012 11:25 PM
Answer

Iker Garcia

Rank: New Member

Posts: 3

Join Date: November 28, 2011

Recent Posts

'com.liferay.portal.verify.VerifyPermission' process had been running for 10+ days and it didn't finish. (we had to stop the process due to some unrelated network issues)

Is there any workaround for this? Can we avoid launching this process and migrate 'permissions' in another way?

Regards
Daniel Tyger
RE: liferay upgrade from 5.2 to 6.0 sp2
June 11, 2013 7:34 PM
Answer

Daniel Tyger

Rank: Junior Member

Posts: 73

Join Date: February 5, 2013

Recent Posts

Iker,

I am experiencing the same issue going from 6.0.6 > 6.1.1 - hung on the VerifyPermission process for days...

Did you ever find a way to work around the VerifyPermission problem?

What did you decide to do?

Thanks for any response.

daniel
Alex Belt
RE: liferay upgrade from 5.2 to 6.0 sp2
June 12, 2013 11:03 AM
Answer

Alex Belt

Rank: Junior Member

Posts: 49

Join Date: October 9, 2012

Recent Posts

I wish I could get as far as the VerifyPermission stage - I keep getting hung up on the population of the AssetEntry and AssetTag tables with no idea why. I'm about to insert my own log statements into the process and rebuild the source so I can see better what's going on.
Daniel Tyger
RE: liferay upgrade from 5.2 to 6.0 sp2
June 12, 2013 11:35 AM
Answer

Daniel Tyger

Rank: Junior Member

Posts: 73

Join Date: February 5, 2013

Recent Posts

Alex Belt:
I wish I could get as far as the VerifyPermission stage - I keep getting hung up on the population of the AssetEntry and AssetTag tables with no idea why. I'm about to insert my own log statements into the process and rebuild the source so I can see better what's going on.


Alex - Are you trying to move from 6.0.6 > 6.1.1 or from an earlier version > 6.0.6?

I stripped down the system to just /webapps/ ROOT and minimalist portal-ext settings.

I had this issue: http://issues.liferay.com/browse/LPS-11981 which the tip from Privalov did help (5.2.3 > 6...)

and this helped me a lot: workaround from Nicolas Grue http://issues.liferay.com/browse/LPS-11543 (5.2.3 > 6...)

All along the way I have had to find workarounds and hacks to get upgraded, usually involving stopping the VerifyProcess, hacking the release_ table and applying the indexes myself afterwards... it has been a rough journey...
Alex Belt
RE: liferay upgrade from 5.2 to 6.0 sp2
June 12, 2013 11:55 AM
Answer

Alex Belt

Rank: Junior Member

Posts: 49

Join Date: October 9, 2012

Recent Posts

I'm trying to move from 5.2.3 > 6.0.6. I didn't get far enough along to encounter either of those problems. Something in the upgrade code is trying to insert more than one entry into the AssetEntry table using the same primary key. I don't know what the entry is, but it always happens at row 246 in the AssetEntry table, because there are always 245 rows in it when it dies. This isn't from a new index, this is from the table's existing index - made up of a single numeric column. What I'm trying to locate at the moment is, where is the data coming from that it's attempting to insert into the table?

Like you, I stripped the system down to just ROOT and tunnel-web, but it hasn't made a difference so far.
Daniel Tyger
RE: liferay upgrade from 5.2 to 6.0 sp2
June 12, 2013 11:59 AM
Answer

Daniel Tyger

Rank: Junior Member

Posts: 73

Join Date: February 5, 2013

Recent Posts

Can you post your stack trace error? I had some things like that but was able to track down the culprits and delete them from the db... like I had to actually create a role that it wanted and we had removed long ago and I had to remove a couple things as well... Sorry if not helpful so far...
Alex Belt
RE: liferay upgrade from 5.2 to 6.0 sp2
June 12, 2013 12:38 PM
Answer

Alex Belt

Rank: Junior Member

Posts: 49

Join Date: October 9, 2012

Recent Posts

I got farther this time. I surmised correctly that the issue was, in fact, that I didn't update the counter in the portal-data-counter.sql file before I built the portal from source. I have encountered another issue around the SocialActivity table.

19:31:08,274 ERROR [MainServlet:202] com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.upgrade.UpgradeException: com.liferay.portal.kernel.upgrade.UpgradeException: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Date
com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.upgrade.UpgradeException: com.liferay.portal.kernel.upgrade.UpgradeException: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Date
at com.liferay.portal.events.StartupAction.run(StartupAction.java:53)
at com.liferay.portal.servlet.MainServlet.processStartupEvents(MainServlet.java:1166)
at com.liferay.portal.servlet.MainServlet.init(MainServlet.java:199)
at javax.servlet.GenericServlet.init(GenericServlet.java:212)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1173)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:993)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4350)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4659)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:546)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:637)
at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:563)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:498)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:785)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:445)
at org.apache.catalina.core.StandardService.start(StandardService.java:519)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
at org.apache.catalina.startup.Catalina.start(Catalina.java:581)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: com.liferay.portal.kernel.upgrade.UpgradeException: com.liferay.portal.kernel.upgrade.UpgradeException: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Date
at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:114)
at com.liferay.portal.upgrade.UpgradeProcessUtil._upgradeProcess(UpgradeProcessUtil.java:80)
at com.liferay.portal.upgrade.UpgradeProcessUtil.upgradeProcess(UpgradeProcessUtil.java:37)
at com.liferay.portal.events.StartupHelper.upgradeProcess(StartupHelper.java:73)
at com.liferay.portal.events.StartupHelperUtil.upgradeProcess(StartupHelperUtil.java:40)
at com.liferay.portal.tools.DBUpgrader.upgrade(DBUpgrader.java:94)
at com.liferay.portal.events.StartupAction.doRun(StartupAction.java:117)
at com.liferay.portal.events.StartupAction.run(StartupAction.java:47)
... 29 more
Caused by: com.liferay.portal.kernel.upgrade.UpgradeException: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Date
at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:114)
at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:130)
at com.liferay.portal.upgrade.UpgradeProcess_6_0_0.doUpgrade(UpgradeProcess_6_0_0.java:56)
at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:111)
... 36 more
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Date
at com.liferay.portal.kernel.upgrade.util.DateUpgradeColumnImpl.getNewValue(DateUpgradeColumnImpl.java:31)
at com.liferay.portal.upgrade.util.DefaultUpgradeTableImpl.getExportedData(DefaultUpgradeTableImpl.java:64)
at com.liferay.portal.upgrade.util.Table.generateTempFile(Table.java:389)
at com.liferay.portal.upgrade.util.Table.generateTempFile(Table.java:347)
at com.liferay.portal.upgrade.util.BaseUpgradeTableImpl.updateTable(BaseUpgradeTableImpl.java:51)
at com.liferay.portal.upgrade.v6_0_0.UpgradeSocial.doUpgrade(UpgradeSocial.java:57)
at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:111)
... 39 more


Time to figure out where this is happening at.