Upgrade from 6.0.6 to 6.1 stops when file not found - bug?

[MICHAIL MOUDATSOS, modificado hace 12 años.]

Hello all,

after trying to upgrade from Liferay 6.0.6 CE to 6.1 CE the upgrade process ended abnormally. I restored my DB backup and retried and when I redirected the output I got the following error

...
Loading file:/C:/Liferay/liferay-portal-6.1.0-ce-ga1/portal-ext.properties
10:54:59,214 INFO  [DialectDetector:80] Determine dialect for Microsoft SQL Server 10
10:54:59,246 INFO  [DialectDetector:145] Found dialect com.liferay.portal.dao.orm.hibernate.SQLServer2008Dialect
Starting Liferay Portal Community Edition 6.1.0 CE (Paton / Build 6100 / January 6, 2012)
10:55:26,450 INFO  [BaseDB:406] Database supports case sensitive queries
10:55:26,690 INFO  [UpgradeProcess:205] Upgrading com.liferay.portal.upgrade.UpgradeProcess_6_1_0
10:55:26,692 INFO  [UpgradeProcess:205] Upgrading com.liferay.portal.upgrade.v6_1_0.UpgradeSchema
10:55:28,614 WARN  [BaseDB:382] The index 'IX_5204C37B' is dependent on column 'active_'.: alter table User_ drop column active_;

10:55:29,119 INFO  [UpgradeProcess:205] Upgrading com.liferay.portal.upgrade.v6_1_0.UpgradeUserName
10:55:29,227 INFO  [UpgradeProcess:205] Upgrading com.liferay.portal.upgrade.v6_1_0.UpgradeAdminPortlets
10:55:29,429 INFO  [UpgradeProcess:205] Upgrading com.liferay.portal.upgrade.v6_1_0.UpgradeBlogs
10:55:29,542 INFO  [UpgradeProcess:205] Upgrading com.liferay.portal.upgrade.v6_1_0.UpgradeCamelCasePortletPreferences
10:55:31,656 INFO  [UpgradeProcess:205] Upgrading com.liferay.portal.upgrade.v6_1_0.UpgradeCountry
10:55:31,788 INFO  [UpgradeProcess:205] Upgrading com.liferay.portal.upgrade.v6_1_0.UpgradeDocumentLibrary
10:55:31,837 INFO  [UpgradeProcess:205] Upgrading com.liferay.portal.upgrade.v6_1_0.UpgradeExpando
10:55:31,869 INFO  [UpgradeProcess:205] Upgrading com.liferay.portal.upgrade.v6_1_0.UpgradeGroup
10:55:31,927 INFO  [UpgradeProcess:205] Upgrading com.liferay.portal.upgrade.v6_1_0.UpgradeImageGallery
10:55:32,375 ERROR [MainServlet:198] com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.upgrade.UpgradeException: com.liferay.portal.kernel.upgrade.UpgradeException: com.liferay.portlet.documentlibrary.NoSuchFileException: C:\Liferay\liferay-portal-6.1.0-ce-ga1\data\images\10\42\10422.png
com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.upgrade.UpgradeException: com.liferay.portal.kernel.upgrade.UpgradeException: com.liferay.portlet.documentlibrary.NoSuchFileException: C:\Liferay\liferay-portal-6.1.0-ce-ga1\data\images\10\42\10422.png
	at com.liferay.portal.events.StartupAction.run(StartupAction.java:58)
...

This implies that a missing file from the data\images folder would break the whole upgrade process! Should this work this way or is it a bug to be issued? Note that the error appeared even after I copied all the data\images file from my 6.0.6 installation to the 6.1 installation - there was no such file in the former either! (Later I found one such file n an older 6.0.5 installation but I don't think I had upgraded from that one to 6.0.6 - in which case its a coincidence- but I ve done various imports and exports of lar files corresponding to web content displays, for which I ve lost track. Maybe those have played a role in this scenario)

So the question remains. Is this a bug to be issued? Or am I doing something wrong?

EDIT: note that if i dont reinitialize the db other sql related problems arise such as duplicate or missing columns but i assume they are a consequence of the first incomplete initialization attempt

[Ricardo Funke Ormieres, modificado hace 12 años.]

Well, I don't know if this is a bug, but I would try to put empty files with the same name just to complete the upgrade process, if they were not too many files

[MICHAIL MOUDATSOS, modificado hace 12 años.]

I actually found those files from earlier installations back to 6.0.5!!! I found out from other posts that image.hook.impl=com.liferay.portal.image.FileSystemHook setting probably plays a role as well. The thing is that 1) even when I did this other errors came up (duplicate file error for a folder and file that do not even exist on disk) and 2) The error I describe should not really terminate the upgrade process. Upgrade should go on even if it couldn't fimd some files that are not operation-critical...

I am thinking of issuing a bug but before I make someone start working on this, I must be sure that I'm not doing something wrong.

[Eike Salow, modificado hace 12 años.]

Hi Michail,
ervertime you get a fail while processsing an update make sure you set your database back.
make sure you make a clear drop database and a clear create database, do NOT doing somthink like : mysql -uxxx -pxxx mydabase < myold606dump.sql
as liferay 6.1. creates a bunch of new tables, which will not beeing droped by your myold606dump.sql

Following tables are created while upgrade and differs from your old DB
if your database is already messed up try to drop following tables if exist:

DDLRecord;
DDLRecordSet;
DDLRecordVersion;
DDMContent;
DDMStorageLink;
DDMStructure;
DDMStructureLink;
DDMTemplate;
DLContent;
DLFileEntryMetadata;
DLFileEntryType;
DLFileEntryTypes_DDMStructures;
DLFileEntryTypes_DLFolders;
DLSync;
LayoutBranch;
LayoutRevision;
LayoutSetBranch;
MBThreadFlag;
MDRAction;
MDRRule;
MDRRuleGroup;
MDRRuleGroupInstance;
PortalPreferences;
Repository;
RepositoryEntry;
ResourceBlock;
ResourceBlockPermission;
ResourceTypePermission;
SocialActivityAchievement;
SocialActivityCounter;
SocialActivityLimit;
SocialActivitySetting;
UserGroups_Teams;
UserNotificationEvent;
VirtualHost;
QUARTZ_SIMPROP_TRIGGERS;


hope this helps

Eike

[MICHAIL MOUDATSOS, modificado hace 12 años.]

@Eike Salow

Eike Salow:

make sure you make a clear drop database and a clear create database

If by that you mean a new clean install of lportal db using the provided sql scripts, then probably you misunderstood me. I want an upgrade. If not, I have to add that I use a MS SQL Server 2008. I backed up my 6.0.6 installations db to a full copy-only backup. Then I used this backup to make a new db on which to test the upgrade process (In order to have my 6.0.6 installation operational any time I wanted to). So after I extracted the Liferay 6.1 bundle and set up my portal-ext.properties correctly, I first got the error of files not found. This is unacceptable in my opinion; the process should find a way to go on. In the worst case missing resources should just be ... missing from site! And that's what this post was originally about.

Furthermore, even when I found out the missing files from the data/images folder, restored db once more to its pre-upgrade state, re-extracted the Liferay 6.1 bundle (I tried this particular step today), and HAVING SET image.hook.impl=com.liferay.portal.image.FileSystemHook, which was the default choice in 6.0.6 and the one I used, I got that duplicate file exception concerning a thumbnail or something. And once more the upgrade process ended abnoramlly (interrupted).

I don't suppose you find this OK, do you guys?

@Juan Gonzalez P
I checked the guide (thanks again, I was expecting that guide to come out) but it doesn't really say anything more than I already know.

[Eike Salow, modificado hace 12 años.]

Hi Michail,
what I ment was:
once your databse upgrade was interrupted you have make a drop and create database. And then import your your 606 db data once again into into the fresh database you created. Do not import the 606 DB again on a DB of a failed upgrade.


regards
Eike

Ps. Are you sure you do not have inconistent data in liferay 6.06. Try to open the image / doc lib files within your Liferay 6.06 version, you facing trouble while upgradingto 6.1.

[MICHAIL MOUDATSOS, modificado hace 12 años.]

OK for the DB part, got it now, though as I said I use MS SQL Server.

Concerning the consistency of data I guess you re right, because the installation I've tried my upgrade on, has undergone many .lar file imports, many of which did not succeed and I don't know what garbage all these attempts may have left.

The fact that enforces the above speculation is that I tried an upgrade on a new installation of my application from 6.0.6 and it partially worked. Partially in the sense that I had some problems with my themes. Although I migrated all my projects to 6.1-plugins-SDK, my themes were not installed properly. I ve found some posts on forums on the subject and I ll study them a bit. Till then, the only thing I have to say, once more, is that the upgrade operation SHOULD NOT BRAKE if there's an inconsistency on some resources. Just point to no resource and do not show. I know this is over-simplistic, development-wise, but this should be the intended behavior, whatsoever.

Thank you for your time

[Trey Howard, modificado hace 12 años.]

Hi All

I'm getting the same error while attempting to upgrade from 6.0.5 to 6.1.
We have been using the default FileSystemHook in 6.0.
I agree completely with Michail that the upgrade process should continue on migration of a failed resource.

It would not surprise me if our image gallery was corrupted, since our image gallery is empty but the database has 11 records and the file system has 195 files across all the images sub-directories.
But our 6.0 install works, with or without a corrupt image gallery (if that's the problem).

We'd *really* like to upgrade to 6.1 but are stopped cold because of some silly images.
Honestly, we don't have that many graphics (<100) and be open to doing some manual steps post-upgrade to repair anything that did not successfully upgrade.

I have seen a lot of activity both in the forums and JIRA about upgrade failures related to the image gallery / document library, so it sounds like whatever issues--corrupt environment or whatever--contributing to this problem are somewhat widespread.
6.1 was only recently released and so I expect not too many companies have tried to upgrade (yet).
I have an itchy feeling that this is a systemic problem that will affect any future upgrade attempts.

Please let me know if there's any more detail about my LR installation that I can provide to help debug this problem.

-Trey

[MICHAIL MOUDATSOS, modificado hace 12 años.]

Trey Howard:

I'm getting the same error while attempting to upgrade from 6.0.5 to 6.1.

Are you really trying to upgrade from 6.0.5 to 6.1 or was that a typo error? Because I ve read somewhere on the community that it is advisable to upgrade from the latter version of a release (i.e. the latter version of the 6.0 release is 6.0.6). So in your case I would advise you to upgrade first from 6.0.5 to 6.0.6 and then from 6.0.6 to 6.1. Still, I would bet all my money (well not all but a lot!) that you would get the same error. I'm just pointing out the correct procedure. Note that I ve successfully and rather painlessly upgraded from 6.0.5 to 6.0.6 in the past and from my experience there's nothing to be afraid of concerning that transition. Moreover, I found those missing files from my older 6.0.5 installation and put them in data/images and then I got other errors... (concerning duplicate files this time, that btw were not at all on disk - a mess!)

If it was just a typo then np , but there has to be a way to go on...

HELP!

[Artur Linhart, modificado hace 12 años.]

I have encountered this problem too already during the upgrade part from 6.0.6 to 6.1.0.

I also feel this is a bug. I have created an issue for it and proposed some solutions of this problems. If you feel there can be other solutions, then please comment to this issue - and vote for it ;-)

http://issues.liferay.com/browse/LPS-25054

[Tom Wackerfuss, modificado hace 12 años.]

I have experience similar upgrade problems in going from 6.0.6 to 6.1.0,

• I set up a db with an bkup from 606
• I expand the LR610-Tomcat bundle and set up my portal-ext
• If I don't copy my 'data/images' files I get a conversion exception that it can't find the image files
• So I copy the image files, reset the db to the 606 bkup
• I get a verify exception

As a point of additional, information it seems that after I copy the image files into 'data/images' and run the conversion, there is a part of the conversion that takes the files from the 'data/images' folder, processes them and removes them from 'data/images'. I think they are being migrated into the 'data/document_library' folder. Because if I restart the migration (reset db, and copy images back into 'data/images', I get a duplicate file in the document_library folder. So I reset again (reset db, copy images, delete document_library) the conversion goes until the verify exception.

[David H Nebinger, modificado hace 12 años.]

Look, guys, this is not a bug.

Liferay uses both a database component and a filesystem component (unless you configure the doc lib and image lib to live in the database).

You cannot just grab a copy of only the database component and do the upgrade; you need to have both.

And this is not a bug in the upgrade process per se. If you started your 6.0.x environment pointed at just your copy of the the database component (w/o the filesystem component), all of these errors would also be reported if you navigated to a page that referred to the (now missing) filesystem components.

What you're actually seeing is Liferay validating that your environment is consistent - that entities defined in the database to exist actually do exist in the filesystem component.

When you look at the process in this light, I'm sure you would agree that the failures are both good and necessary to ensure that both the database and filesystem components are valid and consistent.

[Artur Linhart, modificado hace 12 años.]

I think it is at least a "lazy programmer" functionality... So, maybe not a bug, but in the case there are some files (for example 1000 of 100000 pictures) missing, it is a nightmare to perform such an upgrade. With every single image you have to restore the document library and database, generate the missing file and start it again.... Effectivelly impossible.

Correct would be to check what everything is missing and throw an exception with this COMPLETE information. Unfortunately, the missing files can occur also by previous errors in the portal. If you will be once in the situation you have to upgrade such a Liferay database, I think, you will very fast stop arguing if this behavior is or is not a bug ;-)))... So, I think the LPS-25054 still can be classified as a bug :-)

[David H Nebinger, modificado hace 12 años.]

The question is how did you delete the 1,000 out of 100,000 images anyway? If you did it in a Liferay standard way and Liferay didn't clean up the references in the doc library or something, then that is a bug.

But if you just swept through the filesystem clearing images you didn't think you needed any more, then you really dug your own hole then.

In either case, it still is an indication that the database is no longer consistent, and really shouldn't be trusted anymore. Bailing out w/ exception is the best option at that point.

I know that it's a tough position to be in, but I always try to look at the other side - you have an inconsistent database but don't know it and you've just started the upgrade process. Would you want to resolve your inconsistent database first, or would you like to try it after the upgrade when it becomes more difficult. I'd opt for the 'before' option, myself...

[Artur Linhart, modificado hace 12 años.]

The answer how - is probably caused by previous liferay bugs, but you do not know exactly and you have figured it out first during the upgrade.
So, the upgrade is then your "help tool" which proves your database is consistent or not and even if you want to make it consistent before upgrade and do not want to develop yourself some checking, in such case Liferay lets you without help. You cannot perform the upgrade 1000 times to figure out which 1000 images are gone...

The developers should always consider, if they throw an exception what breaks up the whole process, if the exception is really so impossible in the reality and should bring to the user more support in solution of such problems.

Also helpful would be the "consistency check" method...

[Juan Gonzalez P, modificado hace 12 años.]

MICHAIL MOUDATSOS:

I actually found those files from earlier installations back to 6.0.5!!! I found out from other posts that image.hook.impl=com.liferay.portal.image.FileSystemHook setting probably plays a role as well. The thing is that 1) even when I did this other errors came up (duplicate file error for a folder and file that do not even exist on disk) and 2) The error I describe should not really terminate the upgrade process. Upgrade should go on even if it couldn't fimd some files that are not operation-critical...

I am thinking of issuing a bug but before I make someone start working on this, I must be sure that I'm not doing something wrong.

Check these steps to upgrade:

http://www.liferay.com/es/documentation/liferay-portal/6.1/user-guide/-/ai/upgrading-lifer-5

[Trey Howard, modificado hace 12 años.]

@Michail

While I understand your position I would encourage you to see this from the perspective of community perception of ease-of-upgrade.
I was finally able to hack my way through several upgrade roadblocks (not this one but others similar).
That said, I'm a developer by trade and not all Liferay customers are.

From our perspective, our 6.0 instances are working with or without the corrupt image gallery so this is perceived exclusively as an upgrade issue.
This may not be the correct conclusion but at the end of the day I think being as defensive and helpful (exceptions, etc) will pay dividends with respect to increased Liferay adoption and decreased attrition.

As a side note, you may be interested to know that my boss was open to discussions of abandoning Liferay given the upgrade errors we encountered.
I don't mean to make this sound like a casual topic, but the TCO is increased by scenarios like this one.

-Trey

[David H Nebinger, modificado hace 12 años.]

Trey Howard:

As a side note, you may be interested to know that my boss was open to discussions of abandoning Liferay given the upgrade errors we encountered.
I don't mean to make this sound like a casual topic, but the TCO is increased by scenarios like this one.

Again the question is how did your database get out of sync w/ the filesystem?

If you only used Liferay to remove images, but Liferay didn't clean up after itself, then it is a bug.

If you do things outside of Liferay, then you're the source of your own pain.

@Trey specifically, there have always been problems in the upgrade process. I've been doing them since the 4.x days, and I can't remember a single one that went as smoothly as what happens in the LR lab. I think there will continue to be problems into the future because we often forget the kind of changes that we affect on our local environment through the months/years that we have between major upgrades. But it's because of the customizations that we can do that is such the appealing nature of Liferay.

You're using 6.x CE. That means that your TCO for Liferay increased from $0.00 to whatever your hourly rate is by the hours it took you to resolve. I think if you were to show the boss the difference in that amount vs one of the big boys w/ a limited feature set to Liferay, he'd get a different perspective.

[Artur Linhart, modificado hace 12 años.]

TCO if Liferay CE is never 0.... Like also the definition of TCO says...

So, here is the question, what comes in mind of every boss with enough money - should I go the way which brings me more or less problems? In the US the managers are maybe paid for "good results", but in Europe they are often paid for "not making mistakes". So, classic "save-my-ass" mentality of the "old kind" bosses is - if I take IBM and it goes wrong, then I made no bad decision, because it is IBM. But if I take Liferay and it goes wrong, then everybody says I made a wrong decision - "why did you take Liferay and not the proven one"? It does not matter if it is really so, it is just the argumentation and partially also very near to the reality - at least from the point of view of such a boss.

So, if Liferay would invest more time into the "making problems smaller" also during upgrade, it would be a great benefit... Unfortunately, I also have behind me the upgrades from 4.2 and I can say, every upgrade very near to a real nightmare.
And this makes really a TCO much bigger and brings the people to the question, if they should not rather use IBM or Microsoft Sharepoint for their purposes... And there comes always somebody who says it could be much better than todays solution where you cannot be sure.
The problem is not only the costs, the other problem is the feeling of "unsureness" what the problems bring.

You have also no chance to verify the DB consistency, you have no documentation of the ER model... It is then really pretty difficult to avoid the problems, keeping your database clean, consistent, etc.
And also, my experiences are: the problems in the upgrades were always in our case caused by Liferay previous errors in the functionality, etc. itself, we had never some HW or system failure, what would cause some damages on the DB or file system. Every problems were produced by the product itself. :-(.

And, please, do not say to me I have to use some other product if Liferay is such bad product... Liferay is great product, but there should be put to the users more support in solving of the problems it had in the past and especially during the upgrades, the tests should be made much wider than they are today, especially with various types of data present or not present in the DB... I think., if Liferay would make the ask to the users to send them their databases and then would test the upgrades on their data, I think, the lot of problems would be solved before the product gos to the customers. It could be automatized, so every time before the new version will be releases, the upgrades could be started again to verify the expected upgrade results.

[Alexander Chow, modificado hace 12 años.]

And, please, do not say to me I have to use some other product if Liferay is such bad product... Liferay is great product, but there should be put to the users more support in solving of the problems it had in the past and especially during the upgrades, the tests should be made much wider than they are today, especially with various types of data present or not present in the DB... I think., if Liferay would make the ask to the users to send them their databases and then would test the upgrades on their data, I think, the lot of problems would be solved before the product gos to the customers. It could be automatized, so every time before the new version will be releases, the upgrades could be started again to verify the expected upgrade results.

Artur, thank you for your comments and, honestly, I am very sorry for the difficulties you are running into. Upgrading is perhaps the most difficult task for any enterprise to do and that is actually why we provide scripts to try to get you up and running as quickly as possible. Moreover, this specific upgrade script I had a major hand in and should have considered the possibility of data integrity problems.

In regards to your suggestions, we actually do test our upgrade scripts quite rigorously internally. But, as you suggest, real world data is the best. This is why we have solicited the help of our community base through the Bug Squad and 100 Papercuts and other mechanisms to ask the community to help us find and fix these problems. This is also why we try to have several betas and RC's before a GA is released… so that community members would actually download it and tell us if something is wrong. Obviously, this was not something that was tested thoroughly enough and we need to have a more systematic approach to this. However, your suggestion of asking the community to send us their databases (and in this case, their files as well) is something that very few community members would do. To be honest, it raises quite a number of security and privacy concerns which we don't have the manpower (or legal power) to manage.

In regards to your statement about verifying data integrity before an upgrade, you are correct… we should probably be doing more of that. We already do a lot of verification of data on startup… but that is after an upgrade.

OK, going to your specific problem, namely that some files seem to be missing… why don't you actually download the code and change the one line throwing the exception and swallow it. Then, log how many files are actually missing.

[Artur Linhart, modificado hace 12 años.]

Hello, Alex,

of course, there are ways how to proceed with the source code modifications... The problem I have mentioned consisted from more parts and the problem with the conversion of document library is not the worst problem, much more problematic is the problem in the role conversion (LPS-25086)... Of course, if my customer decides he wants to invest his money into the search of the upgrade errors in the source codes and into the fix of it, it can be done without problems... I have just made a reaction to David and his complains about it what is and should be reported as a bug or not :-).

According to the data - of course, the community members should anonymize the data before and Lifeay Inc. should have no responsibility for the data losses... It could be just the data of the partals, where it does not matter... |Possibly, Liferay could create a "anonymization" functionality, what could change the database before sent to Liferay to be anonymized... But it does not matter - I just wanted to point to the fact, there should be obviously more test cases and should be tested more widely during the upgrade tests and this was just a idea which came to me in mind.

Of course, for consultants like me it is theoretically not bad - they have more work during the solution of the problems of their customers, but I would be really more happier if my customer would not have to invest the money for the solution of such problems and support of Liferay instances would mean less TCO for the customers (even they are much lower then in the case of other commercial products) :-).

[David H Nebinger, modificado hace 12 años.]

Artur Linhart:

I have just made a reaction to David and his complains about it what is and should be reported as a bug or not :-).

Well, Artur, I wasn't trying to ruffle anyone's feathers in particular.

While there can be bugs in the system, there are also plenty of times when the upgrade process is not followed correctly. Many people do not realize that the default installation of Liferay uses both a database component and a filesystem component, and without the complete filesystem component the upgrade will not be successful.

For these folks, allowing the upgrade to continue does them a disservice, leading them to a false belief that their update was successful and they're good to go. Instead they come here and find similar posts about broken/failed upgrades along w/ responses indicating they need to sync up their filesystem stuff too.

As for the bugs in the system, they of course should be addressed but, in the end, the inconsistencies in the database and filesystem still need to be resolved.

It is really the only way to guarantee a consistent and valid Liferay environment.

[Jens Hamisch, modificado hace 10 años.]

Hi,

I do not want to jump on any TCO discussion, because IMHO this is a board to discuss technical issues ...
I'm having a comparable problem during upgrade from 6.1.0 CE to 6.1.2 CE.
The major difference may be, that I've converted the Jackrabbit datastore from local-filesystem to database some months ago. LR 6.1.0 CE is working properly on that.

However it seems, that either "dead" entries reside inside the LR 6.1.0 database or the mapping of UUIDs has become garbeled during migration to 6.1.2. I've already posted a very detailled technical description in
https://www.liferay.com/de/community/forums/-/message_boards/message/36090887

It'll help a lot if somebody could point me to some possibility to check the consistency of the databases before upgrade.

Thanks in advance,
Jens