Hi guys!

This time I address you not as a Liferay core developer but as a community member.

For the last year or so I have had a personal Wordpress blog and I like the tool very much. Apart from that, last week I started a personal project for a non-profit which wanted to have their site built with Wordpress. After creating the web site, I thought "It would be great to be able to import all these pages and content into Liferay". I thought it would be quite cool if I could import all my blog entries, tags and categories into a Liferay portal so I started working in a new portlet that I want to contribute back to the community.

I have decided to call it the "Worpress Importer Portlet for Liferay"

Here you have a list of features it already has in version 1.0:

- Ability to import tags and categories
- Ability to import blog entries
- Add portlet to the control panel in the web content section
- Ability to import pages and replicate page structure from a Wordpress site
- Ability to import web content asociated to wordpress pages

I have created versions for 6.0 CE and 6.0 EE (I have tested it in 6.0.5, 6.0.6 and 6.0.11) and you can download them here: Wordpress Importer Portlet


To use it, after succesful installation, click on "Wordpress importer" in control panel and upload your XML wordpress file of your blog/site.. that's it!

It is licensed with the LGPL license, so you can access the code and improve it as you want.

Remember that this is a beta version and there are many, many things to improve... and that's what I need you for! I'd like those of you interested in this to write me back and tell me ways you'd like the portlet to be improved or just tell me if you like it or not.

So far, I have thought about a few improvements:
- Import user comments
- Import appeareance and formatting of the text
- Import trackbacks
- Import post translations

But I'm willing to hear from you
Hope you like it!
Regards,
Juan Fernández

Sounds great we have loads of wordpress sites here so im going to give this a bash today!

Thanks
William

Nice, William! Don't hesitate in telling me if you find any bugs and your impressions
Thanks!

First off thanks for creating this. I was actually in need of something like this for something i am working on right now. I gave your portlet a try but ran into a fair amount of issues

1. The title element of an item can be empty. Your code throws a Nullpointer exception when this is the case
2. Wordpress sometimes exports it's data as invalid xml. In my case a user posted the following sequence in a comment :
ÿÿÿÿ this caused the DocumentBuilder to fail.
3. Community and guest permissions are not applied to the created posts
4. The publication date from the export is not used so posts are in the wrong order
5. Paragraphs are not copied correctly

I fixed most of the issues on my machine. Is there any chance of you moving this code over to something like github to make it easy to submit patches ?

Hi Jelmer:
thanks for the feedback!

I've committed the code to sourceforge's subversion, so that you can check it out (svn co https://wordpressimport.svn.sourceforge.net/svnroot/wordpressimport/) , create a patch with your improvements and then send it back to me (you can attach it here, in this post) so that I can fix it and we'll go on improving it.

Thanks a lot for your help!
Juan
ps: I love open source

Hey Juan, i've attached 4 patches for you to review

1. fixes a nullpointer exception when the title element is empty
2. makes the invalid xml that wordpress generates readable
3. adds guest and community permissions to the imported blogs
4. uses the element's pubDate element as the display date

I didn't have time to solve the problem with the paragraphs yet. When I do i'll let you know

Attachments:

patch01.patch (0.7k), patch02.patch (3.3k), patch03.patch (0.6k), patch04.patch (1.3k)

Soooo cool! Thanks a lot Jelmer!

About new features for the portlet that we've been discussing, here's a list of features that we are planning to implement and others could be interesting in:

1. The ability to import only blogs or pages, based on the portlet configuration
2. The ability to import only blogs that are tagged with specific tags
3. The ability to automatically approve blogs in the case workflow is configured for blogs

Do you guys have more features you would like to have/contribute? Feedback!

Regards,
Juan

I don't think it works agains trunk. (I used the portlet for 6.0.11):

INFO: Deploying web application directory WordpressImporter-portlet
23:55:45,784 INFO [PluginPackageUtil:1081] Reading plugin package for WordpressImporter-portlet
23:55:46,094 INFO [PortletHotDeployListener:250] Registering portlets for WordpressImporter-portlet
23:55:46,753 INFO [PortletHotDeployListener:409] 1 portlet for WordpressImporter-portlet is available for use
23:55:55,162 ERROR [JDBCExceptionReporter:234] incompatible data type in conversion
23:55:55,221 ERROR [IncludeTag:403] Current URL /user/joebloggs/home generates exception: An exception occurred processing JSP page /html/portlet/sites_admin/view.jsp at line 104

101: groupParams.put("active", Boolean.TRUE);
102: }
103:
104: int total = GroupLocalServiceUtil.searchCount(company.getCompanyId(), classNameIds, searchTerms.getName(), searchTerms.getDescription(), groupParams);
105:
106: searchContainer.setTotal(total);
107:


Stacktrace:
23:55:55,232 ERROR [IncludeTag:154] com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.dao.orm.ORMException: org.hibernate.exception.SQLGrammarException: could not execute query
at com.liferay.portal.service.persistence.GroupFinderImpl.countByC_C_N_D(GroupFinderImpl.java:216)
at com.liferay.portal.service.impl.GroupLocalServiceImpl.searchCount(GroupLocalServiceImpl.java:913)
....................
Caused by: com.liferay.portal.kernel.dao.orm.ORMException: org.hibernate.exception.SQLGrammarException: could not execute query
at com.liferay.portal.dao.orm.hibernate.ExceptionTranslator.translate(ExceptionTranslator.java:30)
at com.liferay.portal.dao.orm.hibernate.SQLQueryImpl.list(SQLQueryImpl.java:96)
at com.liferay.portal.dao.orm.hibernate.SQLQueryImpl.list(SQLQueryImpl.java:80)
...................
Caused by: org.hibernate.exception.SQLGrammarException: could not execute query
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:92)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
... 223 more
Caused by: java.sql.SQLException: incompatible data type in conversion
at org.hsqldb.jdbc.Util.throwError(Util.java:81)
at org.hsqldb.jdbc.JDBCPreparedStatement.setParameter(JDBCPreparedStatement.java:4268)

23:56:03,215 INFO [PluginPackageUtil:1386] Checking for available updates
23:56:03,216 INFO [PluginPackageUtil:1430] Finished checking for available updates in 1 ms

Hi Aaron!

You're right: I'm developing the wordpress importer for Liferay 6.0 (not 6.1), so to test it, you'll have to download it and deploy it on a 6.0.6 (CE) or a 6.0.11(EE)

Trunk is constantly changing, so it is hard to deliver a plugin as you don't know if it will work the following week (these core engineers always changing the APIs... ).
So I'll wait for 6.1 CE release (it'll be out in a month or so) to update the portlet.

6.1 will allow us to import comments from wordpress (as we can soft-create users as guest members just importing name and email address).
Anyway, if for any project constraints you're using trunk, the update wouldn't be that costly: just add a few parameters to some methods or change method names. If you do it, ping me in case you need some help.

Thanks for the feedback!
Juan

Hey Juan, this afternoon I spent some time implementing filtering blog posts on tags. In case you are interested I attached the patch, it should apply cleanly on the latest svn version.

I haven't tested it too thoroughly but so far but I think it should be solid

Attachments:

patch05.patch (10.6k)

And one more patch

I added the ability to map wordpress users to liferay users. That way the blog posts will be imported under the corrrect username.

Unfortunately i didnt have time to create a patch that applies cleanly on the trunk, it will however apply cleanly if you apply patch05 on the trunk, and then apply this

Attachments:

patch06.patch (8.7k)

Awesome, Jelmer:

I'll try those changes this weekend and I'll get to you.
If you've done the tag filtering feature, extending it to categories wouldn't be that hard, so I'll try to improve it so that we can filter by tags and categories.
About the users thing, it's a great approach I didn't think of (I was thinking in creating anonymous user accounts, but for that we'll have to wait for 6.1) so this is quite cool!

will it be able to import via rss, or only xml ?

i need to import 5 years worth of data/forums with about 300k users

Hi Steve!

For the moment, it's importing only the xml file, but after analyzing the rss generated by wordpress, it is aprox. the same xml that I was parsing, so using the RSS utils from Liferay and knowing the xml is quite the same, it should be easy to implement. This is an interesting feature proposal, as it will allow us to import Wordpress sites without needing to export its content to an xml file... cool!

I foresee some problems:
user imports - if there's no user in liferay with that email address: will you like to create a new account with that email address?
forums imports - we'll have to develop module to convert wordpress forums to liferay's (which is the wp module you're using for the forums?)

Will you help us implement this features? I think we can do a really nice plugin if we all participate with ideas and code contributions

Thanks for the feedback, Steve!

Hi guys!
I've just released Wordpress Importer Portlet 1.1!

You can download it here (or directly on sourceforge).

Improvements and new features for Wordpress Importer 1.1:
- Now you can choose what to import (pages, blog entries, categories, tags...) using the configuration panel
- Ability to define the imported category vocabulary name
- Ability to bypass the workflow system
- Respect the text paragraphs separation
- Creation dates are imported
- Community and guest permissions are granted to the imported entries
- Several bug fixes
- Tags filtering on import time
- User account checks: if the user exists in Liferay, all his comments/entries are created with his account.


A huge thanks to Jelmer, who has developed and contributed several cool features and to those of you that gave feedback.

EE version coming soon too

Keep the feedback coming, guys!
Regards,
Juan

any chance in extending this to allow me to import via rss export ?

Hi Steve:
I'm a bit overloaded with tasks for the next release (6.1), so I won't be able to develop this in the short-term.
If somebody wants to implement this feature (import Worpdress content via RSS feeds), please ping me to look for the best way to implement it.
I'll update this thread if I implement this in the future
Regards

Any status on comment imports, or should we wait until 6.1 is out. If so, any idea when 6.1 will be released? You mentioned to me that comments that cannot be mapped to users will be added as "Test". However, I renamed the "Test" user to be my own account. Could that be an issue?

--Aaron

Hi Aaron:
Comments are imported and they use a mapping rule: in case there exist a user with the same email address, it'll be used, else it'll be created with the user that is using the portlet (the administrator, I guess). This is available for Liferay 6.0.6 in Wordpress Importer v1.1

6.1 will be released very soon (at least the first beta version). Stay tunned
Regards!
Juan

It did not import for me. I will try again. Do you know of an easy way to delete all blog entries, so I can re-import? I don't want duplicates.

I must be doing something wrong. I have done this multiple times. I'm using CE 6.0.6, with your portlet importer 1.1. The blog entries get imported great, but no comments are there. I do not see anything in the logs except for Pings from Google. Is there more I can do to troubleshoot/debug for you?

Here is a quick post I made to make it easy to redirect Wordpress links to LIferay. Though, still struggling with comments:
http://teneo.wordpress.com/2011/06/17/redirecting-wordpress-to-liferay-blogs/

Hi Aaron. Ouch - you're right.

We implemented user mappings based on email address for blog entries and (memory failure here :$) I thought we applied this for comments too. So this is not implemented yet. Sorry for any inconveniences I could cause you.

This approach (if a user with the same email address exists in Liferay, create a commment with its account, else with the admin's account) should be quite easy to implement, as all the components are in place. If this is ok, I can have a look at it and release a new version next week with this feature. Would that approach be ok for your pourposes?

The better approach (creating user accounts on the fly so that if the user has comments and in the future he creates an account, all the comments will be linked to his account) will have to wait for verision 6.1, as we don't have the needed infrastructure yet.

Sorry again and tell me if the first approach if ok for you to have a look at it
Thanks for the feedback!

Hey guys, I have published a new version of the Worpdress Importer for Liferay 6.1.
More info here: http://www.liferay.com/community/forums/-/message_boards/message/12363218
Feedback, feedback!
Enjoy!
Juan Fernández

Hi guys!

I think i may arrive once the battle is over, but i'm currently working with the WordPress Importer portlet and I am experiencing some issues. Is there any easy way to make the syntax coloration still available once the articles are imported? The WordPress content I have to manage is full of articles dealing with Java, C++ and so on..

It would be great if this "colored" presentation could appear in our Liferay portal.

I have found my own way to do it but I would first like to know if you had another, easier way to handle it (mine might appear to be quite 'heavy').

Thanks a lot!

Gaël

Hi Gael!
First of all, welcome to the community, I've seen this is your first post and I'm glad it is to share ideas.

I'm sorry, but I didn't have text "colouring" into account. What I propose is that you share here your approach to solve this problem and we can discuss about the better way to implement it, ok?

First of all: what's your generated xml looks like?
Second: how are you implementing the colouring and why do you say it's "heavy"?

Thanks,
Juan Fernández

Hi Juan!

I'm happily surprised to see how fast you have answered! Thanks a lot for paying attention to my issue and also thanks for the welcoming!

First I need to mention that i am using tiny_mce as an editor, with the syntaxhl plugin for syntax highlightning. This solution was very satisfying when we tried to create new blogs on our portal. The syntaxhl tags generated look like this :



for the Java language, for example.

And as you already know, the code we may find in an xml file generated by a WordPress export looks like this :



I could have transformed the xml file with regex, but the problem is also that the WordPress Importer, during the import, generates <p></p> tags in the code parts (which are not effective : the visitor can see them and no "\n" is operated). Thus I found no other solution but to transform the code once imported. (which shall not be harder).

My program does the following :
- it connects to the lportal database, looks for the blogsentry table
- it gets all the articles the table may contain
- it converts all the tags using some regular expressions
- it removes the <p> tags and turns the </p> tags into "\n"
- and finaly, it persists the articles back.

The program should be easily understandable, but i said it was "heavy" because all after these spent efforts I just asked myself "is there no Liferay editor able to understand those tags?". And if such an editor does exist, my program is useless.

If you think my solution is viable and relevant, I would be happy to share it.

Thanks for helping!

Gaël

Ouch, Gaël... I've just seen this post.. three months late... I'm really sorry
Did you implement this improvement for your project? How was it? Was viable? Difficult?
I'm looking forward to hear your feedback and experiences,
Thanks,
Juan