Solr 1.3.0 and Liferay 5.1.2 problem

[Jérôme Schell, modificado hace 15 años.]

Hi all,

I'm in the process of evaluating a multi Liferay instances setup (with Liferay 5.1.2).
For this I have installed a Solr server (v1.3.0) that seems to work fine.

The problem is when installing the solr-web plugin in Liferay. I get these messages when the archive is deployed:

10:51:57,233 INFO  [AutoDeployDir:76] Processing solr-web-5.1.1.1.war
10:51:57,234 INFO  [WebAutoDeployListener:54] Copying web plugin for /var/lib/tomcat/liferay/repositories/liferay-inst1/liferay/deploy/solr-web-5.1.1.1.war
10:51:57,237 INFO  [BaseDeployer:529] Deploying solr-web-5.1.1.1.war
  Expanding: /var/lib/tomcat/liferay/repositories/liferay-inst1/liferay/deploy/solr-web-5.1.1.1.war into /opt/tomcat-instances/liferay-inst1/temp/20081030105157240
  Copying 1 file to /opt/tomcat-instances/liferay-inst1/temp/20081030105157240/WEB-INF/classes
  Copying 1 file to /opt/tomcat-instances/liferay-inst1/temp/20081030105157240/WEB-INF/classes
10:51:58,231 INFO  [BaseDeployer:1244] Modifying Servlet 2.3 /opt/tomcat-instances/liferay-inst1/temp/20081030105157240/WEB-INF/web.xml
  Copying 27 files to /opt/tomcat-instances/liferay-inst1/webapps/portail.cpe.fr/solr-web
  Copying 1 file to /opt/tomcat-instances/liferay-inst1/webapps/portail.cpe.fr/solr-web
  Deleting directory /opt/tomcat-instances/liferay-inst1/temp/20081030105157240
10:51:58,303 INFO  [WebAutoDeployListener:60] Web plugin for /var/lib/tomcat/liferay/repositories/liferay-inst1/liferay/deploy/solr-web-5.1.1.1.war copied successfully
10:51:58,815 INFO  [ClassPathXmlApplicationContext:412] Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@11dc8af: display name [org.springframework.context.support.ClassPathXmlApplicationContext@11dc8af]; startup date [Thu Oct 30 10:51:58 GMT 2008]; root of context hierarchy
10:51:58,887 INFO  [XmlBeanDefinitionReader:323] Loading XML bean definitions from class path resource [META-INF/solr-spring.xml]
10:51:58,924 INFO  [ClassPathXmlApplicationContext:427] Bean factory for application context [org.springframework.context.support.ClassPathXmlApplicationContext@11dc8af]: org.springframework.beans.factory.support.DefaultListableBeanFactory@1cb2378
10:51:58,944 INFO  [DefaultListableBeanFactory:414] Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1cb2378: defining beans [indexSearcher,indexWriter,searchEngine,searchEngineUtil,searchReaderMessageListener,searchWriterMessageListener]; root of factory hierarchy
Exception in thread "liferay/search_writer" java.lang.AbstractMethodError
        at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:69)
        at com.liferay.portal.kernel.messaging.SerialDestination$1.run(SerialDestination.java:53)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)  

The interesting part beeing the exception at the end of the log ;)

That exception is then repeated every time pages are accessed.
When sniffing network traffic with tcpdump, I see that no request are hitting the Solr server. So I suppose the problem to reside elsewhere in the solr-web plugin.
I suspect that the last solr-web plugin available is not compatible with Liferay 5.1.2.
If so, should I get the trunk version?

Regards,

Jérôme

[Gustavo Fernández Gómez, modificado hace 15 años.]

I have the same problem :-(
Have you try to use liferay 5.1.1, the solr plugin is for this version? works fine?

[Jérôme Schell, modificado hace 15 años.]

Gustavo Fernández Gómez:

Have you try to use liferay 5.1.1, the solr plugin is for this version? works fine?

Yes I tried that version quickly some weeks ago. I didn't have this problem at the time.
But I don't want to go back as it implies too much work for me.

[Bruno Farache, modificado hace 15 años.]

Hi guys, Solr 1.3 wasn't released by the time we developed solr-web, but it should work fine with Solr 1.2.

solr-web 5.1.1.1 is not compatible with liferay 5.1.2, but I can tell you which SVN revision is compatible, it is number 21235, above that it's only compatible with 5.2.

[Jérôme Schell, modificado hace 15 años.]

Bruno Farache:

Hi guys, Solr 1.3 wasn't released by the time we developed solr-web, but it should work fine with Solr 1.2.

solr-web 5.1.1.1 is not compatible with liferay 5.1.2, but I can tell you which SVN revision is compatible, it is number 21235, above that it's only compatible with 5.2.

Hello Bruno,

Any hope to see soon a new release of the plugin compatible with Liferay 5.1.2?

[Bruno Farache, modificado hace 15 años.]

Hey, if you take a look at the download section, you will see that the majority of the plugins are compatible with 5.1.1 only. I don't know if there are plans to release them for 5.1.2, sorry. I will try to figure out.

[Jérôme Schell, modificado hace 15 años.]

Bruno Farache:

solr-web 5.1.1.1 is not compatible with liferay 5.1.2, but I can tell you which SVN revision is compatible, it is number 21235, above that it's only compatible with 5.2.

Ok, so I managed to build solr-web from SVN revision 21235. Now I can deploy it in Liferay 5.1.2 without any error message.

I then revert back to Solr 1.2.0.
I still have a problem when I tried to reindex the portal content, I get "org.apache.solr.core.SolrException: missing content stream" exceptions in Solr log. Below is a portion of the log when reindexing the blog portlet:

31 oct. 2008 10:15:49 org.apache.solr.core.SolrCore execute
INFO: /update <!--?xml version="1.0"+encoding%3D"UTF-8"?-->%10<add><doc><field+name%3d"modified">20081031091549<field+name%3d"entryclassname">com.liferay.portlet.blogs.model.BlogsEntry<field+name%3d"groupid">12401<field+name%3d"companyid">10106<field+name%3d"entryclasspk">12555<field+name%3d"title">Nouveau+E-Campus<field+name%3d"userid">10392<field+name%3d"uid">33_PORTLET_12555<field+name%3d"content">Duis+interdum+rhoncus+tellus.+Sed+fringilla+purus+eu+ante.+Curabitur+facilisis+interdum+ante.+Integer+varius?+Nunc+convallis,+nibh+id+convallis+mattis,+massa+est+hendrerit+mi,+eu+fringilla+diam+ligula+non+dui.+Phasellus+ut+nunc.+Aliquam+tincidunt+fermentum+tellus.+Nam+tortor+sapien;+adipiscing+eget,+facilisis+id,+vulputate+quis,+erat?+Donec+volutpat.+Nunc+venenatis+iaculis+nisl.+Quisque+ultrices+arcu+ut+odio.+Praesent+libero+nunc,+venenatis+a,+pulvinar+sed,+vehicula+non,+massa.+Donec+lectus.+Nullam+iaculis+quam+ac+quam.+Suspendisse+lectus.+In+lobortis+aliquet+turpis.+Fusce+semper+diam+quis+nunc+luctus+tempus.+Fusce+posuere,+ipsum+ut+convallis+ultricies,+nibh+elit+volutpat+velit,+et+placerat+dui+lectus+a+quam.+Aenean+pellentesque.+Maecenas+vulputate+massa+a+dolor.+Duis+interdum+rhoncus+tellus.+Sed+fringilla+purus+eu+ante.+Curabitur+facilisis+interdum+ante.+Integer+varius?+Nunc+convallis,+nibh+id+convallis+mattis,+massa+est+hendrerit+mi,+eu+fringilla+diam+ligula+non+dui.+Phasellus+ut+nunc.+Aliquam+tincidunt+fermentum+tellus.+Nam+tortor+sapien;+adipiscing+eget,+facilisis+id,+vulputate+quis,+erat?+Donec+volutpat.+Nunc+venenatis+iaculis+nisl.+Quisque+ultrices+arcu+ut+odio.+Praesent+libero+nunc,+venenatis+a,+pulvinar+sed,+vehicula+non,+massa.+Donec+lectus.+Nullam+iaculis+quam+ac+quam.+Suspendisse+lectus.+In+lobortis+aliquet+turpis.+Fusce+semper+diam+quis+nunc+luctus+tempus.+Fusce+posuere,+ipsum+ut+convallis+ultricies,+nibh+elit+volutpat+velit,+et+placerat+dui+lectus+a+quam.+Aenean+pellentesque.+Maecenas+vulputate+massa+a+dolor.++++<field+name%3d"portletid">33</field+name%3d"portletid"></field+name%3d"content"></field+name%3d"uid"></field+name%3d"userid"></field+name%3d"title"></field+name%3d"entryclasspk"></field+name%3d"companyid"></field+name%3d"groupid"></field+name%3d"entryclassname"></field+name%3d"modified"></doc></add> 0 2
31 oct. 2008 10:15:49 org.apache.solr.core.SolrException log
GRAVE: org.apache.solr.core.SolrException: missing content stream
        at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:75)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:77)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:658)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:191)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:159)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
        at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:835)
        at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:640)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1286)
        at java.lang.Thread.run(Thread.java:595)

Sorry it's wide...
I get that exception for every requests.

If I add the "<defaultSearchField>content</defaultSearchField>" in the schema.xml it seems to be worse.

Any hints for that?

Thanks in advance.
--
Jérôme

[Jérôme Schell, modificado hace 15 años.]

In fact it seems that "Content-type" header in HTTP requests issued by the solr-web portlet are bad.
They look like:
Content-Type: application/x-www-form-urlencoded

but I think they should be:
Content-Type: text/xml; charset=utf-8

So Solr always returns HTTP code 400 (Bad request).

It seems strange because in SolrIndexWriterImpl.java I see in submitRequest method:

Http.Body httpBody = new Http.Body(
			xml, ContentTypes.TEXT_XML, StringPool.UTF8);

with ContentTypes.TEXT_XML = "text/xml"

I've probably missed something there...

[Jérôme Schell, modificado hace 15 años.]

Some news...

Commenting out:
<requestHandler name="/update" class="solr.XmlUpdateRequestHandler" />

in solrconfig.xml seems to resolved the "Content-type" problem.

Now I get lots of NullPointerException in Liferay logs.

To be continued...

[Bruno Farache, modificado hace 15 años.]

I never seen this error locally, maybe it just happens with some specific data, non-latin characters, maybe? Are you able to add a simple blog entry, Jerôme?

It's strange because solr-web never make an <update> call, if you look at SolrIndexWriterImpl.updateDocument you will see that it's a <delete> followed by <add>.

I'm considering in using Solrj: http://wiki.apache.org/solr/Solrj

I guess it will make Solr upgrades easier. Thank you Gustavo for pointing out this minor change in schema.xml!

[Bruno Farache, modificado hace 15 años.]

Sorry Jerôme /update but didn't see the <add>, it's not making a <update> then,

Does it happen only at reindexing or does it also happen when you add a blog entry?

[Jérôme Schell, modificado hace 15 años.]

Bruno Farache:

Does it happen only at reindexing or does it also happen when you add a blog entry?

It happens everytime solr-web is sending requests on /update URL.
I just tried to add a blog entry and I get the exception when the parameter is not commented out in solrconfig.xml.

[Bruno Farache, modificado hace 15 años.]

Just upgraded solr-web to work with Solr 1.3. Code is much cleaner now because we are leveraging the client library:

http://support.liferay.com/browse/LPS-397

[Jérôme Schell, modificado hace 15 años.]

Great Bruno!
Just have to wait for Liferay 5.2 to test that ;)

Thanks again for your reactivity.

[Alex Wallace, modificado hace 14 años.]

Hi bruno!

Do you have an ide of the effort that would be required to make the plugin that works with solr 1.3 work with LR 5.1.2?

Would it be significant architecture change ? Or fixing compile errors by changing packages, etc?

Because as far as I understand, the plugin with revision 21235 won't work with 1.3, correct?

thanks in advance!

[Bruno Farache, modificado hace 14 años.]

I can't remember all the changes that were made, but the plugin is very simple and small, shouldnt be hard to make it work with solr 3.

I dont think that interfaces changed a lot, if it did it was something small. I answered some of your forum message about multithreaded but for some reason I didn't send it. SearchEngineUtil is inherently multithreaded because it uses our message bus service. Look for messaging-spring.xml, at search writer's parallel destination you can set how many threads it will use. Never seen solr not handling many requests at once, maybe it's a solr configuration issue?

[Alex Wallace, modificado hace 14 años.]

Thank you very much for your Reply Bruno!

I am looking at the differences and my best bet seems to be to modify the plugin checked out at r21235... Particularly SolrIndexSearcherImpl and SolrIndexWriterImpl ...

The new plugin in the trunk seem to use a different 'register' and 'unregister' mechanism...

I will give it a try tomorrow, on the surface indeed it looks like clean path to upgrade r21235 to Solr 1.3


Regarding your reply to my multithreaded post. I appreciate it much as well... In that message I was referring to the Lucene searcher Impl, I had not tried Solr (have not made it work yet)... I will look at the spring config you mention to see if I can gain speed there... But I think that my problem is closer to how I create each document, and using multiple threads to create the documents increases performance significantly for me (from one hour to 3 minutes)... However, as I stated in the message, the index ends up with fewer records the more threads I use... This is, again, only if I use SearchEngineUtil, because If I directly grab the lucene's IndexWriter, it works fine...

If I am able to make Solr work, this will probably be a non issue, as Solr should consider requests send by my threads as a regular http request... If I can't get Solr to work, then I will probably revert to writing to the lucene IndexWriter direcltly anyways... But this going back to lucene alone is not a real option, since we are already running the site as a cluster, only the indexes are out of sync until midnight every day...

I also posted a question to a thread dealing with lucene via JDBC, is that really a viable option for clustering? In my understanding it isn't.

Thank you again for your reply bruno, I will post here how it went...

[Alex Wallace, modificado hace 14 años.]

Hi!

Upgrading r21235 code to work with Solr 1.3 was real easy... pretty much copied the code from what's in trunk for the plugin, added a few dependencies (jars) to the plugin and modified the solr-spring to instantiate a SolrServer and pass that to Searcher and Writer instead of using URLs...

It compiles and deploys fine now...

It is also indexing at the moment but I am not sure yet it is indexing all my records... Unless there is some delay in the records becomming available, there is something wrong, because I'm processing record 4k out of 20k and Solr shows only 1k records...

I still want to let it finish indexing, but I don't see any errors being thrown so far... I guess, we'll see!

[Alex Wallace, modificado hace 14 años.]

Indexing finished and only like half of my records made it to solr... not trying multiple threads on my indexer yet... so not sure what to do....

Solr statistics show 0 errors everywhere, and it shows only around half of my records as 'update' requests...

I am not sure where, but my requests to index data are getting dropped... and there are no errors being reported...

I think my next step is to talk to Solr directly from my indexer instead of going trhough the SearchEngine in LR...

Am I perhaps missing something? like syncronizing on some class? I looked at one reindexer that comes with LR and seems to just use SearchEngine for reindexing as it does for one single item... I'm not sure if I'm missing something?

[Wilson Man, modificado hace 14 años.]

hey Alex, sorry to have come onto this thread so late, but I was doing the same for a client recently and we were able to index 1.1 million users in 40 minutes total.

we made a lot of changes from end to end, but one of the biggest bottleneck is found in SOLR committing updates, merging segments, and optimizing its data.

before i outline what should be looked into, let me first say that the bottleneck isn't on the Indexer classes (the ones that query the db and create the Documents). The message bus is also not the bottleneck. We had tried using multi-threading on this and we can't really validate that it helped. The Message Bus is actually really fast and efficient. Most if not all the bottleneck again is on SOLR side.

What you want to do this this:
1. modify SOLR-WEB plugin and comment out the commit() method call from the addDocument method in SolrIndexWriterImpl.
(my current version also comment out the one in deletePortletDocuments(). I don't think you need to, but make sure you don't comment out the one for deleteDocument().)
2. reconfigure your SOLR to auto commit only when 2000 updates have been submitted OR 5 minutes have passed. You'll also have to play around with the merge segment size factor. And also optimization.

You'll probably need to play around with solr settings to find the most optimal setting for your normal day to day needs. However, for initial reindexing of all data, we've setup our SOLR to favor indexing rather than searching. And once that was done, we reconfigured it to favor searching.

I've read you can have a master-slave setup for SOLR so you can have diff. config for search and indexing and hook them separately, but we haven't tried it yet and can't speak more about it.

But yeah, committed after each update was the problem that slowed it down. Once we stopped that, we saw some blazing speed ... however, the performance degraded after indexing ran for a while. But we fixed that with tweaking merge, optimization settings on SOLR and got 1.1 million users indexed in 40 minutes total.

hope that helps.

updated: also for initial reindexing of everything, you can also turn off warming and caches to speed it up even more ...

[Wilson Man, modificado hace 14 años.]

	public void addDocument(long companyId, Document doc)
		throws SearchException {

		try {
			_solrServer.add(getSolrDocument(doc));

//			_solrServer.commit();
		}
		catch (Exception e) {
			throw new SearchException(e);
		}
	}

here's the code snippet of what i meant in SolrIndexWrtierImpl.java

[Wilson Man, modificado hace 14 años.]

one more thing, we should probably make this configurable solrServer.commit() call configurable ... either from a property or thru the method param ...

bruno? =)

[Alex Wallace, modificado hace 14 años.]

Thank you very much for your reply Wilson!

Let me give some background that I've not expressed clearly... My Indexers are not Liferay's... I'm indexing data that comes from a lot of custom entities we created, so I have a good number of custom indexers.

In the past, when indexing against lucene, we gained performance by doing a number of things:

1 - reusing documents and fields (instead of creating new ones, updating one for the whole process)
2 - reusing the writer (not closing it after every addDocument)
3 - using threads to add to the index writer concurrently.

This proved to work real good when dealing with lucene document/field/writter directly... But when i refactored the code to use SearchEngine (from liferay), when using threads, lots of records where dropped... it is still puzzling to me why... But since I'm trying solr, I have not researched more.

This issue was the reason I thought the Messaging Layer could be the cause... But this is only speculation.

Now that I had data dropped in solr (no timeouts, no errors, etc), I was going to skip the Solr plugin and make my custom indexers use SolrJ directly... Indeed, as you point out, the commit() was the first candidate for improvement... As an equivalent of reusing lucene's writer, I was going to have my threads commit when finished with each batch, or potentially after processing all batches...

You bring up an easier path to try that, by using Solr's auto-commit capability, which I had not thought of... Thank you very much I will give it a try...

If that solves the mistery of the dropped records, I'll be very happy... My doubt comes from the fact that it doesn't seem like a performance issue and I don't have any errors on the log, or timeouts... But maybe I'll get lucky...

Also, I'm using LR 5.1.2... against solr 1.3 ... Are these the versions you are using? This would help me know if your Messaging and Plugin look like mine...

I appologize for the long reply, but I felt like I needed to share some background so that my issue could be better understood.

I thank you very, very much for this reply! I will try your suggestion VS rewriting my indexers to use Solrj... I hope I get lucky!

Thanks!r

[Alex Wallace, modificado hace 14 años.]

I also agree very much with your suggestion about improving the plugin, both solr and lucene's could bennefit from the ability to porpusefully add a bunch of documments before a commit/write ...

For some actions, like the addition of one new document or updating one single document it is convenient to commit right away...

But for reindexing a commit/write per document is definitely not advisable...

Thanks to all!

[Wilson Man, modificado hace 14 años.]

i see, yeah, let us know your results. The client i'm working with are using 5.2.x and Solr 1.3. I'm not sure if there's a big difference between 5.2 and 5.1. Maybe Bruno can speak more to that.

Also, we have some custom indexers that we wrote ... they reside in the plugins. But they follow the liferay convention and call SearchEngineUtil ... which then goes thru the message bus to SOLR.

[Alex Wallace, modificado hace 14 años.]

Thanks again for your reply!

After enabling the auto-commit as you recommended, My custome indexer running on the current thread indeed was able to index ALL documents... This is great news...

Evidently because of this 'commit for every doc' somewhere, documents were being lost... What's concerning is that nothing reported that...

I can only assume that it was Solr or the app server completly ignoring threads... But why didn't liferay tell me?

The sad part is now that my custom reindexer takes 1.5 hours to index 20k records... This is pretty bad... I know that much improvement can be done... One of the improvements was using threads... So, i went and turned that feature on, using only 2 threads, it runs in less than 2 minutes! But now the issue is that solr only picked up 1k documents out of the 20k that were 'supposedly sent'...

I will now try it with only 1 thread (still faster than not using threads)...

Right now the first 'culprit' seems to be Solr... Or the app server where it runs... I will focus on improving that... But I don't know for sure if it is solr or LR... Althought evidence (the first usecase with commit for each doc) points to Solr or its's app server... Perhaps I need to play with maxThreads in it's tomcat...

I'll post here what I find out...

Oh, would you mind sharing what settings/values you used to improve performance aside from authoCommit in Solr? I know they depend on mychine specs, but it could give me a baseline...

Thanks!

[Alex Wallace, modificado hace 14 años.]

Just to report, that running my indexer using 1 thread, the process took 3 minutes... Not bad... but Solr recorded 17k documents... No doubt that the issue is related to the speed at which we send documents to it.... regardless of where they are being dropped... I still wish I got informed some how...

I will first try to play with the app server, increase the amount of threads it can handle for example...

In production, given the hardware, 8 threads is what we were using to reindex, which did it in a few seconds directly to lucene...

I'll post here what my findings are...

Thanks for all the help!

[Alex Wallace, modificado hace 14 años.]

Hi all!

I am still fighting this animal...

I think i can say without doubt that it is an issue on the Solr side... This is, because, at least I know for sure that liferay is calling SolrIndexWriterImpl the right number of times, that it actually excecutes _sorlServer.add the right number of times...

However, my solr index ends up sometimes with a bit more, a bit less, but always short by many docs... If my client uses more threads, then the index ends up with less docs...

2 things concerning:

1 - why are these requests being dropped...
2 - why i don't get any errors anywhere

If only i had errors that could lead me to what to tinker with, for now, I can only speculate that it may be tomcat just disregarding the requests, or solr config...

Any thoughts?

[Alex Wallace, modificado hace 14 años.]

Also... Looking at the log in the app server running Solr, I can see the right number of 'INFO: {add...' Logs, so Indeed they are all getting there...

Further more Solr admin shows the right number of cumulative adds.... But # docs is wrong... I think I'm getting closer...

[Alex Wallace, modificado hace 14 años.]

I think I know what is going on... And i is not Solr...

To save a lot in GC cost, in the past, we implemented a practice recommended by Lucene to improve re-indexing, which is to re-use one single document instance to index all your records... This is done by adding fields to documents and keeping a hold of the inscance of each field, adding the document to the index (using IndexWriter) and then updating the values in the fields to add a new record.

This used to work fine when we were grabbing the lucene IndexWriter our selfs, beacuse the add method really blocked the thread's execution until the document was added, then excecution would continue... Since you can use the same writer and not close it for as long as you see convenient, we kept on adding documents, and yielded every once in a while so that other processes could write to the index... All this worked fine...

Since I changed my documents from lucene to LR, and tried to use SearchEngineUtil, it worked fine when not using threads, but when using threads, I was loosing documents just as I am now with solr... I have not confirmed this yet, but I can almost bet that SearchEngineUtil, because of the messaging it uses, releases my process before the document is actually added to the index... This is great so that response is fast, but that's where I'm loosing documents: My threads keep on updating the document fields even before the documents are added to the index.

I can confirm that Solr gets the 20k add requests... But i decided to do look for uniqueness of the data being added... And indeed, threre are only as many unique documents being added as records on the index... That is, my threads are updating the documents before they are indexed and i endup in many cases adding the same document many times...

I can see how reusing documents can't work under this messaging scenario... With or without solr in place, since by the time i update a document, there is not a guarantee that the document was processed...

I will now try to create a new instance of a document with every call... hopefully (will have to see) the loss of power due to GC is less than the gain of using threads...

I will post back what I find...

[Alex Wallace, modificado hace 14 años.]

Yes! That was absolutely it!

So, it is very worth saying that if one uses Liferay's Search Abstraction Layer, one can not reuse documents/fields and needs to create new instances for each record to be indexed...

The practice saves on GC, but I think the messaging layer has greater advantages...

By creating new document/fields for each record to be indexed, I am now able to run reindexing using all the threads my machine can handle, bringing down the time to under a minute instead of an hour+

Whatever I was saving on memory conservation and less GC is minimal compared to multi-threading gains... So I'm happy to give that away...

This issue was impacting me when using either lucene or solr...

It seems all good now with solr... At least the reindexing part..

I thank you all much for your support in this matter!

[Wilson Man, modificado hace 14 años.]

nice! congrats! =)

[Jérôme Schell, modificado hace 15 años.]

Bruno Farache:

I never seen this error locally, maybe it just happens with some specific data, non-latin characters, maybe? Are you able to add a simple blog entry, Jerôme?

It's strange because solr-web never make an <update> call, if you look at SolrIndexWriterImpl.updateDocument you will see that it's a <delete> followed by <add>.

After stopping and starting Liferay and Solr several times, it seems all is working well. I can reindex all the content, add blog entries and search with success.

Concerning the fact that solr-web never makes <update> calls, no it doesn't. But it makes calls on "/solr/update" URL as you can see in the above error message ("INFO: /update <?xml ..."). And as far as I can see, the Content-Type header of the requests is not good for Solr 1.2. Commenting the parameter in solrconfig.xml makes Solr behave like v1.1 and requests are processed successfully.

I still have problem with non-ASCII characters not being handled correctly in search query strings and results.
If I search the string "hé", the search field of the results page contains "hé" (UTF-8 strings interpreted as ISO-8859-1). The string passed to Solr as the query string looks like "hã" in Solr logs.
There should be some piece of code not dealing correctly with that. But this is not related to Solr as I have this problem also when using Lucene in single instance setup.

Thanks Bruno for your work done on this.

[Bruno Farache, modificado hace 15 años.]

Jérôme Schell:

Yes I tried that version quickly some weeks ago. I didn't have this problem at the time.
But I don't want to go back as it implies too much work for me.

Hey Jérôme, you meant that at some time solr-web was working fine with Solr 1.3?

[Jérôme Schell, modificado hace 15 años.]

Bruno Farache:

Hey Jérôme, you meant that at some time solr-web was working fine with Solr 1.3?

Sorry, no, at the time it was Solr 1.2.
According to what you said, I think I will revert back to 1.2.

[Gustavo Fernández Gómez, modificado hace 15 años.]

Bruno Farache:

Hey Jérôme, you meant that at some time solr-web was working fine with Solr 1.3?

Hi Bruno, i am trying now with liferay-5.1.1 and solr 1.3 and it seems to works fine, except that the re-idex all proces throws a null pointer exception, but if you upload files to DL works fine and the web serarch portlet and DL searcher locate the documents from the respones of sorl server.

[Gustavo Fernández Gómez, modificado hace 15 años.]

i forgot commet that i have to add

<defaultSearchField>content</defaultSearchField>

node to schema.xml

[Carlos del Castillo, modificado hace 15 años.]

Hello everybody, I'm upgrading my liferay 5.1.1 to 5.2. I have been using Solr 12. without problems on the 5.1.1 version.

After the upgraded I checked out the revision 21235 and compiled everything made the required modifications to point to our solr server.

But I'm getting the error that was mentioned previously, missing content stream.

SEVERE: org.apache.solr.core.SolrException: missing content stream
	at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:75)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:77)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:658)
	at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:191)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:159)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:209)
	at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
	at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:856)
	at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:565)
	at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1509)
	at java.lang.Thread.run(Thread.java:619)

I'm wondering if somebody was able to make this work. I have created a new blog entry but is not being indexed. I can search for the old content because Solr already has those entries.

Any idas or advise where to look for the problem?

Thank you very much.

[Bruno Farache, modificado hace 15 años.]

Did you try what Jérôme suggested?

Jérôme:

Commenting out:
<requestHandler name="/update" class="solr.XmlUpdateRequestHandler" />

in solrconfig.xml seems to resolved the "Content-type" problem.

[Carlos del Castillo, modificado hace 15 años.]

Ok, that worked, I thought after reading some other post that didn't work after all.

Thanks again!

Just one more question for liferay 5.1.2 should I stay on Solr 1.2 and the revision of the plugin previously mentioned or should I move to Solr 1.3 and update the plugin?

[Bruno Farache, modificado hace 15 años.]

No, you can't do it out of the box, some changes to this plugin depend on changes that happened in Liferay core after 5.1.2.

But should be easy to develop by yourself...

[Carlos del Castillo, modificado hace 15 años.]

I was wondering if you could take a look to this other post, hopefully you can give an idea on how to solve it.
http://www.liferay.com/web/guest/community/forums/-/message_boards/message/2058791
Thanks

[Willy Toscer, modificado hace 15 años.]

Hi,
I have some problems with solr 1.2 and liferay 5.1.2.
When I do a re-index, I have the same errors.
I try to uncomment the line "<requestHandler name="/update" class="solr.XmlUpdateRequestHandler" />" in solrconfig.xml, but it doesn't work anymore.
When I try to add a blog entry, I have the same problem.
Can you help me?

When I go to the solr interface admin, in the statistics, I can see :
[indent]requests : 2162
errors : 2162[/indent]

[Willy Toscer, modificado hace 15 años.]

Finally, I have successfully installed solr.
But i would like to enable the faceted searching, wich is the fonctionnality i want to have, and I don't know how to do.
Is somebody can help me?

[Jeff Williams, modificado hace 15 años.]

Can anyone tell me what revision of solr-web and what version of solr to use with 5.2.2?

I'm having all kinds of issues trying to get this working.

thanks

[BHAVANI TALLA, modificado hace 14 años.]

hi all,

i want to integrate solr search engine with liferay 5.2.3. i hava solr1.4.0 installed on my pc.i am unable to download solr-web-5.2.0.1.war from sourceforge.net. what is the allteranative to this.