Forums

Home » Liferay Portal » English » 3. Development

RSS (Opens New Window)

Combination View Flat View Tree View
Threads [ Previous | Next ]
Andew Jardine
c3p0 Connection Pool -- unreturned connections?
May 23, 2012 7:38 AM
Answer

Andew Jardine

Rank: Junior Member

Posts: 96

Join Date: December 22, 2010

Recent Posts

Hi,

I have a real puzzle I am trying to figure out right now.A quick summary of what I think is going wrong and how I am patching the holes in a sinking ship.

1. 2 node cluster -- 10GB RAM and 500 DB connections per node
2. 10 - 15 minute window until all resources are exhausted and the server hangs
3. I enabled DEBUG level logging for com.mchange and can see the pool steadily increasing
4. I enable several JDBC loggers at DEBUG and can see a global prepared statement count steadily increasing

I suspect that the two coincided and that the prepares statement was not returning the connection to the pool. I turned on 3 additional c3p0 settings in the portal-ext file.

a. jdbc.default.testConnectionOnCheckout=true
b. jdbc.default.unreturnedConnectionTimeout=300 #5 minutes
c. jdbc.default.debugUnreturnedConnectionStackTraces=true

I was hoping that c. would lead me to the source of the leak. Unfortunately, I don't see the stacktraces being pushed out. On top of that I am occasionally having a problem saving a simple WebForm portlet's information. The stack trace that coincides with this error is --

  1
  212:33:27,794 ERROR [JDBCExceptionReporter:101] No operations allowed after connection closed.
  312:33:27,799 ERROR [WebFormPortlet:331] The web form data could not be saved to the database
  4com.liferay.portal.SystemException: com.liferay.portal.kernel.dao.orm.ORMException: could not load an entity: [com.liferay.counter.model.Counter#com.liferay.counter.model.Counter]
  5    at com.liferay.portal.service.persistence.impl.BasePersistenceImpl.processException(BasePersistenceImpl.java:91)
  6    at com.liferay.counter.service.persistence.CounterPersistence.increment(CounterPersistence.java:162)
  7    at com.liferay.counter.service.persistence.CounterPersistence.increment(CounterPersistence.java:121)
  8    at com.liferay.counter.service.persistence.CounterPersistence.increment(CounterPersistence.java:117)
  9    at com.liferay.counter.service.persistence.CounterUtil.increment(CounterUtil.java:42)
 10    at com.liferay.counter.service.impl.CounterLocalServiceImpl.increment(CounterLocalServiceImpl.java:44)
 11    at sun.reflect.GeneratedMethodAccessor596.invoke(Unknown Source)
 12    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 13    at java.lang.reflect.Method.invoke(Method.java:592)
 14    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
 15    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
 16    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
 17    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
 18    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
 19    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
 20    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
 21    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
 22    at $Proxy82.increment(Unknown Source)
 23    at com.liferay.portlet.expando.service.impl.ExpandoValueLocalServiceImpl.addValue(ExpandoValueLocalServiceImpl.java:66)
 24    at com.liferay.portlet.expando.service.impl.ExpandoValueLocalServiceImpl.addValue(ExpandoValueLocalServiceImpl.java:507)
 25    at sun.reflect.GeneratedMethodAccessor597.invoke(Unknown Source)
 26    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 27    at java.lang.reflect.Method.invoke(Method.java:592)
 28    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
 29    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
 30    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
 31    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
 32    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
 33    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
 34    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
 35    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
 36    at $Proxy98.addValue(Unknown Source)
 37    at com.liferay.portlet.expando.service.ExpandoValueLocalServiceUtil.addValue(ExpandoValueLocalServiceUtil.java:259)
 38    at com.liferay.webform.portlet.WebFormPortlet.saveDatabase(WebFormPortlet.java:323)
 39    at com.liferay.webform.portlet.WebFormPortlet.saveData(WebFormPortlet.java:183)
 40    at sun.reflect.GeneratedMethodAccessor730.invoke(Unknown Source)
 41    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 42    at java.lang.reflect.Method.invoke(Method.java:592)
 43    at com.liferay.portal.kernel.portlet.LiferayPortlet.callActionMethod(LiferayPortlet.java:111)
 44    at com.liferay.portal.kernel.portlet.LiferayPortlet.processAction(LiferayPortlet.java:69)
 45    at com.sun.portal.portletcontainer.appengine.filter.FilterChainImpl.doFilter(FilterChainImpl.java:98)
 46    at com.liferay.portal.kernel.portlet.PortletFilterUtil.doFilter(PortletFilterUtil.java:57)
 47    at com.liferay.portal.kernel.servlet.PortletServlet.service(PortletServlet.java:100)
 48    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 49    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 50    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 51    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:630)
 52    at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:535)
 53    at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:472)
 54    at com.liferay.portlet.InvokerPortletImpl.invoke(InvokerPortletImpl.java:618)
 55    at com.liferay.portlet.InvokerPortletImpl.invokeAction(InvokerPortletImpl.java:664)
 56    at com.liferay.portlet.InvokerPortletImpl.processAction(InvokerPortletImpl.java:359)
 57    at com.liferay.portal.action.LayoutAction.processPortletRequest(LayoutAction.java:604)
 58    at com.liferay.portal.action.LayoutAction.processLayout(LayoutAction.java:428)
 59    at com.liferay.portal.action.LayoutAction.execute(LayoutAction.java:195)
 60    at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
 61    at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)
 62    at com.liferay.portal.struts.PortalRequestProcessor.process(PortalRequestProcessor.java:157)
 63    at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
 64    at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
 65    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
 66    at com.liferay.portal.servlet.MainServlet.callParentService(MainServlet.java:608)
 67    at com.liferay.portal.servlet.MainServlet.service(MainServlet.java:846)
 68    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 69    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 70    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 71    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:630)
 72    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
 73    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
 74    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
 75    at com.liferay.portal.servlet.FriendlyURLServlet.service(FriendlyURLServlet.java:143)
 76    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 77    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 78    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 79    at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
 80    at com.liferay.portal.servlet.filters.strip.StripFilter.processFilter(StripFilter.java:142)
 81    at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
 82    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 83    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 84    at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
 85    at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:94)
 86    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 87    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 88    at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
 89    at com.liferay.portal.servlet.filters.gzip.GZipFilter.processFilter(GZipFilter.java:140)
 90    at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
 91    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 92    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 93    at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
 94    at com.liferay.portal.servlet.filters.secure.SecureFilter.processFilter(SecureFilter.java:282)
 95    at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
 96    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 97    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 98    at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
 99    at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:94)
100    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
101    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
102    at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
103    at com.liferay.portal.servlet.filters.cache.CacheFilter.processFilter(CacheFilter.java:425)
104    at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
105    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
106    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
107    at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
108    at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.doFilter(VirtualHostFilter.java:191)
109    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
110    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
111    at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
112    at com.liferay.portal.servlet.filters.threadlocalcache.ThreadLocalCacheFilter.processFilter(ThreadLocalCacheFilter.java:55)
113    at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
114    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
115    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
116    at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
117    at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:94)
118    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
119    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
120    at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738)
121    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
122    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
123    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
124    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
125    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
126    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
127    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
128    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
129    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
130    at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
131    at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
132    at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
133    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
134    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
135    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
136    at java.lang.Thread.run(Thread.java:595)
137Caused by: com.liferay.portal.kernel.dao.orm.ORMException: could not load an entity: [com.liferay.counter.model.Counter#com.liferay.counter.model.Counter]
138    at com.liferay.portal.dao.orm.hibernate.ExceptionTranslator.translate(ExceptionTranslator.java:41)
139    at com.liferay.portal.dao.orm.hibernate.SessionImpl.get(SessionImpl.java:131)
140    at com.liferay.counter.service.persistence.CounterPersistence.increment(CounterPersistence.java:145)
141    ... 130 more


I'm not sure but i suspect that for some reason the unreturned connections are being used in this scenario. When I first boot I don't see the issue but eventually this starts to show up and the "Portet is unavailable message" is displayed to me after submitting the form.

I'm beating my brains trying to figure this problem out. Can anyone help?
Flag
Igor Barma
RE: c3p0 Connection Pool -- unreturned connections?
February 5, 2013 7:48 AM
Answer

Igor Barma

Rank: New Member

Posts: 16

Join Date: September 30, 2009

Recent Posts

Hi,
I've sometimes exactly the same problem with my database c3p0 connector.
Have you find a solution to it? It's very frustating for me to not find the answer.
My site works well for a time (a couple of days) and then my pool become busy on all my connections for a node, i need to restart it and sometimes restart my DB too.
I turned on unreternedConnectionTimeout like you to monitor leak in my code but without result cause the error:
ERROR [JDBCExceptionReporter:101] No operations allowed after connection closed.

It will be very nice if you have found something on this problem....

Thanks,
Igor
Flag
Andew Jardine
RE: c3p0 Connection Pool -- unreturned connections?
February 6, 2013 4:09 PM
Answer

Andew Jardine

Rank: Junior Member

Posts: 96

Join Date: December 22, 2010

Recent Posts

Hi Igor,

To be honest I never fully solved the issue. I managed to get it to the point where the server only ran out of resources once a month which seemed a suitable solution for my client (at the time). In the end I simply mucked with the pool settings as well as the hibernate and mysql settings until the results seemed reasonable. I think I also set some pretty aggressive garbage collection settings on the JVM.

I did try a new install which didn't seem to have an issue until I moved the code over. That seemed to me to be an indication of a problem with the custom code (written by someone else for the record! emoticon ) rather than a problem with liferay (core).

Hope that helps.
Flag