Foren
tomcat gets blocked using almost 100% of CPU after some days running
Alan L., geändert vor 9 Jahren.
tomcat gets blocked using almost 100% of CPU after some days running
New Member Beiträge: 8 Beitrittsdatum: 28.08.12 Neueste Beiträge
Hi,
I have a problem with Liferay. After several days running smoothly, java process (tomcat) suddenly gets stucked using almost 100% of CPU...I'm using the tomcat bundle. It's liferay-6.1.0-ce-ga1. I'm using MySQL 5.1.73. A thread dump shows several threads blocked, I think it shows several threads waiting to lock an object (<0x00000000b5ea4748>) like this one :
"liferay/buffered_increment_parallel-210" daemon prio=10 tid=0x00007fd168258800 nid=0x48e0 waiting for monitor entry [0x00007fd1704b4000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.mchange.v2.resourcepool.BasicResourcePool.checkinResource(BasicResourcePool.java:646)
- waiting to lock <0x00000000b5ea4748> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$ConnectionEventListenerImpl.doCheckinResource(C3P0PooledConnectionPool.java:636)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$ConnectionEventListenerImpl.connectionClosed(C3P0PooledConnectionPool.java:630)
at com.mchange.v2.c3p0.util.ConnectionEventSupport.fireConnectionClosed(ConnectionEventSupport.java:55)
at com.mchange.v2.c3p0.impl.NewPooledConnection.fireConnectionClosed(NewPooledConnection.java:510)
at com.mchange.v2.c3p0.impl.NewPooledConnection.markClosedProxyConnection(NewPooledConnection.java:381)
at com.mchange.v2.c3p0.impl.NewProxyConnection.close(NewProxyConnection.java:1246)
- locked <0x00000000b29eb048> (a com.mchange.v2.c3p0.impl.NewProxyConnection)
at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
at com.sun.proxy.$Proxy4.close(Unknown Source)
at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.closeConnection(LocalDataSourceConnectionProvider.java:95)
at org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:474)
at org.hibernate.jdbc.ConnectionManager.cleanup(ConnectionManager.java:408)
at org.hibernate.jdbc.ConnectionManager.close(ConnectionManager.java:347)
at org.hibernate.impl.SessionImpl.close(SessionImpl.java:343)
at org.springframework.orm.hibernate3.SessionFactoryUtils.closeSession(SessionFactoryUtils.java:791)
at org.springframework.orm.hibernate3.SessionFactoryUtils.closeSessionOrRegisterDeferredClose(SessionFactoryUtils.java:777)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doCleanupAfterCompletion(HibernateTransactionManager.java:737)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion(AbstractPlatformTransactionManager.java:1011)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:804)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:78)
at com.liferay.portal.increment.BufferedIncreasableEntry.proceed(BufferedIncreasableEntry.java:51)
at com.liferay.portal.increment.BufferedIncrementMessageListener.doReceive(BufferedIncrementMessageListener.java:49)
at com.liferay.portal.kernel.messaging.BaseMessageListener.receive(BaseMessageListener.java:25)
at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:65)
at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:106)
at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:669)
at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:580)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
And 3 other threads like this one where that object (<0x00000000b5ea4748>) is blocked:
"ajp-bio-8009-exec-3082" daemon prio=10 tid=0x00007fd168940800 nid=0x45ae in Object.wait() [0x00007fd1441ba000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
- locked <0x00000000b5ea4748> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at com.liferay.portal.dao.jdbc.util.DataSourceWrapper.getConnection(DataSourceWrapper.java:36)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:403)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
at com.sun.proxy.$Proxy4.prepareStatement(Unknown Source)
at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:534)
at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:452)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1700)
at org.hibernate.loader.Loader.doQuery(Loader.java:801)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.doList(Loader.java:2533)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
at org.hibernate.loader.Loader.list(Loader.java:2271)
at org.hibernate.hql.classic.QueryTranslatorImpl.list(QueryTranslatorImpl.java:940)
at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1268)
at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
at com.liferay.portal.dao.orm.hibernate.QueryImpl.list(QueryImpl.java:86)
at com.liferay.portal.dao.orm.hibernate.QueryImpl.list(QueryImpl.java:79)
at com.liferay.portal.kernel.dao.orm.QueryUtil.list(QueryUtil.java:60)
at com.liferay.portal.kernel.dao.orm.QueryUtil.list(QueryUtil.java:52)
at com.liferay.portal.service.persistence.ResourcePermissionPersistenceImpl.findByC_N_S_P_R(ResourcePermissionPersistenceImpl.java:3810)
at com.liferay.portal.service.persistence.ResourcePermissionPersistenceImpl.findByC_N_S_P_R(ResourcePermissionPersistenceImpl.java:3628)
at com.liferay.portal.service.impl.ResourcePermissionLocalServiceImpl.hasResourcePermission(ResourcePermissionLocalServiceImpl.java:709)
at com.liferay.portal.service.impl.ResourcePermissionLocalServiceImpl.hasResourcePermission(ResourcePermissionLocalServiceImpl.java:612)
at sun.reflect.GeneratedMethodAccessor335.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:112)
at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:71)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108)
at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:59)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108)
at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:59)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108)
at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:211)
at com.sun.proxy.$Proxy83.hasResourcePermission(Unknown Source)
at com.liferay.portal.service.impl.PermissionLocalServiceImpl.hasUserPermissions_6(PermissionLocalServiceImpl.java:1489)
at com.liferay.portal.service.impl.PermissionLocalServiceImpl.hasUserPermissions(PermissionLocalServiceImpl.java:724)
at sun.reflect.GeneratedMethodAccessor333.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:112)
at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:71)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108)
at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:59)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108)
at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:59)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108)
at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:211)
at com.sun.proxy.$Proxy54.hasUserPermissions(Unknown Source)
at com.liferay.portal.service.PermissionLocalServiceUtil.hasUserPermissions(PermissionLocalServiceUtil.java:644)
at com.liferay.portal.security.permission.AdvancedPermissionChecker.hasGuestPermission(AdvancedPermissionChecker.java:901)
at com.liferay.portal.security.permission.AdvancedPermissionChecker.hasPermissionImpl(AdvancedPermissionChecker.java:916)
at com.liferay.portal.security.permission.AdvancedPermissionChecker.hasPermission(AdvancedPermissionChecker.java:595)
at com.liferay.portal.service.permission.PortletPermissionImpl.contains(PortletPermissionImpl.java:303)
at com.liferay.portal.service.permission.PortletPermissionImpl.contains(PortletPermissionImpl.java:205)
at com.liferay.portal.service.permission.PortletPermissionImpl.contains(PortletPermissionImpl.java:164)
at com.liferay.portal.service.permission.PortletPermissionImpl.contains(PortletPermissionImpl.java:155)
at com.liferay.portal.service.permission.PortletPermissionUtil.contains(PortletPermissionUtil.java:116)
at org.apache.jsp.html.portal.render_005fportlet_jsp._jspService(render_005fportlet_jsp.java:485)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:72)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:70)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:593)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:530)
at com.liferay.portal.util.PortalImpl.renderPortlet(PortalImpl.java:4735)
at com.liferay.portal.util.PortalUtil.renderPortlet(PortalUtil.java:1455)
at com.liferay.portlet.layoutconfiguration.util.RuntimePortletImpl.processPortlet(RuntimePortletImpl.java:213)
at com.liferay.portlet.layoutconfiguration.util.RuntimePortletImpl.processPortlet(RuntimePortletImpl.java:148)
at com.liferay.portlet.layoutconfiguration.util.RuntimePortletImpl.processTemplate(RuntimePortletImpl.java:309)
at com.liferay.portlet.layoutconfiguration.util.RuntimePortletImpl.processTemplate(RuntimePortletImpl.java:234)
at com.liferay.portlet.layoutconfiguration.util.RuntimePortletUtil.processTemplate(RuntimePortletUtil.java:106)
at org.apache.jsp.html.portal.layout.view.portlet_jsp._jspService(portlet_jsp.java:512)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:72)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:70)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:593)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:530)
at com.liferay.portal.action.LayoutAction.includeLayoutContent(LayoutAction.java:412)
at com.liferay.portal.action.LayoutAction.processLayout(LayoutAction.java:680)
at com.liferay.portal.action.LayoutAction.execute(LayoutAction.java:244)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)
at com.liferay.portal.struts.PortalRequestProcessor.process(PortalRequestProcessor.java:174)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at com.liferay.portal.servlet.MainServlet.callParentService(MainServlet.java:533)
at com.liferay.portal.servlet.MainServlet.service(MainServlet.java:510)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:72)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:113)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:113)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.secure.SecureFilter.processFilter(SecureFilter.java:201)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.autologin.AutoLoginFilter.processFilter(AutoLoginFilter.java:240)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:70)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:471)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:402)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:329)
at com.liferay.portal.servlet.FriendlyURLServlet.service(FriendlyURLServlet.java:135)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:72)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.strip.StripFilter.processFilter(StripFilter.java:304)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.gzip.GZipFilter.processFilter(GZipFilter.java:123)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.secure.SecureFilter.processFilter(SecureFilter.java:201)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.i18n.I18nFilter.processFilter(I18nFilter.java:241)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.cache.CacheFilter.processFilter(CacheFilter.java:401)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.autologin.AutoLoginFilter.processFilter(AutoLoginFilter.java:240)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:70)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:471)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:402)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:329)
at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.processFilter(VirtualHostFilter.java:298)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:184)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:92)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:164)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:92)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:164)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:92)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:184)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:92)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:70)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)
- locked <0x00000000aebc3010> (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- <0x00000000ac90c278> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
I'm not an expert reading thread dumps, so I would appreciate your help in this. As I see, it's getting a problem with c3p0 connections, but I dont know why... Any idea of what can be the problem? I haven't changed any c3p0 config, so it's using the bundle's preconfigured options.
When the server gets blocked, I always have to restart tomcat manually, so it's a big problem, because it can happen any time and the portal can be stuck for a long time before I can restart it... Please help me with this. ANy ideas of what can I do to fix it?
Thanks!
I have a problem with Liferay. After several days running smoothly, java process (tomcat) suddenly gets stucked using almost 100% of CPU...I'm using the tomcat bundle. It's liferay-6.1.0-ce-ga1. I'm using MySQL 5.1.73. A thread dump shows several threads blocked, I think it shows several threads waiting to lock an object (<0x00000000b5ea4748>) like this one :
"liferay/buffered_increment_parallel-210" daemon prio=10 tid=0x00007fd168258800 nid=0x48e0 waiting for monitor entry [0x00007fd1704b4000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.mchange.v2.resourcepool.BasicResourcePool.checkinResource(BasicResourcePool.java:646)
- waiting to lock <0x00000000b5ea4748> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$ConnectionEventListenerImpl.doCheckinResource(C3P0PooledConnectionPool.java:636)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$ConnectionEventListenerImpl.connectionClosed(C3P0PooledConnectionPool.java:630)
at com.mchange.v2.c3p0.util.ConnectionEventSupport.fireConnectionClosed(ConnectionEventSupport.java:55)
at com.mchange.v2.c3p0.impl.NewPooledConnection.fireConnectionClosed(NewPooledConnection.java:510)
at com.mchange.v2.c3p0.impl.NewPooledConnection.markClosedProxyConnection(NewPooledConnection.java:381)
at com.mchange.v2.c3p0.impl.NewProxyConnection.close(NewProxyConnection.java:1246)
- locked <0x00000000b29eb048> (a com.mchange.v2.c3p0.impl.NewProxyConnection)
at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
at com.sun.proxy.$Proxy4.close(Unknown Source)
at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.closeConnection(LocalDataSourceConnectionProvider.java:95)
at org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:474)
at org.hibernate.jdbc.ConnectionManager.cleanup(ConnectionManager.java:408)
at org.hibernate.jdbc.ConnectionManager.close(ConnectionManager.java:347)
at org.hibernate.impl.SessionImpl.close(SessionImpl.java:343)
at org.springframework.orm.hibernate3.SessionFactoryUtils.closeSession(SessionFactoryUtils.java:791)
at org.springframework.orm.hibernate3.SessionFactoryUtils.closeSessionOrRegisterDeferredClose(SessionFactoryUtils.java:777)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doCleanupAfterCompletion(HibernateTransactionManager.java:737)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion(AbstractPlatformTransactionManager.java:1011)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:804)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:78)
at com.liferay.portal.increment.BufferedIncreasableEntry.proceed(BufferedIncreasableEntry.java:51)
at com.liferay.portal.increment.BufferedIncrementMessageListener.doReceive(BufferedIncrementMessageListener.java:49)
at com.liferay.portal.kernel.messaging.BaseMessageListener.receive(BaseMessageListener.java:25)
at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:65)
at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:106)
at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:669)
at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:580)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
And 3 other threads like this one where that object (<0x00000000b5ea4748>) is blocked:
"ajp-bio-8009-exec-3082" daemon prio=10 tid=0x00007fd168940800 nid=0x45ae in Object.wait() [0x00007fd1441ba000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
- locked <0x00000000b5ea4748> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at com.liferay.portal.dao.jdbc.util.DataSourceWrapper.getConnection(DataSourceWrapper.java:36)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:403)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
at com.sun.proxy.$Proxy4.prepareStatement(Unknown Source)
at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:534)
at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:452)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1700)
at org.hibernate.loader.Loader.doQuery(Loader.java:801)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.doList(Loader.java:2533)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
at org.hibernate.loader.Loader.list(Loader.java:2271)
at org.hibernate.hql.classic.QueryTranslatorImpl.list(QueryTranslatorImpl.java:940)
at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1268)
at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
at com.liferay.portal.dao.orm.hibernate.QueryImpl.list(QueryImpl.java:86)
at com.liferay.portal.dao.orm.hibernate.QueryImpl.list(QueryImpl.java:79)
at com.liferay.portal.kernel.dao.orm.QueryUtil.list(QueryUtil.java:60)
at com.liferay.portal.kernel.dao.orm.QueryUtil.list(QueryUtil.java:52)
at com.liferay.portal.service.persistence.ResourcePermissionPersistenceImpl.findByC_N_S_P_R(ResourcePermissionPersistenceImpl.java:3810)
at com.liferay.portal.service.persistence.ResourcePermissionPersistenceImpl.findByC_N_S_P_R(ResourcePermissionPersistenceImpl.java:3628)
at com.liferay.portal.service.impl.ResourcePermissionLocalServiceImpl.hasResourcePermission(ResourcePermissionLocalServiceImpl.java:709)
at com.liferay.portal.service.impl.ResourcePermissionLocalServiceImpl.hasResourcePermission(ResourcePermissionLocalServiceImpl.java:612)
at sun.reflect.GeneratedMethodAccessor335.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:112)
at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:71)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108)
at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:59)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108)
at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:59)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108)
at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:211)
at com.sun.proxy.$Proxy83.hasResourcePermission(Unknown Source)
at com.liferay.portal.service.impl.PermissionLocalServiceImpl.hasUserPermissions_6(PermissionLocalServiceImpl.java:1489)
at com.liferay.portal.service.impl.PermissionLocalServiceImpl.hasUserPermissions(PermissionLocalServiceImpl.java:724)
at sun.reflect.GeneratedMethodAccessor333.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:112)
at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:71)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108)
at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:59)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108)
at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:59)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108)
at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:211)
at com.sun.proxy.$Proxy54.hasUserPermissions(Unknown Source)
at com.liferay.portal.service.PermissionLocalServiceUtil.hasUserPermissions(PermissionLocalServiceUtil.java:644)
at com.liferay.portal.security.permission.AdvancedPermissionChecker.hasGuestPermission(AdvancedPermissionChecker.java:901)
at com.liferay.portal.security.permission.AdvancedPermissionChecker.hasPermissionImpl(AdvancedPermissionChecker.java:916)
at com.liferay.portal.security.permission.AdvancedPermissionChecker.hasPermission(AdvancedPermissionChecker.java:595)
at com.liferay.portal.service.permission.PortletPermissionImpl.contains(PortletPermissionImpl.java:303)
at com.liferay.portal.service.permission.PortletPermissionImpl.contains(PortletPermissionImpl.java:205)
at com.liferay.portal.service.permission.PortletPermissionImpl.contains(PortletPermissionImpl.java:164)
at com.liferay.portal.service.permission.PortletPermissionImpl.contains(PortletPermissionImpl.java:155)
at com.liferay.portal.service.permission.PortletPermissionUtil.contains(PortletPermissionUtil.java:116)
at org.apache.jsp.html.portal.render_005fportlet_jsp._jspService(render_005fportlet_jsp.java:485)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:72)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:70)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:593)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:530)
at com.liferay.portal.util.PortalImpl.renderPortlet(PortalImpl.java:4735)
at com.liferay.portal.util.PortalUtil.renderPortlet(PortalUtil.java:1455)
at com.liferay.portlet.layoutconfiguration.util.RuntimePortletImpl.processPortlet(RuntimePortletImpl.java:213)
at com.liferay.portlet.layoutconfiguration.util.RuntimePortletImpl.processPortlet(RuntimePortletImpl.java:148)
at com.liferay.portlet.layoutconfiguration.util.RuntimePortletImpl.processTemplate(RuntimePortletImpl.java:309)
at com.liferay.portlet.layoutconfiguration.util.RuntimePortletImpl.processTemplate(RuntimePortletImpl.java:234)
at com.liferay.portlet.layoutconfiguration.util.RuntimePortletUtil.processTemplate(RuntimePortletUtil.java:106)
at org.apache.jsp.html.portal.layout.view.portlet_jsp._jspService(portlet_jsp.java:512)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:72)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:70)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:593)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:530)
at com.liferay.portal.action.LayoutAction.includeLayoutContent(LayoutAction.java:412)
at com.liferay.portal.action.LayoutAction.processLayout(LayoutAction.java:680)
at com.liferay.portal.action.LayoutAction.execute(LayoutAction.java:244)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)
at com.liferay.portal.struts.PortalRequestProcessor.process(PortalRequestProcessor.java:174)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at com.liferay.portal.servlet.MainServlet.callParentService(MainServlet.java:533)
at com.liferay.portal.servlet.MainServlet.service(MainServlet.java:510)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:72)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:113)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:113)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.secure.SecureFilter.processFilter(SecureFilter.java:201)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.autologin.AutoLoginFilter.processFilter(AutoLoginFilter.java:240)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:70)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:471)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:402)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:329)
at com.liferay.portal.servlet.FriendlyURLServlet.service(FriendlyURLServlet.java:135)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:72)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.strip.StripFilter.processFilter(StripFilter.java:304)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.gzip.GZipFilter.processFilter(GZipFilter.java:123)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.secure.SecureFilter.processFilter(SecureFilter.java:201)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.i18n.I18nFilter.processFilter(I18nFilter.java:241)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.cache.CacheFilter.processFilter(CacheFilter.java:401)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:121)
at com.liferay.portal.servlet.filters.autologin.AutoLoginFilter.processFilter(AutoLoginFilter.java:240)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:70)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:471)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:402)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:329)
at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.processFilter(VirtualHostFilter.java:298)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:184)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:92)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:203)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:105)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:164)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:92)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:164)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:92)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:184)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:92)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:70)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)
- locked <0x00000000aebc3010> (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- <0x00000000ac90c278> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
I'm not an expert reading thread dumps, so I would appreciate your help in this. As I see, it's getting a problem with c3p0 connections, but I dont know why... Any idea of what can be the problem? I haven't changed any c3p0 config, so it's using the bundle's preconfigured options.
When the server gets blocked, I always have to restart tomcat manually, so it's a big problem, because it can happen any time and the portal can be stuck for a long time before I can restart it... Please help me with this. ANy ideas of what can I do to fix it?
Thanks!
David H Nebinger, geändert vor 9 Jahren.
RE: tomcat gets blocked using almost 100% of CPU after some days running
Liferay Legend Beiträge: 14915 Beitrittsdatum: 02.09.06 Neueste Beiträge
I'm not sure the locks are Liferay's fault, I think it's reflecting a possible issue on the database side. Any issues reported there?
Alan L., geändert vor 9 Jahren.
RE: tomcat gets blocked using almost 100% of CPU after some days running
New Member Beiträge: 8 Beitrittsdatum: 28.08.12 Neueste Beiträge
I haven't seen any issues there, but I'll watch it again next time it happens.
I haven't seen any related errors in mysqld.log... Any other specific check or place to look at that you suggest for mysql?
Thanks for your help
I haven't seen any related errors in mysqld.log... Any other specific check or place to look at that you suggest for mysql?
Thanks for your help
Christoph Rabel, geändert vor 9 Jahren.
RE: tomcat gets blocked using almost 100% of CPU after some days running
Liferay Legend Beiträge: 1554 Beitrittsdatum: 24.09.09 Neueste Beiträge
Hi,
Just wild guessing:
I had a problem once with Firewall dropping connections, leading to a "huge" list of open connections and subsequent hangs.
Maybe take a look with netstat -an if there are lots of connections open.
You could try to use mysql-tuner or tuning-primer. Those tools give hints about optimizing mysql. Maybe they point you in the right direction.
You could try to enable the slow query log:
https://dev.mysql.com/doc/refman/5.0/en/slow-query-log.html
Maybe there are some queries taking really long?
Just wild guessing:
I had a problem once with Firewall dropping connections, leading to a "huge" list of open connections and subsequent hangs.
Maybe take a look with netstat -an if there are lots of connections open.
You could try to use mysql-tuner or tuning-primer. Those tools give hints about optimizing mysql. Maybe they point you in the right direction.
You could try to enable the slow query log:
https://dev.mysql.com/doc/refman/5.0/en/slow-query-log.html
Maybe there are some queries taking really long?
Alan L., geändert vor 9 Jahren.
RE: tomcat gets blocked using almost 100% of CPU after some days running
New Member Beiträge: 8 Beitrittsdatum: 28.08.12 Neueste Beiträge
Hi Christoph, thank you very much for your comments and suggestions.
I've already enabled slow query log and I will have a look at it next time it happens.
And I'll have a look at netstat and also mysql-tuner and tuning-primer as you say.
I'm still open for any ideas anybody can have about possible causes and solutions for my problem.
Thanks
I've already enabled slow query log and I will have a look at it next time it happens.
And I'll have a look at netstat and also mysql-tuner and tuning-primer as you say.
I'm still open for any ideas anybody can have about possible causes and solutions for my problem.
Thanks
William Gosse, geändert vor 8 Jahren.
RE: tomcat gets blocked using almost 100% of CPU after some days running
Liferay Master Beiträge: 533 Beitrittsdatum: 04.07.10 Neueste BeiträgeAlan L.:
Hi Christoph, thank you very much for your comments and suggestions.
I've already enabled slow query log and I will have a look at it next time it happens.
And I'll have a look at netstat and also mysql-tuner and tuning-primer as you say.
I'm still open for any ideas anybody can have about possible causes and solutions for my problem.
Thanks
I have a similar issue on 6.2 GA4 with Postgres. See my other post on this issue: https://www.liferay.com/community/forums/-/message_boards/message/53171438