Performance + Authentication problems

[lou cat, módosítva 10 év-val korábban]

Hello all,

I have an existing big portal based on LR 6.0.6 and I'm experiencing some performance and authentication issues.

Regarding the performance, I've just noticed the site is incredibly slow in these last 2 days and I was wondering if this may be caused by the growing number of users (they are around 1000 at the moment) and documents.
How can I assess this? Any suggestion on how to improve the performance? I guess disabling the live monitor could help (although when I checked only 5 users were online.....), other easy ways?

Regarding the authentication, apparently many times the users receive an "internal server error" even if the password they use to login is correct.
After 2-3 attempts they manage to log in, but how come this happens?

Any suggestion on how to analyse and fix this situation would be really helpful, thanks!!!

[Mika Koivisto, módosítva 10 év-val korábban]

If anyone is going to help you they are going to need more information. You don't tell what kind of environment you have also any stacktraces with the internal server error.

[lou cat, módosítva 10 év-val korábban]

Oh you're right Mika, I wrote this very quickly and forgot to add more info!

The LR6.0.6. portal is under Tomcat (OpenSuse linux), and hosted in Amazon Ec2 (not a microinstance), database is postgreSQL, authentication should be the default one and I'm not using any LDAP or other things. Document Library is the default one and is accessed through webdav, and unfortunately I don't have any stacktrace for the authentication error as it never happened to me and the users informed me 2 weeks after it happened, they just have a screenshot of their error (which mention the login struts action in the url, as far as I remember).

Any suggestion would be really helpful! thanks!

[lou cat, módosítva 10 év-val korábban]

Other info:

The server (which contains both tomcat and the database) has a total of 8GB Ram, which the portal is using extensively, from the Control Panel I can often read this values:
Used Memory: 677,075,008 Bytes
Total Memory: 937,689,088 Bytes
Maximum Memory: 954,466,304 Bytes

Java version: "1.6.0_45"
PostgreSQL version: 8.3.21
Tomcat version is the one bundled with Liferay 6.0.6 CE: 6.0.29

I read some posts in the forum but most of them are very old, I can find people suggesting > 1GB Ram as well as people using 16GB, so I'm not really sure of what is recommended for a production version that has a big document library and more than 1000 users, with usually not more than 10 logged in users at the same time, but in busy times can have up to 200 concurrent guest users.

To me 8GB seems more than enough, we could buy more ram but I have the feeling there's a memory leak somewhere or some other problem.

I am about to restart the server so if anyone has suggestions on these matters:

- portal.properties settings
- setenv.sh settings
(current one is "JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=UTF8 -Duser.timezone=Europe/London -Xmx1024m -XX:MaxPermSize=256m")
- free monitoring tools
- any useful tweak to reduce memory usage (for example is the "live monitoring" a lot memory-consuming?)
- any other related recommendation

please give me a shout

[lou cat, módosítva 10 év-val korábban]

ok I believe it's a memory setting issue but at the same time we will buy more ram and we will add a swap space on ec2, so in total we will have 16GB of ram.
Do you have recommendation on the setenv.sh setting (related to java memory) to use?

[gofri _, módosítva 10 év-val korábban]

Maximum Memory: 954,466,304 Bytes

This looks too low, even for basic setups. Check the administration guide.
We are using Xms/Xmx at least 2GB/2GB, better more (probably 4G/4G).

Also, updating Java to latest version will not harm (although will not resolve the problem too).

[lou cat, módosítva 10 év-val korábban]

gofri _:

Maximum Memory: 954,466,304 Bytes

This looks too low, even for basic setups. Check the administration guide.
We are using Xms/Xmx at least 2GB/2GB, better more (probably 4G/4G).

Also, updating Java to latest version will not harm (although will not resolve the problem too).

Thanks Gofri for replying, I was suspecting I had to tweak my java memory settings and already having a look at the admin guide.
Do you think buying more Ram won't be necessary?

I actually had to downgrade the java version because of some well known problem with Liferay and Java7 (especially with Lucene indexing, as far as I remember)

[gofri _, módosítva 10 év-val korábban]

8GB should be enough, of course, depends on other services on this server (what amount is free from these 8G).
You need to provide more memory to tomcat, currently as Vilmos said, You are using only 1GB. Having 8GB and using 1 is the same as having 16 and using 1.

[Vilmos Papp, módosítva 10 év-val korábban]

Hi,

I think you should take a look at the preformance tuning documentation here: https://www.liferay.com/documentation/liferay-portal/6.0/administration/-/ai/performance-tuni-4

From your settings I suspect, that you don't use most of the memory of your machine as JVM is set to use only 1GB ant that's not for production use.

Further more, could you please let us know about the use case when the error happens?

Regards,
Vilmos

[lou cat, módosítva 10 év-val korábban]

Vilmos Papp:

Hi,

I think you should take a look at the preformance tuning documentation here: https://www.liferay.com/documentation/liferay-portal/6.0/administration/-/ai/performance-tuni-4

From your settings I suspect, that you don't use most of the memory of your machine as JVM is set to use only 1GB ant that's not for production use.

Further more, could you please let us know about the use case when the error happens?

Regards,
Vilmos

Thank you Vilmos!
I'll have a further look at the admin guide and will try to find the best settings for us (we have 8GB ram at the moment but I guess we'll upgrade to 16GB anyway).

Regarding the authentication problem, what happens is that the first login of the day gives an internal server error, and just after one or 2 times the user is able to login.
From the screenshot I can see they were using IE and in the url there is "exception=java.lang.InternalS"
(java.lang.illegalStateException maybe? sorry but I can just guess at the moment...)
I just have this screenshot and didn't find anything in the catalina.out log file, but I suspect is a problem of session (I'm currently having problems where liferay closes my session much before the 30minutes, or the "remember me" checkbox doesn't work in production while it does in my internal server which has the same settings).
If I get more information from the users, I'll post it here (or is it better on another thread?)

Thanks again!

[lou cat, módosítva 10 év-val korábban]

lou cat:

Hello all,

I have an existing big portal based on LR 6.0.6 and I'm experiencing some performance and authentication issues.

Regarding the performance [...]

thank you all, I've changed the java settings in setenv.sh and it seems going fine now.

lou cat:

Regarding the authentication, apparently many times the users receive an "internal server error" even if the password they use to login is correct.
After 2-3 attempts they manage to log in, but how come this happens?

Any suggestion on how to analyse and fix this situation would be really helpful, thanks!!!

I finally have more information on the authentication problem:

this is the URL of the internal server error:

http://www.MYDOMAIN.com/c/portal/status?status=0&exception=java.lang.IllegalStateException&previousURL=%2Fen%2Fcontext-portal-login%3Fp_auth%3D540NHoP3%26p_p_id%3D58%26p_p_lifecycle%3D1%26p_p_state%3Dnormal%26p_p_mode%3Dview%26p_p_col_id%3Dcolumn-1%26p_p_col_count%3D1%26saveLastPath%3D0%26_58_struts_action%3D%252Flogin%252Flogin

and this is what is shown in the page:

An error occurred while accessing the requested resource.

http://www.MYDOMAIN.com/en/context-portal-login?p_auth=540NHoP3&p_p_id=58&p_p_lifecycle=1&p_p_state=normal&p_p_mode=view&p_p_col_id=column-1&p_p_col_count=1&saveLastPath=0&_58_struts_action=%2Flogin%2Flogin

This happened twice, the third time it worked.
She was using IE.

I've found this user has the same problems, but no solutions for him as well

https://www.liferay.com/community/forums/-/message_boards/message/27996525

[gofri _, módosítva 10 év-val korábban]

not sure, but AFAIK there was some related bug in 6.0.6. We set
auth.token.check.enabled=false
in portal-ext.properties
to bypass it. But the behavior was different, so not sure it will help here.

[lou cat, módosítva 10 év-val korábban]

Thank you gofri, I'll have a look!

I'm not sure of what this property does but I believe in the future we may use authentication tokens...
this problem appears randomly and we noticed it just in the production server, so it would be really difficult to investigate 'cause I can't change properties there and restart the server often :/