Jmeter System Preferences Lock Error

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Jmeter System Preferences Lock Error

Lynch Kieran
Hi guys,

Potentially a strange one for you... I'm running a test on an EC2 instance with 64G available memory and jmeter's heap sized appropriately (currently Xmx40g) configured for 6K concurrent users and 200TPS. When the test reaches 4k users I get the following error message output:

WARNING: Could not lock User prefs.  Unix error code 24.
Nov 05, 2019 7:49:00 PM java.util.prefs.FileSystemPreferences syncWorld
WARNING: Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.

And in my jmeter.log a stream of these:

2019-11-05 19:47:44,512 ERROR o.a.j.c.CSVDataSet: java.io.IOException: Stream closed
2019-11-05 19:47:44,612 ERROR o.a.j.c.CSVDataSet: java.io.IOException: Stream closed
2019-11-05 19:47:44,612 ERROR o.a.j.c.CSVDataSet: java.io.IOException: Stream closed
2019-11-05 19:47:44,612 ERROR o.a.j.c.CSVDataSet: java.io.IOException: Stream closed

Test errors are out of control from this point onwards (60 - 70%) despite application health looking good - I'm assuming because jmeter is unable to send CSV parameterised requests?

I did a little digging and it seems to be a Java install specific issue? Although I find it strange that I get these errors at 4K threads every time. I am running jmeter as root and my Java system preferences file is writable for all:

lrwxrwxrwx  1 root root   35 Nov  5 18:15 .systemPrefs -> /usr/java/latest/.java/.systemPrefs

java -version
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)

echo $JAVA_HOME
/usr/java/jre1.8.0_231-amd64

Any help or info from someone who has experienced this before greatly appreciated.

Kind Regards,
Kieran


Kieran Lynch

Portaltech Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Jmeter System Preferences Lock Error

glinius@live.com
Looking into  JMeter Bug 61279
<https://bz.apache.org/bugzilla/show_bug.cgi?id=61279>   it isn't something
connected with JMeter, I can think of the following reasons for the
behaviour:1. You have more than 1  JVM
<https://en.wikipedia.org/wiki/Java_virtual_machine>  running on the EC2
machine and these JVMs are suffering from a form of a  race condition
<https://en.wikipedia.org/wiki/Race_condition>   when trying to obtain the
lock on the same file to store the preferences2. There is a problem with
Linux file permissions and JVM, i.e. you're running out of  free handles
<https://www.cyberciti.biz/tips/linux-procfs-file-descriptors.html>  The
solution for point 1 would be explicitly specifying the store for
systemRoot
<https://docs.oracle.com/javase/7/docs/api/java/util/prefs/PreferencesFactory.html#systemRoot()>
and  userRoot
<https://docs.oracle.com/javase/7/docs/api/java/util/prefs/PreferencesFactory.html#userRoot()>
for the JVM which hosts JMeter so it would use the separate preferences
storage not shared with other JVMs. The properties can be defined either via
-D command-line argument
<https://jmeter.apache.org/usermanual/get-started.html#override>   like:
> ./jmeter -Djava.util.prefs.systemRoot=/some/folder/.jmeter
> -Djava.util.prefs.userRoot=/some/folder/.jmeter/.userPrefs

In order to make the changes persistent you can add the same lines to
/system.properties/ file (lives in "bin" folder of your JMeter installation)
> java.util.prefs.systemRoot=/some/folder/.jmeterjava.util.prefs.userRoot=/some/folder/.jmeter/.userPrefs

More information:  Apache JMeter Properties Customization Guide
<https://www.blazemeter.com/blog/apache-jmeter-properties-customization/>
In case of point 2 I would recommend checking / raising  Linux process
limits <https://gerardnico.com/os/linux/limits.conf>  



--
Sent from: http://www.jmeter-archive.org/JMeter-User-f512775.html
Reply | Threaded
Open this post in threaded view
|

Re: Jmeter System Preferences Lock Error

Lynch Kieran
Hi glinius,

Appreciate the detailed response and links.

You were correct with your second hypothesis, turns out there was a hard limit of 4096 for file descriptors on the EC2 instance. I edited /etc/security/limits.conf and added the following two lines:

root               soft    nofile          6114
root               hard    nofile          12288

and that appears to have resolved the issue.

Thanks again for saving me!

Kieran






Kieran Lynch

Portaltech Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com



From: [hidden email] <[hidden email]>
Sent: 07 November 2019 09:38
To: [hidden email] <[hidden email]>
Subject: Re: Jmeter System Preferences Lock Error
 
Looking into  JMeter Bug 61279
<https://bz.apache.org/bugzilla/show_bug.cgi?id=61279>   it isn't something
connected with JMeter, I can think of the following reasons for the
behaviour:1. You have more than 1  JVM
<https://en.wikipedia.org/wiki/Java_virtual_machine>  running on the EC2
machine and these JVMs are suffering from a form of a  race condition
<https://en.wikipedia.org/wiki/Race_condition>   when trying to obtain the
lock on the same file to store the preferences2. There is a problem with
Linux file permissions and JVM, i.e. you're running out of  free handles
<https://www.cyberciti.biz/tips/linux-procfs-file-descriptors.html>  The
solution for point 1 would be explicitly specifying the store for
systemRoot
<https://docs.oracle.com/javase/7/docs/api/java/util/prefs/PreferencesFactory.html#systemRoot()>
and  userRoot
<https://docs.oracle.com/javase/7/docs/api/java/util/prefs/PreferencesFactory.html#userRoot()>
for the JVM which hosts JMeter so it would use the separate preferences
storage not shared with other JVMs. The properties can be defined either via
-D command-line argument
<https://jmeter.apache.org/usermanual/get-started.html#override>   like:
> ./jmeter -Djava.util.prefs.systemRoot=/some/folder/.jmeter
> -Djava.util.prefs.userRoot=/some/folder/.jmeter/.userPrefs

In order to make the changes persistent you can add the same lines to
/system.properties/ file (lives in "bin" folder of your JMeter installation)
> java.util.prefs.systemRoot=/some/folder/.jmeterjava.util.prefs.userRoot=/some/folder/.jmeter/.userPrefs

More information:  Apache JMeter Properties Customization Guide
<https://www.blazemeter.com/blog/apache-jmeter-properties-customization/>
In case of point 2 I would recommend checking / raising  Linux process
limits <https://gerardnico.com/os/linux/limits.conf>  



--
Sent from: http://www.jmeter-archive.org/JMeter-User-f512775.html


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]