Warming up Java servers before measuring performance

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Warming up Java servers before measuring performance

Isuru Perera
Hi,

We are using JMeter to run load tests on our Java servers and we want to
warm-up the application before taking measurements.

For example, I want to have 10 minutes as the warm up time and 20 minutes
as the measurement time.

I see two approaches to this.

1. Run JMeter for 30 minutes and exclude first 10 minutes data in the JTL
file and generate reports.
2. Run JMeter for 10 minutes and run again for 20 minutes. Generate reports
from 20 minutes JTL file.

We prefer number 1, as the Java server will be in a steady state when
taking measurements. However for approach 1, we need to find a way to
exclude first few minutes of data. We can do this with a simple program.

Number 2 approach may not be useful as we are restarting the JMeter and it
will take some time get a steady throughput.

I searched the web and I found following Stackoverflow question.
http://stackoverflow.com/questions/32237837/how-to-exclude-warmup-time-from-jmeter-summary

With GraphsGeneratorListener
<http://jmeter-plugins.org/wiki/GraphsGeneratorListener/>, it seems that I
can only generate reports. Is there any other way to just exclude first few
minutes in the JTL and then use the JTL to generate reports (for example,
JMeter Dashboard). Is that listener the only standard way to exclude
warm-up time in JMeter?

Why don't I see anything mentioned about Java Warm-up time in official
docs? Since JMeter is based on Java, I expected some guidelines to load
test Java servers and how to exclude Java warm-up time. I think we should
also consider the warm-up time of JMeter. With approach 1, we can also make
sure that JMeter is also performing at its best when taking the
measurements.

I really appreciate your feedback on this.

Thank you.

Best Regards,
--
Isuru Perera
about.me/chrishantha
Reply | Threaded
Open this post in threaded view
|

Re: Warming up Java servers before measuring performance

glinius@live.com
This post has NOT been accepted by the mailing list yet.
You can use Filter Results Tool which provides start-offset and end-offset command-line options so you can precisely set the range you are interested in and focus on the steady phase analysis.

You can install Filter Results Tool using JMeter Plugins Manager:



Upon installation you will see FilterResults.bat and FilterResults.sh scripts for Windows and Linux/Unix/MacOSX operating systems correspondingly.

Reply | Threaded
Open this post in threaded view
|

Re: Warming up Java servers before measuring performance

shettyd
In reply to this post by Isuru Perera
Hi
Your reporting engine of choice will usually support this filter. For most
non trivial load tests you have to analyze the data in either a DB or
something like graphite etc - Almost all of which will trivially allow you
to filter out time frames you dont want.
For out of box listeners , yes you could write a simple program to filter
values from JTL.

Ive not found JMeter *warmup* to be a significant issue to matter in most
normal load tests (i.e. those that run for atleast an hour). The
application itself is a different matter but thats very application
specific.(And sometimes *warmup* hides issues)

On Wed, May 17, 2017 at 11:48 PM, Isuru Perera <[hidden email]>
wrote:

> Hi,
>
> We are using JMeter to run load tests on our Java servers and we want to
> warm-up the application before taking measurements.
>
> For example, I want to have 10 minutes as the warm up time and 20 minutes
> as the measurement time.
>
> I see two approaches to this.
>
> 1. Run JMeter for 30 minutes and exclude first 10 minutes data in the JTL
> file and generate reports.
> 2. Run JMeter for 10 minutes and run again for 20 minutes. Generate reports
> from 20 minutes JTL file.
>
> We prefer number 1, as the Java server will be in a steady state when
> taking measurements. However for approach 1, we need to find a way to
> exclude first few minutes of data. We can do this with a simple program.
>
> Number 2 approach may not be useful as we are restarting the JMeter and it
> will take some time get a steady throughput.
>
> I searched the web and I found following Stackoverflow question.
> http://stackoverflow.com/questions/32237837/how-to-
> exclude-warmup-time-from-jmeter-summary
>
> With GraphsGeneratorListener
> <http://jmeter-plugins.org/wiki/GraphsGeneratorListener/>, it seems that I
> can only generate reports. Is there any other way to just exclude first few
> minutes in the JTL and then use the JTL to generate reports (for example,
> JMeter Dashboard). Is that listener the only standard way to exclude
> warm-up time in JMeter?
>
> Why don't I see anything mentioned about Java Warm-up time in official
> docs? Since JMeter is based on Java, I expected some guidelines to load
> test Java servers and how to exclude Java warm-up time. I think we should
> also consider the warm-up time of JMeter. With approach 1, we can also make
> sure that JMeter is also performing at its best when taking the
> measurements.
>
> I really appreciate your feedback on this.
>
> Thank you.
>
> Best Regards,
> --
> Isuru Perera
> about.me/chrishantha
>
Reply | Threaded
Open this post in threaded view
|

Re: Warming up Java servers before measuring performance

Isuru Perera
Hi Deepak,

Thanks a lot for the reply.

We are just using out of the box listeners for now.

I agree that the excluding the warmup time may hide the issues.

However, I believe that warming up the Java server is very important
especially when considering the JIT compilation in HotSpot JVM. JMH (
http://openjdk.java.net/projects/code-tools/jmh/) tool also warms up the
methods before taking measurements.

That's why I wanted to ask in this list whether other users have the same
requirements to exclude warmup time from the final results.

We can write a simple program to split the JTL, which will write warmup and
measurement requests to two different file. In this way, we don't
completely ignore the warmup time and we can compare the warmup results
with the actual measurement results.

Splitting the JTL seemed to be an additional work and we have to process
large files again after the tests. I was wondering why JMeter didn't
provide such a feature out of the box similar to JMH.

Thanks!

Best Regards,


On Fri, May 19, 2017 at 11:37 PM, Deepak Shetty <[hidden email]> wrote:

> Hi
> Your reporting engine of choice will usually support this filter. For most
> non trivial load tests you have to analyze the data in either a DB or
> something like graphite etc - Almost all of which will trivially allow you
> to filter out time frames you dont want.
> For out of box listeners , yes you could write a simple program to filter
> values from JTL.
>
> Ive not found JMeter *warmup* to be a significant issue to matter in most
> normal load tests (i.e. those that run for atleast an hour). The
> application itself is a different matter but thats very application
> specific.(And sometimes *warmup* hides issues)
>
> On Wed, May 17, 2017 at 11:48 PM, Isuru Perera <[hidden email]>
> wrote:
>
> > Hi,
> >
> > We are using JMeter to run load tests on our Java servers and we want to
> > warm-up the application before taking measurements.
> >
> > For example, I want to have 10 minutes as the warm up time and 20 minutes
> > as the measurement time.
> >
> > I see two approaches to this.
> >
> > 1. Run JMeter for 30 minutes and exclude first 10 minutes data in the JTL
> > file and generate reports.
> > 2. Run JMeter for 10 minutes and run again for 20 minutes. Generate
> reports
> > from 20 minutes JTL file.
> >
> > We prefer number 1, as the Java server will be in a steady state when
> > taking measurements. However for approach 1, we need to find a way to
> > exclude first few minutes of data. We can do this with a simple program.
> >
> > Number 2 approach may not be useful as we are restarting the JMeter and
> it
> > will take some time get a steady throughput.
> >
> > I searched the web and I found following Stackoverflow question.
> > http://stackoverflow.com/questions/32237837/how-to-
> > exclude-warmup-time-from-jmeter-summary
> >
> > With GraphsGeneratorListener
> > <http://jmeter-plugins.org/wiki/GraphsGeneratorListener/>, it seems
> that I
> > can only generate reports. Is there any other way to just exclude first
> few
> > minutes in the JTL and then use the JTL to generate reports (for example,
> > JMeter Dashboard). Is that listener the only standard way to exclude
> > warm-up time in JMeter?
> >
> > Why don't I see anything mentioned about Java Warm-up time in official
> > docs? Since JMeter is based on Java, I expected some guidelines to load
> > test Java servers and how to exclude Java warm-up time. I think we should
> > also consider the warm-up time of JMeter. With approach 1, we can also
> make
> > sure that JMeter is also performing at its best when taking the
> > measurements.
> >
> > I really appreciate your feedback on this.
> >
> > Thank you.
> >
> > Best Regards,
> > --
> > Isuru Perera
> > about.me/chrishantha
> >
>



--
Isuru Perera
about.me/chrishantha
Reply | Threaded
Open this post in threaded view
|

Re: Warming up Java servers before measuring performance

Bo Bodnar-2

The experiments you're doing are basically Monte Carlo discrete event simulations - except that you're doing "hardware in the loop" instead of a mathematical abstraction using a tool like ARENA, Simpy, etc. In each case you have two choices:

  1. Ignore the warm-up time and do a very long run (this will mask the transient startup behavior). I sometimes do this when I'm pressed for time or lazy.
  2. Put in a warm-up time.

Figuring out the appropriate warm-up time is not trivial. In my case, I wait for the system to become "wide-sense stationary"; that is, the mean and variance stabilize. An alternative (favored by our local statistician) is to look at the time series (we look at latency) and see when then the time series appears to stabilize.

Wide-sense stationarity time can be estimated by using an online algorithm such as Welford's algorithm; it's around a dozen lines of Python or C code and doesn't require storing all data points (it is an unbiased estimator and computes mean and variance on-the-fly). One then looks at the difference between the current mean estimate and the previous mean estimate and compares the differences to a predetermined reference (I use a relative absolute value percent difference). This is somewhat subjective.

Both items 1 and 2 assume the system being measured doesn't change behavior with respect to time; as long as the system is not running under heavy load, this will probably be true.

There's at least one other approach I've used but that's really non-trivial and is best left for another time.

Best regards,

Bo


Bohdan L. Bodnar
Lead Performance Engineer
1-312-871-5163

E-mail: [hidden email]
222 South Riverside Plaza
Chicago, IL 60606
United States

Isuru Perera ---05/24/2017 01:37:47 AM---Hi Deepak, Thanks a lot for the reply.

From: Isuru Perera <[hidden email]>
To: JMeter Users List <[hidden email]>, Deepak Shetty <[hidden email]>
Date: 05/24/2017 01:37 AM
Subject: Re: Warming up Java servers before measuring performance




Hi Deepak,

Thanks a lot for the reply.

We are just using out of the box listeners for now.

I agree that the excluding the warmup time may hide the issues.

However, I believe that warming up the Java server is very important
especially when considering the JIT compilation in HotSpot JVM. JMH (
http://openjdk.java.net/projects/code-tools/jmh/) tool also warms up the
methods before taking measurements.

That's why I wanted to ask in this list whether other users have the same
requirements to exclude warmup time from the final results.

We can write a simple program to split the JTL, which will write warmup and
measurement requests to two different file. In this way, we don't
completely ignore the warmup time and we can compare the warmup results
with the actual measurement results.

Splitting the JTL seemed to be an additional work and we have to process
large files again after the tests. I was wondering why JMeter didn't
provide such a feature out of the box similar to JMH.

Thanks!

Best Regards,


On Fri, May 19, 2017 at 11:37 PM, Deepak Shetty <[hidden email]> wrote:

> Hi
> Your reporting engine of choice will usually support this filter. For most
> non trivial load tests you have to analyze the data in either a DB or
> something like graphite etc - Almost all of which will trivially allow you
> to filter out time frames you dont want.
> For out of box listeners , yes you could write a simple program to filter
> values from JTL.
>
> Ive not found JMeter *warmup* to be a significant issue to matter in most
> normal load tests (i.e. those that run for atleast an hour). The
> application itself is a different matter but thats very application
> specific.(And sometimes *warmup* hides issues)
>
> On Wed, May 17, 2017 at 11:48 PM, Isuru Perera <[hidden email]>
> wrote:
>
> > Hi,
> >
> > We are using JMeter to run load tests on our Java servers and we want to
> > warm-up the application before taking measurements.
> >
> > For example, I want to have 10 minutes as the warm up time and 20 minutes
> > as the measurement time.
> >
> > I see two approaches to this.
> >
> > 1. Run JMeter for 30 minutes and exclude first 10 minutes data in the JTL
> > file and generate reports.
> > 2. Run JMeter for 10 minutes and run again for 20 minutes. Generate
> reports
> > from 20 minutes JTL file.
> >
> > We prefer number 1, as the Java server will be in a steady state when
> > taking measurements. However for approach 1, we need to find a way to
> > exclude first few minutes of data. We can do this with a simple program.
> >
> > Number 2 approach may not be useful as we are restarting the JMeter and
> it
> > will take some time get a steady throughput.
> >
> > I searched the web and I found following Stackoverflow question.
> >
http://stackoverflow.com/questions/32237837/how-to-
> > exclude-warmup-time-from-jmeter-summary
> >
> > With GraphsGeneratorListener
> > <
http://jmeter-plugins.org/wiki/GraphsGeneratorListener/>, it seems
> that I
> > can only generate reports. Is there any other way to just exclude first
> few
> > minutes in the JTL and then use the JTL to generate reports (for example,
> > JMeter Dashboard). Is that listener the only standard way to exclude
> > warm-up time in JMeter?
> >
> > Why don't I see anything mentioned about Java Warm-up time in official
> > docs? Since JMeter is based on Java, I expected some guidelines to load
> > test Java servers and how to exclude Java warm-up time. I think we should
> > also consider the warm-up time of JMeter. With approach 1, we can also
> make
> > sure that JMeter is also performing at its best when taking the
> > measurements.
> >
> > I really appreciate your feedback on this.
> >
> > Thank you.
> >
> > Best Regards,
> > --
> > Isuru Perera
> > about.me/chrishantha
> >
>



--
Isuru Perera
about.me/chrishantha



Reply | Threaded
Open this post in threaded view
|

Re: Warming up Java servers before measuring performance

shettyd
In reply to this post by Isuru Perera
>However, I believe that warming up the Java server is very important
especially when considering the JIT compilation in HotSpot JVM
In general , yes. But the question is does it apply to you(referring
specifically to the JMeter part of warm up) ? What are your typical
response times ? As well as what are the numbers you are looking at (We
usually use percentiles(90/95) rather than max which would eliminate
outliers in any case. Or to put it in another way are you really seeing a
difference in your response times if you "warm up" Jmeter v/s if you dont ?

Note that "warm up" for the application server is quite different - because
things like caches (both at application and DB layers) will usually drown
out any JVM warm up unless you have very very low response times or you
have highly specialised low latency type of applications.

On Tue, May 23, 2017 at 11:37 PM, Isuru Perera <[hidden email]>
wrote:

> Hi Deepak,
>
> Thanks a lot for the reply.
>
> We are just using out of the box listeners for now.
>
> I agree that the excluding the warmup time may hide the issues.
>
> However, I believe that warming up the Java server is very important
> especially when considering the JIT compilation in HotSpot JVM. JMH (
> http://openjdk.java.net/projects/code-tools/jmh/) tool also warms up the
> methods before taking measurements.
>
> That's why I wanted to ask in this list whether other users have the same
> requirements to exclude warmup time from the final results.
>
> We can write a simple program to split the JTL, which will write warmup
> and measurement requests to two different file. In this way, we don't
> completely ignore the warmup time and we can compare the warmup results
> with the actual measurement results.
>
> Splitting the JTL seemed to be an additional work and we have to process
> large files again after the tests. I was wondering why JMeter didn't
> provide such a feature out of the box similar to JMH.
>
> Thanks!
>
> Best Regards,
>
>
> On Fri, May 19, 2017 at 11:37 PM, Deepak Shetty <[hidden email]> wrote:
>
>> Hi
>> Your reporting engine of choice will usually support this filter. For most
>> non trivial load tests you have to analyze the data in either a DB or
>> something like graphite etc - Almost all of which will trivially allow you
>> to filter out time frames you dont want.
>> For out of box listeners , yes you could write a simple program to filter
>> values from JTL.
>>
>> Ive not found JMeter *warmup* to be a significant issue to matter in most
>> normal load tests (i.e. those that run for atleast an hour). The
>> application itself is a different matter but thats very application
>> specific.(And sometimes *warmup* hides issues)
>>
>> On Wed, May 17, 2017 at 11:48 PM, Isuru Perera <[hidden email]>
>> wrote:
>>
>> > Hi,
>> >
>> > We are using JMeter to run load tests on our Java servers and we want to
>> > warm-up the application before taking measurements.
>> >
>> > For example, I want to have 10 minutes as the warm up time and 20
>> minutes
>> > as the measurement time.
>> >
>> > I see two approaches to this.
>> >
>> > 1. Run JMeter for 30 minutes and exclude first 10 minutes data in the
>> JTL
>> > file and generate reports.
>> > 2. Run JMeter for 10 minutes and run again for 20 minutes. Generate
>> reports
>> > from 20 minutes JTL file.
>> >
>> > We prefer number 1, as the Java server will be in a steady state when
>> > taking measurements. However for approach 1, we need to find a way to
>> > exclude first few minutes of data. We can do this with a simple program.
>> >
>> > Number 2 approach may not be useful as we are restarting the JMeter and
>> it
>> > will take some time get a steady throughput.
>> >
>> > I searched the web and I found following Stackoverflow question.
>> > http://stackoverflow.com/questions/32237837/how-to-
>> > exclude-warmup-time-from-jmeter-summary
>> >
>> > With GraphsGeneratorListener
>> > <http://jmeter-plugins.org/wiki/GraphsGeneratorListener/>, it seems
>> that I
>> > can only generate reports. Is there any other way to just exclude first
>> few
>> > minutes in the JTL and then use the JTL to generate reports (for
>> example,
>> > JMeter Dashboard). Is that listener the only standard way to exclude
>> > warm-up time in JMeter?
>> >
>> > Why don't I see anything mentioned about Java Warm-up time in official
>> > docs? Since JMeter is based on Java, I expected some guidelines to load
>> > test Java servers and how to exclude Java warm-up time. I think we
>> should
>> > also consider the warm-up time of JMeter. With approach 1, we can also
>> make
>> > sure that JMeter is also performing at its best when taking the
>> > measurements.
>> >
>> > I really appreciate your feedback on this.
>> >
>> > Thank you.
>> >
>> > Best Regards,
>> > --
>> > Isuru Perera
>> > about.me/chrishantha
>> >
>>
>
>
>
> --
> Isuru Perera
> about.me/chrishantha
>