Best way to compare two results of jmeter

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Best way to compare two results of jmeter

Marek Czernek
Hi there,

is there any 'supported' way to compare the results of 2 jmeter runs? I
googled around and found an old email from 2004 [1] basically saying
that there is no recommended solution other than a custom-made analysis.
Have there been any solutions to this problem?

I can also see a Jenkins plugin [2] though I have no idea in what state
the plugin is, and as such, how viable it is to run it. Last but not
least, there's some Grafana integration blog [3].  Does anyone have any
other suggestions? How would you compare two results programmatically to
see if there is degraded performance?

[1]
http://mail-archives.apache.org/mod_mbox/jmeter-user/200401.mbox/%3C99805B014A26D211BC3100A0C9B72CB8059E28EA@...%3E

[2] https://wiki.jenkins.io/display/JENKINS/Performance+Plugin

[3]
http://www.testautomationguru.com/jmeter-real-time-results-influxdb-grafana/

Cheers,
--

Marek Czernek

JWS/JBCS Associate Quality Engineer, RHCA

Find me at www.halfastack.com

Reply | Threaded
Open this post in threaded view
|

Re: Best way to compare two results of jmeter

Bo Bodnar-2

I've been away from jmeter for a while. Working from memory:

The scripts I used had functional breakdown with the topic of interest being latency. Each functional area had latency samples associated with it. For example "login" for run1 would have X samples and the same was for run2. It is important that measurement conditions be identical for each run; otherwise, the performance of interest may be affected by uncontrolled conditions (e.g., time-dependent internet delays). It is also VERY important that the system of interest be in exactly the same condition when the runs are done; if not, there may be memory impacts. I then did paired t-testing to see if the runs were statistically similar (I think I used a confidence interval of 0.95). X should be greater than around 10; otherwise, there may be too much variability in the measurements and you may have false conclusions.

Bohdan L. Bodnar
Lead Performance Engineer
1-312-871-5163

E-mail: [hidden email]
222 South Riverside Plaza
Chicago, IL 60606
United States

Marek Czernek ---08/30/2018 09:11:19 AM---Hi there, is there any 'supported' way to compare the results of 2 jmeter runs? I

From: Marek Czernek <[hidden email]>
To: [hidden email]
Date: 08/30/2018 09:11 AM
Subject: Best way to compare two results of jmeter




Hi there,

is there any 'supported' way to compare the results of 2 jmeter runs? I
googled around and found an old email from 2004 [1] basically saying
that there is no recommended solution other than a custom-made analysis.
Have there been any solutions to this problem?

I can also see a Jenkins plugin [2] though I have no idea in what state
the plugin is, and as such, how viable it is to run it. Last but not
least, there's some Grafana integration blog [3].  Does anyone have any
other suggestions? How would you compare two results programmatically to
see if there is degraded performance?

[1]
http://mail-archives.apache.org/mod_mbox/jmeter-user/200401.mbox/%3C99805B014A26D211BC3100A0C9B72CB8059E28EA@...%3E

[2]
https://wiki.jenkins.io/display/JENKINS/Performance+Plugin

[3]
http://www.testautomationguru.com/jmeter-real-time-results-influxdb-grafana/

Cheers,
--

Marek Czernek

JWS/JBCS Associate Quality Engineer, RHCA

Find me at
www.halfastack.com




Reply | Threaded
Open this post in threaded view
|

Re: Best way to compare two results of jmeter

Alexander Podelko
In reply to this post by Marek Czernek
Hi Marek,
I do use the Jenkins Performance Plugin for that purpose for some time, I'd say that you get quite a lot for free straight out of the box. Pretty decent. Another thing is that I haven't found practically any documentation (although it is pretty straightforward for a simple use - and there is a few posts how to setup it) and it is still not clear for me what to do if I'd need something else from it....

Regards,Alex


   On Thursday, August 30, 2018, 10:11:22 AM EDT, Marek Czernek <[hidden email]> wrote:  
 
 Hi there,

is there any 'supported' way to compare the results of 2 jmeter runs? I
googled around and found an old email from 2004 [1] basically saying
that there is no recommended solution other than a custom-made analysis.
Have there been any solutions to this problem?

I can also see a Jenkins plugin [2] though I have no idea in what state
the plugin is, and as such, how viable it is to run it. Last but not
least, there's some Grafana integration blog [3].  Does anyone have any
other suggestions? How would you compare two results programmatically to
see if there is degraded performance?

[1]
http://mail-archives.apache.org/mod_mbox/jmeter-user/200401.mbox/%3C99805B014A26D211BC3100A0C9B72CB8059E28EA@...%3E

[2] https://wiki.jenkins.io/display/JENKINS/Performance+Plugin

[3]
http://www.testautomationguru.com/jmeter-real-time-results-influxdb-grafana/

Cheers,
--

Marek Czernek

JWS/JBCS Associate Quality Engineer, RHCA

Find me at www.halfastack.com

 
Reply | Threaded
Open this post in threaded view
|

Re: Best way to compare two results of jmeter

niraj
I was searching something similar last week and after spending couple of days, i came up with following options

  • Checkout below project on Github. It will take sometime to set it up but it's will give you comparison across build with some other features as well.  
          https://github.com/innogames/JMeter-Control-Center



  • Performance plugin using Jenkins. Only issue with that, export option is not available and when you have results for few months, it's become difficult to read results inside Jenkins.  
  • I wrote one python script which will give you comparison for response time across different runs. If you want to check that, drop me email i will share code with you. 
Hope this helps.

-Niraj  

On Thu, Aug 30, 2018 at 8:21 PM Alexander Podelko <[hidden email]> wrote:
Hi Marek,
I do use the Jenkins Performance Plugin for that purpose for some time, I'd say that you get quite a lot for free straight out of the box. Pretty decent. Another thing is that I haven't found practically any documentation (although it is pretty straightforward for a simple use - and there is a few posts how to setup it) and it is still not clear for me what to do if I'd need something else from it....

Regards,Alex


   On Thursday, August 30, 2018, 10:11:22 AM EDT, Marek Czernek <[hidden email]> wrote: 

 Hi there,

is there any 'supported' way to compare the results of 2 jmeter runs? I
googled around and found an old email from 2004 [1] basically saying
that there is no recommended solution other than a custom-made analysis.
Have there been any solutions to this problem?

I can also see a Jenkins plugin [2] though I have no idea in what state
the plugin is, and as such, how viable it is to run it. Last but not
least, there's some Grafana integration blog [3].  Does anyone have any
other suggestions? How would you compare two results programmatically to
see if there is degraded performance?

[1]
http://mail-archives.apache.org/mod_mbox/jmeter-user/200401.mbox/%3C99805B014A26D211BC3100A0C9B72CB8059E28EA@...%3E

[2] https://wiki.jenkins.io/display/JENKINS/Performance+Plugin

[3]
http://www.testautomationguru.com/jmeter-real-time-results-influxdb-grafana/

Cheers,
--

Marek Czernek

JWS/JBCS Associate Quality Engineer, RHCA

Find me at www.halfastack.com

 
Reply | Threaded
Open this post in threaded view
|

Re: Best way to compare two results of jmeter

Alexander Podelko
In reply to this post by Alexander Podelko
Just saw another solution in that area https://dzone.com/articles/jmeter-elasticsearch-live-monitoring 

   On Thursday, August 30, 2018, 10:50:56 AM EDT, Alexander Podelko <[hidden email]> wrote:  
 
 Hi Marek,
I do use the Jenkins Performance Plugin for that purpose for some time, I'd say that you get quite a lot for free straight out of the box. Pretty decent. Another thing is that I haven't found practically any documentation (although it is pretty straightforward for a simple use - and there is a few posts how to setup it) and it is still not clear for me what to do if I'd need something else from it....

Regards,Alex


   On Thursday, August 30, 2018, 10:11:22 AM EDT, Marek Czernek <[hidden email]> wrote:  
 
 Hi there,

is there any 'supported' way to compare the results of 2 jmeter runs? I
googled around and found an old email from 2004 [1] basically saying
that there is no recommended solution other than a custom-made analysis.
Have there been any solutions to this problem?

I can also see a Jenkins plugin [2] though I have no idea in what state
the plugin is, and as such, how viable it is to run it. Last but not
least, there's some Grafana integration blog [3].  Does anyone have any
other suggestions? How would you compare two results programmatically to
see if there is degraded performance?

[1]
http://mail-archives.apache.org/mod_mbox/jmeter-user/200401.mbox/%3C99805B014A26D211BC3100A0C9B72CB8059E28EA@...%3E

[2] https://wiki.jenkins.io/display/JENKINS/Performance+Plugin

[3]
http://www.testautomationguru.com/jmeter-real-time-results-influxdb-grafana/

Cheers,
--

Marek Czernek

JWS/JBCS Associate Quality Engineer, RHCA

Find me at www.halfastack.com

   
Reply | Threaded
Open this post in threaded view
|

Re: Best way to compare two results of jmeter

Marek Czernek
I saw the Grafana integration, but in my mind, any solution that
involves external DB seems unsuitable for my simple needs. I am testing
the Jenkins plugin, though I have been running into problems with
comparing two runs. I wonder:

 1. Do you need to fire the JMX testplan using the plugin to be able to
    compare results across builds with the performance plugin?
 2. Do you need any Jenkins plugin other than the performance plugin for
    cross-build comparison?

In the worst case, I'll implement a solution similar to what Bo
suggested, i.e. simply execute calculation on top of the CSVs. The
database solutions seem great if you need to really work with the data;
for my purposes, I mainly want to see whether there's a performance
difference from the previous build and I don't care that much about the
visual output.

Cheers,
--

Marek Czernek

JWS/JBCS Associate Quality Engineer, RHCA

Find me at www.halfastack.com


On 08/31/2018 03:18 PM, Alexander Podelko wrote:

> Just saw another solution in that area https://dzone.com/articles/jmeter-elasticsearch-live-monitoring
>
>     On Thursday, August 30, 2018, 10:50:56 AM EDT, Alexander Podelko <[hidden email]> wrote:
>  
>   Hi Marek,
> I do use the Jenkins Performance Plugin for that purpose for some time, I'd say that you get quite a lot for free straight out of the box. Pretty decent. Another thing is that I haven't found practically any documentation (although it is pretty straightforward for a simple use - and there is a few posts how to setup it) and it is still not clear for me what to do if I'd need something else from it....
>
> Regards,Alex
>
>
>     On Thursday, August 30, 2018, 10:11:22 AM EDT, Marek Czernek <[hidden email]> wrote:
>  
>   Hi there,
>
> is there any 'supported' way to compare the results of 2 jmeter runs? I
> googled around and found an old email from 2004 [1] basically saying
> that there is no recommended solution other than a custom-made analysis.
> Have there been any solutions to this problem?
>
> I can also see a Jenkins plugin [2] though I have no idea in what state
> the plugin is, and as such, how viable it is to run it. Last but not
> least, there's some Grafana integration blog [3].  Does anyone have any
> other suggestions? How would you compare two results programmatically to
> see if there is degraded performance?
>
> [1]
> http://mail-archives.apache.org/mod_mbox/jmeter-user/200401.mbox/%3C99805B014A26D211BC3100A0C9B72CB8059E28EA@...%3E
>
> [2] https://wiki.jenkins.io/display/JENKINS/Performance+Plugin
>
> [3]
> http://www.testautomationguru.com/jmeter-real-time-results-influxdb-grafana/
>
> Cheers,
Reply | Threaded
Open this post in threaded view
|

Re: Best way to compare two results of jmeter

Alexander Podelko
Marek,
My understanding is no for both questions (from a simple user point of view).

The step involving plug in (where you specify what to do) is named "Publish performance test result reports" and the input is a jtl file. So it appears that you just need to get the result file there - and it keeps historic data and do comparison with previous run (so it still should be a Jenkins job). I haven't found anything about internals yet (well, except https://github.com/jenkinsci/performance-plugin ).

Here is explanation on the Jenkins plugin config page:


This plugin understands the JMeter analysis report XMLformat, JMeter Summariser report text format,  SOAPUI report in JUnit format, Iago format as recorded while parrot server is running, and LoadRunner analysis format.
This plug-in does not perform the actual analysis; it onlydisplays useful information about analysis results, such as averageresponding time, historical result trend, web UI for viewing analysisreports, and so on.
To use this feature, first set up your build to run tests, then select the adequate parser for your tests (JMeter, JUnit, or Iago) and finallyyou have to specify the path to the different performance files. By default the plugin will use the **/*.jtl pattern forJMeter, **/TEST*.xml for JUnit tests ,**/*.log pattern for JMeter Summariser, parrot-server-stats.log for Iago, and **/*.mdb for LoadRunner.

There some post with instructions how to setup it like https://www.blazemeter.com/blog/how-to-use-the-jenkins-performance-plugin
As per other plugins, nothing special was needed in my case - but there were some plugins already installed there. My struggle there, for example, was to start JMeter tests remotely and getting results back to Jenkins (most examples assume that this is on the same machine) - which I did using "Send build artifacts over SSH" (which, as far as I understand, uses another plugin) and modifying folder access rights... I guess there are other options to do that.

Regards,Alex



   On Friday, August 31, 2018, 9:30:18 AM EDT, Marek Czernek <[hidden email]> wrote:  
 
 I saw the Grafana integration, but in my mind, any solution that
involves external DB seems unsuitable for my simple needs. I am testing
the Jenkins plugin, though I have been running into problems with
comparing two runs. I wonder:

 1. Do you need to fire the JMX testplan using the plugin to be able to
    compare results across builds with the performance plugin?
 2. Do you need any Jenkins plugin other than the performance plugin for
    cross-build comparison?

In the worst case, I'll implement a solution similar to what Bo
suggested, i.e. simply execute calculation on top of the CSVs. The
database solutions seem great if you need to really work with the data;
for my purposes, I mainly want to see whether there's a performance
difference from the previous build and I don't care that much about the
visual output.

Cheers,
--

Marek Czernek

JWS/JBCS Associate Quality Engineer, RHCA

Find me at www.halfastack.com


On 08/31/2018 03:18 PM, Alexander Podelko wrote:

> Just saw another solution in that area https://dzone.com/articles/jmeter-elasticsearch-live-monitoring
>
>    On Thursday, August 30, 2018, 10:50:56 AM EDT, Alexander Podelko <[hidden email]> wrote:

>  Hi Marek,
> I do use the Jenkins Performance Plugin for that purpose for some time, I'd say that you get quite a lot for free straight out of the box. Pretty decent. Another thing is that I haven't found practically any documentation (although it is pretty straightforward for a simple use - and there is a few posts how to setup it) and it is still not clear for me what to do if I'd need something else from it....
>
> Regards,Alex
>
>
>    On Thursday, August 30, 2018, 10:11:22 AM EDT, Marek Czernek <[hidden email]> wrote:

>  Hi there,
>
> is there any 'supported' way to compare the results of 2 jmeter runs? I
> googled around and found an old email from 2004 [1] basically saying
> that there is no recommended solution other than a custom-made analysis.
> Have there been any solutions to this problem?
>
> I can also see a Jenkins plugin [2] though I have no idea in what state
> the plugin is, and as such, how viable it is to run it. Last but not
> least, there's some Grafana integration blog [3].  Does anyone have any
> other suggestions? How would you compare two results programmatically to
> see if there is degraded performance?
>
> [1]
> http://mail-archives.apache.org/mod_mbox/jmeter-user/200401.mbox/%3C99805B014A26D211BC3100A0C9B72CB8059E28EA@...%3E
>
> [2] https://wiki.jenkins.io/display/JENKINS/Performance+Plugin
>
> [3]
> http://www.testautomationguru.com/jmeter-real-time-results-influxdb-grafana/
>
> Cheers,
 
Reply | Threaded
Open this post in threaded view
|

Re: Best way to compare two results of jmeter

Bo Bodnar-2
In reply to this post by Marek Czernek

Here's an example of a paired t-test; I did this in R using RStudio since this is easy to do. I made up the numbers. The whole thing can be done in Excel but I haven't done that for a few years (R is much easier).

x = c(1,-2, 3, 4, 5, 6)
y = c(5, -6, 9, 8, 4, 6)

Question: is there a statistically significant difference between measured x and measured y? I will use 90% confidence interval.

Result:

Give the command: t.test(x, y, conf=0.9)

The returned information:


Welch Two Sample t-test

data: x and y
t = -0.59894, df = 7.7117, p-value = 0.5664
alternative hypothesis: true difference in means is not equal to 0
90 percent confidence interval:
-6.179799 3.179799
sample estimates:
mean of x mean of y
2.833333 4.333333

Note the interval's lower bound is -6.17... and the upper bound is 3.17...

This includes 0. Consequently, I conclude that there's no statistically valid difference between the two even though the means are different. If the interval did not include 0, then I can conclude that there's a statistically valid difference. The reason for this conclusion is the large variability in the data (that's why we need a lot of data points and we need tightly controlled test environments).

Hope this helps,

Bo



Bohdan L. Bodnar
Lead Performance Engineer
1-312-871-5163

E-mail: [hidden email]
222 South Riverside Plaza
Chicago, IL 60606
United States

Marek Czernek ---08/31/2018 08:31:37 AM---I saw the Grafana integration, but in my mind, any solution that involves external DB seems unsuita

From: Marek Czernek <[hidden email]>
To: [hidden email]
Date: 08/31/2018 08:31 AM
Subject: Re: Best way to compare two results of jmeter




I saw the Grafana integration, but in my mind, any solution that
involves external DB seems unsuitable for my simple needs. I am testing
the Jenkins plugin, though I have been running into problems with
comparing two runs. I wonder:

1. Do you need to fire the JMX testplan using the plugin to be able to
   compare results across builds with the performance plugin?
2. Do you need any Jenkins plugin other than the performance plugin for
   cross-build comparison?

In the worst case, I'll implement a solution similar to what Bo
suggested, i.e. simply execute calculation on top of the CSVs. The
database solutions seem great if you need to really work with the data;
for my purposes, I mainly want to see whether there's a performance
difference from the previous build and I don't care that much about the
visual output.

Cheers,
--

Marek Czernek

JWS/JBCS Associate Quality Engineer, RHCA

Find me at
www.halfastack.com


On 08/31/2018 03:18 PM, Alexander Podelko wrote:

> Just saw another solution in that area
https://dzone.com/articles/jmeter-elasticsearch-live-monitoring
>
>     On Thursday, August 30, 2018, 10:50:56 AM EDT, Alexander Podelko <[hidden email]> wrote:
>  
>   Hi Marek,
> I do use the Jenkins Performance Plugin for that purpose for some time, I'd say that you get quite a lot for free straight out of the box. Pretty decent. Another thing is that I haven't found practically any documentation (although it is pretty straightforward for a simple use - and there is a few posts how to setup it) and it is still not clear for me what to do if I'd need something else from it....
>
> Regards,Alex
>
>
>     On Thursday, August 30, 2018, 10:11:22 AM EDT, Marek Czernek <[hidden email]> wrote:
>  
>   Hi there,
>
> is there any 'supported' way to compare the results of 2 jmeter runs? I
> googled around and found an old email from 2004 [1] basically saying
> that there is no recommended solution other than a custom-made analysis.
> Have there been any solutions to this problem?
>
> I can also see a Jenkins plugin [2] though I have no idea in what state
> the plugin is, and as such, how viable it is to run it. Last but not
> least, there's some Grafana integration blog [3].  Does anyone have any
> other suggestions? How would you compare two results programmatically to
> see if there is degraded performance?
>
> [1]
>
http://mail-archives.apache.org/mod_mbox/jmeter-user/200401.mbox/%3C99805B014A26D211BC3100A0C9B72CB8059E28EA@...%3E
>
> [2]
https://wiki.jenkins.io/display/JENKINS/Performance+Plugin
>
> [3]
>
http://www.testautomationguru.com/jmeter-real-time-results-influxdb-grafana/
>
> Cheers,



Reply | Threaded
Open this post in threaded view
|

Re: Best way to compare two results of jmeter

glinius@live.com
In reply to this post by Marek Czernek
The easiest way to compare 2 JMeter runs is using  Merge Results
<https://jmeter-plugins.org/wiki/MergeResults/>   tool. You will be able to
specify prefixes for your test runs and get an aggregate chart which you can
use for comparison.

<http://www.jmeter-archive.org/file/t340375/merge_results_test1-2.png>

Merge Results tool can be installed using  JMeter Plugins Manager
<https://www.blazemeter.com/blog/how-install-jmeter-plugins-manager>  







--
Sent from: http://www.jmeter-archive.org/JMeter-User-f512775.html

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]