Compare test result to baseline (or previous run)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Compare test result to baseline (or previous run)

David Balažic
Hi!

I'm looking for a way to detect performance regressions in my product.

I looked into JMeter but found no easy way to show the desired results.
I created a test plan, test steps (samplers) that executes, but I just can't find a suitable method to get the desired results.
Those would be:
 - compare execution times (each sampler on its own, the average or maybe the median time, not each individual execution) agains a predefined baseline (or a previous run or similar)
 - an OK flag, if no execution time is significantly slower than in baseline
 - otherwise an easily readable list of samplers, that are slower

Similar questions are asked often on the internet, with similar (mostly partial) answers.
For example on this very mail list : [1]

Was there any progress regarding this since?

In the lack of better solution, I entertained the idea to store  aggregate results using JMeterPluginsCMD [2] and then import them into some external tool (Excel, RStudio or similar) for processing.

[1] Marek Czernek "Best way to compare two results of jmeter" Thu, 30 Aug 2020, 14:11
http://mail-archives.apache.org/mod_mbox/jmeter-user/201808.mbox/%3c8efb6251-f2aa-287e-5793-836d1efd5b59@...%3e

[2] https://jmeter-plugins.org/wiki/JMeterPluginsCMD/

Lep pozdrav,
David Balažic

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Compare test result to baseline (or previous run)

Dan Zaitoun
Use performance plugin for jenkins.it execatly for this.

בתאריך יום ד׳, 27 במאי 2020, 18:25, מאת David Balažic ‏<
[hidden email]>:

> Hi!
>
> I'm looking for a way to detect performance regressions in my product.
>
> I looked into JMeter but found no easy way to show the desired results.
> I created a test plan, test steps (samplers) that executes, but I just
> can't find a suitable method to get the desired results.
> Those would be:
>  - compare execution times (each sampler on its own, the average or maybe
> the median time, not each individual execution) agains a predefined
> baseline (or a previous run or similar)
>  - an OK flag, if no execution time is significantly slower than in
> baseline
>  - otherwise an easily readable list of samplers, that are slower
>
> Similar questions are asked often on the internet, with similar (mostly
> partial) answers.
> For example on this very mail list : [1]
>
> Was there any progress regarding this since?
>
> In the lack of better solution, I entertained the idea to store  aggregate
> results using JMeterPluginsCMD [2] and then import them into some external
> tool (Excel, RStudio or similar) for processing.
>
> [1] Marek Czernek "Best way to compare two results of jmeter" Thu, 30 Aug
> 2020, 14:11
>
> http://mail-archives.apache.org/mod_mbox/jmeter-user/201808.mbox/%3c8efb6251-f2aa-287e-5793-836d1efd5b59@...%3e
>
> [2] https://jmeter-plugins.org/wiki/JMeterPluginsCMD/
>
> Lep pozdrav,
> David Balažic
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Compare test result to baseline (or previous run)

glinius@live.com
In reply to this post by David Balažic
Have you seen  Real-time results
<https://jmeter.apache.org/usermanual/realtime-results.html>   page of
JMeter Documentation, it's very first lines say:


> Since JMeter 2.13 you can get real-time results sent to a backend through
> the Backend Listener using potentially any backend (JDBC, JMS, Webservice,
> …) by providing a class which implements AbstractBackendListenerClient.

So you can setup i.e.  InfluxDB and Grafana combination
<https://www.blazemeter.com/blog/how-to-use-grafana-to-monitor-jmeter-non-gui-results>  
and get both real-time statistics and possibility to compare different test
runs

If you're looking for simpler ad-hoc solution -  Merge Results Tool
<https://jmeter-plugins.org/wiki/MergeResults/>   provides possibility to
compare up to 4 test result sets.



--
Sent from: http://www.jmeter-archive.org/JMeter-User-f512775.html

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Compare test result to baseline (or previous run)

David Balažic
In reply to this post by Dan Zaitoun
Well,

I tried it , but it is like an alpha stage software. It kind of works, but is like a yugo. It drives, but you rather drive another car.
Is there a non-free alternative that is better?

Regards,
David

-----Original Message-----
From: Dan Zaitoun

Use performance plugin for jenkins.it execatly for this.

בתאריך יום ד׳, 27 במאי 2020, 18:25, מאת David Balažic ‏<
[hidden email]>:

> Hi!
>
> I'm looking for a way to detect performance regressions in my product.
>
> I looked into JMeter but found no easy way to show the desired results.
> I created a test plan, test steps (samplers) that executes, but I just
> can't find a suitable method to get the desired results.
> Those would be:
>  - compare execution times (each sampler on its own, the average or
> maybe the median time, not each individual execution) agains a
> predefined baseline (or a previous run or similar)
>  - an OK flag, if no execution time is significantly slower than in
> baseline
>  - otherwise an easily readable list of samplers, that are slower
>
> Similar questions are asked often on the internet, with similar
> (mostly
> partial) answers.
> For example on this very mail list : [1]
>
> Was there any progress regarding this since?
>
> In the lack of better solution, I entertained the idea to store  
> aggregate results using JMeterPluginsCMD [2] and then import them into
> some external tool (Excel, RStudio or similar) for processing.
>
> [1] Marek Czernek "Best way to compare two results of jmeter" Thu, 30
> Aug 2020, 14:11
>
> http://mail-archives.apache.org/mod_mbox/jmeter-user/201808.mbox/%3c8e
> [hidden email]%3e
>
> [2] https://jmeter-plugins.org/wiki/JMeterPluginsCMD/
>
> Lep pozdrav,
> David Balažic
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Compare test result to baseline (or previous run)

Martin Grigorov
Hi,

On Thu, May 28, 2020 at 12:27 AM David Balažic <[hidden email]>
wrote:

> Well,
>
> I tried it , but it is like an alpha stage software. It kind of works, but
> is like a yugo. It drives, but you rather drive another car.
> Is there a non-free alternative that is better?
>

JMeter stores its results in files (CSV, JSON, ...).
I use Logstash to parse and store those into Elasticsearch. And finally I
visualize them with Kibana.
You may use your favorite language / tool to do the same


>
> Regards,
> David
>
> -----Original Message-----
> From: Dan Zaitoun
>
> Use performance plugin for jenkins.it execatly for this.
>
> בתאריך יום ד׳, 27 במאי 2020, 18:25, מאת David Balažic ‏<
> [hidden email]>:
>
> > Hi!
> >
> > I'm looking for a way to detect performance regressions in my product.
> >
> > I looked into JMeter but found no easy way to show the desired results.
> > I created a test plan, test steps (samplers) that executes, but I just
> > can't find a suitable method to get the desired results.
> > Those would be:
> >  - compare execution times (each sampler on its own, the average or
> > maybe the median time, not each individual execution) agains a
> > predefined baseline (or a previous run or similar)
> >  - an OK flag, if no execution time is significantly slower than in
> > baseline
> >  - otherwise an easily readable list of samplers, that are slower
> >
> > Similar questions are asked often on the internet, with similar
> > (mostly
> > partial) answers.
> > For example on this very mail list : [1]
> >
> > Was there any progress regarding this since?
> >
> > In the lack of better solution, I entertained the idea to store
> > aggregate results using JMeterPluginsCMD [2] and then import them into
> > some external tool (Excel, RStudio or similar) for processing.
> >
> > [1] Marek Czernek "Best way to compare two results of jmeter" Thu, 30
> > Aug 2020, 14:11
> >
> > http://mail-archives.apache.org/mod_mbox/jmeter-user/201808.mbox/%3c8e
> > [hidden email]%3e
> >
> > [2] https://jmeter-plugins.org/wiki/JMeterPluginsCMD/
> >
> > Lep pozdrav,
> > David Balažic
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>