Incorrect 90th percentile values in JMeter dashboard HTML report

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Incorrect 90th percentile values in JMeter dashboard HTML report

Abhitosh Patil
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP
Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Mariusz W
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP
Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Abhitosh Patil
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP
Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Lynch Kieran
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:


Aggregate Report:



Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com <A title="" style="TEXT-DECORATION: underline; COLOR: blue" href="http://" target="">



From: Abhitosh Patil <[hidden email]>
Sent: 26 February 2021 14:28
To: JMeter Users List <[hidden email]>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report
 
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Mariusz W
Yes, I have seen this small discrepancies also in my tests but I don't adhere to exact values and this reports are just first step during analysis to find an anomaly or something else to focus on , e.g. Is it worth to look at the mean when data is not symmetrically distributed. If it is not symmetrical should I care about standard deviation at all. If my histogram is strange - what was wrong.
But surely the meaning of the report depends on the client's need:)

Mariusz

On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <[hidden email]> wrote:
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:


Aggregate Report:



Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com

WM Reply

From: Abhitosh Patil <[hidden email]>
Sent: 26 February 2021 14:28
To: JMeter Users List <[hidden email]>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report
 
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Abhitosh Patil
@Kieran Lynch
I'm using JMeter 5.4
I already tried "jmeter.reportgenerator.statistic_window = -1". But it doesn't work.

@Mariusz W
In my case it's a very huge difference (> 40 seconds) which is not acceptable at all as there are client facing reports.

@All
I had a discussion with my colleague. As per him there is difference in formula how Aggregate report and HTMl report calculates 90th percentile.
Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML report is picking 6th value (which is the highest one) as 90 th percentile.
I need to figure it out ASAP now.



On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <[hidden email]> wrote:
Yes, I have seen this small discrepancies also in my tests but I don't adhere to exact values and this reports are just first step during analysis to find an anomaly or something else to focus on , e.g. Is it worth to look at the mean when data is not symmetrically distributed. If it is not symmetrical should I care about standard deviation at all. If my histogram is strange - what was wrong.
But surely the meaning of the report depends on the client's need:)

Mariusz

On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <[hidden email]> wrote:
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:


Aggregate Report:



Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com

WM Reply

From: Abhitosh Patil <[hidden email]>
Sent: 26 February 2021 14:28
To: JMeter Users List <[hidden email]>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report
 
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Mariusz W
I have to admint that I have never seen such big discrepancies in JMeter reports. So in which report data is wrong in your opinion? When you talk about "Aggregate report" do you think about JMeter report generated by Aggregate Report listener? I ask because in your images You have data in Excel and standard deviation is present but (as far as I know it is not present in this listener). Standard deviation is generated in Summary Report listener. So you calculated it yourself or merged reports? We also see only some examples (only 6 samples report) so can you send any more realistic data (more samples)? My hypothesis (based on information I see so far, or my understanding of this information:) ) is that You have some error in automatic report generation (You said: "I have automated Aggregate report and HTML report creation" ). But as I have said,  it is based on my understanding of data you provided:)

Mariusz

On Fri, 26 Feb 2021 at 19:39, Abhitosh Patil <[hidden email]> wrote:
@Kieran Lynch
I'm using JMeter 5.4
I already tried "jmeter.reportgenerator.statistic_window = -1". But it doesn't work.

@Mariusz W
In my case it's a very huge difference (> 40 seconds) which is not acceptable at all as there are client facing reports.

@All
I had a discussion with my colleague. As per him there is difference in formula how Aggregate report and HTMl report calculates 90th percentile.
Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML report is picking 6th value (which is the highest one) as 90 th percentile.
I need to figure it out ASAP now.



On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <[hidden email]> wrote:
Yes, I have seen this small discrepancies also in my tests but I don't adhere to exact values and this reports are just first step during analysis to find an anomaly or something else to focus on , e.g. Is it worth to look at the mean when data is not symmetrically distributed. If it is not symmetrical should I care about standard deviation at all. If my histogram is strange - what was wrong.
But surely the meaning of the report depends on the client's need:)

Mariusz

On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <[hidden email]> wrote:
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:


Aggregate Report:



Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com

WM Reply

From: Abhitosh Patil <[hidden email]>
Sent: 26 February 2021 14:28
To: JMeter Users List <[hidden email]>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report
 
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Felix Schumacher
In reply to this post by Abhitosh Patil

Hi Abhitosh,

it would have been a good idea to share sample data, that shows that behaviour.

I think you stumbled upon https://bz.apache.org/bugzilla/show_bug.cgi?id=61130

One problem is, that JMeter uses two different algorithms to calculate a percentile. One for the report and one for the dashboard. They differ in the point, which they use to display the percentile.

We did some work to use the same algorithm for both components in https://github.com/apache/jmeter/pull/380 which got closed when converting our repo from subversion to git.

In my eyes, that big discrepancy in your values shows, that you are having bad data in the first place. But of course, it would be nice, if JMeter would show consistent values, or even better an indicator of how close the value is to its neighbours.

Felix

Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
@Kieran Lynch
I'm using JMeter 5.4
I already tried "jmeter.reportgenerator.statistic_window = -1". But it doesn't work.

@Mariusz W
In my case it's a very huge difference (> 40 seconds) which is not acceptable at all as there are client facing reports.

@All
I had a discussion with my colleague. As per him there is difference in formula how Aggregate report and HTMl report calculates 90th percentile.
Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML report is picking 6th value (which is the highest one) as 90 th percentile.
I need to figure it out ASAP now.



On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <[hidden email]> wrote:
Yes, I have seen this small discrepancies also in my tests but I don't adhere to exact values and this reports are just first step during analysis to find an anomaly or something else to focus on , e.g. Is it worth to look at the mean when data is not symmetrically distributed. If it is not symmetrical should I care about standard deviation at all. If my histogram is strange - what was wrong.
But surely the meaning of the report depends on the client's need:)

Mariusz

On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <[hidden email]> wrote:
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:


Aggregate Report:



Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com

WM Reply

From: Abhitosh Patil <[hidden email]>
Sent: 26 February 2021 14:28
To: JMeter Users List <[hidden email]>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report
 
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Abhitosh Patil
@Mariusz
As per below post, HTML report has wrong values:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti

I automated aggregate report creation using JMeter plugin: CMDRunner

Also I tried creating HTML and Aggregate reports manually using a JTL file. But the results are showing the same difference.

My suggestion to reproduce this issue is:
Execute a long time JMeter load test with a considerable number of samples. Then compare HTML and aggregate reports for the test.
You can see the difference between them.

@Felix
But not getting a clear idea how to implement this.

On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <[hidden email]> wrote:

Hi Abhitosh,

it would have been a good idea to share sample data, that shows that behaviour.

I think you stumbled upon https://bz.apache.org/bugzilla/show_bug.cgi?id=61130

One problem is, that JMeter uses two different algorithms to calculate a percentile. One for the report and one for the dashboard. They differ in the point, which they use to display the percentile.

We did some work to use the same algorithm for both components in https://github.com/apache/jmeter/pull/380 which got closed when converting our repo from subversion to git.

In my eyes, that big discrepancy in your values shows, that you are having bad data in the first place. But of course, it would be nice, if JMeter would show consistent values, or even better an indicator of how close the value is to its neighbours.

Felix

Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
@Kieran Lynch
I'm using JMeter 5.4
I already tried "jmeter.reportgenerator.statistic_window = -1". But it doesn't work.

@Mariusz W
In my case it's a very huge difference (> 40 seconds) which is not acceptable at all as there are client facing reports.

@All
I had a discussion with my colleague. As per him there is difference in formula how Aggregate report and HTMl report calculates 90th percentile.
Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML report is picking 6th value (which is the highest one) as 90 th percentile.
I need to figure it out ASAP now.



On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <[hidden email]> wrote:
Yes, I have seen this small discrepancies also in my tests but I don't adhere to exact values and this reports are just first step during analysis to find an anomaly or something else to focus on , e.g. Is it worth to look at the mean when data is not symmetrically distributed. If it is not symmetrical should I care about standard deviation at all. If my histogram is strange - what was wrong.
But surely the meaning of the report depends on the client's need:)

Mariusz

On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <[hidden email]> wrote:
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:


Aggregate Report:



Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com

WM Reply

From: Abhitosh Patil <[hidden email]>
Sent: 26 February 2021 14:28
To: JMeter Users List <[hidden email]>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report
 
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Mariusz W
@Abhitosh
To be sure: So you loaded manually JTL file in Aggregate Report listener ( without CMDRunner ) and compared the results with html report?

Mariusz

On Sun, 28 Feb 2021 at 12:27, A P <[hidden email]> wrote:
@Mariusz
As per below post, HTML report has wrong values:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti

I automated aggregate report creation using JMeter plugin: CMDRunner

Also I tried creating HTML and Aggregate reports manually using a JTL file. But the results are showing the same difference.

My suggestion to reproduce this issue is:
Execute a long time JMeter load test with a considerable number of samples. Then compare HTML and aggregate reports for the test.
You can see the difference between them.

@Felix
But not getting a clear idea how to implement this.

On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <[hidden email]> wrote:

Hi Abhitosh,

it would have been a good idea to share sample data, that shows that behaviour.

I think you stumbled upon https://bz.apache.org/bugzilla/show_bug.cgi?id=61130

One problem is, that JMeter uses two different algorithms to calculate a percentile. One for the report and one for the dashboard. They differ in the point, which they use to display the percentile.

We did some work to use the same algorithm for both components in https://github.com/apache/jmeter/pull/380 which got closed when converting our repo from subversion to git.

In my eyes, that big discrepancy in your values shows, that you are having bad data in the first place. But of course, it would be nice, if JMeter would show consistent values, or even better an indicator of how close the value is to its neighbours.

Felix

Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
@Kieran Lynch
I'm using JMeter 5.4
I already tried "jmeter.reportgenerator.statistic_window = -1". But it doesn't work.

@Mariusz W
In my case it's a very huge difference (> 40 seconds) which is not acceptable at all as there are client facing reports.

@All
I had a discussion with my colleague. As per him there is difference in formula how Aggregate report and HTMl report calculates 90th percentile.
Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML report is picking 6th value (which is the highest one) as 90 th percentile.
I need to figure it out ASAP now.



On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <[hidden email]> wrote:
Yes, I have seen this small discrepancies also in my tests but I don't adhere to exact values and this reports are just first step during analysis to find an anomaly or something else to focus on , e.g. Is it worth to look at the mean when data is not symmetrically distributed. If it is not symmetrical should I care about standard deviation at all. If my histogram is strange - what was wrong.
But surely the meaning of the report depends on the client's need:)

Mariusz

On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <[hidden email]> wrote:
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:


Aggregate Report:



Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com

WM Reply

From: Abhitosh Patil <[hidden email]>
Sent: 26 February 2021 14:28
To: JMeter Users List <[hidden email]>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report
 
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Abhitosh Patil
@ Mariusz
Yes.

On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <[hidden email]> wrote:
@Abhitosh
To be sure: So you loaded manually JTL file in Aggregate Report listener ( without CMDRunner ) and compared the results with html report?

Mariusz

On Sun, 28 Feb 2021 at 12:27, A P <[hidden email]> wrote:
@Mariusz
As per below post, HTML report has wrong values:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti

I automated aggregate report creation using JMeter plugin: CMDRunner

Also I tried creating HTML and Aggregate reports manually using a JTL file. But the results are showing the same difference.

My suggestion to reproduce this issue is:
Execute a long time JMeter load test with a considerable number of samples. Then compare HTML and aggregate reports for the test.
You can see the difference between them.

@Felix
But not getting a clear idea how to implement this.

On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <[hidden email]> wrote:

Hi Abhitosh,

it would have been a good idea to share sample data, that shows that behaviour.

I think you stumbled upon https://bz.apache.org/bugzilla/show_bug.cgi?id=61130

One problem is, that JMeter uses two different algorithms to calculate a percentile. One for the report and one for the dashboard. They differ in the point, which they use to display the percentile.

We did some work to use the same algorithm for both components in https://github.com/apache/jmeter/pull/380 which got closed when converting our repo from subversion to git.

In my eyes, that big discrepancy in your values shows, that you are having bad data in the first place. But of course, it would be nice, if JMeter would show consistent values, or even better an indicator of how close the value is to its neighbours.

Felix

Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
@Kieran Lynch
I'm using JMeter 5.4
I already tried "jmeter.reportgenerator.statistic_window = -1". But it doesn't work.

@Mariusz W
In my case it's a very huge difference (> 40 seconds) which is not acceptable at all as there are client facing reports.

@All
I had a discussion with my colleague. As per him there is difference in formula how Aggregate report and HTMl report calculates 90th percentile.
Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML report is picking 6th value (which is the highest one) as 90 th percentile.
I need to figure it out ASAP now.



On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <[hidden email]> wrote:
Yes, I have seen this small discrepancies also in my tests but I don't adhere to exact values and this reports are just first step during analysis to find an anomaly or something else to focus on , e.g. Is it worth to look at the mean when data is not symmetrically distributed. If it is not symmetrical should I care about standard deviation at all. If my histogram is strange - what was wrong.
But surely the meaning of the report depends on the client's need:)

Mariusz

On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <[hidden email]> wrote:
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:


Aggregate Report:



Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com

WM Reply

From: Abhitosh Patil <[hidden email]>
Sent: 26 February 2021 14:28
To: JMeter Users List <[hidden email]>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report
 
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Abhitosh Patil
Hi All,

I'm eagerly waiting for solution to this issue. If anyone has a fix please reply back.

AP

On Sun, Feb 28, 2021 at 11:59 PM A P <[hidden email]> wrote:
@ Mariusz
Yes.

On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <[hidden email]> wrote:
@Abhitosh
To be sure: So you loaded manually JTL file in Aggregate Report listener ( without CMDRunner ) and compared the results with html report?

Mariusz

On Sun, 28 Feb 2021 at 12:27, A P <[hidden email]> wrote:
@Mariusz
As per below post, HTML report has wrong values:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti

I automated aggregate report creation using JMeter plugin: CMDRunner

Also I tried creating HTML and Aggregate reports manually using a JTL file. But the results are showing the same difference.

My suggestion to reproduce this issue is:
Execute a long time JMeter load test with a considerable number of samples. Then compare HTML and aggregate reports for the test.
You can see the difference between them.

@Felix
But not getting a clear idea how to implement this.

On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <[hidden email]> wrote:

Hi Abhitosh,

it would have been a good idea to share sample data, that shows that behaviour.

I think you stumbled upon https://bz.apache.org/bugzilla/show_bug.cgi?id=61130

One problem is, that JMeter uses two different algorithms to calculate a percentile. One for the report and one for the dashboard. They differ in the point, which they use to display the percentile.

We did some work to use the same algorithm for both components in https://github.com/apache/jmeter/pull/380 which got closed when converting our repo from subversion to git.

In my eyes, that big discrepancy in your values shows, that you are having bad data in the first place. But of course, it would be nice, if JMeter would show consistent values, or even better an indicator of how close the value is to its neighbours.

Felix

Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
@Kieran Lynch
I'm using JMeter 5.4
I already tried "jmeter.reportgenerator.statistic_window = -1". But it doesn't work.

@Mariusz W
In my case it's a very huge difference (> 40 seconds) which is not acceptable at all as there are client facing reports.

@All
I had a discussion with my colleague. As per him there is difference in formula how Aggregate report and HTMl report calculates 90th percentile.
Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML report is picking 6th value (which is the highest one) as 90 th percentile.
I need to figure it out ASAP now.



On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <[hidden email]> wrote:
Yes, I have seen this small discrepancies also in my tests but I don't adhere to exact values and this reports are just first step during analysis to find an anomaly or something else to focus on , e.g. Is it worth to look at the mean when data is not symmetrically distributed. If it is not symmetrical should I care about standard deviation at all. If my histogram is strange - what was wrong.
But surely the meaning of the report depends on the client's need:)

Mariusz

On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <[hidden email]> wrote:
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:


Aggregate Report:



Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com

WM Reply

From: Abhitosh Patil <[hidden email]>
Sent: 26 February 2021 14:28
To: JMeter Users List <[hidden email]>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report
 
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
pmd
Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

pmd
Hello,
Have you tried increasing this value in user.properties which defaults to 20000

On Fri, Mar 5, 2021 at 7:12 PM A P <[hidden email]> wrote:
Hi All,

I'm eagerly waiting for solution to this issue. If anyone has a fix please reply back.

AP

On Sun, Feb 28, 2021 at 11:59 PM A P <[hidden email]> wrote:
@ Mariusz
Yes.

On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <[hidden email]> wrote:
@Abhitosh
To be sure: So you loaded manually JTL file in Aggregate Report listener ( without CMDRunner ) and compared the results with html report?

Mariusz

On Sun, 28 Feb 2021 at 12:27, A P <[hidden email]> wrote:
@Mariusz
As per below post, HTML report has wrong values:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti

I automated aggregate report creation using JMeter plugin: CMDRunner

Also I tried creating HTML and Aggregate reports manually using a JTL file. But the results are showing the same difference.

My suggestion to reproduce this issue is:
Execute a long time JMeter load test with a considerable number of samples. Then compare HTML and aggregate reports for the test.
You can see the difference between them.

@Felix
But not getting a clear idea how to implement this.

On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <[hidden email]> wrote:

Hi Abhitosh,

it would have been a good idea to share sample data, that shows that behaviour.

I think you stumbled upon https://bz.apache.org/bugzilla/show_bug.cgi?id=61130

One problem is, that JMeter uses two different algorithms to calculate a percentile. One for the report and one for the dashboard. They differ in the point, which they use to display the percentile.

We did some work to use the same algorithm for both components in https://github.com/apache/jmeter/pull/380 which got closed when converting our repo from subversion to git.

In my eyes, that big discrepancy in your values shows, that you are having bad data in the first place. But of course, it would be nice, if JMeter would show consistent values, or even better an indicator of how close the value is to its neighbours.

Felix

Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
@Kieran Lynch
I'm using JMeter 5.4
I already tried "jmeter.reportgenerator.statistic_window = -1". But it doesn't work.

@Mariusz W
In my case it's a very huge difference (> 40 seconds) which is not acceptable at all as there are client facing reports.

@All
I had a discussion with my colleague. As per him there is difference in formula how Aggregate report and HTMl report calculates 90th percentile.
Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML report is picking 6th value (which is the highest one) as 90 th percentile.
I need to figure it out ASAP now.



On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <[hidden email]> wrote:
Yes, I have seen this small discrepancies also in my tests but I don't adhere to exact values and this reports are just first step during analysis to find an anomaly or something else to focus on , e.g. Is it worth to look at the mean when data is not symmetrically distributed. If it is not symmetrical should I care about standard deviation at all. If my histogram is strange - what was wrong.
But surely the meaning of the report depends on the client's need:)

Mariusz

On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <[hidden email]> wrote:
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:


Aggregate Report:



Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com

WM Reply

From: Abhitosh Patil <[hidden email]>
Sent: 26 February 2021 14:28
To: JMeter Users List <[hidden email]>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report
 
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


--
Cordialement.
Philippe Mouawad.


Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Abhitosh Patil
Yes, I tried " jmeter.reportgenerator.statistic_window = -1"
But it does not work.

On Fri, Mar 5, 2021 at 11:46 PM Philippe Mouawad <[hidden email]> wrote:
Hello,
Have you tried increasing this value in user.properties which defaults to 20000

On Fri, Mar 5, 2021 at 7:12 PM A P <[hidden email]> wrote:
Hi All,

I'm eagerly waiting for solution to this issue. If anyone has a fix please reply back.

AP

On Sun, Feb 28, 2021 at 11:59 PM A P <[hidden email]> wrote:
@ Mariusz
Yes.

On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <[hidden email]> wrote:
@Abhitosh
To be sure: So you loaded manually JTL file in Aggregate Report listener ( without CMDRunner ) and compared the results with html report?

Mariusz

On Sun, 28 Feb 2021 at 12:27, A P <[hidden email]> wrote:
@Mariusz
As per below post, HTML report has wrong values:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti

I automated aggregate report creation using JMeter plugin: CMDRunner

Also I tried creating HTML and Aggregate reports manually using a JTL file. But the results are showing the same difference.

My suggestion to reproduce this issue is:
Execute a long time JMeter load test with a considerable number of samples. Then compare HTML and aggregate reports for the test.
You can see the difference between them.

@Felix
But not getting a clear idea how to implement this.

On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <[hidden email]> wrote:

Hi Abhitosh,

it would have been a good idea to share sample data, that shows that behaviour.

I think you stumbled upon https://bz.apache.org/bugzilla/show_bug.cgi?id=61130

One problem is, that JMeter uses two different algorithms to calculate a percentile. One for the report and one for the dashboard. They differ in the point, which they use to display the percentile.

We did some work to use the same algorithm for both components in https://github.com/apache/jmeter/pull/380 which got closed when converting our repo from subversion to git.

In my eyes, that big discrepancy in your values shows, that you are having bad data in the first place. But of course, it would be nice, if JMeter would show consistent values, or even better an indicator of how close the value is to its neighbours.

Felix

Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
@Kieran Lynch
I'm using JMeter 5.4
I already tried "jmeter.reportgenerator.statistic_window = -1". But it doesn't work.

@Mariusz W
In my case it's a very huge difference (> 40 seconds) which is not acceptable at all as there are client facing reports.

@All
I had a discussion with my colleague. As per him there is difference in formula how Aggregate report and HTMl report calculates 90th percentile.
Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML report is picking 6th value (which is the highest one) as 90 th percentile.
I need to figure it out ASAP now.



On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <[hidden email]> wrote:
Yes, I have seen this small discrepancies also in my tests but I don't adhere to exact values and this reports are just first step during analysis to find an anomaly or something else to focus on , e.g. Is it worth to look at the mean when data is not symmetrically distributed. If it is not symmetrical should I care about standard deviation at all. If my histogram is strange - what was wrong.
But surely the meaning of the report depends on the client's need:)

Mariusz

On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <[hidden email]> wrote:
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:


Aggregate Report:



Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com

WM Reply

From: Abhitosh Patil <[hidden email]>
Sent: 26 February 2021 14:28
To: JMeter Users List <[hidden email]>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report
 
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


--
Cordialement.
Philippe Mouawad.


pmd
Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

pmd
-1 does not work
Did you try increasing it ?

Thanks

On Friday, March 5, 2021, A P <[hidden email]> wrote:
Yes, I tried " jmeter.reportgenerator.statistic_window = -1"
But it does not work.

On Fri, Mar 5, 2021 at 11:46 PM Philippe Mouawad <[hidden email]> wrote:
Hello,
Have you tried increasing this value in user.properties which defaults to 20000

On Fri, Mar 5, 2021 at 7:12 PM A P <[hidden email]> wrote:
Hi All,

I'm eagerly waiting for solution to this issue. If anyone has a fix please reply back.

AP

On Sun, Feb 28, 2021 at 11:59 PM A P <[hidden email]> wrote:
@ Mariusz
Yes.

On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <[hidden email]> wrote:
@Abhitosh
To be sure: So you loaded manually JTL file in Aggregate Report listener ( without CMDRunner ) and compared the results with html report?

Mariusz

On Sun, 28 Feb 2021 at 12:27, A P <[hidden email]> wrote:
@Mariusz
As per below post, HTML report has wrong values:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti

I automated aggregate report creation using JMeter plugin: CMDRunner

Also I tried creating HTML and Aggregate reports manually using a JTL file. But the results are showing the same difference.

My suggestion to reproduce this issue is:
Execute a long time JMeter load test with a considerable number of samples. Then compare HTML and aggregate reports for the test.
You can see the difference between them.

@Felix
But not getting a clear idea how to implement this.

On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <[hidden email]> wrote:

Hi Abhitosh,

it would have been a good idea to share sample data, that shows that behaviour.

I think you stumbled upon https://bz.apache.org/bugzilla/show_bug.cgi?id=61130

One problem is, that JMeter uses two different algorithms to calculate a percentile. One for the report and one for the dashboard. They differ in the point, which they use to display the percentile.

We did some work to use the same algorithm for both components in https://github.com/apache/jmeter/pull/380 which got closed when converting our repo from subversion to git.

In my eyes, that big discrepancy in your values shows, that you are having bad data in the first place. But of course, it would be nice, if JMeter would show consistent values, or even better an indicator of how close the value is to its neighbours.

Felix

Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
@Kieran Lynch
I'm using JMeter 5.4
I already tried "jmeter.reportgenerator.statistic_window = -1". But it doesn't work.

@Mariusz W
In my case it's a very huge difference (> 40 seconds) which is not acceptable at all as there are client facing reports.

@All
I had a discussion with my colleague. As per him there is difference in formula how Aggregate report and HTMl report calculates 90th percentile.
Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML report is picking 6th value (which is the highest one) as 90 th percentile.
I need to figure it out ASAP now.



On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <[hidden email]> wrote:
Yes, I have seen this small discrepancies also in my tests but I don't adhere to exact values and this reports are just first step during analysis to find an anomaly or something else to focus on , e.g. Is it worth to look at the mean when data is not symmetrically distributed. If it is not symmetrical should I care about standard deviation at all. If my histogram is strange - what was wrong.
But surely the meaning of the report depends on the client's need:)

Mariusz

On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <[hidden email]> wrote:
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:


Aggregate Report:



Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com

WM Reply

From: Abhitosh Patil <[hidden email]>
Sent: 26 February 2021 14:28
To: JMeter Users List <[hidden email]>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report
 
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


--
Cordialement.
Philippe Mouawad.




--
Cordialement.
Philippe Mouawad.



Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Abhitosh Patil
I'll try it.

On Sat, Mar 6, 2021 at 2:26 AM Philippe Mouawad <[hidden email]> wrote:
-1 does not work
Did you try increasing it ?

Thanks

On Friday, March 5, 2021, A P <[hidden email]> wrote:
Yes, I tried " jmeter.reportgenerator.statistic_window = -1"
But it does not work.

On Fri, Mar 5, 2021 at 11:46 PM Philippe Mouawad <[hidden email]> wrote:
Hello,
Have you tried increasing this value in user.properties which defaults to 20000

On Fri, Mar 5, 2021 at 7:12 PM A P <[hidden email]> wrote:
Hi All,

I'm eagerly waiting for solution to this issue. If anyone has a fix please reply back.

AP

On Sun, Feb 28, 2021 at 11:59 PM A P <[hidden email]> wrote:
@ Mariusz
Yes.

On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <[hidden email]> wrote:
@Abhitosh
To be sure: So you loaded manually JTL file in Aggregate Report listener ( without CMDRunner ) and compared the results with html report?

Mariusz

On Sun, 28 Feb 2021 at 12:27, A P <[hidden email]> wrote:
@Mariusz
As per below post, HTML report has wrong values:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti

I automated aggregate report creation using JMeter plugin: CMDRunner

Also I tried creating HTML and Aggregate reports manually using a JTL file. But the results are showing the same difference.

My suggestion to reproduce this issue is:
Execute a long time JMeter load test with a considerable number of samples. Then compare HTML and aggregate reports for the test.
You can see the difference between them.

@Felix
But not getting a clear idea how to implement this.

On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <[hidden email]> wrote:

Hi Abhitosh,

it would have been a good idea to share sample data, that shows that behaviour.

I think you stumbled upon https://bz.apache.org/bugzilla/show_bug.cgi?id=61130

One problem is, that JMeter uses two different algorithms to calculate a percentile. One for the report and one for the dashboard. They differ in the point, which they use to display the percentile.

We did some work to use the same algorithm for both components in https://github.com/apache/jmeter/pull/380 which got closed when converting our repo from subversion to git.

In my eyes, that big discrepancy in your values shows, that you are having bad data in the first place. But of course, it would be nice, if JMeter would show consistent values, or even better an indicator of how close the value is to its neighbours.

Felix

Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
@Kieran Lynch
I'm using JMeter 5.4
I already tried "jmeter.reportgenerator.statistic_window = -1". But it doesn't work.

@Mariusz W
In my case it's a very huge difference (> 40 seconds) which is not acceptable at all as there are client facing reports.

@All
I had a discussion with my colleague. As per him there is difference in formula how Aggregate report and HTMl report calculates 90th percentile.
Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML report is picking 6th value (which is the highest one) as 90 th percentile.
I need to figure it out ASAP now.



On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <[hidden email]> wrote:
Yes, I have seen this small discrepancies also in my tests but I don't adhere to exact values and this reports are just first step during analysis to find an anomaly or something else to focus on , e.g. Is it worth to look at the mean when data is not symmetrically distributed. If it is not symmetrical should I care about standard deviation at all. If my histogram is strange - what was wrong.
But surely the meaning of the report depends on the client's need:)

Mariusz

On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <[hidden email]> wrote:
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:


Aggregate Report:



Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com

WM Reply

From: Abhitosh Patil <[hidden email]>
Sent: 26 February 2021 14:28
To: JMeter Users List <[hidden email]>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report
 
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


--
Cordialement.
Philippe Mouawad.




--
Cordialement.
Philippe Mouawad.



Reply | Threaded
Open this post in threaded view
|

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Barcode S K
In reply to this post by pmd
I've also had this problem for a while now. The percentile values never come out correct as my results files grow. My results file usually grows to over 3 GBs.

As a sample, I'm sharing a stripped down version of my results file containing a little over 207k samples. I usually validate numbers using awk. I wrote a Python script ("py_calc.py") which does the same thing, to share with you guys. It looks simpler than the awk command I run.

From the script, these are the numbers I get for the given results file:

D:\> python py_calc.py mytxn.csv MyTXN
File to be analyzed: mytxn.csv
207557 lines in the file
207556 lines contain the sample
90th index: 186800
90th percentile (ms): 1868
Next value after 90th: 1868
Average (ms): 1049.18

So, I know that my 90th percentile should be 1868 ms. JMeter never reports it correctly with the default reportgenerator values. I've tried with JMeter 5.2 and 5.4.1.

90th percentile value with:
statistic_window default: 1783.90
statistic_window 60k: 1839.00
statistic_window 120k: 1900.00
statistic_window 200k: 1884.00
statistic_window 207556 (number of matching samples in my file): 1868.00
statistic_window 240k: 1868.00


So, to get the accurate number, my statistic_window has to be larger than or equal to the number of matching samples. Is the need to utilize less memory- as the reportgenerator.properties files says:
# Caution : higher value provides a better accuracy but needs more memory.
really worth the inaccurate percentile numbers? I don't suppose load generators nowadays are equipped with small memory modules.

-SK
PS: Unable to attach results CSV; file size over 2 MB even after compression.

On Sat, Mar 6, 2021 at 2:26 AM Philippe Mouawad <[hidden email]> wrote:
-1 does not work
Did you try increasing it ?

Thanks

On Friday, March 5, 2021, A P <[hidden email]> wrote:
Yes, I tried " jmeter.reportgenerator.statistic_window = -1"
But it does not work.

On Fri, Mar 5, 2021 at 11:46 PM Philippe Mouawad <[hidden email]> wrote:
Hello,
Have you tried increasing this value in user.properties which defaults to 20000

On Fri, Mar 5, 2021 at 7:12 PM A P <[hidden email]> wrote:
Hi All,

I'm eagerly waiting for solution to this issue. If anyone has a fix please reply back.

AP

On Sun, Feb 28, 2021 at 11:59 PM A P <[hidden email]> wrote:
@ Mariusz
Yes.

On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <[hidden email]> wrote:
@Abhitosh
To be sure: So you loaded manually JTL file in Aggregate Report listener ( without CMDRunner ) and compared the results with html report?

Mariusz

On Sun, 28 Feb 2021 at 12:27, A P <[hidden email]> wrote:
@Mariusz
As per below post, HTML report has wrong values:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti

I automated aggregate report creation using JMeter plugin: CMDRunner

Also I tried creating HTML and Aggregate reports manually using a JTL file. But the results are showing the same difference.

My suggestion to reproduce this issue is:
Execute a long time JMeter load test with a considerable number of samples. Then compare HTML and aggregate reports for the test.
You can see the difference between them.

@Felix
But not getting a clear idea how to implement this.

On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <[hidden email]> wrote:

Hi Abhitosh,

it would have been a good idea to share sample data, that shows that behaviour.

I think you stumbled upon https://bz.apache.org/bugzilla/show_bug.cgi?id=61130

One problem is, that JMeter uses two different algorithms to calculate a percentile. One for the report and one for the dashboard. They differ in the point, which they use to display the percentile.

We did some work to use the same algorithm for both components in https://github.com/apache/jmeter/pull/380 which got closed when converting our repo from subversion to git.

In my eyes, that big discrepancy in your values shows, that you are having bad data in the first place. But of course, it would be nice, if JMeter would show consistent values, or even better an indicator of how close the value is to its neighbours.

Felix

Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
@Kieran Lynch
I'm using JMeter 5.4
I already tried "jmeter.reportgenerator.statistic_window = -1". But it doesn't work.

@Mariusz W
In my case it's a very huge difference (> 40 seconds) which is not acceptable at all as there are client facing reports.

@All
I had a discussion with my colleague. As per him there is difference in formula how Aggregate report and HTMl report calculates 90th percentile.
Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML report is picking 6th value (which is the highest one) as 90 th percentile.
I need to figure it out ASAP now.



On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <[hidden email]> wrote:
Yes, I have seen this small discrepancies also in my tests but I don't adhere to exact values and this reports are just first step during analysis to find an anomaly or something else to focus on , e.g. Is it worth to look at the mean when data is not symmetrically distributed. If it is not symmetrical should I care about standard deviation at all. If my histogram is strange - what was wrong.
But surely the meaning of the report depends on the client's need:)

Mariusz

On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <[hidden email]> wrote:
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:


Aggregate Report:



Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK

phone: +44 (0)20 7730 6000
[hidden email]
www.reply.com

WM Reply

From: Abhitosh Patil <[hidden email]>
Sent: 26 February 2021 14:28
To: JMeter Users List <[hidden email]>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report
 
This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <[hidden email]> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <[hidden email]> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
image.png

HTML report (90th percentile is 44266):
image.png

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


--
Cordialement.
Philippe Mouawad.




--
Cordialement.
Philippe Mouawad.





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]