Issues in JMETER Dashboard Latency Percentile Calculations

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Issues in JMETER Dashboard Latency Percentile Calculations

Malith Jayasinghe
Dear All,

We have been using JMETER extensively for performance testing.

All our performance reports are generated using the JMETER Dashboard.

Recently, we have found out that there is an issue in
the JMETER dashboard's *latency percentile calculation. *

There is a significant difference in the actual percentile values and the
values shown in the dashboard. We have verified this using "R" and in fact,
the values shown the JMETER aggregate report are correct. The values that
appear in the JMETER Dashboard are not.

We understand that JMETER Dashboard may be using a different algorithm
(approximation) for percentile calculation. However, the problem is that
that latency percentiles values calculated using this algorithm are not
accurate (at least for certain scenarios).

We have had a case where the Dashboard showing a 95% percentile value 1000
ms. However, the correct value was 321 ms.

The accuracy of percentile values are of utmost importance to us and we
would kindly ask you to look into this issue and release a patch if
possible.

If you can implement the same algorithm used in the aggregate report within
the dashboard that will resolve this issue.


Thanks

Malith Jayasinghe
pmd
Reply | Threaded
Open this post in threaded view
|

Re: Issues in JMETER Dashboard Latency Percentile Calculations

pmd
Hello,
You can adjust accuracy by setting in user.properties:

   - jmeter.reportgenerator.statistic_window = 20000

Caution : higher value provides a better accuracy but needs more memory.

This is not a bug.

We use DescriptiveStatistics from commons-math with a sliding window.

So if after modifying this value you still face issue, please report and
open a bug at commons-math.

Regards


On Thu, Nov 23, 2017 at 5:04 AM, Malith Jayasinghe <
[hidden email]> wrote:

> Dear All,
>
> We have been using JMETER extensively for performance testing.
>
> All our performance reports are generated using the JMETER Dashboard.
>
> Recently, we have found out that there is an issue in
> the JMETER dashboard's *latency percentile calculation. *
>
> There is a significant difference in the actual percentile values and the
> values shown in the dashboard. We have verified this using "R" and in fact,
> the values shown the JMETER aggregate report are correct. The values that
> appear in the JMETER Dashboard are not.
>
> We understand that JMETER Dashboard may be using a different algorithm
> (approximation) for percentile calculation. However, the problem is that
> that latency percentiles values calculated using this algorithm are not
> accurate (at least for certain scenarios).
>
> We have had a case where the Dashboard showing a 95% percentile value 1000
> ms. However, the correct value was 321 ms.
>
> The accuracy of percentile values are of utmost importance to us and we
> would kindly ask you to look into this issue and release a patch if
> possible.
>
> If you can implement the same algorithm used in the aggregate report within
> the dashboard that will resolve this issue.
>
>
> Thanks
>
> Malith Jayasinghe
>



--
Cordialement.
Philippe Mouawad.
Reply | Threaded
Open this post in threaded view
|

Re: Issues in JMETER Dashboard Latency Percentile Calculations

Malith Jayasinghe
Thanks a lot Philippe. We will check on this and get back.

On Fri, Nov 24, 2017 at 4:02 AM, Philippe Mouawad <
[hidden email]> wrote:

> Hello,
> You can adjust accuracy by setting in user.properties:
>
>    - jmeter.reportgenerator.statistic_window = 20000
>
> Caution : higher value provides a better accuracy but needs more memory.
>
> This is not a bug.
>
> We use DescriptiveStatistics from commons-math with a sliding window.
>
> So if after modifying this value you still face issue, please report and
> open a bug at commons-math.
>
> Regards
>
>
> On Thu, Nov 23, 2017 at 5:04 AM, Malith Jayasinghe <
> [hidden email]> wrote:
>
> > Dear All,
> >
> > We have been using JMETER extensively for performance testing.
> >
> > All our performance reports are generated using the JMETER Dashboard.
> >
> > Recently, we have found out that there is an issue in
> > the JMETER dashboard's *latency percentile calculation. *
> >
> > There is a significant difference in the actual percentile values and the
> > values shown in the dashboard. We have verified this using "R" and in
> fact,
> > the values shown the JMETER aggregate report are correct. The values that
> > appear in the JMETER Dashboard are not.
> >
> > We understand that JMETER Dashboard may be using a different algorithm
> > (approximation) for percentile calculation. However, the problem is that
> > that latency percentiles values calculated using this algorithm are not
> > accurate (at least for certain scenarios).
> >
> > We have had a case where the Dashboard showing a 95% percentile value
> 1000
> > ms. However, the correct value was 321 ms.
> >
> > The accuracy of percentile values are of utmost importance to us and we
> > would kindly ask you to look into this issue and release a patch if
> > possible.
> >
> > If you can implement the same algorithm used in the aggregate report
> within
> > the dashboard that will resolve this issue.
> >
> >
> > Thanks
> >
> > Malith Jayasinghe
> >
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
pmd
Reply | Threaded
Open this post in threaded view
|

Re: Issues in JMETER Dashboard Latency Percentile Calculations

pmd
In reply to this post by pmd
Hello,
Any feedback on this ?
Are you the opener of this ?:

- https://bz.apache.org/bugzilla/show_bug.cgi?id=61725

Thanks

On Thu, Nov 23, 2017 at 11:32 PM, Philippe Mouawad <
[hidden email]> wrote:

> Hello,
> You can adjust accuracy by setting in user.properties:
>
>    - jmeter.reportgenerator.statistic_window = 20000
>
> Caution : higher value provides a better accuracy but needs more memory.
>
> This is not a bug.
>
> We use DescriptiveStatistics from commons-math with a sliding window.
>
> So if after modifying this value you still face issue, please report and
> open a bug at commons-math.
>
> Regards
>
>
> On Thu, Nov 23, 2017 at 5:04 AM, Malith Jayasinghe <
> [hidden email]> wrote:
>
>> Dear All,
>>
>> We have been using JMETER extensively for performance testing.
>>
>> All our performance reports are generated using the JMETER Dashboard.
>>
>> Recently, we have found out that there is an issue in
>> the JMETER dashboard's *latency percentile calculation. *
>>
>> There is a significant difference in the actual percentile values and the
>> values shown in the dashboard. We have verified this using "R" and in
>> fact,
>> the values shown the JMETER aggregate report are correct. The values that
>> appear in the JMETER Dashboard are not.
>>
>> We understand that JMETER Dashboard may be using a different algorithm
>> (approximation) for percentile calculation. However, the problem is that
>> that latency percentiles values calculated using this algorithm are not
>> accurate (at least for certain scenarios).
>>
>> We have had a case where the Dashboard showing a 95% percentile value 1000
>> ms. However, the correct value was 321 ms.
>>
>> The accuracy of percentile values are of utmost importance to us and we
>> would kindly ask you to look into this issue and release a patch if
>> possible.
>>
>> If you can implement the same algorithm used in the aggregate report
>> within
>> the dashboard that will resolve this issue.
>>
>>
>> Thanks
>>
>> Malith Jayasinghe
>>
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>


--
Cordialement.
Philippe Mouawad.
Reply | Threaded
Open this post in threaded view
|

Re: Issues in JMETER Dashboard Latency Percentile Calculations

Malith Jayasinghe
.Hi Phiilppe,

No, I did not create this issue.   There is a significant increase in the
accuracy when using "large" window sizes.

Thanks

Malith


On Wed, Nov 29, 2017 at 3:53 AM, Philippe Mouawad <
[hidden email]> wrote:

> Hello,
> Any feedback on this ?
> Are you the opener of this ?:
>
> - https://bz.apache.org/bugzilla/show_bug.cgi?id=61725
>
> Thanks
>
> On Thu, Nov 23, 2017 at 11:32 PM, Philippe Mouawad <
> [hidden email]> wrote:
>
> > Hello,
> > You can adjust accuracy by setting in user.properties:
> >
> >    - jmeter.reportgenerator.statistic_window = 20000
> >
> > Caution : higher value provides a better accuracy but needs more memory.
> >
> > This is not a bug.
> >
> > We use DescriptiveStatistics from commons-math with a sliding window.
> >
> > So if after modifying this value you still face issue, please report and
> > open a bug at commons-math.
> >
> > Regards
> >
> >
> > On Thu, Nov 23, 2017 at 5:04 AM, Malith Jayasinghe <
> > [hidden email]> wrote:
> >
> >> Dear All,
> >>
> >> We have been using JMETER extensively for performance testing.
> >>
> >> All our performance reports are generated using the JMETER Dashboard.
> >>
> >> Recently, we have found out that there is an issue in
> >> the JMETER dashboard's *latency percentile calculation. *
> >>
> >> There is a significant difference in the actual percentile values and
> the
> >> values shown in the dashboard. We have verified this using "R" and in
> >> fact,
> >> the values shown the JMETER aggregate report are correct. The values
> that
> >> appear in the JMETER Dashboard are not.
> >>
> >> We understand that JMETER Dashboard may be using a different algorithm
> >> (approximation) for percentile calculation. However, the problem is that
> >> that latency percentiles values calculated using this algorithm are not
> >> accurate (at least for certain scenarios).
> >>
> >> We have had a case where the Dashboard showing a 95% percentile value
> 1000
> >> ms. However, the correct value was 321 ms.
> >>
> >> The accuracy of percentile values are of utmost importance to us and we
> >> would kindly ask you to look into this issue and release a patch if
> >> possible.
> >>
> >> If you can implement the same algorithm used in the aggregate report
> >> within
> >> the dashboard that will resolve this issue.
> >>
> >>
> >> Thanks
> >>
> >> Malith Jayasinghe
> >>
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
> >
> >
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
Reply | Threaded
Open this post in threaded view
|

Re: Issues in JMETER Dashboard Latency Percentile Calculations

Malith Jayasinghe
A detailed discussion of this issue can be found in here
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti

On Wed, Nov 29, 2017 at 5:43 AM, Malith Jayasinghe <
[hidden email]> wrote:

> .Hi Phiilppe,
>
> No, I did not create this issue.   There is a significant increase in the
> accuracy when using "large" window sizes.
>
> Thanks
>
> Malith
>
>
> On Wed, Nov 29, 2017 at 3:53 AM, Philippe Mouawad <
> [hidden email]> wrote:
>
>> Hello,
>> Any feedback on this ?
>> Are you the opener of this ?:
>>
>> - https://bz.apache.org/bugzilla/show_bug.cgi?id=61725
>>
>> Thanks
>>
>> On Thu, Nov 23, 2017 at 11:32 PM, Philippe Mouawad <
>> [hidden email]> wrote:
>>
>> > Hello,
>> > You can adjust accuracy by setting in user.properties:
>> >
>> >    - jmeter.reportgenerator.statistic_window = 20000
>> >
>> > Caution : higher value provides a better accuracy but needs more memory.
>> >
>> > This is not a bug.
>> >
>> > We use DescriptiveStatistics from commons-math with a sliding window.
>> >
>> > So if after modifying this value you still face issue, please report and
>> > open a bug at commons-math.
>> >
>> > Regards
>> >
>> >
>> > On Thu, Nov 23, 2017 at 5:04 AM, Malith Jayasinghe <
>> > [hidden email]> wrote:
>> >
>> >> Dear All,
>> >>
>> >> We have been using JMETER extensively for performance testing.
>> >>
>> >> All our performance reports are generated using the JMETER Dashboard.
>> >>
>> >> Recently, we have found out that there is an issue in
>> >> the JMETER dashboard's *latency percentile calculation. *
>> >>
>> >> There is a significant difference in the actual percentile values and
>> the
>> >> values shown in the dashboard. We have verified this using "R" and in
>> >> fact,
>> >> the values shown the JMETER aggregate report are correct. The values
>> that
>> >> appear in the JMETER Dashboard are not.
>> >>
>> >> We understand that JMETER Dashboard may be using a different algorithm
>> >> (approximation) for percentile calculation. However, the problem is
>> that
>> >> that latency percentiles values calculated using this algorithm are not
>> >> accurate (at least for certain scenarios).
>> >>
>> >> We have had a case where the Dashboard showing a 95% percentile value
>> 1000
>> >> ms. However, the correct value was 321 ms.
>> >>
>> >> The accuracy of percentile values are of utmost importance to us and we
>> >> would kindly ask you to look into this issue and release a patch if
>> >> possible.
>> >>
>> >> If you can implement the same algorithm used in the aggregate report
>> >> within
>> >> the dashboard that will resolve this issue.
>> >>
>> >>
>> >> Thanks
>> >>
>> >> Malith Jayasinghe
>> >>
>> >
>> >
>> >
>> > --
>> > Cordialement.
>> > Philippe Mouawad.
>> >
>> >
>> >
>>
>>
>> --
>> Cordialement.
>> Philippe Mouawad.
>>
>
>