> OctoPerf's load testing - Result analysis page. Quickly find out the most relevant information from a report. Optimization tips for your applications.

Source: https://api.octoperf.com/doc/analysis/tips/

# Analysis Tips

On this page, we will explore the most important information you should look at on the report.
These tips will help  you quickly analyze your test report and make the most out of it.

Note that we try to cover the most common use cases but may have missed some of them.
Feel free to report any other use case to us through the chat or [contact@octoperf.com](mailto:contact@octoperf.com).

## Response times

Obviously response times are going to be our first stop. Here a quick overview of how response times are measured:

![response-times](https://api.octoperf.com/doc/analysis/tips/img/response-times.png)

In this situation the container response time is the sum of its children (RT1+RT2+RT3), note that it can differ, if for instance you used the [parallel plugin](https://api.octoperf.com/doc/design/edit-virtual-user/plugins-usage/#parallel-controller).

Also response times of each individual request does not include the think times and other pre/postprocessors computing times.

If your APM shows different response times than OctoPerf, see [Why response times differ between OctoPerf and your APM](#why-response-times-differ-between-octoperf-and-your-apm).

### Basic response time graph

The most common one is the **Hits and response time** graph:

![Hits and response time](https://api.octoperf.com/doc/analysis/tips/img/hits-response-times.png "Hits and response time")

It shows the average response time across all containers (pages or transactions) for the whole test.
The one above is from a test that was successful, we can tell since the response time remained steady during the entire test.

Still, there may be additional things to check since a steady response time does not means it was acceptable.
This is what we're going to explore here.

### Global average response time

The global response time is a very good way to spot obvious issues.
Whenever the curve looks like this, you know something is wrong:
![Global response time increase](https://api.octoperf.com/doc/analysis/tips/img/global-response-time-increase.png "Global response time increase")

Here we clearly see that after 16:21 the response times increases.
It may not have reached a critical point yet but you've reached an **inflection point** and it is not likely to get any better later on.
Investigate for any event around that time, in particular on your [server monitoring](https://api.octoperf.com/doc/monitoring/).

!!! tip
    A response time curve is rarely flat, and it can be hard to tell normal noise from real instability. Add a [standard deviation band](https://api.octoperf.com/doc/analysis/edit-bench-report/report-items/line-chart/#spread) to the chart: a narrow band means a stable response time, a band that widens shows when it starts to vary.

    When your test increases the load by steps, the **Response Time vs Load** chart of the default report shows the same curve [against the number of users](https://api.octoperf.com/doc/analysis/edit-bench-report/report-items/line-chart/#x-axis): you read at which load the response time starts to grow, rather than at which time.

### Request or containers response times

OctoPerf shows the **average containers response time** by default but you can change that by editing the graph.
Be careful since **average container response times** include the response times of scripts and also technical elements like [if](https://api.octoperf.com/doc/design/edit-virtual-user/action-types/logic-actions/if-action/) and [while](https://api.octoperf.com/doc/design/edit-virtual-user/action-types/logic-actions/while-action/).
Because of that this time may be very high, for instance when you have a while that runs for a very long time.

Instead it could be more relevant to look at **average request response time**.
I say "may" because the average value across many requests may be very low and could hide problems because of that.
In the end it is critical to always look at transaction response times.

### Step by step response times

Here my recommendation is to check the result table.
Look for transactions that stand out by sorting by highest average response time:
![Highest average response time](https://api.octoperf.com/doc/analysis/tips/img/result-table-high-response-time.png "Highest average response time")

A good practice is to put these on a dedicated graph to see if they are always high or if there is another **inflection point**.

The [top chart](https://api.octoperf.com/doc/analysis/edit-bench-report/report-items/top-chart/) item does exactly this:

![Highest average response time](https://api.octoperf.com/doc/analysis/tips/img/top-chart.png "Highest average response time")

In this case we can clearly see when the slowest transaction hit a bottleneck, **this info is invaluable for proper root cause analysis**.

!!! tip
    You can spot them even faster using our built-in [SLAs](https://api.octoperf.com/doc/design/edit-virtual-user/action-types/post-processors/sla-action/).

### Min and max response times

Don't forget that you can also look at **min and max response times**.
Obviously that will give you insight on the best and worst transaction times.
If you notice an important difference, you should go have a look at [percentiles](#standard-deviation-and-percentiles).

## Hit rate

The hit rate is closely related to response times.
The **Hit rate** means how many of this ***request or container*** am I running each second.

When you see only **hits** it usually refers to the total number of times a ***request or container*** was run during the test.

### Inflexion point

Now it's easy to understand that when the response times get higher, you virtual users have to wait longer for a response and thus generate less hits overall.

This can take many forms but in general the global hit rate should follow the amount of users running.
When it doesn't it means there was a problem with the application, usually an increase of response times.
Sometimes the increase is small and barely noticeable but it's easier to notice on the hits rate.

On this example:
![Hit rate slows down](https://api.octoperf.com/doc/analysis/tips/img/hit-rate-slowing-down.png "Hit rate slows down")

The green area represents the difference between hits and users running, ideally it should be barely visible. Here we can see after a while that **the hits go down compared to the number of users**. This clearly indicates that the application is getting slower.
This means you must now analyze response times and find out if they were too high. Or user percentiles/maximum response times to check if a few outliers are causing this behavior.
But in any case you've spotted an **inflection point** that you should investigate.

!!! tip
    The **Hit Rate vs Load** chart of the default report plots the hit rate and the error rate [against the number of users](https://api.octoperf.com/doc/analysis/edit-bench-report/report-items/line-chart/#x-axis). As long as the application keeps up, the hit rate grows with the users; the load where the curve flattens is your inflection point.

### Synchronicity

When using no [think time](https://api.octoperf.com/doc/design/edit-virtual-user/action-types/http-actions/#think-time) or fixed [think times](https://api.octoperf.com/doc/design/edit-virtual-user/action-types/http-actions/#think-time), your users might be synchronized on the same pages at the same time.
This is more visible when one of the pages is larger than others or when it has a lot more resources (images, scripts, etc...).

If something goes temporarily wrong with the application it can intensify this phenomenon like this:

![Synchronicity](https://api.octoperf.com/doc/analysis/tips/img/synchronicity.png "Synchronicity")

We clearly see that users are running the same hits intensive pages almost at the same time.
It slowly goes back to normal but the load generated is very chaotic and it is hard to explain to stakeholders.
It is usually better to re-run the test with a bit of randomness in your [think times](https://api.octoperf.com/doc/design/edit-virtual-user/action-types/http-actions/#think-time).

## Error analysis

The error rate is often overlooked but it is also an important metric.
There are mostly two types of situations:

- Peaks of errors
- Recurring errors

We'll see what we can deduce from each one of these and also how to go further.

### Peak of errors

A peak of error would typically look like this:
![Peak of errors](https://api.octoperf.com/doc/analysis/tips/img/peak-errors.png "Peak of errors")

We can see two peaks during this test.
Notice how the hit rate matches the error rate.

This is usually because **the server is overloaded** and answers all requests quickly but with an error.
In which case, all pending requests get an answer over a short period but **that answer is an error message**.
If you do not look at the error rate you might think that the application is doing better when it's the opposite.

### Recurring errors

Some other times you may see recurring errors like this:
![Recurring errors](https://api.octoperf.com/doc/analysis/tips/img/recurring-errors.png "Recurring errors")

See how a small amount of errors occurs every minute?
There are two explanations possible here:

- Your test scripts do something invalid every minute. Try to run another test using random [think times](https://api.octoperf.com/doc/design/edit-virtual-user/action-types/http-actions/#think-time) inside your test scripts. This way errors will not be synchronized anymore.
- The infrastructure or application have a batch or protection that activates on a regular basis. Check for firewalls and proxies, also automatic tasks triggered by a recurring event that may cause this.

To go further you can have a look at the error rate per container or transaction and see if one of them stands out.

### Step by step errors

To see the error rate of a particular container, a good first step is to sort the result table by highest number of errors:
![High error rate](https://api.octoperf.com/doc/analysis/tips/img/result-table-high-error-rate.png "High error rate")

In this example we can see that some transactions fail all the time whereas others do not.
It is already a good indication of what might be the issue.
Try to confront that with server logs, check what the failing steps do in the application and you should have a good idea of what's the issue.

### Overall errors

You should also check the error section of the report for the percentage of response codes per type.
If a 4XX or 5XX code represents more than a few percent of the total response codes it is worth investigating.
Another possibility is that you get "none" as an error code:
![Overall errors](https://api.octoperf.com/doc/analysis/tips/img/overall-errors.png "Overall errors")

This means that the request got no response at all: JMeter raised a Java error instead, or K6 recorded a status `0`.
The most common situation is that there was a timeout or the remote server was not able to answer your request. The **Requests without response** insight of the [insights](https://api.octoperf.com/doc/analysis/edit-bench-report/report-items/insights/) report item counts them.
You can find out by opening the error details in the **Error table** below:
![Error details](https://api.octoperf.com/doc/analysis/tips/img/error.png "Error details")

!!! tip
    To find out more about each error type, check the [error table documentation](https://api.octoperf.com/doc/analysis/edit-bench-report/report-items/errors-table/#understanding-errors).

### Error rate

Another important graph is error rate per error code:

![Error rate](https://api.octoperf.com/doc/analysis/tips/img/error-rate.png "Error rate")

In this example we can clearly see that most 504 errors occur during a very short period of time.
This tells a very different story than just looking at errors across the entire test:

![error-rate-global](https://api.octoperf.com/doc/analysis/tips/img/error-rate-global.png "Error rate global")

You could also have a behavior like this:

![Error rate](https://api.octoperf.com/doc/analysis/tips/img/high-error-rate.png "High error rate")

In this case we see multiple changes in the response codes over time, indicating several problematic behaviors.

## Latency and connect time

Latency and connect time are also often overlooked.

They are subsets of the response time, that correspond to specific events:
![Latency](https://api.octoperf.com/doc/analysis/tips/img/latency.png "Latency")

Latency is the time until we receive the first byte and connect time is how long it took to initiate the HTTP connection.

!!! tip
    The **Connect Time / Latency / Response Time vs Load** chart, in the **SLA** section of the default report, plots these three times [against the number of users](https://api.octoperf.com/doc/analysis/edit-bench-report/report-items/line-chart/#x-axis). It tells you whether the connect time or the latency is the one that grows with the load, which the sections below help you interpret.

### Connect time increase

When the connect time increases like this:
![Connect time increase](https://api.octoperf.com/doc/analysis/tips/img/connect-time-increase.png "Connect time increase")

It means that reaching the application is taking longer and longer.
It usually points to a network issue, but keep in mind that connect time also includes SSL handshake and other technical operations.

So you need to consider if the majority of the time is spent waiting for a connection.
If so it might be an SSL issue.

If instead it gets worse when you add load, it is probably a network issue.
If so you should see **connection reset** errors or **timeouts**.

### Latency increase

When the latency increases:
![Latency increase](https://api.octoperf.com/doc/analysis/tips/img/latency-increase.png "Latency increase")

It means that you spend more and more time waiting for your servers to answer.
So obviously the issue here is that your server is overloaded.
You need to check your [server monitoring](https://api.octoperf.com/doc/monitoring/) to find out why.
The good thing is that you can rule out any netwrok related issue for now.

### Response time increase

When the response time is very high in the beginning of the test:
![Response time increase](https://api.octoperf.com/doc/analysis/tips/img/response-time-increase.png "Response time increase")
Initially latency is also high, but as the test goes, the issue seems to be shifting toward the network. With network being overloaded, the server is no longer facing as much load as he was and the latency goes down.

## Throughput

Throughput can also tell us a lot about the application under test.

In this example we can see that most of the bandwidth is composed of images:
![Throughput images](https://api.octoperf.com/doc/analysis/tips/img/throughput-img.png "Throughput images")

In this situation, if you want to reduce the bandwidth used and also the download time, there are a few solutions available:

- Use an in-memory caching solution to deliver resources faster.
- Use a CDN to deliver resources from a source close to each user.
- Optimize the size of your images. There are many possibilities here and most of them allow for lossless compression that can reduce size significantly.

You may also have noticed that other resources like JS or CSS can take a lot of bandwidth.
This could mean that you did not minify or compress these files.
[Minification](https://en.wikipedia.org/wiki/Minification_(programming)) is a good way to save a lot of space by removing all unnecessary characters. 
Add some compression on top of it and you can reduce by a factor of at least 10 the size of these files.
Some systems have what is called a **production mode** that will take care of this once activated, make sure to do so.

While this is being done, you may want to consider **running another test without [clearing the cache](https://api.octoperf.com/doc/runtime/edit-scenario/edit-user-profile/options/device/#cache) on each iteration**, that way the impact of these resources on the bandwidth will be lowered.

## Standard deviation and percentiles

Average values are a good start but going further it is important to also rely on [variance/deviation](https://api.octoperf.com/doc/analysis/edit-bench-report/performance-metrics/#hit-metrics-types) and [percentiles](https://api.octoperf.com/doc/analysis/edit-bench-report/report-items/percentiles-chart/).

### Standard deviation

First, deviation shows us if response times are very different to each others.
What's interesting is not the value in itself, but **the trend:**
![Deviation](https://api.octoperf.com/doc/analysis/tips/img/deviation.png "Deviation")

We clearly see that it is getting worse during a sizeable portion of the test after 11h51.

A quick look at the **result table** can also tell us if the deviation is high when average and percentile are very different:
![High percentiles](https://api.octoperf.com/doc/analysis/tips/img/high-percentiles.png "High percentiles")

In this case 90th percentile is close to 10x the average.

### Response time distribution

And another interesting graph is **Response Times distribution**:
![Percentiles](https://api.octoperf.com/doc/analysis/tips/img/percentiles.png "Percentiles")

Again we can see a high disparity in response times.
This indicates that the application behavior is very unstable.

Even if you consider at least some of these response times ok, users will have a very different experience.
And a good portion of them will have very high response times.
It is important to spot this and **fix it before releasing the application in production.**

### Flooring

Flooring is easily missed when you only look at averages or aggregated data.
This happens **when you consistently have two very different values for a given metric.**

For example here we can see that almost half of the response times are 0.5 sec higher than the others:
![Flooring percentiles](https://api.octoperf.com/doc/analysis/tips/img/flooring-percentiles.png "Flooring percentiles")
The fact that **the 55th percentile is 1 sec higher** instead of having a progressive increase when going to higher centiles is a strong clue.

Looking at the average response time it is almost impossible to tell:
![Flooring average](https://api.octoperf.com/doc/analysis/tips/img/flooring-average.png "Flooring average")
A small clue could be the instability, but it also has other explanations so it should not be trusted as hard evidence in this case.

But when we filter on the response time from different load generators we can clearly see it:
![Flooring filtered](https://api.octoperf.com/doc/analysis/tips/img/flooring-filtered.png "Flooring filtered")
In this case [bandwidth emulation](https://api.octoperf.com/doc/runtime/edit-scenario/edit-user-profile/options/device/#bandwidth) was used to slow down one of the groups.
We can clearly see that **half the load is actually having all the influence** on the average response time.
The other half is very fast and almost invisible in that regard.

## Why response times differ between OctoPerf and your APM

OctoPerf and your Application Performance Management (APM) tool time the same request from two places, so both numbers can be right and still differ. OctoPerf measures what the user waits for, from the load generator. Your APM measures only the time the request spends in the instrumented application code.

### What each tool measures

![How OctoPerf and an APM time the same request](https://api.octoperf.com/doc/analysis/tips/img/request-timeline.svg "How OctoPerf and an APM time the same request")

| OctoPerf metric       | What it contains                                                                                                   |
|-----------------------|--------------------------------------------------------------------------------------------------------------------|
| Response Time         | The whole trip in the diagram above                                                                                |
| Connect Time          | DNS lookup, TCP and TLS setup, only when a new connection is opened                                                |
| Latency (Server Time) | Everything up to the first byte: connect time, both network trips, what sits in front of the application, and the application code |
| Network Time          | The download of the response after its first byte                                                                  |

!!! note
    Despite its name, **Latency (Server Time)** is measured on the load generator, not on the server. The application's share is usually at most latency minus connect time.

### Where the gap comes from

| What you see                                    | Likely cause                                                                                    | What to check                                                                                                                                                                  |
|-------------------------------------------------|-------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| The same gap at 1 user and at peak load         | Distance to the application, TLS handshakes, load balancers, proxies or firewalls in front of it | Run a short test from a [location](https://api.octoperf.com/doc/runtime/edit-scenario/edit-user-profile/location/) close to the application                                                       |
| Latency grows with load, the APM stays flat     | Requests wait in a queue before the application code runs                                       | Worker threads, connection limits, load balancer; some APMs report it as request queuing                                                                                      |
| High connect time                               | A new connection, and its TLS handshake, at each iteration; if it rises with the load, a network problem | [Connection Keep Alive](https://api.octoperf.com/doc/runtime/edit-scenario/edit-user-profile/options/device/#connection-keep-alive), [DNS options](https://api.octoperf.com/doc/runtime/edit-scenario/edit-user-profile/options/dns/), [Connect time increase](#connect-time-increase) |
| High network time                               | Large responses or limited bandwidth                                                            | Response size, [Bandwidth](https://api.octoperf.com/doc/runtime/edit-scenario/edit-user-profile/options/device/#bandwidth), [Load Generators Hosts](https://api.octoperf.com/doc/analysis/edit-bench-report/report-items/lg-hosts/) |
| Every request slows down at once, not in the APM | A saturated load generator, whose response times are no longer accurate                         | CPU, memory and garbage collection in [Load Generators Hosts](https://api.octoperf.com/doc/analysis/edit-bench-report/report-items/lg-hosts/) and [JVMs](https://api.octoperf.com/doc/analysis/edit-bench-report/report-items/lg-jvms/)     |
| A container is slower than the APM transaction  | A [container](https://api.octoperf.com/doc/design/edit-virtual-user/action-types/logic-actions/container-action/) adds up several requests, resources and redirects included | Compare a single request                                                                                                                       |
| OctoPerf is faster than the APM                 | Errors and cache hits answered before the application: the APM averages only the slower requests | [Errors Table](https://api.octoperf.com/doc/analysis/edit-bench-report/report-items/errors-table/)                                                                                                            |
| Same curves, different summary numbers          | Another statistic, time window or traffic mix                                                   | The steps below                                                                                                                                                                |

### How to compare

1. **Same request**: a single HTTP request, and the APM transaction with the same method and endpoint.
2. **Same window**: narrow the report with its [Time Range](https://api.octoperf.com/doc/analysis/edit-bench-report/configuration/#time-range) to the steady part of the test, and give the APM the same start, end and time zone.
3. **Same statistic**: average with average, or the same percentile, over that whole window.
4. **Test traffic only**: activate the [APM integration](https://api.octoperf.com/doc/runtime/edit-scenario/integrations-automation/apm/) of your project: OctoPerf adds headers to its requests, which your APM can filter on once it is set up to read them. With another APM, add a [request header](https://api.octoperf.com/doc/design/edit-virtual-user/action-types/http-actions/#request-headers) of your own and filter on it.

    ![APM integrations](https://api.octoperf.com/doc/runtime/edit-scenario/integrations-automation/img/application-performance-management-panel.png "APM integrations")

5. **The gap against the load**: the **Connect Time / Latency / Response Time vs Load** chart, in the **SLA** section of the default report, plots these three times [against the number of users](https://api.octoperf.com/doc/analysis/edit-bench-report/report-items/line-chart/#x-axis). Once the [load generators](https://api.octoperf.com/doc/analysis/edit-bench-report/report-items/lg-hosts/) are ruled out, what stays flat is distance and infrastructure, and what grows with the users is queueing or saturation in front of the application.

    ![OctoPerf latency and APM time as the users increase](https://api.octoperf.com/doc/analysis/tips/img/gap-against-load.svg "OctoPerf latency and APM time as the users increase")

### Example

![Of the 420 ms OctoPerf measures, the APM sees 180 ms](https://api.octoperf.com/doc/analysis/tips/img/response-time-split.svg "Of the 420 ms OctoPerf measures, the APM sees 180 ms")

OctoPerf shows 420 ms and the APM 180 ms. Connect time and network time explain 130 ms of the 240 ms gap; the other 110 ms is the network round trip and the wait in front of the application, the part to watch as the load grows.