Elasticsearch¶

Live metrics can be sent to an external Elasticsearch while your scenario is running. To that end we use the JMeter elasticsearch plugin.

Each metric corresponds to a JMeter Sample which will be sent as a JSON document to Elasticsearch. The fields of this JSON document are configurable with:

Standard fields: a list of 35 fields are available
Custom fields: you can add your own fields as key/value pairs

Example of metric sent to Elasticsearch:

{
  "ScenarioName" : "Queen Parrotfish",
  "SampleLabel" : "request_main_page",
  "URL" : "https://octoperf.com/",
  "ThreadName" : "01 - ThreadGroup 1-1",
  "ContentType" : "text/html; charset=utf-8",
  "AllThreads" : 1,
  "TestStartTime" : 1655972857482,
  "SampleStartTime" : "2022-06-23T10:27:38.549+0200",
  "SampleEndTime" : "2022-06-23T10:27:38.770+0200",
  "SentBytes" : 167,
  ...
  "ResponseTime" : 221,
  "Bytes" : 9959,
  "BodySize" : 9460,
  "Latency" : 221,
  "InjectorId" : "turquoise_donovan_howell"
  "InjectorHostname" : "legion",
  "LocationName" : "default",
  "VirtualUserName" : "octoperf.com",
  "ElapsedTime" : 1642892401000,
  "Timestamp" : 1655972858549,
  "ErrorCount" : 0
}

Requirements¶

A running Elasticsearch accessible from the OctoPerf agent where the performance test will be executed:
If you are running the test on our cloud agents your Elasticsearch must be accessible publicly or through a dedicated IP,
If you are running the test from an on premise agent, your Elasticsearch must be accessible from this agent.

Octoperf configuration¶

Connection¶

All fields are pre-filled with default values. In most cases, you only need to fill hostname and username / password. (if Basic Authentication is required). Example with Authentication enabled for an on premise Elasticsearch:

configure-connection

Name	Description
Host Scheme	HTTP or HTTPS. In case your Elasticsearch is accessible via HTTPS and its certificate is self-signed, see SSL verification parameter below.
Host Name	The hostname or IP address of your Elasticsearch.
Host Port	The port of your Elasticsearch (Default port is 9200).
Index	The Elasticsearch index where data will be pushed. The index is supposed to have been correctly configured (mapping of fields to be queryable). Note that if the index doesn't exist, it will be automatically created (without any mapping) when the test starts.
Authentication	Enable Basic authentication. This is when your Elasticsearch is configured with HTTP basic authentication headers.
SSL verification	Only available if scheme is HTTPS. This option allows you to skip the hostname and certificate validation. It is only required to disable this validation if your certificate is self-signed.

Data sent¶

This section let's customize what data has to be sent to Elasticsearch. Example with default data sent and 2 custom fields added:

custom-fields

Name	Description
Verbosity	This parameter defines which sample results will be sent to Elasticsearch during the test. Default level is INFO. Check the documentation for more details.
Standard fields	There are 35 fields available with the default verbosity. OctoPerf adds the following on top of what the JMeter plugin provides: ScenarioName, LocationName, InjectorId, VirtualUserName.
Custom fields	Allows to specify custom data to be sent to Elasticsearch as Key/Value pairs.

Example¶

Let's say we have configured 2 custom fields like this:

custom-fields

In Elasticsearch we will have these fields:

{
  ...
  "CustomFieldKey1" : "CustomFieldValue1",
  "CustomFieldKey2" : "CustomFieldValue2",
 ...
}

Data sent (+)¶

This section is composed of advanced options regarding data sent customization: Filtering Samples by label, Send All Request or Response headers as fields or even changing some formatted date fields format.

Example with default configuration:

data-sent

Samples filtering¶

Allows filtering of specific Samples only (filtering based on Sample labels).

actions

Let's say we have these actions in our Virtual User and we only want to have results for:

01 - Container
request_jmeter_page

With a filter like this: 01;jmeter, only Samples having a label containing 01 or jmeter will be sent to Elasticsearch.

Headers sent as fields¶

This option sends Request/Response headers as fields to Elasticsearch. For example in our VU we have this GET request which has 2 headers: headers

If we activate Request headers sent as fields, in Elasticsearch we will have these fields:

{
  ...
  "headerKey1" : "headerValue1",
  "headerKey2" : "headerValue2",
 ...
}

Dates format¶

The date format to be used. This is based on Java Simple Date Format.

Sending¶

The last section is about changing the default way data is pushed to Elasticsearch. These options are for complex cases and should only be changed if the network between Elasticsearch and the agent is an issue (high latency or slow bandwidth).

Example with default configuration:

sending

Name	Description
Bulk size	Data is sent to Elasticsearch index through groups of JSON documents. This setting defines the number of JSON documents being sent at once. Default value is 100 and should be fine in most cases. You can increase this value if the latency/bandwidth between the location (injector) and your Elasticsearch is really good.
Socket timeout in ms	The time waiting for data – after establishing the connection; maximum time of inactivity between two data packets. Default value is 200 ms. You can increase this value if the network has degraded performance.

Elasticsearch index¶

To make fields pushed to your Elasticsearch searchable, the index mapping must be configured.

Example using Kibana dev console:

kibana

Depending on the standard/custom fields you have selected, configure your index mapping for the fields you want to make queryable.

Below a mapping configuration with all standard fields:

{
  "mappings": {
    "properties": {
      "AllThreads": {
        "type": "long"
      },
      "BodySize": {
        "type": "long"
      },
      "Bytes": {
        "type": "long"
      },
      "ConnectTime": {
        "type": "long"
      },
      "ContentType": {
        "type": "text"
      },
      "DataType": {
        "type": "text"
      },
      "ElapsedTime": {
        "type": "date"
      },
      "ErrorCount": {
        "type": "long"
      },
      "GrpThreads": {
        "type": "long"
      },
      "IdleTime": {
        "type": "long"
      },
       "InjectorHostname": {
        "type": "keyword"
      },
      "Latency": {
        "type": "long"
      },
      "ResponseTime": {
        "type": "long"
      },
      "SampleCount": {
        "type": "long"
      },
      "SampleLabel": {
        "type": "keyword"
      },
      "ThreadName": {
        "type": "keyword"
      },
      "ResponseCode": {
        "type": "text"
      },
      "TestStartTime": {
        "type": "date"
      },
      "SampleStartTime": {
        "type": "date",
        "format": "date_optional_time"
      },
      "SampleEndTime": {
        "type": "date",
        "format": "date_optional_time"
      },
      "SentBytes": {
        "type": "long"
      },
      "Timestamp": {
        "type": "date"
      },
      "URL": {
        "type": "keyword"
      },
      "AssertionResults": {
        "properties": {
          "failure": {
            "type": "boolean"
          },
          "failureMessage": {
            "type": "text"
          },
          "name": {
            "type": "text"
          }
        }
      },
      "FailureMessage": {
        "type": "text"
      },
      "Success": {
        "type": "boolean"
      },
      "RequestHeaders": {
        "type": "text"
      },
      "RequestBody": {
        "type": "text"
      },
      "ResponseHeaders": {
        "type": "text"
      },
      "ResponseBody": {
        "type": "text"
      },
      "ResponseMessage": {
        "type": "text"
      },
      "ScenarioName": {
        "type": "keyword"
      },
      "LocationName": {
        "type": "keyword"
      },
      "InjectorId": {
        "type": "keyword"
      },
      "VirtualUserName": {
        "type": "keyword"
      }
    }
  }
}

In the example of Data sent section, there are 2 custom fields. If we want them to be searchable:

"CustomFieldKey1": {
  "type": "keyword"
},
"CustomFieldKey2": {
  "type": "keyword"
}