Source: https://api.octoperf.com/doc/on-premise-agent/provider-type/on-premise/
# On-Premise Agents
## Introduction
An **On-Premise Agent is a small Docker container running on a physical machine**. It provides the following features:
- [Monitor your system](https://api.octoperf.com/doc/monitoring/): monitor various parts of your system,
- **Load Generator**: start / stop / remove JMeter containers to run your load tests.
The *Agents* table shows the following information:
| **Column** | **Description** |
|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **State** | **Green** when connected to the server. It is **Red** when the agent was [disconnected less than an hour ago](#stop-an-agent). |
| **Name** | Name of the docker container running the docker-agent. |
| **IP** | IP address of the computer running the agent. Several agents can run behind a single IP, but make sure they are not on the same machine otherwise they will step on each other. |
| **Provider** | Name of the Provider running this agent. |
| **Region** | Name of the Region in which this agent was launched. |
| **Free Memory** | Percent and available memory on this machine. Only this amount of memory is available for tests, if it is low even at rest, then you need to stop other processes on the agent machine. |
| **Containers** | Number of Docker containers running on this machine. This counts OctoPerf containers and also other containers you might have on this machine. |
| **Last Update** | When the agent has been updated for the last time. |
!!! warning
The Name, Provider and Region of an agent cannot be updated, you must re-install the agent to update them.
!!! info
Before each launch, the [scenario check](https://api.octoperf.com/doc/runtime/edit-scenario/scenario-check/#agents-and-capacity) places the load generators of the scenario on the agents that are up, the way a test does. A test can only start when its location has an agent up ([No agent ready in this location](https://api.octoperf.com/doc/runtime/edit-scenario/scenario-check/#agent-missing)) with enough memory for its load ([Not enough capacity on the agents](https://api.octoperf.com/doc/runtime/edit-scenario/scenario-check/#capacity-exceeded)). The memory of an agent is what the agent reports: the RAM of its machine, or for an agent on Kubernetes the memory of the nodes of its cluster. The *Amount of memory per agent in MB* of the provider only sizes the load generators: a load generator never takes more than this value times the percentage of the [Memory usage](https://api.octoperf.com/doc/on-premise-agent/providers/#memory-usage). Set it to the real RAM of the machines, see [Memory configuration](https://api.octoperf.com/doc/on-premise-agent/providers/#configuration): a higher value makes load generators too big for the agents.
## Start an agent
The following procedure explains how to launch an on-premise load generator:
1. [Create an On-Premise Provider](https://api.octoperf.com/doc/on-premise-agent/providers/#create-a-new-provider),
2. Back to the providers list, click on the _Add Agent_ button,
4. Select the *Region* where the agent will be running,
5. Select the Operating System (Windows or Linux),
6. Follow the instructions related to the OS.
**This procedure must be followed again every time you want to install another agent** otherwise your agents will share the same display name, making them harder to identify. Changing the name inside the command line will only update the docker container name, not the name we use in OctoPerf (it is embedded in the token). This is especially important if you install several agents behind the same IP.
!!! warning
Only a **single agent per host** must be launched. Having two agents will only use more resources and make them both slower. You may also experience issues like auto-upgrade failure, or race condition during test launch. Each agent also reports the whole memory of the machine, so it counts twice: the **Providers** page and the scenario check report it as [Agents sharing a machine](https://api.octoperf.com/doc/runtime/edit-scenario/scenario-check/#agents-sharing-a-machine).
### Windows
OctoPerf agents run on linux [Docker](https://docker.com) images. For that reason it is only possible to execute OctoPerf agents from windows machines through linux emulation. Once you have a linux VM running, refer to the [linux](#linux) section of this documentation.
!!! warning
Installing an agent on a linux virtual machine is perfectly suitable for evaluation purpose but in the long run it is harder to maintain than a native linux agent.
#### WSL2
If you are running the latest versions of windows 10 you can check our [WSL2 tutorial](https://api.octoperf.com/doc/on-premise-infra/tutorials/wsl2-enterprise-deployment/). WSL2 provides an efficient way to abstract the linux virtual machine and run docker almost seamlessly.
### Linux
OctoPerf agents run on [Docker](https://docker.com). If you already have docker on your Linux machine you can skip the first command line.
1. Click on the `wget` command to copy it,
2. Paste it on your terminal to install Docker,
3. Click on the `docker` command to copy it,
4. Execute the command on the target computer.
The Agent should appear in the list in a couple of minutes. Otherwise you can refer to the [troubleshooting section](#troubleshooting).
### Networking
An OctoPerf Agent requires Internet access for two requirements, fetching docker images and connecting to its server.
!!! warning
Make sure you do not go through a proxy when accessing the application under test. Otherwise you could be load testing the proxy and cause network issues inside your organization.
#### Docker images
Docker images will be fetched from [Dockerhub](https://hub.docker.com/) by default. Because of this you will most likely end up in one of the following situations:
1. Unrestricted Internet access: **No action is required**,
2. Internet access through proxy: You must direct all traffic from docker through the proxy. The procedure is detailed in [docker documentation](https://docs.docker.com/network/proxy/),
3. Your organization has its own private docker registry: You need to [change the docker registry](https://docs.docker.com/engine/reference/commandline/pull/#pull-from-a-different-registry) when executing the command line if it's not already configured and get the [OctoPerf Images](https://api.octoperf.com/doc/release-notes/) to your private registry,
4. No internet access possible: Check our [offline installation guide](https://api.octoperf.com/doc/on-premise-infra/tutorials/offline-deployment/#agent-installation) for more details.
#### OctoPerf server
The OctoPerf agent will need to communicate with its server. The traffic always originates from the agent, so it's a one-way route to open:
- [Saas](https://api.octoperf.com/): Firewall outgoing rules must allow the agent to send HTTPS (443) requests to **https://api.octoperf.com**,
- [On premise infra](https://api.octoperf.com/doc/on-premise-infra/): Firewall rules must allow traffic between the agent and the server over HTTP (80) or HTTPS (443) depending on how your on premise infra is configured.
The Agent itself can be configured to use a proxy to communicate with its OctoPerf server via [environment properties](https://docs.docker.com/engine/reference/commandline/run/) when running the agent. The agent is a [Java Program](https://en.wikipedia.org/wiki/Java_(programming_language)) :
- `-e HTTP_PROXY='hostname'`: the host name of the proxy server (HTTP),
- `-e HTTPS_PROXY='hostname'`: the host name of the proxy server (HTTPS),
- `-e NO_PROXY=''`: a list of hosts that should be reached directly, bypassing the proxy.
If your proxy is *SSL-Secured*, use `https` instead of `http`. For a more complete documentation, see [Configuring Proxies](https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html) documentation.
!!! warning
OctoPerf agents reuse the docker proxy by default. If you already defined the proxy at docker level, this procedure is not required. Except if you want to bypass the docker proxy instead.
### Configuration
The agent can be configured using environement variables. You can specify an environement variable within the agent docker command using `--env` or `-e` parameter. See [Docker Run](https://docs.docker.com/engine/reference/commandline/run/) for more information.
The agent supports the following environment variables:
- `AGENT_TOKEN`: **required**. unique token generated by the server, used by the agent to authenticate itself when registering on the server,
- `SERVER_URL`: **required**. Location of the OctoPerf server,
- `octoperf.auto-update`: **optional**. Default to `true`. Whenever the agent should upgrade itself when a new version is available.
- `-e SERVER_PRIVATE_ROOT_CERTIFICATE="true"`: **optional.** Required to ignore errors caused by SSL certificates signed by self / private root certificates used by the machine hosting the agent.
The agent has the following volume mappings:
- `/var/run/docker.sock:/var/run/docker.sock`: **optional**. Gives access to the docker service on the host machine. It's used to start / stop JMeter containers on the host.
!!! warning
The agent must be upgraded manually when the `docker.sock` volume mapping is not present.
### Kubernetes agent
On a Kubernetes cluster, the agent can run as a pod, installed with the [`kubernetes-agent` Helm chart](https://github.com/OctoPerf/helm-charts/tree/master/kubernetes-agent). It then launches each load generator as a pod of the cluster, instead of a Docker container. An agent only starts load generators on its own node: install one agent per node that runs load generators.
The chart needs the token of the agent and the URL of the OctoPerf server. Both come from the command of the [Add Agent](#start-an-agent) dialog:
1. In the *Installation* step, click **Copy Command**: the token is masked on the screen,
2. Take the value of `AGENT_TOKEN` and the value of `SERVER_URL` from the copied command,
3. Install the chart with them:
```bash
helm install kubernetes-agent octoperf/kubernetes-agent --namespace octoperf --create-namespace \
--set token='' \
--set serverUrl=''
```
As with a Docker agent, take a new command for each agent you install: the token holds the name of the agent. The [Kubernetes tutorial](https://api.octoperf.com/doc/on-premise-infra/tutorials/kubernetes-deployment/) installs OctoPerf and a Kubernetes agent on a local minikube cluster.
The settings of these load generator pods are given in the values of the chart, under `config`: memory and CPU, tolerations, image pull secrets and, from `18.0.0`, the security context of the pod and of its container. The [README of the chart](https://github.com/OctoPerf/helm-charts/tree/master/kubernetes-agent#readme) lists them all.
The security contexts matter when the cluster enforces a security policy. For example, a cluster refusing privilege escalation rejects the load generator pods, and the agent logs `Privilege escalation container is not allowed`. Declare it in the values of the chart:
```yaml
config:
octoperf:
container:
security-context:
allow-privilege-escalation: false
```
!!! info
Both security contexts are empty by default: the load generator pods are created as before `18.0.0`. Give only the fields your policy requires, the README of the chart has an example for the restricted Pod Security Standard.
## Agent Containers
By **clicking on the inspect button** on an agent, the *Inspect Agent* panel appears. It provides a **Containers Table** that displays the containers currently running on this agent machine.
When selecting a container, **container logs are displayed**. Those information are mostly provided for debugging purpose.
You can also hover the container to and click **Terminate** to terminate it. If you terminate the `docker-agent` container, you will [stop this Agent](#stop-an-agent).
!!! warning
Terminating a container can have an impact on tests that are running. Make sure to only terminate containers that are idle or stuck.
## Stop an agent
To stop an agent you can use the [Inspect Agent](#agent-containers) screen to **Terminate** your agent through our UI.
Or from the Linux host:
- Connect on the machine running the container through a Terminal (Linux),
- Find the container within the list of running containers by running `docker ps -a | grep docker-agent` command,
- Copy the `containerId` and run the command `docker stop `.
The agent should be marked as **DOWN** within a few minutes:
- Agents down for a while (more than 1 Hour) are automatically removed, this delay aims to avoid agents flashing in and out of existence upon any network issue,
- Restarting a stopped agent will make it reappear on the server again in `UP` state, as long as its docker container was not entirely removed.
## Troubleshooting
The following steps require that you open a terminal and enter commands on your linux machine.
### My agent does not appear in the agents list
First, make sure you have successfully installed **Docker** and launched the agent command on the machine.
#### Display the Agent Logs
First, display the container status and its `containerId`:
`docker ps -a | grep docker-agent`
You can then proceed to check the agent logs for error messages:
`docker logs -f `
The agent logs will tell you if it's a connectivity issue or something else.
Typically, messages like :
- No route to host,
- Failed to connect,
- Unknown host.
All indicate a network issue when communicating with the OctoPerf server. In that case try:
- Disabling any firewall,
- Revert any change you've made to the iptables,
- Use your server IP instead of DNS name in the agent command line.
Or move on to the next section to further debug the issue.
#### Debug connectivity issue
You should be able to retrieve the list of Workspaces from the machine hosting the OctoPerf docker agent.
Calling the REST API ensures that the Agent is able to communicate with the backend.
!!! info
If you have an On Premise server, replace api.octoperf.com with its address.
Run the following command:
```bash
curl --header "Authorization: Bearer API_KEY" -v https://api.octoperf.com/workspaces/member-of
```
You API key can be found in the [OctoPerf application profile section](https://api.octoperf.com/doc/account/profile/#apikey).
You may also run this command from the inside of the agent Docker container:
```bash
docker exec -it /bin/bash
```
Once connected to the container shell:
```bash
apt-get -qq update
apt-get -qq -y install curl
curl --header "Authorization: Bearer API_KEY" -v https://api.octoperf.com/workspaces/member-of
```
If you can't connect to the API then this is proof that you have a network connectivity issue from the agent to your server. Make sure the network route is enabled and/or whitelisted.
#### Remove and recreate the agent
Delete the existing agent by running the following commands:
```bash
docker ps -a | grep docker-agent
docker rm -f
```
Then proceed to [reinstall the agent](#start-an-agent) using a newly generated command line. If the token used in your installation command line is outdated, this should fix it.
!!! info
On Enterprise Edition, the agents installed before `18.0.0` must be recreated this way after the upgrade: their token is refused by an `18.0.0` server. See the [17.1.x to 18.0.0 migration guide](https://api.octoperf.com/doc/on-premise-infra/migration/17.1.x-to-18.0.0/#your-on-premise-agents-must-be-relaunched-with-a-new-token).
!!! warning
If the agent machine is used for other docker containers, it's better to locate the OctoPerf images and delete them specifically.
#### Switching Logs to Debug
In case the default logs do not give enough information you may start a new agent with this environment variable added: `-e JAVA_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=debug"`.
This will activate DEBUG logs and may help you find out what is going on.
#### Increase heap size
If you messages like this in the log, you might need to increase the docker-agent Heap memory:
```
java.lang.OutOfMemoryError: Java heap space
```
The default memory size of a docker-agent is 256 Mb, to increase it to 1 GB add a environment variable right before the agent token in the installation command line:
``` bash
sudo docker run -d --restart=unless-stopped --name XXXX -v /var/run/docker.sock:/var/run/docker.sock -e JAVA_OPTS="-Xmx1g" -e AGENT_TOKEN='XXXX' -e SERVER_URL='XXXX' octoperf/docker-agent:XX.XX.X
```
!!! warning
This is only required for large monitoring since that's the only operation that can require large amounts of memory on the agent. If you see this message in a JMeter container logs, or during a test, consider changing the [memory configuration](https://api.octoperf.com/doc/on-premise-agent/providers/#memory-usage) of the provider instead.
### I'm using podman instead of docker
In that case there's a few changes you have to make.
#### Edit the command line
Make it look like this:
!!! Example "Command line"
=== "Podman"
``` bash
sudo podman run -d --restart=unless-stopped --name XXXX -v /var/run/podman/podman.sock:/var/run/docker.sock -e AGENT_TOKEN='XXXX' -e SERVER_URL='XXXX' docker.io/octoperf/docker-agent:XX.XX.X
```
=== "Docker"
``` bash
sudo docker run -d --restart=unless-stopped --name XXXX -v /var/run/docker.sock:/var/run/docker.sock -e AGENT_TOKEN='XXXX' -e SERVER_URL='XXXX' octoperf/docker-agent:XX.XX.X
```
!!! warning
Make sure to replace all the XXXX with values from the docker command line you got from OctoPerf's UI.
#### Custom registry
You will also most likely have to use docker.io as a custom registry in the provider:
