Source: https://api.octoperf.com/doc/on-premise-infra/tutorials/rancher-v2xx-deployment/

# Deploy on Rancher v2xx

## On-Premise InfraDeploy On-Premise Infra on Rancher v2.x.x

![Rancher](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-logo.png)

This tutorial explains how to setup a highly available 3-nodes cluster to run OctoPerf On-Premise Infra. As a result, we use [Rancher](https://rancher.com/) to provision a [Kubernetes](https://kubernetes.io) cluster.

We're going to use cloud instance on [Digital Ocean](https://www.digitalocean.com) for both the rancher server and the rancher worker nodes.

### Prerequisites

This tutorial requires:

- A [Digital Ocean](https://www.digitalocean.com) account,
- Basic knowledge around managing [Ubuntu linux](https://ubuntu.com) via SSH,
- And basic knowledge of how [Docker](https://docker.com) works.

### Topology

The infrastructure built in this tutorial works as following:

- **rancher-server**: hosts the rancher server only,
- **rancher-node-1**: acts as etcd, control plane and worker,
- **rancher-node-2**: acts as etcd, control plane and worker,
- **rancher-node-3**: acts as etcd, control plane and worker.

All the pods are running on **rancher-node-X**. Elasticsearch is deployed on all 3 nodes. OctoPerf On-Premise Infra Backend is deployed on all 3 nodes.

### Rancher Server

First, we need to spin-up the [Rancher](https://rancher.com) server. The server is going to run on a dedicated instance:

- Spin up a Digital Ocean instance by selecting `Marketplace` `Docker xxx on Ubuntu` type of instance. Name it `rancher-server` to easily recognize it. A server with `2 CPUs` and `4GB RAM` is enough for testing purpose

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/0-rancher-server-instance.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/0-rancher-server-instance.png)

- SSH into the instance once it up and running, and disable the firewall:

```bash
root@rancher-server:~# sudo ufw disable
Firewall stopped and disabled on system startup
```

- Start the rancher `v2.x.x` server:

```bash
root@rancher-server:~# sudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher
```

- It should be running shortly:

```bash
root@rancher-server:~# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                                      NAMES
b37abe78bfd3        rancher/rancher     "entrypoint.sh"     11 seconds ago      Up 9 seconds        0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   stoic_kalam
```

- Once Rancher is running, we need to configure it to work properly. Browse to `https://RANCHER_SERVER_IP/`, where `RANCHER_SERVER_IP` is the public ip of the instance,

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/1-open-rancher-web-ui.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/1-open-rancher-web-ui.png)

**Firefox** will complain about the invalid certificate. Proceed by pressing `Accept the Risk and Continue`.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/2-rancher-setup-password.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/2-rancher-setup-password.png)

Configure the admin password and click on `Continue`.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/3-rancher-server-url.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/3-rancher-server-url.png)

Set the rancher server url to the public IP address of the instance hosting the server. `165.22.24.244` in my case is the public IP which has been automatically assigned to the `rancher-server` Digital Ocean instance.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/3-rancher-add-cluster.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/3-rancher-add-cluster.png)

Rancher server is up and properly configured. Let's now setup the worker nodes.

### Rancher Nodes

3 rancher nodes are going to host the docker containers to run OctoPerf On-Premise Infra. Launch 3 nodes (with Docker preinstalled from Marketplace) and name them `rancher-node-1`, `rancher-node-2` and `rancher-node-3` respectively.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/4-digital-ocean-rancher-nodes.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/4-digital-ocean-rancher-nodes.png)

`2 CPUs` with `4GB RAM` standard instances are enough for testing purpose. Consider using at least `6 CPUs` with `16GB RAM` per node in a production setup.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/5-digital-ocean-rancher-nodes.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/5-digital-ocean-rancher-nodes.png)

Once the nodes are up and running:

- SSH into each node,
- Increase `vm.max_map_count` permanently:

```bash
root@rancher-node-1:~# sudo nano /etc/sysctl.conf
```

- Add the following line to the end of the `sysctl.conf` file:

```bash
vm.max_map_count=262144
```

- Save and quit, then update `systcl`:

```bash
root@rancher-node-1:~# sudo sysctl -p
vm.max_map_count = 262144
```

- Disable the firewall: (otherwise networking between nodes will fail)

```bash
root@rancher-node-1:~# sudo ufw disable
Firewall stopped and disabled on system startup
```

- Install basic software for proper security and clock synchronization:

```bash
root@rancher-node-1:~# sudo apt install -y fail2ban ntp
```

Repeat those operations for `rancher-node-1`, `rancher-node-2` and `rancher-node-3`. We now have 3 nodes with Docker installed and preconfigured to easily run [Elasticsearch](https://elastic.co).

### Rancher Cluster Configuration

Let's create the Kubernetes cluster using **Rancher** via the Web UI.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/5-rancher-cluster-custom.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/5-rancher-cluster-custom.png)
*Custom Kubernetes Cluster*

Click on `Create Cluster` button, then:

- Select `Custom` flavor,
- In cluster name, enter `octoperf`,
- In cluster options, select Kubernetes version `v1.14.1-rancher-1-2`,
- In `Network Provider`, leave as `Canal`,
- In `Cloud Provider` , lease as `None`.

Then, click on `Next` button. On the next screen:

- Select Node options `etcd`, `Control Plane` and `Worker`. No other option is required (like node public ip or node name) as those are detected automatically,
- Copy the command-line provided by Rancher and run it on each node:

```bash
root@rancher-node-1:~# sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.2.4 --server https://165.22.24.244 --token dh9whpwjgt4ppm92r7xpjfn4s7wrhrgbk8xz29xphg6zmbwgk22l95 --ca-checksum bddf7e0dd7057ac3f36d751fb27e745837dc1b768447677500ba0b675118b0a7 --etcd --controlplane --worker
Unable to find image 'rancher/rancher-agent:v2.2.4' locally
v2.2.4: Pulling from rancher/rancher-agent
...
```

It may take several minutes until the nodes join the cluster. Rancher automatically setups Kubernetes along with networking connectivity on all nodes.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/10-rancher-cluster-provisioning.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/10-rancher-cluster-provisioning.png)

The cluster of 3 nodes is now ready! The next phase consists of deploying each [workload](https://kubernetes.io/fr/docs/concepts/workloads/).

### Elasticsearch Workload

#### Topology

Elasticsearch will be deployed using 3 workloads:

- `es-1`: elasticsearch master + data node running on `rancher-node-1`,
- `es-2`: elasticsearch master + data node running on `rancher-node-2`,
- and `es-3`: elasticsearch master + data node running on `rancher-node-3`.

Each elasticsearch container will use a persistent volume mapped to the node local directory `/root/elasticsearch`.

Also, each elasticsearch container will have its own name along with the shared configuration stored in a [config map](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/).

#### Persistent Volume

First, let's create the `elasticsearch` directory on each rancher node:

```bash
root@rancher-node-1:~# pwd
/root
root@rancher-node-1:~# mkdir elasticsearch
root@rancher-node-1:~# chmod 777 elasticsearch/
root@rancher-node-1:~#
```

Now we have a `/root/elasticsearch` writeable directory on each rancher node. We need to configure this directory as a volume within Rancher.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/12-elasticsearch-data-volume.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/12-elasticsearch-data-volume.png)

The Elasticsearch persistent volume is designed to house the elasticsearch data files:

- In Rancher UI, click on `octoperf` cluster, then select `Storage > Persistent Volumes` menu,
- Select `Local Node Path` volume plugin,
- Set at least `20GiB` of space,
- In plugin configuration, set `/root/elasticsearch` path on the node, and the path on the node myst be `Anything: do not check the target path`,
- Click on `Save` to create the persistent volume.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/13-elasticsearch-data-volume.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/13-elasticsearch-data-volume.png)

Next, we need to add this volume to the `Default` project:

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/14-rancher-cluster-add-volume.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/14-rancher-cluster-add-volume.png)

- Click on `octoperf > Default` project in top left menu in Rancher UI,
- Click on `Workloads` in top menu,
- Then select `Volumes` in sub-menu,
- Click on `Add Volume`,
- Enter `elasticsearch-data` as name and select `elasticsearch-data` persistent volume,
- Click on `Create`.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/15-rancher-cluster-volume-claim.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/15-rancher-cluster-volume-claim.png)

The data volume is now ready to be used by the **Elasticsearch** workload we're going to configure.

#### Config Map

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/14-elasticsearch-config-map.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/14-elasticsearch-config-map.png)

Elasticsearch config map is containing the configuration settings shared by all the elasticsearch nodes:

- `ES_JAVA_OPTS`: `-Xms512m -Xmx512m`,
- `cluster.initial_master_nodes`: `es-1, es-2, es-3`,
- `discovery.seed_hosts`: `es-1, es-2, es-3`,
- `network.host`: `0.0.0.0`.

That way, we will have a cluster of 3 elasticsearch nodes, each master-eligible and storing data. Now, let's launch the workloads.

#### Workloads

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/16-es-1-workload.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/16-es-1-workload.png)

Each Elasticsearch container is launched in a separate workload because each has a unique `node.name`. Repeat the following procedure for `es-1`, `es-2` and `es-3`:

- Select `Default` project inside `octoperf` cluster,
- Click on `Workloads`, then click on `Create`,
- **Name**: `es-X` (where X is 1, 2 then 3),
- **Workload type**: `Stateful Set of 1 pod`,
- **Docker image**: `docker.elastic.co/elasticsearch/elasticsearch-oss:7.1.1`, (version may vary depending on OctoPerf On-Premise Infra version being launched)
- **Environment variables**: add `node.name` with value `es-X`, (replace `X` by relevant number, here it's `es-1`). Click on `Add from source` then select `Config Map`, `elasticsearch-config` config map,
- **Node Scheduling**: select `rancher-node-x` (where X matches the `es-X` value; Example: `rancher-node-1`),
- **Volumes**: click on `Add Volume`, then select `Persistent Volume Claim` and choose `elasticsearch-data` volume. Set `/usr/share/elasticsearch/data` as `Mount Point`,
- Click on `Launch` to deploy workload.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/17-elasticsearch-cluster-operational.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/17-elasticsearch-cluster-operational.png)
*All 3 nodes are up and running*

Once `es-1`, `es-2` and `es-3` workload are deployed, the elasticsearch cluster should stabilise and become operational. You can check this by `Executing Shell` inside one of the workload pods:

```bash
[root@es-1-0 elasticsearch]# curl localhost:9200/_cat/nodes
10.42.1.3  13 91 5 0.15 0.11 0.16 mdi - es-3
10.42.2.4  11 93 6 0.23 0.23 0.30 mdi - es-2
10.42.0.11 22 85 6 0.05 0.21 0.35 mdi * es-1
[root@es-1-0 elasticsearch]# 
```
#### Service Discovery

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/18-elasticsearch-service-discovery.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/18-elasticsearch-service-discovery.png)

The backend is connecting to Elasticsearch through its hostname. The thing is, we don't want to hardcode any of the `es-X` nodes hostname: it may fail to connect if one of the nodes is failing. The solution is to define a **Service Discovery** entry:

- Select `Default` project inside `octoperf` cluster,
- Click on `Service Discovery`, then click on `Add Record`,
- **Name**: `elasticsearch`,
- **Resolves To**: `One or more workloads`,
- **Namespace**: `default`,
- **Target Workloads**: add `es-1`, `es-2` and `es-3`.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/19-rancher-service-discovery.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/19-rancher-service-discovery.png)

That means: a DNS resolution of hostname `elasticsearch` is the IPs of all three nodes.

### On-Premise Infra

The On-Premise Infra backend server connects to Elasticsearch database and exposes an API which can be consumed by OctoPerf web UI.

This tutorial supposes you have already setup an [Amazon S3 bucket for resources storage](https://api.octoperf.com/doc/on-premise-infra/configuration/#resources-storage). The backend is going to run on all nodes in High availability so the S3 bucket is needed to share the files between all the backends.

#### Config Map

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/20-enterprise-edition-config-map.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/20-enterprise-edition-config-map.png)

The config map is containing the configuration shared among all backend containers:

- **clustering.driver**: `hazelcast`,
- **clustering.members**: `enterprise-edition`, (workload DNS entry)
- **clustering.quorum**: `2`, as we have a cluster of 3 nodes,
- **storage.driver**: `s3`, because resources (like project files) are stored on s3,
- **storage.s3.region**: `eu-west-1` in this case (because our S3 bucket is in Ireland),
- **storage.s3.bucket**: `YOUR_S3_BUCKET` (we used `junit.octoperf.com` here),
- **server.hostname**: `IP or DNS entry` (the IP of one of the 3 nodes in our case; the load balancer is going to run on all the nodes)
- **elasticsearch.hostname**: `elasticsearch` (the service discovery entry we previously created)
- **server.public.port**: `80`, because we're going to serve the API in plain text (no SSL in this setup).

For more information regarding these settings, please refer to [Configuration](https://api.octoperf.com/doc/on-premise-infra/configuration/).

#### Secret

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/21-enterprise-edition-secret.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/21-enterprise-edition-secret.png)

The `storage.s3.access-key` and `storage.s3.secret-key` aws keys are provided by a **Secret**:

- Click on `Resources > Secrets`,
- Create a new secret named `enterprise-edition-secret`,
- Add `storage.s3.access-key` and `storage.s3.secret-key` which point to an IAM user which has only the permission to access the given s3 bucket. Here is an example IAM policy which gives access to `junit.octoperf.com` s3 bucket:

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1427454857000",
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:Put*",
                "s3:Delete*"
            ],
            "Resource": [
                "arn:aws:s3:::junit.octoperf.com",
                "arn:aws:s3:::junit.octoperf.com/*"
            ]
        }
    ]
}
```

#### Workload

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/22-enterprise-edition-workload.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/22-enterprise-edition-workload.png)

Let's now deploy the workload which runs the server on all the nodes:

- Click on `Workloads` then `Deploy`,
- **Name**: `enterprise-edition`,
- **Workload type**: `Run one pod on each node`,
- **Docker image**: `octoperf/enterprise-edition:11.0.0` (replace `11.0.0` by latest version available),
- **Namespace**: `default`,
- **Environment variables**: import both `enterprise-edition-config` config map and `enterprise-edition-secret` secret,
- Click on `Launch`.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/23-enterprise-edition-workload-pods.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/23-enterprise-edition-workload-pods.png)

You should see 3 pods, each running on a separate host. All those pods are connected to each other by using [Hazelcast](https://www.hazelcast.org) to form a cluster. 

Those 3 containers are acting in a master / slave configuration:

- One node is elected as master and takes care of scheduling the load tests,
- Should the master fail, another node takes over,
- Should there be a [network partition](https://en.wikipedia.org/wiki/Network_partition), the partition having at least `2` nodes (as stated by the quorum setting) stays operational.

### On-Premise UI

The frontend is a tiny Nginx server running inside a container to serve the statically generated JS and html files.

#### Workload

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/24-enterprise-ui-workload.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/24-enterprise-ui-workload.png)

Let's deploy the frontend container. In this case, a single container is plenty enough to serve the frontend. The workload is pretty simple:

- Click on `Workloads` then `Deploy`,
- **Name**: `enterprise-ui`,
- **Docker Image**: `octoperf/enterprise-ui:11.0.0`, (adjust version to the latest available)
- **Workload Type**: scalable deployment of 1 pod.

No other settings are required. the container can run on any node. Kubernetes will take care of rescheduling it to another node if it fails.

### On-Premise Documentation

The frontend is a tiny Nginx server running inside a container to serve the statically generated documentation.

#### Workload

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/24-enterprise-documentation-workload.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/24-enterprise-documentation-workload.png)

Let's deploy the frontend container. In this case, a single container is plenty enough to serve the frontend. The workload is pretty simple:

- **Name**: `enterprise-documentation`,
- **Docker Image**: `octoperf/enterprise-documentation:11.0.0`, (adjust version to the latest available)
- **Workload Type**: scalable deployment of 1 pod.

No other settings are required. the container can run on any node. Kubernetes will take care of rescheduling it to another node if it fails.

### Load Balancing

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/25-create-ingress.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/25-create-ingress.png)

The final step is to expose the frontend, documentation and backend workloads through a **ingress load balancer**:

- Click on `Load Balancing` then `Add Ingress`,
- **Name**: `load-balancer`,
- **Rules**: set to use generated `nip.io` hostname.


[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/25-ingress-rules.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/25-ingress-rules.png)

Then define 3 workload rules:

- `/doc` which points to `enterprise-documentation` on port `80`,
- `/app` which points to `enterprise-ui` on port `80`,
- `/` which points to `enterprise-edition` on port `8090`.

Click on `Save` to create the ingress load balancer.

### Accessing the Application

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/26-octoperf-running.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v2xx/26-octoperf-running.png)

Now you can browse to `http://octoperf.default.XXX.XXX.XXX.XXX.nip.io` (click on the link from the ingress view to open it automatically from Rancher UI).

### Create an Account

OctoPerf EE comes completely empty. You need to create an account and register a load generator (computer used to generate the load) to be able to run load tests:

- [Signup and create a new account](https://api.octoperf.com/doc/account/#sign-up),
- Create an [On-Premise Provider](https://api.octoperf.com/doc/on-premise-agent/): providers are groups of load generators,
- And [register an On-Premise Agent](https://api.octoperf.com/doc/on-premise-agent/provider-type/on-premise/) within the previously created *On-Premise Provider*.

Congratulations! The setup is now complete.