Source: https://api.octoperf.com/doc/on-premise-infra/tutorials/rancher-v16x-deployment/

# Deploy On-Premise Infra on Rancher v1.6.x

![Rancher](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-logo.png)

This tutorial explains how to setup a highly available 3-nodes cluster to run OctoPerf On-Premise Infra. As a result, we use [Rancher v1.6.x](https://rancher.com/) to orchestrate the cluster.

We're going to use cloud instance on [Digital Ocean](https://www.digitalocean.com) for both the rancher server and the rancher worker nodes.

`Rancher v1.6.x` is still maintained but support will be discontinued end 2019. `Rancher v1.6.x` is still a very good docker cluster orchestration tool because it's very **simple to use**.

## Prerequisites

This tutorial requires:

- A [Digital Ocean](https://www.digitalocean.com) account,
- Basic knowledge around managing [Ubuntu linux](https://ubuntu.com) via SSH,
- And basic knowledge of how [Docker](https://docker.com) works.

## Topology

The infrastructure built in this tutorial works as following:

- **rancher-server**: hosts the rancher server only,
- **rancher-node-1**: acts as rancher node,
- **rancher-node-2**: acts as rancher node,
- **rancher-node-3**: acts as rancher node.

All the containers are running on **rancher-node-X**. Elasticsearch is deployed on all 3 nodes. OctoPerf On-Premise Infra Backend is deployed on all 3 nodes.

## Rancher Server

First, we need to spin-up the [Rancher](https://rancher.com) server. The server is going to run on a dedicated instance:

- Spin up a Digital Ocean instance by selecting `Marketplace` `Docker xxx on Ubuntu` type of instance. Name it `rancher-server` to easily recognise it. A server with `2 CPUs` and `4GB RAM` is enough for testing purpose.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/0-rancher-cloud-instances.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/0-rancher-cloud-instances.png)

- SSH into the instance once it up and running, and disable the firewall:

```bash
root@rancher-server:~# sudo ufw disable
Firewall stopped and disabled on system startup
```

- Start the rancher `v1.6.x` server:

```bash
mkdir rancher-v1.6.x
docker run -d -v /root/rancher-v1.6.x:/var/lib/mysql --restart=unless-stopped -p 8080:8080 rancher/server:stable
```

- Open `http://RANCHER_IP:8080` in your web browser, then go to `ADMIN > Access Control`. Enable `Local Authentication` and provide a username and a password to secure your rancher server.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/1-rancher-access-control.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/1-rancher-access-control.png)

- Verify `Host Registration URL` is pointing to the public IP in `ADMIN > Settings`. Click on `Save` to apply settings,

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/2-rancher-registration-url.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/2-rancher-registration-url.png)

Rancher server is now ready to use.

## Rancher Nodes

### Cloud Configuration

3 rancher nodes are going to host the docker containers to run OctoPerf On-Premise Infra. Launch 3 nodes (with Docker preinstalled from Marketplace) and name them `rancher-node-1`, `rancher-node-2` and `rancher-node-3` respectively.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/4-digital-ocean-rancher-nodes.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/4-digital-ocean-rancher-nodes.png)

`2 CPUs` with `4GB RAM` standard instances are enough for testing purpose. Consider using at least `6 CPUs` with `16GB RAM` per node in a production setup.


### OS Configuration

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/5-digital-ocean-rancher-nodes.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/5-digital-ocean-rancher-nodes.png)

Once the nodes are up and running:

- SSH into each node,
- Increase `vm.max_map_count` permanently:

```bash
root@rancher-node-1:~# sudo nano /etc/sysctl.conf
```

- Add the following line to the end of the `sysctl.conf` file:

```bash
vm.max_map_count=262144
```

- Save and quit, then update `systcl`:

```bash
root@rancher-node-1:~# sudo sysctl -p
vm.max_map_count = 262144
```

- Disable the firewall: (otherwise networking between nodes will fail)

```bash
root@rancher-node-1:~# sudo ufw disable
Firewall stopped and disabled on system startup
```

- Install basic software for proper security and clock synchronization:

```bash
root@rancher-node-1:~# sudo apt install -y fail2ban ntp
```

Repeat those operations for `rancher-node-1`, `rancher-node-2` and `rancher-node-3`. We now have 3 nodes with Docker installed and preconfigured to easily run [Elasticsearch](https://elastic.co).

## Host Registration

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/6-rancher-add-host.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/6-rancher-add-host.png)

The next step is to **add the hosts** to our rancher server:

- Open Rancher Web UI,
- Click on `INFRASTRUCTURE > Hosts`, 
- Click on `Add Host` button,
- Select `Custom` host type,
- Copy and execute the command-line provided by Rancher on each of the nodes through SSH.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/7-rancher-hosts-ui.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/7-rancher-hosts-ui.png)

After a few minutes, all hosts should be registered and visible in Rancher Web UI. All system services should be shown as green dots. At this stage, we have a working rancher server connected to the 3 rancher hosts `rancher-node-1`, `rancher-node-2` and `rancher-node-3`.

Once all hosts are registered:

- Edit each host through Rancher Web UI,
- Add a label with name `name` and value `rancher-node-X` (where `X` is the number of the node).

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/7-rancher-host-labels.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/7-rancher-host-labels.png)
*Each host has the label `name=rancher-node-X`*

Those labels are subsequently used in stacks below to assign containers to specific hosts.

## Elasticsearch Stack

### Topology

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/10-elasticsearch-stack-graph.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/10-elasticsearch-stack-graph.png)
*Elasticsearch Stack Graph*

Each Elasticsearch container will:

- running as 3 separate services, each running a on separate host,
- use a volume mapping from `/usr/share/elasticsearch/data` (inside the container) to `/root/elasticsearch` (on the host),
- be running all roles including `master`, `ingest` and `data`,
- be named `es-1`, `es-2` and `es-3` and respectively run on `rancher-node-1`, `rancher-node-2` and `rancher-node-3`.

### Local Directory

First, let's create the `elasticsearch` directory on each rancher node:

```bash
root@rancher-node-1:~# pwd
/root
root@rancher-node-1:~# mkdir elasticsearch
root@rancher-node-1:~# chmod 777 elasticsearch/
root@rancher-node-1:~#
```

Now we have a `/root/elasticsearch` writeable directory on each rancher node.

### Stack

Let's now create the Elasticsearch stack. Its purpose is to contain all Elasticsearch related services:

- Open Rancher Web UI,
- Click on `STACKS`,
- Click on `Add Stack`,
- In name, enter `elasticsearch`,
- Copy and paste the following content into the `docker-compose.yml` field:

```yml
version: '2'
services:
  elastisearch:
    image: rancher/dns-service
    links:
    - es-3:es-3
    - es-2:es-2
    - es-1:es-1
  es-3:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.1.1
    environment:
      network.host: 0.0.0.0
      ES_JAVA_OPTS: -Xms512m -Xmx512m
      cluster.initial_master_nodes: es-1, es-2, es-3
      discovery.seed_hosts: es-1, es-2, es-3
      node.name: es-3
    stdin_open: true
    volumes:
    - /root/elasticsearch:/usr/share/elasticsearch/data
    tty: true
    labels:
      io.rancher.container.pull_image: always
      io.rancher.scheduler.affinity:host_label: name=rancher-node-3
  es-2:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.1.1
    environment:
      network.host: 0.0.0.0
      ES_JAVA_OPTS: -Xms512m -Xmx512m
      cluster.initial_master_nodes: es-1, es-2, es-3
      discovery.seed_hosts: es-1, es-2, es-3
      node.name: es-2
    stdin_open: true
    volumes:
    - /root/elasticsearch:/usr/share/elasticsearch/data
    tty: true
    labels:
      io.rancher.container.pull_image: always
      io.rancher.scheduler.affinity:host_label: name=rancher-node-2
  es-1:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.1.1
    environment:
      network.host: 0.0.0.0
      ES_JAVA_OPTS: -Xms512m -Xmx512m
      cluster.initial_master_nodes: es-1, es-2, es-3
      discovery.seed_hosts: es-1, es-2, es-3
      node.name: es-1
    stdin_open: true
    volumes:
    - /root/elasticsearch:/usr/share/elasticsearch/data
    tty: true
    labels:
      io.rancher.container.pull_image: always
      io.rancher.scheduler.affinity:host_label: name=rancher-node-1
```

!!! info
    Please adjust Elasticsearch version to the latest one being used by the On-Premise Infra. As of writing, OctoPerf uses Elasticsearch `7.1.1`.

- Copy and paste the following content into the `rancher-compose.yml` field:

```yml
version: '2'
services:
  elastisearch:
    start_on_create: true
  es-3:
    scale: 1
    start_on_create: true
  es-2:
    scale: 1
    start_on_create: true
  es-1:
    scale: 1
    start_on_create: true
```

- Upload those files when creating the stack.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/9-elasticsearch-services-running.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/9-elasticsearch-services-running.png)

This will automatically create all the Elasticsearch services and run them on each node.

The Elasticsearch stack is composed of 4 services:

- `es-1`: Elasticsearch node running on `rancher-node-1`,
- `es-2`: Elasticsearch node running on `rancher-node-2`,
- `es-3`: Elasticsearch node running on `rancher-node-3`,
- `elasticsearch`: service alias pointing to all `es-X` services. (equivalent of a DNS entry)

## OctoPerf Stack

### Topology

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/11-octoperf-stack-graph.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/11-octoperf-stack-graph.png)
*OctoPerf Stack Graph*

OctoPerf stack contains the following services:

- **enterprise-edition**: On-Premise backend server which connects to the database and serves the Rest API used by the web ui,
- **enterprise-ui**: On-Premise frontend server which is a small Nginx web server serving the static HTML / CSS / Javascript files,
- **enterprise-documentation**: On-Premise documentation server which is a small Nginx web server serving the static HTML / CSS / Javascript files,
- **elasticsearch/elasticsearch**: service link to the `elasticsearch` service alias in `elasticsearch` stack previously created,
- **load-balancer**: [NGinx](https://www.nginx.org/) front load balancer managed by Rancher which exposes the On Premise Infra services on all rancher-nodes on HTTP port `80` (or `443` for the HTTPS version).

This tutorial supposes you have already setup an [Amazon S3 bucket for resources storage](https://api.octoperf.com/doc/on-premise-infra/configuration/#resources-storage). The backend is going to run on all nodes in [High availability](https://api.octoperf.com/doc/on-premise-infra/configuration/#high-availability) so the S3 bucket is needed to share the files between all the backends.

### Stack

Let's now create the OctoPerf stack:

- Open Rancher Web UI,
- Click on `STACKS`,
- Click on `Add Stack`,
- In name, enter `octoperf`,
- Copy and paste the following content into the `docker-compose.yml` field:

```yml
version: '2'
services:
  enterprise-documentation:
    image: octoperf/enterprise-documentation:11.0.0
    stdin_open: true
    tty: true
    labels:
      io.rancher.container.pull_image: always
      io.rancher.scheduler.global: 'true'
  enterprise-ui:
    image: octoperf/enterprise-ui:11.0.0
    stdin_open: true
    tty: true
    labels:
      io.rancher.container.pull_image: always
      io.rancher.scheduler.global: 'true'
  enterprise-edition:
    image: octoperf/enterprise-edition:11.0.0
    environment:
      server.hostname: ANY_RANCHER_NODE_PUBLIC_IP
      elasticsearch.hostname: elasticsearch
      server.public.port: '80'
      clustering.driver: hazelcast
      clustering.members: enterprise-edition
      clustering.quorum: '2'
      storage.driver: s3
      storage.s3.region: eu-west-1
      storage.s3.bucket: junit.octoperf.com
      storage.s3.access-key: S3_ACCESS_KEY
      storage.s3.secret-key: S3_SECRET_KEY
    stdin_open: true
    external_links:
    - elasticsearch/elastisearch:elasticsearch
    tty: true
    labels:
      io.rancher.container.pull_image: always
      io.rancher.scheduler.global: 'true'
  load-balancer:
    image: rancher/lb-service-haproxy:v0.9.11
    ports:
    - 80:80/tcp
    labels:
      io.rancher.container.agent.role: environmentAdmin,agent
      io.rancher.container.agent_service.drain_provider: 'true'
      io.rancher.container.create_agent: 'true'
      io.rancher.scheduler.global: 'true'
```

Replace `ANY_RANCHER_NODE_PUBLIC_IP`, `S3_ACCESS_KEY` and `S3_SECRET_KEY` by your own configuration. The backend uses S3 to store resource files (such as JMeter logs, csv files etc). 

`storage.s3.access-key` and `storage.s3.secret-key` point to an IAM user which has only the permission to access the given s3 bucket. Here is an example IAM policy which gives access to `junit.octoperf.com` s3 bucket:

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1427454857000",
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:Put*",
                "s3:Delete*"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}
```

Replace `YOUR_BUCKET_NAME` by the name of your S3 bucket when creating this IAM using AWS Console Web UI.

- Copy and paste the following content into the `rancher-compose.yml` field:

```yml
version: '2'
services:
  enterprise-documentation:
    start_on_create: true
  enterprise-ui:
    start_on_create: true
  enterprise-edition:
    start_on_create: true
  load-balancer:
    start_on_create: true
    lb_config:
      certs: []
      config: |-
        defaults
        compression algo gzip
        compression type application/json text/html text/plain
      port_rules:
      - path: /app
        priority: 1
        protocol: http
        service: enterprise-ui
        source_port: 80
        target_port: 80
      - path: /doc
        priority: 2
        protocol: http
        service: enterprise-documentation
        source_port: 80
        target_port: 80
      - path: /
        priority: 3
        protocol: http
        service: enterprise-edition
        source_port: 80
        target_port: 8090
    health_check:
      healthy_threshold: 2
      response_timeout: 2000
      port: 42
      unhealthy_threshold: 3
      initializing_timeout: 60000
      interval: 2000
      reinitializing_timeout: 60000
```

- Click on `Create` button to create the stack.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/12-octoperf-stack.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/12-octoperf-stack.png)

The stack should be operational after a few minutes. Once all services are up and running (shown as green dots in Rancher web UI), you can browse to `http://ANY_RANCHER_NODE_IP` to access to OctoPerf On-Premise Infra.

[![](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/13-rancher-nodes.png)](https://api.octoperf.com/doc/on-premise-infra/tutorials/img/rancher-v16x/13-rancher-nodes.png)
*Elasticsearch and OctoPerf stack containers spread accross the 3 nodes cluster*

## Create an Account

OctoPerf EE comes completely empty. You need to create an account and register a load generator (computer used to generate the load) to be able to run load tests:

- [Signup and create a new account](https://api.octoperf.com/doc/account/#sign-up),
- Create an [On-Premise Provider](https://api.octoperf.com/doc/on-premise-agent/): providers are groups of load generators,
- And [register an On-Premise Agent](https://api.octoperf.com/doc/on-premise-agent/provider-type/on-premise/) within the previously created *On-Premise Provider*.

Congratulations! The setup is now complete.