Skip to content

Data Persistence

OctoPerf On Premise Infra entirely runs as Docker containers. Containers are inherently volatile: they can be created and destroyed easily. How can we persist data in this context, given the containers may be destroyed and re-created any time? Docker volumes. OctoPerf relies on Docker to let you manage how you want to persist your data.

Warning

Beware that any change in data persistence settings may delete your previously existing data. Always make a backup of your data before any manipulation.

Datas

The datas which need to be persisted are:

  • Elasticsearch: Elasticsearch stores the virtual users, scenarios and test results within the Elasticsearch container on path /usr/share/elasticsearch/data,
  • OctoPerf Resources: This includes Project files, JMeter logs and JMeter JTLs. This is saved within the enterprise edition container inside /root/data.

Warning

Saving these folders from inside the containers is not recommended, please follow the procedures below to backup the docker volumes from the host machine.

By default, when using our docker-compose.yml to setup the On Premise Infra, Named volumes are configured:

  • OctoPerf Data Location: octoperf-data local volume by default. Defines the volume where backend stores data like JMeter Logs and project files.
  • ElasticSearch Data Location: elasticsearch-data local volume by default. Defines the volume where backend stores data like JMeter Logs and project files.

Other Backend Configuration Settings:

  • OctoPerf License Location: ./license local folder by default. Defines where the backend looks up the license.l4j file.
  • OctoPerf Config Location: ./config local folder by default. Defines where the backend looks up the application.yml config file.

Persistence Strategies

Container

This strategy is only suitable for testing purpose. All the data is wiped if a container is deleted.

In this case, the data is saved inside the docker container itself. There is no real advantage to use this method. The major drawback is that any OctoPerf On premise Infra upgrade will entirely wipe off all your data.

Local Directory

Local directory volume mapping allows to map a local directory on your computer to a directory inside a docker container. For example, a Host Directory (Mount a host directory as a data volume) can be mounted directly as a volume. In this case, a local directory on your machine is mapped to the directory inside the container. This setup makes it easy to move data from one physical machine to another because the data is accessible from the host machine.

Let's say the location you want to use to store octoperf-data is /path/to/new/octoperf-data

Here is the part of the docker-compose file that needs to be modified to use the local folder instead of the Named Volume :

docker-compose

  On Premise Infra:
  #...
    volumes:
      - octoperf-data:/home/octoperf/data
      - ./config:/home/octoperf/config
      - ./license:/home/octoperf/license

  On Premise Infra:
  #...
    volumes:
      - /path/to/new/octoperf-data:/home/octoperf/data
      - ./config:/home/octoperf/config
      - ./license:/home/octoperf/license

The same goes for elastic-search-data, the following block should be modified :

docker-compose

  services:
    elasticsearch:
      image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
      container_name: elasticsearch
      environment:
        - network.host=0.0.0.0
        - discovery.type=single-node # clustering and the bootstrap checks are disabled
      stdin_open: true
      tty: true
      restart: unless-stopped
      volumes:
        - elasticsearch-data:/usr/share/elasticsearch/data

  services:
    elasticsearch:
      image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
      container_name: elasticsearch
      environment:
        - network.host=0.0.0.0
        - discovery.type=single-node # clustering and the bootstrap checks are disabled
      stdin_open: true
      tty: true
      restart: unless-stopped
      volumes:
        - /path/to/new/elasticsearch-data:/usr/share/elasticsearch/data

Named Volumes

When setting up OctoPerf On Premise Infra using our docker-compose.yml configuration, data is persisted using Local Docker volumes. When upgrading Elasticsearch or OctoPerf itself, the data is kept safe in a separate volume on the local machine. This default configuration allows you to use OctoPerf without messing with Docker Volume Mapping.

You can list the existing Docker volumes by running docker volume ls

local               On Premise Infra_elasticsearch-data_9a8a2
local               On Premise Infra_octoperf-data_842b2

You can also inspect a single Docker Volume by name:

ubuntu@desktop:~$ docker inspect volume On Premise Infra_elasticsearch-data_9a8a2
[
    {
        "CreatedAt": "2017-10-19T11:01:12+02:00",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/On Premise Infra_elasticsearch-data_9a8a2/_data",
        "Name": "On Premise Infra_elasticsearch-data_9a8a2",
        "Options": {},
        "Scope": "local"
    }
]
Error: No such object: volume

It displays where the volume has been stored along with other information.

You can configure OctoPerf On premise Infra to use a specific path to store the datas, instead of the default octoperf-data volume. You can specify the absolute path on the docker host machine, for Example /path/to/elasticsearch-data for Elasticsearch Data Location.

Use Local Docker Host Path

On the machine running OctoPerf On premise Infra:

  • Create the folder in the wanted location, for Example /path/to/elasticsearch-data,
  • chmod -R 777: recursively allow write permission on the folder for anyone,
  • Specify the absolute path during OctoPerf configuration in the wanted setting. Example: Elasticsearch Data Location.

If you already had data stored in octoperf-data volume:

  • Stop All OctoPerf On premise Infra Services,
  • SSH onto the Docker host,
  • List all volumes by running docker volume ls. There should be at least enterprise-edition_elasticsearch-data_xxxx and enterprise-edition_octoperf-data_xxxx volumes,
  • Inspect the volume containing the existing data. Example: docker inspect volume enterprise-edition_elasticsearch-data_9a8a2,
  • Copy recursively the files and folders from Mountpoint folder to the new target folder by running:

cp -rp MOUNT_POINT NEW_PATH, where MOUNT_POINT is the existing volume folder and NEW_PATH the absolute path to the new folder.

  • Update the configuration files to change the volume settings, Example Elasticsearch Data Location changed from octoperf-data to /path/to/elasticsearch-data,
  • Start OctoPerf On premise Infra.

Your data should still be there, but now it's being stored in another location on the Docker Host.

Managing folder ownership

Make sure to maintain the rights while transferring otherwise the new folder will have to be edited as described in elasticsearch documentation. The command lines in the examples above use -rp to that end. If you need to transfer between server, consider creating a tar archive.

In case it is too late, you can use the following commands to change ownership of the folders to the right users:

chown -R 1000:1000 /path/to/elasticsearch-data

chown -R 9001:9001 /path/to/octoperf-data