Introduction

I have a small dedicated server which I use to host web applications running in docker containers. The context of each app is defined in a single docker-compose.yml file which provides all the benefits docker and docker-compose can offer.

Running self-hosted applications on docker, one of the challenges I wanted to solve was how to backup the volumes safely (without causing trouble to the running application) and efficiently.

This article will demonstrate the solution I came up with leveraging Borg Backup, cron and Ansible.

Requirements

My requirements were as follows - this solution must:

• Make use of open-source software only
• Support encrypted backups
• Support deduplicated backups
• Perform operations before the backup (I have in mind the creation of database dumps that could be backed up at the same time)
• Run as a service of the docker-compose stack. This way, running docker-compose down should not only remove the application from the server but the backup service as well, preventing myself to update a crontab as soon as I add or remove new services - which surely I am going to forget.
• Run in a container
  • It is not recommended to interact with volumes directly form the host filesystem. If you refer to the docker documentation, you will find their recommendations.
  • I want to avoid as much as possible to install new software on my server as I am trying to keep dependencies to a minimum. Restarting everything from scratch becomes a breeze.

After looking around I quickly found Borg that ticks the first 3 boxes and much more. Have a look at its features. For the 4th point, I decided to use Ansible to run the backup as well as extra operations. I find it easy to use, with resulting playbooks much easier to read and upgrade than bash scripts in my opinion, plus I don’t have to log anything as Ansible will provide a nice output by itself. As for the rest, I did it myself and wrote the necessary Dockerfiles to run the Ansible playbooks periodically.

Backup Nextcloud volumes as an example

This tutorial will backup a dockerized Nextcloud application as a use case. To make it simple to test, all instructions required to set up everything on a local machine are provided below. Although it is even better if you already have a running docker stack with volumes you want to back up, as well as a machine you can sshed into and that will host the backups. In that case jump to this section. You can apply the same instructions as below: just update the IPs and ssh keys related to the machine you want to back up to.

Here comes a picture of what will be covered:

You can get the entire setup directly from this github repository

The nextcloud stack

Create a docker-compose.yml file with the following content:

version: '3.7'

volumes:
  mysql:
  data:

services:

  mysql:
    image: mysql:5.7.24
    environment:
      MYSQL_ROOT_PASSWORD_FILE: /run/secrets/db_root_password
      MYSQL_USER: nextcloud
      MYSQL_DATABASE: nextcloud
      MYSQL_PASSWORD_FILE: /run/secrets/db_password
    volumes:
      - mysql:/var/lib/mysql
    secrets:
      - db_root_password
      - db_password

  php:
    image: nextcloud:15.0.5-fpm-alpine
    volumes:
      - data:/var/www/html
    depends_on:
      - mysql

  nginx:
    image: nginx:1.15.8-alpine
    ports:
      - 8042:80
    volumes:
      - data:/var/www/html
      - ./nextcloud-nginx.conf:/etc/nginx/nginx.conf
    depends_on:
      - php

secrets:
  db_password:
    file: secret_db_password.txt
  db_root_password:
    file: secret_db_root_password.txt

Create a file named nextcloud-nginx.conf with this content

Then run these commands after editing the passwords:

echo "KilmpodG65" > secret_db_password.txt
echo "F4gTYjukI" > secret_db_root_password.txt
docker-compose up -d

Wait for the mysql container to be initialized. You can do so by running

docker-compose logs -f mysql

If you can see “MySQL init process done. Ready for start up” you are good to go.

Browse http://localhost:8042/ then fill in the form. Hit finish setup and… be patient. You should have Nextcloud running in a few minutes.

Preparing the backup server

It is recommended to have Borg installed on your backup machine. Follow this guide.

For this example let’s just install it locally. On my laptop running on ubuntu bionic it is as simple as:

sudo apt install borgbackup
borg --version

The container that will send backups needs an SSH access to the backup server. Let’s create a new ssh key pair on the machine running your containers. This one will need to be without passphrase as it will be used in a cron.

ssh-keygen -f ~/.ssh/id_rsa_borg_backup -t rsa -N ''

Now - on your server - authorize this key.

touch ~/.ssh/authorized_keys
chmod g-w ~
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Copy the content of the public key at the end of the ~/.ssh/authorized_keys file.

If the entire setup is in local, this makes it easy:

cat ~/.ssh/id_rsa_borg_backup.pub >> ~/.ssh/authorized_keys

Obviously make sure you have an ssh server running on your server.

sudo apt install openssh-server

Adding the borg backup service

Update the docker-compose.yaml file

services:

  ...

  backup_cron:
    image: ovski/borgbackup-cron:v1.0.0
    volumes:
      - data:/var/docker_volumes/nextcloud/app/data
    environment:
      SSH_CONNECTION: backup_user@your.server.net
      PRIVATE_KEY_PATH: /run/secrets/backup_server_user_private_key
      BORG_REPO_PATH: /home/backup_user/borg_repositories
      BORG_REPO_NAME: nextcloud
      LOCAL_FOLDER: /var/docker_volumes/nextcloud
      MYSQL_USER: nextcloud
      MYSQL_DATABASE: nextcloud
      MYSQL_PASSWORD_FILE: /run/secrets/db_password
      SSH_KNOWN_HOSTS: your.server.net,38.26.55.241
    secrets:
      - backup_server_user_private_key
      - borg_passphrase
      - db_password

secrets:

  ...

  backup_server_user_private_key:
    file: secret_backup_server_user_private_key.txt
  borg_passphrase:
    file: secret_borg_passphrase.txt

There are a few changes we need to apply:

• Update the SSH_CONNECTION environment variable.

Use whichever user you want to use on your backup server. In my case it will be baptiste. Update the server IP or domain names (Same explanation as below).

• Update the SSH_KNOWN_HOSTS variable with the backup server IP and/or domain names.

On my ubuntu laptop, running hostname -I | awk '{print $2}' on the command line prints the host IP which is reachable from within containers. You might also get the piece of information by running ip a and look for the docker0 network.

• Update the BORG_REPO_PATH variable. Set it to whichever path you want your backups to be send to on your backup server.

• In case you are using your own docker stack which does not contain a mysql service, you must get rid of the MYSQL environment variables.

• The parent folder that will contain your backups needs to exist beforehand. On the backup server run mkdir /home/your_user/borg_repositories. On my laptop that will be mkdir /home/baptiste/borg_repositories.

• Finally create the secret files secret_backup_server_user_private_key.txt and secret_borg_passphrase.txt:

cat ~/.ssh/id_rsa_borg_backup > secret_backup_server_user_private_key.txt
echo "mysuperpassphrasefortheborgnextcloudrepo" > secret_borg_passphrase.txt
chmod 400 secret_backup_server_user_private_key.txt

We have to run the chmod command as we can’t use the mode keyword to force permissions outside of a docker swarm

Here is what I personally have:

services:

  ...

  backup_cron:
    image: ovski/borgbackup-cron:v1.0.0
    volumes:
      - data:/var/docker_volumes/nextcloud/app/data
    environment:
      SSH_CONNECTION: baptiste@172.17.0.1
      PRIVATE_KEY_PATH: /run/secrets/backup_server_user_private_key
      BORG_REPO_PATH: /home/baptiste/borg_repositories
      BORG_REPO_NAME: nextcloud
      LOCAL_FOLDER: /var/docker_volumes/nextcloud
      MYSQL_USER: nextcloud
      MYSQL_DATABASE: nextcloud
      MYSQL_PASSWORD_FILE: /run/secrets/db_password
      SSH_KNOWN_HOSTS: 172.17.0.1
    secrets:
      - backup_server_user_private_key
      - borg_passphrase
      - db_password

secrets:

  ...

  backup_server_user_private_key:
    file: ./secret_backup_server_user_private_key.txt
  borg_passphrase:
    file: ./secret_borg_passphrase.txt

Moment of truth

Run docker-compose up -d followed by docker-compose logs -f backup_cron. Every 5 minutes, you should see the following output:

backup_cron_1  | === I'm alive ===

This means our cron works fine!

OK that’s cool. Unfortunately the cron is setup to run every night at 1AM. You might not want to wait this late to ensure the backup runs smoothly. Edit the docker-compose.yml file another time by overriding the default command like this:

services:

  ...

  backup_cron:
    image: ovski/borgbackup-cron:v1.0.0
    ...
    command: /var/backup_script.sh

Let’s have another round of docker-compose up -d and docker-compose logs -f backup_cron. Wait for the script to end and the container to stop. You should eventually have a backup available. Let’s check it out:

cd /home/your_user/borg_repositories
borg list nextcloud

Enter your passphrase (mysuperpassphrasefortheborgnextcloudrepo). You should see the list of backups! If you are not familiar with borg already, head to the documentation

If you found this tutorial helpful, star this repo as a thank you! ⭐