Backup Automation with Paperless-Ngx and Rclone

Regularly creating backups in your paperless office, especially when using Paperless-ngx, is indispensable for several reasons:

• Data recovery: All your documents, including important contracts, invoices, and personal records, are stored digitally. Regular backups protect against data loss due to hardware failures, software issues, and accidental deletion. In the event of natural disasters, theft, or cyberattacks, you can quickly restore your data and maintain your business operations.
• Legal requirements: You may be legally required to retain certain documents for a specific duration. Through backups, you ensure that you have the documents available for legal processes or audits.
• Version control: With regular backups, you can create a chronological archive of your documents. This allows you to access older versions of your documents, which is useful for tracking changes.

You may have already heard of the 3-2-1 backup rule. It's a recommended strategy for creating backups and ensuring high data security. The rule is simple but effective.

The three criteria:

• 3 copies of your data: Create at least three separate copies of your data. This includes your original data in Paperless and two additional backups. If one backup fails, you have another option to restore your data.
• 2 different devices: Store these backups on at least two distinct devices. These can be hard drives or cloud storage. Different storage technologies have different vulnerabilities, and diversifying across multiple types reduces device-specific risks.
• 1 off-site location: At least one of the backups should be stored at a different location. This protects your data in the event of natural disasters or theft at your main office. Cloud storage is often used for off-site backups because you can access it from anywhere.

Implementing the 3-2-1 backup rule helps you effectively protect your data by diversifying across multiple storage types and locations.

If you want to follow the 3-2-1 backup rule, you can now consider where you want to secure your data.

I will show you an example of setting up a Paperless backup automation from Linux to Google Drive using the open-source tool rclone.

Most NAS or VPS run a Linux distribution. Therefore, the steps should be quite similar regardless of the device.

Rclone is a program for creating backups on cloud storage, running on Linux, macOS, and Windows, and supports over 70 cloud providers (Google Drive, OneDrive, Dropbox, Amazon S3, etc.)

Rclone mirrors your data from your server to the cloud storage. Thus, not everything needs to be re-uploaded with each change. To prevent the cloud provider from reading your data, we will also encrypt the backup with rclone.

The prerequisite for this guide is that you have already successfully installed Paperless-ngx and fed it with some documents. You also need an account with a cloud provider. Here I use Google Drive. The following steps are very similar and sometimes simpler for other cloud providers.

There are various ways to make Paperless-ngx backups. I recommend using the native “Paperless Document Exporter” & “Importer” instead of simply copying your entire Paperless folders. Since Paperless is probably running in Docker for you, some of the data might be in Docker volumes. You can run the Document Exporter with just one command, and it exports not only all files but also any configurations, user accounts, tags, and everything the algorithm has learned.

Log in to your server via SSH and execute the command to install rclone via snap:

1sudo snap install rclone

The snap store daemon must be installed for this (it is usually pre-installed on Ubuntu). If the command above fails, install the snap store daemon first:

1sudo apt update
2sudo apt install snapd

For this guide, it's important that rclone is installed on a device with web browser access (in addition) to retrieve a token for connecting with the cloud provider (see Step 3).

Since I am using a VPS without a web browser, I install rclone additionally on my Mac. For this, I use the package manager homebrew:

1brew install rclone

You can find downloads for other operating systems here.

Log into the Google API Console. It must be the same account that you want to use for Google Drive.

Select a project or create a new project.

Under APIs & Services, click on + ENABLE APIS AND SERVICES and search for drive, then activate the Google Drive API.

Click on Credentials in the left sidebar, then click on CONFIGURE CONSENT SCREEN.

Select External and click CREATE. Then enter an app name, the user support email (your own email is okay), and the developer contact information (your own email is okay). Click SAVE AND CONTINUE (all other fields are optional).

Click on ADD OR REMOVE SCOPES. Add the scopes with the scope .../auth/docs, .../auth/drive, and .../auth/drive.metadata.readonly so that you can edit, create, and delete files with rclone. After that, click UPDATE and then SAVE AND CONTINUE.

Add your own account as a test user. Then click SAVE AND CONTINUE.

Then click on Credentials in the left sidebar again. Click on + CREATE CREDENTIALS and select OAuth client ID.

Select Desktop app as the application type and click CREATE.

Now you will see the Client ID and Client secret. Copy both! We will need these in Step 3 for rclone.

Click on OAuth consent screen in the left sidebar, then click on PUBLISH APP and confirm.

Go back to your terminal and create a new remote connection with rclone. The command for this is:

1rclone config

1e) Edit existing remote
2n) New remote
3d) Delete remote
4r) Rename remote
5c) Copy remote
6s) Set configuration password
7q) Quit config
8e/n/d/r/c/s/q>

Type n to create a new remote. Then enter a name for the remote (e.g., gdrive).

After that, select the cloud provider. For this, enter the correct number (17 for Google Drive).

1Option Storage.
2Type of storage to configure.
3Choose a number from below, or type in your own value.
4...
517 / Google Drive
6   \ (drive)
7...
8Storage> 17

Next, enter the Client ID from Step 2:

1Option client_id.
2Google Application Client Id
3Setting your own is recommended.
4See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
5If you leave this blank, it will use an internal key which is low performance.
6Enter a value. Press Enter to leave empty.
7client_id> my-client-id

Next, enter the Client Secret from Step 2:

1Option client_secret.
2OAuth Client Secret.
3Leave blank normally.
4Enter a value. Press Enter to leave empty.
5client_secret> my-client-secret

Choose the access permissions for rclone by entering a number (for the backup to work, number 3 is sufficient).

1Option scope.
2Comma separated list of scopes that rclone should use when requesting access from drive.
3Choose a number from below, or type in your own value.
4Press Enter to leave empty.
5 1 / Full access all files, excluding Application Data Folder.
6   \ (drive)
7 2 / Read-only access to file metadata and file contents.
8   \ (drive.readonly)
9   / Access to files created by rclone only.
10 3 | These are visible in the drive website.
11   | File authorization is revoked when the user deauthorizes the app.
12   \ (drive.file)
13   / Allows read and write access to the Application Data folder.
14 4 | This is not visible in the drive website.
15   \ (drive.appfolder)
16   / Allows read-only access to file metadata but
17 5 | does not allow any access to read or download file content.
18   \ (drive.metadata.readonly)
19scope> 3

For the next step, you can leave it blank by pressing ENTER.

1Option service_account_file.
2Service Account Credentials JSON file path.
3Leave blank normally.
4Needed only if you want use SA instead of interactive login.
5Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
6Enter a value. Press Enter to leave empty.
7service_account_file>

After that, select the default by pressing ENTER.

1Edit advanced config?
2y) Yes
3n) No (default)
4y/n>

If your server does not have a web browser, enter n.

1Use web browser to automatically authenticate rclone with remote?
2 * Say Y if the machine running rclone has a web browser you can use
3 * Say N if running rclone on a (remote) machine without web browser access
4If not sure try Y. If Y failed, try N.
5
6y) Yes (default)
7n) No
8y/n> n

Copy the entire line that starts with rclone authorize "drive" and execute it on your computer. To do this, open a new tab in your terminal. Your web browser should launch, and you can log into your Google account. Then copy the token and paste it back into your server terminal.

1Option config_token.
2For this to work, you will need rclone available on a machine that has
3a web browser available.
4For more help and alternate methods see: https://rclone.org/remote_setup/
5Execute the following on the machine with the web browser (same rclone
6version recommended):
7	rclone authorize "drive" "very-long-code"
8Then paste the result.
9Enter a value.
10config_token> mein-token

Select the default by pressing ENTER:

1Configure this as a Shared Drive (Team Drive)?
2
3y) Yes
4n) No (default)
5y/n> n

You will see a summary of your configuration. Press ENTER.

1Keep this "gdrive" remote?
2y) Yes this is OK (default)
3e) Edit this remote
4d) Delete this remote
5y/e/d> y

Now the connection to your Google Drive account is established.

We will now create another remote. This remote will form an encryption layer over the cloud storage remote that we just created. If you do not wish to encrypt the backup, you can skip this step and proceed to Step 5. The advantage of an unencrypted backup is that you can use Google Drive OCR search to quickly find your documents in Google Drive.

Start the configuration again with rclone config and create a new remote (n). You could name it, for example, gdrive_encrypted.

Next, select crypt from the list of providers by entering number 13.

Then you specify the name of the Google Drive remote along with the desired path where the backup should be stored. My remote is called gdrive and I want to store the backup in a folder named paperless_backup_encrypted. So, I enter gdrive:paperless_backup_encrypted:

1Option remote.
2Remote to encrypt/decrypt.
3Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
4"myremote:bucket" or maybe "myremote:" (not recommended).
5Enter a value.
6remote> gdrive:paperless_backup_encrypted

Then you select the encryption of the file names. I choose the default by pressing ENTER:

1Option filename_encryption.
2How to encrypt the filenames.
3Choose a number from below, or type in your own string value.
4Press Enter for the default (standard).
5   / Encrypt the filenames.
6 1 | See the docs for the details.
7   \ (standard)
8 2 / Very simple filename obfuscation.
9   \ (obfuscate)
10   / Don't encrypt the file names.
11 3 | Adds a ".bin", or "suffix" extension only.
12   \ (off)
13filename_encryption> 1

Then you select the encryption of the folder names. I choose the default again:

1Option directory_name_encryption.
2Option to either encrypt directory names or leave them intact.
3NB If filename_encryption is "off" then this option will do nothing.
4Choose a number from below, or type in your own boolean value (true or false).
5Press Enter for the default (true).
6 1 / Encrypt directory names.
7   \ (true)
8 2 / Don't encrypt directory names, leave them intact.
9   \ (false)
10directory_name_encryption> 1

Then you set two passwords for the encryption. I let the passwords be generated with the maximum length.

1Option password.
2Password or pass phrase for encryption.
3Choose an alternative below.
4y) Yes, type in my own password
5g) Generate random password
6y/g> g
7Password strength in bits.
864 is just about memorable
9128 is secure
101024 is the maximum
11Bits> 1024
12Your password is: dCGprYdlIOtBGa5ozP7H2VM6uXRp_KlwH5OfEoufl-IOgJKXXSWNVKR92K0vf72u1oj3MM9CoEuEgnKTEyxh2na8LtGn-X6v3EGzNkr-UXrBg38pxrijclE0_jtOz1q0ldJe0X9Z918Fd0ZxDmAwT3IqjyTEXhs9bJMedYhyk9w
13Use this password? Please note that an obscured version of this
14password (and not the password itself) will be stored under your
15configuration file, so keep this generated password in a safe place.
16y) Yes (default)
17n) No
18y/n> y
19
20Option password2.
21Password or pass phrase for salt.
22Optional but recommended.
23Should be different to the previous password.
24Choose an alternative below. Press Enter for the default (n).
25y) Yes, type in my own password
26g) Generate random password
27n) No, leave this optional password blank (default)
28y/g/n> g
29Password strength in bits.
3064 is just about memorable
31128 is secure
321024 is the maximum
33Bits> 1024
34Your password is: XuL0KVb8vQWnIodgem7qWTlD6vrZKt18dgUmMAK61v1coMUt7DCc6EMPww4viD7YcQDcE78miAKBg9L9Qm8mx2kXiyquMUXrvND-BC9qcJMp95cJsPrsocVxQ26b1aU7aXa3glZre69phmqICZVb6ijfo_-61KRiOsBNCt-QKkA
35Use this password? Please note that an obscured version of this
36password (and not the password itself) will be stored under your
37configuration file, so keep this generated password in a safe place.
38y) Yes (default)
39n) No
40y/n> y
41
42Edit advanced config?
43y) Yes
44n) No (default)
45y/n> n
46
47Configuration complete.

Copy the passwords and save them in a secure place. This completes the configuration of the crypt remote. Confirm with ENTER and terminate the process with CTRL+C.

I'll now show you how to create a script that will create a backup for you when executed. Go to the folder where you want to save the script. In my case, I'm going to the paperless-ngx folder:

1cd paperless-ngx

Then create a new file:

1sudo nano backup.sh

Then paste the script. My script is deliberately kept very simple. You can copy it, but you need to adjust the absolute path to your docker-compose.yml file and your export folder:

1#!/bin/bash
2
3# path to your docker-compose.yml
4cd /home/tobias
5
6# execute the paperless document exporter
7# save the backup in the export folder
8docker compose exec -T webserver document_exporter ../export
9
10# rclone command to encrypt and sync with the cloud
11/snap/bin/rclone sync /home/tobias/paperless-ngx/export gdrive_encrypted:

Save and close the file afterwards with CTRL+C, Y, and ENTER.

To make the script executable, enter this command:

1chmod +x backup.sh

Test now if your script works by executing it via the command line:

1./backup.sh

Depending on the amount of data, it may take several minutes. The process should terminate automatically.

1100%|██████████| 265/265 [00:00<00:00, 332.70it/s]

If you now open Google Drive in your web browser, you should find a new folder in your storage. In this folder, you will find the encrypted data from Paperless.

Now we will create a new cron job that automatically executes the script at a defined time:

1crontab -e

If the script, for example, is to be executed daily at 3:00 a.m., add the following line at the end of the file:

10 3 * * * /home/tobias/paperless-ngx/backup.sh >> /home/tobias/paperless-ngx/backup.log 2>&1

Here you need to specify the absolute path to your backup script and the path where the log file should be created. Save and close the file. When you execute crontab -l, you should see a list of your cron jobs including the added line.

First and foremost, make sure to securely store the passwords for the backup encryption. If your server becomes unreachable, the backup will be useless if you cannot decrypt it.

If your server has failed and you don't have a snapshot, you'll need to reinstall rclone and Paperless on your server. Follow the same steps and then don't choose a new password for the crypt remote, but instead, enter the previous passwords. This way, rclone can decrypt your backup again.

Afterwards, execute the rclone sync command to import the backup:

1rclone sync gdrive_encrypted: /home/tobias/paperless-ngx/export

Once your Paperless export folder is filled again, you can restore your Paperless instance with the Document Importer:

1docker compose exec webserver document_importer ../export

The presented Paperless backup automation can be similarly modeled with other cloud providers. Typically, an API key or client ID with a client secret is sufficient for authentication.

When you run the Document Exporter, the export folder in your Paperless directory contains everything to restore your instance.

If you want to make backups to local hard drives, you can simply copy the contents of the export folder. You can use tools like rsync paired with a cron job for this purpose. Some NAS operating systems already have integrated local backup functions.