Administration

Benji is an important tool when it’s responsible for keeping backups of your important data. Backups, scrubs, restores and cleanups must run smoothly and need to be monitored closely. Also, the database backend and the collection of data storages need to meet your availability requirements.

Secure your Metadata

This section shows methods of how to keep your metadata safe even when disasters happen.

Metadata Redundancy

Benji already exports the version metadata to the storage, too. You can restore this information with benji metadata-restore:

The metadata-backend-less uses this import from the storage to populate an in-memory database to enable restores when the metadata backend is unavailable, please see section With Unavailable Metadata Backend.

You can also make further copies of the metadata with benji metadata-export and store them somewhere safe to increase your redundancy even more. It is advisable to compress them as the JSON export format is quite redundant.

You can import these exports again with:

If the imported version already exists in the database backend Benji terminates with an error and doesn’t proceed with the import.

So now, even if your backup database server crashes, you’ll still be able to reimport all existing versions again!

Attention

When you remove (benji rm) versions from the database and then call benji cleanup, the blocks containing the backed up data will be removed. No benji metadata-import can bring them back, because Benji’s export format only contains metadata information.

Database High-Availability

An additional option against data loss is to replicate the SQL database. Please refer to the database documentation. You should also have a regular database backup in place.

Caution

DBMS replication only helps when one server crashes or has a failure. It does not help against software-bug related data loss, human error and more. So the automatic metadata backup and benji metadata-export are the only reliable options for long-term data safety.

Secure your block data

Your storage should be redundant in some way, too. Most cloud providers have an SLA which guarantees a certain level of availability. If you manage your storage yourself, then you should look into redundancy and high-availability technologies like:

  • RAID 1, 5 and 6
  • Redundancy provided by a distributed objected store like Ceph or Minio
  • DRBD
  • Filesystem specific data redundancy and replication mechanisms in filesystems like Btrs or ZFS

If your storage fails or has corruptions, at best corrupted restores will be possible. Benji doesn’t store any redundant data and it cannot restore data from stored checksums alone.

Monitoring

Tips & tricks

You should monitor exit codes of Benji closely. Anything != 0 means that there was a problem.

Benji writes all output including possible tracebacks and command lines to the configured logfile (see Configuration). If anything goes wrong, you’ll be able to visit this logfile and get enough information to troubleshoot the problem, even if Benji was called from an automated script.

You should also monitor the success of the backups. In addition to checking the exit code, you can do this with benji ls and see if the column valid is True. For a currently running backup this column is False but it will change to True on successful completion of the backup.

You can also monitor the progress of the backups either by looking at the logfile or by checking your process-tree:

$ ps axfu|grep "[b]acky2"
…  \_ benji [Scrubbing Version V00000001 (0.1%)]

To know which backup took how long and to see how many blocks/bytes have been read and written, you can use the benji stats command:

$ benji stats --help
usage: benji stats [-h] [-l LIMIT] [filter_expression]

positional arguments:
  filter_expression     Statistics filter expression

optional arguments:
  -h, --help            show this help message and exit
  -l LIMIT, --limit LIMIT
                        Limit output to this number of entries

Example:

$ benji stats
    INFO: $ benji stats
+---------------------+-------------+------+---------------+---------+------------+---------+---------+---------+--------+--------------+
|         date        | uid         | name | snapshot_name |   size  | block_size |    read | written |   dedup | sparse | duration (s) |
+---------------------+-------------+------+---------------+---------+------------+---------+---------+---------+--------+--------------+
| 2018-06-13T15:21:55 | V0000000001 | test |               | 40.0MiB |   4.0MiB   | 40.0MiB | 40.0MiB |    0.0B |   0.0B |          00s |
| 2018-06-13T15:21:57 | V0000000002 | test |               | 40.0MiB |   4.0MiB   | 40.0MiB |    0.0B | 40.0MiB |   0.0B |          00s |
| 2018-06-13T15:21:58 | V0000000003 | test |               | 40.0MiB |   4.0MiB   | 40.0MiB |    0.0B | 40.0MiB |   0.0B |          00s |
| 2018-06-13T15:21:59 | V0000000004 | test |               | 40.0MiB |   4.0MiB   | 40.0MiB |    0.0B | 40.0MiB |   0.0B |          00s |
+---------------------+-------------+------+---------------+---------+------------+---------+---------+---------+--------+--------------+

Machine output

Some commands can also produce machine readable JSON output for usage in scripts:

$ benji -m ls
{
  "metadataVersion": "1.0.0",
  "versions": [
    {
      "uid": 1,
      "date": "2018-06-07T12:51:19",
      "name": "test",
      "snapshot_name": "",
      "size": 41943040,
      "block_size": 4194304,
      "valid": true,
      "protected": false,
      "tags": []
    }
  ]
}

Note

Take care to put the -m between benji and ls.

All messages emitted by Benji are written to STDERR. In contrast the machine readable output is written to STDOUT. Also, when using -m the logging level is adjusted to only output errors. The Benji logfile still gets the whole output.

Here’s a table of commands supporting machine readable output and their output:

Command Description of output
ls List of matching versions
stats List of matching statistics
backup List of newly create version
enforce List of removed versions
scrub List of scrubbed versions and of versions with errors
deep-scrub List of scrubbed versions and of versions with errors
batch-scrub List of scrubbed versions and of versions with errors
batch-deep-scrub List of scrubbed versions and of versions with errors

All other commands also accept the -m switch. But for them only the logging level is turned down.

jq is an excellent tool for parsing this data and filtering out the bits you want. Here’s a short example, but see the scripts/ and images/benji-rook/scripts/ directories for more:

$ benji -m ls | jq -r '.versions[0].date'
2018-06-07T12:51:19

With machine readable output you can use the option --include-blocks to ls which includes all blocks of this version in the output.

Version UIDs will be represented as simple integers without the V prefix and being zero-filled. All Benji commands are able to take this representation as well, so you can use such UIDs in further commands as-is.

All timestamps are in UTC and without timezone information.

Debugging

In case something goes wrong, you can use the -v switch to increase the logging verbosity. This outputs much more information.