Administration¶
Benji is an important tool when it’s responsible for keeping backups of your important data. Backups, scrubs, restores and cleanups must run smoothly and need to be monitored closely. Also, the database backend and the collection of data storages need to meet your availability requirements.
Secure your Metadata¶
This section shows methods of how to keep your metadata safe even when disasters happen.
Metadata Redundancy¶
Benji already exports the version metadata to the storage, too. You
can restore this information with benji metadata-restore
:
The metadata-backend-less uses this import from the storage to populate an in-memory database to enable restores when the metadata backend is unavailable, please see section With Unavailable Metadata Backend.
You can also make further copies of the metadata with benji metadata-export
and store them somewhere safe to increase your redundancy even more. It is
advisable to compress them as the JSON export format is quite redundant.
You can import these exports again with:
If the imported version already exists in the database backend Benji terminates with an error and doesn’t proceed with the import.
So now, even if your backup database server crashes, you’ll still be able to reimport all existing versions again!
Attention
When you remove (benji rm
) versions from the database and
then call benji cleanup
, the blocks containing the backed up data will
be removed. No benji metadata-import
can bring them back, because
Benji’s export format only contains metadata information.
Database High-Availability¶
An additional option against data loss is to replicate the SQL database. Please refer to the database documentation. You should also have a regular database backup in place.
Caution
DBMS replication only helps when one server crashes or has a
failure. It does not help against software-bug related data loss, human
error and more. So the automatic metadata backup and benji metadata-export
are the only reliable options for long-term data safety.
Secure your block data¶
Your storage should be redundant in some way, too. Most cloud providers have an SLA which guarantees a certain level of availability. If you manage your storage yourself, then you should look into redundancy and high-availability technologies like:
- RAID 1, 5 and 6
- Redundancy provided by a distributed objected store like Ceph or Minio
- DRBD
- Filesystem specific data redundancy and replication mechanisms in filesystems like Btrs or ZFS
If your storage fails or has corruptions, at best corrupted restores will be possible. Benji doesn’t store any redundant data and it cannot restore data from stored checksums alone.
Monitoring¶
Tips & tricks¶
You should monitor exit codes of Benji closely. Anything != 0 means that there was a problem.
Benji writes all output including possible tracebacks and command lines to the configured logfile (see Configuration). If anything goes wrong, you’ll be able to visit this logfile and get enough information to troubleshoot the problem, even if Benji was called from an automated script.
You should also monitor the success of the backups. In addition to checking the
exit code, you can do this with benji ls
and see if the column valid
is True. For a currently running backup this column is False but it will change
to True on successful completion of the backup.
You can also monitor the progress of the backups either by looking at the logfile or by checking your process-tree:
$ ps axfu|grep "[b]acky2"
… \_ benji [Scrubbing Version V00000001 (0.1%)]
To know which backup took how long and to see how many blocks/bytes have been
read and written, you can use the benji stats
command:
$ benji stats --help
usage: benji stats [-h] [-l LIMIT] [filter_expression]
positional arguments:
filter_expression Statistics filter expression
optional arguments:
-h, --help show this help message and exit
-l LIMIT, --limit LIMIT
Limit output to this number of entries
Example:
$ benji stats
INFO: $ benji stats
+---------------------+-------------+------+---------------+---------+------------+---------+---------+---------+--------+--------------+
| date | uid | name | snapshot_name | size | block_size | read | written | dedup | sparse | duration (s) |
+---------------------+-------------+------+---------------+---------+------------+---------+---------+---------+--------+--------------+
| 2018-06-13T15:21:55 | V0000000001 | test | | 40.0MiB | 4.0MiB | 40.0MiB | 40.0MiB | 0.0B | 0.0B | 00s |
| 2018-06-13T15:21:57 | V0000000002 | test | | 40.0MiB | 4.0MiB | 40.0MiB | 0.0B | 40.0MiB | 0.0B | 00s |
| 2018-06-13T15:21:58 | V0000000003 | test | | 40.0MiB | 4.0MiB | 40.0MiB | 0.0B | 40.0MiB | 0.0B | 00s |
| 2018-06-13T15:21:59 | V0000000004 | test | | 40.0MiB | 4.0MiB | 40.0MiB | 0.0B | 40.0MiB | 0.0B | 00s |
+---------------------+-------------+------+---------------+---------+------------+---------+---------+---------+--------+--------------+
Machine output¶
Some commands can also produce machine readable JSON output for usage in scripts:
$ benji -m ls
{
"metadataVersion": "1.0.0",
"versions": [
{
"uid": 1,
"date": "2018-06-07T12:51:19",
"name": "test",
"snapshot_name": "",
"size": 41943040,
"block_size": 4194304,
"valid": true,
"protected": false,
"tags": []
}
]
}
Note
Take care to put the -m
between benji
and ls
.
All messages emitted by Benji are written to STDERR. In contrast
the machine readable output is written to STDOUT. Also, when using -m
the
logging level is adjusted to only output errors. The Benji logfile still gets
the whole output.
Here’s a table of commands supporting machine readable output and their output:
Command | Description of output |
---|---|
ls | List of matching versions |
stats | List of matching statistics |
backup | List of newly create version |
enforce | List of removed versions |
scrub | List of scrubbed versions and of versions with errors |
deep-scrub | List of scrubbed versions and of versions with errors |
batch-scrub | List of scrubbed versions and of versions with errors |
batch-deep-scrub | List of scrubbed versions and of versions with errors |
All other commands also accept the -m
switch. But for them only the logging
level is turned down.
jq is an excellent tool for parsing this data
and filtering out the bits you want. Here’s a short example, but see the scripts/
and images/benji-rook/scripts/
directories for more:
$ benji -m ls | jq -r '.versions[0].date'
2018-06-07T12:51:19
With machine readable output you can use the option --include-blocks
to ls
which includes all blocks of this version in the output.
Version UIDs will be represented as simple integers without the V prefix and being zero-filled. All Benji commands are able to take this representation as well, so you can use such UIDs in further commands as-is.
All timestamps are in UTC and without timezone information.
Debugging¶
In case something goes wrong, you can use the -v
switch to increase the
logging verbosity. This outputs much more information.