There is a backup strategy that looks correct and is not: stop nothing, tar the volume directory, upload the tarball, sleep well.
The problem is that a running database does not keep a consistent state on disk. Pages are dirty in memory, the write-ahead log is mid-transaction, and the file you copied at 03:00:01 does not match the file you copied at 03:00:04. What you get is a crash-consistent snapshot — sometimes it recovers on restore, sometimes it comes back with a corrupt index, and you find out which on the day it matters.
Do this instead: dump from inside, then take the volume.
PostgreSQL:
MySQL or MariaDB — note the flags, they are not optional:
MongoDB:
Redis is the exception — it has no useful dump format for this purpose, so you trigger a save and take the volume:
The other half nobody does: test the restore.
A backup you have not restored is a hypothesis. Restore into a scratch container once a quarter, run one query against it, throw it away. Fifteen minutes, and it is the only thing that turns the hypothesis into a fact.
Application-aware, not just file-aware
The general principle is that the backup has to know what it is backing up. For a stack — an app container plus its database plus its Redis — a correct backup is a logical dump of each data service, plus the volumes for the file data, plus the compose definition to rebuild the topology. Panelica's stack backup does the detection and picks the right dump command per engine, then packages the volumes alongside it, which is the same three parts assembled automatically. Whether you use tooling or a shell script, that is the shape you are aiming for: dump the data engines properly, archive the rest, and keep the definition that puts it back together.
What are people here using for offsite? Interested in whether anyone has landed on something better than "cron plus rclone".
The problem is that a running database does not keep a consistent state on disk. Pages are dirty in memory, the write-ahead log is mid-transaction, and the file you copied at 03:00:01 does not match the file you copied at 03:00:04. What you get is a crash-consistent snapshot — sometimes it recovers on restore, sometimes it comes back with a corrupt index, and you find out which on the day it matters.
Do this instead: dump from inside, then take the volume.
PostgreSQL:
Code:
docker exec -t pg-container pg_dumpall -c -U postgres > all.sqlMySQL or MariaDB — note the flags, they are not optional:
Code:
docker exec -t mysql-container \
mysqldump -u root -p"$PASS" --all-databases --routines --triggers --events > all.sql--routines --triggers --events is the difference between restoring a database and restoring the tables of a database. Stored procedures, triggers and scheduled events live outside the table data and a default mysqldump leaves all three behind. People discover this when the application restores fine and then quietly stops doing whatever the nightly event used to do.MongoDB:
Code:
docker exec -t mongo-container mongodump --archive=/tmp/dump.archive
docker cp mongo-container:/tmp/dump.archive ./dump.archive
docker exec -t mongo-container rm -f /tmp/dump.archiveRedis is the exception — it has no useful dump format for this purpose, so you trigger a save and take the volume:
Code:
docker exec -t redis-container redis-cli BGSAVE
# wait for it to finish, then archive the volumeThe other half nobody does: test the restore.
A backup you have not restored is a hypothesis. Restore into a scratch container once a quarter, run one query against it, throw it away. Fifteen minutes, and it is the only thing that turns the hypothesis into a fact.
Application-aware, not just file-aware
The general principle is that the backup has to know what it is backing up. For a stack — an app container plus its database plus its Redis — a correct backup is a logical dump of each data service, plus the volumes for the file data, plus the compose definition to rebuild the topology. Panelica's stack backup does the detection and picks the right dump command per engine, then packages the volumes alongside it, which is the same three parts assembled automatically. Whether you use tooling or a shell script, that is the shape you are aiming for: dump the data engines properly, archive the rest, and keep the definition that puts it back together.
What are people here using for offsite? Interested in whether anyone has landed on something better than "cron plus rclone".