From 329fd4bcf6504c74e3906e51c6fc2124bc09cc02 Mon Sep 17 00:00:00 2001 From: Alexandre Alapetite Date: Tue, 30 Apr 2024 08:31:13 +0200 Subject: [PATCH] CLI database backup and restore (#6387) * CLI database backup and restore Can also be used to migrate from one database to another (e.g. MySQL to PostgreSQL) or to ease upgrade to a major PostgreSQL version (e.g. 15 to 16). * +x * Fix some cases * Update to docker-compose-v2 * More documentation --- Docker/README.md | 77 +++++++++++++++++++++++++++++-------- app/Models/DatabaseDAO.php | 22 +++++++++-- cli/README.md | 8 ++++ cli/db-backup.php | 20 ++++++++++ cli/db-restore.php | 65 +++++++++++++++++++++++++++++++ docs/en/admins/05_Backup.md | 46 +++++++++++++++++++++- docs/en/admins/Caddy.md | 2 +- 7 files changed, 219 insertions(+), 21 deletions(-) create mode 100755 cli/db-backup.php create mode 100755 cli/db-restore.php diff --git a/Docker/README.md b/Docker/README.md index 54f60a991..4c45764d1 100644 --- a/Docker/README.md +++ b/Docker/README.md @@ -21,7 +21,7 @@ Example for Linux Debian / Ubuntu: ```sh # Install default Docker Compose and automatically the corresponding version of Docker -apt install docker-compose +apt install docker-compose-v2 ``` ## Quick run @@ -194,6 +194,8 @@ docker run -d --restart unless-stopped --log-opt max-size=10m \ In the FreshRSS setup, you will then specify the name of the container (`freshrss-db`) as the host for the database. +See also the section [Docker Compose with PostgreSQL](#docker-compose-with-postgresql) below. + ### [MySQL](https://hub.docker.com/_/mysql/) or [MariaDB](https://hub.docker.com/_/mariadb) ```sh @@ -285,13 +287,13 @@ See [`docker-compose.yml`](./freshrss/docker-compose.yml) ```sh cd ./FreshRSS/Docker/freshrss/ # Update -docker-compose pull +docker compose pull # Run -docker-compose -f docker-compose.yml -f docker-compose-local.yml up -d --remove-orphans +docker compose -f docker-compose.yml -f docker-compose-local.yml up -d --remove-orphans # Logs -docker-compose logs -f --timestamps +docker compose logs -f --timestamps # Stop -docker-compose down --remove-orphans +docker compose down --remove-orphans ``` Detailed (partial) example of Docker Compose for FreshRSS: @@ -378,13 +380,15 @@ See [`docker-compose-db.yml`](./freshrss/docker-compose-db.yml) ```sh cd ./FreshRSS/Docker/freshrss/ # Update -docker-compose -f docker-compose.yml -f docker-compose-db.yml pull +docker compose -f docker-compose.yml -f docker-compose-db.yml pull # Run -docker-compose -f docker-compose.yml -f docker-compose-db.yml -f docker-compose-local.yml up -d --remove-orphans +docker compose -f docker-compose.yml -f docker-compose-db.yml -f docker-compose-local.yml up -d --remove-orphans # Logs -docker-compose -f docker-compose.yml -f docker-compose-db.yml logs -f --timestamps +docker compose -f docker-compose.yml -f docker-compose-db.yml logs -f --timestamps ``` +See also the section [Migrate database](#migrate-database) below to upgrade to a major PostgreSQL version with Docker Compose. + ### Docker Compose for development Use the local (git) FreshRSS source code instead of the one inside the Docker container, @@ -396,11 +400,11 @@ See [`docker-compose-development.yml`](./freshrss/docker-compose-development.yml cd ./FreshRSS/Docker/freshrss/ # Update git pull --ff-only --prune -docker-compose pull +docker compose pull # Run -docker-compose -f docker-compose-development.yml -f docker-compose.yml -f docker-compose-local.yml up --remove-orphans +docker compose -f docker-compose-development.yml -f docker-compose.yml -f docker-compose-local.yml up --remove-orphans # Stop with [Control]+[C] and purge -docker-compose down --remove-orphans --volumes +docker compose down --remove-orphans --volumes ``` > ℹ️ You can combine it with `-f docker-compose-db.yml` to spin a PostgreSQL database. @@ -446,13 +450,13 @@ See [`docker-compose-proxy.yml`](./freshrss/docker-compose-proxy.yml) ```sh cd ./FreshRSS/Docker/freshrss/ # Update -docker-compose -f docker-compose.yml -f docker-compose-proxy.yml pull +docker compose -f docker-compose.yml -f docker-compose-proxy.yml pull # Run -docker-compose -f docker-compose.yml -f docker-compose-proxy.yml up -d --remove-orphans +docker compose -f docker-compose.yml -f docker-compose-proxy.yml up -d --remove-orphans # Logs -docker-compose -f docker-compose.yml -f docker-compose-proxy.yml logs -f --timestamps +docker compose -f docker-compose.yml -f docker-compose-proxy.yml logs -f --timestamps # Stop -docker-compose -f docker-compose.yml -f docker-compose-proxy.yml down --remove-orphans +docker compose -f docker-compose.yml -f docker-compose-proxy.yml down --remove-orphans ``` > ℹ️ You can combine it with `-f docker-compose-db.yml` to spin a PostgreSQL database. @@ -650,3 +654,46 @@ docker run -d --restart unless-stopped --log-opt max-size=10m \ --name freshrss_cron freshrss/freshrss:alpine \ crond -f -d 6 ``` + +## Migrate database + +Our [CLI](../cli/README.md) offers commands to back-up and migrate user databases, +with `cli/db-backup.php` and `cli/db-restore.php` in particular. + +Here is an example (assuming our [Docker Compose example](#docker-compose-with-postgresql)) +intended for migrating to a newer major version of PostgreSQL, +but which can also be used to migrate between other databases (e.g. MySQL to PostgreSQL). + +```sh +# Stop FreshRSS container (Web server + cron) during maintenance +docker compose down freshrss + +# Optional additional pre-upgrade back-up using PostgreSQL own mechanism +docker compose -f docker-compose-db.yml \ + exec freshrss-db pg_dump -U freshrss freshrss | gzip -9 > freshrss-postgres-backup.sql.gz +# ------↑ Name of your PostgreSQL Docker container +# -----------------------------↑ Name of your PostgreSQL user for FreshRSS +# --------------------------------------↑ Name of your PostgreSQL database for FreshRSS + +# Back-up all users’ respective tables to SQLite files +docker compose -f docker-compose.yml -f docker-compose-db.yml \ + run --rm freshrss cli/db-backup.php + +# Remove old database (PostgreSQL) container and its data volume +docker compose -f docker-compose-db.yml \ + down --volumes freshrss-db + +# Edit your Compose file to use new database (e.g. newest postgres:xx) +nano docker-compose-db.yml + +# Start new database (PostgreSQL) container and its new empty data volume +docker compose -f docker-compose.yml -f docker-compose-db.yml \ + up -d freshrss-db + +# Restore all users’ respective tables from SQLite files +docker compose -f docker-compose.yml -f docker-compose-db.yml \ + run --rm freshrss cli/db-restore.php --delete-backup + +# Restart a new FreshRSS container after maintenance +docker compose -f docker-compose.yml -f docker-compose-db.yml up -d freshrss +``` diff --git a/app/Models/DatabaseDAO.php b/app/Models/DatabaseDAO.php index cdc74fa12..667cb61a2 100644 --- a/app/Models/DatabaseDAO.php +++ b/app/Models/DatabaseDAO.php @@ -48,6 +48,18 @@ class FreshRSS_DatabaseDAO extends Minz_ModelPdo { } } + public function exits(): bool { + $sql = 'SELECT * FROM `_entry` LIMIT 1'; + $stm = $this->pdo->query($sql); + if ($stm !== false) { + $res = $stm->fetchAll(PDO::FETCH_COLUMN, 0); + if ($res !== false) { + return true; + } + } + return false; + } + public function tablesAreCorrect(): bool { $res = $this->fetchAssoc('SHOW TABLES'); if ($res == null) { @@ -242,6 +254,7 @@ SQL; } $error = ''; + $databaseDAO = FreshRSS_Factory::createDatabaseDAO(); $userDAO = FreshRSS_Factory::createUserDao(); $catDAO = FreshRSS_Factory::createCategoryDao(); $feedDAO = FreshRSS_Factory::createFeedDao(); @@ -259,15 +272,18 @@ SQL; $error = 'Error: SQLite import file is not readable: ' . $filename; } elseif ($clearFirst) { $userDAO->deleteUser(); + $userDAO = FreshRSS_Factory::createUserDao(); if ($this->pdo->dbType() === 'sqlite') { //We cannot just delete the .sqlite file otherwise PDO gets buggy. //SQLite is the only one with database-level optimization, instead of at table level. $this->optimize(); } } else { - $nbEntries = $entryDAO->countUnreadRead(); - if (!empty($nbEntries['all'])) { - $error = 'Error: Destination database already contains some entries!'; + if ($databaseDAO->exits()) { + $nbEntries = $entryDAO->countUnreadRead(); + if (isset($nbEntries['all']) && $nbEntries['all'] > 0) { + $error = 'Error: Destination database already contains some entries!'; + } } } break; diff --git a/cli/README.md b/cli/README.md index 1ce70b5c1..366f456d5 100644 --- a/cli/README.md +++ b/cli/README.md @@ -121,6 +121,14 @@ cd /usr/share/FreshRSS ```sh cd /usr/share/FreshRSS +./cli/db-backup.php +# Back-up all users respective database to `data/users/*/backup.sqlite` + +./cli/db-restore.php --delete-backup --force-overwrite +# Restore all users respective database from `data/users/*/backup.sqlite` +# --delete-backup: delete `data/users/*/backup.sqlite` after successful import +# --force-overwrite: will clear the users respective database before import + ./cli/db-optimize.php --user username # Optimize database (reduces the size) for a given user (perform `OPTIMIZE TABLE` in MySQL, `VACUUM` in SQLite) ``` diff --git a/cli/db-backup.php b/cli/db-backup.php new file mode 100755 index 000000000..290b5cc7b --- /dev/null +++ b/cli/db-backup.php @@ -0,0 +1,20 @@ +#!/usr/bin/env php +db['type'] ?? ''); +$ok = true; + +foreach (listUsers() as $username) { + $username = cliInitUser($username); + $filename = DATA_PATH . '/users/' . $username . '/backup.sqlite'; + @unlink($filename); + + echo 'FreshRSS backup database to SQLite for user “', $username, "”…\n"; + + $databaseDAO = FreshRSS_Factory::createDatabaseDAO($username); + $ok &= $databaseDAO->dbCopy($filename, FreshRSS_DatabaseDAO::SQLITE_EXPORT); +} + +done((bool)$ok); diff --git a/cli/db-restore.php b/cli/db-restore.php new file mode 100755 index 000000000..6ea6f4a7d --- /dev/null +++ b/cli/db-restore.php @@ -0,0 +1,65 @@ +#!/usr/bin/env php +db['type'] ?? ''); + +$cliOptions = new class extends CliOptionsParser { + public string $deleteBackup; + public string $forceOverwrite; + + public function __construct() { + $this->addOption('deleteBackup', (new CliOption('delete-backup'))->withValueNone()); + $this->addOption('forceOverwrite', (new CliOption('force-overwrite'))->withValueNone()); + parent::__construct(); + } +}; + +if (!empty($cliOptions->errors)) { + fail('FreshRSS error: ' . array_shift($cliOptions->errors) . "\n" . $cliOptions->usage); +} + +FreshRSS_Context::initSystem(true); +Minz_User::change(Minz_User::INTERNAL_USER); +$ok = false; +try { + $error = initDb(); + if ($error != '') { + $_SESSION['bd_error'] = $error; + } else { + $ok = true; + } +} catch (Exception $ex) { + $_SESSION['bd_error'] = $ex->getMessage(); +} +if (!$ok) { + fail('FreshRSS database error: ' . (empty($_SESSION['bd_error']) ? 'Unknown error' : $_SESSION['bd_error'])); +} + +foreach (listUsers() as $username) { + $username = cliInitUser($username); + $filename = DATA_PATH . "/users/{$username}/backup.sqlite"; + if (!file_exists($filename)) { + fwrite(STDERR, "FreshRSS SQLite backup not found for user “{$username}”!\n"); + $ok = false; + continue; + } + + echo 'FreshRSS restore database from SQLite for user “', $username, "”…\n"; + + $databaseDAO = FreshRSS_Factory::createDatabaseDAO($username); + $clearFirst = isset($cliOptions->forceOverwrite); + $ok &= $databaseDAO->dbCopy($filename, FreshRSS_DatabaseDAO::SQLITE_IMPORT, $clearFirst); + if ($ok) { + if (isset($cliOptions->deleteBackup)) { + unlink($filename); + } + } else { + fwrite(STDERR, "FreshRSS database already exists for user “{$username}”!\n"); + fwrite(STDERR, "If you would like to clear the user database first, use the option --force-overwrite\n"); + } + invalidateHttpCache($username); +} + +done((bool)$ok); diff --git a/docs/en/admins/05_Backup.md b/docs/en/admins/05_Backup.md index 061300a37..d724050e1 100644 --- a/docs/en/admins/05_Backup.md +++ b/docs/en/admins/05_Backup.md @@ -10,9 +10,19 @@ Do this before an upgrade. This following tutorial demonstrates commands for backing up FreshRSS. It assumes that your main FreshRSS directory is `/usr/share/FreshRSS`. If you’ve installed it somewhere else, substitute your path as necessary. +### Creating a database backup + +Back-up all users respective database to `data/users/*/backup.sqlite` + +```sh +cd /usr/share/FreshRSS/ +./cli/db-backup.php +``` + ### Creating a Backup of all Files -First, Enter the directory you wish to save your backup to. Here, for example, we’ll save the backup to the user home directory +Enter the directory you wish to save your backup to. +Here, for example, we’ll save the backup to the user home directory ```sh cd ~ @@ -52,7 +62,39 @@ And optionally, as cleanup, remove the copy of your backup from the FreshRSS dir rm FreshRSS-backup.tgz ``` -## Backing up Feeds +### Restore a database backup + +> ℹ️ It is safer to stop your Web server and cron during maintenance operations. + +Restore all users respective database from `data/users/*/backup.sqlite` + +```sh +cd /usr/share/FreshRSS/ +./cli/db-restore.php --delete-backup --force-overwrite +``` + +## Migrate database + +Start by making an automatic backup of the all the user databases to SQLite files: + +```sh +cd /usr/share/FreshRSS/ +./cli/db-backup.php +``` + +Change your database setup: +- if you like to change database type (e.g. from MySQL to PostgreSQL), edit `data/config.php` accordingly. +- if you upgrade to a major PostgreSQL version, after a PostgreSQL backup, you may delete the old instance and start a new instance (remove the PostgreSQL volume if using Docker). + +Restore all the user databases from the SQLite files: + +```sh +./cli/db-restore.php --delete-backup --force-overwrite +``` + +See also our [Docker documentation to migrate database](https://github.com/FreshRSS/FreshRSS/blob/edge/Docker/README.md#migrate-database). + +## Backing up selected content ### Feed list Export diff --git a/docs/en/admins/Caddy.md b/docs/en/admins/Caddy.md index f3c2be96b..9f19113d4 100644 --- a/docs/en/admins/Caddy.md +++ b/docs/en/admins/Caddy.md @@ -49,7 +49,7 @@ To set up FreshRSS behind a reverse proxy with Caddy and using a subfolder, foll Restart FreshRSS to ensure that it recognizes the new base URL: ```bash - docker-compose restart freshrss + docker compose restart freshrss ``` 4. **Access FreshRSS:**