CLI database backup and restore (#6387)

* CLI database backup and restore Can also be used to migrate from one database to another (e.g. MySQL to PostgreSQL) or to ease upgrade to a major PostgreSQL version (e.g. 15 to 16). * +x * Fix some cases * Update to docker-compose-v2 * More documentation
2024-04-30 08:31:13 +02:00 · 2024-04-30 08:31:13 +02:00 · 329fd4bcf6
parent 173555795a
commit 329fd4bcf6
7 changed files with 219 additions and 21 deletions
--- a/Docker/README.md
+++ b/Docker/README.md
@ -21,7 +21,7 @@ Example for Linux Debian / Ubuntu:

 ```sh
 # Install default Docker Compose and automatically the corresponding version of Docker
-apt install docker-compose
+apt install docker-compose-v2
 ```

 ## Quick run
@ -194,6 +194,8 @@ docker run -d --restart unless-stopped --log-opt max-size=10m \

 In the FreshRSS setup, you will then specify the name of the container (`freshrss-db`) as the host for the database.

+See also the section [Docker Compose with PostgreSQL](#docker-compose-with-postgresql) below.
+
 ### [MySQL](https://hub.docker.com/_/mysql/) or [MariaDB](https://hub.docker.com/_/mariadb)

 ```sh
@ -285,13 +287,13 @@ See [`docker-compose.yml`](./freshrss/docker-compose.yml)
 ```sh
 cd ./FreshRSS/Docker/freshrss/
 # Update
-docker-compose pull
+docker compose pull
 # Run
-docker-compose -f docker-compose.yml -f docker-compose-local.yml up -d --remove-orphans
+docker compose -f docker-compose.yml -f docker-compose-local.yml up -d --remove-orphans
 # Logs
-docker-compose logs -f --timestamps
+docker compose logs -f --timestamps
 # Stop
-docker-compose down --remove-orphans
+docker compose down --remove-orphans
 ```

 Detailed (partial) example of Docker Compose for FreshRSS:
@ -378,13 +380,15 @@ See [`docker-compose-db.yml`](./freshrss/docker-compose-db.yml)
 ```sh
 cd ./FreshRSS/Docker/freshrss/
 # Update
-docker-compose -f docker-compose.yml -f docker-compose-db.yml pull
+docker compose -f docker-compose.yml -f docker-compose-db.yml pull
 # Run
-docker-compose -f docker-compose.yml -f docker-compose-db.yml -f docker-compose-local.yml up -d --remove-orphans
+docker compose -f docker-compose.yml -f docker-compose-db.yml -f docker-compose-local.yml up -d --remove-orphans
 # Logs
-docker-compose -f docker-compose.yml -f docker-compose-db.yml logs -f --timestamps
+docker compose -f docker-compose.yml -f docker-compose-db.yml logs -f --timestamps
 ```

+See also the section [Migrate database](#migrate-database) below to upgrade to a major PostgreSQL version with Docker Compose.
+
 ### Docker Compose for development

 Use the local (git) FreshRSS source code instead of the one inside the Docker container,
@ -396,11 +400,11 @@ See [`docker-compose-development.yml`](./freshrss/docker-compose-development.yml
 cd ./FreshRSS/Docker/freshrss/
 # Update
 git pull --ff-only --prune
-docker-compose pull
+docker compose pull
 # Run
-docker-compose -f docker-compose-development.yml -f docker-compose.yml -f docker-compose-local.yml up --remove-orphans
+docker compose -f docker-compose-development.yml -f docker-compose.yml -f docker-compose-local.yml up --remove-orphans
 # Stop with [Control]+[C] and purge
-docker-compose down --remove-orphans --volumes
+docker compose down --remove-orphans --volumes
 ```

 > ℹ️ You can combine it with `-f docker-compose-db.yml` to spin a PostgreSQL database.
@ -446,13 +450,13 @@ See [`docker-compose-proxy.yml`](./freshrss/docker-compose-proxy.yml)
 ```sh
 cd ./FreshRSS/Docker/freshrss/
 # Update
-docker-compose -f docker-compose.yml -f docker-compose-proxy.yml pull
+docker compose -f docker-compose.yml -f docker-compose-proxy.yml pull
 # Run
-docker-compose -f docker-compose.yml -f docker-compose-proxy.yml up -d --remove-orphans
+docker compose -f docker-compose.yml -f docker-compose-proxy.yml up -d --remove-orphans
 # Logs
-docker-compose -f docker-compose.yml -f docker-compose-proxy.yml logs -f --timestamps
+docker compose -f docker-compose.yml -f docker-compose-proxy.yml logs -f --timestamps
 # Stop
-docker-compose -f docker-compose.yml -f docker-compose-proxy.yml down --remove-orphans
+docker compose -f docker-compose.yml -f docker-compose-proxy.yml down --remove-orphans
 ```

 > ℹ️ You can combine it with `-f docker-compose-db.yml` to spin a PostgreSQL database.
@ -650,3 +654,46 @@ docker run -d --restart unless-stopped --log-opt max-size=10m \
  --name freshrss_cron freshrss/freshrss:alpine \
  crond -f -d 6
 ```
+
+## Migrate database
+
+Our [CLI](../cli/README.md) offers commands to back-up and migrate user databases,
+with `cli/db-backup.php` and `cli/db-restore.php` in particular.
+
+Here is an example (assuming our [Docker Compose example](#docker-compose-with-postgresql))
+intended for migrating to a newer major version of PostgreSQL,
+but which can also be used to migrate between other databases (e.g. MySQL to PostgreSQL).
+
+```sh
+# Stop FreshRSS container (Web server + cron) during maintenance
+docker compose down freshrss
+
+# Optional additional pre-upgrade back-up using PostgreSQL own mechanism
+docker compose -f docker-compose-db.yml \
+  exec freshrss-db pg_dump -U freshrss freshrss | gzip -9 > freshrss-postgres-backup.sql.gz
+# ------↑ Name of your PostgreSQL Docker container
+# -----------------------------↑ Name of your PostgreSQL user for FreshRSS
+# --------------------------------------↑ Name of your PostgreSQL database for FreshRSS
+
+# Back-up all users’ respective tables to SQLite files
+docker compose -f docker-compose.yml -f docker-compose-db.yml \
+  run --rm freshrss cli/db-backup.php
+
+# Remove old database (PostgreSQL) container and its data volume
+docker compose -f docker-compose-db.yml \
+  down --volumes freshrss-db
+
+# Edit your Compose file to use new database (e.g. newest postgres:xx)
+nano docker-compose-db.yml
+
+# Start new database (PostgreSQL) container and its new empty data volume
+docker compose -f docker-compose.yml -f docker-compose-db.yml \
+  up -d freshrss-db
+
+# Restore all users’ respective tables from SQLite files
+docker compose -f docker-compose.yml -f docker-compose-db.yml \
+  run --rm freshrss cli/db-restore.php --delete-backup
+
+# Restart a new FreshRSS container after maintenance
+docker compose -f docker-compose.yml -f docker-compose-db.yml up -d freshrss
+```
--- a/app/Models/DatabaseDAO.php
+++ b/app/Models/DatabaseDAO.php
@ -48,6 +48,18 @@ class FreshRSS_DatabaseDAO extends Minz_ModelPdo {
 		}
 	}

+	public function exits(): bool {
+		$sql = 'SELECT * FROM `_entry` LIMIT 1';
+		$stm = $this->pdo->query($sql);
+		if ($stm !== false) {
+			$res = $stm->fetchAll(PDO::FETCH_COLUMN, 0);
+			if ($res !== false) {
+				return true;
+			}
+		}
+		return false;
+	}
+
 	public function tablesAreCorrect(): bool {
 		$res = $this->fetchAssoc('SHOW TABLES');
 		if ($res == null) {
@ -242,6 +254,7 @@ SQL;
 		}
 		$error = '';

+		$databaseDAO = FreshRSS_Factory::createDatabaseDAO();
 		$userDAO = FreshRSS_Factory::createUserDao();
 		$catDAO = FreshRSS_Factory::createCategoryDao();
 		$feedDAO = FreshRSS_Factory::createFeedDao();
@ -259,15 +272,18 @@ SQL;
 					$error = 'Error: SQLite import file is not readable: ' . $filename;
 				} elseif ($clearFirst) {
 					$userDAO->deleteUser();
+					$userDAO = FreshRSS_Factory::createUserDao();
 					if ($this->pdo->dbType() === 'sqlite') {
 						//We cannot just delete the .sqlite file otherwise PDO gets buggy.
 						//SQLite is the only one with database-level optimization, instead of at table level.
 						$this->optimize();
 					}
 				} else {
-					$nbEntries = $entryDAO->countUnreadRead();
-					if (!empty($nbEntries['all'])) {
-						$error = 'Error: Destination database already contains some entries!';
+					if ($databaseDAO->exits()) {
+						$nbEntries = $entryDAO->countUnreadRead();
+						if (isset($nbEntries['all']) && $nbEntries['all'] > 0) {
+							$error = 'Error: Destination database already contains some entries!';
+						}
 					}
 				}
 				break;
--- a/cli/README.md
+++ b/cli/README.md
@ -121,6 +121,14 @@ cd /usr/share/FreshRSS
 ```sh
 cd /usr/share/FreshRSS

+./cli/db-backup.php
+# Back-up all users respective database to `data/users/*/backup.sqlite`
+
+./cli/db-restore.php --delete-backup --force-overwrite
+# Restore all users respective database from `data/users/*/backup.sqlite`
+# --delete-backup:	delete `data/users/*/backup.sqlite` after successful import
+# --force-overwrite:	will clear the users respective database before import
+
 ./cli/db-optimize.php --user username
 # Optimize database (reduces the size) for a given user (perform `OPTIMIZE TABLE` in MySQL, `VACUUM` in SQLite)
 ```
--- a/cli/db-backup.php
+++ b/cli/db-backup.php
@ -0,0 +1,20 @@
+#!/usr/bin/env php
+<?php
+declare(strict_types=1);
+require(__DIR__ . '/_cli.php');
+
+performRequirementCheck(FreshRSS_Context::systemConf()->db['type'] ?? '');
+$ok = true;
+
+foreach (listUsers() as $username) {
+	$username = cliInitUser($username);
+	$filename = DATA_PATH . '/users/' . $username . '/backup.sqlite';
+	@unlink($filename);
+
+	echo 'FreshRSS backup database to SQLite for user “', $username, "”…\n";
+
+	$databaseDAO = FreshRSS_Factory::createDatabaseDAO($username);
+	$ok &= $databaseDAO->dbCopy($filename, FreshRSS_DatabaseDAO::SQLITE_EXPORT);
+}
+
+done((bool)$ok);
--- a/cli/db-restore.php
+++ b/cli/db-restore.php
@ -0,0 +1,65 @@
+#!/usr/bin/env php
+<?php
+declare(strict_types=1);
+require(__DIR__ . '/_cli.php');
+
+performRequirementCheck(FreshRSS_Context::systemConf()->db['type'] ?? '');
+
+$cliOptions = new class extends CliOptionsParser {
+	public string $deleteBackup;
+	public string $forceOverwrite;
+
+	public function __construct() {
+		$this->addOption('deleteBackup', (new CliOption('delete-backup'))->withValueNone());
+		$this->addOption('forceOverwrite', (new CliOption('force-overwrite'))->withValueNone());
+		parent::__construct();
+	}
+};
+
+if (!empty($cliOptions->errors)) {
+	fail('FreshRSS error: ' . array_shift($cliOptions->errors) . "\n" . $cliOptions->usage);
+}
+
+FreshRSS_Context::initSystem(true);
+Minz_User::change(Minz_User::INTERNAL_USER);
+$ok = false;
+try {
+	$error = initDb();
+	if ($error != '') {
+		$_SESSION['bd_error'] = $error;
+	} else {
+		$ok = true;
+	}
+} catch (Exception $ex) {
+	$_SESSION['bd_error'] = $ex->getMessage();
+}
+if (!$ok) {
+	fail('FreshRSS database error: ' . (empty($_SESSION['bd_error']) ? 'Unknown error' : $_SESSION['bd_error']));
+}
+
+foreach (listUsers() as $username) {
+	$username = cliInitUser($username);
+	$filename = DATA_PATH . "/users/{$username}/backup.sqlite";
+	if (!file_exists($filename)) {
+		fwrite(STDERR, "FreshRSS SQLite backup not found for user “{$username}”!\n");
+		$ok = false;
+		continue;
+	}
+
+	echo 'FreshRSS restore database from SQLite for user “', $username, "”…\n";
+
+	$databaseDAO = FreshRSS_Factory::createDatabaseDAO($username);
+	$clearFirst = isset($cliOptions->forceOverwrite);
+	$ok &= $databaseDAO->dbCopy($filename, FreshRSS_DatabaseDAO::SQLITE_IMPORT, $clearFirst);
+	if ($ok) {
+		if (isset($cliOptions->deleteBackup)) {
+			unlink($filename);
+		}
+	} else {
+		fwrite(STDERR, "FreshRSS database already exists for user “{$username}”!\n");
+		fwrite(STDERR, "If you would like to clear the user database first, use the option --force-overwrite\n");
+	}
+	invalidateHttpCache($username);
+}
+
+done((bool)$ok);
--- a/docs/en/admins/05_Backup.md
+++ b/docs/en/admins/05_Backup.md
@ -10,9 +10,19 @@ Do this before an upgrade.

 This following tutorial demonstrates commands for backing up FreshRSS. It assumes that your main FreshRSS directory is `/usr/share/FreshRSS`. If you’ve installed it somewhere else, substitute your path as necessary.

+### Creating a database backup
+
+Back-up all users respective database to `data/users/*/backup.sqlite`
+
+```sh
+cd /usr/share/FreshRSS/
+./cli/db-backup.php
+```
+
 ### Creating a Backup of all Files

-First, Enter the directory you wish to save your backup to. Here, for example, we’ll save the backup to the user home directory
+Enter the directory you wish to save your backup to.
+Here, for example, we’ll save the backup to the user home directory

 ```sh
 cd ~
@ -52,7 +62,39 @@ And optionally, as cleanup, remove the copy of your backup from the FreshRSS dir
 rm FreshRSS-backup.tgz
 ```

-## Backing up Feeds
+### Restore a database backup
+
+> ℹ️ It is safer to stop your Web server and cron during maintenance operations.
+
+Restore all users respective database from `data/users/*/backup.sqlite`
+
+```sh
+cd /usr/share/FreshRSS/
+./cli/db-restore.php --delete-backup --force-overwrite
+```
+
+## Migrate database
+
+Start by making an automatic backup of the all the user databases to SQLite files:
+
+```sh
+cd /usr/share/FreshRSS/
+./cli/db-backup.php
+```
+
+Change your database setup:
+- if you like to change database type (e.g. from MySQL to PostgreSQL), edit `data/config.php` accordingly.
+- if you upgrade to a major PostgreSQL version, after a PostgreSQL backup, you may delete the old instance and start a new instance (remove the PostgreSQL volume if using Docker).
+
+Restore all the user databases from the SQLite files:
+
+```sh
+./cli/db-restore.php --delete-backup --force-overwrite
+```
+
+See also our [Docker documentation to migrate database](https://github.com/FreshRSS/FreshRSS/blob/edge/Docker/README.md#migrate-database).
+
+## Backing up selected content

 ### Feed list Export

--- a/docs/en/admins/Caddy.md
+++ b/docs/en/admins/Caddy.md
@ -49,7 +49,7 @@ To set up FreshRSS behind a reverse proxy with Caddy and using a subfolder, foll
    Restart FreshRSS to ensure that it recognizes the new base URL:

    ```bash
-    docker-compose restart freshrss
+    docker compose restart freshrss
    ```

 4. **Access FreshRSS:**