Setup backups for auth server #42

Merged

gingermusketeer merged 5 commits from setup-auth-backups into main 2026-09-09 15:14:07 +00:00

Conversation 6 Commits 5 Files changed 10 +200 -3

gingermusketeer commented 2026-07-22 06:17:40 +00:00

Owner

Copy link

TODO

• Setup secrets

Changes

❯ uv run ansible-playbook --check --diff auth/playbooks/site.yaml

PLAY [Setup python] **************************************************************************************************************************************************

TASK [prerequisites : Install python] ********************************************************************************************************************************
skipping: [auth]

PLAY [Setup auto updates] ********************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************
ok: [auth]

TASK [updates : Install unattended-upgrades package] *****************************************************************************************************************
ok: [auth]

TASK [updates : Copy 50unattended-upgrades configuration file] *******************************************************************************************************
ok: [auth]

PLAY [Setup podman] **************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************
ok: [auth]

TASK [podman : Install container runtime] ****************************************************************************************************************************
ok: [auth]

TASK [podman : Install container runtime compose] ********************************************************************************************************************
ok: [auth]

TASK [podman : Login to Dockerhub] ***********************************************************************************************************************************
skipping: [auth]

PLAY [Deploy Auth] ***************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************
ok: [auth]

TASK [Ensure deploy directory] ***************************************************************************************************************************************
ok: [auth]

TASK [Ensure backup directory] ***************************************************************************************************************************************
--- before
+++ after
@@ -1,4 +1,4 @@
 {
     "path": "/opt/deploy/pgdumps",
-    "state": "absent"
+    "state": "directory"
 }

changed: [auth]

TASK [Deploy compose file] *******************************************************************************************************************************************
--- before: /opt/deploy/compose.yaml
+++ after: /Users/maxbrosnahan/Projects/datakollektivet/systemer/auth/compose.yaml
@@ -42,6 +42,7 @@
     image: docker.io/postgres:18.4
     volumes:
     - postgres_data:/var/lib/postgresql
+    - ./pgdumps:/pgdumps
     environment:
       POSTGRES_DB: keycloak
       POSTGRES_USER: keycloak

changed: [auth]

TASK [Populate .env file] ********************************************************************************************************************************************
ok: [auth]

TASK [Deploy nginx config] *******************************************************************************************************************************************
ok: [auth]

TASK [Pull images with Podman] ***************************************************************************************************************************************
skipping: [auth]

TASK [Expose podman socket] ******************************************************************************************************************************************
ok: [auth]

TASK [Package spi-auth.jar] ******************************************************************************************************************************************
skipping: [auth]

TASK [Deploy spi-auth.jar] *******************************************************************************************************************************************
ok: [auth]

TASK [Start application with Podman Compose] *************************************************************************************************************************
skipping: [auth]

TASK [Wait for Keycloak availability] ********************************************************************************************************************************
skipping: [auth]

PLAY [Setup Restic Backup for the database] **************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************
ok: [auth]

TASK [backup : Validate backup role inputs] **************************************************************************************************************************
ok: [auth] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [backup : Install restic] ***************************************************************************************************************************************
The following additional packages will be installed:
  fonts-font-awesome fonts-lato javascript-common libjs-jquery libjs-sphinxdoc
  libjs-underscore sphinx-rtd-theme-common
Suggested packages:
  apache2 | lighttpd | httpd
The following NEW packages will be installed:
  fonts-font-awesome fonts-lato javascript-common libjs-jquery libjs-sphinxdoc
  libjs-underscore restic sphinx-rtd-theme-common
0 upgraded, 8 newly installed, 0 to remove and 36 not upgraded.
changed: [auth]

TASK [backup : Create backup directory] ******************************************************************************************************************************
--- before
+++ after
@@ -1,4 +1,4 @@
 {
     "path": "/opt/backups",
-    "state": "absent"
+    "state": "directory"
 }

changed: [auth]

TASK [backup : Deploy backup secrets] ********************************************************************************************************************************
--- before
+++ after: /opt/backups/.env
@@ -0,0 +1,4 @@
+AWS_ACCESS_KEY_ID=REDACTED
+AWS_SECRET_ACCESS_KEY=REDACTED
+RESTIC_PASSWORD=REDACTED
+RESTIC_REPOSITORY=REDACTED

changed: [auth]

TASK [backup : Initialize restic repository] *************************************************************************************************************************
skipping: [auth]

TASK [backup : Deploy backup script] *********************************************************************************************************************************
--- before
+++ after: /Users/maxbrosnahan/.ansible/tmp/ansible-local-26304dni_45mj/tmp_3hjmgjc/backup.sh.j2
@@ -0,0 +1,23 @@
+#!/bin/bash
+set -euo pipefail
+
+set -a && source "/opt/backups/.env" && set +a
+
+echo "$(date '+%Y-%m-%d %H:%M:%S') - Starting backup"
+
+echo "$(date '+%Y-%m-%d %H:%M:%S') - [Pre-backup] podman-compose -f \"/opt/deploy/compose.yaml\" exec -T postgres pg_dump -O -U keycloak -Fc keycloak -f \"/pgdumps/keycloak.dump\""
+podman-compose -f "/opt/deploy/compose.yaml" exec -T postgres pg_dump -O -U keycloak -Fc keycloak -f "/pgdumps/keycloak.dump"
+
+echo "$(date '+%Y-%m-%d %H:%M:%S') - [Restic] Uploading backup"
+restic backup --verbose "/opt/deploy/pgdumps"
+# Keep 7 daily, 4 weekly, 6 monthly
+echo "$(date '+%Y-%m-%d %H:%M:%S') - [Restic] Pruning backups"
+restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
+
+# Weekly integrity check (Mondays)
+if [ "$(date +%u)" = "1" ]; then
+  echo "$(date '+%Y-%m-%d %H:%M:%S') - Running weekly integrity check"
+  restic check
+fi
+
+echo "$(date '+%Y-%m-%d %H:%M:%S') - Backup completed"
\ No newline at end of file

changed: [auth]

TASK [backup : Deploy restore script] ********************************************************************************************************************************
--- before
+++ after: /Users/maxbrosnahan/.ansible/tmp/ansible-local-26304dni_45mj/tmp20l1vfib/restore.sh.j2
@@ -0,0 +1,35 @@
+#!/bin/bash
+set -euo pipefail
+
+set -a && source "/opt/backups/.env" && set +a
+
+RESTORE_DIR="/opt/backups/restored"
+SNAPSHOT="${1:-latest}"
+
+echo "=== Restic Restore Script ==="
+echo "This script downloads backups to: $RESTORE_DIR"
+echo "It does NOT overwrite production data."
+echo ""
+
+if [ -d "$RESTORE_DIR" ]; then
+  echo "Removing previous restore at $RESTORE_DIR"
+  rm -rf "$RESTORE_DIR"
+fi
+
+mkdir -p "$RESTORE_DIR"
+
+echo "Restoring snapshot: $SNAPSHOT"
+echo ""
+
+restic restore "$SNAPSHOT" --target "$RESTORE_DIR"
+
+echo ""
+echo "=== Restore complete ==="
+echo ""
+echo "Files restored to: $RESTORE_DIR"
+echo ""
+echo "To inspect:"
+echo "  ls -la $RESTORE_DIR/opt/deploy/pgdumps"
+echo ""
+echo "To manually copy to production (CAREFUL!):"
+echo "  cp -r $RESTORE_DIR/opt/deploy/pgdumps /opt/deploy/pgdumps"

changed: [auth]

TASK [backup : Deploy systemd service] *******************************************************************************************************************************
--- before
+++ after: /Users/maxbrosnahan/.ansible/tmp/ansible-local-26304dni_45mj/tmp_vvd_hq2/backup.service.j2
@@ -0,0 +1,17 @@
+[Unit]
+Description=Restic backup
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/opt/backups/backup.sh
+Nice=10
+IOSchedulingClass=idle
+PrivateTmp=true
+ProtectSystem=strict
+ReadWritePaths=/opt/backups
+ProtectHome=true
+NoNewPrivileges=true
+TimeoutStartSec=3000
+OOMPolicy=stop
\ No newline at end of file

changed: [auth]

TASK [backup : Deploy systemd timer] *********************************************************************************************************************************
--- before
+++ after: /Users/maxbrosnahan/.ansible/tmp/ansible-local-26304dni_45mj/tmpl7fniqf5/backup.timer.j2
@@ -0,0 +1,10 @@
+[Unit]
+Description=Run restic backup daily at 04:00:00
+
+[Timer]
+OnCalendar=*-*-* 04:00:00
+RandomizedDelaySec=300
+Persistent=true
+
+[Install]
+WantedBy=timers.target

changed: [auth]

TASK [backup : Enable backup timer] **********************************************************************************************************************************
skipping: [auth]

PLAY RECAP ***********************************************************************************************************************************************************
auth                       : ok=23   changed=9    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0

## TODO - [x] Setup secrets ## Changes ```diff ❯ uv run ansible-playbook --check --diff auth/playbooks/site.yaml PLAY [Setup python] ************************************************************************************************************************************************** TASK [prerequisites : Install python] ******************************************************************************************************************************** skipping: [auth] PLAY [Setup auto updates] ******************************************************************************************************************************************** TASK [Gathering Facts] *********************************************************************************************************************************************** ok: [auth] TASK [updates : Install unattended-upgrades package] ***************************************************************************************************************** ok: [auth] TASK [updates : Copy 50unattended-upgrades configuration file] ******************************************************************************************************* ok: [auth] PLAY [Setup podman] ************************************************************************************************************************************************** TASK [Gathering Facts] *********************************************************************************************************************************************** ok: [auth] TASK [podman : Install container runtime] **************************************************************************************************************************** ok: [auth] TASK [podman : Install container runtime compose] ******************************************************************************************************************** ok: [auth] TASK [podman : Login to Dockerhub] *********************************************************************************************************************************** skipping: [auth] PLAY [Deploy Auth] *************************************************************************************************************************************************** TASK [Gathering Facts] *********************************************************************************************************************************************** ok: [auth] TASK [Ensure deploy directory] *************************************************************************************************************************************** ok: [auth] TASK [Ensure backup directory] *************************************************************************************************************************************** --- before +++ after @@ -1,4 +1,4 @@ { "path": "/opt/deploy/pgdumps", - "state": "absent" + "state": "directory" } changed: [auth] TASK [Deploy compose file] ******************************************************************************************************************************************* --- before: /opt/deploy/compose.yaml +++ after: /Users/maxbrosnahan/Projects/datakollektivet/systemer/auth/compose.yaml @@ -42,6 +42,7 @@ image: docker.io/postgres:18.4 volumes: - postgres_data:/var/lib/postgresql + - ./pgdumps:/pgdumps environment: POSTGRES_DB: keycloak POSTGRES_USER: keycloak changed: [auth] TASK [Populate .env file] ******************************************************************************************************************************************** ok: [auth] TASK [Deploy nginx config] ******************************************************************************************************************************************* ok: [auth] TASK [Pull images with Podman] *************************************************************************************************************************************** skipping: [auth] TASK [Expose podman socket] ****************************************************************************************************************************************** ok: [auth] TASK [Package spi-auth.jar] ****************************************************************************************************************************************** skipping: [auth] TASK [Deploy spi-auth.jar] ******************************************************************************************************************************************* ok: [auth] TASK [Start application with Podman Compose] ************************************************************************************************************************* skipping: [auth] TASK [Wait for Keycloak availability] ******************************************************************************************************************************** skipping: [auth] PLAY [Setup Restic Backup for the database] ************************************************************************************************************************** TASK [Gathering Facts] *********************************************************************************************************************************************** ok: [auth] TASK [backup : Validate backup role inputs] ************************************************************************************************************************** ok: [auth] => { "changed": false, "msg": "All assertions passed" } TASK [backup : Install restic] *************************************************************************************************************************************** The following additional packages will be installed: fonts-font-awesome fonts-lato javascript-common libjs-jquery libjs-sphinxdoc libjs-underscore sphinx-rtd-theme-common Suggested packages: apache2 | lighttpd | httpd The following NEW packages will be installed: fonts-font-awesome fonts-lato javascript-common libjs-jquery libjs-sphinxdoc libjs-underscore restic sphinx-rtd-theme-common 0 upgraded, 8 newly installed, 0 to remove and 36 not upgraded. changed: [auth] TASK [backup : Create backup directory] ****************************************************************************************************************************** --- before +++ after @@ -1,4 +1,4 @@ { "path": "/opt/backups", - "state": "absent" + "state": "directory" } changed: [auth] TASK [backup : Deploy backup secrets] ******************************************************************************************************************************** --- before +++ after: /opt/backups/.env @@ -0,0 +1,4 @@ +AWS_ACCESS_KEY_ID=REDACTED +AWS_SECRET_ACCESS_KEY=REDACTED +RESTIC_PASSWORD=REDACTED +RESTIC_REPOSITORY=REDACTED changed: [auth] TASK [backup : Initialize restic repository] ************************************************************************************************************************* skipping: [auth] TASK [backup : Deploy backup script] ********************************************************************************************************************************* --- before +++ after: /Users/maxbrosnahan/.ansible/tmp/ansible-local-26304dni_45mj/tmp_3hjmgjc/backup.sh.j2 @@ -0,0 +1,23 @@ +#!/bin/bash +set -euo pipefail + +set -a && source "/opt/backups/.env" && set +a + +echo "$(date '+%Y-%m-%d %H:%M:%S') - Starting backup" + +echo "$(date '+%Y-%m-%d %H:%M:%S') - [Pre-backup] podman-compose -f \"/opt/deploy/compose.yaml\" exec -T postgres pg_dump -O -U keycloak -Fc keycloak -f \"/pgdumps/keycloak.dump\"" +podman-compose -f "/opt/deploy/compose.yaml" exec -T postgres pg_dump -O -U keycloak -Fc keycloak -f "/pgdumps/keycloak.dump" + +echo "$(date '+%Y-%m-%d %H:%M:%S') - [Restic] Uploading backup" +restic backup --verbose "/opt/deploy/pgdumps" +# Keep 7 daily, 4 weekly, 6 monthly +echo "$(date '+%Y-%m-%d %H:%M:%S') - [Restic] Pruning backups" +restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune + +# Weekly integrity check (Mondays) +if [ "$(date +%u)" = "1" ]; then + echo "$(date '+%Y-%m-%d %H:%M:%S') - Running weekly integrity check" + restic check +fi + +echo "$(date '+%Y-%m-%d %H:%M:%S') - Backup completed" \ No newline at end of file changed: [auth] TASK [backup : Deploy restore script] ******************************************************************************************************************************** --- before +++ after: /Users/maxbrosnahan/.ansible/tmp/ansible-local-26304dni_45mj/tmp20l1vfib/restore.sh.j2 @@ -0,0 +1,35 @@ +#!/bin/bash +set -euo pipefail + +set -a && source "/opt/backups/.env" && set +a + +RESTORE_DIR="/opt/backups/restored" +SNAPSHOT="${1:-latest}" + +echo "=== Restic Restore Script ===" +echo "This script downloads backups to: $RESTORE_DIR" +echo "It does NOT overwrite production data." +echo "" + +if [ -d "$RESTORE_DIR" ]; then + echo "Removing previous restore at $RESTORE_DIR" + rm -rf "$RESTORE_DIR" +fi + +mkdir -p "$RESTORE_DIR" + +echo "Restoring snapshot: $SNAPSHOT" +echo "" + +restic restore "$SNAPSHOT" --target "$RESTORE_DIR" + +echo "" +echo "=== Restore complete ===" +echo "" +echo "Files restored to: $RESTORE_DIR" +echo "" +echo "To inspect:" +echo " ls -la $RESTORE_DIR/opt/deploy/pgdumps" +echo "" +echo "To manually copy to production (CAREFUL!):" +echo " cp -r $RESTORE_DIR/opt/deploy/pgdumps /opt/deploy/pgdumps" changed: [auth] TASK [backup : Deploy systemd service] ******************************************************************************************************************************* --- before +++ after: /Users/maxbrosnahan/.ansible/tmp/ansible-local-26304dni_45mj/tmp_vvd_hq2/backup.service.j2 @@ -0,0 +1,17 @@ +[Unit] +Description=Restic backup +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +ExecStart=/opt/backups/backup.sh +Nice=10 +IOSchedulingClass=idle +PrivateTmp=true +ProtectSystem=strict +ReadWritePaths=/opt/backups +ProtectHome=true +NoNewPrivileges=true +TimeoutStartSec=3000 +OOMPolicy=stop \ No newline at end of file changed: [auth] TASK [backup : Deploy systemd timer] ********************************************************************************************************************************* --- before +++ after: /Users/maxbrosnahan/.ansible/tmp/ansible-local-26304dni_45mj/tmpl7fniqf5/backup.timer.j2 @@ -0,0 +1,10 @@ +[Unit] +Description=Run restic backup daily at 04:00:00 + +[Timer] +OnCalendar=*-*-* 04:00:00 +RandomizedDelaySec=300 +Persistent=true + +[Install] +WantedBy=timers.target changed: [auth] TASK [backup : Enable backup timer] ********************************************************************************************************************************** skipping: [auth] PLAY RECAP *********************************************************************************************************************************************************** auth : ok=23 changed=9 unreachable=0 failed=0 skipped=8 rescued=0 ignored=0 ```

gingermusketeer added 2 commits 2026-07-22 06:17:40 +00:00

Add basic setup fe30938195

Mark backups as completed ... ad96d72863

Still need to be tested but this is a good start!

gingermusketeer referenced this pull request 2026-08-19 18:47:35 +00:00

setup basic forgejo #59

gingermusketeer force-pushed setup-auth-backups from ad96d72863 to 45abdc9c27 2026-08-25 16:29:25 +00:00 Compare

gingermusketeer added 1 commit 2026-08-25 16:31:56 +00:00

Add backup restore TODO 9db76f55f0

gingermusketeer changed title from Sett opp sikkerhetskopiering på autentiseringsserveren to Setup backups for auth server 2026-08-25 16:34:26 +00:00

luisa commented 2026-08-28 16:53:32 +00:00

Member

Copy link

Probably a dumb question, but why does the log you posted shows files from /Users/maxbrosnahan, especially some weird temp ones? Otherwise it looks good, I guess we will test it properly in the future :)

Probably a dumb question, but why does the log you posted shows files from `/Users/maxbrosnahan`, especially some weird temp ones? Otherwise it looks good, I guess we will test it properly in the future :)

gingermusketeer commented 2026-08-28 17:43:06 +00:00

Author

Owner

Copy link

@luisa I am not 100% sure what is going on here. The docs for this path doesn't have much to say. My guess is that in --check --diff mode it doesn't modify files on the host and instead does this on the local machine.

It could also be that it always generates templated files on the controller and then copies them over when applying the changes. As the files are new there is nothing on the destination machine to compare against hence the empty before path.

@luisa I am not 100% sure what is going on here. The [docs](https://docs.ansible.com/projects/ansible/latest/reference_appendices/config.html#default-local-tmp) for this path doesn't have much to say. My guess is that in `--check --diff` mode it doesn't modify files on the host and instead does this on the local machine. It could also be that it always generates templated files on the controller and then copies them over when applying the changes. As the files are new there is nothing on the destination machine to compare against hence the empty before path.

👍 1

luisa approved these changes 2026-08-28 18:00:37 +00:00

alexanrf commented 2026-08-30 10:29:01 +00:00

Owner

Copy link

Dette ser bra ut. Det ligner på approachen i Loomio. Men kan jeg foreslå at vi renamer rolen til "pg_backup" eller "postgres_backup" e.l. og ikke bare backup?

Jeg vet ikke om dette er verdt det nå, men om vi legger til "engine" = "docker" / "podman" så kan denne generaliseres til Loomio også.

Dette ser bra ut. Det ligner på approachen i Loomio. Men kan jeg foreslå at vi renamer rolen til "pg_backup" eller "postgres_backup" e.l. og ikke bare backup? Jeg vet ikke om dette er verdt det nå, men om vi legger til "engine" = "docker" / "podman" så kan denne generaliseres til Loomio også.

gingermusketeer commented 2026-08-30 12:06:16 +00:00

Author

Owner

Copy link

Men kan jeg foreslå at vi renamer rollen til "pg_backup" eller "postgres_backup" e.l. og ikke bare backup?

@alexanrf Rollen backup har ikke noe med PostgreSQL å gjøre. PostgreSQL backups blir håndtert av backup_pre_commands som står i site.yaml filen til auth.

Relevant del av site.yaml er:

      backup_paths:
        - /opt/deploy/pgdumps
      backup_pre_commands:
        - podman-compose -f "/opt/deploy/compose.yaml" exec -T postgres pg_dump -O -U keycloak -Fc keycloak -f "/pgdumps/keycloak.dump"

Jeg vet ikke om dette er verdt det nå, men om vi legger til "engine" = "docker" / "podman" så kan denne generaliseres til Loomio også.

Fordi backup_pre_commands står i site.yaml bør dette være støttet allerede.

>Men kan jeg foreslå at vi renamer rollen til "pg_backup" eller "postgres_backup" e.l. og ikke bare backup? @alexanrf Rollen `backup` har ikke noe med PostgreSQL å gjøre. PostgreSQL backups blir håndtert av `backup_pre_commands` som står i [site.yaml](https://forge.hornwitser.no/datakollektivet/systemer/src/commit/9db76f55f0bf83af92a518584c4337b1348ddd00/auth/playbooks/site.yaml#L38) filen til `auth`. Relevant del av `site.yaml` er: ```yaml backup_paths: - /opt/deploy/pgdumps backup_pre_commands: - podman-compose -f "/opt/deploy/compose.yaml" exec -T postgres pg_dump -O -U keycloak -Fc keycloak -f "/pgdumps/keycloak.dump" ``` >Jeg vet ikke om dette er verdt det nå, men om vi legger til "engine" = "docker" / "podman" så kan denne generaliseres til Loomio også. Fordi `backup_pre_commands` står i site.yaml bør dette være støttet allerede.

alexanrf commented 2026-08-30 12:49:49 +00:00

Owner

Copy link

Ah ha, elegant. Fikk ikke med meg den detaljen der. Bra greier, absolutt generaliserbart.

Ah ha, elegant. Fikk ikke med meg den detaljen der. Bra greier, absolutt generaliserbart.

gingermusketeer merged commit 16f848ba08 into main 2026-09-09 15:14:07 +00:00

gingermusketeer deleted branch setup-auth-backups 2026-09-09 15:14:07 +00:00

gingermusketeer referenced this pull request from a commit 2026-09-09 15:14:09 +00:00

Merge pull request 'Setup backups for auth server' (#42) from setup-auth-backups into main

Sign in to join this conversation.

Reviewers

No reviewers

luisa

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

datakollektivet/systemer!42

No description provided.