Skip to content

Commit ecdffed

Browse files
authored
Document DB snapshot restore (#168)
1 parent 3d97320 commit ecdffed

1 file changed

Lines changed: 44 additions & 0 deletions

File tree

‎docs/developer-guide/100-system-administration.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,50 @@ mv -v data/old/* $PGDATA
268268
rm -r data/new data/old
269269
```
270270

271+
### Attaching DB snapshot to an EC2 instance
272+
For data recovery or testing purposes, it may be desirable to attach a DB snapshot to an existing instance. The following steps explain how to do this using an AWS EC2 EBS snapshot of the Docker volumes.
273+
274+
#### New instance
275+
A volume based on a snapshot can be easily created and attached to a new EC2 instance using the `Provision Host` GitHub action or the `provision_host.sh` bash script directly.
276+
277+
#### Rollback an existing instance using AWS SSM
278+
When using the SSM documents provided in `cloudformation-create-ssm-documents.yml` the rollback process can be completed automatically in just a few minutes. This reduces the number of manual steps required and is therefore less error-prone.
279+
To start the rollback process, select the `replace_volume` SSM document from the AWS SSM console. The document requires only a few parameters to be filled in:
280+
281+
- `SnapshotId` (The ID of the snapshot to restore from)
282+
- `VolumeId` (The ID of the EBS volume to replace with the snapshot)
283+
- `DeleteVolume` (Specifies whether the existing EBS volume should be deleted after replacement. Set to `true` to delete the volume or `false` to keep it.)
284+
285+
When executing the automation, AWS will perform several actions under the hood.
286+
287+
1. Retrieve the instance details for the specified `VolumeId`
288+
2. Create a new EBS volume from the snapshot specified in `SnapshotId`
289+
3. Detach the current EBS volume and attach the newly created volume.
290+
4. Execute the `attach_volume` SSM automation. This automation runs several commands directly on the EC2 instance to mount the volume and update the `fstab` file. The `tag` attached to the EBS volume is also updated to ensure that AWS Data Lifecycle Manager (DLM) targets the volume when creating daily and weekly backups.
291+
5. Based on the `DeleteVolume` parameter, the original EBS volume is either deleted or retained as backup.
292+
293+
#### Manually rollback an existing instance
294+
1. Create a new EBS volume from the EBS snapshot
295+
2. Attach the new EBS volume to the existing EC2 instance using the AWS console or the `aws ec2 attach-volume` command
296+
3. Check the EBS volume is attached as a disk using `lsblk` (assuming it is attached as `/dev/nvme2n1` for remaining steps)
297+
4. Mount the EBS volume `sudo mount -t xfs -o nouuid /dev/nvme2n1 /mnt/snapshot`
298+
5. Change PostgreSQL data volume permissions `sudo chown -R 70:70 /mnt/snapshot/or_postgresql-data/_data`
299+
6. Start a temporary PostgreSQL instance to connect to the snapshot DB. Locate the PostgreSQL data volume shown by `ls /mnt/snapshot` (generally called `or_postgresql-data`) and use that name in the `-v /mnt/snapshot/<YOUR_POSTGRES_VOLUME_NAME>/_data:/var/lib/postgresql/data` mount below.
300+
```shell
301+
sudo docker run -d \
302+
--rm \
303+
--name temp_recovery_db \
304+
-p 5433:5432 \
305+
-e POSTGRES_PASSWORD=postgres \
306+
-v /mnt/snapshot/<YOUR_POSTGRES_VOLUME_NAME>/_data:/var/lib/postgresql/data \
307+
-v /tmp:/export \
308+
openremote/postgresql:latest
309+
```
310+
7. Wait for the DB to be ready. When starting from a large snapshot, this can take a while; check the Docker container logs for the DB-ready message.
311+
8. Extract/Analyse the data as required either by:
312+
* Exec'ing into the container `docker exec -it temp_recovery_db psql -U postgres -d openremote`
313+
* Modifying the Manager and Keycloak DB settings to connect to the temporary DB on port 5433.
314+
271315
### Useful resources
272316
- [Shared memory](https://www.instaclustr.com/blog/postgresql-docker-and-shared-memory/#:~:text=Docker%20and%20SHM%2DSize&text=This%20means%20that%20instead%20of,default%2C%20this%20limit%20is%2064MB)
273317
- [Index maintenance](https://wiki.postgresql.org/wiki/Index_Maintenance)

0 commit comments

Comments
 (0)