Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/.custom_wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ subfolders
subnet
subnets
SVG
systemd
Terraform
TinyPNG
TLS
Expand Down
16 changes: 13 additions & 3 deletions doc/explanation/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,12 @@ MicroCloud {ref}`further enforces security <exp-security-microcloud>` through th

MicroCloud runs on Ubuntu and benefits from all [Ubuntu platform security measures](https://ubuntu.com/security), including kernel hardening, signed packages, and continuous security maintenance. For production environments, we recommend using a recent Ubuntu LTS release to ensure long-term support and predictable security updates.

Ubuntu LTS releases subscribed to Ubuntu Pro can use the [Ubuntu Security Guide (USG)](https://documentation.ubuntu.com/security/compliance/usg/) for CIS hardening. Refer to the LXD documentation on {ref}`lxd:howto-security-harden-cis` for related details about auditing LXD hosts with USG.

(exp-security-snaps)=
## Snaps
## Snaps and supported versions

The MicroCloud team maintains both Long Term Support (LTS) and feature releases. See {ref}`ref-releases-snaps` and our {ref}`ref-release-notes` for details about the currently supported releases.

MicroCloud and its components are distributed as [snaps](https://snapcraft.io/docs), which enhances security by providing a confined environment with a streamlined update mechanism. Both LTS and feature channels receive regular security updates through Canonical’s official infrastructure.

Expand All @@ -24,15 +28,21 @@ All snaps are digitally signed using {ref}`assertions <snap:explanation-security
(exp-security-reporting)=
## Security reporting and disclosure

Report potential security issues privately through GitHub by [filing a security advisory](https://github.com/canonical/microcloud/security/advisories/new). Please include a clear description of the issue, affected MicroCloud versions, reproduction steps, and any known mitigation strategies.
MicroCloud adheres to the [Ubuntu disclosure policy](https://ubuntu.com/security/disclosure-policy). Report potential security issues privately through GitHub by [filing a security advisory](https://github.com/canonical/microcloud/security/advisories/new). Please include a clear description of the issue, affected MicroCloud versions, reproduction steps, and any known mitigation strategies. Refer to the [MicroCloud security policy](https://github.com/canonical/microcloud/blob/main/SECURITY.md) for details.

(exp-security-microcloud)=
## MicroCloud

### Cryptography

MicroCloud manages cluster membership and encrypted communication through mTLS and certificate-based identities. When a machine joins a cluster, it verifies the cluster’s certificate fingerprint and receives the complete set of member certificates, establishing a consistent trust store.

During the join process, MicroCloud uses an **explicit trust establishment mechanism** designed to prevent secret leakage and mitigate {spellexception}`man-in-the-middle` attacks. This mechanism uses a Hash-Based Message Authentication Code (HMAC) to sign the messages exchanged between the machine that initiates the join process and the joining peers. The shared secret used for joining is never transmitted over the network. The join process also enforces rate limits and session timeouts to reduce the risk of replay and brute-force attacks. For further information, refer to the [public specification](https://discourse.ubuntu.com/t/explicit-trust-establishment-mechanism-for-microcloud/44261).

### Logging

MicroCloud creates logs through systemd. These logs can be accessed with `sudo snap logs microcloud`.

(exp-security-lxd)=
## LXD

Expand All @@ -46,4 +56,4 @@ The {doc}`MicroCeph security documentation <microceph:explanation/security/secur
(exp-security-microovn)=
## MicroOVN

MicroOVN secures its network endpoints using the TLS protocol (version 1.2 or higher), along with P-384 elliptic curve keys. For details, refer to the {doc}`MicroOVN documentation on its use of cryptography <microovn:reference/cryptography>`. Also see the {doc}`MicroOVN security process documentation <microovn:reference/security>`.
MicroOVN secures its network endpoints using the TLS protocol (version 1.2 or higher), along with P-384 elliptic curve keys. For details, refer to the MicroOVN documentation on {doc}`cryptography <microovn:reference/cryptography>`, {doc}`working with TLS <microovn:how-to/tls>`, and the {doc}`MicroOVN security process <microovn:reference/security>`.
276 changes: 276 additions & 0 deletions doc/how-to/decommission.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,276 @@
---
myst:
html_meta:
description: Follow these steps to securely decommission a MicroCloud cluster member or cluster.
---

(howto-decommission)=
# How to securely decommission a MicroCloud deployment

```{important}
This process will erase all data associated with your MicroCloud deployment.
Make copies of any data that you need to preserve before proceeding.
Refer to {ref}`lxd:instances-backup` and {ref}`lxd:howto-storage-backup-volume` for relevant details.
```

This guide walks you through the steps to decommission an entire MicroCloud cluster.

If you only need to decommission a single cluster member, first {ref}`remove the member from the cluster <howto-member-remove>`.
After removing the member, {ref}`update the certificate <lxd:cluster-manage-update-certificate>` on the cluster remaining in production.
Then, return to this guide, skip ahead to {ref}`howto-decommission-remove-microcloud`, and follow the instructions in the sections that follow.

Some commands used to decommission MicroCloud are LXD or MicroCeph commands.
Refer to the {ref}`LXD decommissioning guide <lxd:howto-decommission>` and the {doc}`MicroCeph guide to removing disks <microceph:how-to/remove-disk>` for related information.

(howto-decommission-remove-offline-member)=
## Remove offline cluster members

Use the `--force` flag to {ref}`remove any offline cluster members <howto-member-remove>` (you will remove online cluster members later in the process):

```bash
sudo microcloud remove --force <offline_member_name>
```

(howto-decommission-revoke-remote)=
## Revoke remote access

List all identities that have access to LXD, then delete each identity:

```bash
lxc auth identity list
lxc auth identity delete <type>/<name_or_identifier>
```

(howto-decommission-list-projects)=
## List projects

Instances, profiles, and custom volumes are scoped by {ref}`project <lxd:projects>`.
For deployments with more than one project, you must repeat some steps for **each** project, each time using the `--project` flag.
You do not need to use the `--project` flag to decommission deployments with only one project.

Run this command to get a list of all projects:

```bash
lxc project list
```

````{note}
You can also delete a project (except the `default` project) and all of its project-level entities with:

```bash
lxc project delete <project_name> --force
```
````

(howto-decommission-delete-data)=
## Delete data

You can run the commands in this section on any online cluster member to delete data.

```{important}
Data deleted by LXD physically remains on disks and can be recovered by users with access to the disks.
To prevent unauthorized data recovery, you must {ref}`destroy and sanitize your data <howto-decommission-destroy-data>`.
```

(howto-decommission-delete-instances)=
### Stop and delete instances

For each project, stop all instances:

```bash
lxc stop --all --project <project_name>
```

Next, for each project, list all instances, then delete each instance:

```bash
lxc list --project <project_name>
lxc delete <instance_name> --project <project_name>
```

If you are unable to stop or delete an instance, use the `--force` flag:

```bash
lxc stop --force <instance_name> --project <project_name>
lxc delete --force <instance_name> --project <project_name>
```

(howto-decommission-delete-profiles)=
### Delete profiles

For each project, list all profiles:

```bash
lxc profile list --project <project_name>
```

Each project has a `default` profile that cannot be deleted.
Delete all other profiles:

```bash
lxc profile delete <profile_name> --project <project_name>
```

(howto-decommission-remove-disk-devices)=
### Remove disk devices from `default` profiles

You cannot delete a storage pool used by an instance, profile, or custom volume.
You must, therefore, remove any disk devices used by the `default` profiles in order to delete any storage pools or custom volumes referenced by those devices.

At a minimum, the `default` profile of the `default` project has a disk device named `root` that references a storage pool.
Remove this device with:

```bash
lxc profile device remove default root --project default
```

To check for additional disk devices, view information about the `default` profile of each project:

```bash
lxc profile show default --project <project_name>
```

Remove any remaining disk devices that reference storage pools:

```bash
lxc profile device remove default <device_name> --project <project_name>
```

(howto-decommission-delete-volumes)=
### Delete custom volumes

To delete {ref}`custom volumes <lxd:storage-volume-types>`, you must specify the storage pools used by the volumes.
First, list all storage pools across projects:

```bash
lxc storage list
```

Next, for each storage pool, list the custom volumes.
Use the `--all-projects` flag to view all custom volumes across projects:

```bash
lxc storage volume list <pool_name> type=custom --all-projects
```

Use the `PROJECT` column in the output to identify the project associated with each custom volume.
Then delete each custom volume, specifying both the storage pool and the project:

```bash
lxc storage volume delete <pool_name> <volume_name> --project <project_name>
```

(howto-decommission-delete-pools)=
### Delete storage pools

```{note}
Storage pools are not scoped by project, so you do not need to use the `--project` flag with `lxc storage` commands.
```

List all storage pools, then delete each one:

```bash
lxc storage list
lxc storage delete <pool_name>
```

(howto-decommission-delete-monitoring)=
### Delete monitoring data

Delete data from any external systems that you used to monitor {ref}`LXD events <lxd:logs_loki>`, {ref}`LXD metrics <lxd:metrics>`, or {doc}`Ceph logging <microceph:explanation/security/security-overview>`, such as [Loki](https://grafana.com/oss/loki/), [Prometheus](https://prometheus.io/), or [Grafana](https://grafana.com/).
Refer to the documentation for those systems for details.


(howto-decommission-remove-microceph-osds)=
## Remove MicroCeph OSDs

To {doc}`remove MicroCeph OSDs <microceph:how-to/remove-disk>`, list all disks, then remove each one:

```bash
microceph disk list
sudo microceph disk remove <osd_id>
```

Finally, verify that the OSDs have been removed:

```
microceph disk list
```

````{note}
If you are unable to remove an OSD, use the `--bypass-safety-checks` flag:

```bash
sudo microceph disk remove <osd_id> --bypass-safety-checks
```
````


(howto-decommission-remove-remaining-members)=
## Remove remaining cluster members

After deleting data, you can remove the online cluster members from the cluster.
First, list all cluster members:

```bash
microcloud cluster list
```

You can then {ref}`remove most cluster members <howto-member-remove>` with:

```bash
sudo microcloud remove <member_name>
```

However, before reducing the cluster from two members to one member, you must {ref}`clean up the Ceph monitor map <howto-member-remove-reduce-cluster>`.

```{note}
As you remove each member, you can run `microcloud status` on the remaining cluster members to verify the removal.
```

(howto-decommission-remove-microcloud)=
## Remove snaps

```{important}
Run these commands on **every** machine that you decommission.

Removing MicroCloud **does not** erase {ref}`ZFS pools (zpools) <lxd:storage-zfs>` or dedicated disks used by MicroCeph as Ceph object storage daemons (OSDs).
To securely decommission MicroCloud, you must {ref}`destroy and sanitize your data <howto-decommission-destroy-data>`.
```

Remove the MicroCloud, LXD, MicroCeph, and MicroOVN snaps.
Use the `--purge` flag, or a snapshot of your data will be preserved:

```bash
sudo snap remove microcloud --purge
sudo snap remove lxd --purge
sudo snap remove microceph --purge
sudo snap remove microovn --purge
```

```{note}
The MicroCeph and MicroOVN snaps may not be installed if you deployed a MicroCloud without those components.
```

Verify that the snaps and associated data were removed.
The following commands should report that none of these snaps are installed and that the `/var/snap/microcloud/`, `/var/snap/lxd/`, `/var/snap/microceph/`, and `/var/snap/microovn` directories do not exist:

```bash
snap list microcloud lxd microceph microovn
ls /var/snap/microcloud/ /var/snap/lxd/ /var/snap/microceph/ /var/snap/microovn/
```

(howto-decommission-destroy-data)=
## Destroy and sanitize data

Data deleted with MicroCloud, LXD, or MicroCeph commands remains readable and can be recovered by users with access to disks used in your deployment.
To prevent unauthorized recovery, you must physically overwrite the data.
Follow your data destruction policy to securely erase or destroy the disks that you are decommissioning.

If you are decommissioning an entire MicroCloud, apply your data destruction policy to any machines used to monitor events, logs, or metrics.
For clusters {ref}`configured with OIDC <lxd:authentication-openid>`, consult your OIDC identity provider for the steps to remove any data associated with your profile.
Likewise, if you used {ref}`ACME services to issue server certificates <lxd:authentication-server-certificate>`, refer to the service provider for the steps to remove any associated data.

```{important}
Sanitized data is irreversibly destroyed and cannot be recovered.
```
1 change: 1 addition & 0 deletions doc/how-to/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ Manage cluster members </how-to/members_manage>
Recover MicroCloud </how-to/recover>
Update and upgrade </how-to/update_upgrade>
Manage the snaps </how-to/snaps>
Decommission a MicroCloud </how-to/decommission>
```

## Engage with us
Expand Down
3 changes: 3 additions & 0 deletions doc/how-to/member_remove.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ You can remove a cluster member from a MicroCloud at any time, as long as at lea
sudo microcloud remove <name>
```

Run {command}`microcloud status` on any of the remaining cluster members to verify the removal.

Before removing the cluster member, ensure that there are no LXD instances, storage volumes, or MicroCeph OSDs located on it.

See {ref}`how to remove instances <lxd:instances-manage-delete>` in the LXD documentation.
Expand All @@ -31,6 +33,7 @@ If the machine is no longer reachable over the network, you can also add the `--
Removing a cluster member with `--force` will not attempt to perform any clean-up of the removed machine. All services will need to be fully re-installed before they can be re-initialized. Resources allocated to the MicroCloud like disks and network interfaces may need to be re-initialized as well.
```

(howto-member-remove-reduce-cluster)=
## Reducing the cluster to one member

When shrinking the cluster down to one member, you must also clean up the Ceph monitor map (`monmap`) before proceeding, even when using the `--force` flag.
Expand Down
Loading