doc: add decommissioning guide

[elijahgreenstein]

This PR adds a decommissioning guide to comply with SSDLC requirements.

This guide is based off of the guide to LXD introduced in canonical/lxd#18507, but with several changes:

1. The structure is reorganized: this guide is largely organized around decommissioning an entire cluster. A paragraph at the beginning of the guide instructs user wishing to decommission a single cluster member to first remove the cluster, and provides cross references to relevant pages in the LXD documentation. This paragraph then directs users to proceed to destroy the data on the removed member after removing the cluster.
2. Deleting replicators and cluster links is moved into the "Delete data" section, in order to follow the note about projects and the need to use the --project flag in order to delete project-scoped entities.
3. microcloud commands are used wherever possible, in place of lxc commands.
4. References to MicroCeph logging is included.
5. Details about removing MicroCeph OSDs is included.
6. Instructions about removing storage buckets is removed from this guide, on the assumption that MicroCloud users would not use Ceph Object buckets.
7. Cluster removal includes details about removing the Ceph monmap when reducing the cluster down to one member.
8. Snap removal covers MicroCloud, LXD, MicroCeph, and MicroOVN.
9. A note about ACME services is included, alongside the note about OIDC providers, in the final section.

Checklist

• I have read the contributing guidelines and attest that all commits in this PR are signed off, cryptographically signed, and follow this project's commit structure.
• I have checked and added or updated relevant documentation.

[elijahgreenstein]

@mseralessandri , @roosterfish -- I added the section on MicroCeph OSDs based on my reading of the MicroCloud guide to removing cluster members, which included a reference to the MicroCeph guide to removing disks. However, would these steps be necessary when decommissioning an entire cluster?

[roosterfish]

Thanks, left a few comments.

[roosterfish]

This is not true. LXD blocks the pool deletion because the default profile has a disk device which references the pool. You can remove this device which will then allow you to remove the pool:

root@micro01:~# lxc storage rm remote
Error: The storage pool is currently in use
root@micro01:~# lxc profile show default
name: default
description: Default LXD profile
config: {}
devices:
  eth0:
    name: eth0
    network: default
    type: nic
  root:
    path: /
    pool: remote
    type: disk
used_by: []
project: default
root@micro01:~# lxc profile device rm default root
root@micro01:~# lxc storage rm remote

[elijahgreenstein]

Nice, it was bothering me to leave that profile and its associated storage pool untouched. I've updated the section with the instructions to remove the device.

Two notes:

1. I changed the command to lxc profile device remove (rather than rm), since that is what appears in the man pages.
2. It seems like the device is always named root. Is that correct? I've used that term in the instructions, but, if necessary, I can update the instructions to instruct users to find the device name.

[elijahgreenstein]

Another note: it seems as if the default profile is always associated with the default project. Is that correct? I've added a flag with the goal of making the command work regardless of the currently active project:

lxc profile show default --project default

[roosterfish]

Profiles belong to projects. When you create a new project, it also creates a new profile default. So it could happen that you have multiple projects, and each has their own default profile.

It seems like the device is always named root. Is that correct?

When it comes to MicroCloud, we configure the default profile in the default project to with two devices. One called eth0 (network) and the other called root. On top of this the user can ofc create additional entities we are not aware of when it comes to decommissioning.

[elijahgreenstein]

Thanks for the clarification. I've updated the guide as follows to address this by adding a subsection on removing disk devices from default profiles --- I am assuming here that the issue centers on disk devices. If there are other devices that would be of concern, let me know, but I currently understand the issue as:

• default profiles cannot be deleted
• default profiles may have disk devices that reference storage pools (and the default profile in the default project will definitely have that root device).
• Disk devices reference storage pools and possibly custom volumes (in the case of the root device, it does not have a value for source as would be the case in a custom volume attached to the profile)
• Hence: users must remove disk devices from the default profiles; if those reference custom volumes, the volumes will be deleted in the next section.

[elijahgreenstein]

Updated based on @roosterfish 's comments. I've also changed the title that appears in the navigation menu from "Decommission MicroCloud" to "Decommission a MicroCloud." Wondering what your thoughts on this are. I've also tried out "Decommission a cluster or cluster member."

[elijahgreenstein]

Made an additional update to direct users decommissioning a single cluster member to proceed to the "Remove the snaps" section after removing the member, rather than skipping all the way to "Destroy and sanitize data".

[elijahgreenstein]

Updated to address @roosterfish 's comments about the default profiles and the root device.

Also reorganized following a conversation with @mseralessandri to pull the "List projects" and the "Delete replicators and cluster links" subsections out of the "Delete data" section, and reorganize them above "Delete data".

[roosterfish]

LGTM thanks!

[mseralessandri]

LGTM, thanks!