Editing Ceph Storage Cluster (section)

=== Adding Disks ===
Drives are added individually as OSDs in ceph.  As mentioned above, ceph is rather particular about what it will accept for storage devices. 

An inventory of storage devices on all cluster hosts can be displayed with:

 ceph orch device ls

A storage device is considered available if all of the following conditions are met:
* The device must have no partitions.
* The device must not have any LVM state.
* The device must not be mounted.
* The device must not contain a file system.
* The device must not contain a Ceph BlueStore OSD.
* The device must be larger than 5 GB.

Ceph refuses to provision an OSD on a device that is not available. 

If a disk has been used previously, you will need to zap it so that ceph will accept it. 
 ceph orch device zap --force <host> <device>
or
 dd if=/dev/zero count=10 of=/dev/<drive>
or
 wipefs -a /dev/<drive>

If a drive has been used in a previous incarnation of ceph on that machine (i.e. you tore down a cluster and are trying to build it up again), you will need to do a bit more.  This is especially the case when using the [[Rook Storage for Kubernetes|Rook]] operator to build an internal cluster.  It does more than just check for the magic numbers on the disks, it checks the OS to see what it thinks of the drive.  Since ceph uses LVM to manage its disks, and LVM keeps its own record of its drives.  You can wipe a disk, but LVM still thinks it owns it ... and therefore the Rook operator won't touch it.  The process to make LVM forget about the drive is involved, but it works.  See [[Removing a disk from LVM]] for the process.

Once you have a clean drive, add it by issuing this command on the ceph master (not the system the drive is mounted on):
 ceph orch daemon add osd <host>:<device>

The host must have already been added in the previous section, and the cephadm takes care of installing the OSD software to manage it.  Also, if the host has been rebooted, you need to set the hostname to the base name (NOT the FQDN) before adding the drive/OSD.