Ceph Storage Cluster - Revision history

DrEdWilliams: /* Object Gateway */

2021-05-08T20:43:23Z

Object Gateway

← Older revision		Revision as of 20:43, 8 May 2021
Line 225:		Line 225:
	* https://docs.ceph.com/en/latest/cephadm/rgw/		* https://docs.ceph.com/en/latest/cephadm/rgw/
	* https://docs.ceph.com/en/latest/radosgw/		* https://docs.ceph.com/en/latest/radosgw/

	~~'''''NOTE: the Pacific release (16.2.x) seems to have significantly changed how RGW is done ... need to figure it out ... the rest of this section is suspect and not likely to work ...'''''~~

	Obviously, a working ceph cluster is required; in fact, the cluster must be 'healthy' ... or it won't even start the RADOS gateway.		Obviously, a working ceph cluster is required; in fact, the cluster must be 'healthy' ... or it won't even start the RADOS gateway.

DrEdWilliams: /* Object Gateway */

2021-05-08T20:42:58Z

Object Gateway

← Older revision		Revision as of 20:42, 8 May 2021
Line 236:		Line 236:

	The label can be anything, but 'rgw' does make sense ... repeat this for all hosts that you want to run gateways. Now you can actually start the rgw service:		The label can be anything, but 'rgw' does make sense ... repeat this for all hosts that you want to run gateways. Now you can actually start the rgw service:
	ceph orch apply rgw <~~realm> <zone~~> --port=<port> --placement=label:rgw		ceph orch apply rgw <servicename> --port=<port> --placement=label:rgw

	For the test cluster, this looked like:		For the test cluster, this looked like:
	ceph orch apply rgw ~~williamsnet~~ test --port=8088 --placement=label:rgw		ceph orch apply rgw test-rgw --port=8088 --placement=label:rgw

	If the rgw service(s) do not start, you may need to force the issue ...		If the rgw service(s) do not start, you may need to force the issue ...
	ceph orch daemon add rgw ~~williamsnet~~ test port:8088 <host>		ceph orch daemon add rgw test-rgw port:8088 <host>

	Note the slightly different format of the parameters for port and placement.		Note the slightly different format of the parameters for port and placement.

DrEdWilliams: /* Object Gateway */

2021-05-08T18:26:38Z

Object Gateway

← Older revision		Revision as of 18:26, 8 May 2021
Line 225:		Line 225:
	* https://docs.ceph.com/en/latest/cephadm/rgw/		* https://docs.ceph.com/en/latest/cephadm/rgw/
	* https://docs.ceph.com/en/latest/radosgw/		* https://docs.ceph.com/en/latest/radosgw/

			'''''NOTE: the Pacific release (16.2.x) seems to have significantly changed how RGW is done ... need to figure it out ... the rest of this section is suspect and not likely to work ...'''''

	Obviously, a working ceph cluster is required; in fact, the cluster must be 'healthy' ... or it won't even start the RADOS gateway.		Obviously, a working ceph cluster is required; in fact, the cluster must be 'healthy' ... or it won't even start the RADOS gateway.

DrEdWilliams: /* Adding Disks */

2021-05-05T02:29:43Z

Adding Disks

← Older revision		Revision as of 02:29, 5 May 2021
Line 123:		Line 123:
	wipefs -a /dev/<drive>		wipefs -a /dev/<drive>

	~~Especially if~~ a drive has been used in a previous incarnation of ceph on that machine (i.e. you tore down a cluster and are trying to build it up again), you ~~still may not be able~~ to ~~add the disk~~. This is especially the case when using the [[Rook Storage for Kubernetes\|Rook]] operator to build an internal cluster. It does more than just check for the magic numbers on the disks, it checks the OS to see what it thinks of the drive. ~~Without going into too many details,~~ ceph uses LVM to manage its disks, and LVM keeps its own record of its drives. You can wipe a disk, but LVM still thinks it owns it ... and therefore the Rook operator won't touch it. The ~~only solution is~~ to ~~wipe~~ the ~~disk and then reboot that node~~. ~~The Rook operator will then be happy~~.		If a drive has been used in a previous incarnation of ceph on that machine (i.e. you tore down a cluster and are trying to build it up again), you will need to do a bit more. This is especially the case when using the [[Rook Storage for Kubernetes\|Rook]] operator to build an internal cluster. It does more than just check for the magic numbers on the disks, it checks the OS to see what it thinks of the drive. Since ceph uses LVM to manage its disks, and LVM keeps its own record of its drives. You can wipe a disk, but LVM still thinks it owns it ... and therefore the Rook operator won't touch it. The process to make LVM forget about the drive is involved, but it works. See [[Removing a disk from LVM]] for the process.

	Once you have a clean drive, add it by issuing this command on the ceph master (not the system the drive is mounted on):		Once you have a clean drive, add it by issuing this command on the ceph master (not the system the drive is mounted on):

DrEdWilliams: /* Ceph Object Storage */

2021-05-01T12:21:54Z

Ceph Object Storage

← Older revision		Revision as of 12:21, 1 May 2021
Line 1:		Line 1:
	== Ceph Object Storage ==		== Ceph Object Storage ==
	As an alternative to a parallel file system (i.e. [https://beegfs.io BeeGFS], GPFS/Spectrum Scale, Lustre, etc.) which stores data as a collection of files, ceph is a system that stores data as a series of ~~immutable~~ objects. One well-known implementation of object storage is the Amazon S3 API; like ceph, it also provides additional layers of abstraction that provide alternative approaches to using the storage system:		As an alternative to a parallel file system (i.e. [https://beegfs.io BeeGFS], GPFS/Spectrum Scale, Lustre, etc.) which stores data as a collection of files, ceph is a system that stores data as a series of objects. One well-known implementation of object storage is the Amazon S3 API; like ceph, it also provides additional layers of abstraction that provide alternative approaches to using the storage system:
	* Raw Object Storage (think Amazon S3)		* Raw Object Storage (think Amazon S3)
	* Block Storage (think disk images)		* Block Storage (think disk images)

DrEdWilliams: /* Adding Disks */

2021-04-30T21:49:35Z

Adding Disks

← Older revision		Revision as of 21:49, 30 April 2021
Line 122:		Line 122:
	or		or
	wipefs -a /dev/<drive>		wipefs -a /dev/<drive>

			Especially if a drive has been used in a previous incarnation of ceph on that machine (i.e. you tore down a cluster and are trying to build it up again), you still may not be able to add the disk. This is especially the case when using the [[Rook Storage for Kubernetes\|Rook]] operator to build an internal cluster. It does more than just check for the magic numbers on the disks, it checks the OS to see what it thinks of the drive. Without going into too many details, ceph uses LVM to manage its disks, and LVM keeps its own record of its drives. You can wipe a disk, but LVM still thinks it owns it ... and therefore the Rook operator won't touch it. The only solution is to wipe the disk and then reboot that node. The Rook operator will then be happy.

	Once you have a clean drive, add it by issuing this command on the ceph master (not the system the drive is mounted on):		Once you have a clean drive, add it by issuing this command on the ceph master (not the system the drive is mounted on):

DrEdWilliams: /* Adding Disks */

2021-04-29T01:51:05Z

Adding Disks

← Older revision		Revision as of 01:51, 29 April 2021
Line 120:		Line 120:
	or		or
	dd if=/dev/zero count=10 of=/dev/<drive>		dd if=/dev/zero count=10 of=/dev/<drive>
			or
			wipefs -a /dev/<drive>

	Once you have a clean drive, add it by issuing this command on the ceph master (not the system the drive is mounted on):		Once you have a clean drive, add it by issuing this command on the ceph master (not the system the drive is mounted on):

DrEdWilliams: /* Adding Disks */

2021-04-28T01:16:46Z

Adding Disks

← Older revision		Revision as of 01:16, 28 April 2021
Line 118:		Line 118:
	If a disk has been used previously, you will need to zap it so that ceph will accept it.		If a disk has been used previously, you will need to zap it so that ceph will accept it.
	ceph orch device zap --force <host> <device>		ceph orch device zap --force <host> <device>
			or
			dd if=/dev/zero count=10 of=/dev/<drive>

	Once you have a clean drive, add it by issuing this command on the ceph master (not the system the drive is mounted on):		Once you have a clean drive, add it by issuing this command on the ceph master (not the system the drive is mounted on):

DrEdWilliams: /* Uninstalling Ceph */

2021-04-25T18:21:27Z

Uninstalling Ceph

← Older revision		Revision as of 18:21, 25 April 2021
Line 166:		Line 166:
	/etc/logrotate.d		/etc/logrotate.d

	In the ~~<pre>~~/etc/systemd/system~~</pre>~~ directory, make sure you get all the links out of the 'wants' directories as well as getting rid of the service files.		In the '''/etc/systemd/system''' directory, make sure you get all the links out of the 'wants' directories as well as getting rid of the service files.

	If you are removing ceph completely without intending to reinstall, you can remove the 'ceph' directories themselves, otherwise just clean out the contents.		If you are removing ceph completely without intending to reinstall, you can remove the 'ceph' directories themselves, otherwise just clean out the contents.

DrEdWilliams: /* Upgrading Ceph */

2021-04-25T18:20:47Z

Upgrading Ceph

@@ Line 152: / Line 152: @@
 == Upgrading Ceph ==
 Patch releases can be upgraded in-place with the ceph orch command -- see https://docs.ceph.com/en/latest/cephadm/upgrade/ for details
 == Cluster Concepts and Operations ==