Page MenuHomeSoftware Heritage

Update Ceph pool to use the default CRUSH replication rule
Closed, ResolvedPublic


Make sure the Ceph pool replicates data across hosts instead of only OSDs.

Event Timeline

olasd changed the status of subtask T2504: Add hypervisor3 ssds as Ceph OSDs from Open to Work in Progress.Jul 21 2020, 5:57 PM

The current setup (with only two hosts with OSDs) can't support three replicas using the default replication rule, which separates copies so they're on different hosts.

The temporary "single host" crush rule would allow all three copies of an object to be on the same host, which is not okay either.

Inspired by this ceph-users thread:, we created a custom rule which:

  • allows 3 replicas
  • makes sure the 3 replicas are split across 2 hosts

Dump and decompile the current crushmap

sudo ceph osd getcrushmap -o crushmap.orig
crushtool -d crushmap.orig -o crushmap.orig.plain

Create the new crushmap

cp crushmap.orig.plain

Insert the new rule

rule two_hosts_rule {
	id 2
	type replicated
	min_size 1
	max_size 10
	step take default
	step choose firstn 2 type host
	step chooseleaf firstn 2 type osd
	step emit

Compile and load the new crushmap

crushtool -c -o
sudo ceph osd setcrushmap -i

Now that we have three hosts, the following command has been run to restore replication across all three hosts:

$ sudo ceph osd pool set proxmox crush_rule replicated_rule
set pool 1 crush_rule to replicated_rule

(the data is being shuffled around to match the CRUSH rule)