The current setup (with only two hosts with OSDs) can't support three replicas using the default replication rule, which separates copies so they're on different hosts.

The temporary "single host" crush rule would allow all three copies of an object to be on the same host, which is not okay either.

Inspired by this ceph-users thread: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-April/000599.html, we created a custom rule which:

• allows 3 replicas
• makes sure the 3 replicas are split across 2 hosts

Dump and decompile the current crushmap

sudo ceph osd getcrushmap -o crushmap.orig
crushtool -d crushmap.orig -o crushmap.orig.plain

Create the new crushmap

cp crushmap.orig.plain crushmap.new.plain
edit crushmap.new.plain

Insert the new rule

rule two_hosts_rule {
	id 2
	type replicated
	min_size 1
	max_size 10
	step take default
	step choose firstn 2 type host
	step chooseleaf firstn 2 type osd
	step emit
}

Compile and load the new crushmap

crushtool -c crushmap.new.plain -o crushmap.new
sudo ceph osd setcrushmap -i crushmap.new

Now that we have three hosts, the following command has been run to restore replication across all three hosts:

$ sudo ceph osd pool set proxmox crush_rule replicated_rule
set pool 1 crush_rule to replicated_rule

(the data is being shuffled around to match the CRUSH rule)