When using the new ceph-volume scan and activate commands on ceph Nautilus after an upgrade from Luminous I was getting the following message 1root@ceph2 ~]# ceph-volume simple activate --all 2--> activating OSD specified in /etc/ceph/osd/37-11af5440-dadf-40e3-8924-2bbad3ee5b58.json 3Running command: /bin/ln -snf …

Read More

Original article here Ceph’s default IO priority and class for behind the scene disk operations should be considered required vs best efforts. For those of us who actually utilize our storage for services that require performance will quickly find that deep scrub grinds even the most powerful systems to a halt. Below …

Read More

1ceph osd pool ls | while read line; do echo $line && ceph osd pool get $line crush_rule; done

Read More

Seems like something that should be simple! Step1 – Create the empty image 1glance image-create --container-format=bare --disk-format=raw --min-ram 2048 --name="Windows Server test" 1+------------------+--------------------------------------+ 2| Property | Value | …

Read More

Get status of monitors ceph quorum_status --format json-pretty Set the weight of a disk lower so it gets less IO, or use it to scale a disk in\out of a cluster ceph osd reweight osd.30 .01 --cluster mycluster Move an OSD to a node in the cursh map, usefull if your OSD comes up not attached to a node ceph osd crush move …

Read More

1ceph pg dump | awk ' 2BEGIN { IGNORECASE = 1 } 3/^PG_STAT/ { col=1; while($col!="UP") {col++}; col++ } 4/^[0-9a-f]+\\.[0-9a-f]+/ { match($0,/^[0-9a-f]+/); pool=substr($0, RSTART, RLENGTH); poollist[pool]=0; 5up=$col; i=0; RSTART=0; RLENGTH=0; delete osds; while(match(up,/[0-9]+/)>0) { …

Read More

Find the OSD in the cluster that's bringing the whole party down. 1grep 'slow request' /var/log/ceph/ceph.log | awk '{print $3}' | sort | uniq -c | sort -n

Read More

Just a reminder folks, when you bring online a new Ceph node with a bunch of new(Or existing) OSD's you might see it blocking IO's for some reason. This might show as "Requests are blocked > 32 sec" or simply as the VM's accessing the cluster are stalling and or reporting their own IO issues. FIREWALL!! …

Read More

Lots of talk but not a lot of info online. This worked for me I'm enabling replication t pool level here. Not only images that have the 'journaling' feature enabled will be mirrored irrespective of the state of your pool. So to clarify, you need to enable replication on the pool AND enable the journaling feature on the …

Read More

Check replication See how much journal data is yet to be replicated, where 'sas' is the pool containing the journal data 1rados ls -p sas | grep journal | wc -l Enable mirroring for an image Step 1 - Enable journaling feature for the image(You can place the journal data on a separate pool) 1rbd feature enable …

Read More