Showing posts with label configuring openstack with ceph. Show all posts
Showing posts with label configuring openstack with ceph. Show all posts
Monday, December 23, 2013
Thursday, December 5, 2013
Ceph + OpenStack :: Part-5
OpenStack Instance boot from Ceph Volume
- For a list of images to choose from to create a bootable volume
[root@rdo /(keystone_admin)]# nova image-list
+--------------------------------------+-----------------------------+--------+--------+
| ID | Name | Status | Server |
+--------------------------------------+-----------------------------+--------+--------+
| f61edc8d-c9a1-4ff4-b4fc-c8128bd1a10b | Ubuntu 12.04 cloudimg amd64 | ACTIVE | |
| fcc07414-bbb3-4473-a8df-523664c8c9df | ceph-glance-image | ACTIVE | |
| be62a5bf-879f-4d1f-846c-fdef960224ff | precise-cloudimg.raw | ACTIVE | |
| 3c2db0ad-8d1e-400d-ba13-a506448f2a8e | precise-server-cloudimg | ACTIVE | |
+--------------------------------------+-----------------------------+--------+--------+
[root@rdo /(keystone_admin)]#
- To create a bootable volume from an image, include the image ID in the command: Before the volume builds, its bootable state is false.
[root@rdo qemu(keystone_admin)]# cinder create --image-id be62a5bf-879f-4d1f-846c-fdef960224ff --display-name my-boot-vol 10
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2013-12-05T13:34:38.296723 |
| display_description | None |
| display_name | my-boot-vol |
| id | 5fca6e1b-b494-4773-9c78-63f72703bfdf |
| image_id | be62a5bf-879f-4d1f-846c-fdef960224ff |
| metadata | {} |
| size | 10 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | None |
+---------------------+--------------------------------------+
[root@rdo qemu(keystone_admin)]#
[root@rdo qemu(keystone_admin)]# cinder list
+--------------------------------------+-------------+--------------+------+--------------+----------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-------------+--------------+------+--------------+----------+--------------------------------------+
| 0e2bfced-be6a-44ec-a3ca-22c771c66cdc | in-use | nova-vol_1 | 2 | None | false | 9d3c327f-1893-40ff-8a82-16fad9ce6d91 |
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available | ceph-vol1 | 5 | ceph-storage | false | |
| 5fca6e1b-b494-4773-9c78-63f72703bfdf | downloading | my-boot-vol | 10 | None | false | |
+--------------------------------------+-------------+--------------+------+--------------+----------+--------------------------------------+
- Wait for few minutes the bootable state turns to true. Copy the value in the ID field for your volume.
[root@rdo qemu(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+--------------+----------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+--------------+----------+--------------------------------------+
| 0e2bfced-be6a-44ec-a3ca-22c771c66cdc | in-use | nova-vol_1 | 2 | None | false | 9d3c327f-1893-40ff-8a82-16fad9ce6d91 |
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available | ceph-vol1 | 5 | ceph-storage | false | |
| 5fca6e1b-b494-4773-9c78-63f72703bfdf | available | my-boot-vol | 10 | None | true | |
+--------------------------------------+-----------+--------------+------+--------------+----------+--------------------------------------+
[root@rdo qemu(keystone_admin)]#
- Create a nova instance which will be boot from ceph volume
[root@rdo qemu(keystone_admin)]# nova boot --flavor 2 --image be62a5bf-879f-4d1f-846c-fdef960224ff --block_device_mapping vda=5fca6e1b-b494-4773-9c78-63f72703bfdf::0 --security_groups=default --nic net-id=4fe5909e-02db-4517-89f2-1278248fa26c myInstanceFromVolume
+--------------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------------+--------------------------------------+
| OS-EXT-STS:task_state | scheduling |
| image | precise-cloudimg.raw |
| OS-EXT-STS:vm_state | building |
| OS-EXT-SRV-ATTR:instance_name | instance-0000001e |
| OS-SRV-USG:launched_at | None |
| flavor | m1.small |
| id | f24a0b29-9f1e-444b-b895-c3c694f2f1bc |
| security_groups | [{u'name': u'default'}] |
| user_id | 99f8019ba2694d78a680a5de46aa1afd |
| OS-DCF:diskConfig | MANUAL |
| accessIPv4 | |
| accessIPv6 | |
| progress | 0 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive | |
| status | BUILD |
| updated | 2013-12-05T13:47:34Z |
| hostId | |
| OS-EXT-SRV-ATTR:host | None |
| OS-SRV-USG:terminated_at | None |
| key_name | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| name | myInstanceFromVolume |
| adminPass | qt34izQiLkG3 |
| tenant_id | 0dafe42cfde242ddbb67b681f59bdb00 |
| created | 2013-12-05T13:47:34Z |
| os-extended-volumes:volumes_attached | [] |
| metadata | {} |
+--------------------------------------+--------------------------------------+
[root@rdo qemu(keystone_admin)]#
[root@rdo qemu(keystone_admin)]#
[root@rdo qemu(keystone_admin)]#
[root@rdo qemu(keystone_admin)]# nova list
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| 0043a8be-60d1-43ed-ba43-1ccd0bba7559 | instance2 | SHUTOFF | None | Shutdown | public=172.24.4.228 |
| f24a0b29-9f1e-444b-b895-c3c694f2f1bc | myInstanceFromVolume | BUILD | spawning | NOSTATE | private=10.0.0.3 |
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 | small-ubuntu | ACTIVE | None | Running | public=172.24.4.230 |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
[root@rdo qemu(keystone_admin)]#
- Just in few minutes the instance starts RUNNING , time for a party now
[root@rdo qemu(keystone_admin)]# nova list
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| 0043a8be-60d1-43ed-ba43-1ccd0bba7559 | instance2 | SHUTOFF | None | Shutdown | public=172.24.4.228 |
| f24a0b29-9f1e-444b-b895-c3c694f2f1bc | myInstanceFromVolume | ACTIVE | None | Running | private=10.0.0.3 |
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 | small-ubuntu | ACTIVE | None | Running | public=172.24.4.230 |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
[root@rdo qemu(keystone_admin)]#
OpenStack Instance boot from Ceph Volume :: Troubleshooting
- During boot from volume , i encountered some errors after creating nova instance. The image was not able to get booted up from volume
[root@rdo nova(keystone_admin)]# nova boot --flavor 2 --image be62a5bf-879f-4d1f-846c-fdef960224ff --block_device_mapping vda=dd315dda-b22a-4cf8-8b77-7c2b2f163155:::0 --security_groups=default --nic net-id=4fe5909e-02db-4517-89f2-1278248fa26c myInstanceFromVolume
+--------------------------------------+----------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------+
| OS-EXT-STS:task_state | scheduling |
| image | precise-cloudimg.raw |
| OS-EXT-STS:vm_state | building |
| OS-EXT-SRV-ATTR:instance_name | instance-0000001d |
| OS-SRV-USG:launched_at | None |
| flavor | m1.small |
| id | f324e9b8-ec3a-4174-8b97-bf78dba62932 |
| security_groups | [{u'name': u'default'}] |
| user_id | 99f8019ba2694d78a680a5de46aa1afd |
| OS-DCF:diskConfig | MANUAL |
| accessIPv4 | |
| accessIPv6 | |
| progress | 0 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive | |
| status | BUILD |
| updated | 2013-12-05T12:42:22Z |
| hostId | |
| OS-EXT-SRV-ATTR:host | None |
| OS-SRV-USG:terminated_at | None |
| key_name | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| name | myInstanceFromVolume |
| adminPass | eish5pu56CiE |
| tenant_id | 0dafe42cfde242ddbb67b681f59bdb00 |
| created | 2013-12-05T12:42:21Z |
| os-extended-volumes:volumes_attached | [{u'id': u'dd315dda-b22a-4cf8-8b77-7c2b2f163155'}] |
| metadata | {} |
+--------------------------------------+----------------------------------------------------+
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]# nova list
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| 0043a8be-60d1-43ed-ba43-1ccd0bba7559 | instance2 | SHUTOFF | None | Shutdown | public=172.24.4.228 |
| f324e9b8-ec3a-4174-8b97-bf78dba62932 | myInstanceFromVolume | ERROR | None | NOSTATE | private=10.0.0.3 |
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 | small-ubuntu | ACTIVE | None | Running | public=172.24.4.230 |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
[root@rdo nova(keystone_admin)]#
- After checking up logs from /var/log/libvirt/qemu/instance-0000001d.log
qemu-kvm: -drive file=rbd:ceph-volumes/volume-dd315dda-b22a-4cf8-8b77-7c2b2f163155:id=volumes:key=AQC804xS8HzFJxAAD/zzQ8LMzq9wDLq/5a472g==:auth_supported=cephx\;none:mon_host=192.168.1.31\:6789\;192.168.1.33\:6789\;192.168.1.38\:6789,if=none,id=drive-virtio-disk0,format=raw,serial=dd315dda-b22a-4cf8-8b77-7c2b2f163155,cache=none: could not open disk image rbd:ceph-volumes/volume-dd315dda-b22a-4cf8-8b77-7c2b2f163155:id=volumes:key=AQC804xS8HzFJxAAD/zzQ8LMzq9wDLq/5a472g==:auth_supported=cephx\;none:mon_host=192.168.1.31\:6789\;192.168.1.33\:6789\;192.168.1.38\:6789: No such file or directory
2013-12-05 12:42:29.544+0000: shutting down
- Run qemu-img -h command to check for the supported format , here i found rbd format is not supported by qemu , so there is something fishy in this
Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2 qed vhdx parallels nbd blkdebug host_cdrom host_floppy host_device file gluster gluster gluster gluster
- Check the installed qemu version
[root@rdo qemu(keystone_admin)]# rpm -qa | grep -i qemu
qemu-img-0.12.1.2-2.415.el6_5.3.x86_64
qemu-guest-agent-0.12.1.2-2.415.el6_5.3.x86_64
gpxe-roms-qemu-0.9.7-6.10.el6.noarch
qemu-kvm-0.12.1.2-2.415.el6_5.3.x86_64
qemu-kvm-tools-0.12.1.2-2.415.el6_5.3.x86_64
[root@rdo qemu(keystone_admin)]#
- Have a look on previous post to see the installation of correct version of qemu . After this your nova instance should boot from volume
[root@rdo qemu(keystone_admin)]# rpm -qa | grep -i qemu
qemu-img-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64
qemu-guest-agent-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64
qemu-kvm-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64
gpxe-roms-qemu-0.9.7-6.10.el6.noarch
qemu-kvm-tools-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64
[root@rdo qemu(keystone_admin)]#
[root@rdo /(keystone_admin)]# qemu-img -h | grep -i rbd
Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2 qed parallels nbd blkdebug host_cdrom host_floppy host_device file rbd
[root@rdo /(keystone_admin)]#
[root@rdo qemu(keystone_admin)]# nova list
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| 0043a8be-60d1-43ed-ba43-1ccd0bba7559 | instance2 | SHUTOFF | None | Shutdown | public=172.24.4.228 |
| f24a0b29-9f1e-444b-b895-c3c694f2f1bc | myInstanceFromVolume | ACTIVE | None | Running | private=10.0.0.3 |
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 | small-ubuntu | ACTIVE | None | Running | public=172.24.4.230 |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
[root@rdo qemu(keystone_admin)]#
Ceph + OpenStack :: Part-4
Testing OpenStack Glance + RBD
- To allow glance to keep images on ceph RBD volume , edit /etc/glance/glance-api.conf
default_store = rbd
# ============ RBD Store Options =============================
# Ceph configuration file path
# If using cephx authentication, this file should
# include a reference to the right keyring
# in a client. section
rbd_store_ceph_conf = /etc/ceph/ceph.conf
# RADOS user to authenticate as (only applicable if using cephx)
rbd_store_user = images ## This is the ceph user that we have created above in this document
# RADOS pool in which images are stored
rbd_store_pool = ceph-images ## This is the ceph pool for images that we have created above in this document
# Images will be chunked into objects of this size (in megabytes).
# For best performance, this should be a power of two
rbd_store_chunk_size = 8
- Check ceph auth to make sure keys are present for client.images user . This should present here as we have created them earlier in this document.
[root@rdo ceph(keystone_admin)]# ceph auth list
installed auth entries:
mds.ceph-mon1
key: AQAxp35ScNUxOBAAfAXc+J5F3/v7jUrpztVRBQ==
caps: [mds] allow
caps: [mon] allow profile mds
caps: [osd] allow rwx
osd.0
key: AQCOvWpSsKN4JBAA015Uf53JjGCJS4cgzhxGFg==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.1
key: AQCn+mtSULePJxAACKvSkIqF39f5MaFiwsVR6Q==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.10
key: AQCjNIZSOF7AFxAA3vwLvgaB3PI+WAZPt2eIlQ==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.2
key: AQDHBmxSwKTZBxAAyWlQGj8H48sdPGl4PzlFbQ==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.3
key: AQBv/WtSwH5gOBAAHrSWblzq/n/qPbaurBMC2g==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.4
key: AQCiE2xSgDLQMRAAjWotlPtyqaSgpll1P6NTfw==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.5
key: AQCrFGxSOEnjMRAAnrqLcMR8UHu3rTTTQ5DHjw==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.6
key: AQAXFmxSUAmsJxAA83qr0mZ3sGLQbi+C59LXgw==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.7
key: AQBpFmxSOCZFNBAAONPg5I3QnB3Wd/pr7rSkEg==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.8
key: AQC7M4ZSSP9dMhAAh4HQ0uvKFs9yHiQrobXzUA==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.9
key: AQBmNIZSkAIjMRAA3FFGaMhGiPCmYmQ9REisRQ==
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQBSt2pS4M5cCBAAUd4jWA1vxJT+y5C9X6juzg==
caps: [mds] allow
caps: [mon] allow *
caps: [osd] allow *
client.bootstrap-mds
key: AQBSt2pS8IirKxAAQ27MWZ4pEEBuNhCDrj/FRw==
caps: [mon] allow profile bootstrap-mds
client.bootstrap-osd
key: AQBSt2pSYLXVGRAAYs0R8gXKSEct6ApEy4h6dQ==
caps: [mon] allow profile bootstrap-osd
client.images
key: AQDS04xSEJEYABAA8Kl9eEqIr3Y8pyz+tPRpvQ==
caps: [mon] allow r
caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=ceph-images
client.volumes
key: AQC804xS8HzFJxAAD/zzQ8LMzq9wDLq/5a472g==
caps: [mon] allow r
caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=ceph-volumes, allow rx pool=ceph-images
[root@rdo ceph(keystone_admin)]#
- Copy the keyrings file to glance directory. This is the same file that we have generated above in this document.
cp /etc/ceph/ceph.client.images.keyring /etc/glance
chown glance:glance /etc/glance/ceph.client.images.keyring
# service openstack-glance-api restart
# service openstack-glance-registry restart
# service openstack-glance-scrubber restart
- Before creating a new glance image on ceph volume , check the ceph pool content ( in my case its empty and it should be , this is for the first time we are using this volume )
[root@rdo init.d(keystone_admin)]# rbd -p ceph-images ls
rbd: pool ceph-images doesn't contain rbd images
[root@rdo init.d(keystone_admin)]#
- Download a new image or use if you have existing.
[root@rdo var(keystone_admin)]# wget http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img
[root@rdo var(keystone_admin)]# glance add name="ceph-glance-image" is_public=True disk_format=qcow2 container_format=ovf architecture=x86_64 < ubuntu-12.04.3-desktop-amd64.iso
Added new image with ID: fcc07414-bbb3-4473-a8df-523664c8c9df
[root@rdo var(keystone_admin)]# glance index
ID Name Disk Format Container Format Size
------------------------------------ ------------------------------ -------------------- -------------------- --------------
fcc07414-bbb3-4473-a8df-523664c8c9df ceph-glance-image qcow2 ovf 742391808
3c2db0ad-8d1e-400d-ba13-a506448f2a8e precise-server-cloudimg qcow2 ovf 254738432
f61edc8d-c9a1-4ff4-b4fc-c8128bd1a10b Ubuntu 12.04 cloudimg amd64 qcow2 ovf 254738432
[root@rdo var(keystone_admin)]#
- Now check your ceph pool , it will have image ( even match the glance image ID with pool objects , also compare image size with object size )
[root@rdo var(keystone_admin)]# rbd -p ceph-images ls
fcc07414-bbb3-4473-a8df-523664c8c9df
[root@rdo var(keystone_admin)]#
[root@rdo var(keystone_admin)]# du ubuntu-12.04.3-desktop-amd64.iso
724996 ubuntu-12.04.3-desktop-amd64.iso
[root@rdo var(keystone_admin)]#
[root@rdo var(keystone_admin)]# rados df
pool name category KB objects clones degraded unfound rd rd KB wr wr KB
ceph-images - 724993 92 0 0 0 63 50 98 724993
ceph-volumes - 1 9 0 0 0 284 212 72 8
data - 141557761 34563 0 0 0 71843 131424295 71384 146013188
metadata - 9667 23 0 0 0 72 19346 851 10102
rbd - 1 1 0 0 0 2117 21883 305 226753
total used 287309244 34688
total avail 6222206348
total space 6509515592
- Feeling Happy , you should be , now glance will use ceph to retrieve / store images
Please Follow Ceph + OpenStack :: Part-5 for next step in installation
Ceph + OpenStack :: Part-3
Testing OpenStack Cinder + RBD
- Creating a cinder volume provided by ceph backend
[root@rdo /]#
[root@rdo /]# cinder create --display-name cinder-ceph-vol1 --display-description "first cinder volume on ceph backend" 10
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2013-11-27T19:35:39.481075 |
| display_description | first cinder volume on ceph backend |
| display_name | cinder-ceph-vol1 |
| id | 10cc0855-652a-4a9b-baa1-80bc86dc12ac |
| metadata | {} |
| size | 10 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | None |
+---------------------+--------------------------------------+
[root@rdo /]#
[root@rdo /]#
[root@rdo /(keystone_admin)]# cinder list
+--------------------------------------+-----------+------------------+------+--------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------------------+------+--------------+----------+-------------+
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available | cinder-ceph-vol1 | 5 | ceph-storage | false | |
| 9671edaa-62c8-4f98-a36c-d6e59612141b | available | boot_from_volume | 20 | None | false | |
+--------------------------------------+-----------+------------------+------+--------------+----------+-------------+
[root@rdo /(keystone_admin)]#
[root@rdo /]#
[root@rdo /]# rados lspools
data
metadata
rbd
ceph-images
ceph-volumes
[root@rdo /]#
[root@rdo /]#
[root@rdo /]# rbd -p ceph-volumes ls
volume-10cc0855-652a-4a9b-baa1-80bc86dc12ac
[root@rdo /]#
- Attaching cinder volume to Instance
[root@rdo /(keystone_admin)]# nova list
+--------------------------------------+------------------+---------+--------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------------+---------+--------------+-------------+---------------------+
| 0043a8be-60d1-43ed-ba43-1ccd0bba7559 | instance2 | SHUTOFF | None | Shutdown | public=172.24.4.228 |
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 | small-ubuntu | ACTIVE | None | Running | public=172.24.4.230 |
| 10d1c49f-9fbc-455f-b72d-f731338b2dd5 | small-ubuntu-pwd | ACTIVE | powering-off | Shutdown | public=172.24.4.231 |
+--------------------------------------+------------------+---------+--------------+-------------+---------------------+
[root@rdo /(keystone_admin)]#
[root@rdo /(keystone_admin)]# nova show 9d3c327f-1893-40ff-8a82-16fad9ce6d91
+--------------------------------------+--------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+--------------------------------------------------------------------+
| status | ACTIVE |
| updated | 2013-12-03T15:58:31Z |
| OS-EXT-STS:task_state | None |
| OS-EXT-SRV-ATTR:host | rdo |
| key_name | RDO-admin |
| image | Ubuntu 12.04 cloudimg amd64 (f61edc8d-c9a1-4ff4-b4fc-c8128bd1a10b) |
| hostId | 4a74aa79a23a084f73f49a4fedba7447c132ab45c4701ed7fbbb2286 |
| OS-EXT-STS:vm_state | active |
| OS-EXT-SRV-ATTR:instance_name | instance-00000018 |
| public network | 172.24.4.230 |
| OS-SRV-USG:launched_at | 2013-12-03T08:55:46.000000 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | rdo |
| flavor | m1.small (2) |
| id | 9d3c327f-1893-40ff-8a82-16fad9ce6d91 |
| security_groups | [{u'name': u'default'}] |
| OS-SRV-USG:terminated_at | None |
| user_id | 99f8019ba2694d78a680a5de46aa1afd |
| name | small-ubuntu |
| created | 2013-12-03T08:55:39Z |
| tenant_id | 0dafe42cfde242ddbb67b681f59bdb00 |
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| os-extended-volumes:volumes_attached | [] |
| accessIPv4 | |
| accessIPv6 | |
| progress | 0 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive | |
+--------------------------------------+--------------------------------------------------------------------+
[root@rdo /(keystone_admin)]#
[root@rdo /(keystone_admin)]# virsh list
Id Name State
----------------------------------------------------
2 instance-00000018 running
[root@rdo /(keystone_admin)]# cat disk.xml
<disk type='network'>
<driver name="qemu" type="raw"/>
<source protocol="rbd" name="ceph-volumes/volume-10cc0855-652a-4a9b-baa1-80bc86dc12ac">
<host name='192.168.1.38' port='6789'/>
<host name='192.168.1.31' port='6789'/>
<host name='192.168.1.33' port='6789'/>
</source>
<target dev="vdf" bus="virtio"/>
<auth username='volumes'>
<secret type='ceph' uuid='801a42ec-aec1-3ea8-d869-823c2de56b83'/>
</auth>
</disk>
[root@rdo /(keystone_admin)]#
- Things you should know about this file
- source name=<ceph_pool_name/volume_name> ## ceph pool that we have created above and its cinder volume
- host name=<Your_Monitor_nodes>
- auth username=<user_you_created_in_Ceph_having_rights_to_pools_that_will_be_used_with_OS> ## we have created 2 users client.volumes and client.images for ceph that will have access to pools for openstack
- secret uuid=<secret_generated_by_virsh_secrec_define_command> ## refer above that we have generated.
- Attaching disk device to instance
[root@rdo /(keystone_admin)]# virsh attach-device instance-00000018 disk.xml
Device attached successfully
[root@rdo /(keystone_admin)]#
- Now the ceph volume is attached to your openstack instance , you can use this as a regular block disk.
Making integration more seamless
- To allow openstack create and attach ceph volumes using nova / cinder CLI as well as horizon dashboard , we need to add in /etc/nova/nova.conf the following values
rbd_user=volumes
rbd_secret_uuid=801a42ec-aec1-3ea8-d869-823c2de56b83
- After updating nova.conf , try creating volume from nova cli and attach to instance
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]# nova volume-create --display_name=nova-vol_1 2
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| status | creating |
| display_name | nova-vol_1 |
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2013-12-04T14:13:07.265831 |
| display_description | None |
| volume_type | None |
| snapshot_id | None |
| source_volid | None |
| size | 2 |
| id | 0e2bfced-be6a-44ec-a3ca-22c771c66cdc |
| metadata | {} |
+---------------------+--------------------------------------+
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]# nova volume-list
+--------------------------------------+-----------+------------------+------+--------------+-------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+-----------+------------------+------+--------------+-------------+
| 0e2bfced-be6a-44ec-a3ca-22c771c66cdc | available | nova-vol_1 | 2 | None | |
| 9671edaa-62c8-4f98-a36c-d6e59612141b | available | boot_from_volume | 20 | None | |
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available | ceph-vol1 | 5 | ceph-storage | |
+--------------------------------------+-----------+------------------+------+--------------+-------------+
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]# nova list
+--------------------------------------+------------------+---------+--------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------------+---------+--------------+-------------+---------------------+
| 0043a8be-60d1-43ed-ba43-1ccd0bba7559 | instance2 | SHUTOFF | None | Shutdown | public=172.24.4.228 |
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 | small-ubuntu | ACTIVE | None | Running | public=172.24.4.230 |
| 10d1c49f-9fbc-455f-b72d-f731338b2dd5 | small-ubuntu-pwd | ACTIVE | powering-off | Shutdown | public=172.24.4.231 |
+--------------------------------------+------------------+---------+--------------+-------------+---------------------+
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]# nova volume-attach 9d3c327f-1893-40ff-8a82-16fad9ce6d91 0e2bfced-be6a-44ec-a3ca-22c771c66cdc /dev/vdi
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdi |
| serverId | 9d3c327f-1893-40ff-8a82-16fad9ce6d91 |
| id | 0e2bfced-be6a-44ec-a3ca-22c771c66cdc |
| volumeId | 0e2bfced-be6a-44ec-a3ca-22c771c66cdc |
+----------+--------------------------------------+
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]# nova volume-list
+--------------------------------------+-----------+------------------+------+--------------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+-----------+------------------+------+--------------+--------------------------------------+
| 0e2bfced-be6a-44ec-a3ca-22c771c66cdc | in-use | nova-vol_1 | 2 | None | 9d3c327f-1893-40ff-8a82-16fad9ce6d91 |
| 9671edaa-62c8-4f98-a36c-d6e59612141b | available | boot_from_volume | 20 | None | |
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available | ceph-vol1 | 5 | ceph-storage | |
+--------------------------------------+-----------+------------------+------+--------------+--------------------------------------+
[root@rdo nova(keystone_admin)]#
Please Follow Ceph + OpenStack :: Part-4 for next step in installation
Ceph + OpenStack :: Part-2
Configuring OpenStack
Two parts of openstack integrates with Ceph’s block devices:
- Images: OpenStack Glance manages images for VMs.
- Volumes: Volumes are block devices. OpenStack uses volumes to boot VMs, or to attach volumes to running VMs. OpenStack manages volumes using Cinder services.
- Create pools for volumes and images:
ceph osd pool create volumes 128
ceph osd pool create images 128
- Configure OpenStack Ceph Client - The nodes running glance-api and cinder-volume act as Ceph clients. Each requires the ceph.conf file:
[root@ceph-mon1 ceph]# scp ceph.conf openstack:/etc/ceph
- Installing ceph client packages on openstack node
- First install Python bindings for librbd
yum install python-ceph
- Install ceph
[root@ceph-mon1 ceph]# ceph-deploy install openstack
- Setup Ceph Client Authentication for both pools along with keyrings
- Create a new user for Nova/Cinder and Glance.
ceph auth get-or-create client.volumes mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images'
ceph auth get-or-create client.images mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
- Add these keyrings to glance-api and cinder-volume nodes.
ceph auth get-or-create client.images | ssh openstack tee /etc/ceph/ceph.client.images.keyring
ssh openstack chown glance:glance /etc/ceph/ceph.client.images.keyring
ceph auth get-or-create client.volumes | ssh openstack tee /etc/ceph/ceph.client.volumes.keyring
ssh openstack chown cinder:cinder /etc/ceph/ceph.client.volumes.keyring
- Hosts running nova-compute do not need the keyring. Instead, they store the secret key in libvirt. To create libvirt secret key you will need key from client.volumes.key
ceph auth get-key client.volumes | ssh openstack tee client.volumes.key
- on the compute nodes, add the secret key to
libvirt create a secret.xml file
- on the compute nodes, add the secret key to
cat > secret.xml < <EOF
<secret ephemeral='no' private='no'>
<usage type='ceph'>
<name>client.volumes secret</name>
</usage>
EOF
- Generate secret from created secret.xml file , make a note of uuid of secret output
# virsh secret-define --file secret.xml
- Set libvirt secret using above key
# virsh secret-set-value --secret {uuid of secret} --base64 $(cat client.volumes.key) && rm client.volumes.key secret.xml
- Configure OpenStack-Glance to use CEPH
- Glance can use multiple back ends to store images. To use Ceph block devices by default, edit
/etc/glance/glance-api.confand add:
- Glance can use multiple back ends to store images. To use Ceph block devices by default, edit
default_store=rbd
rbd_store_user=images
rbd_store_pool=images
- If want to enable copy-on-write cloning of images into volumes, also add:
show_image_direct_url=True
- Configure OpenStack - Cinder to use CEPH
- OpenStack requires a driver to interact with Ceph block devices. You must specify the pool name for the block device. On your OpenStack node, edit
/etc/cinder/cinder.confby adding:
- OpenStack requires a driver to interact with Ceph block devices. You must specify the pool name for the block device. On your OpenStack node, edit
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_pool=volumes
glance_api_version=2
- If you’re using cephx authentication also configure the user and uuid of the secret you added to
libvirtearlier:
rbd_user=volumes
rbd_secret_uuid={uuid of secret}
- Restart Openstack
service glance-api restart
service nova-compute restart
service cinder-volume restart
- Once OpenStack is up and running, you should be able to create a volume with OpenStack on a Ceph block device.
- NOTE : Make sure /etc/ceph/ceph.conf file have sufficient rights to be ready by cinder and glance users.
Please Follow Ceph + OpenStack :: Part-3 for next step in installation
Ceph + OpenStack :: Part-1
Ceph & OpenStack Integration
libvirt, which configures the QEMU interface tolibrbd. To use Ceph Block Devices with openstack , we must install QEMU, libvirt, and openstack first. ( we will not cover openstack installation in this document , you can use your existing openstack infrastructure ) The following diagram explains openstack /Ceph technology stack.
Installing QEMU
[ceph-extras]
name=Ceph Extra Packages and Backports $basearch
baseurl=http://ceph.com/packages/ceph-extras/rpm/centos6/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
[ceph-extras-noarch]
name=Ceph Extra Packages and Backports noarch
baseurl=http://ceph.com/packages/ceph-extras/rpm/centos6/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
[ceph-extras-source]
name=Ceph Extra Packages and Backports Sources
baseurl=http://ceph.com/packages/ceph-extras/rpm/centos6/SRPMS
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
centos-extras
#yum update
#yum remove qemu-img
#yum --disablerepo=* --enablerepo=ceph-extras install -y qemu-img
# yum --disablerepo=* --enablerepo=ceph-extras install -y qemu-kvm
# yum --disablerepo=* --enablerepo=ceph-extras install -y qemu-guest-agent
# yum --disablerepo=* --enablerepo=ceph-extras install -y qemu-kvm-tools
--> Check creating a QEMU image it should work
[root@rdo yum.repos.d]# qemu-img create -f rbd rbd:data/foo 10G
Formatting 'rbd:data/foo', fmt=rbd size=10737418240 cluster_size=0
[root@rdo yum.repos.d]#
[root@rdo yum.repos.d]# qemu-img info -f rbd rbd:data/foo
image: rbd:data/foo
file format: rbd
virtual size: 10G (10737418240 bytes)
disk size: unavailable
cluster_size: 4194304
[root@rdo yum.repos.d]#
Installing LIBVIRT
libvirt with Ceph, we must have a running Ceph Storage Cluster, and have installed and configured QEMUyum install libvirt
Please Follow Ceph + OpenStack :: Part-2 for next step in installation
Ceph Installation :: Part-3
Creating Block Device from Ceph
- From monitor node , use ceph-deploy to install Ceph on your ceph-client1 node.
[root@ceph-mon1 ~]# ceph-deploy install ceph-client1
[ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy install ceph-client1
[ceph_deploy.install][DEBUG ] Installing stable version dumpling on cluster ceph hosts ceph-client1
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph-client1 ...
[ceph-client1][DEBUG ] connected to host: ceph-client1
[ceph-client1][DEBUG ] detect platform information from remote host
[ceph-client1][DEBUG ] detect machine type
[ceph_deploy.install][INFO ] Distro info: Ubuntu 13.04 raring
[ceph-client1][INFO ] installing ceph on ceph-client1
[ceph-client1][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive apt-get -q install --assume-yes ca-certificates
[ceph-client1][DEBUG ] Reading package lists...
[ceph-client1][DEBUG ] Building dependency tree...
[ceph-client1][DEBUG ] Reading state information...
[ceph-client1][DEBUG ] ca-certificates is already the newest version.
[ceph-client1][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 105 not upgraded.
[ceph-client1][INFO ] Running command: wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | apt-key add -
[ceph-client1][WARNIN] command returned non-zero exit status: 4
[ceph-client1][DEBUG ] add ceph deb repo to sources.list
[ceph-client1][INFO ] Running command: apt-get -q update
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security Release.gpg
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security Release
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring Release.gpg
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates Release.gpg
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring Release
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates Release
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/main Sources
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/main Sources
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/universe Sources
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/main amd64 Packages
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/universe Sources
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/universe amd64 Packages
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/main i386 Packages
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/universe i386 Packages
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/main amd64 Packages
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/main Translation-en
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/universe Translation-en
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/universe amd64 Packages
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/main Sources
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/universe Sources
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/main amd64 Packages
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/main i386 Packages
[ceph-client1][DEBUG ] Get:1 http://ceph.com raring Release.gpg [836 B]
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/universe amd64 Packages
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/universe i386 Packages
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/main i386 Packages
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/universe i386 Packages
[ceph-client1][DEBUG ] Hit http://ceph.com raring Release
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/main Translation-en
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/main Translation-en
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/universe Translation-en
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/universe Translation-en
[ceph-client1][DEBUG ] Hit http://ceph.com raring/main amd64 Packages
[ceph-client1][DEBUG ] Hit http://ceph.com raring/main i386 Packages
[ceph-client1][DEBUG ] Ign http://ceph.com raring/main Translation-en
[ceph-client1][DEBUG ] Fetched 836 B in 15s (55 B/s)
[ceph-client1][DEBUG ] Reading package lists...
[ceph-client1][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get -q -o Dpkg::Options::=--force-confnew --no-install-recommends --assume-yes install -- ceph ceph-mds ceph-common ceph-fs-common gdisk
[ceph-client1][DEBUG ] Reading package lists...
[ceph-client1][DEBUG ] Building dependency tree...
[ceph-client1][DEBUG ] Reading state information...
[ceph-client1][DEBUG ] gdisk is already the newest version.
[ceph-client1][DEBUG ] The following extra packages will be installed:
[ceph-client1][DEBUG ] binutils libaio1 libboost-thread1.49.0 libgoogle-perftools4 libjs-jquery
[ceph-client1][DEBUG ] libleveldb1 libnspr4 libnss3 librados2 librbd1 libreadline5 libsnappy1
[ceph-client1][DEBUG ] libtcmalloc-minimal4 libunwind8 python-ceph python-flask python-jinja2
[ceph-client1][DEBUG ] python-markupsafe python-werkzeug xfsprogs
[ceph-client1][DEBUG ] Suggested packages:
[ceph-client1][DEBUG ] binutils-doc javascript-common python-jinja2-doc ipython python-genshi
[ceph-client1][DEBUG ] python-lxml python-memcache libjs-sphinxdoc xfsdump acl attr quota
[ceph-client1][DEBUG ] Recommended packages:
[ceph-client1][DEBUG ] btrfs-tools ceph-fuse libcephfs1
[ceph-client1][DEBUG ] The following NEW packages will be installed:
[ceph-client1][DEBUG ] binutils ceph ceph-common ceph-fs-common ceph-mds libaio1
[ceph-client1][DEBUG ] libboost-thread1.49.0 libgoogle-perftools4 libjs-jquery libleveldb1 libnspr4
[ceph-client1][DEBUG ] libnss3 librados2 librbd1 libreadline5 libsnappy1 libtcmalloc-minimal4
[ceph-client1][DEBUG ] libunwind8 python-ceph python-flask python-jinja2 python-markupsafe
[ceph-client1][DEBUG ] python-werkzeug xfsprogs
[ceph-client1][DEBUG ] 0 upgraded, 24 newly installed, 0 to remove and 105 not upgraded.
[ceph-client1][DEBUG ] Need to get 40.9 MB of archives.
[ceph-client1][DEBUG ] After this operation, 192 MB of additional disk space will be used.
[ceph-client1][DEBUG ] Get:1 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libaio1 amd64 0.3.109-3 [6328 B]
[ceph-client1][DEBUG ] Get:2 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libsnappy1 amd64 1.0.5-2 [13.2 kB]
[ceph-client1][DEBUG ] Get:3 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libleveldb1 amd64 1.9.0-1 [138 kB]
[ceph-client1][DEBUG ] Get:4 http://ceph.com/debian-dumpling/ raring/main librados2 amd64 0.67.4-1raring [1635 kB]
[ceph-client1][DEBUG ] Get:5 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libnspr4 amd64 2:4.9.5-1ubuntu1 [134 kB]
[ceph-client1][DEBUG ] Get:6 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libnss3 amd64 2:3.14.3-0ubuntu1 [1044 kB]
[ceph-client1][DEBUG ] Get:7 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libreadline5 amd64 5.2+dfsg-1 [131 kB]
[ceph-client1][DEBUG ] Get:8 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libunwind8 amd64 1.0.1-4ubuntu2 [55.9 kB]
[ceph-client1][DEBUG ] Get:9 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main binutils amd64 2.23.2-2ubuntu1 [2393 kB]
[ceph-client1][DEBUG ] Get:10 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libboost-thread1.49.0 amd64 1.49.0-3.2ubuntu1 [41.6 kB]
[ceph-client1][DEBUG ] Get:11 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libjs-jquery all 1.7.2+debian-1ubuntu1 [115 kB]
[ceph-client1][DEBUG ] Get:12 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main python-werkzeug all 0.8.3+dfsg-1 [1333 kB]
[ceph-client1][DEBUG ] Get:13 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main python-markupsafe amd64 0.15-1build3 [13.8 kB]
[ceph-client1][DEBUG ] Get:14 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main python-jinja2 amd64 2.6-1build3 [158 kB]
[ceph-client1][DEBUG ] Get:15 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main python-flask all 0.9-1 [56.1 kB]
[ceph-client1][DEBUG ] Get:16 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main xfsprogs amd64 3.1.9 [1238 kB]
[ceph-client1][DEBUG ] Get:17 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libtcmalloc-minimal4 amd64 2.0-4ubuntu1 [163 kB]
[ceph-client1][DEBUG ] Get:18 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libgoogle-perftools4 amd64 2.0-4ubuntu1 [412 kB]
[ceph-client1][DEBUG ] Get:19 http://ceph.com/debian-dumpling/ raring/main librbd1 amd64 0.67.4-1raring [276 kB]
[ceph-client1][DEBUG ] Get:20 http://ceph.com/debian-dumpling/ raring/main python-ceph amd64 0.67.4-1raring [39.7 kB]
[ceph-client1][DEBUG ] Get:21 http://ceph.com/debian-dumpling/ raring/main ceph-common amd64 0.67.4-1raring [6090 kB]
[ceph-client1][DEBUG ] Get:22 http://ceph.com/debian-dumpling/ raring/main ceph amd64 0.67.4-1raring [22.8 MB]
[ceph-client1][DEBUG ] Get:23 http://ceph.com/debian-dumpling/ raring/main ceph-fs-common amd64 0.67.4-1raring [28.2 kB]
[ceph-client1][DEBUG ] Get:24 http://ceph.com/debian-dumpling/ raring/main ceph-mds amd64 0.67.4-1raring [2676 kB]
[ceph-client1][DEBUG ] Fetched 40.9 MB in 12s (3212 kB/s)
[ceph-client1][DEBUG ] Selecting previously unselected package libaio1:amd64.
[ceph-client1][DEBUG ] (Reading database ... 58918 files and directories currently installed.)
[ceph-client1][DEBUG ] Unpacking libaio1:amd64 (from .../libaio1_0.3.109-3_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libsnappy1.
[ceph-client1][DEBUG ] Unpacking libsnappy1 (from .../libsnappy1_1.0.5-2_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libleveldb1:amd64.
[ceph-client1][DEBUG ] Unpacking libleveldb1:amd64 (from .../libleveldb1_1.9.0-1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libnspr4:amd64.
[ceph-client1][DEBUG ] Unpacking libnspr4:amd64 (from .../libnspr4_2%3a4.9.5-1ubuntu1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libnss3:amd64.
[ceph-client1][DEBUG ] Unpacking libnss3:amd64 (from .../libnss3_2%3a3.14.3-0ubuntu1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libreadline5:amd64.
[ceph-client1][DEBUG ] Unpacking libreadline5:amd64 (from .../libreadline5_5.2+dfsg-1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libunwind8.
[ceph-client1][DEBUG ] Unpacking libunwind8 (from .../libunwind8_1.0.1-4ubuntu2_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package binutils.
[ceph-client1][DEBUG ] Unpacking binutils (from .../binutils_2.23.2-2ubuntu1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libboost-thread1.49.0.
[ceph-client1][DEBUG ] Unpacking libboost-thread1.49.0 (from .../libboost-thread1.49.0_1.49.0-3.2ubuntu1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package librados2.
[ceph-client1][DEBUG ] Unpacking librados2 (from .../librados2_0.67.4-1raring_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package librbd1.
[ceph-client1][DEBUG ] Unpacking librbd1 (from .../librbd1_0.67.4-1raring_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libjs-jquery.
[ceph-client1][DEBUG ] Unpacking libjs-jquery (from .../libjs-jquery_1.7.2+debian-1ubuntu1_all.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package python-werkzeug.
[ceph-client1][DEBUG ] Unpacking python-werkzeug (from .../python-werkzeug_0.8.3+dfsg-1_all.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package python-markupsafe.
[ceph-client1][DEBUG ] Unpacking python-markupsafe (from .../python-markupsafe_0.15-1build3_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package python-jinja2.
[ceph-client1][DEBUG ] Unpacking python-jinja2 (from .../python-jinja2_2.6-1build3_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package python-flask.
[ceph-client1][DEBUG ] Unpacking python-flask (from .../python-flask_0.9-1_all.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package python-ceph.
[ceph-client1][DEBUG ] Unpacking python-ceph (from .../python-ceph_0.67.4-1raring_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package ceph-common.
[ceph-client1][DEBUG ] Unpacking ceph-common (from .../ceph-common_0.67.4-1raring_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package xfsprogs.
[ceph-client1][DEBUG ] Unpacking xfsprogs (from .../xfsprogs_3.1.9_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libtcmalloc-minimal4.
[ceph-client1][DEBUG ] Unpacking libtcmalloc-minimal4 (from .../libtcmalloc-minimal4_2.0-4ubuntu1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libgoogle-perftools4.
[ceph-client1][DEBUG ] Unpacking libgoogle-perftools4 (from .../libgoogle-perftools4_2.0-4ubuntu1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package ceph.
[ceph-client1][DEBUG ] Unpacking ceph (from .../ceph_0.67.4-1raring_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package ceph-fs-common.
[ceph-client1][DEBUG ] Unpacking ceph-fs-common (from .../ceph-fs-common_0.67.4-1raring_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package ceph-mds.
[ceph-client1][DEBUG ] Unpacking ceph-mds (from .../ceph-mds_0.67.4-1raring_amd64.deb) ...
[ceph-client1][DEBUG ] Processing triggers for man-db ...
[ceph-client1][DEBUG ] Processing triggers for ureadahead ...
[ceph-client1][DEBUG ] Setting up libaio1:amd64 (0.3.109-3) ...
[ceph-client1][DEBUG ] Setting up libsnappy1 (1.0.5-2) ...
[ceph-client1][DEBUG ] Setting up libleveldb1:amd64 (1.9.0-1) ...
[ceph-client1][DEBUG ] Setting up libnspr4:amd64 (2:4.9.5-1ubuntu1) ...
[ceph-client1][DEBUG ] Setting up libnss3:amd64 (2:3.14.3-0ubuntu1) ...
[ceph-client1][DEBUG ] Setting up libreadline5:amd64 (5.2+dfsg-1) ...
[ceph-client1][DEBUG ] Setting up libunwind8 (1.0.1-4ubuntu2) ...
[ceph-client1][DEBUG ] Setting up binutils (2.23.2-2ubuntu1) ...
[ceph-client1][DEBUG ] Setting up libboost-thread1.49.0 (1.49.0-3.2ubuntu1) ...
[ceph-client1][DEBUG ] Setting up librados2 (0.67.4-1raring) ...
[ceph-client1][DEBUG ] Setting up librbd1 (0.67.4-1raring) ...
[ceph-client1][DEBUG ] Setting up libjs-jquery (1.7.2+debian-1ubuntu1) ...
[ceph-client1][DEBUG ] Setting up python-werkzeug (0.8.3+dfsg-1) ...
[ceph-client1][DEBUG ] Setting up python-markupsafe (0.15-1build3) ...
[ceph-client1][DEBUG ] Setting up python-jinja2 (2.6-1build3) ...
[ceph-client1][DEBUG ] Setting up python-flask (0.9-1) ...
[ceph-client1][DEBUG ] Setting up python-ceph (0.67.4-1raring) ...
[ceph-client1][DEBUG ] Setting up ceph-common (0.67.4-1raring) ...
[ceph-client1][DEBUG ] Setting up xfsprogs (3.1.9) ...
[ceph-client1][DEBUG ] Setting up libtcmalloc-minimal4 (2.0-4ubuntu1) ...
[ceph-client1][DEBUG ] Setting up libgoogle-perftools4 (2.0-4ubuntu1) ...
[ceph-client1][DEBUG ] Setting up ceph (0.67.4-1raring) ...
[ceph-client1][DEBUG ] ceph-all start/running
[ceph-client1][DEBUG ] Setting up ceph-fs-common (0.67.4-1raring) ...
[ceph-client1][DEBUG ] Processing triggers for ureadahead ...
[ceph-client1][DEBUG ] Setting up ceph-mds (0.67.4-1raring) ...
[ceph-client1][DEBUG ] ceph-mds-all start/running
[ceph-client1][DEBUG ] Processing triggers for libc-bin ...
[ceph-client1][DEBUG ] ldconfig deferred processing now taking place
[ceph-client1][DEBUG ] Processing triggers for ureadahead ...
[ceph-client1][INFO ] Running command: ceph --version
[ceph-client1][DEBUG ] ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
[root@ceph-mon1 ~]#
- From monitor node , use ceph-deploy to copy the Ceph configuration file and the ceph.client.admin.keyring to the ceph-client1.
[root@ceph-mon1 ceph]# ceph-deploy admin ceph-client1
[ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy admin ceph-client1
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-client1
[ceph-client1][DEBUG ] connected to host: ceph-client1
[ceph-client1][DEBUG ] detect platform information from remote host
[ceph-client1][DEBUG ] detect machine type
[ceph-client1][DEBUG ] get remote short hostname
[ceph-client1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[root@ceph-mon1 ceph]#
- On the ceph-client1 node, create a block device image.
rbd create block1-ceph-client1 --size 200
- On the ceph-client1 node, load the rbd client module.
modprobe rbd
- On the ceph-client1 node, map the image to a block device.
root@ceph-client1:~# rbd map block1-ceph-client1
root@ceph-client1:~#
root@ceph-client1:~#
root@ceph-client1:~# rbd showmapped
id pool image snap device
1 rbd block1-ceph-client1 - /dev/rbd1
root@ceph-client1:~#
- Use the block device by creating a file system on the ceph-client1 node.
root@ceph-client1:/dev/rbd/rbd# mkfs.ext4 /dev/rbd/rbd/block1-ceph-client1
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=4096 blocks, Stripe width=4096 blocks
51200 inodes, 204800 blocks
10240 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
25 block groups
8192 blocks per group, 8192 fragments per group
2048 inodes per group
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729
Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
root@ceph-client1:/dev/rbd/rbd#
- Mount the file system on the ceph-client1 node.
root@ceph-client1:/dev/rbd/rbd# mount /dev/rbd/rbd/block1-ceph-client1 /rbd-1/
root@ceph-client1:/dev/rbd/rbd#
root@ceph-client1:/dev/rbd/rbd#
root@ceph-client1:/dev/rbd/rbd#
root@ceph-client1:/dev/rbd/rbd# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 9.8G 1.3G 8.0G 14% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 487M 8.0K 487M 1% /dev
tmpfs 100M 256K 100M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 498M 0 498M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/vdb 109G 188M 103G 1% /mnt
/dev/rbd1 190M 1.6M 179M 1% /rbd-1
root@ceph-client1:/dev/rbd/rbd#
Subscribe to:
Posts (Atom)