読者です 読者をやめる 読者になる 読者になる

5400rpm

人生シングルスレッド

夏休みの自由工作(2)

日記

続・Raspberry Pi3でCephを動かす

e6zout.hatenablog.jp

2. Ceph環境の構築

  • 基本はありもののパッケージを利用する
  • まずはオブジェクトストレージの検証を第一目標にする
    • 手順についてはビットアイルのこちらの記事を多々参考にさせていただきました

    Ceph ~ブロックアクセス可能な分散ストレージ

2-1. 接続環境

  • ズレてたらごめん
    • ローカルネットワークはルータデフォルトの 192.168.10.0/24 を利用
    • IPは自宅サーバCentOS以外は明示での固定せずDHCP任せ(48Hリリースなので実質固定状態)
    • 自宅サーバのhostnameは comicsong を利用しているが以降手順の中では簡略化のため Client にて表記する
+----------------------------------+
|個人用PC(Win10),タブレット機器etc.|
+-+--------------------------------+
  |
+-+-----------------------+  +----------- ... ..  .  
|NECのブロードバンドルータ   +--+ The Internet
+-+---------------------+-+  +----------- ... ..  . 
  |                     |
+-+----------------+    |
|自宅サーバ(録画用)|    |
|Client兼Deployer  |    |
|CentOS7           |    |
+------------------+    | 
+-----------------------++
|Netgear スイッチングハブ|
+------------------------+
|  +----------------------------------------+
+--+※Ceph-node1(MON.0) OSD.0,3(各USB 128GB)|
|  +----------------------------------------+
|
|  +----------------------------------------+
+--+※Ceph-node2(MON.1) OSD.1,4(各USB 128GB)|
|  +----------------------------------------+
|
|  +----------------------------------------+
+--+※Ceph-node3(MON.2) OSD.2,5(各USB 128GB)|
|  +----------------------------------------+
|
|  +--------------------+
+--+※Ceph-node4(MDS.0) |
   +--------------------+

※4台共通で Raspberry Pi3/Ubuntu16.04

2-2. CephDeployerのインストール

  • Clinetで実行
$ sudo yum install ceph-deploy
  • バージョンは 1.5.25-1 でした
[ubuntu@comicsong ~]$ rpm -q ceph-deploy
ceph-deploy-1.5.25-1.el7.noarch
  • 各ceph-nodeに鍵配布してsshでパスなしでのログインを設定
  • ceph-deploy を実施
[ubuntu@comicsong ~]$ ceph-deploy new ceph-node1 ceph-node2 ceph-node3
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ubuntu/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.25): /bin/ceph-deploy new ceph-node1 ceph-node2 ceph-node3
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[ceph-node1][DEBUG ] connected to host: comicsong.jp
[ceph-node1][INFO  ] Running command: ssh -CT -o BatchMode=yes ceph-node1
[ceph-node1][DEBUG ] connection detected need for sudo
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph-node1][DEBUG ] find the location of an executable
[ceph-node1][INFO  ] Running command: sudo /bin/ip link show
[ceph-node1][INFO  ] Running command: sudo /bin/ip addr show
[ceph-node1][DEBUG ] IP addresses found: ['192.168.10.7']
[ceph_deploy.new][DEBUG ] Resolving host ceph-node1
[ceph_deploy.new][DEBUG ] Monitor ceph-node1 at 192.168.10.7
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[ceph-node2][DEBUG ] connected to host: comicsong.jp
[ceph-node2][INFO  ] Running command: ssh -CT -o BatchMode=yes ceph-node2
[ceph-node2][DEBUG ] connection detected need for sudo
[ceph-node2][DEBUG ] connected to host: ceph-node2
[ceph-node2][DEBUG ] detect platform information from remote host
[ceph-node2][DEBUG ] detect machine type
[ceph-node2][DEBUG ] find the location of an executable
[ceph-node2][INFO  ] Running command: sudo /bin/ip link show
[ceph-node2][INFO  ] Running command: sudo /bin/ip addr show
[ceph-node2][DEBUG ] IP addresses found: ['192.168.10.8']
[ceph_deploy.new][DEBUG ] Resolving host ceph-node2
[ceph_deploy.new][DEBUG ] Monitor ceph-node2 at 192.168.10.8
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[ceph-node3][DEBUG ] connected to host: comicsong.jp
[ceph-node3][INFO  ] Running command: ssh -CT -o BatchMode=yes ceph-node3
[ceph-node3][DEBUG ] connection detected need for sudo
[ceph-node3][DEBUG ] connected to host: ceph-node3
[ceph-node3][DEBUG ] detect platform information from remote host
[ceph-node3][DEBUG ] detect machine type
[ceph-node3][DEBUG ] find the location of an executable
[ceph-node3][INFO  ] Running command: sudo /bin/ip link show
[ceph-node3][INFO  ] Running command: sudo /bin/ip addr show
[ceph-node3][DEBUG ] IP addresses found: ['192.168.10.9']
[ceph_deploy.new][DEBUG ] Resolving host ceph-node3
[ceph_deploy.new][DEBUG ] Monitor ceph-node3 at 192.168.10.9
[ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph-node1', 'ceph-node2', 'ceph-node3']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.10.7', '192.168.10.8', '192.168.10.9']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
  • 設定ファイルが作成された
    • 何処に作られるのかなと探したがまさかの実行ディレクトリだった
[ubuntu@comicsong ~]$ pwd
/home/ubuntu
[ubuntu@comicsong ~]$ cat ceph.conf
[global]
fsid = 37164a6b-7187-4ba7-8b3f-4eccc58a9cf0
mon_initial_members = ceph-node1, ceph-node2, ceph-node3
mon_host = 192.168.10.7,192.168.10.8,192.168.10.9
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
  • ceph-deploy install を実施
$ ceph-deploy install ceph-node1 ceph-node2 ceph-node3 ceph-node4
  • ceph-deploy mon create-initial を実施
$ ceph-deploy mon create-initial
  • ここで問題発生。対象で sudo initctl emit ceph-mon cluster=ceph id=ceph-node1 を実施する箇所でエラー出る
  • 実行される各nodeは initctl じゃなくて systemd だからね…
[ceph-node1][INFO  ] Running command: sudo initctl emit ceph-mon cluster=ceph id=ceph-node1
[ubuntu@comicsong ~]$ wget http://download.ceph.com/rpm/el7/noarch/ceph-deploy-1.5.34-0.noarch.rpm
.
.
.
`ceph-deploy-1.5.34-0.noarch.rpm' に保存中

100%[=========================================================================>] 288,604     95.8KB/s 時間 2.9s

2016-08-16 16:48:37 (95.8 KB/s) - `ceph-deploy-1.5.34-0.noarch.rpm' へ保存完了 [288604/288604]

[ubuntu@comicsong ~]$ rpm --test -Uvh ceph-deploy-1.5.34-0.noarch.rpm
警告: ceph-deploy-1.5.34-0.noarch.rpm: ヘッダー V4 RSA/SHA1 Signature、鍵 ID 460f3994: NOKEY
準備しています...              ################################# [100%]
[ubuntu@comicsong ~]$ sudo rpm -Uvh ceph-deploy-1.5.34-0.noarch.rpm
警告: ceph-deploy-1.5.34-0.noarch.rpm: ヘッダー V4 RSA/SHA1 Signature、鍵 ID 460f3994: NOKEY
準備しています...              ################################# [100%]
更新中 / インストール中...
   1:ceph-deploy-1.5.34-0             ################################# [ 50%]
整理中 / 削除中...
   2:ceph-deploy-1.5.25-1.el7         ################################# [100%]
  • 旧バージョンで作成されたconfを削除しておく
rm ceph.conf
  • newからやり直す
$ ceph-deploy new ceph-node1 ceph-node2 ceph-node3
$ ceph-deploy install ceph-node1 ceph-node2 ceph-node3 ceph-node4
  • 続く

e6zout.hatenablog.jp