-
如何配置CEPH RGW对象存储与公有云同步
云和安全管理服务专家 祝祥原创
容灾 (Disaster Recovery)，即容灾备份或灾备，是业务连续性系统的一个子集，用于保障 IT 系统在遭受自然灾害、人为操作失误或蓄意破坏后的数据还原和业务恢复。
Ceph RGW 多数据中心（multisite）功能旨在实现异地双活，提供了备份容灾的能力。并且具有多个数据中心供用户选择，存放资源。
主节点在对外提供服务时，用户数据在主节点落盘后即向用户回应“写成功”应答，然后实时记录数据变化的相关日志信息。备节点则实时比较主备数据差异，并及时将差异化数据拉回备节点。异步复制技术适用于远距离的容灾方案，对系统性能影响较小。
以往的容灾都是在单该或者多该ceph集群之间的，本次主要由于模拟 ceph 的 RGW 可以与其他所有的S3云供应商(
https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/)进行数据同步，从而实现Ceph的RGW对象存储的灾备。内容
要求：
- Ceph 集群
- 2 个 RGW 守护进程正在运行
- S3 目标
我们将使用三个端点：
- http://192.168.112.5:80
- RGW 为我们现有的集群管理的端点
- http://192.168.112.6:80
- 我们将创建一个新端点，它将依赖 RGW 将数据同步到另一个 S3 提供者
- http://192.168.105.5:80
- 我们将用于推送/同步数据的 S3 目标
检查我们现有的存储池：
所有 rgw 都有一个区域和区域组，它可能是default.
您可以检查池的名称：
(docker-croit)@mycephcluster / $ ceph osd lspools 1 device_health_metrics 2 .rgw.root 3 default.rgw.control 4 default.rgw.log 5 default.rgw.meta 6 default.rgw.buckets.non-ec 7 default.rgw.buckets.index 8 default.rgw.buckets.data或者直接检查区域或区域组：
(docker-croit)@mycephcluster / $ radosgw-admin zone get --rgw-zone=default { "id": "303a00f5-f50d-43fd-afee-aa0503926952", "name": "default", ... }(docker-croit)@mycephcluster / $ radosgw-admin zonegroup get --rgw-zonegroup=default { "id": "881cf806-f6d2-47a0-b7dc-d65ee87f8ef4", "name": "default", "api_name": "default", "is_master": "true", ... "zones": [ { "id": "303a00f5-f50d-43fd-afee-aa0503926952", "name": "default",准备存储池：
我们的新区将需要一些池。我们将手动创建它们以确保没有问题（例如每个 osd 有太多 PG）阻止它们的创建。
(docker-croit)@mycephcluster / $ for pool in sync.rgw.meta sync.rgw.log sync.rgw.control sync.rgw.buckets.non-ec sync.rgw.buckets.index sync.rgw.buckets.data; do ceph osd pool create $pool 16 16 replicated; done pool 'sync.rgw.meta' created pool 'sync.rgw.log' created pool 'sync.rgw.control' created pool 'sync.rgw.buckets.non-ec' created pool 'sync.rgw.buckets.index' created pool 'sync.rgw.buckets.data' created创建新区域：
我们的新区域将命名为sync：
(docker-croit)@mycephcluster / $ radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=sync --endpoints=http://192.168.112.6/ --tier-type=cloud { "id": "7ead9532-0938-4698-9b4a-2d84d0d00869", "name": "sync", "domain_root": "sync.rgw.meta:root", "control_pool": "sync.rgw.control", "gc_pool": "sync.rgw.log:gc", "lc_pool": "sync.rgw.log:lc", "log_pool": "sync.rgw.log", "intent_log_pool": "sync.rgw.log:intent", "usage_log_pool": "sync.rgw.log:usage", "roles_pool": "sync.rgw.meta:roles", "reshard_pool": "sync.rgw.log:reshard", "user_keys_pool": "sync.rgw.meta:users.keys", "user_email_pool": "sync.rgw.meta:users.email", "user_swift_pool": "sync.rgw.meta:users.swift", "user_uid_pool": "sync.rgw.meta:users.uid", "otp_pool": "sync.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "sync.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "sync.rgw.buckets.data" } }, "data_extra_pool": "sync.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "46669d35-f7ed-4374-8247-2b8f41218109" }- rgw-zonegroup：我们的新区域将成为默认区域组的一部分。
- 端点：我们的新区域需要它自己的 RGW，因此它使用新的端点。
- tier-type ：我们使用cloud层类型，请参阅文档(https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/)了解更多设置
修改现有区域：
我们需要添加现有default区域的端点。
(docker-croit)@mycephcluster / $ radosgw-admin zone modify --rgw-zonegroup=default --rgw-zone=default --endpoints=http://192.168.112.5:80 { "id": "303a00f5-f50d-43fd-afee-aa0503926952", "name": "default", "domain_root": "default.rgw.meta:root", "control_pool": "default.rgw.control", "gc_pool": "default.rgw.log:gc", "lc_pool": "default.rgw.log:lc", "log_pool": "default.rgw.log", "intent_log_pool": "default.rgw.log:intent", "usage_log_pool": "default.rgw.log:usage", "roles_pool": "default.rgw.meta:roles", "reshard_pool": "default.rgw.log:reshard", "user_keys_pool": "default.rgw.meta:users.keys", "user_email_pool": "default.rgw.meta:users.email", "user_swift_pool": "default.rgw.meta:users.swift", "user_uid_pool": "default.rgw.meta:users.uid", "otp_pool": "default.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "default.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "default.rgw.buckets.data" } }, "data_extra_pool": "default.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "46669d35-f7ed-4374-8247-2b8f41218109" }创建系统用户：
系统用户将用于同步数据，可以通过 CLI 创建此用户。
(docker-croit)@mycephcluster / $ radosgw-admin user create --uid=syncuser --display-name=syncuser --system { "user_id": "syncuser", "display_name": "syncuser", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "syncuser", "access_key": "VGIF31FGOHZ0Q6MQRBQR", "secret_key": "1FwPZH0ICfV1e1zi8okXApJJJEB0XHfiOxe1mmTr" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "system": "true", "default_placement": "", "default_storage_class": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] }配置同步区域以使用此系统用户：
我们将更改两个区域以使用我们的新系统用户。
(docker-croit)@mycephcluster / $ radosgw-admin user info --uid syncuser| jq '.keys' [ { "user": "syncuser", "access_key": "VGIF31FGOHZ0Q6MQRBQR", "secret_key": "1FwPZH0ICfV1e1zi8okXApJJJEB0XHfiOxe1mmTr" } ](docker-croit)@mycephcluster / $ radosgw-admin zone modify --rgw-zonegroup=default --rgw-zone=default --access-key=VGIF31FGOHZ0Q6MQRBQR --secret=1FwPZH0ICfV1e1zi8okXApJJJEB0XHfiOxe1mmTr { "id": "303a00f5-f50d-43fd-afee-aa0503926952", "name": "default", "domain_root": "default.rgw.meta:root", "control_pool": "default.rgw.control", "gc_pool": "default.rgw.log:gc", "lc_pool": "default.rgw.log:lc", "log_pool": "default.rgw.log", "intent_log_pool": "default.rgw.log:intent", "usage_log_pool": "default.rgw.log:usage", "roles_pool": "default.rgw.meta:roles", "reshard_pool": "default.rgw.log:reshard", "user_keys_pool": "default.rgw.meta:users.keys", "user_email_pool": "default.rgw.meta:users.email", "user_swift_pool": "default.rgw.meta:users.swift", "user_uid_pool": "default.rgw.meta:users.uid", "otp_pool": "default.rgw.otp", "system_key": { "access_key": "VGIF31FGOHZ0Q6MQRBQR", "secret_key": "1FwPZH0ICfV1e1zi8okXApJJJEB0XHfiOxe1mmTr" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "default.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "default.rgw.buckets.data" } }, "data_extra_pool": "default.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "46669d35-f7ed-4374-8247-2b8f41218109" }(docker-croit)@mycephcluster / $ radosgw-admin zone modify --rgw-zonegroup=default --rgw-zone=sync --access-key=VGIF31FGOHZ0Q6MQRBQR --secret=1FwPZH0ICfV1e1zi8okXApJJJEB0XHfiOxe1mmTr { "id": "7ead9532-0938-4698-9b4a-2d84d0d00869", "name": "sync", "domain_root": "sync.rgw.meta:root", "control_pool": "sync.rgw.control", "gc_pool": "sync.rgw.log:gc", "lc_pool": "sync.rgw.log:lc", "log_pool": "sync.rgw.log", "intent_log_pool": "sync.rgw.log:intent", "usage_log_pool": "sync.rgw.log:usage", "roles_pool": "sync.rgw.meta:roles", "reshard_pool": "sync.rgw.log:reshard", "user_keys_pool": "sync.rgw.meta:users.keys", "user_email_pool": "sync.rgw.meta:users.email", "user_swift_pool": "sync.rgw.meta:users.swift", "user_uid_pool": "sync.rgw.meta:users.uid", "otp_pool": "sync.rgw.otp", "system_key": { "access_key": "VGIF31FGOHZ0Q6MQRBQR", "secret_key": "1FwPZH0ICfV1e1zi8okXApJJJEB0XHfiOxe1mmTr" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "sync.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "sync.rgw.buckets.data" } }, "data_extra_pool": "sync.rgw.buckets.non-ec", "index_type": 0 } } ], "tier_config": { "connection": { "access_key": "JO4RQ1787A6OGI6XMFDW", "endpoint": "http://192.168.105.5:80", "secret": "Dx5kKGUUeR0DaSRYueBWhV6oDRvJ9oXH2gPcVJ6s" } }, "realm_id": "46669d35-f7ed-4374-8247-2b8f41218109" }确保默认区域是主区域：
(docker-croit)@mycephcluster / $ radosgw-admin zonegroup get { "id": "881cf806-f6d2-47a0-b7dc-d65ee87f8ef4", "name": "default", "api_name": "default", "is_master": "true", "endpoints": [], "hostnames": [], "hostnames_s3website": [], "master_zone": "303a00f5-f50d-43fd-afee-aa0503926952", "zones": [ { "id": "303a00f5-f50d-43fd-afee-aa0503926952", "name": "default",如果defaultzone 不是 master，则可以通过执行强制它radosgw-admin zone modify –rgw-zonegroup=default –rgw-zone=default –master –default
提交更改并验证配置：
(docker-croit)@mycephcluster / $ radosgw-admin period update --commit { "id": "1861622f-b748-410d-b4a9-7338f4b6842b", "epoch": 3, "predecessor_uuid": "b6cd42db-6567-4a4b-9433-aee238da0c9d", "sync_status": [], "period_map": { "id": "1861622f-b748-410d-b4a9-7338f4b6842b", "zonegroups": [ { "id": "881cf806-f6d2-47a0-b7dc-d65ee87f8ef4", "name": "default", "api_name": "default", "is_master": "true", "endpoints": [], "hostnames": [], "hostnames_s3website": [], "master_zone": "303a00f5-f50d-43fd-afee-aa0503926952", "zones": [ { "id": "303a00f5-f50d-43fd-afee-aa0503926952", "name": "default", "endpoints": [ "http://192.168.112.5:80" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "7ead9532-0938-4698-9b4a-2d84d0d00869", "name": "sync", "endpoints": [ "http://192.168.112.6/" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "cloud", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "46669d35-f7ed-4374-8247-2b8f41218109", "sync_policy": { "groups": [] } } ], "short_zone_ids": [ { "key": "303a00f5-f50d-43fd-afee-aa0503926952", "val": 2796720163 }, { "key": "7ead9532-0938-4698-9b4a-2d84d0d00869", "val": 2175446857 } ] }, "master_zonegroup": "881cf806-f6d2-47a0-b7dc-d65ee87f8ef4", "master_zone": "303a00f5-f50d-43fd-afee-aa0503926952", "period_config": { "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } }, "realm_id": "46669d35-f7ed-4374-8247-2b8f41218109", "realm_name": "default", "realm_epoch": 2 }配置新区域：
我们的云同步模块需要一些配置(
https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/#cloud-sync-tier-type-configuration)。我们将定义用于同步数据的端点和 S3 用户凭据。注意：如果您的密钥以 0 开头，您将无法配置它。例如，使用05XXXXXXXX时 ，访问密钥将被错误地存储：
(docker-croit)@mycephcluster / $ radosgw-admin zone modify --rgw-zonegroup=default --rgw-zone=sync --tier-config=connection.endpoint=http://192.168.105.5:80,connection.access_key=05XXXXXXXX,connection.secret=56NwS1p7krU0IMYaXXXXXXXXXXXXX (docker-croit)@mycephcluster / $ radosgw-admin zone get --rgw-zone=sync | jq '.tier_config' { "connection": { "access_key": 5, "endpoint": "http://192.168.105.5:80", "secret": 56NwS1p7krU0IMYaXXXXXXXXXXXXX } }(docker-croit)@mycephcluster / $ radosgw-admin zone modify --rgw-zonegroup=default --rgw-zone=sync --tier-config=connection.endpoint=http://192.168.105.5:80,connection.access_key=JO4RQ1787A6OGI6XMFDW,connection.secret=Dx5kKGUUeR0DaSRYueBWhV6oDRvJ9oXH2gPcVJ6s { "id": "7ead9532-0938-4698-9b4a-2d84d0d00869", "name": "sync", "domain_root": "sync.rgw.meta:root", "control_pool": "sync.rgw.control", "gc_pool": "sync.rgw.log:gc", "lc_pool": "sync.rgw.log:lc", "log_pool": "sync.rgw.log", "intent_log_pool": "sync.rgw.log:intent", "usage_log_pool": "sync.rgw.log:usage", "roles_pool": "sync.rgw.meta:roles", "reshard_pool": "sync.rgw.log:reshard", "user_keys_pool": "sync.rgw.meta:users.keys", "user_email_pool": "sync.rgw.meta:users.email", "user_swift_pool": "sync.rgw.meta:users.swift", "user_uid_pool": "sync.rgw.meta:users.uid", "otp_pool": "sync.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "sync.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "sync.rgw.buckets.data" } }, "data_extra_pool": "sync.rgw.buckets.non-ec", "index_type": 0 } } ], "tier_config": { "connection": { "access_key": "JO4RQ1787A6OGI6XMFDW", "endpoint": "http://192.168.105.5:80", "secret": "Dx5kKGUUeR0DaSRYueBWhV6oDRvJ9oXH2gPcVJ6s" } }, "realm_id": "46669d35-f7ed-4374-8247-2b8f41218109" }检查配置是否已正确应用。
(docker-croit)@mycephcluster / $ radosgw-admin zone get --rgw-zone=sync | jq '.tier_config' { "connection": { "access_key": "JO4RQ1787A6OGI6XMFDW", "endpoint": "http://192.168.105.5:80", "secret": "Dx5kKGUUeR0DaSRYueBWhV6oDRvJ9oXH2gPcVJ6s" } }提交更改：
(docker-croit)@mycephcluster / $ radosgw-admin zone get --rgw-zone=sync { "id": "7ead9532-0938-4698-9b4a-2d84d0d00869", "name": "sync", "domain_root": "sync.rgw.meta:root", "control_pool": "sync.rgw.control", "gc_pool": "sync.rgw.log:gc", "lc_pool": "sync.rgw.log:lc", "log_pool": "sync.rgw.log", "intent_log_pool": "sync.rgw.log:intent", "usage_log_pool": "sync.rgw.log:usage", "roles_pool": "sync.rgw.meta:roles", "reshard_pool": "sync.rgw.log:reshard", "user_keys_pool": "sync.rgw.meta:users.keys", "user_email_pool": "sync.rgw.meta:users.email", "user_swift_pool": "sync.rgw.meta:users.swift", "user_uid_pool": "sync.rgw.meta:users.uid", "otp_pool": "sync.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "sync.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "sync.rgw.buckets.data" } }, "data_extra_pool": "sync.rgw.buckets.non-ec", "index_type": 0 } } ], "tier_config": { "connection": { "access_key": "JO4RQ1787A6OGI6XMFDW", "endpoint": "http://192.168.105.5:80", "secret": "Dx5kKGUUeR0DaSRYueBWhV6oDRvJ9oXH2gPcVJ6s" } }, "realm_id": "46669d35-f7ed-4374-8247-2b8f41218109" }配置 RGW：
我们需要修改每个 radosgw 配置以通过删除rgw zone = default和添加来管理正确的区域
[client.rgw.$(hostname)] host = $(hostname) rgw zone = default [client.rgw.$(hostname)] host = $(hostname) rgw zone = sync在我们的基础设施上，我们/etc/ceph/ceph.conf通过添加以下内容进行编辑：
[client.rgw.new-croit-host-C0DE01] host = new-croit-host-C0DE01 rgw zone = default [client.rgw.new-croit-host-C0DE02] host = new-croit-host-C0DE02 rgw zone = sync如果你使用 croit，你可以简单地ceph.conf用这个内容替换模板。
[global] auth cluster required = cephx auth service required = cephx auth client required = cephx auth supported = cephx mon client hunt interval = {{huntInterval}} mon client hunt parallel = {{huntParallel}} fsid = {{fsid}} mon host = {{commaSeparatedList mons}} {{~#if full}} {{~#unless server.managementHost}} crush location = host={{server.hostname}} {{~/unless}} {{~#if publicNets}} public network = {{publicNets}} {{~/if}} {{~#if privateNets}} cluster network = {{privateNets}} {{~/if}} log file = /dev/null mon cluster log file = /dev/null mon cluster log to syslog = true log to syslog = true err to syslog = true {{~#replaceAll "rgw zone = default" ""~}} {{~#options}} {{key}} = {{value}} {{~/options}} {{~/replaceAll}} [client.rgw.new-croit-host-C0DE01] host = new-croit-host-C0DE01 rgw zone = default [client.rgw.new-croit-host-C0DE02] host = new-croit-host-C0DE02 rgw zone = sync {{~/if}}要应用更改，您必须重新启动 RGW。
root@new-croit-host-C0DE01 ~ $ systemctl restart ceph-radosgw@rgw.new-croit-host-C0DE01.service root@new-croit-host-C0DE02 ~ $ systemctl restart ceph-radosgw@rgw.new-croit-host-C0DE02.service测试 S3 同步
我们将使用 s3cmd 进行测试。
在源和目标上生成用户并配置他们的 s3cmd 配置文件。检查 API 访问是否有效。
~ s3cmd -c .s3cfg_source ls ~ s3cmd -c .s3cfg_target ls ~如果您有 API 错误错误，请检查您的凭据、端点。
在源上创建一个存储桶并添加对象：
我们首先创建一个bucket：
~ s3cmd -c .s3cfg_source mb s3://mystetbucket Bucket 's3://mystetbucket/' created ~ s3cmd -c .s3cfg_source ls 2021-06-24 15:06 s3://mystetbucket现在我们在源上添加一个对象：
~ s3cmd -c .s3cfg_source put /tmp/myobject s3://mystetbucket/synctest 64 WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. upload: '/tmp/myobject' -> 's3://mystetbucket/synctest' [1 of 1] 13 of 13 100% in 0s 325.90 B/s done并检查它是否已同步到目标：
~ s3cmd -c .s3cfg_target ls s3:// 2021-06-24 15:30 s3://rgw-default-271b93c16a9565d8 ~ s3cmd -c .s3cfg_target ls s3://rgw-default-271b93c16a9565d8 DIR s3://rgw-default-271b93c16a9565d8/mystetbucket/ ~ s3cmd -c .s3cfg_target ls s3://rgw-default-271b93c16a9565d8/mystetbucket/ 2021-06-24 15:36 13 s3://rgw-default-271b93c16a9565d8/mystetbucket/synctest技巧和窍门
您可以随时增加调试日志记录以便于调试：
root@new-croit-host-C0DE02 ~ $ ceph --admin-daemon /var/run/ceph/ceph-client.rgw.new-croit-host-C0DE02.96866.94534872347832.asok config set debug_rgw_sync 5 root@new-croit-host-C0DE02 ~ $ ceph --admin-daemon /var/run/ceph/ceph-client.rgw.new-croit-host-C0DE02.96866.94534872347832.asok config set debug_rgw 5检查同步状态：
(docker-croit)@mycephcluster / $ radosgw-admin sync status --rgw-zone=sync realm 46669d35-f7ed-4374-8247-2b8f41218109 (default) zonegroup 881cf806-f6d2-47a0-b7dc-d65ee87f8ef4 (default) zone 7ead9532-0938-4698-9b4a-2d84d0d00869 (sync) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 303a00f5-f50d-43fd-afee-aa0503926952 (default) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source参考文章：
- https://docs.ceph.com/en/latest/radosgw/multisite/
- https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/#cloud-sync-tier-type-configuration
- https://croit.io/blog/setting-up-ceph-cloud-sync-module
云和安全管理服务专家
