DAOS-在docker中搭建daos开发调试环境
简介
本节介绍如何在 Docker 容器中构建和运行 DAOS 服务。 至少需要 4GB DRAM 和 16GB 磁盘空间。 在 Mac 上，请确保“Preferences/{Disk, Memory}”下的 Docker 设置已进行相应配置
步骤
拉取项目
git clone https://github.com/ssbandjl/daos.git
下载依赖包(缓存cache)
cd daos; wget http://rz2fg6ogr.hn-bkt.clouddn.com/cache_tgz
编译镜像或下载我这边已经编好的镜像
编译镜像: docker build . -f utils/docker/Dockerfile.el.8 -t daos
# 已编好的镜像: wget http://rz2fg6ogr.hn-bkt.clouddn.com/daos_main.tgz; docker load -i daos_main.tgz
查看docker镜像:
[root@node3 daos]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
daos latest 8a64df225def 2 weeks ago 3.16GB给宿主机配置大页(如8GB):
永久生效
tee<<EOF>>/etc/sysctl.conf
vm.nr_hugepages = 4096
EOF
sysctl -p启动docker镜像:
docker run -it -d --privileged --cap-add=ALL --name daos_main1 -v /home/xb/project/stor/daos/main/daos:/home/xb/project/stor/daos/main/daos -v /dev:/dev -v /dev/hugepages:/dev/hugepages -v /sys/fs/cgroup:/sys/fs/cgroup:ro daos:latest进入镜像: docker exec -u root -it daos_main1 bash -c 'cd /home/xb/project/stor/daos/main/daos;exec "${SHELL:-sh}"'重新编译并安装DAOS:
./build.sh 或
scons --jobs 1 install PREFIX=/opt/daos BUILD_TYPE=debug TARGET_TYPE=debug --build-deps=yeswhich dmg拷贝证书和配置(启动时根据提示修改权限):
cp -r daosCA/certs /etc/daos/参考配置文件:
daos_server_docker_main.yml
# For a single-server systemname: daos_server
access_points: ['172.17.0.2']
provider: ofi+sockets
control_log_mask: DEBUG
control_log_file: /tmp/daos_server.log
transport_config:allow_insecure: falseclient_cert_dir: /etc/daos/certs/clientsca_cert: /etc/daos/certs/daosCA.crtcert: /etc/daos/certs/server.crtkey: /etc/daos/certs/server.keyengines:
-targets: 1first_core: 0nr_xs_helpers: 0fabric_iface: eth0fabric_iface_port: 31416log_mask: debuglog_file: /tmp/daos_engine.0.logenv_vars:- FI_LOG_LEVEL=warn- FI_SOCKETS_MAX_CONN_RETRY=1- FI_SOCKETS_CONN_TIMEOUT=2000- D_LOG_MASK=DEBUG- DD_SUBSYS=all- DD_MASK=all- HG_LOG_LEVEL=debug# Storage definitions# When scm_class is set to ram, tmpfs will be used to emulate SCM.# The size of ram is specified by scm_size in GB units.scm_mount: /mnt/daos # map to -s /mnt/daosscm_class: ramscm_size: 4 #内存模拟scm# When bdev_class is set to file, Linux AIO will be used to emulate NVMe.# The size of file that will be created is specified by bdev_size in GB units.# The location of the files that will be created is specified in bdev_list.bdev_class: filebdev_size: 16 #文件模拟nvmebdev_list: [/tmp/daos-bdev]启动集群:
./start.sh
或参考:
echo -e "start"# source /root/.bashrc
# run_all "pkill daos_agent;pkill daos_server"source /root/.bashrc
pkill daos_agent
pkill daos_server
rm -f /tmp/daos*.log
rm -f /tmp/.daos_engine.0.log.swp
rm -f /tmp/daos_engine*
umount /mnt/daos#export FI_LOG_LEVEL=debug
#export FI_LOG_LEVEL=warn#export HG_LOG_LEVEL=debug
#export HG_LOG_LEVEL=warndaos_agent &
# echo -e "dmg storage format"
daos_server start &# count=0
# while true;do
# joined_num=`dmg sys query -v|grep Joined|wc -l`
# if [[ $joined_num -eq 3 ]];then
# break
# fi
# echo -e "wait all rank join, $count times"
# count=$((count+1))
# sleep 1
# donesleep 5
dmg storage format创建池,容器, 挂载daos文件系统:
参考文件: daos_create_pool_cont_mount_dfuse
dmg pool create sxb -z 4g; dmg pool list --verbose
daos container create sxb --type POSIX sxb; daos container query sxb sxb --verbose; daos cont get-prop sxb sxb
mkdir -p /tmp/sxb; dfuse --mountpoint=/tmp/sxb --pool=sxb --cont=sxb; df -h
#cd /tmp/sxb最终效果:
[root@8b90fa1190f1 daos]# dmg pool list --verbose
Label UUID State SvcReps SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled UpgradeNeeded?
----- ---- ----- ------- -------- -------- ------------- --------- --------- -------------- -------- --------------
sxb 9b4157a8-7bab-493c-aa4d-5145144d92b3 Ready 0 240 MB 213 kB 0% 3.8 GB 52 MB 0% 0/1 None [root@8b90fa1190f1 daos]# daos container query sxb sxb --verbose; daos cont get-prop sxb sxbContainer UUID : ccc2f4f7-1e46-4709-8f28-09282a9f82ed Container Label : sxb Container Type : POSIX Pool UUID : 9b4157a8-7bab-493c-aa4d-5145144d92b3 Container redundancy factor : 0 Number of open handles : 2 Latest open time : 0x126c7703f0840000 (2023-08-19 08:02:08.06070272 +0000 UTC)Latest close/modify time : 0x126c76d527b80000 (2023-08-19 08:01:55.50212096 +0000 UTC)Number of snapshots : 0 Object Class : UNKNOWN Dir Object Class : UNKNOWN File Object Class : UNKNOWN Chunk Size : 1.0 MiB Properties for container sxb
Name Value
---- -----
Highest Allocated OID (alloc_oid) 11
Checksum (cksum) off
Checksum Chunk Size (cksum_size) 32 KiB
Compression (compression) off
Deduplication (dedup) off
Dedupe Threshold (dedup_threshold) 4.0 KiB
EC Cell Size (ec_cell_sz) 64 KiB
Performance domain affinity level of EC (ec_pda) 1
Encryption (encryption) off
Global Version (global_version) 3
Group (group) root@
Label (label) sxb
Layout Type (layout_type) POSIX (1)
Layout Version (layout_version) 1
Max Snapshot (max_snapshot) 0
Object Version (obj_version) 1
Owner (owner) root@
Performance domain level (perf_domain) root (255)
Redundancy Factor (rd_fac) rd_fac0
Redundancy Level (rd_lvl) node (2)
Performance domain affinity level of RP (rp_pda) 4294967295
Server Checksumming (srv_cksum) off
Health (status) HEALTHY
Access Control List (acl) A::OWNER@:rwdtTaAo, A:G:GROUP@:rwtT
[root@8b90fa1190f1 daos]# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 50G 25G 26G 50% /
devtmpfs 63G 0 63G 0% /dev
tmpfs 63G 0 63G 0% /dev/shm
/dev/mapper/xxx-var 50G 25G 26G 50% /etc/hosts
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/mapper/xxx-root 120G 56G 65G 47% /home/xb/project/stor/daos/main/daos
tmpfs 4.0G 500M 3.6G 13% /mnt/daos
dfuse 3.8G 51M 3.7G 2% /tmp/sxb
[root@8b90fa1190f1 daos]#
参考
在docker中搭建开发环境: https://docs.daos.io/v2.0/dev/development/#daos-development-in-docker
DAOS IO全路径详解(视频)
DAOS 项目简介(视频)
晓兵
博客: https://logread.cn | https://blog.csdn.net/ssbandjl | https://cloud.tencent.com/developer/user/5060293/articles
weixin: ssbandjl
公众号: 云原生云
欢迎对DAOS, SPDK, RDMA等高性能技术感兴趣的朋友加入[DAOS技术交流(群)]
本文来自互联网用户投稿，文章观点仅代表作者本人，不代表本站立场，不承担相关法律责任。如若转载，请注明出处。 如若内容造成侵权/违法违规/事实不符，请点击【内容举报】进行投诉反馈！