- 博客/
P2P文件与镜像分发开源解决方案Dragonfly安装部署
·859 字·5 分钟
Docker
前言#
随着公司容器管理平台部署的系统越来越多，所纳管集群主机也越来越多，面对大规模的文件和镜像分发，由于单一文件服务器、镜像仓库实例会受到部署主机的网络带宽限制，导致分发耗时长、成功率低的问题。利用P2P技术搭建文件和镜像的分发系统，文件和镜像可在集群主机之间共享传输，可有效解决上述问题。
Dragonfly简介#
阿里开源，托管在github上 https://github.com/dragonflyoss/Dragonfly
核心组件： 1) cluster manger: 超级节点（supernode）集群 - 管理peer节点注册 - 负责CDN，从源文件服务器下载文件并生成数据块存储 - 调度peer节点在P2P网络中选择合适的下载源 2) dfget proxy： dfget是p2p客户端，部署在peer节点上，负责数据块的下载和共享。在镜像分发系统中也叫dfdaemon
Dragonfly镜像分发的工作模式: 1) dfdaemon 拦截peer节点docker pull的HTTP请求，从镜像仓库服务器获取到镜像manifest文件后，使用dfget向supernode集群请求下载镜像的各layer文件 2) supernode从镜像仓库6下载镜像layer文件并生成数据块 3) peer节点通过dfget下载数据块，下载完成后supernode调度后续peer节点以P2P方式进行数据块下载 4) 所有layer文件通过dfget下载完成后，docker进程根据manifests文件整合为本地镜像，并按照镜像规范保存各镜像layer
Dragonfly镜像分发系统的搭建#
一、节点规划
| IP | roles | services |
|---|---|---|
| 192.168.196.134 | supernode,registry,webserver | docker,supernode,docker-distribution,nginx |
| 192.168.196.2 | supernode | docker,supernode |
| 192.168.196.3 | peer | docker |
二、下载源码 最近一次release的版本0.2.0，在节点192.168.196.134上执行编译。
$ mkdir /data/dragonfly
$ cd /data/dragonfly
$ wget https://github.com/dragonflyoss/Dragonfly/archive/v0.2.0.tar.gz -O Dragonfly-0.2.0.tar.gz
$ tar xf Dragonfly-0.2.0.tar.gz
三、编译安装supernode
3.1 编译环境准备
supernode.jar包编译依赖maven (yum install maven)，镜像定制依赖docker服务
3.2 修改默认注册端口8001为48001及默认下载服务端口8002为48002
$cd /data/dragonfly/Dragonfly-0.2.0/src/supernode/src/main/java/com/alibaba/dragonfly/supernode/common
$ sed -i 's/8001/48001/' Constants.java
$ cd /data/dragonfly/Dragonfly-0.2.0/src/supernode/src/main/docker/source
$ cat nginx.conf
......
server {
listen 48001;
location / {
root /home/admin/supernode/repo;
}
}
server {
listen 48002;
location /peer {
proxy_pass http://127.0.0.1:8080;
}
......
3.3 修改Dockerfile及配置文件
本例中主机docker版本为1.13不支持多阶段定制，需要修改Dockerfile文件
新建config.properties配置文件， 参考官方文档 https://d7y.io/user_guide/supernode_configuration/
$ cd /data/dragonfly/Dragonfly-0.2.0/src/supernode/src/main/docker
$ cat Dockerfile
#私有centos7.2镜像(已配置阿里yum源)
FROM 192.168.196.134:5000/library/centos7.2:01
COPY sources /tmp/sources
RUN yum install -y epel-release && \
yum install -y -q telnet nginx java-1.8.0-openjdk.x86_64 && \
cp /tmp/sources/start.sh /root/start.sh && \
cp /tmp/sources/nginx.conf /etc/nginx/nginx.conf && \
cp /tmp/sources/config.properties /root/config.properties &&\
mkdir -p /home/admin/supernode/bin && \
cp /tmp/sources/start.sh /home/admin/supernode/bin/ && \
yum clean all && \
rm -rf /var/cache/yum /tmp/* && \
history -c
ADD supernode.jar supernode.jar
EXPOSE 48001 48002
ENTRYPOINT ["/root/start.sh"]
$ cat source/config.properties
supernode.cluster[0].ip = '192.168.196.134'
supernode.cluster[0].registerPort = 48001
supernode.cluster[1].ip = '192.168.196.2'
supernode.cluster[1].registerPort = 48001
$ cat source/start.sh
#!/usr/bin/env bash
nginx
java -Dspring.config.location=/root/config.properties -Djava.security.egd=file:/dev/./urandom -jar supernode.jar
3.4 编译supernode
参考官方文档 https://d7y.io/user_guide/install_server/
$ cd /data/dragonfly/Dragonfly-0.2.0
$ ./build/build.sh supernode
$ docker images supernode
REPOSITORY TAG IMAGE ID CREATED SIZE
supernode 0.2.0 aac72150e4a0 2 minutes ago 493 MB
3.5 启动supernode服务
$ docker run -d -p 48001:48001 -p 48002:48002 supernode:0.2.0
394df40d13eded44f78e943b49bbd233bf19258f777c0c42586ae969db39f142
$ docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bafcf15d15ae supernode:0.2.0 "/root/start.sh" 30 seconds ago Up 26 seconds 0.0.0.0:48001-48002->48001-48002/tcp stupefied_meninsky
$ telnet localhost 48001
Trying ::1...
Connected to localhost.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
$ telnet localhost 48002
Trying ::1...
Connected to localhost.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
# 将supernode镜像推到私有仓库
$ docker tag supernode:0.2.0 192.168.196.134:5000/library/supernode:0.2.0
$ docker push 192.168.196.134:5000
# 在192.168.196.2上pull镜像并启动supernode实例
$ docker pull 192.168.196.134:5000/library/supernode:0.2.0
$ docker run -d --net host 192.168.196.134:5000/library/supernode:0.2.0
1c9bef1f4c14f4427291ecff793bef6665a617c71e10cb13c481d56763430b2d
$ ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 128 *:48001 *:*
LISTEN 0 128 *:48002 *:*
LISTEN 0 100 :::8080 :::*
LISTEN 0 128 :::22 :::*
四、编译安装df-client
4.1 编译环境准备
依赖go环境，需要安装golint
$ yum install -y make go
#默认GOPATH
$ cd /root/go/src
#GFW下的安装方法
$ git clone https://github.com/golang/tools.git tools
$ git clone https://github.com/golang/lint.git
$ mkdir -p golang.org/x
$ cp github.com/golang/lint/ golang.org/x/ -a
$ go install /root/go/src/golang.org/x/lint/golint
$ cp /root/go/bin/golint /usr/local/sbin/
4.2 修改dfget请求supernode下载端口8002为48002
$ cd/data/dragonfly/Dragonfly-0.2.0/src/getter
$ sed -i 's/8002/48002/' env.py
$ sed -i 's/8002/48002/' component/httputil.py
#core/server.py中有关peer节点数据块存储时间的代码，默认180s
#98 expire_time = 180
#135 alive_qu.get(True, 5 * 60.0)
4.3 编译df-client
参考文档 https://d7y.io/user_guide/install_client/
$ cd /data/dragonfly/Dragonfly-0.2.0/build/client
$ ./configure --prefix=/data/dragonfly
$ make && make install
# 将编译好的df-client，打包传到192.168.196.3节点上
$ cd /data/dragonfly
$ tar cf df-client.tar df-client/
$ scp df-client.tar 192.168.196.3:/data/
4.4 在peer节点192.168.196.3安装df-client
1) 新建/etc/dragonfly.conf配置文件，指定要注册的supernode集群
$ cat /etc/dragonfly.conf
[node]
address=192.168.196.2,192.168.196.134
2) 修改docker daemon.json文件
# 配置registry-mirrors
$ cat /etc/docker/daemon.json
{
"insecure-registries": ["0.0.0.0/0"],
"registry-mirrors": ["http://127.0.0.1:65001"]
}
$ systemctl restart docker.service
3) 启动dfdaemon
$ export PATH=/data/df-client:$PATH
$ nohup dfdaemon --registry 192.168.196.134:5000 &
$ cat nohup.out
launch dfdaemon on port:65001
$ ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 128 :::22 :::*
LISTEN 0 128 :::65001 :::*
$ cd /root/.small-dragonfly/logs
$ cat dfdaemon.log
time="2018-09-12 21:10:00" level=info msg=init...
time="2018-09-12 21:10:00" level=info msg="dfget version:0.2.0\n"
time="2018-09-12 21:10:00" level=info msg="init finish"
time="2018-09-12 21:10:00" level=info msg="start dfdaemon param:&{DfPath:/data/df-client/dfget DFRepo:/root/.small-dragonfly/dfdaemon/data/ RateLimit:20M CallSystem:com_ops_dragonfly URLFilter:Signature&Expires&OSSAccessKeyId Notbs:true MaxProcs:4 Version:false Verbose:false Help:false HostIP:127.0.0.1 Port:65001 Registry:192.168.196.134:5000 DownRule: CertFile: KeyFile:}"
五、镜像分发测试
私有镜像仓库镜像：
192.168.196.134:5000/library/supernode:0.2.0 192.168.196.134:5000/testnasp/nginx:0.1
在peer节点上 pull镜像：
$ docker pull testnasp/nginx:0.1
Trying to pull repository docker.io/testnasp/nginx ...
0.1: Pulling from docker.io/testnasp/nginx
ed070c36b93c: Pull complete
07e00549345e: Pull complete
c68fedf2fd94: Pull complete
Digest: sha256:2ff1929d16fb3df64564eeb91686bb07e585ed365ab3601e1fed164ff3dc4bc4
Status: Downloaded newer image for docker.io/testnasp/nginx:0.1
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/testnasp/nginx 0.1 cd5239a0906a 6 months ago 109 MB
# pull镜像时可以查看dfget日志
$ cd /root/.small-dragonfly/logs
$ tailf dfdaemon.log
time="2018-09-12 21:45:00" level=info msg="start download url:http://192.168.196.134:5000/v2/testnasp/nginx/blobs/sha256:07e00549345efcbf93e50fc43440a05315c3ed049c28b7e7b709c1515b6550e0 to f5da0b91-b417-4f90-8a66-7ab99c5f581b in repo"
time="2018-09-12 21:45:00" level=info msg="start download url:http://192.168.196.134:5000/v2/testnasp/nginx/blobs/sha256:ed070c36b93c24fc135fd9541fc687a8fb0664192a6c8e3397644e365944221a to 0afb237f-c0d1-4ee4-a8d0-ec43cfe69109 in repo"
time="2018-09-12 21:45:00" level=info msg="dfget url:http://192.168.196.134:5000/v2/testnasp/nginx/blobs/sha256:07e00549345efcbf93e50fc43440a05315c3ed049c28b7e7b709c1515b6550e0 [SUCCESS] cost:20s"
time="2018-09-12 21:45:00" level=info msg="dfget url:http://192.168.196.134:5000/v2/testnasp/nginx/blobs/sha256:ed070c36b93c24fc135fd9541fc687a8fb0664192a6c8e3397644e365944221a [SUCCESS] cost:22s"
$ cat dfclient.log
[2018-09-12 21:45:45,160] INFO sign:1719-1544622329.923 lineno:51 : cmd params:Namespace(callsystem='com_ops_dragonfly', console=False, dfdaemon=True, filter='Signature&Expires&OSSAccessKeyId', header=['User-Agent:docker/1.13.1 go/go1.8.3 kernel/3.10.0-693.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))', 'Accept-Encoding:identity', 'X-Forwarded-For:127.0.0.1'], identifier=None, locallimit='20M', md5=None, node=None, notbs=True, output='/root/.small-dragonfly/dfdaemon/data/f5da0b91-b417-4f90-8a66-7ab99c5f581b', pattern=None, showbar=False, timeout=None, totallimit='20M', url='http://192.168.196.134:5000/v2/testnasp/nginx/blobs/sha256:07e00549345efcbf93e50fc43440a05315c3ed049c28b7e7b709c1515b6550e0', version=False),python version:sys.version_info(major=2, minor=7, micro=5, releaselevel='final', serial=0)
...中间省略...
[2018-09-12 21:45:50,965] INFO sign:1720-1544622330.044 lineno:109 : move src:/root/.small-dragonfly/data/0afb237f-c0d1-4ee4-a8d0-ec43cfe69109-1720-1544622330.044 to dst:/root/.small-dragonfly/dfdaemon/data/0afb237f-c0d1-4ee4-a8d0-ec43cfe69109 result:True cost:0.057
[2018-09-12 21:45:50,965] INFO sign:1720-1544622330.044 lineno:195 : download successfully from dragonfly
[2018-09-12 21:45:51,003] INFO sign:1720-1544622330.044 lineno:60 : local http result:success for path:/client/ and cost:0.002
[2018-09-12 21:45:51,003] INFO sign:1720-1544622330.044 lineno:94 : |7773d958cc784318c36095a405f4aa918d0c7ca45f7cad12c3e08c68fbca68fb|http://192.168.196.134:5000/v2/testnasp/nginx/blobs/sha256:ed070c36b93c24fc135fd9541fc687a8fb0664192a6c8e3397644e365944221a|22496029|22496059|192.168.196.134|com_ops_dragonfly|20.957|
[2018-09-12 21:45:51,003] INFO sign:1720-1544622330.044 lineno:111 : download SUCCESS cost:20.959s length:22496029 reason:0
Tips：
要分发的镜像命名空间是library时，要注意peer节点pull下的镜像名字
$ docker pull library/supernode:0.2.0
Trying to pull repository docker.io/library/supernode ...
0.2.0: Pulling from docker.io/library/supernode
89ac44988659: Already exists
58f1be4f44ca: Already exists
8de7fe9b2274: Pull complete
adfad6e78041: Pull complete
802d3ffa4c82: Pull complete
Digest: sha256:4b49d1655f518be7fa5b450df887a54519a57a462ca289ea21ab3d9e54fc846c
Status: Downloaded newer image for docker.io/supernode:0.2.0
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/supernode 0.2.0 aac72150e4a0 5 days ago 493 MB
dfdaemon也可以容器化部署，使用host模式运行即可。还有要注意,目前一个dfdaemon只能指定一个仓库地址。
Related
Docker配置TLS安全认证
·516 字·3 分钟
Docker
CA
Openssl
批量export Docker镜像脚本
·395 字·2 分钟
Docker
Docker存储驱动direct-lvm的配置
·518 字·3 分钟
Docker
Direct-Lvm