- 博客/
LVS DR 负载均衡实现
上篇关于 LVS-nat 模式的负载均衡实现博客中已经简单介绍了有关LVS集群的概念、四种IP负载均衡技术以及十种调度算法，本文将详细介绍LVS-DR(直接路由)模式的实现
LVS-DR：Direct Routing, LVS默认模式, 通过为请求报文重新封装一个MAC首部进行转发，源MAC是DIP所在的接口的MAC，目标MAC是某挑选出的RS的RIP所在接口的MAC地址；源IP/PORT，以及目标IP/PORT均保持不变。请求报文同样要经由 Director，但与nat不同的是DR模式下的响应报文不经由 Director，而由RS直接发往Client
DR模式调度流程#
网络传输时，经过路由转发，会修改源mac为该转发路由的mac
Director和各RS都配置有VIP
确保RS响应报文中源IP为VIP，进而走直接路由返回client
解决地址冲突, 确保前端路由器将目标IP为VIP的请求报文发往Director,而不是RS，三种方法：
在前端网关做静态绑定VIP和Director的MAC地址
在RS上使用arptables工具
arptables -A IN -d VIP -j DROP arptables -A OUT -s VIP -j mangle –mangle -ip -s RIP
在各RS修改内核参数，来限制arp响应和通告的级别
限制响应级别： arp_ignore
1 –> 仅在请求的目标IP配置在本地主机的接收到请求报文的接口上时，才给予响应
eg:
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore限制通告级别：arp_announce
2 –> 必须避免将接口信息向非本网络进行通告
eg:
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
HTTP服务负载均衡实验#
网络拓扑搭建#
使用VMware搭建实验网络拓扑
VIP与RIP可以在一个网段也可以不在同一网段。此实验中，VIP是与RIP在同一网段，前端路由临近的接口只用配一个地址，否则要额外配置一个vip网段的地址
调度器Director要打开核心转发功能
$ echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf如果要启用sorry server, VS 的网关要指向前端路由临近的接口；否则，指向RIP所在网段任意IP皆可
RS 网关要指向前端路由
配置RS httpd服务并启动
RS1 –>
$ echo “RS1 SRV” > /var/www/html/index.htmlRS2 –>
$ echo “RS2 SRV” > /var/www/html/index.html
脚本实现RS配置
$ bash lvs_dr_rs.sh start
#!/bin/bash
vip=192.168.196.186
dev=lo:1 <-- 配置VIP在回环lo的别名上
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask 255.255.255.255 broadcast $vip up
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac
DIRECTOR上管理集群服务并测试#
脚本实现集群服务管理
$ bash lvs_dr_vs.sh start
#!/bin/bash
vip=192.168.196.186
port=80
service=$vip:$port
dev=ens33:1
RS1=192.168.196.155
RS2=192.168.196.196
scheduler=rr
tp="-g"
case $1 in
start)
ifconfig $dev $vip/32 broadcast $vip up
ipvsadm -A -t $service -s $scheduler
ipvsadm -a -t $service -r $RS1 $tp -w 1
ipvsadm -a -t $service -r $RS2 $tp -w 1
;;
stop)
ipvsadm -C
ifconfig $dev down
;;
*)
echo "Usage $(basename $0) start|stopr";exit 1
;;
esac
查看当前状态
$ ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.196.186:80 rr
-> 192.168.196.155:80 Route 1 0 0
-> 192.168.196.196:80 Route 1 0 1
客户端测试
$ for i in {1..10}; do curl 192.168.196.186;sleep 1;done
RS2 SRV
RS1 SRV
RS2 SRV
RS1 SRV
RS2 SRV
RS1 SRV
RS2 SRV
RS1 SRV
RS2 SRV
RS1 SRV
ldirectord工具实现高可用的lvs#
ldirectord：监控和控制LVS守护进程，可管理LVS规则 rpm包 –> ldirectord-3.9.6-0rc1.1.1.x86_64.rpm
$ rpm -ql ldirectord
/etc/ha.d
/etc/ha.d/resource.d
/etc/ha.d/resource.d/ldirectord
/etc/logrotate.d/ldirectord
/usr/lib/ocf/resource.d/heartbeat/ldirectord
/usr/lib/systemd/system/ldirectord.service
/usr/sbin/ldirectord
/usr/share/doc/ldirectord-3.9.6
/usr/share/doc/ldirectord-3.9.6/COPYING
/usr/share/doc/ldirectord-3.9.6/ldirectord.cf <-- 配置文件模板
/usr/share/man/man8/ldirectord.8.gz
$cp /usr/share/doc/ldirectord-3.9.6/ldirectord.cf /etc/ha.d
修改配置文件#
$ vim /etc/ha.d/ldirectord.cf
checktimeout=3 <-- 响应超时时间
checkinterval=1 <-- 检查间隔时间
#fallback=127.0.0.1:80
#fallback6=[::1]:80
autoreload=yes <-- 自动读取加载配置文件
logfile="/var/log/ldirectord.log"
#logfile="local0" <-- 指定rsyslog日志服务的facility名称
#emailalert="admin@x.y.z"
#emailalertfreq=3600
#emailalertstatus=all
quiescent=no <-- no代表一旦宕机则从集群中移除RS；yes代表一旦宕机则将该RS权重设置为0
# Sample for an http virtual service <-- Director Server的配置(以上文实验配置为例)
virtual=192.168.196.186:80
real=192.168.196.155:80 gate <-- gate 对应-g dr模式
real=192.168.196.196:80 gate
fallback=127.0.0.1:80 gate <-- sorry server; RS全部宕机时的响应
service=http
scheduler=rr
#persistent=600
#netmask=255.255.255.255
protocol=tcp
checktype=negotiate
checkport=80
request="index.html"
receive="SRV" <-- 抓取网页关键字，判断
$ echo "Sorry, server is down" > /var/www/html/index.html
$ systemctl start httpd <-- 启动调度服务器的http服务
启动ldirector服务#
$ ifconfig ens33:1 192.168.196.186/32 broadcast 192.168.196.186 up <-- VS上手动添加VIP
$ systemctl start ldirectord
$ ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.196.186:80 rr
-> 192.168.196.155:80 Route 1 0 0
-> 192.168.196.196:80 Route 1 0 0
客户端测试#
- lvs集群已经实现轮询算法的调度
$ for i in {1..5}; do curl --connect-timeout 1 192.168.196.186;sleep 1; done
RS2 SRV
RS1 SRV
RS2 SRV
RS1 SRV
RS2 SRV
停止RS1的http服务模拟宕机
VS上查看集群服务状态
$ ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.196.186:80 rr
-> 192.168.196.196:80 Route 1 0 0 <--自动移除宕机的RS1
客户端测试显示RS1宕机后，DR将请求自动调度到RS2上
$ for i in {1..5}; do curl --connect-timeout 1 192.168.196.186;sleep 1; done
RS2 SRV
RS1 SRV
RS2 SRV
RS2 SRV <-- 此时RS1 http服务停止
RS2 SRV
停止所有RS的http服务
VS上查看集群服务状态
$ ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.196.186:80 rr
-> 127.0.0.1:80 Route 1 0 0 <--自动启用本机sorry server
客户端测试结果
[root@Centos7 ~]#for i in {1..100}; do curl --connect-timeout 1 192.168.196.186;sleep 1; done
RS1 SRV
RS2 SRV
RS1 SRV
RS2 SRV
RS1 SRV
RS1 SRV <--RS2宕机
RS1 SRV
sorry,server is down <--RS1同时宕机，启用sorry server
sorry,server is down