阅读量:0
目录
1.5.VRRP：Virtual Router Redundancy Protocol
3.7延迟抢占模式（需要停用全局配置中的vrrp_stric）
3.8组播变单播（需要停用全局配置中的vrrp_stric）
3.10实现 master/master 的 Keepalived 双主架构
高可用集群KEEPALIVED
解决高可用的问题，同样也支持后端的健康检测，加强版的haproxy
一.高可用keepalived介绍
1.1 集群类型
LB：Load Balance 负载均衡 LVS/HAProxy/nginx（http/upstream, stream/upstream） HA：High Availability 高可用集群 数据库、Redis SPoF: Single Point of Failure，解决单点故障 HPC：High Performance Computing 高性能集群1.2 系统可用性
SLA：Service-Level Agreement 服务等级协议（提供服务的企业与客户之间就服务的品质、 水准、性能等方面所达成的双方共同认可的协议或契约） A = MTBF / (MTBF+MTTR） 指标 ：99.9%, 99.99%, 99.999%,99.9999%1.3 系统故障
硬件故障：设计缺陷、wear out（损耗）、非人为不可抗拒因素 、软件故障：设计缺陷 bug1.4 实现高可用
提升系统高用性的解决方案：降低MTTR- Mean Time To Repair(平均故障时间) 解决方案：建立冗余机制 active/passive 主/备 active/active 双主 active --> HEARTBEAT --> passive active <--> HEARTBEAT <--> active1.5.VRRP：Virtual Router Redundancy Protocol
虚拟路由冗余协议,解决静态网关单点风险 物理层:路由器、三层交换机 软件层:keepalived1.5.1 VRRP 相关术语
虚拟路由器：Virtual Router 虚拟路由器标识：VRID(0-255)，唯一标识虚拟路由器 VIP：Virtual IP VMAC：Virutal MAC (00-00-5e-00-01-VRID) 物理路由器： master：主设备 backup：备用设备 priority：优先级1.5.2 VRRP 相关技术
通告：心跳，优先级等；周期性 工作方式：抢占式，非抢占式 安全认证： 无认证 简单字符认证：预共享密钥 MD5 工作模式： 主/备：单虚拟路由器 主/主：主/备（虚拟路由器1），备/主（虚拟路由器2）
二.keepalived安装和配置文件
2.1 Keepalived 安装
[root@KA1 ~]# dnf install keepalived -y [root@KA1 ~]# systemctl start keepalived [root@KA1 ~]# ps axf | grep keepalived 2385 pts/0 S+ 0:00 \_ grep --color=auto keepalived 2326 ? Ss 0:00 /usr/sbin/keepalived -D 2327 ? S 0:00 \_ /usr/sbin/keepalived -D 配置文件：/etc/keepalived/keepalived.conf 配置文件组成 GLOBAL CONFIGURATION Global definitions： 定义邮件配置，route_id，vrrp配置，多播地址等 VRRP CONFIGURATION VRRP instance(s)： 定义每个vrrp虚拟路由器 LVS CONFIGURATION Virtual server group(s) Virtual server(s)： LVS集群的VS和RS 用户空间核心组件： vrrp stack：VIP消息通告 checkers：监测real server system call：实现 vrrp 协议状态转换时调用脚本的功能 SMTP：邮件组件 IPVS wrapper：生成IPVS规则 Netlink Reflector：网络接口 WatchDog：监控进程 控制组件：提供keepalived.conf 的解析器，完成Keepalived配置 IO复用器：针对网络目的而优化的自己的线程抽象 内存管理组件：为某些通用的内存管理功能（例如分配，重新分配，发布等）提供访问权限三.keepalived实验
3.1环境配置
keep1 ：172.25.254.10 keep2 ：172.25.254.20 rserver1:172.25.254.110 rserver2:172.25.254.120 #rserver1 [root@rserver1 ~]# yum install httpd -y [root@rserver1 ~]# echo 172.25.254.110 > /var/www/html/index.html [root@rserver1 ~]# systemctl enable --now httpd #rserver2 [root@rserver2 ~]# yum install httpd -y [root@rserver2 ~]# echo 172.25.254.120 > /var/www/html/index.html [root@rserver2 ~]# systemctl enable --now httpd3.2keepalived虚拟路由器
[root@keep1 ~]# yum install keepalived -y [root@keep1 ~]# vim /etc/keepalived/keepalived.conf global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 #邮件服务器地址 smtp_connect_timeout 30 router_id keep1.timinglee.org #每个keepalived主机唯一标识#建议使用当前主机名，但多节点重名不影响 vrrp_skip_check_adv_addr #启用此配置后，如果收到的通告报文和上一个报文是同一 个路由器，则跳过检查，默认值为全检查 vrrp_strict vrrp_garp_interval 0 #报文发送延迟，0表示不延迟 vrrp_gna_interval 0 #消息发送延迟 vrrp_mcast_group4 224.0.0.18 # #指定组播IP地址范围 } vrrp_instance VI_1 { state MASTER interface eth0 #绑定为当前虚拟路由器使用的物理接口，如：eth0,可以和VIP不在一个网卡 virtual_router_id 100 #每个虚拟路由器惟一标识,范围：0-255，每个虚拟路由器此值必须唯一 #否则服务无法启动 #同属一个虚拟路由器的多个keepalived节点必须相同 #务必要确认在同一网络中此值必须唯一 priority 100 #值越大优先级越高,每个keepalived主机节点此值不同 advert_int 1 #vrrp通告的时间间隔，默认1s authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 #用于定义虚拟IP地址（VIP），这是高可用性（HA）解决方案中常见的做法，以确保在服务器故障时，服务能够通过另一个服务器上的相同IP地址继续提供。 } } [root@keep1 ~]# enable --now keepalived.service [root@keep1 ~]# restart keepalived.service [root@keep2 ~]# yum install keepalived -y [root@keep2 ~]# vim /etc/keepalived/keepalived.conf global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id keep1.timinglee.org vrrp_skip_check_adv_addr vrrp_strict vrrp_garp_interval 0 vrrp_gna_interval 0 vrrp_mcast_group4 224.0.0.18 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 100 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } } [root@keep2 ~]# tcpdump -i eth0 -nn host 224.0.0.18 11:02:40.120434 IP 172.25.254.10 > 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20 ######## [root@keep1 ~]# systemctl stop keepalived.service [root@keep2 ~]# tcpdump -i eth0 -nn host 224.0.0.18 11:01:58.059105 IP 172.25.254.20 > 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 80, authtype simple, intvl 1s, length 20 3.3如何ping通虚拟vip
在keep1和keep2主机里面的global全局参数中添加 vrrp_iptables，重启服务可以ping通vip了；也可以#vrrp_strict vim /etc/keepalived/keepalived.conf global_defs { vrrp_strict vrrp_iptables }3.4独立子配置文件
[root@keep1 ~]# vim /etc/keepalived/keepalived.conf include "/etc/keepalived/conf.d/*.conf" [root@keep1 ~]# mkdir -p /etc/keepalived/conf.d/ [root@keep1 ~]# vim /etc/keepalived/conf.d/keep1.conf vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 100 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } } [root@keep1 ~]# systemctl restart keepalived.service 3.5独立日志
[root@keep1 ~]# vim /etc/sysconfig/keepalived KEEPALIVED_OPTIONS="-D -S 6" [root@keep1 ~]# vim /etc/rsyslog.conf local6.* /var/log/keepalived.log [root@keep1 ~]# systemctl restart keepalived.service [root@keep1 ~]# systemctl restart rsyslog.service [root@keep1 ~]# ll /var/log/keepalived.log -rw------- 1 root root 724 8月 12 14:02 /var/log/keepalived.log
3.6非抢占式优先级
默认为抢占模式preempt，即当高优先级的主机恢复在线后，会抢占低先级的主机的master角色， 这样会使vip在KA主机中来回漂移，造成网络抖动， 建议设置为非抢占模式 nopreempt ，即高优先级主机恢复后，并不会抢占低优先级主机的master角色 非抢占模块下,如果原主机down机, VIP迁移至的新主机, 后续也发生down时,仍会将VIP迁移回原主机 注意：要关闭 VIP抢占，必须将各 keepalived 服务器state配置为BACKUP [root@keep1 ~]# vim /etc/sysconfig/keepalived vrrp_instance VI_1 { state BACKUP #两台都要修改为backup模式 nopreempt #非抢占优先级 virtual_router_id 100 priority 100 } [root@keep1 ~]# systemctl restart keepalived.service [root@keep2 ~]# vim /etc/sysconfig/keepalived vrrp_instance VI_1 { state BACKUP #两台都要修改为backup模式 nopreempt #非抢占优先级 interface eth0 virtual_router_id 100 priority 80 } [root@keep2 ~]# systemctl restart keepalived.service 3.7延迟抢占模式（需要停用全局配置中的vrrp_stric）
抢占延迟模式，即优先级高的主机恢复后，不会立即抢回VIP，而是延迟一段时间（默认300s）再抢回 VIP 注意：需要各keepalived服务器state为BACKUP,并且不要启用 vrrp_strict [root@keep1 ~]# vim /etc/sysconfig/keepalived vrrp_instance VI_1 { state BACKUP #修改为backup preempt_delay 5s #恢复后，延迟5s interface eth0 virtual_router_id 100 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } } [root@keep1 ~]# systemctl restart keepalived.service [root@keep2 ~]# vim /etc/sysconfig/keepalived vrrp_instance VI_1 { state BACKUP #修改为backup preempt_delay 5s #恢复后，延迟5s interface eth0 virtual_router_id 100 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } } [root@keep2 ~]# systemctl restart keepalived.service 3.8组播变单播（需要停用全局配置中的vrrp_stric）
[root@keep1 ~]# vim /etc/sysconfig/keepalived vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 100 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } unicast_src_ip 172.25.254.10 #广播源地址ip unicast_peer { #接受广播端ip 172.25.254.20 } } [root@keep1 ~]# systemctl restart keepalived.service [root@keep1 ~]# tcpdump -i eth0 -nn src host 172.25.254.20 and dst 172.25.254.10 #在vip在本机的时候 [root@keep2 ~]# vim /etc/sysconfig/keepalived vrrp_instance VI_1 { state BACKUP #preempt_delay 5s interface eth0 virtual_router_id 100 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } unicast_src_ip 172.25.254.20 #广播源地址ip unicast_peer { #接受广播端ip 172.25.254.10 } } [root@keep2 ~]# systemctl restart keepalived.service [root@keep1 ~]# tcpdump -i eth0 -nn src host 172.25.254.20 and dst 172.25.254.10
3.9实现Keepalived状态切换的通知脚本
[root@keep1 ~]#dnf install mailx -y [root@keep1 ~]#vim /etc/mail.rc set from=2784117361@qq.com set smtp=smtp.qq.com set smtp-auth-user=2784117361@qq.com set smtp-auth-password=gjtqeiajudzldfdd set smtp-auth=login set ssl-verify=ignore ~ [root@keep1 ~]# vim /etc/keepalived/mail.sh #!/bin/bash mail_who=2784117361@qq.com hostname='keep1' date1=$(date +'%F %T') mail_send() { mail_subj="$hostname to be $1 vip 转移" mail_mess="$date1发生了vrrp 转移，$hostname 变为$1" echo "$mail_mess" | mail -s "$mail_subj" $mail_who } case $1 in master) mail_send master ;; backup) mail_send backup ;; fault) mail_send fault ;; *) ;; esac [root@keep1 ~]# vim /etc/keepalived/keepalived.conf vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 100 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } notify_master "/etc/keepalived/mail.sh master" notify_backup "/etc/keepalived/mail.sh backup" notify_fault "/etc/keepalived/mail.sh fault" unicast_src_ip 172.25.254.10 unicast_peer { 172.25.254.20 } } [root@keep1 ~]# systemctl restart keepalived.service3.10实现master/master的Keepalived双主架构
[root@keep1 ~]# vim /etc/keepalived/keepalived.conf vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 100 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } unicast_src_ip 172.25.254.10 unicast_peer { 172.25.254.20 } } vrrp_instance VI_2 { state BACKUP interface eth0 virtual_router_id 200 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.200/24 dev eth0 label eth0:2 } unicast_src_ip 172.25.254.10 unicast_peer { 172.25.254.20 } } [root@keep1 ~]# systemctl restart keepalived.service [root@keep2 ~]# vim /etc/keepalived/keepalived.conf vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 100 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } unicast_src_ip 172.25.254.20 unicast_peer { 172.25.254.10 } } vrrp_instance VI_2 { state MASTER interface eth0 virtual_router_id 200 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.200/24 dev eth0 label eth0:2 } unicast_src_ip 172.25.254.10 unicast_peer { 172.25.254.20 } } [root@keep2 ~]# systemctl restart keepalived.service 3.11实现单主的LVS-DR模式（keep+lvs）
keepalived服务器1 ######################################################## [root@keep1 ~]# vim /etc/keepalived/keepalived.conf vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 100 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } unicast_src_ip 172.25.254.10 unicast_peer { 172.25.254.20 } } virtual_server 172.25.254.100 80 { delay_loop 6 lb_algo wrr lb_kind DR #persistence_timeout 50 protocol TCP real_server 172.25.254.110 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 172.25.254.120 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } [root@keep1 ~]# systemctl restart keepalived.service ############################################################ keepalived服务器 [root@keep2 ~]# vim /etc/keepalived/keepalived.conf vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 100 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } unicast_src_ip 172.25.254.20 unicast_peer { 172.25.254.10 } } virtual_server 172.25.254.100 80 { delay_loop 6 lb_algo wrr lb_kind DR #persistence_timeout 50 protocol TCP real_server 172.25.254.110 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 172.25.254.120 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } [root@keep2 ~]# systemctl restart keepalived.service ######################################################## [root@rserver1 ~]# ip a a 172.25.254.100/32 dev lo [root@rserver1 ~]# vim /etc/sysctl.d/arp.conf net.ipv4.conf.all.arp_ignore=1 net.ipv4.conf.all.arp_announce=2 net.ipv4.conf.lo.arp_ignore=1 net.ipv4.conf.lo.arp_announce=2 [root@rserver1 ~]# sysctl --system [root@rserver1 ~]# sysctl -p ########################################################### [root@rserver2 ~]# ip a a 172.25.254.100/32 dev lo [root@rserver2 ~]# vim /etc/sysctl.d/arp.conf net.ipv4.conf.all.arp_ignore=1 net.ipv4.conf.all.arp_announce=2 net.ipv4.conf.lo.arp_ignore=1 net.ipv4.conf.lo.arp_announce=2 [root@rserver2 ~]# sysctl --system [root@rserver2 ~]# sysctl -p [root@rserver2 ~]# sysctl --system 3.12利用keepalived实现HAProxy高可用
HAProxy：HAProxy是一个高性能的TCP/HTTP反向代理服务器和负载均衡器。在这个实验中，HAProxy被配置为监听所有发往172.25.254.100（虚拟IP，VIP）的HTTP请求，并将这些请求以轮询（roundrobin）的方式分发到后端的两台Web服务器（172.25.254.110和172.25.254.120）上。 Keepalived：Keepalived用于实现服务器的健康检查和故障转移。它通过VRRP（Virtual Router Redundancy Protocol）协议来确保服务的高可用性。在这个配置中，Keepalived监控HAProxy的健康状态，如果HAProxy出现故障，则可以将VIP转移到另一台健康的服务器上。 VIP（虚拟IP）：VIP是浮动的，不直接绑定在任何物理服务器上。它根据Keepalived的配置和状态，动态地绑定到当前的MASTER服务器上。这样，无论MASTER服务器是否发生故障，客户端都可以通过VIP访问到后端的Web服务。 健康检查：HAProxy和Keepalived都配置了健康检查。HAProxy通过check指令定期检查后端服务器的状态；Keepalived则通过执行自定义脚本（如/etc/keepalived/haproxy.sh）来检查HAProxy的运行状态。 #rserver1 [root@rserver1 ~]# yum install httpd -y [root@rserver1 ~]# echo 172.25.254.110 > /var/www/html/index.html [root@rserver1 ~]# systemctl enable --now httpd #rserver2 [root@rserver2 ~]# yum install httpd -y [root@rserver2 ~]# echo 172.25.254.120 > /var/www/html/index.html [root@rserver2 ~]# systemctl enable --now httpd ######################################################################################### #keep1 [root@keep1 ~]# vim /etc/sysctl.conf net.ipv4.ip_nonlocal_bind=1 [root@keep1 ~]# sysctl -p [root@keep1 ~]# yum install haproxy -y [root@keep1 ~]# vim /etc/haproxy/haproxy.cfg listen webserver bind 172.25.254.100:80 mode http balance roundrobin server web1 172.25.254.110:80 check inter 2 fall 3 rise 5 weight 1 server web2 172.25.254.120:80 check inter 2 fall 3 rise 5 weight 1 [root@keep1 ~]# systemctl restart haproxy.service vrrp_script check_haproxy { script "/etc/keepalived/haproxy.sh" interval 1 weight -30 fall 2 rise 2 timeout 2 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 100 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } track_script { check_haproxy } unicast_src_ip 172.25.254.10 unicast_peer { 172.25.254.20 } } [root@keep1 ~]# systemctl restart keepalived.service ############################################################################################ #keep2 [root@keep2 ~]# vim /etc/sysctl.conf net.ipv4.ip_nonlocal_bind=1 #因为我们使用keepalived，vip会到两台keep服务器，指定这个参数，可以实现没有vip也可以调用haproxy [root@keep2 ~]# sysctl -p [root@keep2 ~]# yum install haproxy -y [root@keep2 ~]# vim /etc/haproxy/haproxy.cfg listen webserver bind 172.25.254.100:80 mode http balance roundrobin server web1 172.25.254.110:80 check inter 2 fall 3 rise 5 weight 1 server web2 172.25.254.120:80 check inter 2 fall 3 rise 5 weight 1 [root@keep2 ~]# systemctl restart haproxy.service [root@keep2 ~]# vim /etc/keepalived/haproxy.sh #!/bin/bash killall -0 haproxy [root@keep2 ~]# chmod +x /etc/keepalived/haproxy.sh [root@keep2 ~]# vim /etc/keepalived/keepalived.conf vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 100 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } track_script { check_haproxy } unicast_src_ip 172.25.254.20 unicast_peer { 172.25.254.10 } } [root@keep2 ~]# systemctl restart keepalived.service 