confd+zookeeper实现LVS节点发现

实现的功能为:当realserver上线时(nginx启动)自动向zookeeper注册节点,通过confd读取zookeeper并自动生成keepalived配置文件并reload keepalived进程。

confd 是一个配置管理工具,根据模板从后端拉取数据,生成配置文件并可以自动检查配置文件的合法性、reload相关进程。支持如下后端服务:

  • etcd
  • consul
  • environment variables
  • redis
  • zookeeper
  • dynamodb

遗憾的是当confd的后端为zookeeper时不支持设置watch标志,不能实时监控。不过etcd和zookeeper类似,可以方便的将zookeeper迁移到etcd.

confd的编译、安装和配置请参考官方教程。

zookeeper注册

这里使用一个简单的c程序来判断nginx进程并注册,可以写入nginx的启动脚本中。当该程序启动时会持续判断nginx进程是否启动,如果启动则向zookeeper注册节点/realserver/$ip$ip为realserver的IP地址。当nginx进程退出时,程序退出,节点销毁。

程序如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <zookeeper/zookeeper.h>
#include <zookeeper/zookeeper_log.h>
void My_watcher_g(zhandle_t* zh, int type, int state,
const char* path, void* watcherCtx)
{
}
void My_string_completion(int rc, const char *name, const void *data)
{
fprintf(stderr,"successfully register node %s\n",name);
}
void monitor()
{
while(! system("ps -ef | grep nginx | grep -q -v grep"))
sleep(2);
exit(1);
}
int main(int argc, const char *argv[])
{
char* host = "192.168.33.130:2181,192.168.33.131:2181";
int timeout = 2000;
char node[30]="/realserver/";
int flag=0;
flag |= ZOO_EPHEMERAL;// flag |= ZOO_SEQUENCE;
if(argc>4 || argc ==1) {
fprintf(stderr,"Usage: %s <realserver ip> <zk server ip> <timeout>\n",argv[0]);
fprintf(stderr,"Default Value: zk server ip=%s\ttimeout=%d\n","192.168.33.130:2181,192.168.33.131:2181",timeout);
exit(1);
}
if(argc<4)
timeout=2000;
else
timeout=atoi(argv[3]);
if(argc<3)
host="192.168.33.130:2181,192.168.33.131:2181";
else
strcpy(host,argv[2]);
strcat(node,argv[1]);
if(system("ps -ef | grep nginx | grep -q -v grep"))
exit(2);
zoo_set_debug_level(ZOO_LOG_LEVEL_WARN);
zhandle_t* zkhandle = zookeeper_init(host,
My_watcher_g, timeout, 0, "hello zookeeper.", 0);
if (zkhandle == NULL) {
fprintf(stderr, "Error when connecting to zookeeper servers...\n");
exit(EXIT_FAILURE);
}
int ret = zoo_acreate(zkhandle,node, "alive", 5,
&ZOO_OPEN_ACL_UNSAFE, flag,
My_string_completion, "zoo_acreate");
if (ret) {
fprintf(stderr, "Error %d for %s\n", ret, "acreate");
exit(EXIT_FAILURE);
}
monitor();
zookeeper_close(zkhandle);
}

需要在程序中修改 zookeeper服务器的地址、超时时间,不过也提供了默认值。
需要向程序提供realserver的监听ip。
运行

1
2
3
[root@localhost zookeeper]# ./a.out
Usage: ./a.out <realserver ip> <zk server ip> <timeout>
Default Value: zk server ip=192.168.33.130:2181,192.168.33.131:2181 timeout=2000

1
2
[root@localhost zookeeper]# ./a.out 192.168.62.138
successfully register node /realserver/192.168.62.138

在zookeeper服务器上可以看到:

1
2
[zk: localhost:2181(CONNECTED) 25] ls /realserver
[192.168.150.129, 192.168.62.138]

confd配置

需要根据keepalived配置模板,自动生成realserver部分的配置。

创建 template resource config /etc/confd/conf.d/myconfig.toml

这部分指定模板文件、产生的模板配置文件、检测的key、reload命令等。

1
2
3
4
5
6
7
[template]
src = "myconfig.conf.tmpl"
dest = "/tmp/myconfig.conf"
keys = [
"/realserver/"
]
reload_cmd = "/etc/init.d/keepalived reload"

创建source template /etc/confd/templates/myconfig.conf.tmpl

这部分为配置模板,由固定部分和可变部分组成,遵从golang语法。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
! Configuration File for keepalived
global_defs {
notification_email {
i@tyr.so
}
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface eth2
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass password
}
virtual_ipaddress {
192.168.33.200
}
}
virtual_server 192.168.33.200 80 {
delay_loop 6
lb_algo rr
lb_kind TUN
persistence_timeout 50
protocol TCP
sorry_server 192.168.33.140 80
{{range $dir :=ls "/realserver/"}}
real_server {{base $dir}} 80 {
weight 1
HTTP_GET {
url {
path /test.html
digest 99d17aaa9df48db49605a368fde7cda9
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
{{end}}
}

于是,每注册一个realserver,将自动产生这部分的配置块,更新keepalived配置,并自动reload keepalived进程。

配置更新

通过在realserver上执行zookeeper注册程序来模拟,如:

1
2
3
[root@localhost zookeeper]# ./a.out 1.1.1.1 &
[root@localhost zookeeper]# ./a.out 2.2.2.2 &
[root@localhost zookeeper]# ./a.out 3.3.3.3 &

当confd检测到配置变化时:

1
2
3
4
5
6
7
root@localhost conf.d]# /root/confd -onetime -backend zookeeper -node 127.0.0.1:2181
2015-08-19T23:08:54+08:00 localhost.localdomain /root/confd[23654]: INFO Backend set to zookeeper
2015-08-19T23:08:54+08:00 localhost.localdomain /root/confd[23654]: INFO Starting confd
2015-08-19T23:08:54+08:00 localhost.localdomain /root/confd[23654]: INFO Backend nodes set to 127.0.0.1:2181
2015-08-19T23:08:54+08:00 localhost.localdomain /root/confd[23654]: INFO /etc/keepalived/keepalived.conf has md5sum 14e54bfe21e32873027726ec2859742e should be 4b869dadb547c687d13129a73ceb0cc9
2015-08-19T23:08:54+08:00 localhost.localdomain /root/confd[23654]: INFO Target config /etc/keepalived/keepalived.conf out of sync
2015-08-19T23:08:54+08:00 localhost.localdomain /root/confd[23654]: INFO Target config /etc/keepalived/keepalived.conf has been updated

会自动更新配置文件并reload keepalived进程。通过ipvsdm -Ln可以看到后端的realserver确实被更新了。

更新后的配置文件为:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
virtual_server 192.168.33.200 80 {
delay_loop 6
lb_algo rr
lb_kind TUN
persistence_timeout 50
protocol TCP
sorry_server 192.168.33.140 80
real_server 1.1.1.1 80 {
weight 1
HTTP_GET {
url {
path /test.html
digest 5cf26930ff1fa28f7b0ce102acb38637
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 2.2.2.2 80 {
weight 1
HTTP_GET {
url {
path /test.html
digest 5cf26930ff1fa28f7b0ce102acb38637
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 3.3.3.3 80 {
weight 1
HTTP_GET {
url {
path /test.html
digest 5cf26930ff1fa28f7b0ce102acb38637
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
如果您觉得这篇文章对您有帮助,不妨支持我一下!