Puppet是当前devops中常用于管理系统配置和应用部署，多数会使用其C/S架构的方式来进行部署，其中puppetmaster是集群中配置管理的核心节点。在实际的生产环境中，如果因为master节点性能不够或者发生意外宕机，可能会影响到实际业务，因此维护一个高可用和可扩展的puppetmaster池子是一个首要任务。
这里我使用了一种常规的方案：前端使用apache/nginx做负载均衡，使用packmaker/keepalived来做健康检查和故障切换，来做HA，后端起多个puppetmaster实例做横向扩展，来提高处理能力。
方案验证
这里，我将在在每台Master Node上起两个puppetmaster实例，前端使用Apache作负载均衡，keepalived做健康检查。唯一的难点是证书同步问题，在部署中将把我们将证书设成自动认证，只接受fqdn是*.clustername.ustack.com的机器，就不需同步证书了。
| IP | 主机名 | 角色 | vip |
|---|---|---|---|
| 192.168.1.53 | ha-puppet1.ustack.com | puppet master | 192.168.1.103 |
| 192.168.1.54 | ha-puppet2.ustack.com | puppet master | 192.168.1.104 |
配置细节
客户端 配置文件
[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates
pluginsync=true[agent]
server = ha-puppet.ustack.com
report = true
pluginsync = true
listen = true
runinterval = 300
Master node的配置选项
[master]
autosign = $confdir/autosign.conf { mode = 664 }
autosign.conf
*.ustack.com
服务器端配置信息
192.168.2.53
loadbalancer配置
apache的proxy监听在8140端口，后面可以配置多个puppetmaster进程
<Proxy balancer://puppetmaster>
BalancerMember http://127.0.0.1:18140
BalancerMember http://127.0.0.1:18141
</Proxy>Listen 8140
<VirtualHost *:8140>SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXPSSLCertificateFile /var/lib/puppet/ssl/certs/ha-puppet.ustack.com.pem
SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/ha-puppet.ustack.com.pem
SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
# CRL checking should be enabled; if you have problems with Apache complaining about the CRL, disable the next line
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars +ExportCertData# The following client headers allow the same configuration to work with Pound.
RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
RequestHeader unset X-Forwarded-For# you probably want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120RackAutoDetect On
DocumentRoot /etc/puppet/rack/public/
<Directory /etc/puppet/rack>
Options None
AllowOverride None
Order allow,deny
allow from all
</Directory>
<Location />
SetHandler balancer-manager
Order allow,deny
Allow from all
</Location>ProxyPass / balancer://puppetmaster/
ProxyPassReverse / balancer://puppetmaster/
ProxyPreserveHost OnCustomLog /var/log/httpd/balance-8140-access.log combined
ErrorLog /var/log/httpd/balance-8140-error.log
CustomLog /var/log/httpd/balancer_ssl_requests.log "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"</VirtualHost>
第一个puppetmaster的vhost配置文件，puppetmaster实例的数量可以水平扩展：
Listen 18140
<VirtualHost *:18140> SSLEngine off # The following client headers allow the same configuration to work with Pound.
SetEnvIf set X-Client-Verify "(.*)" SSL_CLIENT_VERIFY=$1
SetEnvIf set X-Client-DN "(.*)" SSL_CLIENT_S_DN=$1 # you probably want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120 RackAutoDetect On
DocumentRoot /etc/puppet/rack/18140/public/
<Directory /etc/puppet/rack/18140/>
Options None
AllowOverride None
Order allow,deny
allow from all
</Directory>
CustomLog /var/log/httpd/puppetmaster-18140-access.log combined
ErrorLog /var/log/httpd/puppetmaster-18140-error.log
</VirtualHost>
rack配置文件
拷贝rack配置文件给第一个puppetmaster：
rsync -avxH /etc/puppet/rack/{,18140}/
Keepalived配置文件
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc }
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server localhost
smtp_connect_timeout 30
router_id route-45
}vrrp_script chk_http_port {
script "/usr/bin/killall -0 httpd"
interval 2
weight 2
}vrrp_instance 45 {
virtual_router_id 45 priority 100
state BACKUP interface eth0 virtual_ipaddress {
192.168.2.103
} track_script {
chk_http_port
}
}vrrp_instance 46 {
virtual_router_id 46 priority 101
state MASTER interface eth0 virtual_ipaddress {
192.168.2.104
} track_script {
chk_http_port
}
}
192.168.2.54
apache的配置文件和53相同，唯一区别就是keepalived的配置文件上的IP地址互为热备，在此就不再赘述。
验证
在dnspod上绑定192.168.1.103/4到ha-puppet.ustack.com。
并在ha-puppet1上使用以下manifests文件：
node /default/ { notify {'Hello,I am Master 1':}
}node 'nginx.novalocal' inherits default{}
在ha-puppet2上使用以下manifests文件：
node /default/ { notify {'Hello,I am Master 2':}
}node 'nginx.novalocal' inherits default{}
测试结果，前两次是：
notice: Hello,I am Master 2
notice: /Stage[main]//Node[default]/Notify[Hello,I am Master 2]/message: defined 'message' as 'Hello,I am Master 2'
notice: Finished catalog run in 0.12 seconds
随后出现：
notice: Hello,I am Master 1
notice: /Stage[main]//Node[default]/Notify[Hello,I am Master 1]/message: defined 'message' as 'Hello,I am Master 1'
notice: Finished catalog run in 0.10 seconds
结论
以上方案验证通过，手动部署很简单，难点在于把它们设计成puppet module来进行部署时，需要考虑到所有的服务器需要使用同一个证书，因此需要在启动第一台puppetmaster的时候，根据设定的fqdn生成证书，然后修改hostname，同时把证书和配置文件同步到其他服务器上去。
