...
The current infrastructure tightly couples services with the server it is deployed upon. The following describe describes a setup which maintains the coupling of the service to a specific server but uses service discovery tools and DNS to advertise services dynamically.
...
The Consul agent operates in either client or server mode. Consul clients, running on GÉANT linux servers, register services with the Consul server. The Consul server then has a record of services running on the network and where they are located. With this information it can either act as a source for DNS lookups (via a DNS forward zone) or can update the network DNS server with information it knows about network services.
In this setup the consul client agent is installed on every node. Included with the service when it is deployed on a server, is the consul configuration for the service, and the agent uses this to register the service with the Consul Server. The Consul agent configuration includes a health check stanza which the Consul Server uses to confirm the health of a particular service and, thus, determines whether the service is advertised for use.
Services are advertised in DNS. The Consul Servers can by queried directly via a DNS forwarder, for instance, alternatively the Consul server can update GÉANT's DNS infrastructure but by periodically pushing DNS zone files to Infoblox. A DNS query port on a consul server is a single point of failure so pushing a zone file to Infoblox is a more appropriate resilient solution.
Example for the
...
dashboard service (using the image above)
In the image above there are two linux (VMs) each running dashboard. In addition node 1 also hosts an instance of cacti and node 2 the bodportal. Each node runs the Consul agent in "client" mode and registers all the services it knows about with the Consul server. The consul server periodically assembles all registered services into a DNS zonefile which it pushes to the DNS infrastructure (which in GÉANT is Infoblox). In the example here, the zonefile will contain two entries for dashboard and one each for cacti and bodportal.
When a web client loads dashboard, the DNS query returns one of the IPs it has for dashboard; above, node2 is returned.
The consul server continues to accept service registrations from consul clients as well as actively monitoring the health of registered services. So, for instance, should the dashboard instance on node 2 fail, the consul server will identify this and remove the node 2 IP for dashboard from the DNS zonefile. The next time a client loads dashboard it will only be offered the IP for node 1.
Todo - Low-level design
Components:
- Consul: provides service discovery: client and server modes.
- Infoblox: DNS for services - infoblox api used for deploying zonefile
- consul agent (client) installed
As an example consider the Atlassian Crowd service deployed onto two servers:
prod-crowd01.geant.org
- location: Frankfurt PoP
- IP: 1.1.1.10
- service name: crowd
- consul agent
prod-crowd02.geant.org
- location: City House PoP
- IP: 1.1.1.11
- service name: crowd
- consul agent
consul-server01(02 and 03)
- consul agent
Infoblox (DNS):
- zone: geant.services
- crowd
- IP: 1.1.1.10
- IP: 1.1.1.11
- crowd
In the above setup, a client connection connects to crowd.geant.services, the DNS lookup will return either 1.1.1.10 or 1.1.1.11 (it doesn't matter as long as crowd works on them both). If the crowd service on prod-crowd02 fails (or the server fails, or if there are connectivity problems), the consul server detects this and updates the zone file to look like this:
Infloblox:
- zone: geant.services
- crowd
- IP: 1.1.1.10
- crowd
In this case, a lookup of crowd.geant.services will return 1.1.1.10 only.
Components
Consul: provides service discovery
Infoblox: DNS for services
...
- on all servers which have services to be discovered
...
- quorum of consul servers (3 or 5 Consul servers is recommended)
- Consul server creates/generates DNS zonefile which is pushed to infoblox. Zonefile generated often (every minute?) but only pushed to infoblox if changed (need to detect and protect against flapping)
- service checks for every service include