Service Discovery: Consul, etcd, and DNS Patterns

Why Service Discovery?

In a monolithic application, one service calls another at a known address. In a microservices architecture, services are dynamic: they start, stop, scale up, and move across machines. Hardcoding IP addresses and ports breaks immediately when a service scales from three instances to five, or when an instance is replaced after a crash.

Service discovery solves the fundamental question: how does Service A find the current addresses of Service B?

Without service discovery, you end up maintaining static configuration files or load balancer entries that someone must update every time the infrastructure changes. This is error-prone, slow, and does not scale. Service discovery automates the process: services register themselves when they start and deregister when they stop. Other services query the registry to find available instances.

Discovery Patterns

There are two primary patterns for service discovery:

Client-Side Discovery

The client is responsible for querying the service registry and selecting an instance:

┌─────────┐     1. Query     ┌──────────────────┐
│ Service A├────────────────▶│  Service Registry │
│ (client) │◀────────────────│ (Consul, etcd)    │
└────┬─────┘  2. Return list └──────────────────┘
     │
     │ 3. Direct call to chosen instance
     ▼
┌──────────┐  ┌──────────┐  ┌──────────┐
│Service B │  │Service B │  │Service B │
│Instance 1│  │Instance 2│  │Instance 3│
└──────────┘  └──────────┘  └──────────┘

The client gets a list of instances and applies its own load-balancing logic (round-robin, least connections, random). Libraries like Netflix Ribbon or gRPC’s built-in load balancer implement this pattern.

Advantages: No single-point-of-failure load balancer, lower latency (direct connection), client can use smart routing.

Disadvantages: Every client needs discovery logic, each language/framework needs its own implementation.

Server-Side Discovery

A load balancer or proxy sits between the client and the service instances:

┌─────────┐      Request      ┌───────────────┐
│ Service A├──────────────────▶│ Load Balancer  │
│ (client) │                   │ (Nginx, HAProxy│
└──────────┘                   │  Envoy, Traefik)
                               └───────┬───────┘
                                       │
                          ┌────────────┼────────────┐
                          ▼            ▼            ▼
                    ┌──────────┐ ┌──────────┐ ┌──────────┐
                    │Service B │ │Service B │ │Service B │
                    │Instance 1│ │Instance 2│ │Instance 3│
                    └──────────┘ └──────────┘ └──────────┘

The load balancer queries the registry and routes traffic. Kubernetes Services, AWS ALB with service discovery, and Traefik all implement this pattern.

Advantages: Client is simple (just knows one address), language-agnostic, centralized routing policy.

Disadvantages: Additional infrastructure component, potential bottleneck, added latency hop.

HashiCorp Consul

Consul is the most feature-rich service discovery tool. It provides service registration, health checking, a DNS interface, a key-value store, and multi-datacenter support.

Setting Up Consul

Deploy a Consul cluster using Docker Compose:

version: "3.8"

services:
  consul-server-1:
    image: hashicorp/consul:1.18
    container_name: consul-server-1
    command: agent -server -bootstrap-expect=3 -node=server-1 -client=0.0.0.0 -ui
    ports:
      - "8500:8500"
      - "8600:8600/udp"
    volumes:
      - consul_data_1:/consul/data
    networks:
      - consul-net

  consul-server-2:
    image: hashicorp/consul:1.18
    container_name: consul-server-2
    command: agent -server -node=server-2 -client=0.0.0.0 -retry-join=consul-server-1
    volumes:
      - consul_data_2:/consul/data
    networks:
      - consul-net

  consul-server-3:
    image: hashicorp/consul:1.18
    container_name: consul-server-3
    command: agent -server -node=server-3 -client=0.0.0.0 -retry-join=consul-server-1
    volumes:
      - consul_data_3:/consul/data
    networks:
      - consul-net

volumes:
  consul_data_1:
  consul_data_2:
  consul_data_3:

networks:
  consul-net:
    driver: bridge

docker compose up -d

# Verify the cluster
curl http://localhost:8500/v1/status/leader
curl http://localhost:8500/v1/catalog/nodes

The Consul UI is available at http://localhost:8500.

Registering Services

Services register themselves via the HTTP API or a configuration file.

Using the HTTP API:

curl -X PUT http://localhost:8500/v1/agent/service/register \
  -H 'Content-Type: application/json' \
  -d '{
    "ID": "order-api-1",
    "Name": "order-api",
    "Tags": ["v2.1", "production"],
    "Port": 8080,
    "Address": "192.168.1.10",
    "Meta": {
      "version": "2.1.0",
      "environment": "production"
    },
    "Check": {
      "HTTP": "http://192.168.1.10:8080/health",
      "Interval": "10s",
      "Timeout": "3s",
      "DeregisterCriticalServiceAfter": "90s"
    }
  }'

Using a service definition file on the Consul agent:

{
  "service": {
    "name": "order-api",
    "id": "order-api-1",
    "port": 8080,
    "tags": ["v2.1", "production"],
    "check": {
      "http": "http://localhost:8080/health",
      "interval": "10s",
      "timeout": "3s"
    }
  }
}

Discovering Services

Query registered services via the HTTP API:

# Get all instances of a service
curl http://localhost:8500/v1/catalog/service/order-api

# Get only healthy instances
curl http://localhost:8500/v1/health/service/order-api?passing=true

The response includes addresses, ports, tags, and health status:

[
  {
    "Node": "server-1",
    "ServiceID": "order-api-1",
    "ServiceName": "order-api",
    "ServiceAddress": "192.168.1.10",
    "ServicePort": 8080,
    "ServiceTags": ["v2.1", "production"],
    "Checks": [
      {
        "Status": "passing",
        "Output": "HTTP GET http://192.168.1.10:8080/health: 200 OK"
      }
    ]
  }
]

Consul DNS Interface

Consul exposes a DNS interface on port 8600. Services are available at <service-name>.service.consul:

# Query via DNS
dig @localhost -p 8600 order-api.service.consul SRV

# Response includes the host and port
# order-api.service.consul. 0 IN SRV 1 1 8080 server-1.node.consul.

# A record query returns just the IP
dig @localhost -p 8600 order-api.service.consul A

Configure your system to use Consul for .consul domain resolution by adding it to your DNS configuration or using dnsmasq:

# /etc/dnsmasq.d/consul.conf
server=/consul/127.0.0.1#8600

Now services can discover each other using standard DNS:

curl http://order-api.service.consul:8080/orders

Consul Key-Value Store

Consul includes a key-value store useful for shared configuration:

# Set a value
curl -X PUT http://localhost:8500/v1/kv/config/database/connection-string \
  -d 'postgres://user:pass@db.example.com:5432/mydb'

# Get a value
curl http://localhost:8500/v1/kv/config/database/connection-string?raw

# Watch for changes (long polling)
curl "http://localhost:8500/v1/kv/config/database/connection-string?wait=30s&index=42"

Consul Connect (Service Mesh)

Consul Connect adds mutual TLS and authorization between services:

{
  "service": {
    "name": "order-api",
    "port": 8080,
    "connect": {
      "sidecar_service": {
        "proxy": {
          "upstreams": [
            {
              "destination_name": "payment-api",
              "local_bind_port": 9191
            }
          ]
        }
      }
    }
  }
}

The order-api can now reach payment-api at localhost:9191, with traffic automatically encrypted and authorized through Consul Connect proxies.

etcd

etcd is a distributed key-value store used as the backbone of Kubernetes. While not a purpose-built service discovery tool like Consul, it provides the primitives needed to build service discovery: consistent key-value storage, watches, and leases.

Setting Up etcd

# docker-compose.yml
version: "3.8"

services:
  etcd:
    image: quay.io/coreos/etcd:v3.5.12
    container_name: etcd
    command:
      - etcd
      - --name=etcd-1
      - --listen-client-urls=http://0.0.0.0:2379
      - --advertise-client-urls=http://etcd:2379
      - --listen-peer-urls=http://0.0.0.0:2380
    ports:
      - "2379:2379"
    volumes:
      - etcd_data:/etcd-data

volumes:
  etcd_data:

Service Registration with Leases

etcd uses leases to automatically remove registrations when a service stops sending heartbeats:

# Create a lease (TTL 30 seconds)
LEASE_ID=$(etcdctl lease grant 30 | awk '{print $2}')

# Register the service with the lease
etcdctl put /services/order-api/instances/order-api-1 \
  '{"address":"192.168.1.10","port":8080,"version":"2.1.0"}' \
  --lease=$LEASE_ID

# Keep the lease alive (run in background)
etcdctl lease keep-alive $LEASE_ID

If the service crashes and stops sending keep-alive requests, etcd automatically removes the registration after the TTL expires.

Discovering Services

# Get all instances of a service
etcdctl get /services/order-api/instances/ --prefix

# Watch for changes (new registrations, deregistrations)
etcdctl watch /services/order-api/instances/ --prefix

Building Discovery in Application Code

Here is a Python example using etcd for service discovery:

import etcd3
import json
import threading

class ServiceRegistry:
    def __init__(self, etcd_host='localhost', etcd_port=2379):
        self.client = etcd3.client(host=etcd_host, port=etcd_port)

    def register(self, service_name, instance_id, address, port, ttl=30):
        lease = self.client.lease(ttl)
        key = f'/services/{service_name}/instances/{instance_id}'
        value = json.dumps({
            'address': address,
            'port': port,
            'instance_id': instance_id
        })
        self.client.put(key, value, lease=lease)

        # Keep lease alive in background
        def keep_alive():
            while True:
                lease.refresh()

        thread = threading.Thread(target=keep_alive, daemon=True)
        thread.start()
        return lease

    def discover(self, service_name):
        prefix = f'/services/{service_name}/instances/'
        instances = []
        for value, metadata in self.client.get_prefix(prefix):
            instances.append(json.loads(value))
        return instances

    def watch(self, service_name, callback):
        prefix = f'/services/{service_name}/instances/'
        events_iterator, cancel = self.client.watch_prefix(prefix)
        for event in events_iterator:
            instances = self.discover(service_name)
            callback(instances)

DNS-Based Service Discovery

For simpler architectures, DNS itself can serve as a service discovery mechanism. This approach works well when you have a relatively stable set of services and can tolerate DNS caching delays.

Internal DNS with CoreDNS

CoreDNS can serve dynamic DNS records from various backends:

# Corefile
.:53 {
    forward . 8.8.8.8 8.8.4.4
    log
    errors
}

service.internal:53 {
    file /etc/coredns/zones/service.internal.zone
    log
    errors
    reload 10s
}

; /etc/coredns/zones/service.internal.zone
$ORIGIN service.internal.
@   IN SOA  ns1.service.internal. admin.service.internal. (
        2026070701 ; serial
        3600       ; refresh
        600        ; retry
        86400      ; expire
        60         ; minimum TTL
    )
    IN NS ns1.service.internal.

ns1     IN A 10.0.0.1

order-api   IN A 10.0.1.10
order-api   IN A 10.0.1.11
order-api   IN A 10.0.1.12

payment-api IN A 10.0.2.10
payment-api IN A 10.0.2.11

Services resolve using standard DNS:

dig order-api.service.internal A
# Returns all three A records; client picks one

Kubernetes DNS

Kubernetes provides built-in DNS-based service discovery. Every Service gets a DNS name:

# Service DNS format
<service-name>.<namespace>.svc.cluster.local

# Examples
order-api.production.svc.cluster.local
payment-api.production.svc.cluster.local
redis.cache.svc.cluster.local

From within a pod in the same namespace, you can use just the service name:

curl http://order-api:8080/orders

For headless services (without a ClusterIP), DNS returns the individual pod IPs:

apiVersion: v1
kind: Service
metadata:
  name: order-api-headless
spec:
  clusterIP: None  # Headless service
  selector:
    app: order-api
  ports:
    - port: 8080

# Returns individual pod IPs instead of a single ClusterIP
dig order-api-headless.production.svc.cluster.local A
# 10.244.1.5
# 10.244.2.8
# 10.244.3.12

This is useful for stateful workloads where clients need to connect to specific instances (e.g., database replicas).

Choosing a Discovery Approach

Feature	Consul	etcd	DNS
Health checking	Built-in, rich	Manual (via leases)	External (load balancer)
Multi-datacenter	Native support	Requires setup	Possible with GeoDNS
Service mesh	Consul Connect	No	No
KV store	Built-in	Core feature	No
DNS interface	Yes	No (use CoreDNS)	Native
Complexity	Medium	Low-Medium	Low
Best for	Multi-service, multi-DC	Kubernetes backing, simple registry	Simple architectures, K8s

Use Consul when you need a full-featured service discovery platform with health checking, multi-datacenter support, and a service mesh. It is the most complete solution but adds operational complexity.

Use etcd when you are already running Kubernetes (which uses etcd internally) or need a simple, consistent key-value store for registration. Build your own discovery logic on top.

Use DNS when your architecture is simple enough that DNS round-robin with health-checked backends is sufficient. Kubernetes DNS covers most containerized workloads without any additional tooling.

Wrapping Up

Service discovery is a foundational capability for any distributed system. At its simplest, DNS provides basic name-to-address resolution that works for many use cases, especially within Kubernetes. Consul adds health checking, a key-value store, and multi-datacenter awareness for complex microservice architectures. etcd provides the low-level primitives to build custom discovery logic when you need fine-grained control. Start with the simplest approach that meets your needs: Kubernetes DNS if you are on Kubernetes, Consul if you need health checking and multi-environment support, and etcd if you need a lightweight registry you can build upon. The right choice depends on your architecture’s complexity and your operational capacity to manage additional infrastructure.