For a recent project I was tasked with designing a way for junior engineers to restart services across several nodes in an organized iterative fashion. Since these were legacy applications(not containerized yet) we need to connect to several ec2 instances, issue a restart command and wait for the app to come online before proceeding. In the past I would have used an array of predefined hostnames but in this case I was able to leverage Consul’s service discovery endpoints.

serverList=curl -s "https://consul.somedomain.com/v1/catalog/service/$tomcatService-rest" | jq -r '.[] | select(.ServiceMeta.configBucket == "'$configBucket'_'$bounceArea'") | .Node'

We store a json file with our healthchecks in s3 for shared consumption which we can use here:

restHealth=healthCheckEndpoints -s https://healthCheckEndpoints.s3.amazonaws.com/health.json | jq -r .\"$tomcatService\".rest_health
restPort=healthCheckEndpoints -s https://healthCheckEndpoints.s3.amazonaws.com/health.json | jq -r .\"$tomcatService\".rest_port

I think you can see where this is going, we can now iterate over the nodes in serverList, ssh in, restart and use a health check curl similar to this and proceed once the app is live:

httpResp=curl -s -m 10 http://$host:$restPort$restHealth | jq -r .httpCode

It’s much easier this way when the hosts are populated dynamically, we now rely on the app registration and not a potentially incorrect host list.