WardtermSSH client
EN
Download

How to manage many SSH servers

Give servers names that show role and environment, such as web-prod-1. Keep one ssh config file per project under ~/.ssh/config.d/, and apply shared settings with patterns such as Host web-*. Use a separate key per purpose with IdentitiesOnly yes, keep the list of servers in one inventory file, and use a tool such as Ansible once you repeat the same change on many servers.

Updated . Config examples were run through ssh -G with OpenSSH 10.3 on macOS and 10.0 on Debian 13.

Use names that carry information

Pick a scheme and keep to it, for example role, environment and number:

web-prod-1
web-prod-2
db-staging-1
acme-web-prod-1

A name like this tells you what you are about to touch, and it works with patterns. A name like server3 or a bare IP address does neither. Add a project or customer prefix if you look after more than one.

Use patterns, not repeated lines

Host *-prod-*
  User deploy
  IdentityFile ~/.ssh/prod_ed25519
  IdentitiesOnly yes

Host web-prod-1
  HostName 203.0.113.11

Host web-prod-2
  HostName 203.0.113.12

Remember that the first value wins and every matching block applies, so the order matters. See the ssh config guide. Check any host with ssh -G web-prod-1.

If your names resolve in DNS, you can drop the per-host lines:

Host web-*
  HostName %h.example.com

One file per project

# top of ~/.ssh/config
Include ~/.ssh/config.d/*
~/.ssh/config.d/
  acme
  internal
  lab

Each file holds the hosts of one project, so you can share it with a teammate, keep it in that project's repository, or delete it when the project ends. Keep Include at the top of the main file, and Host * defaults below it.

Groups

ssh has no groups as such. Patterns are the simple way, so naming does the work. Tags, from OpenSSH 9.4, group hosts whose names do not line up:

Host legacy-api billing-1
  Tag prod

Match tagged prod
  User deploy

To run a read-only command across a group, a plain shell loop is fine at small scale:

for host in web-prod-1 web-prod-2; do
  printf '%s: ' "$host"
  ssh -n -o BatchMode=yes -o ConnectTimeout=5 "$host" uptime
done

-n keeps ssh off standard input, so the same loop also works when the names come from a file. BatchMode fails instead of waiting at a prompt, and ConnectTimeout stops one dead server from holding up the rest. For parallel runs and safe roll-outs, see how to run a command on multiple servers.

One key per purpose

Using a key per purpose, such as work, a customer, or production, means losing one does not open everything. It also means you can remove one customer's access by removing one key.

Host acme-*
  IdentityFile ~/.ssh/acme_ed25519
  IdentitiesOnly yes

Without IdentitiesOnly yes, ssh also offers the keys in your agent, which leads to Too many authentication failures. Do not put IdentityFile in Host * if you want this separation: that value is added to the list and not replaced, so the other key is tried as well.

A key per server is usually more work than it is worth. If the list of people and servers keeps changing, look at SSH certificates instead.

Hashed known_hosts: a trade-off

HashKnownHosts yes stores host names in ~/.ssh/known_hosts as hashes, so whoever steals the file does not get a ready list of your servers. Debian and Ubuntu turn it on in /etc/ssh/ssh_config. Upstream OpenSSH leaves it off.

  • What you gain: a stolen file does not list your servers in plain text. Hashes can still be tested against a guess, so this slows an attacker down and does not stop one.
  • What you lose: you cannot read or search the file by eye, and shell completion based on it stops working.

The tools still work with hashed entries:

ssh-keygen -F web-prod-1      # find a host
ssh-keygen -R web-prod-1      # remove a host
ssh-keygen -H                 # hash an existing file

ssh-keygen -H keeps the original, with plain names, as known_hosts.old. Delete that file if privacy was the reason you did this. Note that your ssh config, shell history and inventory also contain the names, so hashing alone hides less than it seems.

Keep an inventory

Past a dozen servers, keep one list that says what exists: name, address, role, environment, owner. A text file in a private Git repository is enough. Keep it free of passwords and keys. The list is what you check when someone leaves, or when you ask which servers are still in use.

An Ansible inventory is a good format for this even if you do not use Ansible yet:

[web]
web-prod-1
web-prod-2

[db]
db-prod-1 ansible_user=postgres

If you already have a list like this, you can generate the config.d files from it, instead of keeping two lists by hand. The same files work for an app that imports hosts from ~/.ssh/config, including files named by Include.

When a tool such as Ansible fits

ssh is good for reaching a server. It does not keep servers the same. Move to Ansible or a similar tool when you make the same change on more than a few servers, when you need to repeat a setup, or when you need a record of what changed.

ansible all -i inventory.ini -m ping
ansible web -i inventory.ini -a 'uptime'

Ansible uses the ssh command by default, so your config, jump hosts and keys keep working. The target only needs SSH and Python. Check what a playbook will do with --check before running it on production, and read how to avoid mistakes on production servers.

Questions

How do I organize many hosts in ~/.ssh/config?

Give hosts names that show role and environment, put shared settings in patterns such as Host web-*, and keep one file per project under ~/.ssh/config.d/, loaded with Include.

How do I run one command on many SSH servers?

For a few servers, a shell loop with ssh -n -o BatchMode=yes is enough. For changes you repeat, use Ansible, which runs over your existing ssh setup.

Should I use one SSH key for all servers?

Use one key per purpose, such as work, a customer or production, with IdentitiesOnly yes. A key per server is usually more work than it is worth.

Sources

More in Configuration and many servers

From the blog

All guides