Run a command on multiple servers over SSH

Before you start: a host list and quiet logins

Every method below reads the same thing: one server name per line. Use the names from your SSH config, so users, ports and keys come from there and not from each command. See ssh config file examples if you do not have one yet.

web1
web2
web3

Save it as hosts. Two options make batch runs behave:

-o BatchMode=yes fails at once instead of waiting at a password or passphrase prompt that nobody will answer.
-o ConnectTimeout=5 stops a dead server from holding up the rest.

Load your key into the agent first with ssh-add, so no run stops to ask for it. For many runs in a row, SSH multiplexing with ControlMaster reuses one connection per server and makes each command start faster.

One server after another: a loop

A loop is the simplest way, and the output stays in order. This one also keeps a list of the servers where the command failed:

failed=()
while read -r host; do
  printf '%s: ' "$host"
  ssh -n -o BatchMode=yes -o ConnectTimeout=5 "$host" 'uptime -p' || failed+=("$host")
done < hosts
echo "failed: ${failed[*]}"

web1: up 4 days, 9 hours, 24 minutes
web2: up 4 days, 9 hours, 24 minutes
web3: up 4 days, 9 hours, 24 minutes
failed:

The -n matters. Without it, ssh reads the rest of hosts as its own input, and the loop ends after the first server. The quoting rules and exit codes are the same as for one server, see how to run a command over SSH.

A loop is slow with many servers: fifty servers at two seconds each take almost two minutes.

In parallel: background jobs, xargs and GNU parallel

Plain bash can start every ssh in the background and wait for all of them. Each server writes to its own file, so outputs do not mix:

mkdir -p out
while read -r host; do
  ssh -n -o BatchMode=yes "$host" 'hostname; uname -r' > "out/$host.log" 2>&1 &
done < hosts
wait
head out/*.log

This starts all of them at once. To cap how many run together, use xargs -P:

xargs -P 10 -I{} ssh -n -o BatchMode=yes {} 'echo "$(hostname): $(nproc) cpu"' < hosts

Lines come back in the order the servers finish, so have the command print its own host name.

GNU parallel does both jobs better: --tag puts the server name before each line, and --joblog writes the exit code of each server to a file:

parallel -j 10 --tag --joblog run.log ssh -n -o BatchMode=yes -o ConnectTimeout=5 {} hostname :::: hosts
cut -f 7,9 run.log

Exitval	Command
255	ssh -n -o BatchMode=yes -o ConnectTimeout=5 web3 hostname
0	ssh -n -o BatchMode=yes -o ConnectTimeout=5 web2 hostname
0	ssh -n -o BatchMode=yes -o ConnectTimeout=5 web1 hostname

Here web3 was down. parallel itself exits with the number of jobs that failed, 1 in this case, so a script can stop on it.

parallel-ssh (pssh)

parallel-ssh is made for this job. It is apt install pssh on Debian and Ubuntu, where the command is parallel-ssh. Other systems name the command pssh.

parallel-ssh -h hosts -i -t 10 'test -d /srv/app'

[1] 15:20:02 [FAILURE] web3 Exited with error code 255
Stderr: ssh: Could not resolve hostname web3: Name or service not known
[2] 15:20:02 [FAILURE] web1 Exited with error code 1
[3] 15:20:02 [SUCCESS] web2

The command checks that a folder exists: it is missing on web1 and web3 is down. -i prints each server's output under its status line, and -t 10 gives up on a server after 10 seconds. To keep the output, write one file per server instead:

parallel-ssh -h hosts -t 10 -o out -e err 'df -h /'

It exits with a non-zero code when any server failed. Add -p to limit how many run at once.

Typing into many terminals at once: tmux

Sometimes you need to watch each server, or answer a prompt on each. tmux can open one pane per server and send every key you type to all of them:

tmux new-session -d -s multi 'ssh web1'
tmux split-window -t multi 'ssh web2'
tmux split-window -t multi 'ssh web3'
tmux select-layout -t multi tiled
tmux set-window-option -t multi synchronize-panes on
tmux attach -t multi

Turn it off from inside tmux with Ctrl-b : and setw synchronize-panes off. While it is on, everything goes everywhere, a typo included. The tmux cheat sheet lists the keys for moving between panes.

Before a command that changes things

A mistake on fifty servers costs fifty times as much. A few habits keep it small:

Preview first. Many tools have a dry run. apt-get -s upgrade | grep -c ^Inst counts the packages that would change, without changing them.
One server, then the rest. Run the change on the first server, check it, then on the others.
Ask before the run. In a script, show the count and wait for an answer.
Read every exit code. A run that worked on 49 servers and failed on one is a failed run until you know why.

first=$(head -n 1 hosts)
ssh -n "$first" 'apt-get -s upgrade | grep -c ^Inst'
n=$(wc -l < hosts)
read -r -p "Run on $n servers? [y/N] " ok
[ "$ok" = y ] || exit 1

Keep production in a separate host file, so a command for staging never reaches it by accident. More habits are in how to avoid running commands on the wrong server.

When Ansible fits better

When the same change comes back every week, a tool that describes the result beats a shell command. Ansible's ad hoc mode reads the same list:

ansible all -i 'web1,web2,web3,' -m ansible.builtin.command -a 'uptime -p' --forks 10

web1 | CHANGED | rc=0 >>
up 4 days, 9 hours, 25 minutes

It reports rc for each server, and modules such as apt or copy can show what they would change with --check. For a quick look at a few servers, the loop or parallel-ssh is less work. See how to manage many SSH servers for keeping the list itself in order.

Questions

Why does my ssh loop stop after the first server?

ssh reads standard input and takes the rest of the host list with it. Add -n to the ssh command, or use a for loop that does not read from standard input.

How many servers can I run on at the same time?

As many as your machine and network allow. Ten to fifty at once is common. Lower the number if the command is heavy or servers share a resource such as a database.

What is the difference between parallel-ssh and Ansible?

parallel-ssh runs one shell command on many servers. Ansible runs modules that describe the result you want, can check before changing, and keeps inventories and playbooks for repeated work.

Can I type into several SSH sessions at once?

Yes. In tmux, put each session in a pane and turn on synchronize-panes. Every key goes to all panes until you turn it off.

Sources

More in Sessions and tmux

All guides