Running bash commands in parallel

Let's imagine you have a script like below and you want to make it faster.

#!/bin/bash

echo "Started at `date +%Y-%m-%d\ %H:%M:%S`"

echo "Starting job 1"
sleep 5

echo "Starting job 2"
sleep 5

echo "Finished at `date +%Y-%m-%d\ %H:%M:%S`"

If it's ok to run jobs in parallel, you can easily do it with bash background jobs.

Simple case

By adding and ampersand sign (&) in the end of command you tell bash to run this command in background. Then you can wait command to wait until child background jobs finish.

#!/bin/bash

echo "Started at `date +%Y-%m-%d\ %H:%M:%S`"

echo "Starting job 1"
sleep 5 &

echo "Starting job 2"
sleep 5 &

wait

echo "Finished at `date +%Y-%m-%d\ %H:%M:%S`"

Multiple commands in background

That worked fine with single command, but can you run several sequential commands in one subprocess? Yes, by grouping them with { ... }.

#!/bin/bash

echo "Started at `date +%Y-%m-%d\ %H:%M:%S`"

{
    echo "Starting job 1"
    sleep 5
    echo "Finished job 1"
} &

{
    echo "Starting job 2"
    sleep 5
    echo "Finished job 2"
} &

wait

echo "Finished at `date +%Y-%m-%d\ %H:%M:%S`"

Limiting number of background jobs

Now let's imagine these commands are heavy and running them all will cause big load on your server. In that case it's possible to to limit background processes to a certain number.

The best way I found to do this is just to wait until there are less than JOBS_LIMIT jobs before running next one. We use jobs -rp command to list running child processes and to see their count.

#!/bin/bash

JOBS_LIMIT=5

echo "Started at `date +%Y-%m-%d\ %H:%M:%S`"

jobs=(1 2 3 4 5 6 7 8 9 10)
for job in "${jobs[@]}"
do
    while [ `jobs -rp | wc -l` -ge $JOBS_LIMIT ]
    do
        sleep 1
    done

    {
        echo "Starting job $job"
        sleep $(( ( RANDOM % 5 )  + 1 ))
        echo "Finished job $job"
    } &
done

wait

echo "Finished at `date +%Y-%m-%d\ %H:%M:%S`"

Conclusion

These tricks give you power to parallelize your scripts and make them much faster. If you want more tricks when running commands manually - check this article. If you have better ideas about running commands from bash scripts - don't hesitate to enlighten me in cooments.

And here's a bit of Nicholas Cage for you: .