Let's imagine you have a script like below and you want to make it faster.
#!/bin/bash
echo "Started at `date +%Y-%m-%d\ %H:%M:%S`"
echo "Starting job 1"
sleep 5
echo "Starting job 2"
sleep 5
echo "Finished at `date +%Y-%m-%d\ %H:%M:%S`"
If it's ok to run jobs in parallel, you can easily do it with bash background jobs.
Simple case
By adding and ampersand sign (&
) in the end of command you tell bash to run this command in background. Then you can wait
command to wait until child background jobs finish.
#!/bin/bash
echo "Started at `date +%Y-%m-%d\ %H:%M:%S`"
echo "Starting job 1"
sleep 5 &
echo "Starting job 2"
sleep 5 &
wait
echo "Finished at `date +%Y-%m-%d\ %H:%M:%S`"
Multiple commands in background
That worked fine with single command, but can you run several sequential commands in one subprocess? Yes, by grouping them with { ... }
.
#!/bin/bash
echo "Started at `date +%Y-%m-%d\ %H:%M:%S`"
{
echo "Starting job 1"
sleep 5
echo "Finished job 1"
} &
{
echo "Starting job 2"
sleep 5
echo "Finished job 2"
} &
wait
echo "Finished at `date +%Y-%m-%d\ %H:%M:%S`"
Limiting number of background jobs
Now let's imagine these commands are heavy and running them all will cause big load on your server. In that case it's possible to to limit background processes to a certain number.
The best way I found to do this is just to wait until there are less than JOBS_LIMIT
jobs before running next one. We use jobs -rp
command to list running child processes and to see their count.
#!/bin/bash
JOBS_LIMIT=5
echo "Started at `date +%Y-%m-%d\ %H:%M:%S`"
jobs=(1 2 3 4 5 6 7 8 9 10)
for job in "${jobs[@]}"
do
while [ `jobs -rp | wc -l` -ge $JOBS_LIMIT ]
do
sleep 1
done
{
echo "Starting job $job"
sleep $(( ( RANDOM % 5 ) + 1 ))
echo "Finished job $job"
} &
done
wait
echo "Finished at `date +%Y-%m-%d\ %H:%M:%S`"
Conclusion
These tricks give you power to parallelize your scripts and make them much faster. If you want more tricks when running commands manually - check this article. If you have better ideas about running commands from bash scripts - don't hesitate to enlighten me in cooments.
And here's a bit of Nicholas Cage for you: .