It’s more like a trick, but it’s better than repeat the same operation linearly. Some explanations:
TH_NUM=`ps aux | grep Python | grep -v "grep" | wc -l`
- TH_MAX is the maximum number of “threads” that can be executed at the same time.
- The first
grepselects the threads that make use ofpython(you can change this, it depends on your script) - The second
grepexcludes the command you issued above
wccounts the number of lines. The first time the result of the pipe is empty, sowcgives “0″ as result.
#!/bin/bash
TH_MAX=10
for sample in `ls ./data`
do
while [ TRUE ]; do
TH_NUM=`ps aux | grep Python | grep -v "grep" | wc -l`
if [ "$TH_NUM" -le "$TH_MAX" ]
then
echo $( ./analyze_sample.py -s ${sample} ) > /dev/null &
echo -en " ${sample} "
break
else
echo -en "."
sleep 1
fi
done
done
Advertisement
Tags: Bash, grep, loop, maximum number of threads, ps, pseudothreading, script, threads, wc
January 22, 2010 at 9:14 pm
Another way to do this would be to use xargs as a one-time-makefile alternative:
find . -type f -print0 | xargs -0 -P 3 -n 10 lzma –
This would lzma every file in the current directory and all subdirectories, with up to 3 parallel invocations of lzma with up to 10 files as arguments each, and correctly handle anything that can appear in a filename
I used to abuse make for that kind of thing until I found these options to xargs
May 6, 2010 at 6:17 pm
You can avoid the ‘grep -v “grep”‘ by simply grepping for “[P]ython” which will match Python in the process list, but not the [P]ython that shows in the process list for your grep command. Additionally, GNU grep includes a -c flag to give you the count, so you can eliminate the wc -l.
That would change:
TH_NUM=`ps aux | grep Python | grep -v “grep” | wc -l`
to
TH_NUM=`ps aux | grep -c “[P]ython”`
June 28, 2011 at 9:54 am
Also, using backquotes is decapricated, and the preferred method (that also allows nesting) is the much more readable
TH_NUM=$(ps aux | grep -c “[P]ython”)