Pseudothreading with BASH

by

It’s more like a trick, but it’s better than repeat the same operation linearly. Some explanations:

TH_NUM=`ps aux | grep Python | grep -v "grep" | wc -l`

  • TH_MAX is the maximum number of “threads” that can be executed at the same time.
  • The first grep selects the threads that make use of python (you can change this, it depends on your script)
  • The second grep excludes the command you issued above ๐Ÿ˜‰
  • wc counts the number of lines. The first time the result of the pipe is empty, so wc gives “0” as result.

#!/bin/bash
TH_MAX=10
for sample in `ls ./data`
do
while [ TRUE ]; do
TH_NUM=`ps aux | grep Python | grep -v "grep" | wc -l`
if [ "$TH_NUM" -le "$TH_MAX" ]
then
echo $( ./analyze_sample.py -s ${sample} ) > /dev/null &
echo -en " ${sample} "
break
else
echo -en "."
sleep 1
fi
done
done

Advertisements

Tags: , , , , , , , ,

5 Responses to “Pseudothreading with BASH”

  1. wrtlprnft Says:

    Another way to do this would be to use xargs as a one-time-makefile alternative:

    find . -type f -print0 | xargs -0 -P 3 -n 10 lzma —

    This would lzma every file in the current directory and all subdirectories, with up to 3 parallel invocations of lzma with up to 10 files as arguments each, and correctly handle anything that can appear in a filename ๐Ÿ™‚

    I used to abuse make for that kind of thing until I found these options to xargs ๐Ÿ™‚

  2. NevDull Says:

    You can avoid the ‘grep -v “grep”‘ by simply grepping for “[P]ython” which will match Python in the process list, but not the [P]ython that shows in the process list for your grep command. Additionally, GNU grep includes a -c flag to give you the count, so you can eliminate the wc -l.

    That would change:
    TH_NUM=`ps aux | grep Python | grep -v “grep” | wc -l`

    to

    TH_NUM=`ps aux | grep -c “[P]ython”`

  3. ReaphSharc Says:

    Also, using backquotes is decapricated, and the preferred method (that also allows nesting) is the much more readable
    TH_NUM=$(ps aux | grep -c โ€œ[P]ythonโ€)

  4. Gergely Pihentagy Says:

    ps aux | grep Python | grep -v “grep” | wc -l –> TH_NUM=$(pgrep Python |wc -l) or, if using non BSD version of pgrep: TH_NUM=$(pgrep -c Ptyhon)

  5. Nollie Says:

    Pleasing you should think of sonhmeitg like that

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: