Posts Tagged ‘threads’

Pseudothreading with BASH

January 20, 2010

It’s more like a trick, but it’s better than repeat the same operation linearly. Some explanations:

TH_NUM=`ps aux | grep Python | grep -v "grep" | wc -l`

  • TH_MAX is the maximum number of “threads” that can be executed at the same time.
  • The first grep selects the threads that make use of python (you can change this, it depends on your script)
  • The second grep excludes the command you issued above 😉
  • wc counts the number of lines. The first time the result of the pipe is empty, so wc gives “0” as result.

#!/bin/bash
TH_MAX=10
for sample in `ls ./data`
do
while [ TRUE ]; do
TH_NUM=`ps aux | grep Python | grep -v "grep" | wc -l`
if [ "$TH_NUM" -le "$TH_MAX" ]
then
echo $( ./analyze_sample.py -s ${sample} ) > /dev/null &
echo -en " ${sample} "
break
else
echo -en "."
sleep 1
fi
done
done

pthreads – Some useful links and solution for maximum thread number

July 21, 2009

Manual Reference Pages (including a list of functions that are not thread-safe)

Tutorial on pthreads

And if you wonder why you cannot create more than X threads on your system (for me this was always 382), this forum provides a solution.

Basically, the problem is that each thread created will occupy space for its stack. On my system, the default thread stack size is 8MB. Therefore, after 382, I simply run out of space.

Solution: Change the stack size to a smaller value, unless you really need 8MB.

pthread_attr_t tattr;
size_t size;
size = PTHREAD_STACK_MIN + WHAT_ELSE_YOU_NEED;
pthread_attr_init(&tattr);
pthread_attr_setstacksize(&tattr, size);

Another interesting page on pthreads is this.