multithreading - Python subprocess return code without waiting -

my question particular enough not relate of other ones i've read. i'm wanting use subprocess , multiprocessing spawn bunch of jobs serially , return return code me. problem don't want wait() can spawn jobs @ once, want know when finishes can return code. i'm having weird problem if poll() process won't run. hangs out in activity monitor without running (i'm on mac). thought use watcher thread, i'm hanging on q_out.get() leading me believe maybe i'm filling buffer , deadlocking. i'm not sure how around this. code looks like. if has better ideas on how happy change approach.

def watchjob(p1,out_q):     while p1.poll() == none:         pass     print "job done"     out_q.put(p1.returncode)  def runjob(out_q):     logfile = open('job_to_run.log','w')     p1 = popen(['../../bin/jobexe','job_to_run'], stdout = logfile)     t = threading.thread(target=watchjob, args=(p1,out_q))     t.start()  out_q= queue() outlst=[] in range(len(nprocs)):     proc = process(target=runjob, args=(out_q,))     proc.start()     outlst.append(out_q.get()) # hangs indefinitely     proc.join() 

you don't need neither multiprocessing nor threading here. run multiple child processes in parallel , collect statutes in single thread:

#!/usr/bin/env python3 subprocess import popen  def run(cmd, log_filename):     open(log_filename, 'wb', 0) logfile:         return popen(cmd, stdout=logfile)  # start several subprocesses processes = {run(['echo', c], 'subprocess.%s.log' % c) c in 'abc'} # run in parallel # report child process exits while processes:      p in processes:          if p.poll() not none:            processes.remove(p)             print('{} done, status {}'.format(p.args, p.returncode))            break 

p.args stores cmd in python 3.3+, keep track of cmd on earlier python versions.

see also:

• python threading multiple bash subprocesses?
• python subprocess in parallel
• python: execute cat subprocess in parallel
• using python's multiprocessing module execute simultaneous , separate seawat/modflow model runs

to limit number of parallel jobs threadpool used (as shown in the first link):

#!/usr/bin/env python3 multiprocessing.dummy import pool # use threads subprocess import popen  def run_until_done(args):     cmd, log_filename = args     try:         open(log_filename, 'wb', 0) logfile:             p = popen(cmd, stdout=logfile)         return cmd, p.wait(), none     except exception e:         return cmd, none, str(e)  commands = ((('echo', str(d)), 'subprocess.%03d.log' % d) d in range(500)) pool = pool(128) # 128 concurrent commands @ time cmd, status, error in pool.imap_unordered(run_until_done, commands):     if error none:        fmt = '{cmd} done, status {status}'     else:        fmt = 'failed run {cmd}, reason: {error}'     print(fmt.format_map(vars())) # or fmt.format(**vars()) on older versions 

the thread pool in example has 128 threads (no more, no less). can't execute more 128 jobs concurrently. of threads frees (done job), takes another, etc. total number of jobs executed concurrently limited number of threads. new job doesn't wait 128 previous jobs finish. started when any of old jobs done.