multiprocessing - Python: Using Multiprocess to switch functions -
i'm trying learn multiprocessing in python.
what want happen have x increase time, when hit 'enter' decrease time , can keep hitting enter switch increasing decreasing.
all i've managed make far this
import time multiprocessing import pool def f(x): return (x+1) def f1(x): return (x-1) if __name__ == '__main__': pool = pool(processes=2) x=0 while x<100: x = pool.apply_async(f, [10]) time.sleep(0.05)
i'm having trouble looking though documentation because don't give explicit examples. please have no idea
threw example give ideas.
#!/usr/bin/python3 import time multiprocessing import pool, manager threading import thread num_processes = 4 def consumer(q): out = [] while true: val = q.get() if val none: #poison pill break #doing work here time.sleep(0.01) out.append(val) return out def producer(queue): flip = true val = 5 def flipper(): nonlocal flip input('enter flip it!') while true: flip = not flip txt = 'up' if flip else 'down' input('val {}, counting {}'.format(val, txt)) t = thread(target=flipper, args=(), daemon=true) t.start() while val > 0: _ in range(num_processes): queue.put(val) val = val + (1 if flip else -1) time.sleep(0.2) print() print('hit zero, shutting down.') _ in range(num_processes): #poison pills queue.put(none) if __name__ == '__main__': pool = pool(processes=num_processes) m = manager() # n.b.: multiprocessing.queue doesn't work pool.apply_async q = m.queue() results = pool.apply_async(consumer, args=(q,)) producer(q) # running in main thread because i'm lazy print(results.get())
output:
ben@nixbox:~$ python3 multithread.py enter flip it! val 13, counting down val 4, counting val 8, counting down hit zero, shutting down. [5, 5, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7, 8, 8, 8, 8, 9, 9, 9, 9, 10, 10, 10, 10, 11, 11, 11, 11, 12, 12, 12, 12, 13, 13, 13, 13, 12, 12, 12, 12, 11, 11, 11, 11, 10, 10, 10, 10, 9, 9, 9, 9, 8, 8, 8, 8, 7, 7, 7, 7, 6, 6, 6, 6, 5, 5, 5, 5, 4, 4, 4, 4, 5, 5, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7, 8, 8, 8, 8, 7, 7, 7, 7, 6, 6, 6, 6, 5, 5, 5, 5, 4, 4, 4, 4, 3, 3, 3, 3, 2, 2, 2, 2, 1, 1, 1, 1]
the way original code set up, parallelizing didn't make whole lot of sense. switched producer/consumer paradigm: producer produces "work units" increase until hit enter, decrease until hits zero, loads work queue poison pills tell consumers shut down. there's few tricks input/flipping thing work, i'll let @ code that. note used threads wait on blocking i/o operations , processes (simulated) cpu-heavy work. on purpose.
there lots of ways structure multiprocessing code find producer/consumer useful pattern. 1 thing don't want doing rapidly creating processes teeny tiny amount of work , tearing them down. slow! (i note using pool
correctly avoids - keeps workers alive you. still, worth mentioning since lot of people start out firing off bunch of process
es in loop, later wonder why parallel code slower single-threaded code :-))
Comments
Post a Comment