ruby on rails - Is it possible to force concurrent jobs to run in separate Sidekiq processes? -

-

one of benefits of sidekiq on resqueue can run multiple jobs in same process. drawback, however, can't figure out how force set of concurrent jobs run in different processes.

here's use case: have generate 64m rows of data, , have 8 vcpus on amazon ec2 instance. i'd carve task 8 concurrent jobs generating 8m rows each. problem if i'm running 8 sidekiq processes, sidekiq decide run 2 or more of jobs in same process, , doesn't use 8 vcpus , takes longer finish. there way tell sidekiq worker use or force spread jobs in group evenly amongst processes?

answer can't easily, design. specialization leads spofs.

  1. you can create custom queue each process , create 1 job each queue.
  2. you can use jruby doesn't suffer same flaw.
  3. you can execute processing rake task spawn 1 process per job, ensuring load.
  4. you can carve 64 jobs instead of 8 , more load way.

i latter unless resulting i/o crushes machine.


Comments

Post a Comment

Popular posts from this blog

javascript - how to protect a flash video from refresh？ -

-
<!doctype html> <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> <title>flash video refresh</title> </head> <body> <div id="playerzone"><embed height="496" width="580" flashvars="flvid=19742822&amp;createtime=2012-4-19 13:46:33" wmode="opaque" allowscriptaccess="always" allowfullscreen="true" quality="high" bgcolor="#000000" name="player" id="player" src="swf/mine.swf" type="application/x-shockwave-flash"></div> <a onclick="return test();" id="test">click me popup div,and hide scrollbar</a> </body> the code upon： for example,when click a#test button,popup div,it's no problem,but want hide scrollbar while div popup. i used $('html').css('overflow','hidden') in
Read more

android - Associate same looper with different threads -

-
Image
in application, want create multiple threads working on single queue. create queue know need call looper.prepare() . i want other threads create should associated looper created first thread, how can achieve this? use 1 of java thread safe queue class java.util.concurrent package instead of looper achieve goal. it's more common practice. share queue between threads , post task queue entry 1 thread can entry thread. blockingqueue can - in 1 thread call take() , block thread until thread put() entry queue. that's all. producer consumer pattern example blockingqueue or concurrentlinkedqueue about blockingqueue
Read more

visual studio 2010 - Connect to informix database windows form application -

-
i trying figure out how connect ibm informix database. have been doing research , have found threads 5 years ago examples not working. i have installed latest sdk ibm informix. i have included ibm.data.informix.dll references in project. i have included using ibm.data.informix; i adding button , on click testing conenction. debug error "sql0035n file "c:\users\adam\documents\visual studio 2010\projects\test\test\msg\en_us\db2nmp.xml" cannot opened." this file not exist , dont see anywhere in program files (x86)\ibm informix client sdk directory. my on click code is private void button1_click(object sender, eventargs e) { const string host = "192.168.obfuscated"; const string servicenum = "1525"; //port? const string server = "serverobfuscated"; const string database = "dbobfuscatedy"; const string user = "myusername"; const string password = &
Read more