scala - SPARK_EXECUTOR_INSTANCES not working in SPARK SHELL, YARN CLIENT MODE -

-

i new on spark.

trying run spark on yarn in yarn-client mode.

spark version = 1.0.2 hadoop version = 2.2.0

cluster of yarn has 3 live nodes.

properties set in spark-env.sh

spark_executor_memory=1g

spark_executor_instances=3

spark_executor_cores=1

spark_driver_memory=2g

command used : /bin/spark-shell --master yarn-client

but after logging spark-shell, registers 1 executor default mem assign it.

i confirmed via spark-web ui has 1 executor , on master node ( yarn resource manager node ) only.

info yarn.client: command starting spark applicationmaster: list($java_home/bin/java, -server, -xmx2048m, -djava.io.tmpdir=$pwd/tmp, -dspark.tachyonstore.foldername=\"spark-fc6383cc-0904-4af9-8abd-3b66b3f0f461\", -dspark.yarn.secondary.jars=\"\", -dspark.home=\"/home/impadmin/spark-1.0.2-bin-hadoop2\", -dspark.repl.class.uri=\"http://master_node:46823\", -dspark.driver.host=\"master_node\", -dspark.app.name=\"spark shell\", -dspark.jars=\"\", -dspark.fileserver.uri=\"http://master_node:46267\", -dspark.master=\"yarn-client\", -dspark.driver.port=\"41209\", -dspark.httpbroadcast.uri=\"http://master_node:36965\", -dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.deploy.yarn.executorlauncher, --class, notused, --jar , null, --args 'master_node:41209' , --executor-memory, 1024, --executor-cores, 1, --num-executors , 3, 1>, /stdout, 2>, /stderr)

...  ...  ...  14/09/10 22:21:24 info cluster.yarnclientschedulerbackend: registered executor:

actor[akka.tcp://sparkexecutor@master_node:53619/user/executor#1075999905] id 1 14/09/10 22:21:24 info storage.blockmanagerinfo: registering block manager master_node:40205 589.2 mb ram 14/09/10 22:21:25 info cluster.yarnclientclusterscheduler: yarnclientclusterscheduler.poststarthook done 14/09/10 22:21:25 info repl.sparkiloop: created spark context.. spark context available sc.

and after running spark action amount of parallelization, runs tasks in series on node only!!

ok solved way. have 4 data nodes on cluster

spark-shell --num-executors 4 --master yarn-client


Comments

Post a Comment

Popular posts from this blog

javascript - how to protect a flash video from refresh？ -

-
<!doctype html> <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> <title>flash video refresh</title> </head> <body> <div id="playerzone"><embed height="496" width="580" flashvars="flvid=19742822&amp;createtime=2012-4-19 13:46:33" wmode="opaque" allowscriptaccess="always" allowfullscreen="true" quality="high" bgcolor="#000000" name="player" id="player" src="swf/mine.swf" type="application/x-shockwave-flash"></div> <a onclick="return test();" id="test">click me popup div,and hide scrollbar</a> </body> the code upon： for example,when click a#test button,popup div,it's no problem,but want hide scrollbar while div popup. i used $('html').css('overflow','hidden') in...
Read more

visual studio 2010 - Connect to informix database windows form application -

-
i trying figure out how connect ibm informix database. have been doing research , have found threads 5 years ago examples not working. i have installed latest sdk ibm informix. i have included ibm.data.informix.dll references in project. i have included using ibm.data.informix; i adding button , on click testing conenction. debug error "sql0035n file "c:\users\adam\documents\visual studio 2010\projects\test\test\msg\en_us\db2nmp.xml" cannot opened." this file not exist , dont see anywhere in program files (x86)\ibm informix client sdk directory. my on click code is private void button1_click(object sender, eventargs e) { const string host = "192.168.obfuscated"; const string servicenum = "1525"; //port? const string server = "serverobfuscated"; const string database = "dbobfuscatedy"; const string user = "myusername"; const string password = ...
Read more

android - Associate same looper with different threads -

-
Image
in application, want create multiple threads working on single queue. create queue know need call looper.prepare() . i want other threads create should associated looper created first thread, how can achieve this? use 1 of java thread safe queue class java.util.concurrent package instead of looper achieve goal. it's more common practice. share queue between threads , post task queue entry 1 thread can entry thread. blockingqueue can - in 1 thread call take() , block thread until thread put() entry queue. that's all. producer consumer pattern example blockingqueue or concurrentlinkedqueue about blockingqueue
Read more