How to check the number of cores Spark uses? -

-

i have spark.cores.max set 24 [3 worker nodes], if inside worker node , see there 1 process [command = java] running consumes memory , cpu. suspect not use 8 cores (on m2.4x large).

how know number?

you can see number of cores occupied on each worker in cluster under spark web ui: spark web ui


Comments

Post a Comment

Popular posts from this blog

c# - HttpResponseMessage System.InvalidOperationException -

-
quite simple problem i've got easy solve reckon brains. it's tried different queries on google , nothing popped hey that's why here. here error: system.invalidoperationexception basically error thrown on piece of code here string test = response.content.headers.getvalues("location").firstordefault(); location in fiddler looks this: location: https://www.redirecturlishere.com/blah here entire function: private async task<string> getrequest() { //httpcontent postcontent = new stringcontent(post, encoding.ascii, "application/x-www-form-urlencoded"); using (httpresponsemessage response = await httpclient.getasync( "http://www.cant-share-url.com")) { using (httpcontent content = response.content) { string test = response.content.headers.getvalues("location").firstordefault(); } } return ""; ...
Read more

sql - Postgresql error: "failed to find conversion function from unknown to text" -

-
i have following postgresql query: ( select '201405mase04' testid, count(*) totq, count(distinct case when se1 = '' null else se1 end) totse, case when count(*)=0 7 else count(distinct rc) end totrc eivtestitems ti, eivtests t ti.test_id = t.test_id , t.test_type_id = 1 , t.test_id= '201405mase04' , ti.omit <> 1 ), b ( select '201405mase04' testid, count(*) totq eivtestitems ti, eivtests t ti.test_id = t.test_id , t.test_type_id = 1 , t.test_id= '201405mase04' , ti.omit = 1 ), c ( select '201405mase04' testid, count(*) totq eivtestitems ti, eivtests t ti.test_id = t.test_id , t.test_type_id = 1 , t.test_id= '201405mase04' , ti....
Read more