Apache spark: setting spark.eventLog.enabled and spark.eventLog.dir at submit or Spark start -

-

i set spark.eventlog.enabled , spark.eventlog.dir @ spark-submit or start-all level -- not require enabled in scala/java/python code. have tried various things no success:

setting spark-defualts.conf as

spark.eventlog.enabled           true spark.eventlog.dir               hdfs://namenode:8021/directory

or

spark.eventlog.enabled           true spark.eventlog.dir               file:///some/where

running spark-submit as:

spark-submit --conf "spark.eventlog.enabled=true" --conf "spark.eventlog.dir=file:///tmp/test" --master spark://server:7077 examples/src/main/python/pi.py

starting spark environment variables:

spark_daemon_java_opts="-dspark.eventlog.enabled=true -dspark.history.fs.logdirectory=$sparkhistorydir -dspark.history.provider=org.apache.spark.deploy.history.fshistoryprovider -dspark.history.fs.cleaner.enabled=true -dspark.history.fs.cleaner.interval=2d"

and overkill:

spark_history_opts="-dspark.eventlog.enabled=true -dspark.history.fs.logdirectory=$sparkhistorydir -dspark.history.provider=org.apache.spark.deploy.history.fshistoryprovider -dspark.history.fs.cleaner.enabled=true -dspark.history.fs.cleaner.interval=2d"

where , how must these things set history on arbitrary jobs?

i solved problem, yet strangely had tried before... same, seems stable solution:

create directory in hdfs logging, /eventlogging

hdfs dfs -mkdir /eventlogging

then spark-shell or spark-submit (or whatever) can run following options: 

--conf spark.eventlog.enabled=true --conf spark.eventlog.dir=hdfs://<hdfsnamenodeaddress>:8020/eventlogging

such as:

spark-shell --conf spark.eventlog.enabled=true --conf spark.eventlog.dir=hdfs://<hdfsnamenodeaddress>:8020/eventlogging

Comments

Post a Comment

Popular posts from this blog

c# - Better 64-bit byte array hash -

-
i need hash algorithm produces 64-bit hash code ( long ) fewer collisions string.gethashcode() , fast (no expensive calls cryptographic functions). here's implementation of fnv still shows 3% of collisions after testing 2 million random strings. need number way lower. void main() { const string chars = "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz!#@$%^&*()_+}{\":?><,./;'[]0123456789\\"; const int n = 2000000; var random = new random(); var hashes = new hashset<long>(); int collisions = 0; for(int = 0; < n; i++) { var len = random.next(chars.length); var str = new char[len]; (int j = 0; j < len; j++) { str[j] = chars[random.next(chars.length)]; } var s = new string(str); if(!hashes.add(get64bithash( s ))) collisions++; } console.writeline("collision percentage after " + n + " random strings: " + ((doubl
Read more

webrtc - Which ICE candidate am I using and why? -

-
[questions in bold below] i have setup kurento media server 5.1.3 in datacenter behind firewall running os ubuntu 14.04. has 2 network cards: 222.222.222.222 (eth0 - private ip) 111.111.111.111 (eth1 - public ip) attached below sdp (setremotedescription) when browser connected kurento media server type: answer, sdp: v=0 o=- 5487318114793304426 0 in ip4 0.0.0.0 s=kurento media server c=in ip4 0.0.0.0 t=0 0 a=group:bundle audio video m=audio 59068 rtp/savpf 111 0 c=in ip4 111.111.111.111 a=rtpmap:111 opus/48000/2 a=rtpmap:0 pcmu/8000 a=sendrecv a=rtcp:59068 in ip4 111.111.111.111 a=rtcp-mux a=ssrc:669011897 cname:user39019747@host-6e83e4c2 a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=mid:audio b=as:20 a=ice-ufrag:ymdk a=ice-pwd:lylifk5ueqzpwm91ddj37e a=fingerprint:sha-256 ff:0f:81:8c:41:4e:b4:b6:c6:d8:36:f3:d6:5f:09:fd:5f:af:13:b3:9d:fc:12:66:ac:f3:56:d6:5b:0a:73:5d a=candidate:1 1 udp 2013266431 111.111.111.111 55239 typ host a=candidate:2 1 udp
Read more

php - Zend Framework / Skeleton-Application / Composer install issue -

-
i trying create zend framework application using skeleton in netbeans. when run composer install, following error message: "c:\php\php.exe" "c:\composer\composer.phar" "--ansi" "--no-interaction" "update" "--dev" using deprecated option "dev". dev packages installed default now. loading composer repositories package information updating dependencies (including require-dev) requirements not resolved installable set of packages. problem 1 - package requires php >=5.5 php version (5.4.42) not satisfy requirement. problem 2 - installation request zendframework/zendframework 2.5.1 -> satisfiable zendframework/zendframework[2.5.1]. - zendframework/zendframework 2.5.1 requires php >=5.5 -> php version (5.4.42) not satisfy requirement. done. apparently, zend framework only needs php 5.3+ contradicts above. need use php 5.4. my composer.json following: { "name": "
Read more