Apache spark: setting spark.eventLog.enabled and spark.eventLog.dir at submit or Spark start -

-

i set spark.eventlog.enabled , spark.eventlog.dir @ spark-submit or start-all level -- not require enabled in scala/java/python code. have tried various things no success:

setting spark-defualts.conf as

spark.eventlog.enabled           true spark.eventlog.dir               hdfs://namenode:8021/directory

or

spark.eventlog.enabled           true spark.eventlog.dir               file:///some/where

running spark-submit as:

spark-submit --conf "spark.eventlog.enabled=true" --conf "spark.eventlog.dir=file:///tmp/test" --master spark://server:7077 examples/src/main/python/pi.py

starting spark environment variables:

spark_daemon_java_opts="-dspark.eventlog.enabled=true -dspark.history.fs.logdirectory=$sparkhistorydir -dspark.history.provider=org.apache.spark.deploy.history.fshistoryprovider -dspark.history.fs.cleaner.enabled=true -dspark.history.fs.cleaner.interval=2d"

and overkill:

spark_history_opts="-dspark.eventlog.enabled=true -dspark.history.fs.logdirectory=$sparkhistorydir -dspark.history.provider=org.apache.spark.deploy.history.fshistoryprovider -dspark.history.fs.cleaner.enabled=true -dspark.history.fs.cleaner.interval=2d"

where , how must these things set history on arbitrary jobs?

i solved problem, yet strangely had tried before... same, seems stable solution:

create directory in hdfs logging, /eventlogging

hdfs dfs -mkdir /eventlogging

then spark-shell or spark-submit (or whatever) can run following options: 

--conf spark.eventlog.enabled=true --conf spark.eventlog.dir=hdfs://<hdfsnamenodeaddress>:8020/eventlogging

such as:

spark-shell --conf spark.eventlog.enabled=true --conf spark.eventlog.dir=hdfs://<hdfsnamenodeaddress>:8020/eventlogging

Comments

Post a Comment

Popular posts from this blog

python - argument must be rect style object - Pygame -

-
disclaimer: know other people on site have had error it's been in different contexts , fixes don't seem work here. hey stackoverflow, i've been making sideways space invaders game, ship follows mouse's y position stays on left, , clicking shoots bullet. enemies spawn randomly right , fly left, , objects removed once each side of screen. works fine, when tried implement collision detection see when shot hits enemy, error: typeerror: argument must rect style object. it points line "return hitbox.colliderect(target.rect)". try , work out causing made 'print' lines below see 'target.rect' (and whether proper rect object), , seemed same style shot's own rect, i'm lost why colliderect doesn't accept it. i've annotated pretty everything, sorry if it's insult intelligence should make things easier. class causes problems: class gameobject: def __init__(self, image, height, speed): self.speed = speed self.image...
Read more

c++ - Qt setGeometry: Unable to set geometry -

-
why? setgeometry: unable set geometry 22x22+320+145 on qwidgetwindow/'widgetclasswindow'. resulting geometry: 116x22+320+145 (frame: 8, 30, 8, 8, custom margin: 0, 0, 0, 0, minimum size: 22x22, maximum size: 16777215x16777215). the project is: project.pro qt += core gui greaterthan(qt_major_version, 4): qt += widgets target = untitled5 template = app sources += main.cpp\ widget.cpp headers += widget.h widget.h #ifndef widget_h #define widget_h #include <qwidget> class widget : public qwidget { q_object public: explicit widget(qwidget *parent = 0); ~widget(); private: }; #endif // widget_h widget.cpp #include "widget.h" #include <qvboxlayout> widget::widget(qwidget *parent) : qwidget(parent) { qvboxlayout *vlayout = new qvboxlayout(this); } widget::~widget() { } main.cpp #include "widget.h" #include <qapplication> int main(int argc, char *argv[]) { qapplication a...
Read more

How to resolve Delphi error: Incompatible types: 'PWideChar' and 'Pointer' -

-
i have piece of code delphi 7: var lprgndata: prgndata; pc: pchar; pr: prect; ... pc := @(lprgndata^.buffer[0]); in delphi xe4 gives following compile error: incompatible types: 'pwidechar' , 'pointer' how should code updated work correctly in xe4? thanks whether or not compiles depends upon setting of type-checked pointers option . have enabled option excellent decision. doing results in stricter type checking. with type-checked pointers disabled, code compile. type-checked pointers enabled, code not compile, want because code not valid. now, on types in question. defined in windows unit this: type prgndata = ^trgndata; {$externalsym _rgndata} _rgndata = record rdh: trgndataheader; buffer: array[0..0] of byte; reserved: array[0..2] of byte; end; trgndata = _rgndata; {$externalsym rgndata} rgndata = _rgndata; the benefit of using type-checked pointers compiler can tell doing not valid. knows lprgndat...
Read more