r - Can I use readLines in mapreduce job in Rhadoop? -

-

i'm trying read text or gz file hdfs , run simple mapreduce job (actually map job) got error seems readlines part doesn't work. i'm seeking answers of whether can use readlines function in mapreduce. ps. there no problem if use readlines function parse hdfs files outside of mapreduce job. thanks.

counts <- function(path){         ct.map <- function(., lines) {         line <- readlines(lines)         word <- unlist(strsplit(line, pattern = " "))         keyval(word, 1)     }      mapreduce(     input = path,     input.format = "text",     map = ct.map         ) } counts("/user/ychen/100.txt")

not - mapping function expects dfs formatted data come in. rewrite function this, formatting in input step:

counts <- function(path){   ct.map <- function(.,line) {     word <- unlist(strsplit(line, split = " "))     keyval(word, 1)   }    mapreduce(     input = to.dfs(readlines(path)),     map = function(k,v){ct.map(k,v)},     reduce = function(k,v){keyval(k,length(v))}   ) } output<-from.dfs(counts("/user/ychen/100.txt"))

i added in reduce step, sum values.


Comments

Post a Comment

Popular posts from this blog

c++ - Qt setGeometry: Unable to set geometry -

-
why? setgeometry: unable set geometry 22x22+320+145 on qwidgetwindow/'widgetclasswindow'. resulting geometry: 116x22+320+145 (frame: 8, 30, 8, 8, custom margin: 0, 0, 0, 0, minimum size: 22x22, maximum size: 16777215x16777215). the project is: project.pro qt += core gui greaterthan(qt_major_version, 4): qt += widgets target = untitled5 template = app sources += main.cpp\ widget.cpp headers += widget.h widget.h #ifndef widget_h #define widget_h #include <qwidget> class widget : public qwidget { q_object public: explicit widget(qwidget *parent = 0); ~widget(); private: }; #endif // widget_h widget.cpp #include "widget.h" #include <qvboxlayout> widget::widget(qwidget *parent) : qwidget(parent) { qvboxlayout *vlayout = new qvboxlayout(this); } widget::~widget() { } main.cpp #include "widget.h" #include <qapplication> int main(int argc, char *argv[]) { qapplication a...
Read more

python - argument must be rect style object - Pygame -

-
disclaimer: know other people on site have had error it's been in different contexts , fixes don't seem work here. hey stackoverflow, i've been making sideways space invaders game, ship follows mouse's y position stays on left, , clicking shoots bullet. enemies spawn randomly right , fly left, , objects removed once each side of screen. works fine, when tried implement collision detection see when shot hits enemy, error: typeerror: argument must rect style object. it points line "return hitbox.colliderect(target.rect)". try , work out causing made 'print' lines below see 'target.rect' (and whether proper rect object), , seemed same style shot's own rect, i'm lost why colliderect doesn't accept it. i've annotated pretty everything, sorry if it's insult intelligence should make things easier. class causes problems: class gameobject: def __init__(self, image, height, speed): self.speed = speed self.image...
Read more

How to resolve Delphi error: Incompatible types: 'PWideChar' and 'Pointer' -

-
i have piece of code delphi 7: var lprgndata: prgndata; pc: pchar; pr: prect; ... pc := @(lprgndata^.buffer[0]); in delphi xe4 gives following compile error: incompatible types: 'pwidechar' , 'pointer' how should code updated work correctly in xe4? thanks whether or not compiles depends upon setting of type-checked pointers option . have enabled option excellent decision. doing results in stricter type checking. with type-checked pointers disabled, code compile. type-checked pointers enabled, code not compile, want because code not valid. now, on types in question. defined in windows unit this: type prgndata = ^trgndata; {$externalsym _rgndata} _rgndata = record rdh: trgndataheader; buffer: array[0..0] of byte; reserved: array[0..2] of byte; end; trgndata = _rgndata; {$externalsym rgndata} rgndata = _rgndata; the benefit of using type-checked pointers compiler can tell doing not valid. knows lprgndat...
Read more