r - Can I use readLines in mapreduce job in Rhadoop? -

-

i'm trying read text or gz file hdfs , run simple mapreduce job (actually map job) got error seems readlines part doesn't work. i'm seeking answers of whether can use readlines function in mapreduce. ps. there no problem if use readlines function parse hdfs files outside of mapreduce job. thanks.

counts <- function(path){         ct.map <- function(., lines) {         line <- readlines(lines)         word <- unlist(strsplit(line, pattern = " "))         keyval(word, 1)     }      mapreduce(     input = path,     input.format = "text",     map = ct.map         ) } counts("/user/ychen/100.txt")

not - mapping function expects dfs formatted data come in. rewrite function this, formatting in input step:

counts <- function(path){   ct.map <- function(.,line) {     word <- unlist(strsplit(line, split = " "))     keyval(word, 1)   }    mapreduce(     input = to.dfs(readlines(path)),     map = function(k,v){ct.map(k,v)},     reduce = function(k,v){keyval(k,length(v))}   ) } output<-from.dfs(counts("/user/ychen/100.txt"))

i added in reduce step, sum values.


Comments

Post a Comment

Popular posts from this blog

python - argument must be rect style object - Pygame -

-
disclaimer: know other people on site have had error it's been in different contexts , fixes don't seem work here. hey stackoverflow, i've been making sideways space invaders game, ship follows mouse's y position stays on left, , clicking shoots bullet. enemies spawn randomly right , fly left, , objects removed once each side of screen. works fine, when tried implement collision detection see when shot hits enemy, error: typeerror: argument must rect style object. it points line "return hitbox.colliderect(target.rect)". try , work out causing made 'print' lines below see 'target.rect' (and whether proper rect object), , seemed same style shot's own rect, i'm lost why colliderect doesn't accept it. i've annotated pretty everything, sorry if it's insult intelligence should make things easier. class causes problems: class gameobject: def __init__(self, image, height, speed): self.speed = speed self.image...
Read more

c++ - Qt setGeometry: Unable to set geometry -

-
why? setgeometry: unable set geometry 22x22+320+145 on qwidgetwindow/'widgetclasswindow'. resulting geometry: 116x22+320+145 (frame: 8, 30, 8, 8, custom margin: 0, 0, 0, 0, minimum size: 22x22, maximum size: 16777215x16777215). the project is: project.pro qt += core gui greaterthan(qt_major_version, 4): qt += widgets target = untitled5 template = app sources += main.cpp\ widget.cpp headers += widget.h widget.h #ifndef widget_h #define widget_h #include <qwidget> class widget : public qwidget { q_object public: explicit widget(qwidget *parent = 0); ~widget(); private: }; #endif // widget_h widget.cpp #include "widget.h" #include <qvboxlayout> widget::widget(qwidget *parent) : qwidget(parent) { qvboxlayout *vlayout = new qvboxlayout(this); } widget::~widget() { } main.cpp #include "widget.h" #include <qapplication> int main(int argc, char *argv[]) { qapplication a...
Read more

php - Zend Framework / Skeleton-Application / Composer install issue -

-
i trying create zend framework application using skeleton in netbeans. when run composer install, following error message: "c:\php\php.exe" "c:\composer\composer.phar" "--ansi" "--no-interaction" "update" "--dev" using deprecated option "dev". dev packages installed default now. loading composer repositories package information updating dependencies (including require-dev) requirements not resolved installable set of packages. problem 1 - package requires php >=5.5 php version (5.4.42) not satisfy requirement. problem 2 - installation request zendframework/zendframework 2.5.1 -> satisfiable zendframework/zendframework[2.5.1]. - zendframework/zendframework 2.5.1 requires php >=5.5 -> php version (5.4.42) not satisfy requirement. done. apparently, zend framework only needs php 5.3+ contradicts above. need use php 5.4. my composer.json following: { "name": ...
Read more