r - Can I use readLines in mapreduce job in Rhadoop? -

-

i'm trying read text or gz file hdfs , run simple mapreduce job (actually map job) got error seems readlines part doesn't work. i'm seeking answers of whether can use readlines function in mapreduce. ps. there no problem if use readlines function parse hdfs files outside of mapreduce job. thanks.

counts <- function(path){         ct.map <- function(., lines) {         line <- readlines(lines)         word <- unlist(strsplit(line, pattern = " "))         keyval(word, 1)     }      mapreduce(     input = path,     input.format = "text",     map = ct.map         ) } counts("/user/ychen/100.txt")

not - mapping function expects dfs formatted data come in. rewrite function this, formatting in input step:

counts <- function(path){   ct.map <- function(.,line) {     word <- unlist(strsplit(line, split = " "))     keyval(word, 1)   }    mapreduce(     input = to.dfs(readlines(path)),     map = function(k,v){ct.map(k,v)},     reduce = function(k,v){keyval(k,length(v))}   ) } output<-from.dfs(counts("/user/ychen/100.txt"))

i added in reduce step, sum values.


Comments

Post a Comment

Popular posts from this blog

android - questions about switching from C2DM to GCM -

-
i'm converting app c2dm gcm , have questions. i've modified gcmquickstart example found here: git clone https://github.com/googlesamples/google-services.git in class extends application, i've got: public class myapp extends application { @override public void oncreate() { super.oncreate(); if (this.checkplayservices()) { intent intent = new intent(this, registrationintentservice.class); startservice(intent); } else{ log.e(tag,"failed checkforgcm"); } } } afaik, starts service , it'll run forever or if operating system shuts down whereupon operating system restart it. correct? checkplayservices() makes sure android 2.3 or greater , google play services installed these requirements use gcm. correct? public class registrationintentservice extends intentservice{ private static final string tag = "regintentservice"; public registratio...
Read more

c++ - rosrun via ssh doesnt start a node properly -

-
we working ros , try run node via ssh in shell script. have controlling (master) computer on execute script following line: ssh user@xxx.xxx.x.x -x "/bin/bash -c 'source /etc/profile; rosrun stargazer_alter stargazer_node'" the node getting started (it occurs in "rosnode list" , node listed publisher in "rostopic info") main method seems not executed (neither print lines show nor messages published). #include "stargazer_listener.h" enum states {config, readdata}; int main(int argc, char** argv) { ros_info("test!!!! stargazer starts!"); ros::init(argc, argv, "stargazer_node"); stargazer_listener starg; int ifd = 0; char *rec_msg = (char*)malloc(buffsize * sizeof(char)); ifd = starg.portsetup(); states state = config; while(ros::ok()) { switch(state) { case config: starg.cmd(ifd, rec_msg, "~#calcstop`"); printf("calcstop\n"); s...
Read more

Error C2280 mutex in a class C++ -

-
i'm having issue declaring mutex in class, whenever attempt instantiate class error c2280 appears. project create simulation of dining philosopher's problem. declaration: class fork { public: struct _position { float x; float y; }; fork(); fork(int num_phil_, int new_index_); _position returnworkposition(){ _position return_position_; /*position_mutex_.lock();*/ return_position_ = work_position_; /*position_mutex_.unlock();*/ return return_position_; }; float returnrotation(){ return rotation_; }; void calculateworkposition(int master_index_); void resetworkposition(){ /*position_mutex_.lock();*/ work_position_ = orig_position_; /*position_mutex_.unlock();*/ }; void takeownership(int master_index_); void relinquishownership(); private: int index_; int num_philosophers_; float rotation_; _position orig_position_; _position work_position_; //std::mutex master_mutex_; //std::mutex position_mutex_; }; instantiation: f...
Read more