python 2.7 - Problems using Scrapy to scrape craigslist.org -

-

i'm using following set-up scrape prices http://puertorico.craigslist.org/search/sya

def parse(self, response):         items = selector(response).xpath("//p[@class='row']")         items in items:             item = stackitem()             item['prices'] = items.xpath("//a[@class='i']/span[@class='price']/text()").extract()             item['url'] = items.xpath("//span[@class='href']/@href").extract()               yield item

my problem when run script, of prices shown each item.. how can price of particular item url each available item?

you can try using realtive xpath expression based in context of previous one, traversing nodes until 1 want. also, use different variable in for loop, like:

import scrapy craiglist.items import stackitem  class craiglistspider(scrapy.spider):     name = "craiglist_spider"     allowed_domains = ["puertorico.craigslist.org"]     start_urls = (         'http://puertorico.craigslist.org/search/sya',     )      def parse(self, response):         items = response.selector.xpath("//p[@class='row']")         in items:             item = stackitem()             item['prices'] = i.xpath("./a/span[@class='price']/text()").extract()             item['url'] = i.xpath("./span[@class='txt']/span[@class='pl']/a/@href").extract()               yield item

it yields:

{'prices': [u'$130'], 'url': [u'/sys/5105448465.html']} 2015-07-05 23:58:22 [scrapy] debug: scraped <200 http://puertorico.craigslist.org/search/sya> {'prices': [u'$250'], 'url': [u'/sys/5096083890.html']} 2015-07-05 23:58:22 [scrapy] debug: scraped <200 http://puertorico.craigslist.org/search/sya> {'prices': [u'$100'], 'url': [u'/sys/5069848699.html']} 2015-07-05 23:58:22 [scrapy] debug: scraped <200 http://puertorico.craigslist.org/search/sya> {'prices': [u'$35'], 'url': [u'/syd/5007870110.html']} ...

Comments

Post a Comment

Popular posts from this blog

c++ - Qt setGeometry: Unable to set geometry -

-
why? setgeometry: unable set geometry 22x22+320+145 on qwidgetwindow/'widgetclasswindow'. resulting geometry: 116x22+320+145 (frame: 8, 30, 8, 8, custom margin: 0, 0, 0, 0, minimum size: 22x22, maximum size: 16777215x16777215). the project is: project.pro qt += core gui greaterthan(qt_major_version, 4): qt += widgets target = untitled5 template = app sources += main.cpp\ widget.cpp headers += widget.h widget.h #ifndef widget_h #define widget_h #include <qwidget> class widget : public qwidget { q_object public: explicit widget(qwidget *parent = 0); ~widget(); private: }; #endif // widget_h widget.cpp #include "widget.h" #include <qvboxlayout> widget::widget(qwidget *parent) : qwidget(parent) { qvboxlayout *vlayout = new qvboxlayout(this); } widget::~widget() { } main.cpp #include "widget.h" #include <qapplication> int main(int argc, char *argv[]) { qapplication a...
Read more

python - argument must be rect style object - Pygame -

-
disclaimer: know other people on site have had error it's been in different contexts , fixes don't seem work here. hey stackoverflow, i've been making sideways space invaders game, ship follows mouse's y position stays on left, , clicking shoots bullet. enemies spawn randomly right , fly left, , objects removed once each side of screen. works fine, when tried implement collision detection see when shot hits enemy, error: typeerror: argument must rect style object. it points line "return hitbox.colliderect(target.rect)". try , work out causing made 'print' lines below see 'target.rect' (and whether proper rect object), , seemed same style shot's own rect, i'm lost why colliderect doesn't accept it. i've annotated pretty everything, sorry if it's insult intelligence should make things easier. class causes problems: class gameobject: def __init__(self, image, height, speed): self.speed = speed self.image...
Read more

How to resolve Delphi error: Incompatible types: 'PWideChar' and 'Pointer' -

-
i have piece of code delphi 7: var lprgndata: prgndata; pc: pchar; pr: prect; ... pc := @(lprgndata^.buffer[0]); in delphi xe4 gives following compile error: incompatible types: 'pwidechar' , 'pointer' how should code updated work correctly in xe4? thanks whether or not compiles depends upon setting of type-checked pointers option . have enabled option excellent decision. doing results in stricter type checking. with type-checked pointers disabled, code compile. type-checked pointers enabled, code not compile, want because code not valid. now, on types in question. defined in windows unit this: type prgndata = ^trgndata; {$externalsym _rgndata} _rgndata = record rdh: trgndataheader; buffer: array[0..0] of byte; reserved: array[0..2] of byte; end; trgndata = _rgndata; {$externalsym rgndata} rgndata = _rgndata; the benefit of using type-checked pointers compiler can tell doing not valid. knows lprgndat...
Read more