Skip to content

Instantly share code, notes, and snippets.

@nyov

nyov/requests_from_generator.py

Created February 28, 2014 06:38
Download ZIP

Revisions

  1. nyov created this gist Feb 28, 2014.
    9 changes: 9 additions & 0 deletions requests_from_generator.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,9 @@
    class MySpider(Spider):
    # [...]

    # start requests from generator
    def start_requests(self):
    url = 'http://some.page.tld/%s/category'
    for page in xrange(1, 247):
    link = url % page
    yield Request(url=link)