Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #54619 > unrolled thread

How To Build Scrapy Environment

Started byYetToCome <wjw15129@gmail.com>
First post2013-09-23 01:12 -0700
Last post2013-09-23 01:55 -0700
Articles 6 — 2 participants

Back to article view | Back to comp.lang.python

Contents

  How To Build Scrapy Environment YetToCome <wjw15129@gmail.com> - 2013-09-23 01:12 -0700
    Re: How To Build Scrapy Environment Peter Otten <__peter__@web.de> - 2013-09-23 10:37 +0200
      Re: How To Build Scrapy Environment YetToCome <wjw15129@gmail.com> - 2013-09-23 01:56 -0700
        Re: How To Build Scrapy Environment Peter Otten <__peter__@web.de> - 2013-09-23 11:25 +0200
          Re: How To Build Scrapy Environment YetToCome <wjw15129@gmail.com> - 2013-09-23 03:11 -0700
    Re: How To Build Scrapy Environment YetToCome <wjw15129@gmail.com> - 2013-09-23 01:55 -0700

#54619 — How To Build Scrapy Environment

FromYetToCome <wjw15129@gmail.com>
Date2013-09-23 01:12 -0700
SubjectHow To Build Scrapy Environment
Message-ID<1dad0f6e-0241-4892-ad32-afb2097f09a3@googlegroups.com>
I have already install Twisted, zope.interface, w3lib, libxml2, etc,but it still can not be built,here is the error message:

Traceback (most recent call last):
  File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 167, in <module>
    execute()
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 142, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 88, in _run_print
_help
    func(*a, **kw)
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 149, in _run_comm
and
    cmd.run(args, opts)
  File "C:\Python27\lib\site-packages\scrapy\commands\crawl.py", line 47, in run
    crawler = self.crawler_process.create_crawler()
  File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 142, in create_cr
awler
    self.crawlers[name] = Crawler(self.settings)
  File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 23, in __init__
    self.spiders = spman_cls.from_crawler(self)
  File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 35, in from
_crawler
    sm = cls.from_settings(crawler.settings)
  File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 31, in from
_settings
    return cls(settings.getlist('SPIDER_MODULES'))
  File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 22, in __in
it__
    for module in walk_modules(name):
  File "C:\Python27\lib\site-packages\scrapy\utils\misc.py", line 66, in walk_mo
dules
    submod = __import__(fullpath, {}, {}, [''])
  File "blog_crawl\spiders\dmoz_spider.py", line 1, in <module>
    class DmozSpider(BaseSpider):
NameError: name 'BaseSpider' is not defined

someone tells me why, pls

[toc] | [next] | [standalone]

#54621

FromPeter Otten <__peter__@web.de>
Date2013-09-23 10:37 +0200
Message-ID<mailman.256.1379925407.18130.python-list@python.org>
In reply to#54619
YetToCome wrote:

> I have already install Twisted, zope.interface, w3lib, libxml2, etc,but it
> still can not be built,here is the error message:
> 
> Traceback (most recent call last):
>   File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
>     "__main__", fname, loader, pkg_name)
>   File "C:\Python27\lib\runpy.py", line 72, in _run_code
>     exec code in run_globals
>   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 167, in
>   <module>
>     execute()
>   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 142, in
>   execute
>     _run_print_help(parser, _run_command, cmd, args, opts)
>   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 88, in
>   _run_print
> _help
>     func(*a, **kw)
>   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 149, in
>   _run_comm
> and
>     cmd.run(args, opts)
>   File "C:\Python27\lib\site-packages\scrapy\commands\crawl.py", line 47,
>   in run
>     crawler = self.crawler_process.create_crawler()
>   File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 142, in
>   create_cr
> awler
>     self.crawlers[name] = Crawler(self.settings)
>   File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 23, in
>   __init__
>     self.spiders = spman_cls.from_crawler(self)
>   File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 35,
>   in from
> _crawler
>     sm = cls.from_settings(crawler.settings)
>   File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 31,
>   in from
> _settings
>     return cls(settings.getlist('SPIDER_MODULES'))
>   File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 22,
>   in __in
> it__
>     for module in walk_modules(name):
>   File "C:\Python27\lib\site-packages\scrapy\utils\misc.py", line 66, in
>   walk_mo
> dules
>     submod = __import__(fullpath, {}, {}, [''])
>   File "blog_crawl\spiders\dmoz_spider.py", line 1, in <module>
>     class DmozSpider(BaseSpider):
> NameError: name 'BaseSpider' is not defined
> 
> someone tells me why, pls

Assuming you are working your way through the tutorial at

http://doc.scrapy.org/en/latest/intro/tutorial.html#our-first-spider

you probably forgot to import the BaseSpider class with

from scrapy.spider import BaseSpider

as shown in the code snippet in the "Our first Spider" section of the 
tutorial.

[toc] | [prev] | [next] | [standalone]

#54623

FromYetToCome <wjw15129@gmail.com>
Date2013-09-23 01:56 -0700
Message-ID<cee80741-f886-4845-ae1a-38171d87d69d@googlegroups.com>
In reply to#54621
在 2013年9月23日星期一UTC+8下午4时37分22秒，Peter Otten写道：
> YetToCome wrote:
> 
> 
> 
> > I have already install Twisted, zope.interface, w3lib, libxml2, etc,but it
> 
> > still can not be built,here is the error message:
> 
> > 
> 
> > Traceback (most recent call last):
> 
> >   File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
> 
> >     "__main__", fname, loader, pkg_name)
> 
> >   File "C:\Python27\lib\runpy.py", line 72, in _run_code
> 
> >     exec code in run_globals
> 
> >   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 167, in
> 
> >   <module>
> 
> >     execute()
> 
> >   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 142, in
> 
> >   execute
> 
> >     _run_print_help(parser, _run_command, cmd, args, opts)
> 
> >   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 88, in
> 
> >   _run_print
> 
> > _help
> 
> >     func(*a, **kw)
> 
> >   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 149, in
> 
> >   _run_comm
> 
> > and
> 
> >     cmd.run(args, opts)
> 
> >   File "C:\Python27\lib\site-packages\scrapy\commands\crawl.py", line 47,
> 
> >   in run
> 
> >     crawler = self.crawler_process.create_crawler()
> 
> >   File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 142, in
> 
> >   create_cr
> 
> > awler
> 
> >     self.crawlers[name] = Crawler(self.settings)
> 
> >   File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 23, in
> 
> >   __init__
> 
> >     self.spiders = spman_cls.from_crawler(self)
> 
> >   File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 35,
> 
> >   in from
> 
> > _crawler
> 
> >     sm = cls.from_settings(crawler.settings)
> 
> >   File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 31,
> 
> >   in from
> 
> > _settings
> 
> >     return cls(settings.getlist('SPIDER_MODULES'))
> 
> >   File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 22,
> 
> >   in __in
> 
> > it__
> 
> >     for module in walk_modules(name):
> 
> >   File "C:\Python27\lib\site-packages\scrapy\utils\misc.py", line 66, in
> 
> >   walk_mo
> 
> > dules
> 
> >     submod = __import__(fullpath, {}, {}, [''])
> 
> >   File "blog_crawl\spiders\dmoz_spider.py", line 1, in <module>
> 
> >     class DmozSpider(BaseSpider):
> 
> > NameError: name 'BaseSpider' is not defined
> 
> > 
> 
> > someone tells me why, pls
> 
> 
> 
> Assuming you are working your way through the tutorial at
> 
> 
> 
> http://doc.scrapy.org/en/latest/intro/tutorial.html#our-first-spider
> 
> 
> 
> you probably forgot to import the BaseSpider class with
> 
> 
> 
> from scrapy.spider import BaseSpider
> 
> 
> 
> as shown in the code snippet in the "Our first Spider" section of the 
> 
> tutorial.

it had an another error: No module named queuelib, but i have installed all the libs mentioned in that passage...

2013-09-23 16:44:17+0800 [scrapy] INFO: Scrapy 0.18.2 started (bot: tutorial)
2013-09-23 16:44:17+0800 [scrapy] DEBUG: Optional features available: ssl, http1
1, libxml2
2013-09-23 16:44:17+0800 [scrapy] DEBUG: Overridden settings: {'NEWSPIDER_MODULE
': 'tutorial.spiders', 'SPIDER_MODULES': ['tutorial.spiders'], 'BOT_NAME': 'tuto
rial'}
2013-09-23 16:44:17+0800 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetCon
sole, CloseSpider, WebService, CoreStats, SpiderState
Traceback (most recent call last):
  File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 167, in <module>
    execute()
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 142, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 88, in _run_print
_help
    func(*a, **kw)
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 149, in _run_comm
and
    cmd.run(args, opts)
  File "C:\Python27\lib\site-packages\scrapy\commands\crawl.py", line 50, in run

    self.crawler_process.start()
  File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 93, in start
    if self.start_crawling():
  File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 168, in start_cra
wling
    return self.start_crawler() is not None
  File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 151, in start_cra
wler
    crawler.configure()
  File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 45, in configure
    self.engine = ExecutionEngine(self, self._spider_closed)
  File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 61, in __init
__
    self.scheduler_cls = load_object(self.settings['SCHEDULER'])
  File "C:\Python27\lib\site-packages\scrapy\utils\misc.py", line 40, in load_ob
ject
    raise ImportError, "Error loading object '%s': %s" % (path, e)
ImportError: Error loading object 'scrapy.core.scheduler.Scheduler': No module n
amed queuelib

[toc] | [prev] | [next] | [standalone]

#54624

FromPeter Otten <__peter__@web.de>
Date2013-09-23 11:25 +0200
Message-ID<mailman.257.1379928294.18130.python-list@python.org>
In reply to#54623
YetToCome wrote:

[snip]

No need to quote the whole turd -- just confirm that it worked...

> it had an another error: No module named queuelib, but i have installed
> all the libs mentioned in that passage...

> ImportError: Error loading object 'scrapy.core.scheduler.Scheduler': No
> module n amed queuelib

Googling "queuelib site:scrapy.org" leads to

http://doc.scrapy.org/en/latest/news.html

containing

"""
0.18.0 (released 2013-08-09)¶
[...]
Moved persistent (on disk) queues to a separate project (queuelib) which 
scrapy now depends on
"""

which in turn leads to

https://github.com/scrapy/queuelib

[toc] | [prev] | [next] | [standalone]

#54626

FromYetToCome <wjw15129@gmail.com>
Date2013-09-23 03:11 -0700
Message-ID<82583230-c40d-4916-9c7b-060f09a49906@googlegroups.com>
In reply to#54624
在 2013年9月23日星期一UTC+8下午5时25分25秒，Peter Otten写道：
> YetToCome wrote:
> 
> 
> 
> [snip]
> 
> 
> 
> No need to quote the whole turd -- just confirm that it worked...
> 
> 
> 
> > it had an another error: No module named queuelib, but i have installed
> 
> > all the libs mentioned in that passage...
> 
> 
> 
> > ImportError: Error loading object 'scrapy.core.scheduler.Scheduler': No
> 
> > module n amed queuelib
> 
> 
> 
> Googling "queuelib site:scrapy.org" leads to
> 
> 
> 
> http://doc.scrapy.org/en/latest/news.html
> 
> 
> 
> containing
> 
> 
> 
> """
> 
> 0.18.0 (released 2013-08-09)¶
> 
> [...]
> 
> Moved persistent (on disk) queues to a separate project (queuelib) which 
> 
> scrapy now depends on
> 
> """
> 
> 
> 
> which in turn leads to
> 
> 
> 
> https://github.com/scrapy/queuelib

it works, and i also add some mudules, thank you very much!!!

[toc] | [prev] | [next] | [standalone]

#54622

FromYetToCome <wjw15129@gmail.com>
Date2013-09-23 01:55 -0700
Message-ID<d78896c3-f8ff-4678-964e-9393d7e64696@googlegroups.com>
In reply to#54619
在 2013年9月23日星期一UTC+8下午4时12分21秒，YetToCome写道：
> I have already install Twisted, zope.interface, w3lib, libxml2, etc,but it still can not be built,here is the error message:
> 
> 
> 
> Traceback (most recent call last):
> 
>   File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
> 
>     "__main__", fname, loader, pkg_name)
> 
>   File "C:\Python27\lib\runpy.py", line 72, in _run_code
> 
>     exec code in run_globals
> 
>   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 167, in <module>
> 
>     execute()
> 
>   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 142, in execute
> 
>     _run_print_help(parser, _run_command, cmd, args, opts)
> 
>   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 88, in _run_print
> 
> _help
> 
>     func(*a, **kw)
> 
>   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 149, in _run_comm
> 
> and
> 
>     cmd.run(args, opts)
> 
>   File "C:\Python27\lib\site-packages\scrapy\commands\crawl.py", line 47, in run
> 
>     crawler = self.crawler_process.create_crawler()
> 
>   File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 142, in create_cr
> 
> awler
> 
>     self.crawlers[name] = Crawler(self.settings)
> 
>   File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 23, in __init__
> 
>     self.spiders = spman_cls.from_crawler(self)
> 
>   File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 35, in from
> 
> _crawler
> 
>     sm = cls.from_settings(crawler.settings)
> 
>   File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 31, in from
> 
> _settings
> 
>     return cls(settings.getlist('SPIDER_MODULES'))
> 
>   File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 22, in __in
> 
> it__
> 
>     for module in walk_modules(name):
> 
>   File "C:\Python27\lib\site-packages\scrapy\utils\misc.py", line 66, in walk_mo
> 
> dules
> 
>     submod = __import__(fullpath, {}, {}, [''])
> 
>   File "blog_crawl\spiders\dmoz_spider.py", line 1, in <module>
> 
>     class DmozSpider(BaseSpider):
> 
> NameError: name 'BaseSpider' is not defined
> 
> 
> 
> someone tells me why, pls



在 2013年9月23日星期一UTC+8下午4时12分21秒，YetToCome写道：
> I have already install Twisted, zope.interface, w3lib, libxml2, etc,but it still can not be built,here is the error message:
> 
> 
> 
> Traceback (most recent call last):
> 
>   File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
> 
>     "__main__", fname, loader, pkg_name)
> 
>   File "C:\Python27\lib\runpy.py", line 72, in _run_code
> 
>     exec code in run_globals
> 
>   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 167, in <module>
> 
>     execute()
> 
>   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 142, in execute
> 
>     _run_print_help(parser, _run_command, cmd, args, opts)
> 
>   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 88, in _run_print
> 
> _help
> 
>     func(*a, **kw)
> 
>   File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 149, in _run_comm
> 
> and
> 
>     cmd.run(args, opts)
> 
>   File "C:\Python27\lib\site-packages\scrapy\commands\crawl.py", line 47, in run
> 
>     crawler = self.crawler_process.create_crawler()
> 
>   File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 142, in create_cr
> 
> awler
> 
>     self.crawlers[name] = Crawler(self.settings)
> 
>   File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 23, in __init__
> 
>     self.spiders = spman_cls.from_crawler(self)
> 
>   File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 35, in from
> 
> _crawler
> 
>     sm = cls.from_settings(crawler.settings)
> 
>   File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 31, in from
> 
> _settings
> 
>     return cls(settings.getlist('SPIDER_MODULES'))
> 
>   File "C:\Python27\lib\site-packages\scrapy\spidermanager.py", line 22, in __in
> 
> it__
> 
>     for module in walk_modules(name):
> 
>   File "C:\Python27\lib\site-packages\scrapy\utils\misc.py", line 66, in walk_mo
> 
> dules
> 
>     submod = __import__(fullpath, {}, {}, [''])
> 
>   File "blog_crawl\spiders\dmoz_spider.py", line 1, in <module>
> 
>     class DmozSpider(BaseSpider):
> 
> NameError: name 'BaseSpider' is not defined
> 
> 
> 
> someone tells me why, pls

it had an another error: No module named queuelib, but i have installed all the libs mentioned in that passage...

2013-09-23 16:44:17+0800 [scrapy] INFO: Scrapy 0.18.2 started (bot: tutorial)
2013-09-23 16:44:17+0800 [scrapy] DEBUG: Optional features available: ssl, http1
1, libxml2
2013-09-23 16:44:17+0800 [scrapy] DEBUG: Overridden settings: {'NEWSPIDER_MODULE
': 'tutorial.spiders', 'SPIDER_MODULES': ['tutorial.spiders'], 'BOT_NAME': 'tuto
rial'}
2013-09-23 16:44:17+0800 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetCon
sole, CloseSpider, WebService, CoreStats, SpiderState
Traceback (most recent call last):
  File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 167, in <module>
    execute()
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 142, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 88, in _run_print
_help
    func(*a, **kw)
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 149, in _run_comm
and
    cmd.run(args, opts)
  File "C:\Python27\lib\site-packages\scrapy\commands\crawl.py", line 50, in run

    self.crawler_process.start()
  File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 93, in start
    if self.start_crawling():
  File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 168, in start_cra
wling
    return self.start_crawler() is not None
  File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 151, in start_cra
wler
    crawler.configure()
  File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 45, in configure
    self.engine = ExecutionEngine(self, self._spider_closed)
  File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 61, in __init
__
    self.scheduler_cls = load_object(self.settings['SCHEDULER'])
  File "C:\Python27\lib\site-packages\scrapy\utils\misc.py", line 40, in load_ob
ject
    raise ImportError, "Error loading object '%s': %s" % (path, e)
ImportError: Error loading object 'scrapy.core.scheduler.Scheduler': No module n
amed queuelib

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web