scrapy.core.downloader.handlers.s3.S3DownloadHandler

Here are the examples of the python api scrapy.core.downloader.handlers.s3.S3DownloadHandler taken from open source projects. By voting up you can indicate which examples are most useful and appropriate.

3 Examples 7

Example 1

Project: scrapy
Source File: test_downloader_handlers.py
View license
    def setUp(self):
        skip_if_no_boto()
        self.s3reqh = S3DownloadHandler(Settings(),
                httpdownloadhandler=HttpDownloadHandlerMock,
                #anon=True, # is implicit
        )
        self.download_request = self.s3reqh.download_request
        self.spider = Spider('foo')

Example 2

Project: scrapy
Source File: test_downloader_handlers.py
View license
    def setUp(self):
        skip_if_no_boto()
        s3reqh = S3DownloadHandler(Settings(), self.AWS_ACCESS_KEY_ID,
                self.AWS_SECRET_ACCESS_KEY,
                httpdownloadhandler=HttpDownloadHandlerMock)
        self.download_request = s3reqh.download_request
        self.spider = Spider('foo')

Example 3

Project: scrapy
Source File: test_downloader_handlers.py
View license
    def test_extra_kw(self):
        try:
            S3DownloadHandler(Settings(), extra_kw=True)
        except Exception as e:
            self.assertIsInstance(e, (TypeError, NotConfigured))
        else:
            assert False