scrapi.processing.cassandra

Here are the examples of the python api scrapi.processing.cassandra taken from open source projects. By voting up you can indicate which examples are most useful and appropriate.

1 Examples 7

Example 1

Project: scrapi Source File: conftest.py
def pytest_runtest_setup(item):
    TIMEOUT = 20

    marker = item.get_marker('cassandra')
    if marker is not None:
        from scrapi.processing.cassandra import DocuementModel
        if not database.setup():
            pytest.skip('No connection to Cassandra')

        start = time.time()
        while True:
            try:
                DocuementModel.all().limit(1).get()
                break
            except NoHostAvailable as e:
                now = time.time()
                if (now - start) > TIMEOUT:
                    raise e
                continue
            except Exception:
                break


    marker = item.get_marker('elasticsearch')
    if marker is not None:
        if not use_es:
            pytest.skip('No connection to Elasticsearch')
        con.indices.create(index='test', body={}, ignore=400)

        # This is done to let the test index finish being created before connecting to search
        start = time.time()
        while True:
            try:
                scrapi.processing.elasticsearch.ElasticsearchProcessor.manager.es.search(index='test')
                break
            except TransportError as e:
                now = time.time()
                if (now - start) > TIMEOUT:
                    raise e
                continue