scrapelib.Scraper.get.text

Here are the examples of the python api scrapelib.Scraper.get.text taken from open source projects. By voting up you can indicate which examples are most useful and appropriate.

3 Examples 7

Example 1

Project: openstates Source File: __init__.py
Function: session_list
def session_list():
    import scrapelib
    import lxml.html

    url = 'http://www.legis.nd.gov/assembly/'
    html = scrapelib.Scraper().get(url).text
    doc = lxml.html.fromstring(html)
    doc.make_links_absolute(url)
    return doc.xpath("//div[@class='view-content']//a/text()")

Example 2

Project: openstates Source File: __init__.py
Function: session_list
def session_list():
    import scrapelib
    text = scrapelib.Scraper().get('ftp://ftp.cga.ct.gov').text
    sessions = [line.split()[-1] for line in text.splitlines()]
    
    for not_session_name in ('incoming', 'pub', 'CGAAudio', 'rba', 'NCSL',"apaac"):
        sessions.remove(not_session_name)
    return sessions

Example 3

Project: openstates Source File: __init__.py
Function: session_list
def session_list():
    html = scrapelib.Scraper().get('http://legis.sd.gov/Legislative_Session/'
        'Menu.aspx').text
    doc = lxml.html.fromstring(html)
    sessions = doc.xpath('//div[@id="ContentPlaceHolder1_BlueBoxLeft"]//ul/li'
        '/a/div/text()')
    return sessions