Trying to extract UNSC Resolutions with various Python scraping frameworks

[jnothman]

I've tried implementing a standalone scraper for the target suggested by the Library Carpentry folks (https://github.com/data-lessons/library-webscraping/issues/11).

Note that there are a few quirks here (to be expected for scraping manually-constructed pages):

• Most of the link HREFs are relative (e.g. 1980.shtml) in the index page but others are absolute on the domain (e.g. /en/sc/documents/resolutions/2015.shtml)
• 2013 has an unusual quirk in its table where the first content row is actually column headers
• 1960 and 1964 include the entire page in duplicate!
• 2014 has a date column
• 2017 has a loose </tr>. this happens to break bs4's parsing with 'html.parser' engine, such that the data is not recoverable!
• Varying use of whitespace, particularly in resolution codes: S/RES/1939 (2010) vs S/RES/2025 (2011) vs S/RES/2132\n (2013)
• It's possible I missed other quirks because they didn't upset my scraping technique

Implemented so far:

• Beautiful Soup with CSS Selectors
• selenium.webdriver with CSS Selectors
• Scrapy with CSS Selectors
• lxml with CSS Selectors
• selenium.webdriver with XPath
• Scrapy with XPath
• lxml with XPath
• pandas.read_html ?

[jnothman]

Yes, am adding another quirk.

[jnothman]

I've pushed a webdriver version (ignoring the 2014 quirk), but I should note that I tried it at first with PhantomJS and seemed to land on a bug (which I'm not sure how to even characterise and report), which is not a great sign.

The webdriver version is much slower. Getting the index plus 10 years takes:

• 3m56s with webdriver
• 43s with bs4

(But both could benefit from parallelism)

This suggests not selenium.

[ostephens]

Does it have to be Python?
Another option might be PhantomJS + CasperJS (Casper can use CSS selectors or XPath)

[jnothman]

I don't require it to be Python (though this is by far my expertise), but in the context of *Carpentry, requiring javascript knowledge rather than Python (beyond DOM) is a bit strange.

I don't expect Casper + Phantom will be much faster than using webdriver, but I may be wrong. I also don't see why you can't just use Phantom without Casper. But my experiments so far in that direction have not been a success. And as I said above, I landed on a bug on Phantom (or perhaps in webdriver.PhantomJS()) and ran away.

[jnothman]

Yes, PhantomJS + CasperJS appears to be much faster than PhantomJS + webdriver + python. But I don't think we can afford to teach JavaScript in this context.

[ostephens]

I'm not strongly arguing for this, but I don't think we should feel limited in our choice of languages in general terms - as long is this isn't going to create problems in terms of setup and ability for users to go away with something they can use.

There is obviously a question of any pre-requisites for people attending the course - and if we feel that experience with a specific language is a pre-requisite to doing this web scraping course it would definitely make sense for that to be Python as we already have that course in the carpentries.

[jnothman]

FWIW, CasperJS is not proving the answer for me: trying to write it, I land up in confusion over whether some fragment of code is being run within the page or within the driver. I land up with a whole lot of closures and asynchronous futures that are just not going to work for people with an ounce of Python and little more. Pjscrape might offer something more tailored (its scraper definitions look a little more like scrapy's).

[jnothman]

And I should say I'm really appreciating your feedback @ostephens.

Should I presume that those suggesting UNSC resolutions as a target knew, or did not know, of its quirks?

[jnothman]

Added another quirk for 2017

[ostephens]

Thanks @jnothman :)

I don't really know if those suggesting UNSC res. as a target knew the quirks - but they definitely felt it was representative of the type of data they needed to scrape as librarians - so perhaps quirks and all is good?

That said, I don't think you should necessarily feel bound by those decisions - if no-one is here defending them we've got to still feel able to make progress :)

[jnothman]

Okay. I'm thinking lxml is the best pick. It has some minor usability flaws relative to bs4, but:

• You need to use lxml underneath to get bs4 to parse 2014.shtml (also for scrapy), so we're not going to avoid any installation difficulties. I could be persuaded to stay with bs4 for installation reasons.
• It supports both xpath and css selectors so can be used to illustrate both if need be

And I'm wishing webdriver wasn't so sluggish as it gets around lots of pitfalls and quirks.

[ostephens]

I've just been having a dig around and I'm inclined to agree. Potentially a mix of lxml and bs4 would be an option, but then I guess we end up trying to cover two approaches in the lesson which seems like a recipe for confusion.

Of course we could use Ruby Mechanize and Nokogiri instead :) (only joking!)

[ldko]

@runderwood @vphill, were the quirks of scraping the UNSC resolutions pages something you had taken into account when looking at them for use with the web scraping lesson?