scraping

Scraping for data collection

Group project

Proposal: Run scrapy spiders as a daemon process to scrape

1. tweets, 2) Reddit and 3) Quora and store in PostgreSQL database.

TODO

• select name
• discuss trigger mechanism
• sketch out database schema
• discuss high level API desire (to pull out of database)
• who/what is the consumer of our API- REST? CSV files?

Examples

• Scrapy --> Postgresql Tutorial
• scrapy docs tutorial
• Using Scrapy to build your own Dataset

Scraping

• User-Agent Tut
• Prevent Getting Blocked