Skip to content

[Dev Research] NUWorks scraping tech plan #102

@joshw2048

Description

@joshw2048

We want a good way to get the appropriate roles into Cooper - and one idea worth exploring is scraping NUWorks to pull all co-ops into our database. This is a huge undertaking, so let's consider:

  • pros and cons of scraping
  • feasibility of scraping (is it worth it)
  • how it would be done
    • is it easier to get all the companies are get all the roles & reverse engineer the companies?
    • what does the data look like on NUWorks?
    • How can we build some adapter that gives us seed data for our database?

Think of some more questions and come up with some answers that will educate us on next steps.
This ticket requires some knowledge of our database schema, so familiarize yourself with that and ask Alton any questions!

Metadata

Metadata

Labels

StoryUser stories

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions