Spider doesn't work with YouTube's new layout

[m-sterling]

In 2019, YouTube updated its layout dubbed the "polymer layout." Because of this, the HTML structure changed and the spider no longer functions correctly. However, I have found two workarounds.

1. According to this thread post, you can modify your cookies to include &f6=8 on the PREF cookie's value to revert the page structure back to the pre-2019 layout. Modifying the cookies in youtube_cookies.json to include this works as expected and the spider crawls through the history successfully.

2. Adding the GET parameter disable_polymer=1 to any YouTube URL reverts the page back to the old layout. However, this does not carry over when clicking on links, as it defaults back to the new layout. Modifying the start_url variable in youtube_history_spider.py to have ?disable_polymer=1 at the end successfully returns the first history page, but prematurely ends the crawl.

I'm sure the code can be modified to support the new layout, but I'm not sure how much work that could be. Additionally, while the first workaround is the easiest to do, it may not work forever. ¯\_(ツ)_/¯ Great project though, love it!

[AB1908]

I can confirm it still works. If it doesn't, give my fork a shot.