Member-only story
How to scrape any website with Python and Beautiful Soup (dynamic web)
Python Beautiful soup tutorial for scraping any webpage, with youtube video — and a focus on web drivers
Note: This is a purely technical tutorial. Please check with the policies of the website before engaging in any scraping. In the example I use, it’s not legal to scrape the site: https://gitcoin.co//legal/terms. Please do not use.
For those who want to see it done in front of your eyes, check out my YouTube video at the bottom of the page.
Now if you didn’t read my part one, go back and read it! This is building on the concepts learnt there. And remember, the documentation for this is very strong, so be sure to check it out after this tutorial!
We are now going to learn how to deal with dynamic web pages or web pages that have a lot going on and are more than just HTML/CSS. These pages are tricker to scrape with beautifulsoup, because sometimes, code happens on the server side, and beautifulsoup needs to let a browser run it.
Wouldn’t it be nice if there was a way for us to code and have our code pretend it was a browser?… Well it just so happens that’s exactly what this article is about!
