Member-only story
How to scrape ANY website with python and beautiful soup (static web)
Python Beautiful soup tutorial for scraping any web page, with youtube video
Note: This is a purely technical tutorial. Please check with the policies of the website before engaging in any scraping.
For those who want to see it done in front of your eyes, check out my YouTube video at the bottom of the page.
Scraping the web can be done for a TON of reasons.
Do you want to get stats on your football team so you can algorithmically manage your fantasy team? Boom, make a web scraper that scrapes ESPN. Track your competitor's activity on different social media? Great, that’s covered here too.
Or maybe you’re a Developer Advocate who is looking for good ways to measure his OKR of hackathon involvement and there is no current good tool out there so you want to build your own.
That last one was oddly specific, and is what we are going to be looking for! This tutorial shows how you can get all the hackathons from devpost that are ending in the next 50 days, based on the keyword blockchain .
Anyway, let’s jump right into how we can scrape anything with python. I’m going to assume you have space where you can code, and are familiar…
