Tony Wang, A Software Engineer | RSS Feed

The Infra to handle 10M Requests in 10 Minutes for $0.0116

27.6% of the Top 10 Million Sites are Dead

Web Crawling at Scale: Navigating Billions of URLs with Efficiency

The Architecture of a Web Crawler: Building a Google-Inspired Distributed Web Crawler. Part 1

How to efficiently scrape millions of Google Businesses on a large scale using a distributed crawler

A Step-by-Step Guide to Building a Scalable Distributed Crawler for Scraping Millions of Top TikTok Profiles

Deploy your distributed system efficiently with fabric

How to build a scalable crawler to crawl million pages with a single machine in just 2 hours

How to build docker cluster with celery and RabbitMQ in 10 minutes