Practical Python Web Crawler

This book describes the core technology of using Python to develop web crawlers step by step from principle to practice. The whole book can be logically divided into three parts: basics, practical and crawler framework. The Basics chapter mainly introduces the basic knowledge required for writing web crawlers, including website analysis, data crawling, data cleaning and data warehousing. Website analysis describes how to use Chrome and Fiddler packet capture tools to conduct a comprehensive analysis of the website; data capture introduces the basic knowledge of the Python crawler module Urllib and Requests; data cleaning mainly introduces the use of string operations, regular expressions and BeautifulSoup; data warehousing describes the operations of MySQL and MongoDB, achieving data persistence through the ORM framework SQLAlchemy, and conducting enterprise-level development. The practical chapter explains in depth the development of distributed crawlers, crawler software, the 12306 ticket grabbing program, and Weibo crawling. The framework chapter mainly talks about the popular crawler framework Scrapy, and uses project cases that combine Scrapy with Selenium, Splash, and Redis to give readers an in-depth understanding of the use of Scrapy. In addition, this book also introduces the online deployment of crawlers, how to develop a crawler framework yourself, anti-crawler technology solutions, etc. This book is written in Python 3.