Python web scraping : fetching data from the web

Successfully scrape data from any website with the power of Python 3.x About This Book • A hands-on guide to web scraping using Python with solutions to real-world problems • Create a number of different web scrapers in Python to extract information • This book includes practical examples on using the popular and well-maintained libraries in Python for your web scraping needs Who This Book Is For This book is aimed at developers who want to use web scraping for legitimate purposes. Prior programming experience with Python would be useful but not essential. Anyone with general knowledge of programming languages should be able to pick up the book and understand the principals involved. What You Will Learn • Extract data from web pages with simple Python programming • Build a concurrent crawler to process web pages in parallel • Follow links to crawl a website • Extract features from the HTML • Cache downloaded HTML for reuse • Compare concurrent models to determine the fastest crawler • Find out how to parse JavaScript-dependent websites • Interact with forms and sessions In Detail The Internet contains the most useful set of data ever assembled, most of which is publicly accessible for free. However, this data is not easily usable. It is embedded within the structure and style of websites and needs to be carefully extracted. Web scraping is becoming increasingly useful as a means to gather and make sense of the wealth of information available online. This book is the ultimate guide to using the latest features of Python 3.x to scrape data from websites. In the early chapters, you'll see how to extract data from static web pages. You'll learn to use caching with databases and files to save time and manage the load on servers. After covering the basics, you'll get hands-on practice building a more sophisticated crawler using browsers, crawlers, and concurrent scrapers. You'll determine when and how to scrape data from a JavaScript-dependent website using PyQt and Selenium. You'll get a better understanding of how to submit forms on complex websites protected by CAPTCHA. You'll find out how to automate these actions with Python packages such as mechanize. You'll also learn how to create class-based scrapers with Scrapy libraries and implement your learning on real websites. By the end of the book, you will have explored testing websites with scrapers, remote scraping, best practices, working with images, and many other relevant topics. Style and approach This hands-on guide is full of real-life examples and solutions starting simple and then progressively becoming more complex. Each chapter in this book introduces a problem and then provides one or more possible solutions. Cover......Page 1 Credits......Page 4 Copyright......Page 3 About the Authors......Page 5 About the Reviewers......Page 6 www.PacktPub.com......Page 7 Customer Feedback......Page 8 Table of Contents......Page 9 Preface......Page 13 When is web scraping useful?......Page 19 Is web scraping legal?......Page 20 Python 3......Page 21 Checking robots.txt......Page 22 Estimating the size of a website......Page 23 Identifying the technology used by a website......Page 25 Finding the owner of a website......Page 28 Scraping versus crawling......Page 29 Downloading a web page......Page 30 Retrying downloads......Page 31 Setting a user agent......Page 32 Sitemap crawler......Page 33 ID iteration crawler......Page 34 Link crawlers......Page 37 Parsing robots.txt......Page 40 Supporting proxies......Page 41 Throttling downloads......Page 42 Avoiding spider traps......Page 43 Final version......Page 44 Using the requests library......Page 45 Summary......Page 46 Chapter 2: Scraping the Data......Page 47 Analyzing a web page......Page 48 Regular expressions......Page 51 Beautiful Soup......Page 53 Lxml......Page 56 CSS selectors and your Browser Console......Page 57 XPath Selectors......Page 60 LXML and Family Trees......Page 63 Comparing performance......Page 64 Scraping results......Page 65 Overview of Scraping......Page 67 Adding a scrape callback to the link crawler......Page 68 Summary......Page 71 When to use caching?......Page 72 Adding cache support to the link crawler......Page 73 Disk Cache......Page 75 Implementing DiskCache......Page 77 Testing the cache......Page 79 Saving disk space......Page 80 Expiring stale data......Page 81 Drawbacks of DiskCache......Page 82 Key-value storage cache......Page 83 Installing Redis......Page 84 Overview of Redis......Page 85 Redis cache implementation......Page 87 Compression......Page 88 Testing the cache......Page 89 Exploring requests-cache......Page 90 Summary......Page 92 One million web pages......Page 93 Parsing the Alexa list......Page 94 Sequential crawler......Page 95 How threads and processes work......Page 97 Implementing a multithreaded crawler......Page 98 Multiprocessing crawler......Page 100 Performance......Page 104 Python multiprocessing and the GIL......Page 105 Summary......Page 106 An example dynamic web page......Page 107 Reverse engineering a dynamic web page......Page 110 Edge cases......Page 114 Rendering a dynamic web page......Page 116 Debugging with Qt......Page 117 Executing JavaScript......Page 118 Website interaction with WebKit......Page 119 Waiting for results......Page 122 The Render class......Page 123 Selenium......Page 125 Selenium and Headless Browsers......Page 127 Summary......Page 129 Chapter 6: Interacting with Forms......Page 131 The Login form......Page 132 Loading cookies from the web browser......Page 136 Extending the login script to update content......Page 140 Automating forms with Selenium......Page 144 BookmarkTitle:......Page 146 Summary......Page 147 Chapter 7: Solving CAPTCHA......Page 148 Registering an account......Page 149 Loading the CAPTCHA image......Page 150 Optical character recognition......Page 152 Further improvements......Page 155 Getting started with 9kw......Page 156 The 9kw CAPTCHA API......Page 157 Reporting errors......Page 162 Integrating with registration......Page 163 CAPTCHAs and machine learning......Page 164 Summary......Page 165 Installing Scrapy......Page 166 Starting a project......Page 167 Defining a model......Page 168 Creating a spider......Page 169 Tuning settings......Page 170 Testing the spider......Page 171 Different Spider Types......Page 173 Scraping with the shell command......Page 174 Checking results......Page 176 Interrupting and resuming a crawl......Page 178 Visual scraping with Portia......Page 180 Installation......Page 181 Annotation......Page 183 Running the Spider......Page 188 Checking results......Page 189 Automated scraping with Scrapely......Page 190 Summary......Page 191 Google search engine......Page 192 Facebook......Page 197 The website......Page 198 Facebook API......Page 200 Gap......Page 202 BMW......Page 206 Summary......Page 210 Index......Page 211 Cover -- Credits -- Copyright -- About the Authors -- About the Reviewers -- www.PacktPub.com -- Customer Feedback -- Table of Contents -- Preface -- Chapter 1: Introduction to Web Scraping -- When is web scraping useful? -- Is web scraping legal? -- Python 3 -- Background research -- Checking robots.txt -- Examining the Sitemap -- Estimating the size of a website -- Identifying the technology used by a website -- Finding the owner of a website -- Crawling your first website -- Scraping versus crawling -- Downloading a web page -- Retrying downloads -- Setting a user agent -- Sitemap crawler -- ID iteration crawler -- Link crawlers -- Advanced features -- Parsing robots.txt -- Supporting proxies -- Throttling downloads -- Avoiding spider traps -- Final version -- Using the requests library -- Summary -- Chapter 2: Scraping the Data -- Analyzing a web page -- Three approaches to scrape a web page -- Regular expressions -- Beautiful Soup -- Lxml -- CSS selectors and your Browser Console -- XPath Selectors -- LXML and Family Trees -- Comparing performance -- Scraping results -- Overview of Scraping -- Adding a scrape callback to the link crawler -- Summary -- Chapter 3: Caching Downloads -- When to use caching? -- Adding cache support to the link crawler -- Disk Cache -- Implementing DiskCache -- Testing the cache -- Saving disk space -- Expiring stale data -- Drawbacks of DiskCache -- Key-value storage cache -- What is key-value storage? -- Installing Redis -- Overview of Redis -- Redis cache implementation -- Compression -- Testing the cache -- Exploring requests-cache -- Summary -- Chapter 4: Concurrent Downloading -- One million web pages -- Parsing the Alexa list -- Sequential crawler -- Threaded crawler -- How threads and processes work -- Implementing a multithreaded crawler -- Multiprocessing crawler -- Performance About the Author Katharine Jarmul is a data scientist and Pythonista based in Berlin, Germany. She runs a data science consulting company, Kjamistan, that provides services such as data extraction, acquisition, and modelling for small and large companies. She has been writing Python since 2008 and scraping the web with Python since 2010, and has worked at both small and large start-ups who use web scraping for data analysis and machine learning. When she's not scraping the web, you can follow her thoughts and activities via Twitter (@kjam) Richard Lawson is from Australia and studied Computer Science at the University of Melbourne. Since graduating, he built a business specializing in web scraping while travelling the world, working remotely from over 50 countries. He is a fluent Esperanto speaker, conversational in Mandarin and Korean, and active in contributing to and translating open source software. He is currently undertaking postgraduate studies at Oxford University and in his spare time enjoys developing autonomous drones Chapter 5: Dynamic Content ; An example dynamic web page; Reverse engineering a dynamic web page; Edge cases; Rendering a dynamic web page; PyQt or PySide; Debugging with Qt; Executing JavaScript; Website interaction with WebKit; Waiting for results; The Render class; Selenium; Selenium and Headless Browsers; Summary; Chapter 6: Interacting with Forms ; The Login form; Loading cookies from the web browser; Extending the login script to update content; Automating forms with Selenium; ; Summary; Chapter 7: Solving CAPTCHA ; Registering an account; Loading the CAPTCHA image