Python Scrapy 爬虫框架：分布式数据采集实战指南

Python Scrapy 爬虫框架：分布式数据采集实战指南

Scrapy 是 Python 生态中最强大的爬虫框架，它异步、可扩展、内置数据管道。本文将带你从单站点爬虫到分布式集群，掌握生产级数据采集的完整技术栈。

一、Scrapy 核心架构

Engine ──▶ Scheduler ──▶ Downloader ──▶ Spiders ──▶ Item Pipeline
   │           │              │             │              │
   └───────────┴──────────────┴─────────────┴──────────────┘
                         数据流

| 组件 | 职责 | |------|------| | Engine | 控制数据流，触发事件 | | Scheduler | URL 队列管理，去重 | | Downloader | 异步下载页面 | | Spiders | 解析规则，提取数据 | | Item Pipeline | 数据清洗、存储 | | Middleware | 请求/响应处理 |

二、快速开始

pip install scrapy
scrapy startproject ecommerce_crawler
cd ecommerce_crawler

基础 Spider

# spiders/product_spider.py
import scrapy
from ecommerce_crawler.items import ProductItem

class ProductSpider(scrapy.Spider):
    name = 'products'
    allowed_domains = ['example-shop.com']
    start_urls = ['https://example-shop.com/products']
    
    custom_settings = {
        'DOWNLOAD_DELAY': 1,
        'CONCURRENT_REQUESTS': 16,
        'RETRY_TIMES': 3,
        'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
    }
    
    def parse(self, response):
        """解析商品列表页"""
        products = response.css('div.product-card')
        
        for product in products:
            item = ProductItem()
            item['name'] = product.css('h2.title::text').get(default='').strip()
            item['price'] = self.extract_price(product.css('span.price::text').get())
            item['url'] = response.urljoin(product.css('a::attr(href)').get())
            item['category'] = response.css('h1.category-title::text').get()
            
            # 进入详情页
            yield scrapy.Request(
                url=item['url'],
                callback=self.parse_detail,
                meta={'item': item}
            )
        
        # 分页
        next_page = response.css('a.next-page::attr(href)').get()
        if next_page:
            yield scrapy.Request(response.urljoin(next_page), callback=self.parse)
    
    def parse_detail(self, response):
        """解析商品详情页"""
        item = response.meta['item']
        item['description'] = response.css('div.description::text').get(default='')
        item['sku'] = response.css('span.sku::text').get()
        item['stock'] = response.css('span.stock::text').get()
        item['images'] = response.css('img.product-image::attr(src)').getall()
        item['crawl_time'] = datetime.now().isoformat()
        
        yield item
    
    def extract_price(self, price_text):
        """提取价格数值"""
        if not price_text:
            return None
        match = re.search(r'[\d,]+\.?\d*', price_text.replace(',', ''))
        return float(match.group()) if match else None

Item 定义

# i

转载自 CSDN-专业IT技术社区

原文链接：https://blog.csdn.net/qq_16593231/article/details/163732107