1.. _topics-architecture:
2
3=====================
4Architecture overview
5=====================
6
7This document describes the architecture of Scrapy and how its components
8interact.
9
10Overview
11========
12
13The following diagram shows an overview of the Scrapy architecture with its
14components and an outline of the data flow that takes place inside the system
15(shown by the red arrows). A brief description of the components is included
16below with links for more detailed information about them. The data flow is
17also described below.
18
19.. _data-flow:
20
21Data flow
22=========
23
24.. image:: _images/scrapy_architecture_02.png
25   :width: 700
26   :height: 470
27   :alt: Scrapy architecture
28
29The data flow in Scrapy is controlled by the execution engine, and goes like
30this:
31
321. The :ref:`Engine <component-engine>` gets the initial Requests to crawl from the
33   :ref:`Spider <component-spiders>`.
34
352. The :ref:`Engine <component-engine>` schedules the Requests in the
36   :ref:`Scheduler <component-scheduler>` and asks for the
37   next Requests to crawl.
38
393. The :ref:`Scheduler <component-scheduler>` returns the next Requests
40   to the :ref:`Engine <component-engine>`.
41
424. The :ref:`Engine <component-engine>` sends the Requests to the
43   :ref:`Downloader <component-downloader>`, passing through the
44   :ref:`Downloader Middlewares <component-downloader-middleware>` (see
45   :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_request`).
46
475. Once the page finishes downloading the
48   :ref:`Downloader <component-downloader>` generates a Response (with
49   that page) and sends it to the Engine, passing through the
50   :ref:`Downloader Middlewares <component-downloader-middleware>` (see
51   :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_response`).
52
536. The :ref:`Engine <component-engine>` receives the Response from the
54   :ref:`Downloader <component-downloader>` and sends it to the
55   :ref:`Spider <component-spiders>` for processing, passing
56   through the :ref:`Spider Middleware <component-spider-middleware>` (see
57   :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_input`).
58
597. The :ref:`Spider <component-spiders>` processes the Response and returns
60   scraped items and new Requests (to follow) to the
61   :ref:`Engine <component-engine>`, passing through the
62   :ref:`Spider Middleware <component-spider-middleware>` (see
63   :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`).
64
658. The :ref:`Engine <component-engine>` sends processed items to
66   :ref:`Item Pipelines <component-pipelines>`, then send processed Requests to
67   the :ref:`Scheduler <component-scheduler>` and asks for possible next Requests
68   to crawl.
69
709. The process repeats (from step 1) until there are no more requests from the
71   :ref:`Scheduler <component-scheduler>`.
72
73Components
74==========
75
76.. _component-engine:
77
78Scrapy Engine
79-------------
80
81The engine is responsible for controlling the data flow between all components
82of the system, and triggering events when certain actions occur. See the
83:ref:`Data Flow <data-flow>` section above for more details.
84
85.. _component-scheduler:
86
87Scheduler
88---------
89
90The Scheduler receives requests from the engine and enqueues them for feeding
91them later (also to the engine) when the engine requests them.
92
93.. _component-downloader:
94
95Downloader
96----------
97
98The Downloader is responsible for fetching web pages and feeding them to the
99engine which, in turn, feeds them to the spiders.
100
101.. _component-spiders:
102
103Spiders
104-------
105
106Spiders are custom classes written by Scrapy users to parse responses and
107extract :ref:`items <topics-items>` from them or additional requests to
108follow. For more information see :ref:`topics-spiders`.
109
110.. _component-pipelines:
111
112Item Pipeline
113-------------
114
115The Item Pipeline is responsible for processing the items once they have been
116extracted (or scraped) by the spiders. Typical tasks include cleansing,
117validation and persistence (like storing the item in a database). For more
118information see :ref:`topics-item-pipeline`.
119
120.. _component-downloader-middleware:
121
122Downloader middlewares
123----------------------
124
125Downloader middlewares are specific hooks that sit between the Engine and the
126Downloader and process requests when they pass from the Engine to the
127Downloader, and responses that pass from Downloader to the Engine.
128
129Use a Downloader middleware if you need to do one of the following:
130
131* process a request just before it is sent to the Downloader
132  (i.e. right before Scrapy sends the request to the website);
133* change received response before passing it to a spider;
134* send a new Request instead of passing received response to a spider;
135* pass response to a spider without fetching a web page;
136* silently drop some requests.
137
138For more information see :ref:`topics-downloader-middleware`.
139
140.. _component-spider-middleware:
141
142Spider middlewares
143------------------
144
145Spider middlewares are specific hooks that sit between the Engine and the
146Spiders and are able to process spider input (responses) and output (items and
147requests).
148
149Use a Spider middleware if you need to
150
151* post-process output of spider callbacks - change/add/remove requests or items;
152* post-process start_requests;
153* handle spider exceptions;
154* call errback instead of callback for some of the requests based on response
155  content.
156
157For more information see :ref:`topics-spider-middleware`.
158
159Event-driven networking
160=======================
161
162Scrapy is written with `Twisted`_, a popular event-driven networking framework
163for Python. Thus, it's implemented using a non-blocking (aka asynchronous) code
164for concurrency.
165
166For more information about asynchronous programming and Twisted see these
167links:
168
169* :doc:`twisted:core/howto/defer-intro`
170* `Twisted - hello, asynchronous programming`_
171* `Twisted Introduction - Krondo`_
172
173.. _Twisted: https://twistedmatrix.com/trac/
174.. _Twisted - hello, asynchronous programming: http://jessenoller.com/blog/2009/02/11/twisted-hello-asynchronous-programming/
175.. _Twisted Introduction - Krondo: http://krondo.com/an-introduction-to-asynchronous-programming-and-twisted/
176