Scrapy Cores

Now that we learned about Scrapy, let's dive into more detail about its core modules.

The Spider Class

The scrapy.Spider class is the heart of any Scrapy project. It defines how to crawl and extract information from a website. Let's delve into some of the critical parameters that can be utilized within this class to fine-tune our web scraping process.

• name

  • It uniquely identifies our spider, which is crucial when running multiple spiders within a single project. This name differentiates the output files, logs, and other spider-related resources. We should choose a descriptive and meaningful name for our spider.

• allowed_domains

  • The allowed_domains parameter is a list of domains that our spider is allowed to crawl. Any links outside these domains will be ignored. This handy feature ensures our spider stays focused on the relevant content.

• start_urls

  • This parameter is a list of URLs where the spider begins crawling.

  • start_urls serves as a shortcut for start_requests() functions. If this parameter is defined and we didn't define the start_requests() function, Scrapy will internally initialize this function for us with this list of URLs.

• custom_settings

  • Since Scrapy is designed to run multiple spiders, it gives us the capability to modify the project's default settings for each ...