Scrapy Plugins

All

32 repositories

scrapy-zyte-api
Public
Zyte API integration for Scrapy
plugin crawler proxy scraping scrapy
Python
•
BSD 3-Clause "New" or "Revised" License
•19•36•10•5•Updated Nov 19, 2024Nov 19, 2024
scrapy-playwright
Public
🎭 Playwright integration for Scrapy
hacktoberfest chrome-headless python-asyncio headless-browser javascript-renderer firefox-headless playwright playwright-python webkit-headless python
Python
•
BSD 3-Clause "New" or "Revised" License
•113•1k•19•2•Updated Nov 14, 2024Nov 14, 2024
scrapy-zyte-smartproxy
Public
Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
plugin crawler proxy scraping scrapy crawler-detection
Python
•
BSD 3-Clause "New" or "Revised" License
•88•356•6•1•Updated Nov 7, 2024Nov 7, 2024
scrapy-spider-metadata
Public
Python
•
MIT License
•2•4•2•0•Updated Oct 16, 2024Oct 16, 2024
zyte-spidermon
Public
Python
•0•0•0•0•Updated Sep 5, 2024Sep 5, 2024
scrapy-settings-log
Public
An extension that allows a user to display all or some of their scrapy spider settings at runtime.
Python
•
GNU General Public License v3.0
•4•0•0•0•Updated Aug 20, 2024Aug 20, 2024
scrapy-feedexporter-dropbox
Public
Scrapy feed exporter for Dropbox
Python
•3•1•0•0•Updated Jul 11, 2024Jul 11, 2024
scrapy-crawlera-fetch
Public archive
Scrapy Downloader Middleware for Crawlera Fetch API
Python
•
BSD 3-Clause "New" or "Revised" License
•10•8•5•5•Updated Jun 5, 2024Jun 5, 2024
scrapy-feedexporter-azure-storage
Public
Python
•5•2•0•3•Updated May 18, 2024May 18, 2024
scrapy-feedexporter-onedrive
Public
Export to OneDrive
Python
•1•1•0•0•Updated Feb 23, 2024Feb 23, 2024
scrapy-djangoitem
Public
Scrapy extension to write scraped items using Django models
Python
•
BSD 3-Clause "New" or "Revised" License
•87•498•10•3•Updated Oct 15, 2023Oct 15, 2023
scrapy-feedexporter-google-sheets
Public
Python
•0•1•0•0•Updated Aug 15, 2023Aug 15, 2023
scrapy-incremental
Public
Python
•
BSD 3-Clause "New" or "Revised" License
•1•1•0•0•Updated May 1, 2023May 1, 2023
scrapy-splash
Public
Scrapy+Splash for JavaScript integration
scrapy headless-browsers
Python
•
BSD 3-Clause "New" or "Revised" License
•451•3.2k•64•15•Updated Feb 6, 2023Feb 6, 2023
scrapy-feedexporter-google-drive
Public
Python
•2•3•1•0•Updated Jan 4, 2023Jan 4, 2023
scrapy-statsd
Public
Python
•
BSD 3-Clause "New" or "Revised" License
•4•6•1•3•Updated Oct 19, 2022Oct 19, 2022
scrapy-dotpersistence
Public
A scrapy extension to sync `.scrapy` folder to an S3 bucket
Python
•
BSD 3-Clause "New" or "Revised" License
•8•17•2•1•Updated Mar 28, 2022Mar 28, 2022
scrapy-magicfields
Public
Scrapy middleware to add extra fields to items, like timestamp, response fields, spider attributes etc.
Python
•
BSD 3-Clause "New" or "Revised" License
•7•56•0•0•Updated Mar 16, 2022Mar 16, 2022
scrapy-pagestorage
Public
A scrapy extension to store requests and responses information in storage service
Python
•
BSD 3-Clause "New" or "Revised" License
•6•26•2•0•Updated Mar 11, 2022Mar 11, 2022
scrapy-snowflake-stage-exporter
Public
Snowflake database loading utility with Scrapy integration
snowflake scrapy etl
Python
•2•4•0•0•Updated Dec 3, 2021Dec 3, 2021
scrapy-deltafetch
Public
Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls
hacktoberfest hacktoberfest2021
Python
•49•268•14•5•Updated Oct 17, 2021Oct 17, 2021
scrapy-headless
Public
Python
•
BSD 3-Clause "New" or "Revised" License
•8•29•2•2•Updated Apr 28, 2021Apr 28, 2021
scrapy-jsonschema
Public
Scrapy schema validation pipeline and Item builder using JSON Schema
Python
•
BSD 3-Clause "New" or "Revised" License
•12•44•1•0•Updated Mar 26, 2021Mar 26, 2021
scrapy-feedexporter-sftp
Public
Python
•
BSD 3-Clause "New" or "Revised" License
•3•6•1•1•Updated May 14, 2020May 14, 2020
scrapy-jsonrpc
Public
Scrapy extension to control spiders using JSON-RPC
Python
•72•296•7•2•Updated Aug 26, 2019Aug 26, 2019
scrapy-hcf
Public
Scrapy spider middleware to use Scrapinghub's Hub Crawl Frontier as a backend for URLs
Python
•
BSD 3-Clause "New" or "Revised" License
•6•4•1•1•Updated Aug 28, 2018Aug 28, 2018
scrapy-monkeylearn
Public
A Scrapy pipeline to categorize items using MonkeyLearn
Python
•13•37•0•0•Updated Apr 28, 2017Apr 28, 2017
scrapy-splitvariants
Public
Scrapy spider middleware to split an item into multiple items using a multi-valued key
Python
•
BSD 3-Clause "New" or "Revised" License
•1•20•0•0•Updated Feb 8, 2017Feb 8, 2017
scrapy-streaming
Public
Python
•5•18•2•6•Updated Oct 12, 2016Oct 12, 2016
scrapy-querycleaner
Public
Scrapy spider middleware to clean up query parameters in request URLs
Python
•
BSD 3-Clause "New" or "Revised" License
•1•25•0•0•Updated Jun 30, 2016Jun 30, 2016