chore(release): v3.10.0

apify · May 16, 2024 · 5a888d7 · 5a888d7
1 parent d73756b
commit 5a888d7
Show file tree

Hide file tree

Showing 35 changed files with 335 additions and 124 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,6 +3,40 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.10.0](https://github.com/apify/crawlee/compare/v3.9.2...v3.10.0) (2024-05-16)
+
+
+### Bug Fixes
+
+* `EnqueueStrategy.All` erroring with links using unsupported protocols ([#2389](https://github.com/apify/crawlee/issues/2389)) ([8db3908](https://github.com/apify/crawlee/commit/8db39080b7711ba3c27dff7fce1170ddb0ee3d05))
+* **core:** conversion between tough cookies and browser pool cookies ([#2443](https://github.com/apify/crawlee/issues/2443)) ([74f73ab](https://github.com/apify/crawlee/commit/74f73ab77a94ecd285d587b7b3532443deda07b4))
+* **core:** fire local `SystemInfo` events every second ([#2454](https://github.com/apify/crawlee/issues/2454)) ([1fa9a66](https://github.com/apify/crawlee/commit/1fa9a66388846505f84dcdea0393e7eaaebf84c3))
+* **core:** use createSessionFunction when loading Session from persisted state ([#2444](https://github.com/apify/crawlee/issues/2444)) ([3c56b4c](https://github.com/apify/crawlee/commit/3c56b4ca1efe327138aeb32c39dfd9dd67b6aceb))
+* do not drop statistics on migration/resurrection/resume ([#2462](https://github.com/apify/crawlee/issues/2462)) ([8ce7dd4](https://github.com/apify/crawlee/commit/8ce7dd4ae6a3718dac95e784a53bd5661c827edc))
+* double tier decrement in tiered proxy ([#2468](https://github.com/apify/crawlee/issues/2468)) ([3a8204b](https://github.com/apify/crawlee/commit/3a8204ba417936570ec5569dc4e4eceed79939c1))
+* Fixed double extension for screenshots ([#2419](https://github.com/apify/crawlee/issues/2419)) ([e8b39c4](https://github.com/apify/crawlee/commit/e8b39c41764726280c995e52fa7d79a9240d993e)), closes [#1980](https://github.com/apify/crawlee/issues/1980)
+* malformed sitemap url when sitemap index child contains querystring ([#2430](https://github.com/apify/crawlee/issues/2430)) ([e4cd41c](https://github.com/apify/crawlee/commit/e4cd41c49999af270fbe2476a61d92c8e3502463))
+* return true when robots.isAllowed returns undefined ([#2439](https://github.com/apify/crawlee/issues/2439)) ([6f541f8](https://github.com/apify/crawlee/commit/6f541f8c4ea9b1e94eb506383019397676fd79fe)), closes [#2437](https://github.com/apify/crawlee/issues/2437)
+* sitemap `content-type` check breaks on `content-type` parameters ([#2442](https://github.com/apify/crawlee/issues/2442)) ([db7d372](https://github.com/apify/crawlee/commit/db7d37256a49820e3e584165fff42377042ec258))
+
+
+### Features
+
+* add `FileDownload` "crawler" ([#2435](https://github.com/apify/crawlee/issues/2435)) ([d73756b](https://github.com/apify/crawlee/commit/d73756bb225d9ed8f58cf0a3b2e0ce96f6188863))
+* implement ErrorSnapshotter for error context capture ([#2332](https://github.com/apify/crawlee/issues/2332)) ([e861dfd](https://github.com/apify/crawlee/commit/e861dfdb451ae32fb1e0c7749c6b59744654b303)), closes [#2280](https://github.com/apify/crawlee/issues/2280)
+* make `RequestQueue` v2 the default queue, see more on [Apify blog](https://blog.apify.com/new-apify-request-queue/) ([#2390](https://github.com/apify/crawlee/issues/2390)) ([41ae8ab](https://github.com/apify/crawlee/commit/41ae8abec1da811ae0750ac2d298e77c1e3b7b55)), closes [#2388](https://github.com/apify/crawlee/issues/2388)
+
+
+### Performance Improvements
+
+* improve scaling based on memory ([#2459](https://github.com/apify/crawlee/issues/2459)) ([2d5d443](https://github.com/apify/crawlee/commit/2d5d443da5fa701b21aec003d4d84797882bc175))
+* optimize `RequestList` memory footprint ([#2466](https://github.com/apify/crawlee/issues/2466)) ([12210bd](https://github.com/apify/crawlee/commit/12210bd191b50c76ecca23ea18f3deda7b1517c6))
+* optimize adding large amount of requests via `crawler.addRequests()` ([#2456](https://github.com/apify/crawlee/issues/2456)) ([6da86a8](https://github.com/apify/crawlee/commit/6da86a85d848cd1cf860a28e5f077b8b14cdb213))
+
+
+
+
+
 ## [3.9.2](https://github.com/apify/crawlee/compare/v3.9.1...v3.9.2) (2024-04-17)
 
 

diff --git a/lerna.json b/lerna.json
@@ -2,7 +2,7 @@
  "packages": [
  "packages/*"
  ],
- "version": "3.9.2",
+ "version": "3.10.0",
  "command": {
  "version": {
  "conventionalCommits": true,

diff --git a/packages/basic-crawler/CHANGELOG.md b/packages/basic-crawler/CHANGELOG.md
@@ -3,6 +3,24 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.10.0](https://github.com/apify/crawlee/compare/v3.9.2...v3.10.0) (2024-05-16)
+
+
+### Bug Fixes
+
+* `EnqueueStrategy.All` erroring with links using unsupported protocols ([#2389](https://github.com/apify/crawlee/issues/2389)) ([8db3908](https://github.com/apify/crawlee/commit/8db39080b7711ba3c27dff7fce1170ddb0ee3d05))
+* do not drop statistics on migration/resurrection/resume ([#2462](https://github.com/apify/crawlee/issues/2462)) ([8ce7dd4](https://github.com/apify/crawlee/commit/8ce7dd4ae6a3718dac95e784a53bd5661c827edc))
+
+
+### Features
+
+* implement ErrorSnapshotter for error context capture ([#2332](https://github.com/apify/crawlee/issues/2332)) ([e861dfd](https://github.com/apify/crawlee/commit/e861dfdb451ae32fb1e0c7749c6b59744654b303)), closes [#2280](https://github.com/apify/crawlee/issues/2280)
+* make `RequestQueue` v2 the default queue, see more on [Apify blog](https://blog.apify.com/new-apify-request-queue/) ([#2390](https://github.com/apify/crawlee/issues/2390)) ([41ae8ab](https://github.com/apify/crawlee/commit/41ae8abec1da811ae0750ac2d298e77c1e3b7b55)), closes [#2388](https://github.com/apify/crawlee/issues/2388)
+
+
+
+
+
 ## [3.9.2](https://github.com/apify/crawlee/compare/v3.9.1...v3.9.2) (2024-04-17)
 
 

diff --git a/packages/basic-crawler/package.json b/packages/basic-crawler/package.json
@@ -1,6 +1,6 @@
 {
  "name": "@crawlee/basic",
- "version": "3.9.2",
+ "version": "3.10.0",
  "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
  "engines": {
  "node": ">=16.0.0"
@@ -48,9 +48,9 @@
  "@apify/log": "^2.4.0",
  "@apify/timeout": "^0.3.0",
  "@apify/utilities": "^2.7.10",
- "@crawlee/core": "3.9.2",
- "@crawlee/types": "3.9.2",
- "@crawlee/utils": "3.9.2",
+ "@crawlee/core": "^3.10.0",
+ "@crawlee/types": "^3.10.0",
+ "@crawlee/utils": "^3.10.0",
  "csv-stringify": "^6.2.0",
  "fs-extra": "^11.0.0",
  "got-scraping": "^4.0.0",

diff --git a/packages/browser-crawler/CHANGELOG.md b/packages/browser-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.10.0](https://github.com/apify/crawlee/compare/v3.9.2...v3.10.0) (2024-05-16)
+
+**Note:** Version bump only for package @crawlee/browser
+
+
+
+
+
 ## [3.9.2](https://github.com/apify/crawlee/compare/v3.9.1...v3.9.2) (2024-04-17)
 
 **Note:** Version bump only for package @crawlee/browser

diff --git a/packages/browser-crawler/package.json b/packages/browser-crawler/package.json
@@ -1,6 +1,6 @@
 {
  "name": "@crawlee/browser",
- "version": "3.9.2",
+ "version": "3.10.0",
  "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
  "engines": {
  "node": ">=16.0.0"
@@ -54,10 +54,10 @@
  },
  "dependencies": {
  "@apify/timeout": "^0.3.0",
- "@crawlee/basic": "3.9.2",
- "@crawlee/browser-pool": "3.9.2",
- "@crawlee/types": "3.9.2",
- "@crawlee/utils": "3.9.2",
+ "@crawlee/basic": "^3.10.0",
+ "@crawlee/browser-pool": "^3.10.0",
+ "@crawlee/types": "^3.10.0",
+ "@crawlee/utils": "^3.10.0",
  "ow": "^0.28.1",
  "tslib": "^2.4.0",
  "type-fest": "^4.0.0"

diff --git a/packages/browser-pool/CHANGELOG.md b/packages/browser-pool/CHANGELOG.md
@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.10.0](https://github.com/apify/crawlee/compare/v3.9.2...v3.10.0) (2024-05-16)
+
+**Note:** Version bump only for package @crawlee/browser-pool
+
+
+
+
+
 ## [3.9.2](https://github.com/apify/crawlee/compare/v3.9.1...v3.9.2) (2024-04-17)
 
 **Note:** Version bump only for package @crawlee/browser-pool

diff --git a/packages/browser-pool/package.json b/packages/browser-pool/package.json
@@ -1,6 +1,6 @@
 {
  "name": "@crawlee/browser-pool",
- "version": "3.9.2",
+ "version": "3.10.0",
  "description": "Rotate multiple browsers using popular automation libraries such as Playwright or Puppeteer.",
  "engines": {
  "node": ">=16.0.0"
@@ -38,8 +38,8 @@
  "dependencies": {
  "@apify/log": "^2.4.0",
  "@apify/timeout": "^0.3.0",
- "@crawlee/core": "3.9.2",
- "@crawlee/types": "3.9.2",
+ "@crawlee/core": "^3.10.0",
+ "@crawlee/types": "^3.10.0",
  "fingerprint-generator": "^2.0.6",
  "fingerprint-injector": "^2.0.5",
  "lodash.merge": "^4.6.2",

diff --git a/packages/cheerio-crawler/CHANGELOG.md b/packages/cheerio-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.10.0](https://github.com/apify/crawlee/compare/v3.9.2...v3.10.0) (2024-05-16)
+
+**Note:** Version bump only for package @crawlee/cheerio
+
+
+
+
+
 ## [3.9.2](https://github.com/apify/crawlee/compare/v3.9.1...v3.9.2) (2024-04-17)
 
 **Note:** Version bump only for package @crawlee/cheerio

diff --git a/packages/cheerio-crawler/package.json b/packages/cheerio-crawler/package.json
@@ -1,6 +1,6 @@
 {
  "name": "@crawlee/cheerio",
- "version": "3.9.2",
+ "version": "3.10.0",
  "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
  "engines": {
  "node": ">=16.0.0"
@@ -53,9 +53,9 @@
  "access": "public"
  },
  "dependencies": {
- "@crawlee/http": "3.9.2",
- "@crawlee/types": "3.9.2",
- "@crawlee/utils": "3.9.2",
+ "@crawlee/http": "^3.10.0",
+ "@crawlee/types": "^3.10.0",
+ "@crawlee/utils": "^3.10.0",
  "cheerio": "^1.0.0-rc.12",
  "htmlparser2": "^9.0.0",
  "tslib": "^2.4.0"

diff --git a/packages/cli/CHANGELOG.md b/packages/cli/CHANGELOG.md
@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.10.0](https://github.com/apify/crawlee/compare/v3.9.2...v3.10.0) (2024-05-16)
+
+**Note:** Version bump only for package @crawlee/cli
+
+
+
+
+
 ## [3.9.2](https://github.com/apify/crawlee/compare/v3.9.1...v3.9.2) (2024-04-17)
 
 **Note:** Version bump only for package @crawlee/cli

diff --git a/packages/cli/package.json b/packages/cli/package.json
@@ -1,6 +1,6 @@
 {
  "name": "@crawlee/cli",
- "version": "3.9.2",
+ "version": "3.10.0",
  "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
  "engines": {
  "node": ">=16.0.0"
@@ -51,7 +51,7 @@
  "access": "public"
  },
  "dependencies": {
- "@crawlee/templates": "3.9.2",
+ "@crawlee/templates": "^3.10.0",
  "ansi-colors": "^4.1.3",
  "fs-extra": "^11.0.0",
  "inquirer": "^8.2.4",

diff --git a/packages/core/CHANGELOG.md b/packages/core/CHANGELOG.md
@@ -3,6 +3,34 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.10.0](https://github.com/apify/crawlee/compare/v3.9.2...v3.10.0) (2024-05-16)
+
+
+### Bug Fixes
+
+* `EnqueueStrategy.All` erroring with links using unsupported protocols ([#2389](https://github.com/apify/crawlee/issues/2389)) ([8db3908](https://github.com/apify/crawlee/commit/8db39080b7711ba3c27dff7fce1170ddb0ee3d05))
+* **core:** conversion between tough cookies and browser pool cookies ([#2443](https://github.com/apify/crawlee/issues/2443)) ([74f73ab](https://github.com/apify/crawlee/commit/74f73ab77a94ecd285d587b7b3532443deda07b4))
+* **core:** fire local `SystemInfo` events every second ([#2454](https://github.com/apify/crawlee/issues/2454)) ([1fa9a66](https://github.com/apify/crawlee/commit/1fa9a66388846505f84dcdea0393e7eaaebf84c3))
+* **core:** use createSessionFunction when loading Session from persisted state ([#2444](https://github.com/apify/crawlee/issues/2444)) ([3c56b4c](https://github.com/apify/crawlee/commit/3c56b4ca1efe327138aeb32c39dfd9dd67b6aceb))
+* double tier decrement in tiered proxy ([#2468](https://github.com/apify/crawlee/issues/2468)) ([3a8204b](https://github.com/apify/crawlee/commit/3a8204ba417936570ec5569dc4e4eceed79939c1))
+
+
+### Features
+
+* implement ErrorSnapshotter for error context capture ([#2332](https://github.com/apify/crawlee/issues/2332)) ([e861dfd](https://github.com/apify/crawlee/commit/e861dfdb451ae32fb1e0c7749c6b59744654b303)), closes [#2280](https://github.com/apify/crawlee/issues/2280)
+* make `RequestQueue` v2 the default queue, see more on [Apify blog](https://blog.apify.com/new-apify-request-queue/) ([#2390](https://github.com/apify/crawlee/issues/2390)) ([41ae8ab](https://github.com/apify/crawlee/commit/41ae8abec1da811ae0750ac2d298e77c1e3b7b55)), closes [#2388](https://github.com/apify/crawlee/issues/2388)
+
+
+### Performance Improvements
+
+* improve scaling based on memory ([#2459](https://github.com/apify/crawlee/issues/2459)) ([2d5d443](https://github.com/apify/crawlee/commit/2d5d443da5fa701b21aec003d4d84797882bc175))
+* optimize `RequestList` memory footprint ([#2466](https://github.com/apify/crawlee/issues/2466)) ([12210bd](https://github.com/apify/crawlee/commit/12210bd191b50c76ecca23ea18f3deda7b1517c6))
+* optimize adding large amount of requests via `crawler.addRequests()` ([#2456](https://github.com/apify/crawlee/issues/2456)) ([6da86a8](https://github.com/apify/crawlee/commit/6da86a85d848cd1cf860a28e5f077b8b14cdb213))
+
+
+
+
+
 ## [3.9.2](https://github.com/apify/crawlee/compare/v3.9.1...v3.9.2) (2024-04-17)
 
 

diff --git a/packages/core/package.json b/packages/core/package.json
@@ -1,6 +1,6 @@
 {
  "name": "@crawlee/core",
- "version": "3.9.2",
+ "version": "3.10.0",
  "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
  "engines": {
  "node": ">=16.0.0"
@@ -59,9 +59,9 @@
  "@apify/pseudo_url": "^2.0.30",
  "@apify/timeout": "^0.3.0",
  "@apify/utilities": "^2.7.10",
- "@crawlee/memory-storage": "3.9.2",
- "@crawlee/types": "3.9.2",
- "@crawlee/utils": "3.9.2",
+ "@crawlee/memory-storage": "^3.10.0",
+ "@crawlee/types": "^3.10.0",
+ "@crawlee/utils": "^3.10.0",
  "@sapphire/async-queue": "^1.5.1",
  "@types/tough-cookie": "^4.0.2",
  "@vladfrangu/async_event_emitter": "^2.2.2",

diff --git a/packages/crawlee/CHANGELOG.md b/packages/crawlee/CHANGELOG.md
@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.10.0](https://github.com/apify/crawlee/compare/v3.9.2...v3.10.0) (2024-05-16)
+
+**Note:** Version bump only for package crawlee
+
+
+
+
+
 ## [3.9.2](https://github.com/apify/crawlee/compare/v3.9.1...v3.9.2) (2024-04-17)
 
 **Note:** Version bump only for package crawlee

diff --git a/packages/crawlee/package.json b/packages/crawlee/package.json
@@ -1,6 +1,6 @@
 {
  "name": "crawlee",
- "version": "3.9.2",
+ "version": "3.10.0",
  "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
  "engines": {
  "node": ">=16.0.0"
@@ -54,18 +54,18 @@
  "access": "public"
  },
  "dependencies": {
- "@crawlee/basic": "3.9.2",
- "@crawlee/browser": "3.9.2",
- "@crawlee/browser-pool": "3.9.2",
- "@crawlee/cheerio": "3.9.2",
- "@crawlee/cli": "3.9.2",
- "@crawlee/core": "3.9.2",
- "@crawlee/http": "3.9.2",
- "@crawlee/jsdom": "3.9.2",
- "@crawlee/linkedom": "3.9.2",
- "@crawlee/playwright": "3.9.2",
- "@crawlee/puppeteer": "3.9.2",
- "@crawlee/utils": "3.9.2",
+ "@crawlee/basic": "^3.10.0",
+ "@crawlee/browser": "^3.10.0",
+ "@crawlee/browser-pool": "^3.10.0",
+ "@crawlee/cheerio": "^3.10.0",
+ "@crawlee/cli": "^3.10.0",
+ "@crawlee/core": "^3.10.0",
+ "@crawlee/http": "^3.10.0",
+ "@crawlee/jsdom": "^3.10.0",
+ "@crawlee/linkedom": "^3.10.0",
+ "@crawlee/playwright": "^3.10.0",
+ "@crawlee/puppeteer": "^3.10.0",
+ "@crawlee/utils": "^3.10.0",
  "import-local": "^3.1.0",
  "tslib": "^2.4.0"
  },

diff --git a/packages/http-crawler/CHANGELOG.md b/packages/http-crawler/CHANGELOG.md
@@ -3,6 +3,17 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.10.0](https://github.com/apify/crawlee/compare/v3.9.2...v3.10.0) (2024-05-16)
+
+
+### Features
+
+* add `FileDownload` "crawler" ([#2435](https://github.com/apify/crawlee/issues/2435)) ([d73756b](https://github.com/apify/crawlee/commit/d73756bb225d9ed8f58cf0a3b2e0ce96f6188863))
+
+
+
+
+
 ## [3.9.2](https://github.com/apify/crawlee/compare/v3.9.1...v3.9.2) (2024-04-17)
 
 **Note:** Version bump only for package @crawlee/http

diff --git a/packages/http-crawler/package.json b/packages/http-crawler/package.json
@@ -1,6 +1,6 @@
 {
  "name": "@crawlee/http",
- "version": "3.9.2",
+ "version": "3.10.0",
  "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
  "engines": {
  "node": ">=16.0.0"
@@ -55,9 +55,9 @@
  "dependencies": {
  "@apify/timeout": "^0.3.0",
  "@apify/utilities": "^2.7.10",
- "@crawlee/basic": "3.9.2",
- "@crawlee/types": "3.9.2",
- "@crawlee/utils": "3.9.2",
+ "@crawlee/basic": "^3.10.0",
+ "@crawlee/types": "^3.10.0",
+ "@crawlee/utils": "^3.10.0",
  "@types/content-type": "^1.1.5",
  "cheerio": "^1.0.0-rc.12",
  "content-type": "^1.0.4",

diff --git a/packages/jsdom-crawler/CHANGELOG.md b/packages/jsdom-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.10.0](https://github.com/apify/crawlee/compare/v3.9.2...v3.10.0) (2024-05-16)
+
+**Note:** Version bump only for package @crawlee/jsdom
+
+
+
+
+
 ## [3.9.2](https://github.com/apify/crawlee/compare/v3.9.1...v3.9.2) (2024-04-17)
 
 **Note:** Version bump only for package @crawlee/jsdom