release-6450

What's Changed

• Adopt list_streaming in tenant deletion by @arpad-m in #8504
• Add a test for clickhouse as a logical replication consumer by @a-masterov in #8408
• test: deflake test_duplicate_creation by @koivunej in #8536
• pageserver: fix return code from secondary_download_handler by @jcsp in #8508
• pageserver: fail if id is present in pageserver.toml by @problame in #8489
• l0_flush: use mode=direct by default => coverage in automated tests by @problame in #8534
• scrubber: enable cleaning up garbage tenants from known deletion bugs, add object age safety check by @jcsp in #8461
• Add metrics for input data considered and taken for compression by @arpad-m in #8522
• scrubber: add remote_storage based listing APIs and use them in find-large-objects by @arpad-m in #8541
• CI(benchmarking): make neonvm default provisioner by @bayandin in #8538
• [neon/acr] impr: push to ACR while building images by @antonyc in #8545
• feat(storcon): store scrubber metadata scan result by @yliang412 in #8480
• feat(scrubber): post scan_metadata results to storage controller by @yliang412 in #8502
• cleanup(compact_level0_phase1): some commentary and wrapping into block expressions by @problame in #8544
• cicd: change Azure storage details [1/2] by @fcdm in #8553
• compaction_level0_phase1: bypass PS PageCache for data blocks by @problame in #8543
• CI(regress-tests): run less regression tests by @bayandin in #8561
• feat(pageserver): support btm-gc-compaction for child branches by @skyzh in #8519
• refactor(page_service): Timeline gate guard holding + cancellation + shutdown by @problame in #8339
• Fix negative replication delay metric by @tristan957 in #8520
• cicd: change Azure storage details [2/2] by @fcdm in #8562
• controller: simplify reconciler generation increment logic by @jcsp in #8560
• storage_scrubber: migrate FindGarbage to remote_storage by @arpad-m in #8548
• CI(create-test-report): fix missing benchmark results in Allure report by @bayandin in #8540
• test(pageserver): add test_gc_feedback_with_snapshots by @skyzh in #8474
• pageserver: shutdown all walredo managers 8s into shutdown by @problame in #8572
• Reduce linux-raw-sys duplication by @arpad-m in #8577
• pageserver: add layer visibility calculation by @jcsp in #8511
• storage-scrubber: log version on start by @skyzh in #8571
• fix(pageserver): skip existing layers for btm-gc-compaction by @skyzh in #8498
• test_runner: don't create artifacts if Allure is not enabled by @bayandin in #8580
• pageserver: refine how we delete timelines after shard split by @jcsp in #8436
• storage_controller: start adding chaos hooks by @jcsp in #7946
• Wait for completion of the upload queue in flush_frozen_layer by @arpad-m in #8550
• Improve safekeepers eviction rate limiting by @petuhovskiy in #8456
• tests: add test_historic_storage_formats by @jcsp in #8423
• fix(pageserver): deadlock in gc-compaction by @skyzh in #8590
• feat(pageserver): support auto split layers based on size by @skyzh in #8574
• safekeeper: remove unused partial_backup_enabled option by @dotdister in #8547
• feat(pageserver): support split delta layers by @skyzh in #8599
• Update Postgres versions to use smgrexists() instead of access() to check if Oid is used by @knizhnik in #8597
• CI(trigger-e2e-tests): wait for promote-images job from the last commit by @bayandin in #8592
• CI(trigger-e2e-tests): fix deadlock with Build and Test workflow by @bayandin in #8606
• fix: allow awaiting logical size for root timelines by @koivunej in #8604
• CI(benchmarking): set pub/sub projects for LR tests by @bayandin in #8483
• feat(pageserver): support dry-run for gc-compaction, add statistics by @skyzh in #8557
• fix: make Timeline::set_disk_consistent_lsn use fetch_max by @koivunej in #8311
• feat: persistent gc blocking by @koivunej in #8600
• pageserver: remove legacy read path by @VladLazar in #8601
• chore: bump index part version by @koivunej in #8611
• storage_scrubber: migrate scan_safekeeper_metadata to remote_storage by @arpad-m in #8595
• Add package-mode=false to poetry. by @arssher in #8609
• Add a test using Debezium as a client for the logical replication by @a-masterov in #8568
• enable rum test by @a-masterov in #8380
• tests: improve stability of test_storage_controller_many_tenants by @jcsp in #8607
• pageserver: fixes for layer visibility metric by @jcsp in #8603
• pageserver: use layer visibility when composing heatmap by @jcsp in #8616
• pageserver: add bench_ingest by @jcsp in #7409
• scrubber: clean up scan_metadata before prod by @yliang412 in #8565
• storage broker: only print one line for version and build tag in init by @arpad-m in #8624
• Lower level for timeline cancellations during gc by @arpad-m in #8626
• Add missing colon to ArchivalConfigRequest specification by @arpad-m in #8627
• proxy: random changes by @conradludgate in #8602
• fix: EphemeralFiles can outlive their Timeline via enum LayerManager by @koivunej in #8229
• fix(pageserver): dump the key when it's invalid by @skyzh in #8633
• refactor(timeline_detach_ancestor): replace ordered reparented with a hashset by @koivunej in #8629
• cicd: push build-tools image to ACR as well by @fcdm in #8638
• fix: drain completed page_service connections by @koivunej in #8632
• feat(pageserver): add direct io pageserver config by @yliang412 in #8622
• pageserver: make bench_ingest build (but panic) on macOS by @jcsp in #8641
• Use sycnhronous commit for logical replicaiton worker by @knizhnik in #8645
• fix(test): do not fail test for filesystem race by @koivunej in #8643
• fix: stop leaking BackgroundPurges by @koivunej in #8650
• pageserver: remove paranoia double-calculation of retain_lsns by @jcsp in #8617
• tests: don't require kafka client for regular tests by @jcsp in #8662
• CI(build-tools): update Rust, Python, Mold by @bayandin in #8667
• Dockerfiles: fix LegacyKeyValueFormat & JSONArgsRecommended by @bayandin in #8664
• Run a subset of benchmarking job steps on GitHub action runners in Azure - closer to the system under test by @Bodobolero in #8651
• Revert "proxy: update tokio-postgres to allow arbitrary config params (#8076)" by @conradludgate in #8654
• fix(neon): disable create tablespace stmt by @skyzh in #8657
• CI(pin-build-tools-image): fix permissions for Azure login by @bayandin in #8671
• pageserver: don't treat NotInitialized::Stopped as unexpected by @jcsp in #8675
• storcon: skip draining shard if it's secondary is lagging too much by @VladLazar in #8644
• Dockerfiles: remove cachepot by @bayandin in #86...

release-6299

What's Changed

• Storage & Compute release 2024-08-12 by @vipvap in #8688

Full Changelog: release-6250...release-6299

release-proxy-6107

Merge pull request #8647 from neondatabase/rc/proxy/2024-08-08

Proxy release 2024-08-08

release-6250

What's Changed

• Storage release 2024-08-07 by @arpad-m in #8642

Full Changelog: release-6248...release-6250

release-6248

What's Changed

• Storage & Compute release 2024-08-05 by @vipvap in #8598

Full Changelog: release-6210...release-6248

release-6210

What's Changed

• [proxy/redis] impr: use redis_auth_type to switch between auth types by @antonyc in #8428
• fix(docs): race on monotonic rfc id by @skyzh in #8445
• tests: more generous memory allowance in test_compaction_l0_memory by @jcsp in #8446
• pageserver: remove absolute-order disk usage eviction by @jcsp in #8454
• tests: make test_change_pageserver more robust by @jcsp in #8442
• Start using remote_storage in S3 scrubber for PurgeGarbage by @arpad-m in #7932
• fix(scrubber): more robust metadata consistency checks by @yliang412 in #8344
• fix(docs): clearify the admin URL and token used in scrubber by @skyzh in #8441
• refactor(pageserver) remove task_mgr for most global tasks by @problame in #8449
• build: mark target/ and pg_install/ with CACHEDIR.TAG by @problame in #8448
• Mark body of archival_config endpoint as required by @arpad-m in #8458
• Change default version of Neon extensio to 1.4 by @knizhnik in #8405
• vm-image: Expose new LFC working set size metrics by @sharnoff in #8298
• fix(docs): remove incorrect flags for scrubber purge-garbage command by @yliang412 in #8463
• pageserver: use identity file as node id authority and remove init command and config-override flags by @VladLazar in #7766
• tests: make test_scrubber_physical_gc_ancestors more stable by @jcsp in #8453
• build(deps): bump openssl from 0.10.64 to 0.10.66 in /test_runner/pg_clients/rust/tokio-postgres by @dependabot in #8464
• pageserver: add LayerVisibilityHint by @jcsp in #8432
• Use smgrexists() instead of access() to enforce uniqueness of generated relfilenumber by @knizhnik in #7992
• Fix to use a tab instead of spaces by @shinyaaa in #8394
• tests: allow-list a controller heartbeat error by @jcsp in #8471
• temporarily disable cache saving in the registry as it is very slow by @cicdteam in #8475
• feat(pageserver): support retain_lsn in bottommost gc-compaction by @skyzh in #8328
• remote_storage: add list_streaming API call by @arpad-m in #8466
• followup(#8475): also disable 'cache-to' for neon-image-arch and neon-test-extensions by @problame in #8478
• Make async connect work on MacOS: it is necessary top call WaitLatchOrSocket before PQconnectPoll by @knizhnik in #8472
• pageserver: simplify LayerAccessStats by @jcsp in #8431
• Implement retry support for list_streaming by @arpad-m in #8481
• tests: simplify code around unstable test_basebackup_with_high_slru_count by @jcsp in #8477
• pageserver: maintain gc_info incrementally by @jcsp in #8427
• proxy: switch to leaky bucket by @conradludgate in #8470
• CI(build-and-test): move part of the pipeline to a reusable workflow by @bayandin in #8241
• pageserver: faster and simpler inmem layer vec read by @VladLazar in #8469
• pageserver: fix a compilation error by @jcsp in #8487
• devx: nicer diff hunk headers by @koivunej in #8482
• CI: Run ARM checks in the main pipeline by @bayandin in #8185
• followup(#8359): pre-initialize circuitbreaker metrics by @problame in #8491
• fix: remote timeline client shutdown trips circuit breaker by @problame in #8495
• Temporarily remove week-end test for res-aurora from pgbench-compare benchmarking runs by @Bodobolero in #8493
• storcon: make heartbeats restart aware by @VladLazar in #8222
• tests: always scrub on test exit when using S3Storage by @jcsp in #8437
• tests: adjust threshold in test_partial_evict_tenant by @jcsp in #8509
• storcon: shutdown with clean observed state by @VladLazar in #8494
• fix(pageserver): allow incomplete history in btm-gc-compaction by @skyzh in #8500
• Update pgrx to v 0.11.3 by @lubennikovaav in #8515
• Update links in markdown. by @MihaiBojin in #8501
• Handle NotInitialized::ShuttingDown error in shard split by @arpad-m in #8506
• Using own registry to cache layers in docker build workflows by @cicdteam in #8521
• remote_storage: expose last_modified in listings by @jcsp in #8497
• Update Rust to 1.80.0 by @arpad-m in #8518
• tests: checkpoint instead of compact in test_sharding_split_compaction by @jcsp in #8473
• .github: reduce test concurrency by @jcsp in #8444
• pageserver: make vectored get the default read path for the pageserver by @VladLazar in #8384
• CODEOWNERS: collapse safekeepers into storage by @jcsp in #8510
• storcon: introduce step down primitive by @VladLazar in #8512
• Fix sql-exporter-autoscaling for pg < 16 by @sharnoff in #8523
• refactor(layer load API): all errors are permanent by @problame in #8527
• raise wait_lsn timeout from 60s to 300s by @problame in #8529
• fix(pageserver): give L0 compaction priorities over image layer creation by @skyzh in #8443
• Persist archival information by @arpad-m in #8479
• CI(test_runner): Upload all test artifacts if preserve_database_files is enabled by @bayandin in #7990
• safekeeper: include application name in wal service root span by @problame in #8525
• test_runner: fix user_property usage in benchmarks by @bayandin in #8531
• Storage & Compute release 2024-07-29 by @vipvap in #8533

New Contributors

• @MihaiBojin made their first contribution in #8501

Full Changelog: release-6144...release-6210

release-6144

What's Changed

• Compute release 2024-06-26 by @kelvich in #8530

Full Changelog: release-6142...release-6144

release-6142

What's Changed

• Compute release 2024-07-25 by @lubennikovaav in #8514

Full Changelog: release-6138...release-6142

release-6138

What's Changed

• Storage release 2024-07-23 by @problame in #8468

Full Changelog: release-6136...release-6138

release-6136

What's Changed

• tests: remove a leftover 'running' flag by @hlinnaka in #8216
• Restore running xacts from CLOG on replica startup by @knizhnik in #7288
• pageserver: add metric pageserver_secondary_resident_physical_size by @jcsp in #8204
• CI(build-tools): Remove libpq from build image by @bayandin in #8206
• fix(pageserver): include aux file in basebackup only once by @skyzh in #8207
• CI(gather-rust-build-stats): fix build with libpq by @bayandin in #8219
• tests: Make neon_xlogflush() flush all WAL, if you omit the LSN arg by @hlinnaka in #8215
• docs: Graceful storage controller cluster restarts RFC by @VladLazar in #7704
• Check status of connection after PQconnectStartParams by @knizhnik in #8210
• remote_storage config: move handling of empty inline table {} to callers by @problame in #8193
• pageserver: simpler, stricter config error handling by @jcsp in #8177
• fix: noisy logging when download gets cancelled during shutdown by @problame in #8224
• pageserver: rate limit log for loads of layers visited by @VladLazar in #8228
• Add support for reading and writing compressed blobs by @arpad-m in #8106
• L0 flush: opt-in mechanism to bypass PageCache reads and writes by @problame in #8190
• CI(benchmarking): move psql queries to actions/run-python-test-set by @bayandin in #8230
• tense of errors by @jcsp in #8234
• fix(pageserver): comments about metadata key range by @skyzh in #8236
• Add test for proper handling of connection failure to avoid 'cannot wait on socket event without a socket' error by @knizhnik in #8231
• docker: add storage_scrubber into the docker image by @skyzh in #8239
• Simplify test_wal_page_boundary_start test by @hlinnaka in #8214
• CI: update docker/* actions to latest versions by @bayandin in #7694
• pageserver: don't try to flush if shutdown during attach by @jcsp in #8235
• Only support compressed reads if the compression setting is present by @arpad-m in #8238
• add pagebench test cases for periodic pagebench on dedicated hardware by @Bodobolero in #8233
• pageserver: reduce ops tracked at per-timeline detail by @jcsp in #8245
• Fix outdated comment by @japinli in #8149
• page_service: stop exposing get_last_record_rlsn by @problame in #8244
• fix(pageserver): ensure test creates valid layer map by @skyzh in #8191
• pageserver: add supplementary branch usage stats by @jcsp in #8131
• tests: perform graceful rolling restarts in storcon scale test by @VladLazar in #8173
• proxy: cache certain non-retriable console errors for a short time by @conradludgate in #8201
• CI(build-and-test): add conclusion job by @bayandin in #8246
• pageserver: increase rate limit duration for layer visit log by @VladLazar in #8263
• Use bool param for round_trip_test_compressed by @arpad-m in #8252
• CI(pg-clients): unify workflow with build-and-test by @bayandin in #8160
• pageserver: downgrade stale generation messages to INFO by @jcsp in #8256
• Add find-large-objects subcommand to scrubber by @arpad-m in #8257
• feat(pageserver): integrate lsn lease into synthetic size by @yliang412 in #8220
• Flatten compression algorithm setting by @arpad-m in #8265
• implement rolling hyper-log-log algorithm by @knizhnik in #8068
• add checkout depth1 to workflow to access local github actions like generate allure report by @Bodobolero in #8259
• tests: extend allow list in deletion test by @jcsp in #8268
• correct error handling for periodic pagebench runner status by @Bodobolero in #8274
• tests: make location_conf_churn more robust by @jcsp in #8271
• safekeeper: add separate tombstones map for deleted timelines by @jcsp in #8253
• pageserver: add time based image layer creation check by @VladLazar in #8247
• SELECT 💣(); by @bayandin in #8270
• Increase timeout for wating subscriber caught-up by @knizhnik in #8118
• pageserver_live_connections: track as counter pair by @problame in #8227
• Improve parsing of ImageCompressionAlgorithm by @arpad-m in #8281
• Add concurrency to the find-large-objects scrubber subcommand by @arpad-m in #8291
• build(deps): bump certifi from 2023.7.22 to 2024.7.4 by @dependabot in #8301
• fix(pageserver): ensure sparse keyspace is ordered by @skyzh in #8285
• pageserver: switch to jemalloc by @jcsp in #8307
• Long-running replication tests by @save-buffer in #7896
• pageserver: respect has_relmap_file in collect_keyspace by @jcsp in #8276
• refactor: move part of sharding API from pageserver_api to utils by @yliang412 in #8254
• compute_ctl: Use 'fast' shutdown for Postgres termination by @ololobus in #8289
• tests: use smaller layers in test_pg_regress, do compaction after tests by @jcsp in #8232
• fix(pageserver): write to both v1+v2 for aux tenant import by @skyzh in #8316
• tests: stabilize test_timeline_size_quota_on_startup by @jcsp in #8255
• tests: increase approx size equal threshold to avoid test_lsn_lease_size flakiness by @yliang412 in #8282
• CI(promote-compatibility-data): take into account commit sha by @bayandin in #8283
• proxy/http: avoid spurious vector reallocations by @lucab in #8319
• chore(storage-scrubber): allow disable file logging by @skyzh in #8297
• chore: fix nightly build by @conradludgate in #8142
• fix(storage-scrubber): use default AWS authentication by @skyzh in #8299
• Refactor how migrations are ran by @tristan957 in #8326
• refactor: postgres_backend: replace abstract shutdown_watcher with CancellationToken by @problame in #8295
• fix(l0_flush): drops permit before fsync, potential cause for OOMs by @problame in #8327
• pageserver: move page_service's import basebackup / import wal to mgmt API by @problame in #8292
• proxy: pg17 fixes by @conradludgate in #8321
• tests: stabilize test_sharding_split_compaction by @jcsp in #8318
• proxy: remove some trace logs by @conradludgate in #8334
• build(deps-dev): bump zipp from 3.8.1 to 3.19.1 by @dependabot in #8329
• Remove ImageCompressionAlgorithm::DisabledNoDecompress by @arpad-m in #8300
• feat(pageserver): add k-merge layer iterator with lazy loading by @skyzh in #8053
• safekeeper: add eviction_min_resident to stop evictions thrashing by @jcsp in #8335
• Enable core dumps for postgres by @kelvich in #8272
• rfcs: add RFC for timeline archival by @jcsp in #8221
• Run Performance bench on more platforms by @Bodobolero in #8312
• pageserver: remove trace_read_requests by @problame in #8338
• storage_controller: fix ReconcilerWaiter::get_status by @VladLazar in #8341
• safekeeper: eviction metrics by @jcsp in #8348
• storage controller: add nod...