release-7044

What's Changed

• Add --replica parameter to basebackup by @knizhnik in #9553
• feat(pageserver): support partial gc-compaction for lowest retain lsn by @skyzh in #9134
• fix(pagectl): layer parsing and image layer dump by @skyzh in #9571
• Add tests for sql-exporter metrics by @tristan957 in #9184
• pageserver: generation-aware storage for TenantManifest by @jcsp in #9555
• Remove non-gzipped basebackup code path by @tristan957 in #9453
• chore(compute): Bump pg_mooncake to the latest version by @ololobus in #9576
• postgres_ffi: add WAL generator for tests/benchmarks by @erikgrinaker in #9503
• Collect compute backpressure throttling time by @tristan957 in #9176
• Disallow archived timelines to be detached or reparented by @arpad-m in #9578
• Don't keep around the timeline's remote_client by @arpad-m in #9583
• refactor(pageserver): use JSON field encoding for consumption metrics cache by @skyzh in #9470
• Disallow offloaded children during timeline deletion by @arpad-m in #9582
• pageserver: lift decoding and interpreting of wal into wal_decoder by @VladLazar in #9524
• pageserver: don't return option from DeletionQueue::new by @erikgrinaker in #9588
• Add support of extensions for v17 (part 4) by @lubennikovaav in #9568
• increase lifetime of AWS session token to 12 hours by @Bodobolero in #9590
• pageserver: fix spurious error logs in timeline lifecycle by @jcsp in #9589
• auth_broker: fix local_proxy conn count by @conradludgate in #9593
• update pg_session_jwt to use pgrx 0.12 for pg17 by @conradludgate in #9595
• storage_controller: allow deployment infra to use infra token by @VladLazar in #9596
• fix pre-commit hook with python stubs by @conradludgate in #9602
• test_runner: fix tenant_get_shards with one pageserver by @erikgrinaker in #9603
• revert higher token expiration by @Bodobolero in #9605
• safekeeper: log slow WalAcceptor sends by @erikgrinaker in #9564
• storcon: skip non-active projects in chaos injection by @jcsp in #9606
• Do not assign page LSN to new (uninitialized) page in ClearVisibilityMapFlags redo handler by @knizhnik in #9287
• test_runner/performance: add sharded ingest benchmark by @erikgrinaker in #9591
• pageserver: make local timeline deletion infallible by @jcsp in #9594
• Fix a time zone issue in PG17 test case by @MMeent in #9618
• remove aws-lc-rs dep and fix storage_broker tls by @conradludgate in #9613
• [proxy]: parse proxy protocol TLVs with aws/azure support by @conradludgate in #9610
• Storage & Compute release 2024-11-04 by @vipvap in #9617

Full Changelog: release-7011...release-7044

release-7011

What's Changed

• Compute hotfix release 2024-10-30 by @ololobus in #9577

Full Changelog: release-7009...release-7011

release-7009

What's Changed

• build(deps): bump werkzeug from 3.0.3 to 3.0.6 by @dependabot in #9527
• Don't consider archived timelines for synthetic size calculation by @arpad-m in #9497
• safekeeper: don't un-evict timelines during snapshot API handler by @jcsp in #9428
• storcon: add an API to cancel ongoing reconciler by @jcsp in #9520
• tests: add test_storage_controller_onboard_detached by @jcsp in #9431
• Route pgbouncer logs to virtio-serial by @petuhovskiy in #9488
• ci(proxy): Update GH action flag on proxy deployment by @rahulinux in #9535
• [proxy] demote warnings and remove dead-argument by @conradludgate in #9512
• Add pg_mooncake to compute-node.Dockerfile by @tristan957 in #9498
• We don't have legacy proxies anymore by @SergeyMelnikov in #9544
• safekeeper: refactor WalAcceptor to be event-driven by @erikgrinaker in #9462
• refactor(pageserver): remove aux v1 configs by @skyzh in #9494
• fix(pageserver): bump tokio-epoll-uring by @skyzh in #9546
• Build pgrag extensions x3 by @jawj in #8486
• Create the notion of unstable extensions by @knizhnik in #9540
• Get role name when not a C string by @tristan957 in #9526
• Assert the tenant to be active in unoffload_timeline by @arpad-m in #9539
• pageserver: move things around to prepare for decoding logic by @VladLazar in #9504
• Offloaded timeline deletion by @arpad-m in #9519
• temporarily increase timeout for clickbench benchmark until regression is resolved by @Bodobolero in #9554
• [proxy]: fix JWT handling for AWS cognito. by @conradludgate in #9536
• [auth_broker]: regress test by @conradludgate in #9541
• pageserver: refactor generation-aware loading code into generic by @jcsp in #9545
• pageserver: make concurrent offloaded timeline operations safe wrt manifest uploads by @jcsp in #9557
• fix allow_unstable_extensions GUC - make it USERSET, by @lubennikovaav in #9563
• proxy: more granular http status codes for sql-over-http errors by @mrl5 in #9549
• Use pg_mooncake from our fork. by @lubennikovaav in #9565
• Compute release 2024-10-29 by @lubennikovaav in #9573

Full Changelog: release-6981...release-7009

release-6981

What's Changed

• proxy: Make tests fail fast when test proxy exited early by @cloneable in #9432
• Rename dockerfiles from Dockerfile.<something> to <something>.Dockerfile by @bayandin in #9446
• CI: use build-tools on Debian 12 whenever we use Neon artifact by @bayandin in #9463
• Use '-f' for make clean in Makefile compute by @awarus in #9464
• proxy: Use common error interface for error handling with cplane by @awarus in #9454
• fix(pageserver): make image split layer writer finish atomic by @skyzh in #8841
• pageserver: use direct IO for delta and image layer reads by @yliang412 in #9326
• offloaded timeline list API by @arpad-m in #9461
• chore(compute): bumps pg_session_jwt to latest version by @davidgomes in #9474
• safekeeper: refactor timeline initialization by @arssher in #9362
• Use make's builtin RM variable by @tristan957 in #9468
• storage controller: add metrics for tenant shard, node count by @jcsp in #9475
• Fix local errors for the tests with the versions mix by @a-masterov in #9477
• Fix copy-paste error in checkpoints_timed metric by @tristan957 in #9483
• Timeline offloading persistence by @arpad-m in #9444
• fix(pageserver): make delta split layer writer finish atomic by @skyzh in #9048
• Rename IndexPart::{from_s3_bytes,to_s3_bytes} by @arpad-m in #9481
• proxy: clear lib.rs of code items by @cloneable in #9479
• Protect call to pg_current_wal_lsn() in retained_wal query by @tristan957 in #9425
• tests: avoid using background_process when invoking pg_ctl by @jcsp in #9469
• pageserver: add metric for number of zeroed pages on rel extend by @VladLazar in #9492
• Fix buggy sizeof by @tristan957 in #9495
• refactor(pageserver(load_remote_timeline)) remove dead code handling absence of IndexPart by @problame in #9408
• storcon: respect Reconciler::cancel during await_lsn by @jcsp in #9486
• CI: remove git config --add safe.directory hack by @bayandin in #9391
• refactor(pageserver): refactor split writers using batch layer writer by @skyzh in #9493
• pageserver: refactor ingest inplace to decouple decoding and handling by @VladLazar in #9472
• refactor(timeline creation): represent bootstrap vs branch using enum by @problame in #9366
• pageserver/walingest: log on gappy rel extend by @VladLazar in #9502
• Support offloaded timelines during shard split by @arpad-m in #9489
• fix(test): remove too strict layers_removed==0 check in test_readonly_node_gc by @yliang412 in #9506
• proxy: don't follow redirects for user provided JWKS urls + set custom user agent by @mrl5 in #9514
• pageserver: add testing API to scan layers for disposable keys by @problame in #9393
• remote_storage(local_fs): return correct file sizes by @problame in #9511
• Make test_pageserver_lsn_wait_error_safekeeper_stop less aggressive. by @arssher in #9517
• Move logical replication slot monitor by @tristan957 in #9496
• refactor(timeline creation): idempotency checking by @problame in #9501
• pageserver: fix N^2 I/O when processing relation drops in transaction abort by @jcsp in #9507
• proxy: propagate session IDs by @conradludgate in #9509
• safekeeper: retry eviction. by @arssher in #9485
• safekeeper: fix version in TimelinePersistentState::empty() by @erikgrinaker in #9521
• safekeeper: don't pass conf into storage constructors by @erikgrinaker in #9523
• Fix unoffload_timeline races with creation by @arpad-m in #9525
• pageserver: add tokio-epoll-uring slots waiters queue depth metrics by @yliang412 in #9482
• Storage & Compute release 2024-10-25 by @vipvap in #9528

Full Changelog: release-6935...release-6981

release-6935

What's Changed

• Compute release 2024-10-21 by @tristan957 in #9476

Full Changelog: release-6933...release-6935

release-6933

What's Changed

• feat(pageserver): do space check before gc-compaction by @skyzh in #9250
• storage_controller: delete stale shards when deleting tenant by @erikgrinaker in #9333
• safekeeper: flush WAL on compute disconnect by @erikgrinaker in #9436
• Add logging of installed_extensions by @lubennikovaav in #9438
• refactor(pageserver): remove aux v1 code path by @skyzh in #9424
• pageserver: detect & warn on loading an old index which is probably the result of a bad generation by @jcsp in #9383
• Update Rust to 1.82.0 and mold to 2.34.0 by @arpad-m in #9445
• update rustls by @conradludgate in #9396
• 2024 oct new clippy lints by @conradludgate in #9448
• compute_ctl: Add endpoint that allows setting role grants by @devjv in #9395
• storcon: fix case where we might fail to send compute notifications after two opposite migrations by @jcsp in #9435
• Increase shared_buffers in test_subscriber_synchronous_commit. by @arssher in #9427
• Update ruff to much newer version by @cloneable in #9433
• compute_ctl: Add endpoint that allows extensions to be installed by @devjv in #9344
• walproposer: immediately exit if sync-safekeepers collected 0/0. by @arssher in #9442
• pageserver: disconnect when asking client to reconnect by @VladLazar in #9390
• [local_proxy]: install pg_session_jwt extension on demand by @conradludgate in #9370
• storcon: handle ongoing deletions gracefully by @VladLazar in #9449
• Take the collector name as argument when generating sql_exporter configs by @tristan957 in #9447
• Accept basebackup --gzip requests by @arpad-m in #9456
• bump pg-session-jwt version by @conradludgate in #9455
• Compute release 2024-10-20 by @ololobus in #9459

Full Changelog: release-6911...release-6933

release-6911

What's Changed

• proxy: split out the console-redirect backend flow by @conradludgate in #9270
• Ignore pg_dynshmem fiel when comparing directories by @knizhnik in #9374
• CI(build-build-tools): fix unexpected cancellations by @bayandin in #9357
• Add test for fixed storage broker issue by @arpad-m in #9311
• tests: make size comparisons more lenient by @VladLazar in #9388
• Also consider offloaded timelines for obtaining retain_lsn by @arpad-m in #9308
• Add more timing histogram and gauge metrics to the Neon extension by @MMeent in #9116
• Switch CI, Storage and Proxy to Debian 12 (Bookworm) by @bayandin in #9170
• pageserver: stabilize & refine controller scale test by @jcsp in #8971
• Shut down timelines during offload and add offload tests by @arpad-m in #9289
• [local_proxy] update api for pg_session_jwt by @conradludgate in #9359
• proxy: Migrate jwt module away from anyhow by @cloneable in #9361
• Remove redundant code by @knizhnik in #9373
• Generate sql_exporter config files with Jsonnet by @tristan957 in #9368
• fix(pageserver): use a buffer for basebackup; add aux basebackup metrics log by @skyzh in #9401
• Add newly added metrics from #9116 to exports by @MMeent in #9402
• Fix jsonnet_files wildcard by @tristan957 in #9403
• Add jsonnetfmt targets by @tristan957 in #9404
• don't trigger workflows in two repos by @fcdm in #9340
• pageserver: improve handling of archival_config calls during Timeline shutdown by @jcsp in #9415
• proxy: format imports with nightly rustfmt by @cloneable in #9414
• storcon: use the same trace fields for reconciler and results by @VladLazar in #9410
• tests: stabilize test_timelines_parallel_endpoints by @jcsp in #9413
• Remove allow(clippy::unknown_lints) by @arpad-m in #9416
• Add support of extensions for v17 (part 2) by @lubennikovaav in #9389
• Activate timelines during unoffload by @arpad-m in #9399
• refactor(pageserver): upgrade remote_storage to use hyper1 by @skyzh in #9405
• proxy: merge AuthError and AuthErrorImpl by @cloneable in #9418
• Fix beta clippy warnings by @arpad-m in #9419
• Fix typo in sql_exporter generator by @tristan957 in #9420
• Fix some sql_exporter metrics for PG 17 by @tristan957 in #9406
• readme: fix build instructions for debian 12 by @problame in #9371
• Check access_count in lfc_evict by @knizhnik in #9407
• storage_scrubber: fixes to garbage commands by @jcsp in #9409
• proxy: move the connection pools to separate file by @awarus in #9398
• Add config variable for timeline offloading by @arpad-m in #9421
• Add support of extensions for v17 (part 3) by @lubennikovaav in #9430
• test_runner: Update the README.md to build neon with 'testing' by @awarus in #9437
• Releases/2024 10 17 compute kq only by @lubennikovaav in #9451

Full Changelog: release-6872...release-6911

release-6872

What's Changed

• Fix update of statistic for LFC/prefetch by @knizhnik in #9272
• proxy: rename console -> control_plane, rename web -> console_redirect by @conradludgate in #9266
• proxy: Move module base files into module directory by @cloneable in #9297
• Make sure BufferTag padding bytes are cleared in hash keys by @hlinnaka in #9292
• feat(compute_tools): Create JWKS Postgres roles without attributes by @devjv in #9031
• Replace BUFFERTAGS_EQUAL compatibility macro with new-style function by @hlinnaka in #9294
• Improve logging on changes in a compute's status by @tristan957 in #9265
• storage_broker: update hyper and tonic again by @arpad-m in #9299
• vm-monitor: Ignore LFC in postgres cgroup memory threshold by @sharnoff in #8668
• pgxn: Fix compiler warnings, and enable them by @hlinnaka in #9290
• Remove non-existent entries from .dockerignore by @hlinnaka in #9209
• Remove get_self_dir() by @tristan957 in #9306
• Fix compiler warnings on macOS by @bayandin in #9319
• storcon: apply all node status changes before handling transitions by @VladLazar in #9281
• Enable all pyupgrade checks in ruff by @tristan957 in #9323
• Add timeline offload mechanism by @arpad-m in #8907
• tests: Remove redundant log lines when starting an endpoint by @hlinnaka in #9316
• remote_storage: add DownloadOpts::byte_(start|end) by @erikgrinaker in #9293
• tests: Reduce noise from logging renamed files by @hlinnaka in #9315
• proxy: Unalias hyper1 and replace one use of hyper0 in test by @cloneable in #9324
• storcon: make observed state updates more granular by @VladLazar in #9276
• test_runner: fix path to tpc-h queries by @bayandin in #9327
• safekeeper: add evicted_timelines gauge. by @arssher in #9318
• Add /installed_extensions endpoint to collect statistics about extension usage. by @lubennikovaav in #8917
• pageserver: add direct io config to virtual file by @yliang412 in #9214
• tests: Use ThreadedMotoServer python class to launch mock S3 server by @hlinnaka in #9313
• tests: Remove redundant log lines when stopping storage nodes by @hlinnaka in #9317
• added workflow Report Workflow Stats by @fedordikarev in #9330
• Add partial support of extensions for v17: by @lubennikovaav in #9322
• local_proxy: integrate with pg_session_jwt extension by @conradludgate in #9086
• Move PgBenchInitResult.EXTRACTORS to a private module constant by @tristan957 in #9334
• Improve some typing in test_runner by @tristan957 in #9303
• add local-proxy suffix to wake-compute requests, respect the returned port by @conradludgate in #9298
• storage controller: use 'infra' JWT scope for node registration by @jcsp in #9343
• CODEOWNERS: assign storage_scrubber to storage by @erikgrinaker in #9346
• storage_scrubber: fix faulty assertion when no timelines by @erikgrinaker in #9345
• Move path-related fixtures into their own module by @tristan957 in #9304
• tests: allow a log warning in test_cli_start_stop_multi by @jcsp in #9320
• Enable cargo caching in check-codestyle-rust by @tristan957 in #9307
• Add compute_config_dir fixture by @tristan957 in #9352
• Use valid type hints for Python 3.9 by @tristan957 in #9349
• Introduce "quota" ErrorKind by @awarus in #9300
• tests: stabilize test_storage_controller_heartbeats by @jcsp in #9347
• proxy: Demote some errors to warnings in logs by @cloneable in #9354
• ci: gh-workflow-stats-action v0.1.4: remove debug output and proper pagination by @fedordikarev in #9356
• Test versions mismatch by @a-masterov in #9167
• CI(run-python-test-set): allow to skip missing compatibility snapshot by @bayandin in #9365
• proxy: refactor auth backends by @conradludgate in #9271
• Storage & Compute release 2024-10-14 by @vipvap in #9372

New Contributors

• @devjv made their first contribution in #9031
• @fedordikarev made their first contribution in #9330

Full Changelog: release-6812...release-6872

release-6812

What's Changed

• Expose more granular wait event data to the user by @MMeent in #9163
• proxy: Add timeout to webauth confirmation wait by @cloneable in #9227
• pageserver: tweak logging of "became visible" for layers by @jcsp in #9224
• storcon: ignore deleted timelines on new location catch-up by @VladLazar in #9244
• storcon: add metric for long running reconciles by @VladLazar in #9207
• fix(test): only test num of L1 layers in compaction smoke test by @skyzh in #9186
• tests: sync_after_each_test -> sync_between_tests by @jcsp in #9239
• Fix metric name of the 'getpage_wait_seconds_bucket' metric by @hlinnaka in #9242
• Change parentheses to clarify conditions in walproposer by @dotdister in #9180
• test_runner: bump requests and psycopg2-binary by @bayandin in #9248
• Add compute_tools/ to the list of paths that trigger an E2E run on a PR by @hlinnaka in #9251
• Use hyper 1.0 and tonic 0.12 in storage broker by @arpad-m in #9234
• Suppress wal lag timeout warnings right after tenant attachment by @arpad-m in #9232
• Remove unnecessary dependencies from postgis-build image by @hlinnaka in #9211
• chore: smaller layer changes by @koivunej in #9247
• vm-monitor: Upgrade axum from 0.6 to 0.7 by @hlinnaka in #9257
• Rename hyper 1.0 to hyper and hyper 0.14 to hyper0 by @arpad-m in #9254
• pageserver: tweak oversized key read path warning by @VladLazar in #9221
• safekeeper: fix panic in debug_dump. by @arssher in #9097
• Revert hyper and tonic updates by @arpad-m in #9268
• chore: remove unnecessary comments in compute/Dockerfile.compute-node by @davidgomes in #9253
• chore: makes some onboarding document improvements by @davidgomes in #9216
• tests: Refactor NeonCli test class by @hlinnaka in #9195
• Cargo.toml: enable sso for aws-config by @erikgrinaker in #9261
• pageserver: use conditional GET for secondary tenant heatmaps by @erikgrinaker in #9236
• storcon: do timeline creation on all attached location by @VladLazar in #9237
• remote_storage: add head_object integration test by @erikgrinaker in #9274
• neon walsender_hooks: take basebackup LSN directly. by @arssher in #9263
• proxy: add local-proxy to compute image by @conradludgate in #8823
• proxy: exclude triple logging of connect compute errors by @awarus in #9277
• safekeeper: remove local WAL files ignoring peer_horizon_lsn. by @arssher in #8900
• Storage & Compute release 2024-10-07 by @vipvap in #9291

New Contributors

• @awarus made their first contribution in #9277

Full Changelog: release-6777...release-6812

release-6777

What's Changed

• fix: ignore tonic to resolve advisories by @skyzh in #9230
• Update aws SDK crates by @arpad-m in #9233
• Fix compute metrics collector config by @hlinnaka in #9235
• Compute release 2024-10-02 (2) by @lubennikovaav in #9249

Full Changelog: release-6773...release-6777