release-4394

What's Changed

• pageserver: tweak logs for index_part loading by @jcsp in #6005
• test_pg_clients: fix test that reads from stdout by @bayandin in #6021
• Perform L0 compaction before creating new image layers by @ivaxer in #5950
• Safekeepers: persist state before timeline deactivation. by @arssher in #5994
• Enable dynamic rate limiter by @khanova in #6029
• walredo: stderr cleanup & make explicitly cancel safe by @problame in #6031
• concurrency-limit initial logical size calculation [v2] by @problame in #6000
• [compute_ctl] Introduce feature flags in the compute spec by @ololobus in #6016
• Remove trusted from wal2json by @save-buffer in #6035
• Rename metric by @khanova in #6030
• proxy: fix panic in startup packet by @conradludgate in #6032
• pageserver: add Key->Shard mapping logic & use it in page service by @jcsp in #5980
• [compute_ctl] Notify waiters when Postgres failed to start by @ololobus in #6034
• pageserver: remove redundant unsafe_create_dir_all by @jcsp in #6040
• pageserver: add a /reset API for tenants by @jcsp in #6014
• pageserver: logging tweaks by @jcsp in #6039
• Track size of FSM fork while applying records at replica by @knizhnik in #5901
• virtual_file: fix the metric for close() calls done by VirtualFile::drop by @problame in #6051
• Revert timescaledb for pg14 and pg15 by @vadim2404 in #6056
• fix: avoid allocations with logging a slug by @koivunej in #6047
• Make simple_rcu::RcuWaitList::wait() async by @hlinnaka in #6046
• temp: increase the wait tenant activation timeout by @koivunej in #6058
• fix(layer): metric splitting, span rename by @koivunej in #5902
• build: update tokio to 1.34.0, tokio-utils 0.7.10 by @koivunej in #6061
• proxy: fix channel binding error messages by @conradludgate in #6054
• tests: update for tenant generations by @jcsp in #5449
• tests: use tenant_create() helper in test_bulk_insert by @jcsp in #6064
• fix(remote_storage): buffering, by using streams for upload and download by @koivunej in #5446
• virtual_file metrics: distinguish first and subsequent open() syscalls by @problame in #6066
• Add support for Python 3.12 by @tristan957 in #6067
• fix: use larger buffers for remote storage by @koivunej in #6069
• Buffer initdb.tar.zst to a temporary file before upload by @arpad-m in #5944
• test_tenant_reattach: fix reattach mode names by @bayandin in #6070
• pgindent and target for it of neon and core code by @arssher in #5618
• pageserver: filtered WAL ingest for sharding by @jcsp in #6024
• proxy: make auth more type safe by @conradludgate in #5689
• tests: update tests that used local_fs&mock_s3 to use one or the other by @jcsp in #6015
• page cache: improve eviction-related metrics by @problame in #6077
• proxy simplify cancellation by @conradludgate in #5916
• virtual_file metrics: expose max size of the fd cache by @problame in #6078
• tests: allow-lists for occasional failures by @jcsp in #6074
• proxy: pass neon options in deep object format by @prepor in #6068
• Fix RLS/REPLICATION granting by @save-buffer in #6083
• Release 2023-12-11 by @vipvap in #6090

Full Changelog: release-4344...release-4394

release-4344

What's Changed

• Revert timescaledb for pg14 and pg15 (#6056) by @vadim2404 in #6057

Full Changelog: release-4342...release-4344

release-4342

What's Changed

• pageserver: remove defunct local timeline delete markers by @jcsp in #5699
• Refactor remote extensions feature to request extensions from proxy by @lubennikovaav in #5836
• remove confusing no-op .take() of init_tenant_load_remote by @problame in #5923
• deflake: test_live_reconfig_get_evictions_low_residence_... by @koivunej in #5926
• Use test specific directory in test_remote_extensions by @lubennikovaav in #5938
• bootstrap_timeline: rename initdb_path to pgdata_path by @arpad-m in #5931
• channel binding by @conradludgate in #5683
• Make neon extension relocatable to allow SET SCHEMA by @lubennikovaav in #5942
• Update neon extension relocatable for existing installations by @lubennikovaav in #5943
• Compile wal2json by @save-buffer in #5893
• Fix check for if shmem is valid to take into account detached shmem by @save-buffer in #5937
• test suite: add method for generation-aware detachment of a tenant by @problame in #5939
• pageserver: add shard indices to layer metadata by @jcsp in #5928
• pageserver: add sharding metadata to LocationConf by @jcsp in #5932
• Move run_initdb to be async and guarded by max of 8 running tasks. Fixes #5895. Use tenant.cancel for cancellation by @shanyp in #5921
• fix: remove layer_removal_cs by @koivunej in #5108
• perf readme fix by @bojanserafimov in #5956
• build(deps): bump aiohttp from 3.8.6 to 3.9.0 by @dependabot in #5946
• Add walingest test by @bojanserafimov in #5892
• neonvm: add pgbouncer patch to support DEALLOCATD/DISCARD ALL by @bayandin in #5958
• build(deps): bump openssl from 0.10.55 to 0.10.60 by @dependabot in #5965
• build(deps): bump cryptography from 41.0.4 to 41.0.6 by @dependabot in #5970
• build(deps): bump openssl from 0.10.57 to 0.10.60 in /test_runner/pg_clients/rust/tokio-postgres by @dependabot in #5966
• pageserver: fix race between deletion completion and incoming requests by @jcsp in #5941
• Notify safekeeper readiness with systemd. by @arssher in #5972
• Optimize storing of null page in WAL by @knizhnik in #5910
• pageserver: remove redundant serialization helpers on DeletionList by @jcsp in #5960
• pageserver: refactor TenantId to TenantShardId in Tenant & Timeline by @jcsp in #5957
• neon_local: fix endpoint api to prevent two primary endpoints by @rmodpur in #5520
• walredo: latency histogram for spawn duration by @problame in #5925
• proxy: add more db error info by @conradludgate in #5951
• proxy: add flag to enable http pool for all users by @conradludgate in #5959
• refactor: use serde for TenantConf deserialization Fixes: #5300 by @rmodpur in #5310
• pagectl: add subcommand to rewrite layer file summary by @problame in #5933
• update aws sdk to 1.0.x by @ivaxer in #5976
• fix: flush loop should also keep the gate open by @koivunej in #5987
• IP allowlist on the proxy side by @khanova in #5906
• Pageserver disaster recovery RFC by @arpad-m in #5248
• proxy: fix wake_compute error prop by @conradludgate in #5989
• tests: add a log allow list entry in test_timeline_deletion_with_files_stuck_in_upload_queue by @jcsp in #5981
• walingest: refactor if-cascade on decoded.xl_rmid into match statement by @problame in #5974
• pageserver: add flush_ms and document /location_config API by @jcsp in #5860
• test: python needs thread safety as well by @koivunej in #5992
• Proxy fix metrics record by @khanova in #5996
• test: include cmdline in captured output by @koivunej in #5977
• proxy: chore: replace strings with SmolStr by @conradludgate in #5786
• Correct RFC number for Pageserver WAL DR RFC by @arpad-m in #5997
• Add existing_initdb_timeline_id param to timeline creation by @arpad-m in #5912
• Update timescaledb to 2.13.0 by @vadim2404 in #5975
• Clean up local state if index_part.json request gives 404 by @arpad-m in #6009
• initial logical size calculation: add a bunch of metrics by @problame in #5995
• init logical size: infallible accessor + type-level cleanups by @problame in #5999
• test_runner: use separate directory for each rerun by @bayandin in #6004
• add sql_exporter to vm-image by @lubennikovaav in #5949
• Speed up rel extend by @bojanserafimov in #5983
• Use TEST_OUTPUT envvar in pageserver by @bojanserafimov in #5984
• fix: use create_new instead of create for mutex file by @koivunej in #6012
• [compute_ctl] User correct HTTP code in the /configure errors by @ololobus in #6017
• initial logical size: remove CALLS metric from hot path by @problame in #6018
• Override neon.max_cluster_size for the time of compute_ctl config ope… by @lubennikovaav in #5998
• Release 2023-12-04 by @vipvap in #6022

Full Changelog: release-4277...release-4342

release-4277

What's Changed

• fix: use create_new instead of create for mutex file by @koivunej in #6013

Full Changelog: release-4275...release-4277

release-4275

What's Changed

• Notify safekeeper readiness with systemd. by @arssher in #5986

Full Changelog: release-4273...release-4275

release-4273

What's Changed

• Release 2023 11 28 hotfix by @lubennikovaav in #5945

Full Changelog: release-4270...release-4273

release-4270

What's Changed

• chore(background_tasks): missed allowed_error change, logging change by @koivunej in #5883
• build: back to opt-level=0 in debug builds, for faster compile times by @problame in #5751
• Disallow CREATE DATABASE WITH OWNER neon_superuser by @save-buffer in #5887
• Update Rust to 1.74.0 by @arpad-m in #5873
• Cleanup compact_level0_phase1 fsyncing by @koivunej in #5852
• refer to our rust-postgres.git fork by branch name by @problame in #5894
• logging: support output to stderr by @problame in #5896
• Add documentation for our sample pre-commit hook by @lubennikovaav in #5868
• LFC fixes + statistics by @knizhnik in #5727
• fix: cleanup of layers from the future can race with their re-creation by @problame in #5890
• Support cancellation for find_lsn_for_timestamp API by @shanyp in #5904
• Move weekly release PR trigger to Monday morning by @shanyp in #5908
• Upload initdb results to S3 by @arpad-m in #5390
• Create extension neon by @lubennikovaav in #5741
• proxy: fix memory leak again by @conradludgate in #5909
• Check if LFC was intialized in local_cache_pages function by @knizhnik in #5911
• test: support validating allowed_errors against a logfile by @koivunej in #5905
• fix: log cancelled request handler errors by @koivunej in #5915
• tests: fix allowed_error for compaction detecting a shutdown by @koivunej in #5919
• fix: long timeline create cancelled by tenant delete by @koivunej in #5917
• Only create neon extension in postgres database; by @lubennikovaav in #5918
• Release 2023-11-27 by @vipvap in #5922

Full Changelog: release-4245...release-4270

release-4245

What's Changed

• Logical replication startup fixes by @knizhnik in #5750
• s3_scrubber: prepare for scrubbing buckets with generation-aware content by @jcsp in #5700
• refactor: use streaming in safekeeper /v1/debug_dump http response by @duguorong009 in #5731
• Update control plane API path for getting compute spec. by @hlinnaka in #5357
• feat: improve the serde impl for several types(Lsn, TenantId, TimelineId ...) by @duguorong009 in #5335
• pageserver: add Gate as a partner to CancellationToken for safe shutdown of Tenant & Timeline by @jcsp in #5711
• Layer flush minor speedup by @bojanserafimov in #5765
• Add API description for safekeeper copy endpoint by @arpad-m in #5770
• pageserver: add InProgress tenant map state, use a sync lock for the map by @jcsp in #5367
• Make azure blob storage not do extra metadata requests by @arpad-m in #5777
• proxy: remove unsafe by @conradludgate in #5805
• Fix comments in 'receive_wal.rs'. by @wangxuqi in #5807
• pageserver: start refactoring into TenantManager by @jcsp in #5797
• pageserver: avoid converting an error to anyhow::Error by @jcsp in #5803
• refactor: remove unnecessary unsafe by @koivunej in #5802
• Use self.tenants instead of TENANTS by @shanyp in #5811
• build: Add warning when missing postgres submodule during the build by @fluz in #5614
• Dockerfile: Set BUILD_TAG for storage services by @bayandin in #5812
• pageserver: move deletion failpoint inside backoff by @jcsp in #5814
• Support JWT key reload in pageserver by @arpad-m in #5594
• Passing neon options to the console by @prepor in #5781
• pageserver: improve the shutdown log error by @duguorong009 in #5792
• vm-monitor: Remove dependency on workspace_hack by @sharnoff in #5752
• extend test_change_pageserver for failure case, rework changing pageserver by @jcsp in #5693
• test: de-flake test_deletion_queue_recovery by @jcsp in #5822
• pageserver: use Gate for stronger safety check in SlotGuard by @jcsp in #5793
• cleanup unused RemoteStorage fields by @problame in #5830
• JWT logging improvements by @arpad-m in #5823
• Add test simulating bad connection between pageserver and compute by @save-buffer in #5728
• Fix walproposer build on aarch64 by @petuhovskiy in #5827
• remote_storage: use S3 SDK's adaptive retry policy by @jcsp in #5813
• proxy: limit concurrent wake_compute requests per endpoint by @conradludgate in #5799
• Update epic-template.md by @stepashka in #5709
• fix(metric): remove pageserver_wal_redo_wait_seconds by @koivunej in #5791
• Fix blob_io test by @bojanserafimov in #5818
• pageserver: refined Timeline shutdown by @jcsp in #5833
• Lsn for get_timestamp_of_lsn should be string, not integer by @arpad-m in #5840
• temp: log all layer loading errors while we lose them by @koivunej in #5816
• fix(pageserver): spawn all kinds of tenant shutdowns by @koivunej in #5841
• find_lsn_for_timestamp fixes by @arpad-m in #5844
• workflows/neon_extra_builds: run check-codestyle-rust & build-neon on arm64 by @bayandin in #5832
• Set BUILD_TAG for build-neon job by @bayandin in #5847
• metric: add started and killed walredo processes counter by @rmodpur in #5809
• pageserver: error type for collect_keyspace by @jcsp in #5846
• build: remove async-std dependency by @koivunej in #5848
• build: deny async-std and friends by @koivunej in #5849
• build: upgrade ahash by @koivunej in #5851
• Fix comment in find_lsn_for_timestamp by @arpad-m in #5855
• neon_local: use remote storage by default, add cargo neon tenant migrate by @jcsp in #5760
• Migrate links from wiki to notion by @arpad-m in #5862
• Smaller test addition and change by @koivunej in #5858
• Proxy control plane rate limiter by @khanova in #5785
• build(deps): bump aiohttp from 3.8.5 to 3.8.6 by @dependabot in #5864
• Fix baseline commit and branch for code coverage by @bayandin in #5769
• pageserver: make TenantsMap shard-aware by @jcsp in #5819
• Collect number of connections by sni type by @khanova in #5867
• Update vm-builder to v0.19.0 and move its customization here by @sharnoff in #5783
• Proxy log pid hack by @khanova in #5869
• Improve error message when neon.max_cluster_size reached by @sharnoff in #4173
• compute_ctl: Remove deprecated flag --file-cache-on-disk by @sharnoff in #5622
• fix(background_tasks): first backoff, compaction error stacktraces by @koivunej in #5881
• pageserver: shut down idle walredo processes by @jcsp in #5877
• fix(layer): VirtualFile opening and read errors can be caused by contention by @koivunej in #5880
• Release 2023-11-17 by @vipvap in #5876

New Contributors

• @wangxuqi made their first contribution in #5807
• @fluz made their first contribution in #5614
• @prepor made their first contribution in #5781

Full Changelog: release-4179...release-4245

release-4179

What's Changed

• Run real Azure tests on CI by @bayandin in #5627
• proxy: filter out more quota exceeded errors by @conradludgate in #5640
• pageserver: schedule frozen layer uploads inside the layers lock by @jcsp in #5639
• Use tuples instead of manual comparison chain by @arpad-m in #5637
• Bump vm-builder v0.18.1 -> v0.18.2 by @sharnoff in #5646
• move wake compute after the auth quirks logic by @conradludgate in #5642
• tests: de-flake test_eviction_across_generations by @jcsp in #5650
• misc smaller improvements by @koivunej in #5527
• refactor(remote_timeline_client): Split deletion into unlinking + deletion by @koivunej in #5645
• fix http pool test by @conradludgate in #5653
• tests: don't fail tests on torn log lines by @jcsp in #5655
• Switch to querying new tests results DB by @bayandin in #5616
• pageserver: suppress some of the most common spurious warnings by @jcsp in #5658
• proxy refactor serverless by @conradludgate in #4685
• Fix test instructions readme by @bojanserafimov in #5644
• Optionally return json from get_lsn_by_timestamp by @arpad-m in #5608
• Ignore missed AUX_FILES_KEY when generating image layer by @knizhnik in #5660
• fix: trampling on global physical size metric by @koivunej in #5663
• refactor(rtc): schedule compaction update by @koivunej in #5649
• Get env var for real Azure tests from GitHub by @bayandin in #5662
• build(deps): bump werkzeug from 2.2.3 to 3.0.1 by @dependabot in #5665
• pageserver: quieten "Failed to get info about AUX files" by @jcsp in #5669
• reimpl Layer, remove remote layer, trait Layer, trait PersistentLayer by @koivunej in #4938
• pageserver: suppress compaction/gc errors while stopping by @jcsp in #5670
• WALRedo: Limit logging to log_level = ERROR and above by @MMeent in #5587
• pageserver: exponential backoff on compaction/GC failures by @jcsp in #5672
• pageserver: always load remote metadata (no more spawn_load) by @jcsp in #5580
• pageserver: fix the second "AUX files" warning by @jcsp in #5673
• Support changing pageserver dynamically by @save-buffer in #5542
• Bump vm-builder v0.18.2 -> v0.18.5 by @sharnoff in #5666
• scrubber: add separate find/purge garbage commands by @jcsp in #5409
• proxy: prepare to remove high cardinality metrics by @conradludgate in #5461
• tests: revise perf test that interfered with local disk state by @jcsp in #5682
• feat: add build_tag env support for set_build_info_metric by @duguorong009 in #5576
• fix: schedule_compaction_update must only unlink by @koivunej in #5675
• Fix safekeeper log spans by @petuhovskiy in #5643
• Some minor renames in attachment service API by @NanoBjorn in #5687
• pageserver: add generation fields in openapi spec by @jcsp in #5690
• proxy: exclude client latencies in metrics by @conradludgate in #5688
• fix: possible page_service hang on cancel by @koivunej in #5696
• Fix Rust 1.74 warnings by @arpad-m in #5702
• proxy: dont retry server errors by @conradludgate in #5694
• pageserver: cleaner shutdown in timeline delete by @jcsp in #5701
• Prevent SIGSEGV in apply_error_callback when record was not decoded by @knizhnik in #5703
• pageserver: fix error logging on stray timeline files by @jcsp in #5712
• Layer: logging fixes by @koivunej in #5676
• tests: fix missing comma in test_timeline_deletion_with_files_stuck_… by @jcsp in #5713
• Grant BYPASSRLS and REPLICATION to exited roles by @knizhnik in #5657
• fix nightly 1.75 by @conradludgate in #5719
• pageserver: downgrade slow task warnings from warn to info by @jcsp in #5724
• Move release PR creation to Friday by @shanyp in #5721
• pageserver: /attach: clarify semantics of 409 by @problame in #5698
• tests: minor improvements by @jcsp in #5674
• refactor(layer): remove version checking with atomics by @koivunej in #5742
• compute_ctl: Assume --vm-monitor-addr arg is always present by @sharnoff in #5611
• Add docs for updating Postgres for new minor versions by @tristan957 in #5054
• Always handle POLLHUP in walredo error poll loop by @knizhnik in #5716
• fix: Add bearer prefix to Authorization header by @mtyazici in #5740
• fix(layer): reinit on access before eviction happens by @koivunej in #5743
• Record wal size in import benchmark by @bojanserafimov in #5755
• feat(layer): cancellable get_or_maybe_download by @koivunej in #5744
• pageserver: helpers for explicitly dying on fatal I/O errors by @jcsp in #5651
• fix(layer): error out early if layer path is non-file by @koivunej in #5756
• fix(layer): get_and_upgrade and metrics by @koivunej in #5767
• Remove outdated and flaky perf test by @bojanserafimov in #5762
• vm-monitor: Remove support for file cache in tmpfs by @sharnoff in #5617
• Proxy introduce allowed ips by @khanova in #5729
• chore(layer): restore logging, doc changes by @koivunej in #5766
• proxy: no more statements by @conradludgate in #5747
• Release 2023-11-03 by @vipvap in #5778

New Contributors

• @mtyazici made their first contribution in #5740

Full Changelog: release-4108...release-4179

release-4108

What's Changed

• Release 2023-10-26 by @vipvap in #5685

Full Changelog: release-4105...release-4108