release-5578

What's Changed

• Metrics hotfix release by @save-buffer in #7869

Full Changelog: release-5577...release-5578

release-5577

What's Changed

• pageserver: use the heatmap upload interval to set the secondary download interval by @jcsp in #7793
• Upgrade of pgvector to 0.7.0 by @Bodobolero in #7726
• safekeeper: log LSNs on connection exit by @arssher in #7728
• feat(pageserver): generate image layers for sparse keyspace by @skyzh in #7567
• Update banner image in Readme by @ruf-io in #7801
• feat(pageserver): do not read past image layers for vectored get by @skyzh in #7773
• Add some more replication slot metrics by @save-buffer in #7761
• build(deps): bump requests from 2.31.0 to 2.32.0 by @dependabot in #7816
• pageserver: fix secondary progress stats when layers are 404 by @jcsp in #7814
• Propagate apply_lsn from SK to PS to prevent GC from collecting objects which may be still requested by replica by @knizhnik in #7368
• Postgres 14.12, 15.7, and 16.3 updates by @tristan957 in #7688
• neon_local: run controller's postgres with fsync=off by @jcsp in #7817
• test(detach_ancestor): ensure L0 compaction in history is ok by @koivunej in #7813
• chore(pageserver): remove metrics for in-memory ingestion by @skyzh in #7823
• Add dummy lsn lease http and page service APIs by @arpad-m in #7815
• compute_ctl: Fix handling of missing /neonvm/bin/resize-swap by @sharnoff in #7832
• Implement timeline_manager for safekeeper background tasks by @petuhovskiy in #7768
• Refactor PageServerHandler::process_query parsing by @arpad-m in #7835
• Another iteration on making walcraft more robust by @arssher in #7588
• Fix zenith_test_evict mode and clear_buffer_cache() function by @hlinnaka in #7803
• Compute-Only Release 2024-05-22 by @vipvap in #7837

New Contributors

• @ruf-io made their first contribution in #7801

Full Changelog: release-5545...release-5577

release-5545

What's Changed

• test(ancestor detach): verify with fullbackup by @koivunej in #7706
• test: fix confusing limit and logging by @koivunej in #7589
• test: allow vectored get validation failure during shutdown by @koivunej in #7716
• chore(pageserver): plumb through RequestContext to VirtualFile read methods by @problame in #7720
• pageserver: refuse to run without remote storage by @jcsp in #7722
• python: allow using allowed_errors.py by @koivunej in #7719
• Fix test_lock_time_tracing flakiness by @jbajic in #7712
• Revert protocol version upgrade by @VladLazar in #7727
• chore(pageserver): plumb through RequestContext to VirtualFile open methods by @problame in #7725
• CI(report-benchmarks-failures): report benchmarks failures to slack by @bayandin in #7678
• fix: do not create metrics contention from background task permit by @koivunej in #7730
• feat(pageserver): add metrics for aux file size by @skyzh in #7623
• pageserver: improve API for invoking GC by @jcsp in #7655
• pagectl: fix diagrams generation for paths containing generations by @VladLazar in #7739
• pageserver: use adaptive concurrency in secondary layer downloads by @jcsp in #7675
• fix(test): ensure compatibility test uses the correct compute node by @skyzh in #7741
• Tiered compaction: cut deltas along lsn as well if needed by @arpad-m in #7671
• Always use Lsn::MAX as the request LSN in the primary by @hlinnaka in #7708
• pageserver: do fewer heatmap uploads for tiny tenants by @jcsp in #7731
• pageserver: switch on new-style local layer paths by @jcsp in #7660
• CI(report-benchmarks-failures): fix condition by @bayandin in #7745
• chore(test): add version check for forward compat test by @skyzh in #7685
• pageserver: skip waiting for logical size on shard >0 by @jcsp in #7744
• Update mold to 2.31 by @arpad-m in #7757
• test suite: allow overriding default compaction algorithm via env var by @problame in #7747
• Add restart_lsn metric for logical slots. by @arssher in #7698
• CI(neon-extra-builds): Use small-arm64 runners instead of large-arm64 by @bayandin in #7740
• pageserver: remove Option<> around remote storage, clean up metadata file refs by @jcsp in #7752
• chore(deps): use upstream svg_fmt after they merged our PR by @problame in #7764
• chore!: always use async walredo, warn if sync is configured by @problame in #7754
• Add test_uploads_and_deletions test by @arpad-m in #7758
• fixup #7747: actually use the fixture for neon_env_builder by @problame in #7767
• Add performance regress test_ondemand_download_churn.py by @jbajic in #7242
• feat(pageserver): use fnv hash for aux file encoding by @skyzh in #7742
• fix(test): ensure fixtures are correctly used for pageserver_aux_file_policy by @skyzh in #7769
• pageserver: refine tenant_id->shard lookup by @jcsp in #7762
• compute_ctl: catalog API endpoints by @prepor in #7575
• proxy: swap tungstenite for a simpler impl by @conradludgate in #7353
• tiered compaction: introduce PAGE_SZ constant and use it by @arpad-m in #7785
• Remove SSO_ACCOUNT_ID from scrubber docs and BucketConfig by @arpad-m in #7774
• Tiered compaction: improvements to the windows by @arpad-m in #7787
• test_suite: patch, don't replace, the tenant_config field, where appropriate by @problame in #7771
• chore: update defaults for timeline_detach_ancestor by @koivunej in #7779
• pageserver: fixes for layer path changes by @jcsp in #7786
• storage controller: use SERIALIZABLE isolation level by @jcsp in #7792
• pageserver: implement auto-splitting by @jcsp in #7681
• feat(pageserver): persist aux file policy in index part by @skyzh in #7668
• feat(pagebench): add aux file bench by @skyzh in #7746
• Fix calculation in test by @hlinnaka in #7805
• proxy: bump parquet by @conradludgate in #7782
• Release 2024-05-20 by @vipvap in #7807

Full Changelog: release-5493...release-5545

release-5493

What's Changed

• Add option allowing running multiple endpoints on the same branch. by @arssher in #7581
• Make 'neon.protocol_version = 2' the default by @hlinnaka in #7616
• pageserver: call maybe_freeze_ephemeral_layer from a dedicated task by @jcsp in #7594
• pageserver: label tenant_slots metric by slot type by @jcsp in #7603
• Fix grammar in provider.rs error message by @tristan957 in #6864
• refactor(pageserver): remove --update-init flag by @problame in #7612
• test: less flaky test_synthetic_size_while_deleting by @koivunej in #7622
• Fix test_ts_of_lsn_api flakiness by @arpad-m in #7599
• On-demand WAL download for walsender by @save-buffer in #6872
• proxy: reduce number of concurrent connections by @khanova in #7620
• use less neon_local --pageserver-config-override / pageserver -c by @problame in #7613
• pageserver: post-shard-split layer trimming (1/2) by @jcsp in #7572
• feat: Timeline detach ancestor by @koivunej in #7456
• build(deps): bump werkzeug from 3.0.1 to 3.0.3 by @dependabot in #7625
• build(deps): bump jinja2 from 3.1.3 to 3.1.4 by @dependabot in #7626
• Add Postgres commit sha to Postgres version by @bayandin in #4603
• build(deps): bump flask-cors from 3.0.10 to 4.0.1 by @dependabot in #7633
• fix(test suite): forward compat test is not using latest neon_local by @problame in #7637
• refactor(rtc): remove excess cloning by @koivunej in #7635
• remove neon_local --pageserver-config-override by @problame in #7614
• feat(pageserver): generate basebackup from aux file v2 storage by @skyzh in #7517
• pageserver: include generation number in local layer paths by @jcsp in #7609
• proxy: cull http connections by @conradludgate in #7632
• compute_ctl: Break up main() by @sharnoff in #7577
• compute_ctl: Fix unused variable on non-Linux by @sharnoff in #7646
• tests: common log allow list for ancestor detach tests by @jcsp in #7645
• neon_local init: write pageserver.toml directly; no pageserver --init --config-override by @problame in #7638
• proxy: do not invalidate cache for permit errors by @conradludgate in #7652
• Tiered compaction: add order asserts after delta key k-merge by @arpad-m in #7648
• Improve tiered compaction tests by @arpad-m in #7643
• build(deps): bump moto from 4.1.2 to 5.0.6 by @bayandin in #7653
• neon_local: use pageserver.toml as source of truth for struct PageServerConf by @problame in #7642
• Fix flaky test_gc_of_remote_layers by @petuhovskiy in #7647
• Expose LSN and replication delay as metrics by @save-buffer in #7610
• chore(neon_test_utils): restrict installation to superuser by @skyzh in #7624
• pageserver: local layer path followups by @jcsp in #7640
• Ignore page header when comparing VM pages in test_vm_bits.py by @knizhnik in #7499
• pageserver: fix division by zero in layer counting metric by @VladLazar in #7662
• remote_storage: AWS_PROFILE with endpoint overrides in ~/.aws/config (updates AWS SDKs) by @problame in #7664
• tests: adjust log allow list since reqwest upgrade by @jcsp in #7666
• storage controller: fix handing of tenants with no timelines during scheduling optimization by @jcsp in #7673
• Fix tiered compaction k-merge bugs and use in-memory alternative by @arpad-m in #7661
• Revert "chore(neon_test_utils): restrict installation to superuser" by @skyzh in #7679
• build(deps): bump Npgsql from 8.0.2 to 8.0.3 in /test_runner/pg_clients/csharp/npgsql by @dependabot in #7680
• fix(test): update the config for neon_binpath in from_repo_dir by @skyzh in #7684
• Proxy added per ep rate limiter by @khanova in #7636
• pageserver: don't call get_vectored with empty keyspace by @jcsp in #7686
• use own arm64 gha runners by @cicdteam in #7373
• Fix permissions for safekeeper failpoints by @petuhovskiy in #7669
• openapi: resolve the synthetic_size duplication by @koivunej in #7651
• draw_timeline_dir: draw branch points and gc cutoff lines by @koivunej in #7657
• build: run doctests by @koivunej in #7697
• Tiered compaction: fix early exit check in main loop by @arpad-m in #7702
• Fix checkpoint metric by @lubennikovaav in #7701
• feat: allow detaching from ancestor for timelines without writes by @koivunej in #7639
• Handmade Release 2024-05-13 by @VladLazar in #7735
• Release 2024-05-13 by @vipvap in #7713

Full Changelog: release-5434...release-5493

release-5434

What's Changed

• neon_local: add "tenant import" by @jcsp in #7399
• storage controller: send startup compute notifications in background by @jcsp in #7495
• proxy: Exclude compute and retries by @khanova in #7529
• s3_scrubber: add tenant-snapshot by @jcsp in #7444
• proxy: Adjust retry wake compute by @khanova in #7537
• proxy: Create disconnect events by @khanova in #7535
• pagserver: use vectored read path in benchmarks by @VladLazar in #7498
• chore(vm-image): specify sql exporter listen port by @skyzh in #7526
• pageserver: tweak vec get validation for ancestor lsn wait by @VladLazar in #7533
• Update azure_* crates to 0.19 by @arpad-m in #7539
• chore(pageserver): improve in-memory layer vectored get by @skyzh in #7467
• pageserver: shard-aware keyspace partitioning by @jcsp in #6778
• pageserver: remove workarounds from #7454 by @jcsp in #7550
• docs: fix unintentional file link by @fcdm in #7506
• feat(pageserver): separate sparse and dense keyspace by @skyzh in #7503
• pageserver: reduce runtime of init_tenant_mgr by @jcsp in #7553
• Make test_random_updates and test_read_at_max_lsn compatible with new compaction by @arpad-m in #7551
• Make s3_scrubber for safekeepers useful. by @arssher in #7297
• storage controller: test for large shard counts by @jcsp in #7475
• tests: tweak log allow list in test_sharding_split_failures by @jcsp in #7549
• Makefile: DISABLE_HOMEBREW variable by @prepor in #7556
• Discover safekeepers via broker request by @petuhovskiy in #7279
• chore(pageserver): concise error message for layer traversal by @skyzh in #7565
• chore(pageserver): categorize basebackup errors by @skyzh in #7523
• Add retry loops and bump test timeout in test_pageserver_connection_stress by @save-buffer in #7281
• update to reqwest 0.12 by @conradludgate in #7561
• proxy: keep track on the number of events from redis by type. by @khanova in #7582
• Fix usage of pg_waldump --ignore option by @lubennikovaav in #7578
• fix(pageserver): properly propagate missing key error for vectored get by @skyzh in #7569
• storage controller: make the initial database wait configurable by @mattpodraza in #7591
• vm-image: add sqlexporter for autoscaling metrics by @skyzh in #7514
• chore(pageserver): plumb through RequestContext to VirtualFile write methods by @problame in #7566
• Remove tenant_id from TenantLocationConfigRequest by @arpad-m in #7469
• Use streaming downloads for Azure as well by @arpad-m in #7579
• Proxy reconnect pubsub before expiration by @khanova in #7562
• Store operation identifier in IdLockMap on exclusive lock by @jbajic in #7397
• fix(pageserver): compile warning of download_object.ctx on macos by @skyzh in #7596
• refactor(update_gc_info): split GcInfo to compose out of GcCutoffs by @koivunej in #7584
• Increase Azure parallelism limit to 100 by @arpad-m in #7597
• tests: relax test download remote layers api by @VladLazar in #7604
• storage_controller: fix non-timeline passthrough GETs by @jcsp in #7602
• fix: find gc cutoff points without holding Tenant::gc_cs by @koivunej in #7585
• pageserver: jitter secondary periods by @jcsp in #7544
• Update rust to 1.78.0 by @arpad-m in #7598
• test_runner: remove unused NeonPageserver.config_override field by @problame in #7605
• storage controller: check warmth of secondary before doing proactive migration by @jcsp in #7583
• feat(pageserver): add scan interface by @skyzh in #7468
• proxy: add connect compute concurrency lock by @conradludgate in #7607
• fix(pageserver): remove update_gc_info calls in tests by @skyzh in #7608
• refactor: move NodeMetadata to pageserver_api; use it from neon_local by @problame in #7606
• compute_ctl: Add support for swap resizing by @sharnoff in #7434
• compute_ctl: Be more precise in how startup time is calculated by @hlinnaka in #7601
• Allow bad state (not active) pageserver error/warns in walcraft test. by @arssher in #7592
• Properly initialize first WAL segment on safekeepers. by @arssher in #7574
• Release 2024-05-06 by @vipvap in #7615

New Contributors

• @mattpodraza made their first contribution in #7591

Full Changelog: release-5373...release-5434

release-5373

What's Changed

• Make test less sensitive to exact WAL positions by @hlinnaka in #7436
• Make another walcraft test more robust by @hlinnaka in #7439
• refactor(Timeline): simpler metadata updates by @koivunej in #7422
• relaxation: allow using layers across timelines by @koivunej in #7453
• proxy: Make retries configurable and record it. by @khanova in #7438
• pageserver: fix unlogged relations with sharding by @jcsp in #7454
• pagectl draw-timeline-dir: include layer file name as an SVG comment by @problame in #7455
• chore(pageserver): separate missing key error by @skyzh in #7393
• Refactor updating relation size cache on reads by @hlinnaka in #7376
• pageserver: remove import/export script previously used for breaking format changes by @VladLazar in #7458
• Don't pass tenant_id in location_config requests from storage controller by @arpad-m in #7476
• pageserver: fix vectored read aux key handling by @VladLazar in #7404
• pageserver: add read depth metrics and test by @VladLazar in #7464
• storage_controller: wait for db on startup by @jcsp in #7479
• feat(pageserver): add metadata key range and aux key encoding by @skyzh in #7401
• remote_storage: fix prefix handling in remote storage & clean up by @jcsp in #7431
• pageserver: change pitr_interval=0 behavior by @jcsp in #7423
• get_lsn_by_timestamp: clamp commit_lsn to be >= min_lsn by @arpad-m in #7488
• fix: avoid starving background task permits in eviction task by @koivunej in #7471
• Fix a bug with retries by @khanova in #7494
• pageserver: publish the same metrics from both read paths by @VladLazar in #7486
• pageserver: finish vectored get early by @VladLazar in #7490
• fix(metrics): correct maxrss metrics on macos by @skyzh in #7487
• Ability to specify the upload_storage_class in S3 bucket configuration by @arpad-m in #7461
• controller: limit Reconciler concurrency by @jcsp in #7493
• reduce complexity of proxy protocol parse by @conradludgate in #7078
• Update connect to compute and wake compute retry configs by @khanova in #7509
• proxy: Fix cancellations by @khanova in #7510
• pageserver: coalesce read paths by @VladLazar in #7477
• proxy: Workaround switch to the regional redis by @khanova in #7513
• chore(pageserver): temporary metrics on ingestion time by @skyzh in #7515
• Compute <-> pageserver protocol version 2 by @hlinnaka in #7377
• storage controller: log hygiene & better error type by @jcsp in #7508
• refactor(owned_buffers_io::BufferedWriter): be generic over the type of buffer by @problame in #7482
• refactor(owned_buffer_io::util::size_tracking_writer): make generic over underlying writer by @problame in #7483
• refactor(ephemeral_file): reuse owned_buffers_io::BufferedWriter by @problame in #7484
• Remove unused import by @jcsp in #7519
• perf!: use larger buffers for blob_io and ephemeral_file by @problame in #7485
• pageserver: fix image layer creation check that inhibited compaction by @VladLazar in #7420
• Return Past instead of Present or Future when commit_lsn < min_lsn by @arpad-m in #7520
• feat(pageserver): add aux-file-v2 flag on tenant level by @skyzh in #7505
• chore(pageserver): shrink aux keyspace to 0x60-0x7F by @skyzh in #7502
• fix(virtual_file): compile warnings on macos by @skyzh in #7525
• metrics: record time to update gc info as a per timeline metric by @koivunej in #7473
• Release 2024-04-29 by @vipvap in #7530

Full Changelog: release-5324...release-5373

release-5324

What's Changed

• pageserver: improve readability of shard.rs by @jcsp in #7330
• storage release: stop using no-op deployProxy / deployPgSniRouter by @problame in #7382
• proxy: Exclude private ip errors from recorded metrics by @khanova in #7389
• add async walredo mode (disabled-by-default, opt-in via config) by @problame in #6548
• pageserver: return ACCEPTED when deletion already in flight by @jcsp in #7384
• storage_controller: suppress misleading log by @jcsp in #7395
• proxy: rate limit authentication by masked IPv6. by @conradludgate in #7316
• pageserver/http: remove status code boilerplate from swagger spec by @VladLazar in #7385
• feat(neon_local): support listen addr for safekeeper by @skyzh in #7328
• chore(compute_ctl): add error context to apply_spec by @skyzh in #7374
• CI(check-build-tools-image): fix getting build-tools image tag by @bayandin in #7402
• proxy: Cleanup unused rate limiter by @khanova in #7400
• Add for add-help-for-timeline-arg for timeline command by @jbajic in #7361
• pageserver: demote range end fallback log by @VladLazar in #7403
• proxy: Improve logging by @khanova in #7405
• proxy: Record role to span by @khanova in #7407
• proxy: move endpoint rate limiter by @conradludgate in #7413
• proxy: Record only valid rejected events by @khanova in #7415
• feat: copy delta layer prefix or "truncate" by @koivunej in #7228
• pageserver: fix secondary download scheduling by @jcsp in #7396
• Add a docs page for storage controller by @jcsp in #7392
• fix(Layer): metric regression with too many canceled evictions by @koivunej in #7363
• build(deps): bump aiohttp from 3.9.2 to 3.9.4 by @dependabot in #7429
• pageserver: fix cont lsn jump on vectored read path by @VladLazar in #7412
• storcon_cli: tenant-warmup command by @jcsp in #7432
• CI: fix docker/setup-buildx-action & update rustls by @bayandin in #7445
• Bump vm-builder v0.23.2 -> v0.28.1 by @sharnoff in #7433
• Release 2024-04-22 by @vipvap in #7447

Full Changelog: release-5294...release-5324

release-5294

What's Changed

• refactor(pageserver): use tokio::signal instead of spawn_blocking by @problame in #7332
• Remove async_trait from CompactionDeltaLayer by @arpad-m in #7342
• pageserver: option to run with just one tokio runtime by @problame in #7331
• implement Serialize/Deserialize for SystemTime with RFC3339 format by @kevinmingtarja in #7203
• update measured with some more convenient features by @conradludgate in #7334
• CI(flaky-tests): remove outdated restriction by @bayandin in #7345
• Update staging hostname by @bayandin in #7347
• Proxy read ids from redis by @khanova in #7205
• pageserver: time based rolling based on the first write timestamp by @VladLazar in #7346
• Add command to time travel recover prefixes by @arpad-m in #7322
• proxy: hyper1 for only proxy by @conradludgate in #7073
• proxy: fix credentials cache lookup by @khanova in #7349
• Revert "Proxy read ids from redis (#7205)" by @khanova in #7350
• compute_ctl: Auto-set dynamic_shared_memory_type by @sharnoff in #7348
• Remember last written LSN when it is first requested by @knizhnik in #7343
• Reenable test_forward_compatibility by @petuhovskiy in #7358
• compute hook: use shared client with explicit timeout by @jcsp in #7359
• CI(build-build-tools-image): Do not cancel concurrent workflows by @bayandin in #7226
• proxy: replace prometheus with measured by @conradludgate in #6717
• Read cplane events from regional redis by @khanova in #7352
• proxy: fix overloaded db connection closure by @conradludgate in #7364
• CI: speed up Allure reports upload by @bayandin in #7362
• Fix typo in pg_ctl shutdown mode by @tristan957 in #7365
• build(deps): bump idna from 3.3 to 3.7 by @dependabot in #7367
• Release 2024-04-15 by @vipvap in #7378

New Contributors

• @kevinmingtarja made their first contribution in #7203

Full Changelog: release-5269...release-5294

release-5269

What's Changed

• Release 2024-04-11(ish) cherry-pick #7348 by @sharnoff in #7356

Full Changelog: release-5267...release-5269

release-5267

What's Changed

• update tokio 1.37 by @conradludgate in #7276
• Standardizing Logging Format for Improved Middleware Integration by @macdoos in #7194
• CI(benchmarking): Add Sharded Tenant for pgbench by @bayandin in #7186
• tests: clean up compat test workarounds by @jcsp in #7097
• pageserver: drop the layer map lock after planning reads by @VladLazar in #7215
• proxy: upload consumption events to S3 by @khanova in #7213
• controller: add storcon-cli by @jcsp in #7114
• remove http timeout by @conradludgate in #7291
• pageserver: even more debug for test_secondary_downloads by @jcsp in #7295
• refactor(walreceiver): eliminate task_mgr usage by @problame in #7260
• wal_ingest: global counter for bytes received by @koivunej in #7240
• Schedule image layer uploads in tiered compaction by @arpad-m in #7282
• pageserver: on-demand activation of tenant on GET tenant status by @jcsp in #7250
• Upload partial segments by @petuhovskiy in #6530
• pageserver: replace the locked tenant config with arcsawps by @VladLazar in #7292
• refactor(Timeline::shutdown): rely more on Timeline::cancel; use it from deletion code path by @problame in #7233
• update ordered-multimap by @conradludgate in #7306
• tests: reinstate some syntethic size tests by @VladLazar in #7294
• Fix bug in prefetch cleanup by @knizhnik in #7277
• Fix length by @khanova in #7308
• storage controller: grant 'admin' access to all APIs by @jcsp in #7307
• storage_controller: setup sentry reporting by @VladLazar in #7311
• fix(walredo spawn): coalescing stalls other executors std::sync::RwLock by @problame in #7310
• pageserver: timeout on deletion queue flush in timeline deletion by @jcsp in #7315
• pageserver: handle WAL gaps on sharded tenants by @jcsp in #6788
• pageserver: update mgmt_api to use TenantShardId by @jcsp in #7313
• storage controller: fix repeated location_conf returning no shards by @jcsp in #7314
• tests: extend log allow list in test_storcon_cli by @jcsp in #7321
• Fix zero segment upload by @petuhovskiy in #7318
• Update test-context dependency to 0.3 by @arpad-m in #7303
• proxy: report metrics based on cold start info by @conradludgate in #7324
• Clean up 'attachment service' names to storage controller by @jcsp in #7326
• storage controller: rename TenantState to TenantShard by @jcsp in #7329
• tests: improve stability of test_deletion_queue_recovery by @jcsp in #7325
• pageserver: apply shard filtering to blocks ingested during initdb by @jcsp in #7319
• fixup: PR #7319 defined workload.py def stop() twice by @problame in #7333
• control_plane: revise compute_hook locking (don't serialise all calls) by @jcsp in #7088
• tests: further stabilize test_deletion_queue_recovery by @jcsp in #7335
• fix(utilization): return used as does df by @koivunej in #7337
• Release 2024-04-08 by @vipvap in #7338

New Contributors

• @macdoos made their first contribution in #7194

Full Changelog: release-5227...release-5267