SRE update 2024-10-04 (#478)

* SRE update 2024-10-04

* SRE quarterly update 2024-Q3

blog/2024-10-04-sre.md

@@ -0,0 +1,61 @@
+---
+title: SRE Team Update
+slug: 2024-10-04-sre
+authors: johnalotoski
+tags: [sre]
+hide_table_of_contents: false
+---
+
+## High level summary
+
+The SRE team continues work on Cardano environment improvements and general maintenance.
+
+Some notable recent changes, updates or improvements include:
+
+* All environments have been upgraded to cardano-node `9.2.1`.
+
+* Cardano-faucet `9.2` is available and deployed which is compatible with node
+  `9.2.x` and has fixed ipv6 functionality.
+
+* All deployed machines now default to nix `2.24-maint` after an upstream bug
+  causing a hash miscalculation in submodules has been fixed.  Nix `2.21`
+  forward required some rework of the colmena deployment recipes as dirty git
+  trees now force an impure colmena deployment.
+
+* To ease the process of upgrading cardano-parts,
+  [releases](https://github.com/input-output-hk/cardano-parts/releases) will
+  now be made instead of only PR merges with migration notes.
+
+## Repository Work
+
+### Cardano-faucet
+* Binds ipv6 interface in addition to ipv4, parses and logs all ips to a
+  unified ipv6 format and applies hlint and fmt updates.
+  [cardano-faucet-pull-14](https://github.com/input-output-hk/cardano-faucet/pull/14)
+
+* Makes required changes for cardano-api `9.2.0.0` and `9.3.0.0`,  removes void
+  type sig constraints, bumps haskellNix, CHaP, cardano-api -> `9.3.0.0` for
+  node `9.2.x` compatibility. Disables mingw32 builds until alex in current
+  haskellNix pin is updated.
+  [cardano-faucet-pull-15](https://github.com/input-output-hk/cardano-faucet/pull/15)
+
+### Cardano-parts
+* Sets cardano-node to `9.2.1`, cardano-faucet to `9.2`. Bumps nix to `2.24-maint`
+  and adds ipv6 and nix versioning fixes and other improvements. Begins
+  cardano-parts date-based releases for an improved upgrade process.
+  Adds misc fixes and improvements. More detail is available in
+  the release notes:
+  [cardano-parts-release-v2024-10-07](https://github.com/input-output-hk/cardano-parts/releases/tag/v2024-10-07)
+
+### Cardano-playground
+* Deploys cardano-node to `9.2.1`, cardano-faucet to `9.2`. Bumps nix to `2.24-maint`
+  and adds ipv6 and nix versioning fixes and other improvements.  Adds a wip
+  node pparams api server. More detail is available in the PR description:
+  [cardano-playground-pull-33](https://github.com/input-output-hk/cardano-playground/pull/33)
+
+### Cardano-mainnet
+* Deploys cardano-node to `9.2.1`, bumps nix to `2.24-maint` and adds ipv6 and
+  nix versioning fixes and other improvements. Converts bootstraps to a new
+  cached-index-patch branch and upgrades CF canary sql queries. More detail is
+  available in the PR description:
+  [cardano-mainnet-pull-23](https://github.com/input-output-hk/cardano-mainnet/pull/23)

quarterly/2024-Q3-sre.md

@@ -0,0 +1,85 @@
+---
+title: SRE Q3 2024 Update
+slug: 2024-Q3-sre
+authors: johnalotoski
+tags: [sre]
+hide_table_of_contents: false
+---
+
+## 2024-07 - 2024-09
+
+### Main achievements
+
+In addition to ongoing general maintenance and support of cardano environments,
+SRE achievements for this quarter include:
+
+* All IOE cardano-parts supported node environments, including preview,
+  preprod, sanchonet, mainnet and other clusters were upgraded through various
+  cardano-node releases of `9.0.0`, `9.1.0`, `9.1.1`, `9.2.0`, and finally into
+  `9.2.1` by the end of September.
+
+* All IOE cardano-parts supported node environments had dual stack ipv4/ipv6
+  capability added and configured, including supporting scripts and recipes,
+  module updates, terraform/openTofu resource changes and software updates to
+  make previously ipv6 incompatible software ipv6 compatible, example:
+  cardano-faucet. Cardano-parts clusters can now seamlessly participate in ipv6
+  cardano-node traffic and other ipv6 traffic.
+
+* Preview, preprod and mainnet networks were hard forked to Conway.
+
+* Legacy mainnet cluster shelley-era high-load relays were scaled down over the
+  quarter and stopped now that p2p has removed the need for them.
+
+* Legacy cardano [explorer](https://explorer.cardano.org/) was retired and
+  Cardano Foundation is now providing the replacement landing page which links
+  to several community explorers.
+
+* Cardano-smash production load was retired from equinix metal hosting from the
+  cardano-world repo and transferred to the new cardano-mainnet cluster.
+
+* New cardano-mainnet cluster scaling capability was added for the bootstrap
+  machines.  Block performance analysis was used to tune RTS parameters on the
+  bootstraps and other mainnet pool machines.
+
+* Sanchonet environment was re-spun for cardano-node `9.1.0` and greater compatibility.
+
+* Private chain was stopped and re-spun with 2 hr epochs for testing.
+
+* New nixosModules were added to cardano-parts and cardano-playground,
+  including: `profile-blockperf`, `profile-tcpdump` (for saving node traffic
+  pcaps to s3) and `ogmios`.
+
+* Documentation for playground and mainnet cluster operations was improved,
+  such as documents for: debugging of peer-to-peer connections; governance
+  voting with the playground stakepools; faucet setup; faucet pool
+  de-delegation and mainnet dbsync cardano-snapshot operations. See the
+  `docs/explain` directory of both the
+  [cardano-playground](https://github.com/input-output-hk/cardano-playground)
+  and [cardano-mainnet](https://github.com/input-output-hk/cardano-mainnet)
+  repos for details.
+
+* The [cardano-monitoring](https://github.com/input-output-hk/cardano-monitoring)
+  repository received a lot of documentation and improvements and now also
+  serves as the home for `devx-ci` metrics after migration away from Grafana
+  cloud hosting.
+
+* An improved [cardano-airgap](https://github.com/Intersectmbo/cardano-airgap)
+  image for secure signing operations was created and made available.
+
+* Hydra CI performance was improved with changes to our custom Nix evaluator
+  and optimized resource usage while waiting for IFDs.
+
+### Next steps
+
+* Add a production protocol-parameters cardano-api based server to facilitate
+  community transaction creation without requiring a live node.
+
+* Migrate from deprecated grafana agent to grafana alloy.
+
+* Finalize support for the new cardano-node tracing system once the service is
+  rewritten for general consumption.
+
+* Extend govtool frontend and backend to a process-compose stack once govtool
+  is publicly buildable again.
+
+* Continue cardano-parts and operations improvements.