v2023.12.0

This release is a significant, backwards incompatible change with respect to coffea 0.7 analysis code and patterns. We have made sure the translation from coffea 0.7 to this version of coffea is not unendurably difficult.

The coffea 202x series is a major upgrade to coffea that incorporates uproot5, awkward2, dask-awkward, dask-histogram, and fsspec. It uses all of these packages to create a cohesive, scalable, and fluidly evolving analysis and analysis-framework design platform that is convenient, well performing, and robust. Using this release, analysis design and execution is made significantly more smooth and manageable at all stages from initial ideation and data exploration to producing publication quality results.

It includes significant new features like: failed file handling and partial job completion, skimming, high-level dataset handling, streamlined interfaces to machine learning inference at scale, access to columnar jet-clustering, ... .

Many thanks to everyone who made this release possible including: @jpivarski @agoose77 @nsmith- @douglasdavis @yimuchen @iasonkrom @valsdav

What's Changed

• build(deps): bump pypa/gh-action-pypi-publish from 1.7.1 to 1.8.1 by @dependabot in #774
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #781
• build(deps): bump pypa/gh-action-pypi-publish from 1.8.1 to 1.8.4 by @dependabot in #780
• Awkward v2 transition by @lgray in #736
• build(deps): bump pypa/gh-action-pypi-publish from 1.8.4 to 1.8.5 by @dependabot in #782
• Get tests in passing state, mark heisenbugs by @lgray in #789
• update awkward2 pin by @lgray in #785
• Use codecov action now by @lgray in #791
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #787
• update awkward, dask_awkward, dask_histogram pins by @lgray in #792
• Use uproot chunking for uproot.dask, repin uproot by @lgray in #793
• Pin dask < 2023.4.0 while dask_awkward isn't compatible by @lgray in #794
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #799
• pins for coffea 2023.4.0rc3 by @lgray in #796
• Anticipate form.length_zero_array() highlevel deprecation by @lgray in #801
• Check for Weight.partial_weight() by @rkansal47 in #803
• build(deps): bump pypa/gh-action-pypi-publish from 1.8.5 to 1.8.6 by @dependabot in #804
• next coffea will be 2023.5.0.rc0 by @lgray in #802
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #805
• Repin for ak + dak + dhi by @lgray in #806
• The next coffea release will be 2023.5.0.rc1 by @lgray in #808
• fix: Allow lookup_base args to be in any order by @lgray in #811
• Use maintained action for conda and graphviz by @lgray in #812
• chore: remove ci based release by @lgray in #813
• chore: Update CONTRIBUTING.md to indicate calver, release process by @lgray in #815
• feat: pr labeller by @lgray in #816
• fix: use correct secret name by @lgray in #817
• fix(ci): no need to run outdated PR hashes by @lgray in #818
• build: dask-awkward/histogram 2023.5.1 by @lgray in #820
• feat: added rapidity getter in coffea/nanoevents/methods/vector.py by @raeubaen in #825
• build: move to pyproject style setup by @lgray in #819
• docs: Added example code for correctionlib_wrapper by @yimuchen in #790
• build: use hatch to build release instead of setup.py by @lgray in #828
• chore: update to empty_if_typetracer by @lgray in #821
• ci(pre-commit.ci): pre-commit autoupdate by @pre-commit-ci in #831
• fix: Added proper behavior for PFNanoAODSchema by @yimuchen in #832
• fix: serialize dask_awkward.Array to None and store hard ref in closure in behavior by @lgray in #824
• refactor: use length_zero/one_if_typetracer by @lgray in #834
• build: repin dask_awkward >= 2023.6.1 by @lgray in #836
• fix: PFnano format for dask by @yimuchen in #835
• feat: ML inference tools by @yimuchen in #798
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #838
• fix: remove now-incorrect documentation from ml_tools by @lgray in #839
• feat: Pass list of files to NanoEventsFactory by @chrispap95 in #837
• build: pin to awkward 2.2.3, dask-histogram 2023.6.0 by @lgray in #840
• build(pre-commit.ci): pre-commit autoupdate by @pre-commit-ci in #841
• ci: get CI jobs completing again by @lgray in #842
• fix: adjustments to callable wrap to deal with typetracers in nested python structures by @lgray in #843
• ci: don't use url in test_fix823 by @lgray in #844
• chore: remove references to old awkward1 classes by @lgray in #846
• docs: update readme versions by @lgray in #847
• feat: New features to PackedSelection by @iasonkrom in #797
• ci: Bump pypa/gh-action-pypi-publish from 1.8.6 to 1.8.7 by @dependabot in #849
• build: pin awkward up to 2.2.4 by @lgray in #850
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #851
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #864
• Bump pypa/gh-action-pypi-publish from 1.8.7 to 1.8.8 by @dependabot in #866
• build: pin adjustments by @lgray in #852
• docs: Create CITATION.cff by @nsmith- in #853
• chore: coffea uses pyproject.toml now update in readme by @lgray in #868
• chore: Update README.rst with citation by @nsmith- in #869
• build: ignore dask-awkward 2023.8.0 by @lgray in #873
• ci: Bump pypa/gh-action-pypi-publish from 1.8.8 to 1.8.10 by @dependabot in #877
• fix: Make LumiMask compatible with dask-awkward by @jrueb in #879
• build: bump awkward pin to >= 2.3.3 by @lgray in #880
• feat: Dataset querying features using rucio by @valsdav in #883
• feat: dataset discovery CLI by @valsdav in #884
• ci: Bump actions/checkout from 3 to 4 by @dependabot in #889
• build: repin to latest fixed awkward by @lgray in #890
• feat: also pass uproot_options to uproot.dask in from_root() by @iasonkrom in #887
• fix: Daskify Elementlinks in PHYSLITE schema by @nikoladze in #872
• fix: allow for collections that contain non-jagged arrays in PHYSLITE schema by @nikoladze in #888
• ci: Bump crazy-max/ghaction-github-pages from 3 to 4 by @dependabot in #893
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #894
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #896
• fix: comply with boost-histogram 1.4.0 by @iasonkrom in #895
• ci: Bump amannn/action-semantic-pull-request from 5.2.0 to 5.3.0 by @dependabot in #897
• fix: Make rochester_lookup compatible with dask_awkward by @jrueb in #875
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #898
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #905
• feat: support shape_touched from Dask by @agoose77 in #900
• fix: added hasattr check to NanoAODEvents repr by @pvisco...

v2023.12.0.rc1

What's Changed

• feat: Dataset querying features using rucio by @valsdav in #883
• feat: dataset discovery CLI by @valsdav in #884
• build: depend on fsspec-xrootd for convenience by @lgray in #955
• fix: re-enable steps_per_file, warn about inefficiencies by @lgray in #956
• build(deps): pin to dask-awkward with reports by @lgray in #957
• fix: use awkward.mask instead of some_array.mask by @lgray in #959
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #958
• fix(ci): skip broken test in parsl by @lgray in #960
• fix: improvements to dataset_query tools by @valsdav in #940
• feat!: switch over to dask-based processing idioms, improve dataset handling by @lgray in #882

Full Changelog: v2023.12.0.rc0...v2023.12.0.rc1

v2023.12.0.rc0

What's Changed

• build: unpin numpy from above, pin numba >= 0.58.1 by @lgray in #913
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #914
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #919
• ci: Bump amannn/action-semantic-pull-request from 5.3.0 to 5.4.0 by @dependabot in #920
• build: pin up to awkward 2.4.8, dask-awkward 2023.11.0 by @lgray in #921
• feat: switch permit_dask to True by default by @lgray in #922
• fix: remove all eager len calls by @iasonkrom in #928
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #927
• feat: add public delta_r and delta_phi kernels for user-specified phis and etas by @iasonkrom in #936
• ci: Bump conda-incubator/setup-miniconda from 2 to 3 by @dependabot in #938
• ci: Bump pypa/gh-action-pypi-publish from 1.8.10 to 1.8.11 by @dependabot in #941
• feat: Handle LumiData files that contain average instantaneous luminosity. by @lgray in #942
• ci: pin on major version for actions by @lgray in #943
• ci: pre-commit only has 3.0.0 by @lgray in #944
• feat: TreeMakerSchema Skimming (r2023) by @yimuchen in #935
• ci: Bump actions/setup-java from 3 to 4 by @dependabot in #946
• feat: use new dask methods by @agoose77 in #926
• fix: adjust dtypes for constants by @lgray in #948
• feat!: drop use of opt_touch_all=False by @agoose77 in #929
• chore: make dependabot semantic-titles friendly by @lgray in #947
• chore: fix gitter link by @lgray in #950
• fix: make .metric_table() and .nearest() work when return_combinations and return_metric are set to True respectively by @iasonkrom in #925
• fix: dispatch of _apply_global_index wasn't re-implemented correctly by @lgray in #951
• feat: adapt to dask from_map report feature by @lgray in #945
• test: taskvine+coffea+dask with pytest by @btovar in #952
• ci: bump actions/setup-python from 4 to 5 by @dependabot in #953
• fix: remove copyreg workaround, use attrs to store reference to original array by @lgray in #949
• fix(ci): only full version for publish by @lgray in #954

Full Changelog: v2023.10.0.rc1...v2023.12.0.rc0

v2023.10.0.rc1

What's Changed

• ci: remove unnecessary python version warnings, reactivate nanoevents tests by @lgray in #912
• fix: test that demoted version of missing event id actually tries to extract from file by @lgray in #911

Full Changelog: v2023.10.0.rc0...v2023.10.0.rc1

v2023.10.0.rc0

What's Changed

• docs: Create CITATION.cff by @nsmith- in #853
• chore: coffea uses pyproject.toml now update in readme by @lgray in #868
• chore: Update README.rst with citation by @nsmith- in #869
• build: ignore dask-awkward 2023.8.0 by @lgray in #873
• ci: Bump pypa/gh-action-pypi-publish from 1.8.8 to 1.8.10 by @dependabot in #877
• fix: Make LumiMask compatible with dask-awkward by @jrueb in #879
• build: bump awkward pin to >= 2.3.3 by @lgray in #880
• ci: Bump actions/checkout from 3 to 4 by @dependabot in #889
• build: repin to latest fixed awkward by @lgray in #890
• feat: also pass uproot_options to uproot.dask in from_root() by @iasonkrom in #887
• fix: Daskify Elementlinks in PHYSLITE schema by @nikoladze in #872
• fix: allow for collections that contain non-jagged arrays in PHYSLITE schema by @nikoladze in #888
• ci: Bump crazy-max/ghaction-github-pages from 3 to 4 by @dependabot in #893
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #894
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #896
• fix: comply with boost-histogram 1.4.0 by @iasonkrom in #895
• ci: Bump amannn/action-semantic-pull-request from 5.2.0 to 5.3.0 by @dependabot in #897
• fix: Make rochester_lookup compatible with dask_awkward by @jrueb in #875
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #898
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #905
• feat: support shape_touched from Dask by @agoose77 in #900
• fix: added hasattr check to NanoAODEvents repr by @pviscone in #909
• fix: remove binder setup problems in coffea 2023 by @lgray in #908

New Contributors

• @agoose77 made their first contribution in #900
• @pviscone made their first contribution in #909

Full Changelog: v2023.7.0.rc0...v2023.10.0.rc0

v0.7.22

What's Changed

• build(deps): bump pypa/gh-action-pypi-publish from 1.7.1 to 1.8.1 by @dependabot in #774
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #781
• build(deps): bump pypa/gh-action-pypi-publish from 1.8.1 to 1.8.4 by @dependabot in #780
• fix: remove use of deprecated numpy.bool by @lgray in #901
• fix: remove strange definition of base schema behavior by @lgray in #902
• fix: linting error in tests by @lgray in #903

Full Changelog: v0.7.21...v0.7.22

v2023.7.0.rc0

What's Changed

• ci: Bump pypa/gh-action-pypi-publish from 1.8.6 to 1.8.7 by @dependabot in #849
• build: pin awkward up to 2.2.4 by @lgray in #850
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #851
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci in #864
• Bump pypa/gh-action-pypi-publish from 1.8.7 to 1.8.8 by @dependabot in #866
• build: pin adjustments by @lgray in #852

Full Changelog: v2023.6.0.rc1...v2023.7.0.rc0

v2023.6.0.rc1

What's Changed

• chore: update to empty_if_typetracer by @lgray in #821
• ci(pre-commit.ci): pre-commit autoupdate by @pre-commit-ci in #831
• fix: Added proper behavior for PFNanoAODSchema by @yimuchen in #832
• fix: serialize dask_awkward.Array to None and store hard ref in closure in behavior by @lgray in #824
• refactor: use length_zero/one_if_typetracer by @lgray in #834
• build: repin dask_awkward >= 2023.6.1 by @lgray in #836
• fix: PFnano format for dask by @yimuchen in #835
• feat: ML inference tools by @yimuchen in #798
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #838
• fix: remove now-incorrect documentation from ml_tools by @lgray in #839
• feat: Pass list of files to NanoEventsFactory by @chrispap95 in #837
• build: pin to awkward 2.2.3, dask-histogram 2023.6.0 by @lgray in #840
• build(pre-commit.ci): pre-commit autoupdate by @pre-commit-ci in #841
• ci: get CI jobs completing again by @lgray in #842
• fix: adjustments to callable wrap to deal with typetracers in nested python structures by @lgray in #843
• ci: don't use url in test_fix823 by @lgray in #844
• chore: remove references to old awkward1 classes by @lgray in #846
• docs: update readme versions by @lgray in #847
• feat: New features to PackedSelection by @iasonkrom in #797

New Contributors

• @chrispap95 made their first contribution in #837
• @iasonkrom made their first contribution in #797

Full Changelog: v2023.6.0.rc0...v2023.6.0.rc1

v2023.6.0.rc0

What's Changed

• The next coffea release will be 2023.5.0.rc1 by @lgray in #808
• fix: Allow lookup_base args to be in any order by @lgray in #811
• Use maintained action for conda and graphviz by @lgray in #812
• chore: remove ci based release by @lgray in #813
• chore: Update CONTRIBUTING.md to indicate calver, release process by @lgray in #815
• feat: pr labeller by @lgray in #816
• fix: use correct secret name by @lgray in #817
• fix(ci): no need to run outdated PR hashes by @lgray in #818
• build: dask-awkward/histogram 2023.5.1 by @lgray in #820
• feat: added rapidity getter in coffea/nanoevents/methods/vector.py by @raeubaen in #825
• build: move to pyproject style setup by @lgray in #819
• docs: Added example code for correctionlib_wrapper by @yimuchen in #790
• build: use hatch to build release instead of setup.py by @lgray in #828

New Contributors

• @raeubaen made their first contribution in #825

Full Changelog: v2023.5.0.rc0...v2023.6.0.rc0

Release v2023.5.0.rc0

 - Merge pull request #806 from CoffeaTeam/pin-skooch

 - Merge pull request #805 from CoffeaTeam/pre-commit-ci-update-config
 - Merge pull request #802 from CoffeaTeam/calver-update
 - Merge pull request #804 from CoffeaTeam/dependabot/github_actions/pypa/gh-action-pypi-publish-1.8.6
 - Merge pull request #803 from rkansal47/weights_check
 - Merge pull request #801 from CoffeaTeam/remove_zla_highlevel