Rates matrix solver

[andrewfullard]

📝 Description

Type: 🚀 feature

Initial plans for replacing the old "NLTE" setup with a hopefully cleaner structure.

🚦 Testing

How did you test these changes?

• Testing pipeline
• Other method (describe)
• My changes can't be tested (explain why)

☑️ Checklist

• I requested two reviewers for this pull request
• I updated the documentation according to my changes
• I built the documentation by applying the build_docs label

Note: If you are not allowed to perform any of these actions, ping (@) a contributor.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[tardis-bot]

*beep* *bop*
Hi human,
I ran ruff on the latest commit (4bd4206).
Here are the outputs produced.
Results can also be downloaded as artifacts here.
Summarised output:

21	E999  	[ ] SyntaxError: Expected ',', found ':'
8	W293  	[*] Blank line contains whitespace
7	F401  	[*] `astropy.constants` imported but unused
5	RET505	[ ] Unnecessary `else` after `return` statement
3	G004  	[ ] Logging statement uses f-string
2	E902  	[ ] No such file or directory (os error 2)
1	ICN001	[*] `matplotlib` should be imported as `mpl`
1	G001  	[ ] Logging statement uses `str.format`
1	RET506	[ ] Unnecessary `else` after `raise` statement
1	I001  	[*] Import block is un-sorted or un-formatted
1	E701  	[ ] Multiple statements on one line (colon)
1	W291  	[*] Trailing whitespace
1	W292  	[*] No newline at end of file
1	F811  	[ ] Redefinition of unused `opacity_state_initialize` from line 13
1	F821  	[ ] Undefined name `tardis_githash`

Complete output(might be large):

.github/workflows/build-docs.yml:7:4: E999 SyntaxError: Expected an expression
.github/workflows/post-release.yml:7:4: E999 SyntaxError: Expected an expression
.github/workflows/post-release.yml:124:37: W291 [*] Trailing whitespace
.github/workflows/pre-release.yml:7:4: E999 SyntaxError: Expected an expression
.github/workflows/release.yml:7:4: E999 SyntaxError: Expected an expression
.github/workflows/release.yml:16:7: E701 Multiple statements on one line (colon)
.github/workflows/release.yml:48:1: W293 [*] Blank line contains whitespace
.github/workflows/release.yml:51:1: W293 [*] Blank line contains whitespace
.github/workflows/release.yml:58:1: W293 [*] Blank line contains whitespace
.github/workflows/release.yml:61:1: W293 [*] Blank line contains whitespace
.github/workflows/release.yml:64:1: W293 [*] Blank line contains whitespace
.github/workflows/release.yml:69:1: W293 [*] Blank line contains whitespace
.github/workflows/release.yml:73:1: W293 [*] Blank line contains whitespace
.github/workflows/release.yml:89:1: W293 [*] Blank line contains whitespace
.github/workflows/release.yml:97:23: W292 [*] No newline at end of file
.mailmap:1:38: E999 SyntaxError: Expected an expression
CHANGELOG.md:4:15: E999 SyntaxError: Expected ',', found ':'
CITATION.cff:3:1: E999 SyntaxError: Invalid annotated assignment target
README.rst:1:1: E999 SyntaxError: Expected a statement
docs/physics/plasma/equilibrium/collisional_rate_coefficients.hdf:1:1: E902 stream did not contain valid UTF-8
docs/physics/plasma/equilibrium/comparison.ipynb:cell 31:1:1: I001 [*] Import block is un-sorted or un-formatted
docs/physics/plasma/equilibrium/tardis_solver.ipynb:cell 11:1:29: F401 [*] `astropy.constants` imported but unused
docs/physics/plasma/equilibrium/test_continuum_template_wkerzen_rate_coeffs.yml:2:26: E999 SyntaxError: Simple statements must be separated by newlines or semicolons
docs/resources/credits.rst:1:1: E999 SyntaxError: Expected a statement
tardis/io/atom_data/base.py:197:34: G004 Logging statement uses f-string
tardis/io/atom_data/base.py:259:17: G004 Logging statement uses f-string
tardis/io/atom_data/base.py:263:21: G001 Logging statement uses `str.format`
tardis/io/atom_data/base.py:702:17: G004 Logging statement uses f-string
tardis/plasma/detailed_balance/rates/__init__.py:1:1: E902 No such file or directory (os error 2)
tardis/plasma/electron_energy_distribution/__init__.py:2:5: F401 `tardis.plasma.electron_energy_distribution.base.ElectronEnergyDistribution` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
tardis/plasma/electron_energy_distribution/__init__.py:3:5: F401 `tardis.plasma.electron_energy_distribution.base.ThermalElectronEnergyDistribution` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
tardis/plasma/equilibrium/rates/__init__.py:2:5: F401 `tardis.plasma.equilibrium.rates.collision_strengths.UpsilonCMFGENSolver` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
tardis/plasma/equilibrium/rates/__init__.py:3:5: F401 `tardis.plasma.equilibrium.rates.collision_strengths.UpsilonRegemorterSolver` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
tardis/plasma/equilibrium/rates/__init__.py:6:5: F401 `tardis.plasma.equilibrium.rates.collisional_rates.ThermalCollisionalRateSolver` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
tardis/plasma/equilibrium/rates/__init__.py:9:5: F401 `tardis.plasma.equilibrium.rates.radiative_rates.RadiativeRatesSolver` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
tardis/spectrum/formal_integral.py:23:5: F811 Redefinition of unused `opacity_state_initialize` from line 13
tardis/spectrum/formal_integral.py:340:13: RET506 Unnecessary `else` after `raise` statement
tardis/spectrum/formal_integral.py:681:5: RET505 Unnecessary `else` after `return` statement
tardis/spectrum/formal_integral.py:717:5: RET505 Unnecessary `else` after `return` statement
tardis/spectrum/formal_integral_cuda.py:375:5: RET505 Unnecessary `else` after `return` statement
tardis/spectrum/formal_integral_cuda.py:420:5: RET505 Unnecessary `else` after `return` statement
tardis/tests/integration_tests/at/abundances.dat:2:1: E999 SyntaxError: Unexpected indentation
tardis/tests/integration_tests/at/config.yml:2:7: E999 SyntaxError: Expected an expression
tardis/tests/integration_tests/at/densities.dat:1:10: E999 SyntaxError: Simple statements must be separated by newlines or semicolons
tardis/tests/integration_tests/conftest.py:43:13: F821 Undefined name `tardis_githash`
tardis/tests/integration_tests/conftest.py:88:5: RET505 Unnecessary `else` after `return` statement
tardis/tests/integration_tests/paper1_downbranch/config.yml:7:4: E999 SyntaxError: Expected an expression
tardis/tests/integration_tests/paper1_macroatom/abundances.dat:1:1: E999 SyntaxError: Unexpected indentation
tardis/tests/integration_tests/paper1_macroatom/config.yml:7:4: E999 SyntaxError: Expected an expression
tardis/tests/integration_tests/paper1_macroatom/densities.dat:1:1: E999 SyntaxError: Unexpected indentation
tardis/tests/integration_tests/paper1_scatter/config.yml:7:4: E999 SyntaxError: Expected an expression
tardis/tests/integration_tests/test_integration.py:3:8: ICN001 `matplotlib` should be imported as `mpl`
tardis/tests/integration_tests/w7/abundances.dat:2:1: E999 SyntaxError: Unexpected indentation
tardis/tests/integration_tests/w7/config.yml:2:7: E999 SyntaxError: Expected an expression
tardis/tests/integration_tests/w7/densities.dat:1:11: E999 SyntaxError: Simple statements must be separated by newlines or semicolons
Found 55 errors.
[*] 12 fixable with the `--fix` option (1 hidden fix can be enabled with the `--unsafe-fixes` option).

[tardis-bot]

*beep* *bop*

Hi, human.

The docs workflow has failed ❌

Click here to see the build log.

[tardis-bot]

*beep* *bop*
Hi human,
I ran benchmarks as you asked comparing master (a1f621f) and the latest commit (4bd4206).
Here are the logs produced by ASV.
Results can also be downloaded as artifacts here.

Significantly changed benchmarks:

All benchmarks:

Benchmarks that have stayed the same:

| Change   | Before [6ee870e0] <master>   | After [4bd4206c]    | Ratio   | Benchmark (Parameter)                                                                                                               |
|----------|------------------------------|---------------------|---------|-------------------------------------------------------------------------------------------------------------------------------------|
|          | 7.07±2μs                     | 9.50±0.3μs          | ~1.34   | transport_montecarlo_vpacket.BenchmarkMontecarloMontecarloNumbaVpacket.time_trace_vpacket_volley                                    |
|          | 45.0±30μs                    | 52.9±30μs           | ~1.17   | transport_montecarlo_interaction.BenchmarkTransportMontecarloInteraction.time_line_scatter                                          |
|          | 3.12±1μs                     | 1.94±0.9μs          | ~0.62   | transport_montecarlo_estimators_radfield_estimator_calcs.BenchmarkMontecarloMontecarloNumbaPacket.time_update_line_estimators       |
|          | 4.99±1μs                     | 2.54±0.7μs          | ~0.51   | transport_montecarlo_vpacket.BenchmarkMontecarloMontecarloNumbaVpacket.time_trace_bad_vpacket                                       |
|          | 1.45±0.4μs                   | 1.59±0.3μs          | 1.10    | transport_geometry_calculate_distances.BenchmarkTransportGeometryCalculateDistances.time_calculate_distance_line                    |
|          | 5.60±1μs                     | 6.08±1μs            | 1.09    | transport_montecarlo_vpacket.BenchmarkMontecarloMontecarloNumbaVpacket.time_trace_vpacket                                           |
|          | 501±100ns                    | 541±100ns           | 1.08    | opacities_opacity.BenchmarkMontecarloMontecarloNumbaOpacities.time_compton_opacity_calculation                                      |
|          | 1.17±0μs                     | 1.24±0μs            | 1.06    | transport_geometry_calculate_distances.BenchmarkTransportGeometryCalculateDistances.time_calculate_distance_boundary                |
|          | 601±100ns                    | 631±100ns           | 1.05    | opacities_opacity.BenchmarkMontecarloMontecarloNumbaOpacities.time_photoabsorption_opacity_calculation                              |
|          | 62.6±0.3ms                   | 65.8±0.3ms          | 1.05    | transport_montecarlo_packet_trackers.BenchmarkTransportMontecarloPacketTrackers.time_rpacket_trackers_to_dataframe                  |
|          | 621±200ns                    | 641±100ns           | 1.03    | opacities_opacity.BenchmarkMontecarloMontecarloNumbaOpacities.time_pair_creation_opacity_calculation                                |
|          | 40.9±20μs                    | 41.7±20μs           | 1.02    | transport_montecarlo_interaction.BenchmarkTransportMontecarloInteraction.time_line_emission                                         |
|          | 1.64±0ms                     | 1.66±0.02ms         | 1.01    | transport_montecarlo_main_loop.BenchmarkTransportMontecarloMontecarloMainLoop.time_montecarlo_main_loop                             |
|          | 1.05±0m                      | 1.06±0m             | 1.00    | run_tardis.BenchmarkRunTardis.time_run_tardis_rpacket_tracking                                                                      |
|          | 2.08±0m                      | 2.07±0m             | 1.00    | spectrum_formal_integral.BenchmarkTransportMontecarloFormalIntegral.time_FormalIntegrator_functions                                 |
|          | 206±0.1ns                    | 206±0.2ns           | 1.00    | spectrum_formal_integral.BenchmarkTransportMontecarloFormalIntegral.time_intensity_black_body                                       |
|          | 30.8±0.03μs                  | 30.9±0.01μs         | 1.00    | transport_montecarlo_packet_trackers.BenchmarkTransportMontecarloPacketTrackers.time_generate_rpacket_tracker_list                  |
|          | 3.23±0.6μs                   | 3.23±0.5μs          | 1.00    | transport_montecarlo_vpacket.BenchmarkMontecarloMontecarloNumbaVpacket.time_trace_vpacket_within_shell                              |
|          | 40.0±0.05s                   | 39.8±0.06s          | 0.99    | run_tardis.BenchmarkRunTardis.time_run_tardis                                                                                       |
|          | 2.49±0.4ms                   | 2.47±0.4ms          | 0.99    | transport_montecarlo_single_packet_loop.BenchmarkTransportMontecarloSinglePacketLoop.time_single_packet_loop                        |
|          | 3.52±0.02ms                  | 3.44±0.01ms         | 0.98    | opacities_opacity_state.BenchmarkOpacitiesOpacityState.time_opacity_state_initialize('macroatom')                                   |
|          | 758±2ns                      | 725±1ns             | 0.96    | transport_montecarlo_interaction.BenchmarkTransportMontecarloInteraction.time_thomson_scatter                                       |
|          | 23.1±6μs                     | 21.8±5μs            | 0.94    | transport_montecarlo_packet_trackers.BenchmarkTransportMontecarloPacketTrackers.time_generate_rpacket_last_interaction_tracker_list |
|          | 2.73±0.01ms                  | 2.50±0.01ms         | 0.92    | opacities_opacity_state.BenchmarkOpacitiesOpacityState.time_opacity_state_initialize('scatter')                                     |

If you want to see the graph of the results, you can check it here

[codecov]

Codecov Report

Attention: Patch coverage is 95.58824% with 6 lines in your changes missing coverage. Please review.

Project coverage is 70.18%. Comparing base (a1f621f) to head (4bd4206).
Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
tardis/io/atom_data/base.py	78.94%	4 Missing ⚠️
...rdis/plasma/equilibrium/rates/collisional_rates.py	80.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2865      +/-   ##
==========================================
+ Coverage   69.97%   70.18%   +0.20%     
==========================================
  Files         216      219       +3     
  Lines       16077    16094      +17     
==========================================
+ Hits        11250    11295      +45     
+ Misses       4827     4799      -28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[wkerzendorf]

adding more tests to get coverage up probably

[andrewfullard]

Note to self: make sure the tests pull the extra atom data file from the regression data repo

[wkerzendorf]

I think this should be .h5