Investigate scipy/Apple Accelerate (for Sparse Solvers)

[michaelweinold]

scipy in December 2018 dropped and in May 2021 re-introduced support for the Apple Accelerate library (compare also the SciPy roadmap and their GitHub repo wiki page).

This library

provides high-performance, energy-efficient computation on the CPU by leveraging its vector-processing capability.

It could therefore provide a highly optimized alternative to the current bw25nosolver workaround (compare the documentation).

Investigation

• how to check if local scipy installation has Accelerate support
• how to install scipy with Accelerate support
• update installation instructions on the Setup documentation page
• investigate if Accelerate includes alternatives to UMFPACK and pypardiso

Implementation

• set up Jupyter Notebook with a benchmarking example from bw2calc

Compare @cmutel's existing notebook "Compare speed of iterative solvers".

Documentation

Installation

• Manual #1
• Manual #2

Sparse Solvers

Discussion on Hackernews

[michaelweinold]

This affects the following sections in the documentation:

LCA > Normal Static LCA

Due to licensing conflicts, recent versions of SciPy do not include UMFpack. UMFpack is faster than SuperLU, especially for repeated calculations. Python wrappers for UMFpack must be installed separately using scikits.umfpack.

Setup > macOS (Apple Silicon/ARM)

Brightway runs on the new Apple Silicon ARM architecture. However, the super-fast linear algebra software library pypardiso is not compatible with the ARM processor architecture. To avoid critical errors during instruction that would break core functionality, a different version of Brightway (brightway_nosolver) must be installed, which includes a different linear algebra software library (scikit-umfpack).

[cmutel]

The documentation for Accelerate sparse matrices is here.

I also added a warning to let people know if they don't have a fast LA library.

Given that using Accelerate would mean a lot of development work on a platform with has scikit-umfpack already, this is a low priority.

[michaelweinold]

Numpy 2.0.0 now (again) has Apple Accelerate support out-of-the-box.

So I think this may not be the case:

Given that using Accelerate would mean a lot of development work on a platform with has scikit-umfpack already, this is a low priority.

And this might potentially be interesting for:

• Matrix Inversion (Solution of Linear System) for Dense Hybrid Matrices bw_hybrid#45

I'll try to check if I can compare performance on Apple Silicon and post the results here.

Aside from performance, we could further simplify/unify the installation instructions.

[cmutel]

Sure, I saw this too. We are currently pinning stuff to numpy<2 because we don't have enough testing to see if it works, but this looks promising.

[michaelweinold]

Having done some testing, I quickly realized that the actual magic happens in the sp.sparse.linalg.spsolve(technosphere_matrix, demand_array) function, which is called by bw2calc.lca_base.LCABase.solve_linear_system, which is called by bw2calc.lca.LCA.lci_calculation.

Unfortunately, the SciPy documentation (1.13.X) mentions that:

A recent version of OpenBLAS, MKL or ATLAS. The Accelerate BLAS library is no longer supported.
- Scipy Toolchain Roadmap

An archived SciPy GitHub-wiki entry details the reasoning for this missing support.

There are some efforts:

• Slack discussion
• Add support for BLAS and LAPACK dependencies mesonbuild/meson#10921

However, it looks like this might still take a while 😢...

[michaelweinold]

Ok, before @romainsacchi posts his findings here:

For some reason I did not find this previously: SciPy from 1.14.0 DOES now have Apple Accelerate support.

Installing SciPy such that is is actually built against Accelerate is not yet straightforward. Conda failed to do so when I tried... I had better luck with just pip (yeah, yeah, @cmutel 😜).

I found performance did improve for some operations - a quick testing notebook is on this branch:

• https://github.com/brightway-lca/brightway2-calc/tree/numpy_2_apple_silicon_testing

[cmutel]

a quick testing notebook is on this branch

Is it though?