[Hardware][CPU] Add ARM CPU backend

[ShawnD200]

Add ARM CPU backend support

(1) Based on the groundbreaking x86 CPU backend support, implemented the vector types on ARM neon
(2) Added tested cases for explicit implementation verification

# cd tests/cpu
# pip install .
# pytest . -v

Tested on MacOS with M2 Pro chip with docker image build, should work on generic ARM64/Linux.

Future work:

• Built from source on bare-metal MacOS
• Improve performance with advanced SIMD, and a lot other stuff
• Test on ARM/Linux and with more models

Tested simple models, just works. Have fun. Feedback is welcome.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ShawnD200.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[Isotr0py]

Thanks for implementing arm64 support! The cpu kernel tests look pretty good! Just have some questions.

(I don't have an arm64 device. See if others having arm64 can run the test on native Linux)

[Isotr0py]

Why can't we use torch 2.5.0 or higher here? PyTorch have aarch64 distribution for 2.5.1: https://download.pytorch.org/whl/cpu/torch-2.5.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl#sha256=269b10c34430aa8e9643dbe035dc525c4a9b1d671cd3dbc8ecbcaed280ae322d

[ShawnD200]

Honestly I haven't checked through the versions so that I don't know which is supported or not. This torch version is just copied from the x86 CPU backend. This could be future work.

[Isotr0py]

Since FP16 is not supported on arm, we might need to add a fallback to BF16 for arm CPU if user specified dtype="float16". (Just like Intel CPU before)

[ShawnD200]

Yes, good point. X86 already added the switch case for Half type, ARM needs to add it too. This brings to a general question: how to automatically serve all ISA's, as a starter, those data types should be interfaces. I will add this to future work too. Thank you so much.

[ShawnD200]

Pushed a few updates:

Inspired by @Isotr0py, I realized there is tremendous complexity induced by combination of hardware supports and model inference features. Therefore, I think maybe it requires a generic layer that can abstract away the underlying hardware differences. To that end, here comes CPU types interfaces which can be implemented by various ISAs and features within ISA. I haven't sorted many things out; your feedback is so desired.

BTW, before @mgoin introduced #9228 to me, which already implemented ARM CPU backend basically the same way, I had absolutely no idea of that PR. I started this work with a very simple goal that is to run vLLM on my macbook. Kudos to @sanketkaleoss, I learnt a lot from your work. Thanks.

[ShawnD200]

@mgoin @bigPYJ1151 @tlrmchlsmth @sanketkaleoss @Isotr0py, your feedback would be very helpful. Thanks.