Performance of llama.cpp on Apple Silicon M-series

[ggerganov]

Summary

LLaMA 7B

	BW [GB/s]	GPU Cores	F16 PP [t/s]	F16 TG [t/s]	Q8_0 PP [t/s]	Q8_0 TG [t/s]	Q4_0 PP [t/s]	Q4_0 TG [t/s]
✅ M1 1	68	7			108.21	7.92	107.81	14.19
✅ M1 1	68	8			117.25	7.91	117.96	14.15
✅ M1 Pro 1	200	14	262.65	12.75	235.16	21.95	232.55	35.52
✅ M1 Pro 1	200	16	302.14	12.75	270.37	22.34	266.25	36.41
✅ M1 Max 1	400	24	453.03	22.55	405.87	37.81	400.26	54.61
✅ M1 Max 1	400	32	599.53	23.03	537.37	40.2	530.06	61.19
✅ M1 Ultra 1	800	48	875.81	33.92	783.45	55.69	772.24	74.93
✅ M1 Ultra 1	800	64	1168.89	37.01	1042.95	59.87	1030.04	83.73
✅ M2 2	100	8			147.27	12.18	145.91	21.7
✅ M2 2	100	10	201.34	6.72	181.4	12.21	179.57	21.91
✅ M2 Pro 2	200	16	312.65	12.47	288.46	22.7	294.24	37.87
✅ M2 Pro 2	200	19	384.38	13.06	344.5	23.01	341.19	38.86
✅ M2 Max 2	400	30	600.46	24.16	540.15	39.97	537.6	60.99
✅ M2 Max 2	400	38	755.67	24.65	677.91	41.83	671.31	65.95
✅ M2 Ultra 2	800	60	1128.59	39.86	1003.16	62.14	1013.81	88.64
✅ M2 Ultra 2	800	76	1401.85	41.02	1248.59	66.64	1238.48	94.27
🟥 M3 3	100	8
🟨 M3 3	100	10			187.52	12.27	186.75	21.34
🟨 M3 Pro 3	150	14			272.11	17.44	269.49	30.65
✅ M3 Pro 3	150	18	357.45	9.89	344.66	17.53	341.67	30.74
✅ M3 Max 3	300	30	589.41	19.54	566.4	34.3	567.59	56.58
✅ M3 Max 3	400	40	779.17	25.09	757.64	42.75	759.7	66.31
❌ M3 Ultra
🟥 M4 4	120	8
✅ M4 4	120	10	230.18	7.43	223.64	13.54	221.29	24.11
🟥 M4 Pro 4	273	16
✅ M4 Pro 4	273	20	464.48	17.18	449.62	30.69	439.78	50.74
🟥 M4 Max 4	410	32
✅ M4 Max 4	546	40	922.83	31.64	891.94	54.05	885.68	83.06
🟥 M4 Ultra	820	64
🟥 M4 Ultra	1092	80

plot.py

# GPT-4 Generated Code

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Creating DataFrame from the provided data
data = {
    "Chip": ["M1", "M1", "M1 Pro", "M1 Pro", "M1 Max", "M1 Max", "M1 Ultra", "M2", "M2 Pro", "M2 Pro", "M2 Max", "M2 Max", "M2 Ultra", "M2 Ultra", "M3", "M3 Pro", "M3 Pro", "M3 Max"],
    "BW (GB/s)":     [68, 68, 200, 200, 400, 400, 800, 100, 200, 200, 400, 400, 800, 800, 100, 150, 150, 400],
    "GPU Cores":     [7, 8, 14, 16, 24, 32, 48, 10, 16, 19, 30, 38, 60, 76, 10, 14, 18, 40],
    "F16 PP (t/s)":  [None, None, None, 302.14, 453.03, 599.53, 875.81, 201.34, 312.65, 384.38, 600.46, 755.67, 1128.59, 1401.85, None, None, 357.45, 779.17],
    "F16 TG (t/s)":  [None, None, None, 12.75, 22.55, 23.03, 33.92, 6.72, 12.47, 13.06, 24.16, 24.65, 39.86, 41.02, None, None, 9.89, 25.09],
    "Q8_0 PP (t/s)": [108.21, 117.25, 235.16, 270.37, 405.87, 537.37, 783.45, 181.4, 288.46, 344.5, 540.15, 677.91, 1003.16, 1248.59, 187.52, 272.11, 344.66, 757.64],
    "Q8_0 TG (t/s)": [7.92, 7.91, 21.95, 22.34, 37.81, 40.2, 55.69, 12.21, 22.7, 23.01, 39.97, 41.83, 62.14, 66.64, 12.27, 17.44, 17.53, 42.75],
    "Q4_0 PP (t/s)": [107.81, 117.96, 232.55, 266.25, 400.26, 530.06, 772.24, 179.57, 294.24, 341.19, 537.6, 671.31, 1013.81, 1238.48, 186.75, 269.49, 341.67, 759.7],
    "Q4_0 TG (t/s)": [14.19, 14.15, 35.52, 36.41, 54.61, 61.19, 74.93, 21.91, 37.87, 38.86, 60.99, 65.95, 88.64, 94.27, 21.34, 30.65, 30.74, 66.31]
}
df = pd.DataFrame(data)

# Helper function to plot and annotate multiple data series in the same plot
def plot_multi_series(ax, x, y_series, labels, xlabel, ylabel, title, poly_power=1):
    colors = ['r', 'g', 'b']  # Colors for different series
    for i, y in enumerate(y_series):
        # Sorting data for regression
        sorted_indices = np.argsort(x)
        x_sorted = x[sorted_indices]
        y_sorted = y[sorted_indices]

        # Masking NaN values
        mask = ~np.isnan(y_sorted)
        x_sorted = x_sorted[mask]
        y_sorted = y_sorted[mask]

        # Fitting a polynomial regression model
        coefficients = np.polyfit(x_sorted, y_sorted, poly_power)
        polynomial = np.poly1d(coefficients)

        # Creating a range of x-values for a smoother trendline
        x_range = np.linspace(x_sorted.min(), x_sorted.max(), 500)
        trendline = polynomial(x_range)

        # Plotting
        ax.scatter(x, y, color=colors[i], label=labels[i], s=20)
        ax.plot(x_range, trendline, f"{colors[i]}-", linewidth=1)  # Trendline in the same color

    ax.set_xlabel(xlabel)
    ax.set_ylabel(ylabel)
    ax.set_title(title)
    ax.legend()

    # Annotating points with the number of GPU cores and Bandwidth
    for i, txt in enumerate(df["Chip"]):
        ax.annotate(f"{df['GPU Cores'][i]} Cores, {df['BW (GB/s)'][i]} GB/s", (x[i], y_series[0][i]))


# Creating plots for PP vs Cores and TG vs Bandwidth
fig, axs = plt.subplots(1, 2, figsize=(15, 6))
fig.suptitle('PP vs GPU Cores and TG vs Bandwidth for F16, Q8_0, and Q4_0')

# PP vs GPU Cores
y_series_cores_pp = [df["F16 PP (t/s)"], df["Q8_0 PP (t/s)"], df["Q4_0 PP (t/s)"]]
plot_multi_series(axs[0], df["GPU Cores"], y_series_cores_pp,
                  ['F16 PP', 'Q8_0 PP', 'Q4_0 PP'], 'GPU Cores', 'Performance (t/s)',
                  'PP Performance vs GPU Cores', 1)

# TG vs Bandwidth
y_series_bw_tg = [df["F16 TG (t/s)"], df["Q8_0 TG (t/s)"], df["Q4_0 TG (t/s)"]]
plot_multi_series(axs[1], df["BW (GB/s)"], y_series_bw_tg,
                  ['F16 TG', 'Q8_0 TG', 'Q4_0 TG'], 'Bandwidth (GB/s)', 'Performance (t/s)',
                  'TG Performance vs Bandwidth', 2)

plt.tight_layout(rect=[0, 0.03, 1, 0.95])
plt.show()

---

Description

This is a collection of short llama.cpp benchmarks on various Apple Silicon hardware. It can be useful to compare the performance that llama.cpp achieves across the M-series chips and hopefully answer questions of people wondering if they should upgrade or not. Collecting info here just for Apple Silicon for simplicity. Similar collection for A-series chips is available here: #4508

If you are a collaborator to the project and have an Apple Silicon device, please add your device, results and optionally username for the following command directly into this post (requires LLaMA 7B v2):

git checkout 8e672efe
make clean && make -j llama-bench && ./llama-bench \
  -m ./models/llama-7b-v2/ggml-model-f16.gguf  \
  -m ./models/llama-7b-v2/ggml-model-q8_0.gguf \
  -m ./models/llama-7b-v2/ggml-model-q4_0.gguf \
  -p 512 -n 128 -ngl 99 2> /dev/null

• Make sure to run the benchmark on commit 8e672ef
• Please also include the F16 model as shown, not just the quantum models
• Contributors can post the same results in the comments below
• If a device is already benchmarked and your results are comparable, there is no need to add it again
• PP means "prompt processing" (bs = 512), TG means "text-generation" (bs = 1), t/s means "tokens per second"
• ✅ means the data has been added to the summary

Note that in this benchmark we are evaluating the performance against the same build 8e672ef (2023 Nov 13) in order to keep all performance factors even. Since then, there have been multiple improvements resulting in better absolute performance. As an example, here is how the same test compares against the build 86ed72d (2024 Nov 21) on M2 Ultra:

	BW [GB/s]	GPU Cores	F16 PP [t/s]	F16 TG [t/s]	Q8_0 PP [t/s]	Q8_0 TG [t/s]	Q4_0 PP [t/s]	Q4_0 TG [t/s]
M2 Ultra 8e672ef	800	76	1401.85	41.02	1248.59	66.64	1238.48	94.27
M2 Ultra 86ed72d + FA	800	76	1525.95	43.15	1368.18	73.11	1391.78	108.80

---

M1 Pro, 8+2 CPU, 16 GPU (@ggerganov) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	302.14 ± 0.07
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	12.75 ± 0.00
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	270.37 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	22.34 ± 0.00
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	266.25 ± 0.07
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	36.41 ± 0.01

build: 8e672ef (1550)

M2 Ultra, 16+8 CPU, 76 GPU (@ggerganov) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1401.85 ± 1.75
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	41.02 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	1248.59 ± 0.73
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	66.64 ± 0.02
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	1238.48 ± 0.76
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	94.27 ± 0.05

build: 8e672ef (1550)

M3 Max (MBP 14), 12+4 CPU, 40 GPU (@slaren) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	794.26 ± 3.16
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	25.27 ± 0.07
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	749.37 ± 8.35
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	43.00 ± 0.12
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	690.99 ± 33.76
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	65.85 ± 0.22

build: d103d93 (1553)

Footnotes

1. https://en.wikipedia.org/wiki/Apple_M1#Variants ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

2. https://en.wikipedia.org/wiki/Apple_M2#Variants ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

3. https://en.wikipedia.org/wiki/Apple_M3#Variants ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

4. https://en.wikipedia.org/wiki/Apple_M4#Variants ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

[QueryType]

M2 Mac Mini, 4+4 CPU, 10 GPU, 24 GB Memory (@QueryType) ✅

---

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	201.34 ± 0.21
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	6.72 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	181.40 ± 0.05
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	12.21 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	179.57 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	21.91 ± 0.02

build: 8e672ef (1550)

[brozkrut]

M2 Max Studio, 8+4 CPU, 38 GPU ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	755.67 ± 0.11
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	24.65 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	677.91 ± 0.26
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	41.83 ± 0.03
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	671.31 ± 0.20
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	65.95 ± 0.08

build: 8e672ef (1550)

[marcingomulkiewicz]

5800X3D, Linux Mint, kernel 6.2.0-37-generic, NV driver 535.129.03:

build: 8e672ef (1550):

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	pp 512	9905.28 ± 614.92
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	tg 128	60.91 ± 0.17
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	pp 512	6602.12 ± 299.45
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	tg 128	99.90 ± 0.21
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	6927.76 ± 78.51
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	156.47 ± 0.24

newer build basically the same:

build: 33e171d (1604):

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	pp 512	9918.85 ± 597.74
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	tg 128	60.66 ± 0.88
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	pp 512	6616.48 ± 287.31
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	tg 128	100.23 ± 0.42
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	6970.15 ± 38.83
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	157.91 ± 0.36

for comparison, CPU only (1604):

model	size	params	backend	threads	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CPU	8	pp 512	30.69 ± 0.14
llama 7B mostly F16	12.55 GiB	6.74 B	CPU	8	tg 128	2.92 ± 0.00
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CPU	8	pp 512	33.72 ± 0.13
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CPU	8	tg 128	5.40 ± 0.18
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	8	pp 512	28.29 ± 0.11
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	8	tg 128	10.24 ± 0.03

[AndreasKunar]

Thanks a lot. very interesting data and to me in-line with Apple silicon. If you add a GPU FP32 TFLOPS column (pure GPUs is not comparable cross architecture), the PP F16 scales with TFLOPS (FP16 with FP32 accumulate = 165.2 TFLOPS for the 4090), the TG F16 scales with memory-bandwidth (1008 GB/s for 4090). And quantization adds speed due to reduced memory, but requires more compute. What's amazing to me, is that this might enable a good predictor for model-performance based on easily available GPU FP32 TFLOPS + memory-bandwidth information.

[maver1ck]

Wow. I wasn't aware that 4090 is so fast.

[vitali-fridman]

This is from one/two generations old hardware but it's for 70B model which might be of interest.

CPU: AMD 3995WX, GPU: 2x Nvidia 3090, Ubuntu 23.10, Kernel 6.5.0-14, NV Driver: 545.23.08, CUDA: 12.3.1

model	size	params	backend	ngl	test	t/s
llama 70B Q4_0	36.20 GiB	68.98 B	CUDA	99	pp 512	179.29 ± 2.83
llama 70B Q4_0	36.20 GiB	68.98 B	CUDA	99	tg 128	21.17 ± 0.04

For comparison, 7B model on the same hardware

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	1178.60 ± 88.08
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	87.34 ± 0.89

[zotona]

could you try at 7b model for correct comparation? Thanks!

[crasm]

M2 Ultra, 16+8 CPU, 60 GPU (@crasm) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1128.59 ± 0.82
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	39.86 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	1003.16 ± 0.39
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	62.14 ± 0.03
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	1013.81 ± 0.92
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	88.64 ± 0.06

build: 8e672ef (1550)

[ymcui]

M3 Max (MBP 16), 12+4 CPU, 40 GPU (@ymcui) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	779.17 ± 0.49
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	25.09 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	757.64 ± 1.03
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	42.75 ± 0.06
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	759.70 ± 2.26
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	66.31 ± 0.12

build: 55978ce (1555)

---

Short Note: mostly similar to the one reported by @slaren . But for Q4_0 pp 512, my result is 759.70 ± 2.26, while the one in the main post is 690.99 ± 33.76. Not sure about the source of the difference.

[slaren]

I am not sure why, but the results that I get are not very consistent. I suspect that it may due to the cooling limitations of the smaller laptop. I repeated the test now and the results are very similar to yours.

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	787.24 ± 0.84
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	25.15 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	755.88 ± 1.56
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	42.64 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	760.65 ± 0.77
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	66.35 ± 0.24

[Azirine]

In the graph, why is PP t/s plotted against bandwidth and TG t/s plotted against GPU cores? Seems like GPU cores have more effect on PP t/s.

[Azirine]

How about also sharing the largest model sizes and context lengths people can run with their amount of RAM? It's important to get the amount of RAM right when buying Apple computers because you can't upgrade later.

[ggerganov]

You can compute these. By default, you can use ~75% of the total RAM with the GPU. You can use more if you do some tricks

[minosvasilias]

M2 Pro, 6+4 CPU, 16 GPU (@minosvasilias) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	312.65 ± 15.75
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	12.47 ± 0.71
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	288.46 ± 0.06
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	22.70 ± 0.12
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	294.24 ± 0.10
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	37.87 ± 0.10

build: e9c13ff (1560)

[to3d]

Would love to see how M1 Max and M1 Ultra fare given their high memory bandwidth.

[MrSparc]

M2 MAX (MBP 16) 8+4 CPU, 38 GPU, 96 GB RAM (@MrSparc) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	674.50 ± 0.58
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	41.79 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	669.51 ± 1.17
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	64.55 ± 1.36

build: e9c13ff (1560)

[rlippmann]

I'm also using a MBP16 M2Max with the same CPU/GPU specs, but only 32 gb ram and my results are roughly the same:

M2 MAX (MBP 16) 8+4 CPU, 38 GPU, 32 GB RAM ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	747.99 ± 0.28
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	24.54 ± 0.22
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	674.37 ± 0.63
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	40.67 ± 0.05
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	668.28 ± 0.24
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	62.98 ± 0.06

build: 22da055 (1566)

[MrSparc]

Yes, it is expected that the same cpu/gpu spec will have similar performance values for same models to be compared regardless of RAM, as long as the size of the model to be used can be loaded into memory.
The amount of RAM is a limiting factor in the size of the model that can be loaded, as only 75% (by default) of the unified memory can be used as VRAM on the GPU
https://github.com/ggerganov/llama.cpp#memorydisk-requirements

[CedricYauLBD]

M1 Max (MBP 16) 8+2 CPU, 32 GPU, 64GB RAM (@CedricYauLBD) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	599.53 ± 0.86
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	23.03 ± 0.09
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	537.37 ± 0.19
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	40.20 ± 0.03
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	530.06 ± 0.17
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	61.19 ± 0.15

build: e9c13ff (1560)

Note: M1 Max RAM Bandwidth is 400GB/s

[philipturner]

Look at what I started

[yxzwayne]

off topic, but your benchmark output is my desktop rn :D

[paramaggarwal]

M3 Pro (MBP 14), 5+6 CPU, 14 GPU (@paramaggarwal) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	272.11 ± 1.40
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	17.44 ± 0.42
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	269.49 ± 1.14
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	30.65 ± 0.20

build: e9c13ff (1560)

[ggerganov]

This one has 150 GB/s memory bandwidth, correct?

[paramaggarwal]

Yes, that's correct. (source)

[Kaszebe]

Could it run a Q5 quant of llama3 70b Instruct at ~2 tokens per second?

[brozkrut]

Chip (vs. Predecessor)	F16 PP	F16 TG	Q8_0 PP	Q8_0 TG	Q4_0 PP	Q4_0 TG
M2 Pro (16) vs. M1 Pro (16)	312.65 302.14	12.47 12.75	288.46 270.37	22.7 22.34	294.24 266.25	37.87 36.41
	+3.48%	-2.20%	+6.69%	+1.61%	+10.51%	+4.01%
M2 Max (38) vs. M1 Max (32)	755.67 599.53	24.65 23.03	677.91 537.37	41.83 40.2	671.31 530.06	65.95 61.19
	+26.04%	+7.03%	+26.15%	+4.05%	+26.65%	+7.78%
M2 Ultra (60) vs. M2 Max (38)	1128.59 755.67	39.86 24.65	1003.16 677.91	62.14 41.83	1013.81 671.31	88.64 65.95
	+49.34%	+61.90%	+48.04%	+48.48%	+51.03%	+34.41%
M2 Ultra (76) vs. M2 Max (38)	1401.85 755.67	41.02 24.65	1248.59 677.91	66.64 41.83	1238.48 671.31	94.27 65.95
	+85.67%	+66.45%	+84.24%	+59.47%	+84.53%	+43.06%
M2 Ultra (76) vs. M2 Ultra (60)	1401.85 1128.59	41.02 39.86	1248.59 1003.16	66.64 62.14	1238.48 1013.81	94.27 88.64
	+24.25%	+2.91%	+24.43%	+7.23%	+22.19%	+6.33%
M3 Pro (14) vs. M2 Pro (16)			272.11 288.46	17.44 22.7	269.49 294.24	30.65 37.87
			-5.67%	-23.17%	-8.41%	-19.07%
M3 Max (40) vs. M2 Max (38)	779.17 755.67	25.09 24.65	757.64 677.91	42.75 41.83	759.7 671.31	66.31 65.95
	+3.11%	+1.78%	+11.76%	+2.20%	+13.17%	+0.55%

[pudepiedj]

### M2 MAX (MBP 16) 38 Core 32GB ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	754.39 ± 0.36
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	24.31 ± 0.38
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	671.33 ± 2.65
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	40.85 ± 0.32
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	664.07 ± 9.11
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	63.29 ± 0.15

build: 795cd5a (1493)

[MrSparc]

I'm looking at the summary plot about "PP performance vs GPU cores" and evidence that original unquantised fp16 model always delivers more performance than quantized models.
Sorry if my question is silly, I'm new to this area, but can someone explain to me why original model delivers more performance than quantized models? Thanks

[ggerganov]

The question is not silly - the observation is expected. At large batch size (PP means batch size of 512) the computation is compute bound. I.e. the speed depends on how many FLOPS you can utilize. For quantum models, the existing kernels require extra compute to dequantize the data compared to F16 models where the data is already in F16 format.

[ptimmons]

M1 Ultra, 16+4 core CPU, 64 core GPU, 128GB RAM ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1158.95 ± 1.67
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	36.31 ± 0.18

build: 8e672efe (1550)

I'll work on the quantized models soon, I'm having some challenges getting them working.

[ptimmons]

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1168.89 ± 0.49
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	37.01 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	1042.95 ± 1.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	59.87 ± 0.09
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	1030.04 ± 0.72
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	83.73 ± 0.13

[ai-bits]

Major edit!
With this bench I wanted to share the results of my X86 + 2x Nvidia compared to my
M3 Max 10+4 CPU, 30 GPU (MacBook Pro 14 36GB RAM); here results from @soleblaze

Model	size	params	backend	ngl	test	t/s
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	566.40 ± 3.32
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	34.30 ± 0.24

Xeon 3435X w/ 256GB RAM and 2x 20GB Nvidia RTX 4000, compiled w/ CUDA and tooling 12.3:
Ubuntu 22.04 kernel 6.5.0-14-generic driver 545.23.08
Looks much better than on WSL, especially tg! Does anyone have an educated guess re the reason?
Still, if one 4090 is sooo much better I need to adjust my HW plans. Or is it optimization?
Cudos MacBook Pro M3!

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	pp 512	962.94 ± 10.27
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	tg 128	35.28 ± 0.08
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	pp 512	1049.96 ± 26.63
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	tg 128	50.47 ± 0.16
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	1087.59 ± 32.54
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	69.49 ± 0.29

On WSL2 Ubuntu 22.04 kernel 5.15.133.1 driver 546.01 (on Windows).

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	pp 512	648.07 ± 2.41
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	tg 128	18.46 ± 0.67
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	pp 512	637.51 ± 102.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	tg 128	19.11 ± 0.53
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	671.10 ± 94.51
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	19.23 ± 1.21

There is one bench output that worries me:
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no
Optimization problem? Impact?

I use the low-cores 20GB VRAM RTX 4000 because I thought the combined VRAM would be important.

Wonder why a 5800X3D w/ RTX 4090 @marcingomulkiewicz shows roughly 10x the speed, albeit on Linux directly...

@vitali-fridman Glad that my 2x RTX 4000 seem to be roughly on par with 2x 3090 when run directly on Ubuntu and not via WSL!

Thanks for any insights
G.

[slaren]

There was a recent change that should have improved multi GPU performance in some cases, at worst it should only be slightly slower than with a single GPU. You can also set the environment variable CUDA_VISIBLE_DEVICES=0 to use a single GPU.

[marcingomulkiewicz]

Felt like my results were significantly off, so I re-run this (in the meantime I upgraded kernel to 6.5.0-14, NV drivers to 545.23.08, CUDA tools to 12.3 (was 11.x before), and llama.cpp to 1930, but results still remain similar:

model	size	params	backend	ngl	test	t/s
llama 7B F16	12.55 GiB	6.74 B	CUDA	99	pp 512	10384.14 ± 829.23
llama 7B F16	12.55 GiB	6.74 B	CUDA	99	tg 128	61.99 ± 0.07
llama 7B Q8_0	6.67 GiB	6.74 B	CUDA	99	pp 512	7786.36 ± 5.41
llama 7B Q8_0	6.67 GiB	6.74 B	CUDA	99	tg 128	101.58 ± 0.25
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	8040.05 ± 9.72
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	156.48 ± 0.48

I have no direct access to 'original' llamas, but I took the ones by TheBloke - but I guess it shouldn't matter...?

[ai-bits]

Thanks guys!
@marcingomulkiewicz

I have no direct access to 'original' llamas, but I took the ones by TheBloke...

Same here. And the F16 one is from here.

@slaren
Please excuse the naive q from a 71y old hobby-programmer: export LLAMA_CUBLAS=1 and compile.
What did I overlook that I got
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no

MTIA
G.

[slaren]

You didn't overlook anything, it is just printing the setting used to compile llama.cpp. MMQ disabled is the default, as generally it is faster for newer GPUs with good FP16 performance, but you can try MMQ by compiling with LLAMA_CUDA_FORCE_MMQ defined.

[ai-bits]

Hey, @slaren, love weekend night support! ;-) Wishing / Having a VERY good Sunday morning!
export LLAMA_CUDA_FORCE_MMQ=1
compile & run:
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: yes
ggml_init_cublas: CUDA_USE_TENSOR_CORES: no
ggml_init_cublas: found 2 CUDA devices:
Device 0: NVIDIA RTX 4000 Ada Generation, compute capability 8.9
Device 1: NVIDIA RTX 4000 Ada Generation, compute capability 8.9
ggml_cuda_set_main_device: using device 0 (NVIDIA RTX 4000 Ada Generation) as main device

F16 didn't change, but look at Q8 & Q4:

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	pp 512	925.62 ± 23.50
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	tg 128	35.42 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	pp 512	1937.77 ± 109.23
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	tg 128	50.75 ± 0.02
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	1956.45 ± 134.41
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	69.83 ± 0.04

Oops, now we get there. <vbg> (Does anyone still know these abbreviations like <very broad grin> - with all the icons out there?)
(Variance is very bad, but we took the values be4, take it now too and second runs showed very similar values.)

export CUDA_VISIBLE_DEVICES=0 #use only 1 GPU

ggml_init_cublas: GGML_CUDA_FORCE_MMQ: yes
ggml_init_cublas: CUDA_USE_TENSOR_CORES: no
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA RTX 4000 Ada Generation, compute capability 8.9

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	pp 512	3022.37 ± 94.50
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	tg 128	23.93 ± 0.00
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	pp 512	1688.37 ± 51.25
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	tg 128	42.43 ± 0.02
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	1816.45 ± 68.24
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	73.38 ± 0.03

Now look at F16 PP!
1 GPU 3x faster than 2 GPUs with FORCE_MMQ instead of USE_TENSOR_CORES!!!
Guess this needs some major investigation / profiling!?
OMG!
G.

[dariox1337]

Since people already posted some non-apple results. Here is one more just for comparison.

CPU: AMD 7840u, GPU: 780m, RAM: LPDDR5-7500

model	size	params	backend	ngl	mmq	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	99	0	pp 512	310.95 ± 2.10
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	99	0	tg 128	21.47 ± 0.01

build 213d143

ROCm compiled without shared memory because the latter affects generation very badly.
So, this system has the same bandwidth as Apple M3, which results in almost the same tg, but pp is better.

[dariox1337]

Where did you get the formula? With /3.56 you mean scanning the whole model? Are you sure the algo is that primitive?

I may be wrong, but that's my understanding about how transformers work, gathered from many sources that I've read over months.

If that is correct, why do we need the bench?

Because the theoretical limit is unreachable. It implies that actual computation is instant, but it's not. The computation speed depends on the GPU's processing power and the specific implementation of the transformer algo.

And why does one of my RTX 4000 score 3022 pp w/ F16 and two only 925?

I guess the overhead of transferring the model's internal state between two gpus slows things down. It's happening over pcie lanes after all.

And why don't 8 v 4 threads change very much in LM Studio?

I think the accepted explanation is that on most systems, when running in cpu-only mode, 4 threads are enough to saturate available-for-cpu memory bandwidth.

[ai-bits]

And why does one of my RTX 4000 score 3022 pp w/ F16 and two only 925?

I guess the overhead of transferring the model's internal state between two gpus slows things down. It's happening over pcie lanes after all.

So, does the optimization using two GPUs when the model easily fits into one confirm my suspicion that the optimization on amd64 leaves sth to be desired or is your assumption wrong that it uses two GPUs?

[ai-bits]

Good morning, without an undertone:
I think we can agree the op is really highly memory-(throughput-)bound and the simple formula may not be too far off.

Hope we can agree that inference (search for the next, most probable token) is highly serial (not parallizable), so using both GPUs would rather hinder than help!?

Neither had the time nor the capacity to check if the algo uses the 2nd GPU where I think it's detrimental to perf, but my numbers suggest it is so.

Are you in the position to investigate this?
Thanks
G.

[dariox1337]

my suspicion that the optimization on amd64 leaves sth to be desired

I don't know the internals of llama.cpp, but amd64 isn't related to your issue. It's mostly CUDA. If apple systems supported discrete nvida gpus in pcie slots, you'd still have the same issue.

both GPUs would rather hinder than help!?

Using a single gpu is almost universally better. It's just harder to properly utilize multiple gpus. The only reason why you'd want multiple gpus is to increase available vram pool.

Are you in the position to investigate this?

Not me.

[ai-bits]

Using a single gpu is almost universally better.

The situation changed by the month over the last year. First it looked like you need to shell out sth like 20 or 50k, primarily to get 80GB VRAM to run sth locally. NvLink on consumer RTX died with the 2000 series. With 3.25-slot 4090s you could get max 2x 24GB into a box. There are water-cooled 1-slot ones, but you need room for the coolers, so same outcome for affordable.
Etc.
I went for a Xeon w/ 112 PCIe 5 lanes so I could stick in 6x 20GB RTX 4000, alas only PCIe 4, if necessary. In hindsight 2 RTX 4090 would have been better, cuz model sizes went down by the week with quantization.

We'll see how my hw does with OLMo fine-tuning - just this week's development.

I don't know the internals of llama.cpp, but amd64 isn't related to your issue. It's mostly CUDA.

AMD is extremely lucky with the development in this project. They couldn't have closed the >10-year-dev-gap otherwise.
I guess I'll need to look for other quantizations and better Nvidia-specific optimizations.

Regards
G.

[Borobo]

I have the same performance on an M3 Max as OP when running the benchmark, but there seems to be a big gap between llama and Llava performance. As the Llama and Llava 1.5 architectures don't differ much (Llava 1.5 = ViT + Projection Layer + Llama), I'm quite surprised. I'm sorry I can't give you any figures, as I don't think there's a benchmark script for multimodal in llama.cpp. Does anyone have the same felling?

[Borobo]

It seems that the issue is rather linked to metal than Llava, here's my result after running batched-bench with TheBloke/Mistral-7B-v0.1-GGUF:

First of all, is "default.metallib not found" normal?
Then, I think there is a problem concerning S_PP S_TG t/s as their performances are even worse when I'm using llama.cpp batches. How is it possible that my S_TG t/s is divided by two when using two batches?
Is it linked to my configuration?

[ggerganov]

First of all, is "default.metallib not found" normal?

normal

How is it possible that my S_TG t/s is divided by two when using two batches?

This is a known limitation - I've written more about it here: #3524

Unfortunately, I haven't found a universal solution yet to achieve optimal performance for all batches for all Apple devices and models. The best option you have atm is to enable this #if and adjust the values for your use case:

llama.cpp/ggml-metal.m

Lines 1244 to 1270 in e00d2a6

	// find the break-even point where the matrix-matrix kernel becomes more efficient compared
	// to the matrix-vector kernel
	int ne11_mm_min = 1;
	#if 0
	// the numbers below are measured on M2 Ultra for 7B and 13B models
	// these numbers do not translate to other devices or model sizes
	// TODO: need to find a better approach
	if ([ctx->device.name isEqualToString:@"Apple M2 Ultra"]) {
	switch (src0t) {
	case GGML_TYPE_F16: ne11_mm_min = 2; break;
	case GGML_TYPE_Q8_0: ne11_mm_min = 7; break;
	case GGML_TYPE_Q2_K: ne11_mm_min = 15; break;
	case GGML_TYPE_Q3_K: ne11_mm_min = 7; break;
	case GGML_TYPE_Q4_0:
	case GGML_TYPE_Q4_1: ne11_mm_min = 15; break;
	case GGML_TYPE_Q4_K: ne11_mm_min = 11; break;
	case GGML_TYPE_Q5_0: // not tested yet
	case GGML_TYPE_Q5_1: ne11_mm_min = 13; break; // not tested yet
	case GGML_TYPE_Q5_K: ne11_mm_min = 7; break;
	case GGML_TYPE_Q6_K: ne11_mm_min = 7; break;
	default: ne11_mm_min = 1; break;
	}
	}
	#endif

It can lead to significant improvements for bs > 1

Additionally, in this branch I'm currently writing new matrix multiplication kernels aiming to resolve this problem: https://github.com/ggerganov/llama.cpp/tree/gg/metal-batched
However, the results are not looking great so far

[agnosticlines]

Hey I know this is focused on a very specific benchmark but wanted to draw attention to this as it is performance related albeit only affecting the new IQ quant types #5617 and didn't want it being lost in discussions forever

[grigio]

can somebody run the benchmark on Linux with Apple M1 or M2 and Asahi Linux ?

[AndreasKunar]

Results for M2 Max 38 GPU 96 GB RAM MacOs with Ubuntu 24.04 in Parallels 19.4.0 - 8 CPUs + 32Gb allocated to VM. This does not support GPU-virtualization, only runs CPU-native.

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	CPU	8	pp512	26.18 ± 0.06
llama 7B F16	12.55 GiB	6.74 B	CPU	8	tg128	8.19 ± 0.09
llama 7B Q8_0	6.67 GiB	6.74 B	CPU	8	pp512	52.83 ± 0.36
llama 7B Q8_0	6.67 GiB	6.74 B	CPU	8	tg128	13.79 ± 0.49
llama 7B Q4_0	3.56 GiB	6.74 B	CPU	8	pp512	49.04 ± 0.06
llama 7B Q4_0	3.56 GiB	6.74 B	CPU	8	tg128	22.86 ± 0.16

Results for M2 Max 38 GPU 96 GB RAM MacOs with Fedora Container in Podman 4.9 - 8 CPUs + 32Gb allocated to VM. This DOES support GPU-acceleration via the Vulkan driver. Still not as fast as native.

Vulkan0: Virtio-GPU Venus (Apple M2 Max) | uma: 1 | fp16: 1 | warp size: 32

model	size	params	backend	ngl	test	t/s
llama 7B F16	12.55 GiB	6.74 B	Vulkan	99	pp 512	80.40 ± 0.43
llama 7B F16	12.55 GiB	6.74 B	Vulkan	99	tg 128	2.02 ± 0.00
llama 7B Q8_0	6.67 GiB	6.74 B	Vulkan	99	pp 512	79.79 ± 0.07
llama 7B Q8_0	6.67 GiB	6.74 B	Vulkan	99	tg 128	19.79 ± 0.01
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	99	pp 512	79.94 ± 0.13
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	99	tg 128	24.07 ± 0.01

[aneeshmb02]

What data/prompts are used for this?

[beebopkim]

llama-bench generates prompts by itself. So there is no given data or prompts.

[ymcui]

I have just tested the latest Apple M4 chip equipped on iPad Pro 2024 11-inch (256GB). The main difference between two different versions of M4 is the number of performance cores. Also 1TB/2TB iPad Pro has a doubled memory size (16GB).

Capacity	Performance Cores	Efficiency Cores	Graphics Cores	Bandwidth	Memory
256GB / 512GB	3	6	10	120 GB/s	8GB
1TB/2TB	4	6	10	120 GB/s	16GB

The following is a quick test for benchmarking M4.

M4 (iPad Pro 2024 256GB), 3+6 CPU, 10 GPU

tinyllama 1.1b

model	size	params	backend	test	t/s
llama 1B F16	2.05 GiB	1.10 B	Metal	pp 512	1375.87 ± 2.49
llama 1B F16	2.05 GiB	1.10 B	Metal	tg 128	46.21 ± 0.22
llama 1B Q8_0	1.09 GiB	1.10 B	Metal	pp 512	1333.82 ± 1.85
llama 1B Q8_0	1.09 GiB	1.10 B	Metal	tg 128	75.33 ± 0.19
llama 1B Q4_0	0.59 GiB	1.10 B	Metal	pp 512	1334.54 ± 6.87
llama 1B Q4_0	0.59 GiB	1.10 B	Metal	tg 128	121.20 ± 0.08

phi-2 2.7B

TBA

mistral 7b

model	size	params	backend	test	t/s
llama 7B Q4_0	3.83 GiB	7.24 B	Metal	pp 512	212.13 ± 0.11
llama 7B Q4_0	3.83 GiB	7.24 B	Metal	tg 128	23.29 ± 0.03

[tristan-k]

ROCm compiled without shared memory because the latter affects generation very badly.

Can you explain how to do that?

[pudepiedj]

Well, the HuggingFace repo says that Gemma-7B-it.gguf is 34.7GB, so I haven't tried because it looks obviously too big to run, but if you can buy a 64GB machine, I'd recommend that you do.

…

On Sat, Jun 15, 2024 at 2:53 PM Alptekin ***@***.***> wrote: @pudepiedj <https://github.com/pudepiedj> So you cannot run Gemma-7B-it.gguf on your M2 max 12-core CPU 38-core GPU? I am considering to buy a similar config (with 64gb ram) on mac studio so I am curious. Thanks. — Reply to this email directly, view it on GitHub <#4167 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGG22YNBCPRBRG43ZURHI5TZHRBOLAVCNFSM6AAAAAA7V5XCOKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOOBRHE2DI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[J-Siu]

Where can I get the models listed in the command line?

  -m ./models/llama-7b-v2/ggml-model-f16.gguf  \
  -m ./models/llama-7b-v2/ggml-model-q8_0.gguf \
  -m ./models/llama-7b-v2/ggml-model-q4_0.gguf \

[AndreasKunar]

This is Llama V2 (an older model), in order to compare apples to apples. You have to accept Meta's terms for using Llama 2. and you get the quantized versions at https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF. The f16 version you might get from https://huggingface.co/atwine/Llama-2-7b-chat-f16-gguf

Hope this helps.

[J-Siu]

Thank you! I believe I got the correct files as I got almost identical results as the 2nd M2 in the table above.

[AndreasKunar]

Yes, this should give you comparable benchmarks on your hardware. Some variations are also caused by different versions of llama.cpp

Please note, that llama-2 is not really this much useful anymore outside benchmark-comparisons. For using llama as LLM, I would suggest rather llama 3.1 8B, and - if you use llama.cpp on Snapdragon X or other newer ARM chip, the Q4_0_4_8 quantization (2-3x faster prompt-processing).

[miniak]

There won't be an M3 Ultra, you should remove that from the list

[bluemoehre]

Should be replaced with the M4 Ultra - this is what we're waiting for 🍿😁

[miccou]

M4 Pro, 10+4 CPU, 20 GPU, 24 GB Memory (@miccou) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	464.48 ± 2.62
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	17.18 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	449.62 ± 0.12
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	30.69 ± 0.03
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	439.78 ± 6.51
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	50.74 ± 0.07

build: 8e672ef (1550)

[miccou]

@ggerganov here are the results for the M4 Pro, as released today

[ndeldev]

@miccou thanks for the updated performance. Do you think the base model 12-Core CPU 20-Core GPU 48GB Unified Memory would perform any differently or is it largely dependent on the model size.

I extrapolated from the current data.

[sanddudu]

M4 (Mac Mini 2024), 4+6 CPU, 10GPU, 32 GB Memory ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	230.18 ± 0.16
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	7.43 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	223.64 ± 0.05
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	13.54 ± 0.01
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	221.29 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	24.11 ± 0.01

build: 8e672ef (1550)

[sekstini]

M4 Max (Macbook Pro 16" 2024), 12+4 CPU, 40GPU, 128 GB Memory ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	922.83 ± 1.12
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	31.64 ± 0.08
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	891.94 ± 0.28
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	54.05 ± 0.16
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	885.68 ± 1.11
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	83.06 ± 0.22

build: 8e672ef (1550)

[atlas5301]

You are mostly correct about this, if a high bandwidth network is available. In fact, even for the largest models, it's still possible to fit one layer in vram , calculate the kv-cache and the hidden states, and then load the next layer to the vram. The major overheads are pcie transfer of the model checkpoint and the transfer of kv-cache from nvidia gpu device to apple silicon devices. For very long prompts, the former one is negligible. The kv-dimension is 1024 for the 32-layer llama-3.1-8b. Hence, the size of kv-cache per token should be 2*32*1024*2Byte, or 128KiB. To match the 1000 token/s prompt evaluation speed, a 1Gbps ethernet is required. For Qwen2.5-7B-Instruct, about half of the speed is required. The size of kv-cache per Llama-3.1-70b token should be 2*80*1024*2Byte, or 320KiB. To match the 100 token/s prompt evaluation speed, we only need to have a 250Mbps network connection. We can always expect some models, for example, deepseek series, with smaller kv-cache size per token, will benefit from such offloading, especially when the prompt itself is long enough. A rough estimate of the prompt length that will benefit from such kind of offloading is about 1000 tokens, where it takes much longer to process the prompt on m4max than on a nvidia device considering the extra time spent loading the model weights from ram.

[bluemoehre]

@ggerganov insane configuration having 192GB RAM. I was thinking on getting a used Studio M2 Max (32-64GB) or 2x Mac Mini M4 (16GB + 24GB Pro) to run a Chat Model + a Vision Model / Embedding Model (RAG for docs) for daily tasks. Right now it is a similar price tag. Would you recommend getting a huge machine over multiple small ones? How is your subjective experience in quality of answers with larger models / better Q?

[ggerganov]

The M2 Max has 400GB/s memory bandwidth while the M4 Pro has 273GB/s, so ~1.5x faster for text generation. M2 Max with 38 GPU cores would be ~1.5x faster than M4 Pro with 20 GPU cores for prompt processing (see the table above). But with the Minis, you will have 2 of them which can work in parallel. I don't have experience with Vision/Embeddings and RAG so can't really recommend anything.

How is your subjective experience in quality of answers with larger models / better Q?

Can't give you much info here as I am not a heavy chat user and don't have a good base for comparison. I'm mainly using the Qwen 2.5 Coder models and my primary usage is FIM completion. Regarding Q - I've settled on Q8 because prompt processing speed is much more important for FIM and since I have practically infinite VRAM there is no need to use lower quantizations as they don't improve the prompt processing speed. I would probably even use F16 for the extra PP speed, but there is an annoying bug in MacOS that is kind of a deal breaker at the moment (#10119).

[bluemoehre]

Thanks for your feedback. Yeh, I saw that bandwidth difference, but right now I am with a M2 Air and text generation speed is already fine after latest OS updates. The M4 Pro would make it >2x faster, so I would like to put more focus on flexibility and the ability to run more than one model in parallel 24/7. For the heavy loads I still gonna use the cloud, later on (after RTX 5000 series release next weeks) I consider to get some little brick and attach my current RTX 4070 Ti S to it. Unfortunately a Studio M2 (>128GB) is still too expensive.

[gardner]

after RTX 5000 series release next weeks

Can you tell me more about next week?