Skip to content

Latest commit

 

History

History
59 lines (48 loc) · 6.71 KB

Pipeline_Benchmark_CN.md

File metadata and controls

59 lines (48 loc) · 6.71 KB
Raw

Python Pipeline 性能测试

测试环境

测试环境如下表所示：

GPU 显存 CPU 内存
Serving端 4x Tesla P4-8GB 7611MiB Intel(R) Xeon(R) Gold 5117 CPU @ 2.00GHz 48核 216G
Client端 4x Tesla P4-8GB 7611MiB Intel(R) Xeon(R) Gold 5117 CPU @ 2.00GHz 48核 216G

使用单卡GPU，未开启TensorRT。 模型：ResNet_v2_50

性能指标与结论

通过测试，使用 Python Pipeline 模式通过多进程并发，充分利用 GPU 显卡，具有较好的吞吐性能。

测试数据如下：

model_name thread_num batch_size CPU_util(%) GPU_memory(mb) GPU_util(%) qps(samples/s) total count mean(ms) median(ms) 80 percent(ms) 90 percent(ms) 99 percent(ms) total cost(s) each cost(s)
ResNet_v2_50 1 1 2.2 3327 17.25 17.633658869240787 355 56.428481238996476 38.646728515625 39.496826171875 39.98369140625 1273.1911083984373 20.131953477859497 20.033540725708008
ResNet_v2_50 1 4 2.7 3617 28.122 53.50748430453522 268 74.71539215543378 74.6181640625 75.3138671875 75.6051025390625 77.85322998046874 20.03458046913147 20.024930953979492
ResNet_v2_50 1 8 1.7 3877 25.7869 59.60582783086999 150 133.5897119140625 132.7503662109375 134.968310546875 136.470703125 140.79039062499996 20.132259607315063 20.03933620452881
ResNet_v2_50 1 16 7.0 4777 27.0175 63.2627646819339 80 252.30162048339844 251.8448486328125 253.046630859375 253.91142578125 263.361640625 20.233070850372314 20.18476152420044
ResNet_v2_50 1 32 7.5 6567 38.532 62.945314687348024 40 506.8969482421875 507.3531494140625 510.562353515625 511.421240234375 536.8068920898437 20.335111618041992 20.276386737823486
ResNet_v2_50 2 1 4.7 6567 49.4828 50.40600094376044 1010 39.63352195815285 39.5345458984375 40.452880859375 41.1375 42.940522460937494 20.037296772003174 20.01696753501892
ResNet_v2_50 2 4 2.7 6567 44.4744 83.4255836891382 420 95.38548002697172 95.7069091796875 97.599951171875 98.098583984375 102.39680908203125 20.137707471847534 20.03199553489685
ResNet_v2_50 2 8 2.2 6567 42.898 91.3727510505176 230 174.89108568274457 175.0452880859375 175.82001953125 176.7634033203125 178.64064453125002 20.13729453086853 20.1132071018219
ResNet_v2_50 2 16 2.2 6567 45 97.5591285698611 124 327.16720088835683 328.6126708984375 329.75185546875 330.386962890625 336.86397460937496 20.336385011672974 20.284939169883728
ResNet_v2_50 2 32 3.2 6567 59.5714 100.70765418116333 64 633.9812698364258 637.8568115234375 648.103515625 650.7439697265625 659.2212915039062 20.336090803146362 20.28787398338318
ResNet_v2_50 4 1 3.1 6567 64.3333 80.27845081929433 1617 49.56464230756223 49.4873046875 51.5537109375 52.693408203125 55.207568359374996 20.142391681671143 20.038144528865814
ResNet_v2_50 4 4 3.3 6567 70.4563 136.62061939701394 688 116.51574919944586 121.8629150390625 129.8181640625 133.384423828125 142.69500732421875 20.143372297286987 20.041599333286285
ResNet_v2_50 4 8 3.0 6567 70.896 158.46554975132275 399 201.30669079926378 210.69775390625 228.51748046875 236.427294921875 252.24822753906233 20.143179416656494 20.081032752990723
ResNet_v2_50 4 16 3.2 6567 66.3832 156.4935247130092 197 407.6668608224937 423.974609375 450.368212890625 464.45986328125 482.93658203125 20.141408443450928 20.078101694583893
ResNet_v2_50 4 32 3.3 6567 72.4791 162.01742190796557 104 785.5079204852765 813.0341796875 887.107958984375 909.6556640625 935.3334838867188 20.541000843048096 20.423666059970856
ResNet_v2_50 8 1 3.5 6567 93.977 115.9749228558386 2337 68.5580409078145 65.45849609375 76.13930664062501 83.542041015625 91.45666015624998 20.15090799331665 20.028797417879105
ResNet_v2_50 8 4 4.2 6567 90.0952 175.58748591910316 889 180.7330482920592 170.5810546875 218.99931640625 240.06337890625002 254.413759765625 20.252012729644775 20.084695398807526
ResNet_v2_50 8 8 2.6 6567 93.8693 206.76595246418208 526 306.52158695119414 303.043212890625 321.0791015625 350.5477294921875 400.32452392578125 20.351513147354126 20.15437400341034
ResNet_v2_50 8 16 3.2 6567 85.7273 205.31850043117367 265 614.1745522553066 552.372314453125 775.89169921875 802.022607421875 902.2763183593761 20.650842428207397 20.345011442899704
ResNet_v2_50 8 32 5.0 6567 89.8717 219.8410273718835 146 1138.4533474020761 1039.640869140625 1364.289794921875 1474.6744384765625 1788.2614379882834 21.251720190048218 20.777225106954575
ResNet_v2_50 12 1 5.0 6567 89.4762 110.00858327847862 2218 108.50048552943953 103.015625 121.09404296875003 137.1392333984375 151.80401123046872 20.162063121795654 20.055511037508648
ResNet_v2_50 12 4 4.1 6567 77.7619 153.7824464757549 779 309.68895575507463 285.585205078125 378.07421875 413.481640625 424.70853515625 20.262390613555908 20.104551911354065
ResNet_v2_50 12 8 3.6 6567 72.6977 165.36021780846013 425 571.1991590073529 510.995849609375 731.9383300781251 747.6568359375 757.304716796875 20.56117272377014 20.230452219645183
ResNet_v2_50 12 16 1.5 6567 76.2222 189.6414991568285 252 987.7153136238219 926.00390625 1080.99130859375 1249.4956298828126 1434.4802392578124 21.26116919517517 20.74245794614156
ResNet_v2_50 12 32 2.8 6567 84.25 203.868228281784 138 1811.640237559443 1764.2760009765625 1855.28046875 2023.56826171875 2586.8038134765625 21.66105055809021 20.834286351998646
ResNet_v2_50 16 1 4.8 6567 94.3333 116.34927733312234 2347 136.7957122373642 135.959716796875 144.1568359375 146.105517578125 175.05707519531248 20.172020435333252 20.067057371139526
ResNet_v2_50 16 4 15.4 6567 83.6364 160.59012047270738 822 393.3079394412447 396.446533203125 426.272216796875 429.777734375 564.1119360351562 20.47448492050171 20.206754431128502
ResNet_v2_50 16 8 6.8 6567 81.0233 169.95774070621547 437 741.5512622684854 751.521484375 763.199169921875 948.8041992187501 1001.156142578125 20.56981921195984 20.254074171185493
ResNet_v2_50 16 16 3.5 6567 77.8706 186.56600081516 248 1332.1007946383568 1365.2745361328125 1399.212255859375 1432.4037353515625 1771.4374853515626 21.26861262321472 20.64799252152443
ResNet_v2_50 16 32 4.3 6567 83.6371 201.1293408638195 140 2419.3400198800223 2561.09228515625 2616.081103515625 2642.0835205078124 2883.8197412109366 22.274224042892456 21.169659316539764