More Continuous Benchmarks

[avik-pal]

• Simple Chains Benchmarks
• Test AD performance

Tapir seems very slow -- open an issue upstream once the benchmarks are deployed

Needs LuxDL/LuxLib.jl#49

[github-actions]

Benchmark Results

Benchmark suite	Current: 8c83975	Previous: 849ec40	Ratio
Dense(2 => 2)/cpu/reverse/ReverseDiff (compiled)/(2, 128)	3219.125 ns
Dense(2 => 2)/cpu/reverse/Zygote/(2, 128)	9227.5 ns
Dense(2 => 2)/cpu/reverse/Tracker/(2, 128)	15193.5 ns
Dense(2 => 2)/cpu/reverse/ReverseDiff/(2, 128)	4724.571428571428 ns
Dense(2 => 2)/cpu/reverse/Flux/(2, 128)	6950 ns
Dense(2 => 2)/cpu/reverse/SimpleChains/(2, 128)	1524.3 ns
Dense(2 => 2)/cpu/forward/NamedTuple/(2, 128)	1076 ns	2001.7 ns	0.54
Dense(2 => 2)/cpu/forward/ComponentArray/(2, 128)	740.275167785235 ns	1590.4709677419355 ns	0.47
Dense(2 => 2)/cpu/forward/Flux/(2, 128)	1809.171052631579 ns
Dense(2 => 2)/cpu/forward/SimpleChains/(2, 128)	179.72624113475177 ns
Dense(20 => 20)/cpu/reverse/ReverseDiff (compiled)/(20, 128)	18324 ns
Dense(20 => 20)/cpu/reverse/Zygote/(20, 128)	28132.5 ns
Dense(20 => 20)/cpu/reverse/Tracker/(20, 128)	41728 ns
Dense(20 => 20)/cpu/reverse/ReverseDiff/(20, 128)	25993.5 ns
Dense(20 => 20)/cpu/reverse/Flux/(20, 128)	21750 ns
Dense(20 => 20)/cpu/reverse/SimpleChains/(20, 128)	14718 ns
Dense(20 => 20)/cpu/forward/NamedTuple/(20, 128)	5056.571428571428 ns	4869.142857142857 ns	1.04
Dense(20 => 20)/cpu/forward/ComponentArray/(20, 128)	5045.142857142857 ns	4718.857142857143 ns	1.07
Dense(20 => 20)/cpu/forward/Flux/(20, 128)	5022.285714285714 ns
Dense(20 => 20)/cpu/forward/SimpleChains/(20, 128)	1672.15 ns
Conv((3, 3), 3 => 3)/cpu/reverse/ReverseDiff (compiled)/(64, 64, 3, 128)	38774148 ns
Conv((3, 3), 3 => 3)/cpu/reverse/Zygote/(64, 64, 3, 128)	89025940.5 ns
Conv((3, 3), 3 => 3)/cpu/reverse/Tracker/(64, 64, 3, 128)	94106067.5 ns
Conv((3, 3), 3 => 3)/cpu/reverse/ReverseDiff/(64, 64, 3, 128)	39580475 ns
Conv((3, 3), 3 => 3)/cpu/reverse/Flux/(64, 64, 3, 128)	89538924 ns
Conv((3, 3), 3 => 3)/cpu/reverse/SimpleChains/(64, 64, 3, 128)	11795817.5 ns
Conv((3, 3), 3 => 3)/cpu/forward/NamedTuple/(64, 64, 3, 128)	10338375 ns	10357033 ns	1.00
Conv((3, 3), 3 => 3)/cpu/forward/ComponentArray/(64, 64, 3, 128)	10386003 ns	10383988.5 ns	1.00
Conv((3, 3), 3 => 3)/cpu/forward/Flux/(64, 64, 3, 128)	10421130 ns
Conv((3, 3), 3 => 3)/cpu/forward/SimpleChains/(64, 64, 3, 128)	6413360.5 ns
vgg16/cpu/reverse/Zygote/(32, 32, 3, 1)	4043574679 ns
vgg16/cpu/reverse/Zygote/(32, 32, 3, 16)	12018294210 ns
vgg16/cpu/reverse/Zygote/(32, 32, 3, 64)	15214721419 ns
vgg16/cpu/reverse/Tracker/(32, 32, 3, 1)	1367913576 ns
vgg16/cpu/reverse/Tracker/(32, 32, 3, 16)	2569075480 ns
vgg16/cpu/reverse/Tracker/(32, 32, 3, 64)	5246057918 ns
vgg16/cpu/reverse/Flux/(32, 32, 3, 1)	83986821 ns
vgg16/cpu/reverse/Flux/(32, 32, 3, 16)	711473019 ns
vgg16/cpu/reverse/Flux/(32, 32, 3, 64)	3129579942 ns
vgg16/cpu/forward/NamedTuple/(32, 32, 3, 1)	24457835.5 ns	22706894 ns	1.08
vgg16/cpu/forward/NamedTuple/(32, 32, 3, 16)	246050431 ns	251415835 ns	0.98
vgg16/cpu/forward/NamedTuple/(32, 32, 3, 64)	998537165 ns	990621987 ns	1.01
vgg16/cpu/forward/ComponentArray/(32, 32, 3, 1)	23391433 ns	22766979.5 ns	1.03
vgg16/cpu/forward/ComponentArray/(32, 32, 3, 16)	248845730 ns	249637390 ns	1.00
vgg16/cpu/forward/ComponentArray/(32, 32, 3, 64)	973341188 ns	991675056.5 ns	0.98
vgg16/cpu/forward/Flux/(32, 32, 3, 1)	21821208 ns
vgg16/cpu/forward/Flux/(32, 32, 3, 16)	226034195 ns
vgg16/cpu/forward/Flux/(32, 32, 3, 64)	925498943 ns
Conv((3, 3), 64 => 64)/cpu/reverse/ReverseDiff (compiled)/(64, 64, 64, 128)	1049543801 ns
Conv((3, 3), 64 => 64)/cpu/reverse/Zygote/(64, 64, 64, 128)	2327844024 ns
Conv((3, 3), 64 => 64)/cpu/reverse/Tracker/(64, 64, 64, 128)	2713014494 ns
Conv((3, 3), 64 => 64)/cpu/reverse/ReverseDiff/(64, 64, 64, 128)	1383161011 ns
Conv((3, 3), 64 => 64)/cpu/reverse/Flux/(64, 64, 64, 128)	2294746315 ns
Conv((3, 3), 64 => 64)/cpu/forward/NamedTuple/(64, 64, 64, 128)	372674399 ns	371776615 ns	1.00
Conv((3, 3), 64 => 64)/cpu/forward/ComponentArray/(64, 64, 64, 128)	381371753 ns	372349398 ns	1.02
Conv((3, 3), 64 => 64)/cpu/forward/Flux/(64, 64, 64, 128)	376596385 ns
Conv((3, 3), 1 => 1)/cpu/reverse/ReverseDiff (compiled)/(64, 64, 1, 128)	13631262.5 ns
Conv((3, 3), 1 => 1)/cpu/reverse/Zygote/(64, 64, 1, 128)	30731203 ns
Conv((3, 3), 1 => 1)/cpu/reverse/Tracker/(64, 64, 1, 128)	31237330 ns
Conv((3, 3), 1 => 1)/cpu/reverse/ReverseDiff/(64, 64, 1, 128)	13821574.5 ns
Conv((3, 3), 1 => 1)/cpu/reverse/Flux/(64, 64, 1, 128)	30578198 ns
Conv((3, 3), 1 => 1)/cpu/reverse/SimpleChains/(64, 64, 1, 128)	1148259.5 ns
Conv((3, 3), 1 => 1)/cpu/forward/NamedTuple/(64, 64, 1, 128)	3878724 ns	3900400 ns	0.99
Conv((3, 3), 1 => 1)/cpu/forward/ComponentArray/(64, 64, 1, 128)	3880672 ns	3899295 ns	1.00
Conv((3, 3), 1 => 1)/cpu/forward/Flux/(64, 64, 1, 128)	3879132 ns
Conv((3, 3), 1 => 1)/cpu/forward/SimpleChains/(64, 64, 1, 128)	193155.5 ns
Dense(200 => 200)/cpu/reverse/ReverseDiff (compiled)/(200, 128)	293858.5 ns
Dense(200 => 200)/cpu/reverse/Zygote/(200, 128)	309051.5 ns
Dense(200 => 200)/cpu/reverse/Tracker/(200, 128)	379484 ns
Dense(200 => 200)/cpu/reverse/ReverseDiff/(200, 128)	337104.5 ns
Dense(200 => 200)/cpu/reverse/Flux/(200, 128)	272443.5 ns
Dense(200 => 200)/cpu/reverse/SimpleChains/(200, 128)	395288 ns
Dense(200 => 200)/cpu/forward/NamedTuple/(200, 128)	153111 ns	87314 ns	1.75
Dense(200 => 200)/cpu/forward/ComponentArray/(200, 128)	88560.5 ns	87424 ns	1.01
Dense(200 => 200)/cpu/forward/Flux/(200, 128)	86376.5 ns
Dense(200 => 200)/cpu/forward/SimpleChains/(200, 128)	104635.5 ns
Conv((3, 3), 16 => 16)/cpu/reverse/ReverseDiff (compiled)/(64, 64, 16, 128)	198192737 ns
Conv((3, 3), 16 => 16)/cpu/reverse/Zygote/(64, 64, 16, 128)	495951741 ns
Conv((3, 3), 16 => 16)/cpu/reverse/Tracker/(64, 64, 16, 128)	516213058 ns
Conv((3, 3), 16 => 16)/cpu/reverse/ReverseDiff/(64, 64, 16, 128)	210183666 ns
Conv((3, 3), 16 => 16)/cpu/reverse/Flux/(64, 64, 16, 128)	497070792 ns
Conv((3, 3), 16 => 16)/cpu/reverse/SimpleChains/(64, 64, 16, 128)	318630123 ns
Conv((3, 3), 16 => 16)/cpu/forward/NamedTuple/(64, 64, 16, 128)	61024992 ns	56912462 ns	1.07
Conv((3, 3), 16 => 16)/cpu/forward/ComponentArray/(64, 64, 16, 128)	64276993 ns	56953662 ns	1.13
Conv((3, 3), 16 => 16)/cpu/forward/Flux/(64, 64, 16, 128)	64321306 ns
Conv((3, 3), 16 => 16)/cpu/forward/SimpleChains/(64, 64, 16, 128)	27978897 ns
Dense(2000 => 2000)/cpu/reverse/ReverseDiff (compiled)/(2000, 128)	20995755 ns
Dense(2000 => 2000)/cpu/reverse/Zygote/(2000, 128)	25981622.5 ns
Dense(2000 => 2000)/cpu/reverse/Tracker/(2000, 128)	30682768 ns
Dense(2000 => 2000)/cpu/reverse/ReverseDiff/(2000, 128)	22113332 ns
Dense(2000 => 2000)/cpu/reverse/Flux/(2000, 128)	19696628.5 ns
Dense(2000 => 2000)/cpu/forward/NamedTuple/(2000, 128)	6579599 ns	6515785 ns	1.01
Dense(2000 => 2000)/cpu/forward/ComponentArray/(2000, 128)	6568808 ns	6560088 ns	1.00
Dense(2000 => 2000)/cpu/forward/Flux/(2000, 128)	6552518 ns

This comment was automatically generated by workflow using github-action-benchmark.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.33%. Comparing base (849ec40) to head (8c83975).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #575   +/-   ##
=======================================
  Coverage   87.32%   87.33%           
=======================================
  Files          33       33           
  Lines        1728     1729    +1     
=======================================
+ Hits         1509     1510    +1     
  Misses        219      219           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.