CapsNet-vs-CNN

Research focused on comparison between CapsNet and CNN architectures in case of rotated image classification on CIFAR-10 dataset.

Our Experiments

CapsNet with single Digit Layer

• MNIST:

$ python capsnet/singledigitcapsule/mnist_capsulenet.py

• CIFAR10:

$ python capsnet/singledigitcapsule/cifar_capsulenet.py

CapsNet with two Digit Layers

• MNIST:

$ python capsnet/twodigitcapsules/mnist_capsulenet.py

• CIFAR10:

$ python capsnet/twodigitcapsules/cifar_capsulenet.py

ResNet 110

• MNIST:

$ python cnn/mnist_resnet.py

• CIFAR10:

$ python cnn/cifar_resnet.py

Research results

In our work we decided to check CapsNets ability to understand image rotations. We used CIFAR10 and MINIST datasets in series of experiments, training ResNet 110 architecture as classical CNN model representative to compare it with two CapsNet approaches (single/souble caps layers). Models were trained using two kinds of training datasets: unrotated and rotated CIFAR/MNIST.

Results on CIFAR10

Rotated training dataset

Weights are available here:

• CapsNet with Digit Layer.
• CapsNet with two Digit Layers.
• Resnet 110.

Rotation degrees	Accuracy - Resnet110	Accuracy - CapsNet (Single layer)	Accuracy - CapsNet (Double layer)
-180°	0.4162	0.3316	0.337
-165°	0.4185	0.3184	0.3283
-150°	0.4171	0.3115	0.323
-135°	0.3926	0.29	0.2941
-120°	0.3847	0.2629	0.299
-105°	0.3929	0.2661	0.3152
-90°	0.4284	0.2873	0.3315
-75°	0.4992	0.3111	0.3703
-60°	0.6246	0.3774	0.457
-45°	0.7697	0.5106	0.6071
-30°	0.8791	0.6544	0.7463
-15°	0.9145	0.726	0.8006
0°	0.9076	0.7358	0.791
15°	0.91	0.7192	0.7965
30°	0.8829	0.659	0.7504
45°	0.7757	0.5224	0.6058
60°	0.6423	0.3749	0.4494
75°	0.5137	0.3031	0.3682
90°	0.4459	0.2793	0.3285
105°	0.4049	0.2569	0.3053
120°	0.3834	0.2545	0.298
135°	0.384	0.2651	0.2968
150°	0.4127	0.2978	0.3244
165°	0.4217	0.3134	0.3346

---

In relation to the 0° accuracy score:

Rotation degrees	Accuracy - Resnet110	Accuracy - CapsNet (Single layer)	Accuracy - CapsNet (Double layer)
-180°	0.4586	0.4507	0.4260
-165°	0.4611	0.4327	0.4150
-150°	0.4596	0.4233	0.4083
-135°	0.4326	0.3941	0.3718
-120°	0.4239	0.3573	0.3780
-105°	0.4329	0.3616	0.3985
-90°	0.472	0.3905	0.4191
-75°	0.55	0.4228	0.4681
-60°	0.6882	0.5129	0.5778
-45°	0.8480	0.6939	0.7675
-30°	0.9686	0.8894	0.9435
-15°	1.008	0.9867	1.0121
0°	1	1	1
15°	1.003	0.9774	1.007
30°	0.9728	0.8956	0.9487
45°	0.8547	0.71	0.7659
60°	0.7077	0.5095	0.5681
75°	0.566	0.4119	0.4655
90°	0.4913	0.3796	0.4153
105°	0.4461	0.3491	0.386
120°	0.4224	0.3459	0.3767
135°	0.4231	0.3603	0.3752
150°	0.4547	0.4047	0.4101
165°	0.4646	0.4259	0.4230

Without rotations in training dataset

Weights are available here:

• CapsNet with single Digit Layer.
• CapsNet with two Digit Layers.
• Resnet 110.

Rotation degrees	Accuracy - Resnet110	Accuracy - CapsNet (Single layer)	Accuracy - CapsNet (Double layer)
-180°	0.4162	0.2729	0.3493
-165°	0.3578	0.2431	0.3259
-150°	0.2859	0.22	0.2809
-135°	0.2569	0.2042	0.2537
-120°	0.2769	0.1959	0.2461
-105°	0.3288	0.2006	0.2624
-90°	0.3989	0.249	0.304
-75°	0.3905	0.2437	0.3043
-60°	0.3864	0.2559	0.3227
-45°	0.4226	0.297	0.3966
-30°	0.6191	0.4019	0.5314
-15°	0.87	0.566	0.7331
0°	0.9185	0.6805	0.8144
15°	0.8697	0.5802	0.7431
30°	0.6332	0.4143	0.536
45°	0.4304	0.3	0.4
60°	0.3909	0.2462	0.3309
75°	0.3863	0.2345	0.3023
90°	0.4138	0.2465	0.3082
105°	0.3345	0.2083	0.259
120°	0.2717	0.1897	0.236
135°	0.2482	0.2	0.2406
150°	0.2858	0.2146	0.2553
165°	0.3536	0.2382	0.3046

---

In relation to the 0° accuracy score:

Rotation degrees	Accuracy - Resnet110	Accuracy - CapsNet (Single layer)	Accuracy - CapsNet (Double layer)
-180°	0.4531	0.401	0.4289
-165°	0.3895	0.3572	0.4002
-150°	0.3113	0.3233	0.3449
-135°	0.2797	0.3001	0.3115
-120°	0.3015	0.2879	0.3022
-105°	0.358	0.2948	0.3222
-90°	0.4343	0.3659	0.3733
-75°	0.4252	0.3581	0.3736
-60°	0.4207	0.376	0.3962
-45°	0.4601	0.4364	0.4870
-30°	0.6740	0.5906	0.6525
-15°	0.9472	0.8317	0.9002
0°	1	1	1
15°	0.9469	0.8526	0.9125
30°	0.6894	0.6088	0.6582
45°	0.4686	0.4409	0.4912
60°	0.4256	0.3618	0.4063
75°	0.4206	0.3446	0.3712
90°	0.4505	0.3622	0.3784
105°	0.3642	0.3061	0.318
120°	0.2958	0.2788	0.2898
135°	0.2702	0.2939	0.2954
150°	0.3112	0.3154	0.3135
165°	0.385	0.3500	0.374

Results on MNIST

Rotated training dataset

Weights are available here:

• CapsNet with single Digit Layer.
• CapsNet with two Digit Layers.
• Resnet 110.

Rotation degrees	Accuracy - Resnet110	Accuracy - CapsNet (Single layer)	Accuracy - CapsNet (Double layer)
-180°	0.4216	0.4261	0.4274
-165°	0.4307	0.4236	0.443
-150°	0.4305	0.4247	0.4343
-135°	0.4181	0.4011	0.4008
-120°	0.3825	0.3427	0.3155
-105°	0.3649	0.2936	0.2448
-90°	0.4119	0.3306	0.2548
-75°	0.6214	0.5177	0.4569
-60°	0.8523	0.802	0.7752
-45°	0.9673	0.9582	0.944
-30°	0.9901	0.9901	0.9836
-15°	0.9951	0.993	0.9899
0°	0.9956	0.9929	0.9871
15°	0.9961	0.9919	0.9913
30°	0.9926	0.989	0.9883
45°	0.9654	0.9433	0.9483
60°	0.7894	0.6914	0.7153
75°	0.496	0.3606	0.3844
90°	0.337	0.2284	0.235
105°	0.3524	0.2705	0.2653
120°	0.3983	0.3785	0.3577
135°	0.4316	0.4427	0.423
150°	0.4296	0.4572	0.4478
165°	0.4226	0.444	0.4369

Without rotations in training dataset

Weights are available here:

• CapsNet with single Digit Layer.
• CapsNet with two Digit Layers.
• Resnet 110.

Rotation degrees	Accuracy - Resnet110	Accuracy - CapsNet (Single layer)	Accuracy - CapsNet (Double layer)
-180°	0.4107	0.4408	0.3968
-165°	0.4202	0.4346	0.405
-150°	0.4109	0.4049	0.3984
-135°	0.3589	0.3402	0.3422
-120°	0.2669	0.2597	0.26
-105°	0.1897	0.1881	0.182
-90°	0.1697	0.1733	0.1637
-75°	0.2291	0.2488	0.2363
-60°	0.4264	0.3991	0.3928
-45°	0.7224	0.644	0.6486
-30°	0.931	0.9004	0.8928
-15°	0.989	0.9829	0.9803
0°	0.9959	0.9932	0.9913
15°	0.9874	0.9815	0.9726
30°	0.8769	0.8631	0.8099
45°	0.5646	0.5472	0.5073
60°	0.2712	0.2637	0.2289
75°	0.1516	0.147	0.1222
90°	0.1531	0.1468	0.134
105°	0.2012	0.21	0.2024
120°	0.2718	0.2807	0.2806
135°	0.3484	0.3534	0.3336
150°	0.3939	0.4108	0.3734
165°	0.4108	0.4376	0.3965

Used model implementations

• Resnet v2 by Keras Team

• CapsNet-Keras by XifengGuo based upon Sara Sabour, Nicholas Frosst, Geoffrey E Hinton. Dynamic Routing Between Capsules. NIPS 2017