Training on custom dataset of EfficientDet-0 model crash : TypeError: 'NoneType' object is not callable

[Pentar0o]

Hi, there is the problem :

penta@dell-r740:~$ python3 automl/efficientdet/tf2/train.py --train_file_pattern=train_tf.tfrecord --val_file_pattern=val_tf.tfrecord --model_name=efficientdet-d0 --model_dir=/tmp/efficientdet-d0
-finetune --pretrained_ckpt=efficientdet-d0 --batch_size=24 --eval_samples=1024 --num_examples_per_epoch=33707 --num_epochs=50 --hparams=voc_config.yaml --strategy=gpus
2024-02-13 15:12:30.743921: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2024-02-13 15:12:30.792061: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-13 15:12:30.792093: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-13 15:12:30.793253: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-13 15:12:30.800038: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-02-13 15:12:34.323310: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:236] Using CUDA malloc Async allocator for GPU: 0
2024-02-13 15:12:34.325068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13764 MB memory: -> device: 0, name: Tesla T4, pci bus id: 0000:5e:00.0, compute capability: 7.5
2024-02-13 15:12:34.325313: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:236] Using CUDA malloc Async allocator for GPU: 1
2024-02-13 15:12:34.327016: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13764 MB memory: -> device: 1, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5
2024-02-13 15:12:34.327235: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:236] Using CUDA malloc Async allocator for GPU: 2
2024-02-13 15:12:34.329015: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 13764 MB memory: -> device: 2, name: Tesla T4, pci bus id: 0000:d8:00.0, compute capability: 7.5
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2')
I0213 15:12:34.331413 140653502825600 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2')
I0213 15:12:34.387622 140653502825600 train.py:198] All devices: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU')]
I0213 15:12:34.390356 140653502825600 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x7feb559ce320>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=False)
Traceback (most recent call last):
File "/home/penta/automl/efficientdet/tf2/train.py", line 325, in
app.run(main)
File "/home/penta/.local/lib/python3.10/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/home/penta/.local/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/home/penta/automl/efficientdet/tf2/train.py", line 255, in main
model = train_lib.EfficientDetNetTrain(config=config)
File "/home/penta/automl/efficientdet/tf2/train_lib.py", line 474, in init
super().init(*args, **kwargs)
File "/home/penta/automl/efficientdet/tf2/efficientdet_keras.py", line 819, in init
self.backbone = backbone_factory.get_model(
File "/home/penta/automl/efficientdet/backbone/backbone_factory.py", line 80, in get_model
return efficientnet_model.Model(blocks_args, global_params, model_name)
File "/home/penta/automl/efficientdet/backbone/efficientnet_model.py", line 633, in init
self._build()
File "/home/penta/automl/efficientdet/backbone/efficientnet_model.py", line 644, in _build
self._stem = Stem(self._global_params, self._blocks_args[0].input_filters)
File "/home/penta/automl/efficientdet/backbone/efficientnet_model.py", line 520, in init
self._bn = global_params.batch_norm(
TypeError: 'NoneType' object is not callable