Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Training efficientdet-d0 with custom datasets #238

Closed
molyswu opened this issue Apr 15, 2020 · 5 comments
Closed

Training efficientdet-d0 with custom datasets #238

molyswu opened this issue Apr 15, 2020 · 5 comments

Comments

@molyswu
Copy link

molyswu commented Apr 15, 2020

Tred to training with custom datasets :

python main.py --mode=train_and_eval --num_classes=8 --training_file_pattern=tfrecord/train/train*.tfrecord --validation_file_pattern=tfrecord/val/val*.tfrecord --val_json_file=annotations/instances_minival2014.json --model_name=efficientdet-d0 --model_dir=/tmp/efficientdet-d0-scratch --ckpt=efficientdet-d0 --train_batch_size=8 --eval_batch_size=8 --eval_samples=1024 --num_examples_per_epoch=15717 --num_epochs=1 --hparams="use_bfloat16=false,num_classes=8,moving_average_decay=0" --use_tpu=False

15 11:29:47.601560 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode6/op_after_combine11/bn/gamma/Momentum (fpn_cells/cell_2/fnode6/op_after_combine11/bn/gamma/Momentum) -- not in ckpt
I0415 11:29:47.601620 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode6/op_after_combine11/bn/beta/Momentum (fpn_cells/cell_2/fnode6/op_after_combine11/bn/beta/Momentum) -- not in ckpt
I0415 11:29:47.601680 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode7/WSM/Momentum (fpn_cells/cell_2/fnode7/WSM/Momentum) -- not in ckpt
I0415 11:29:47.601739 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode7/WSM_1/Momentum (fpn_cells/cell_2/fnode7/WSM_1/Momentum) -- not in ckpt
I0415 11:29:47.601799 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode7/op_after_combine12/conv/depthwise_kernel/Momentum (fpn_cells/cell_2/fnode7/op_after_combine12/conv/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.601859 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode7/op_after_combine12/conv/pointwise_kernel/Momentum (fpn_cells/cell_2/fnode7/op_after_combine12/conv/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.601919 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode7/op_after_combine12/conv/bias/Momentum (fpn_cells/cell_2/fnode7/op_after_combine12/conv/bias/Momentum) -- not in ckpt
I0415 11:29:47.601978 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode7/op_after_combine12/bn/gamma/Momentum (fpn_cells/cell_2/fnode7/op_after_combine12/bn/gamma/Momentum) -- not in ckpt
I0415 11:29:47.602038 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode7/op_after_combine12/bn/beta/Momentum (fpn_cells/cell_2/fnode7/op_after_combine12/bn/beta/Momentum) -- not in ckpt
I0415 11:29:47.602097 140068141688640 utils.py:107] skip class_net/class-0/depthwise_kernel/Momentum (class_net/class-0/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.602158 140068141688640 utils.py:107] skip class_net/class-0/pointwise_kernel/Momentum (class_net/class-0/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.602234 140068141688640 utils.py:107] skip class_net/class-0/bias/Momentum (class_net/class-0/bias/Momentum) -- not in ckpt
I0415 11:29:47.602357 140068141688640 utils.py:107] skip class_net/class-0-bn-3/gamma/Momentum (class_net/class-0-bn-3/gamma/Momentum) -- not in ckpt
I0415 11:29:47.602454 140068141688640 utils.py:107] skip class_net/class-0-bn-3/beta/Momentum (class_net/class-0-bn-3/beta/Momentum) -- not in ckpt
I0415 11:29:47.602572 140068141688640 utils.py:107] skip class_net/class-1/depthwise_kernel/Momentum (class_net/class-1/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.602673 140068141688640 utils.py:107] skip class_net/class-1/pointwise_kernel/Momentum (class_net/class-1/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.602771 140068141688640 utils.py:107] skip class_net/class-1/bias/Momentum (class_net/class-1/bias/Momentum) -- not in ckpt
I0415 11:29:47.602869 140068141688640 utils.py:107] skip class_net/class-1-bn-3/gamma/Momentum (class_net/class-1-bn-3/gamma/Momentum) -- not in ckpt
I0415 11:29:47.602966 140068141688640 utils.py:107] skip class_net/class-1-bn-3/beta/Momentum (class_net/class-1-bn-3/beta/Momentum) -- not in ckpt
I0415 11:29:47.603064 140068141688640 utils.py:107] skip class_net/class-2/depthwise_kernel/Momentum (class_net/class-2/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.603162 140068141688640 utils.py:107] skip class_net/class-2/pointwise_kernel/Momentum (class_net/class-2/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.603262 140068141688640 utils.py:107] skip class_net/class-2/bias/Momentum (class_net/class-2/bias/Momentum) -- not in ckpt
I0415 11:29:47.603360 140068141688640 utils.py:107] skip class_net/class-2-bn-3/gamma/Momentum (class_net/class-2-bn-3/gamma/Momentum) -- not in ckpt
I0415 11:29:47.603464 140068141688640 utils.py:107] skip class_net/class-2-bn-3/beta/Momentum (class_net/class-2-bn-3/beta/Momentum) -- not in ckpt
I0415 11:29:47.603558 140068141688640 utils.py:96] skip class_net/class-predict/depthwise_kernel/Momentum -- excluded by ./class-predict/.
I0415 11:29:47.603650 140068141688640 utils.py:96] skip class_net/class-predict/pointwise_kernel/Momentum -- excluded by ./class-predict/.
I0415 11:29:47.603742 140068141688640 utils.py:96] skip class_net/class-predict/bias/Momentum -- excluded by ./class-predict/.
I0415 11:29:47.603844 140068141688640 utils.py:107] skip class_net/class-0-bn-4/gamma/Momentum (class_net/class-0-bn-4/gamma/Momentum) -- not in ckpt
I0415 11:29:47.603933 140068141688640 utils.py:107] skip class_net/class-0-bn-4/beta/Momentum (class_net/class-0-bn-4/beta/Momentum) -- not in ckpt
I0415 11:29:47.604003 140068141688640 utils.py:107] skip class_net/class-1-bn-4/gamma/Momentum (class_net/class-1-bn-4/gamma/Momentum) -- not in ckpt
I0415 11:29:47.604066 140068141688640 utils.py:107] skip class_net/class-1-bn-4/beta/Momentum (class_net/class-1-bn-4/beta/Momentum) -- not in ckpt
I0415 11:29:47.604130 140068141688640 utils.py:107] skip class_net/class-2-bn-4/gamma/Momentum (class_net/class-2-bn-4/gamma/Momentum) -- not in ckpt
I0415 11:29:47.604193 140068141688640 utils.py:107] skip class_net/class-2-bn-4/beta/Momentum (class_net/class-2-bn-4/beta/Momentum) -- not in ckpt
I0415 11:29:47.604268 140068141688640 utils.py:107] skip class_net/class-0-bn-5/gamma/Momentum (class_net/class-0-bn-5/gamma/Momentum) -- not in ckpt
I0415 11:29:47.604344 140068141688640 utils.py:107] skip class_net/class-0-bn-5/beta/Momentum (class_net/class-0-bn-5/beta/Momentum) -- not in ckpt
I0415 11:29:47.604443 140068141688640 utils.py:107] skip class_net/class-1-bn-5/gamma/Momentum (class_net/class-1-bn-5/gamma/Momentum) -- not in ckpt
I0415 11:29:47.604542 140068141688640 utils.py:107] skip class_net/class-1-bn-5/beta/Momentum (class_net/class-1-bn-5/beta/Momentum) -- not in ckpt
I0415 11:29:47.604641 140068141688640 utils.py:107] skip class_net/class-2-bn-5/gamma/Momentum (class_net/class-2-bn-5/gamma/Momentum) -- not in ckpt
I0415 11:29:47.604739 140068141688640 utils.py:107] skip class_net/class-2-bn-5/beta/Momentum (class_net/class-2-bn-5/beta/Momentum) -- not in ckpt
I0415 11:29:47.604839 140068141688640 utils.py:107] skip class_net/class-0-bn-6/gamma/Momentum (class_net/class-0-bn-6/gamma/Momentum) -- not in ckpt
I0415 11:29:47.604938 140068141688640 utils.py:107] skip class_net/class-0-bn-6/beta/Momentum (class_net/class-0-bn-6/beta/Momentum) -- not in ckpt
I0415 11:29:47.605038 140068141688640 utils.py:107] skip class_net/class-1-bn-6/gamma/Momentum (class_net/class-1-bn-6/gamma/Momentum) -- not in ckpt
I0415 11:29:47.605138 140068141688640 utils.py:107] skip class_net/class-1-bn-6/beta/Momentum (class_net/class-1-bn-6/beta/Momentum) -- not in ckpt
I0415 11:29:47.605237 140068141688640 utils.py:107] skip class_net/class-2-bn-6/gamma/Momentum (class_net/class-2-bn-6/gamma/Momentum) -- not in ckpt
I0415 11:29:47.605336 140068141688640 utils.py:107] skip class_net/class-2-bn-6/beta/Momentum (class_net/class-2-bn-6/beta/Momentum) -- not in ckpt
I0415 11:29:47.605421 140068141688640 utils.py:107] skip class_net/class-0-bn-7/gamma/Momentum (class_net/class-0-bn-7/gamma/Momentum) -- not in ckpt
I0415 11:29:47.605488 140068141688640 utils.py:107] skip class_net/class-0-bn-7/beta/Momentum (class_net/class-0-bn-7/beta/Momentum) -- not in ckpt
I0415 11:29:47.605551 140068141688640 utils.py:107] skip class_net/class-1-bn-7/gamma/Momentum (class_net/class-1-bn-7/gamma/Momentum) -- not in ckpt
I0415 11:29:47.605614 140068141688640 utils.py:107] skip class_net/class-1-bn-7/beta/Momentum (class_net/class-1-bn-7/beta/Momentum) -- not in ckpt
I0415 11:29:47.605678 140068141688640 utils.py:107] skip class_net/class-2-bn-7/gamma/Momentum (class_net/class-2-bn-7/gamma/Momentum) -- not in ckpt
I0415 11:29:47.605741 140068141688640 utils.py:107] skip class_net/class-2-bn-7/beta/Momentum (class_net/class-2-bn-7/beta/Momentum) -- not in ckpt
I0415 11:29:47.605806 140068141688640 utils.py:107] skip box_net/box-0/depthwise_kernel/Momentum (box_net/box-0/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.605870 140068141688640 utils.py:107] skip box_net/box-0/pointwise_kernel/Momentum (box_net/box-0/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.605935 140068141688640 utils.py:107] skip box_net/box-0/bias/Momentum (box_net/box-0/bias/Momentum) -- not in ckpt
I0415 11:29:47.605999 140068141688640 utils.py:107] skip box_net/box-0-bn-3/gamma/Momentum (box_net/box-0-bn-3/gamma/Momentum) -- not in ckpt
I0415 11:29:47.606063 140068141688640 utils.py:107] skip box_net/box-0-bn-3/beta/Momentum (box_net/box-0-bn-3/beta/Momentum) -- not in ckpt
I0415 11:29:47.606127 140068141688640 utils.py:107] skip box_net/box-1/depthwise_kernel/Momentum (box_net/box-1/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.606190 140068141688640 utils.py:107] skip box_net/box-1/pointwise_kernel/Momentum (box_net/box-1/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.606254 140068141688640 utils.py:107] skip box_net/box-1/bias/Momentum (box_net/box-1/bias/Momentum) -- not in ckpt
I0415 11:29:47.606318 140068141688640 utils.py:107] skip box_net/box-1-bn-3/gamma/Momentum (box_net/box-1-bn-3/gamma/Momentum) -- not in ckpt
I0415 11:29:47.606381 140068141688640 utils.py:107] skip box_net/box-1-bn-3/beta/Momentum (box_net/box-1-bn-3/beta/Momentum) -- not in ckpt
I0415 11:29:47.606445 140068141688640 utils.py:107] skip box_net/box-2/depthwise_kernel/Momentum (box_net/box-2/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.606509 140068141688640 utils.py:107] skip box_net/box-2/pointwise_kernel/Momentum (box_net/box-2/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.606585 140068141688640 utils.py:107] skip box_net/box-2/bias/Momentum (box_net/box-2/bias/Momentum) -- not in ckpt
I0415 11:29:47.606654 140068141688640 utils.py:107] skip box_net/box-2-bn-3/gamma/Momentum (box_net/box-2-bn-3/gamma/Momentum) -- not in ckpt
I0415 11:29:47.606718 140068141688640 utils.py:107] skip box_net/box-2-bn-3/beta/Momentum (box_net/box-2-bn-3/beta/Momentum) -- not in ckpt
I0415 11:29:47.606782 140068141688640 utils.py:107] skip box_net/box-predict/depthwise_kernel/Momentum (box_net/box-predict/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.606846 140068141688640 utils.py:107] skip box_net/box-predict/pointwise_kernel/Momentum (box_net/box-predict/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.606911 140068141688640 utils.py:107] skip box_net/box-predict/bias/Momentum (box_net/box-predict/bias/Momentum) -- not in ckpt
I0415 11:29:47.606975 140068141688640 utils.py:107] skip box_net/box-0-bn-4/gamma/Momentum (box_net/box-0-bn-4/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607038 140068141688640 utils.py:107] skip box_net/box-0-bn-4/beta/Momentum (box_net/box-0-bn-4/beta/Momentum) -- not in ckpt
I0415 11:29:47.607116 140068141688640 utils.py:107] skip box_net/box-1-bn-4/gamma/Momentum (box_net/box-1-bn-4/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607175 140068141688640 utils.py:107] skip box_net/box-1-bn-4/beta/Momentum (box_net/box-1-bn-4/beta/Momentum) -- not in ckpt
I0415 11:29:47.607234 140068141688640 utils.py:107] skip box_net/box-2-bn-4/gamma/Momentum (box_net/box-2-bn-4/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607294 140068141688640 utils.py:107] skip box_net/box-2-bn-4/beta/Momentum (box_net/box-2-bn-4/beta/Momentum) -- not in ckpt
I0415 11:29:47.607353 140068141688640 utils.py:107] skip box_net/box-0-bn-5/gamma/Momentum (box_net/box-0-bn-5/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607413 140068141688640 utils.py:107] skip box_net/box-0-bn-5/beta/Momentum (box_net/box-0-bn-5/beta/Momentum) -- not in ckpt
I0415 11:29:47.607472 140068141688640 utils.py:107] skip box_net/box-1-bn-5/gamma/Momentum (box_net/box-1-bn-5/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607531 140068141688640 utils.py:107] skip box_net/box-1-bn-5/beta/Momentum (box_net/box-1-bn-5/beta/Momentum) -- not in ckpt
I0415 11:29:47.607589 140068141688640 utils.py:107] skip box_net/box-2-bn-5/gamma/Momentum (box_net/box-2-bn-5/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607648 140068141688640 utils.py:107] skip box_net/box-2-bn-5/beta/Momentum (box_net/box-2-bn-5/beta/Momentum) -- not in ckpt
I0415 11:29:47.607713 140068141688640 utils.py:107] skip box_net/box-0-bn-6/gamma/Momentum (box_net/box-0-bn-6/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607773 140068141688640 utils.py:107] skip box_net/box-0-bn-6/beta/Momentum (box_net/box-0-bn-6/beta/Momentum) -- not in ckpt
I0415 11:29:47.607832 140068141688640 utils.py:107] skip box_net/box-1-bn-6/gamma/Momentum (box_net/box-1-bn-6/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607892 140068141688640 utils.py:107] skip box_net/box-1-bn-6/beta/Momentum (box_net/box-1-bn-6/beta/Momentum) -- not in ckpt
I0415 11:29:47.607951 140068141688640 utils.py:107] skip box_net/box-2-bn-6/gamma/Momentum (box_net/box-2-bn-6/gamma/Momentum) -- not in ckpt
I0415 11:29:47.608011 140068141688640 utils.py:107] skip box_net/box-2-bn-6/beta/Momentum (box_net/box-2-bn-6/beta/Momentum) -- not in ckpt
I0415 11:29:47.608071 140068141688640 utils.py:107] skip box_net/box-0-bn-7/gamma/Momentum (box_net/box-0-bn-7/gamma/Momentum) -- not in ckpt
I0415 11:29:47.608131 140068141688640 utils.py:107] skip box_net/box-0-bn-7/beta/Momentum (box_net/box-0-bn-7/beta/Momentum) -- not in ckpt
I0415 11:29:47.608190 140068141688640 utils.py:107] skip box_net/box-1-bn-7/gamma/Momentum (box_net/box-1-bn-7/gamma/Momentum) -- not in ckpt
I0415 11:29:47.608249 140068141688640 utils.py:107] skip box_net/box-1-bn-7/beta/Momentum (box_net/box-1-bn-7/beta/Momentum) -- not in ckpt
I0415 11:29:47.608309 140068141688640 utils.py:107] skip box_net/box-2-bn-7/gamma/Momentum (box_net/box-2-bn-7/gamma/Momentum) -- not in ckpt
I0415 11:29:47.608368 140068141688640 utils.py:107] skip box_net/box-2-bn-7/beta/Momentum (box_net/box-2-bn-7/beta/Momentum) -- not in ckpt
INFO:tensorflow:Done calling model_fn.
I0415 11:29:49.044274 140068141688640 estimator.py:1171] Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
I0415 11:29:49.045373 140068141688640 basic_session_run_hooks.py:546] Create CheckpointSaverHook.
WARNING:tensorflow:Issue encountered when serializing edsummaries.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'tuple' object has no attribute 'name'
W0415 11:29:50.779332 140068141688640 meta_graph.py:437] Issue encountered when serializing edsummaries.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'tuple' object has no attribute 'name'
INFO:tensorflow:Graph was finalized.
I0415 11:29:52.982281 140068141688640 monitored_session.py:246] Graph was finalized.
2020-04-15 11:29:52.982752: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-04-15 11:29:53.010787: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2596920000 Hz
2020-04-15 11:29:53.013378: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f61c0000b20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-04-15 11:29:53.013423: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-04-15 11:29:53.272964: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x556164d73ab0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-04-15 11:29:53.273009: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla M60, Compute Capability 5.2
2020-04-15 11:29:53.273020: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (1): Tesla M60, Compute Capability 5.2
2020-04-15 11:29:53.273936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
vice: 1, name: Tesla M60, pci bus id: 0000:07:00.0, compute capability: 5.2)
INFO:tensorflow:Restoring parameters from /tmp/efficientdet-d0-scratch/model.ckpt-6248
I0415 11:29:53.287226 140068141688640 saver.py:1293] Restoring parameters from /tmp/efficientdet-d0-scratch/model.ckpt-6248
INFO:tensorflow:training_loop marked as finished
I0415 11:29:56.248856 140068141688640 error_handling.py:115] training_loop marked as finished
WARNING:tensorflow:Reraising captured error
W0415 11:29:56.249081 140068141688640 error_handling.py:149] Reraising captured error
Traceback (most recent call last):
File "/home/yy/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/yy/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/yy/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [72] rhs shape= [180]
[[{{node save/Assign_222}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ai/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1299, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/yy/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 958, in run
run_metadata_ptr)
File "/home/yy/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1181, in _run
feed_dict_tensor, options, run_metadata)
File "/home/yy/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run
run_metadata)
@mingxingtan
Copy link
Member

Hi @molyswu looks like this is an old issue that should have been fixed a while ago. Could you check out the latest code and retry it?

Since I can run it well in colab now, I am going to close this issue.

@adarsh0906
Copy link

2020-05-22 16:29:19.583405: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
WARNING:tensorflow:From main.py:229: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.

W0522 16:29:22.477168 140217329719168 module_wrapper.py:138] From main.py:229: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.

WARNING:tensorflow:From main.py:234: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.

W0522 16:29:22.477524 140217329719168 module_wrapper.py:138] From main.py:234: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.

WARNING:tensorflow:From main.py:237: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.

W0522 16:29:22.477794 140217329719168 module_wrapper.py:138] From main.py:237: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.

WARNING:tensorflow:From main.py:354: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.

W0522 16:29:22.478182 140217329719168 module_wrapper.py:138] From main.py:354: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.

WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
W0522 16:29:22.478967 140217329719168 tpu_context.py:218] eval_on_tpu ignored because use_tpu is False.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
W0522 16:29:22.497153 140217329719168 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0522 16:29:22.497616 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
2020-05-22 16:29:22.514062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-22 16:29:22.563861: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:22.564640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2020-05-22 16:29:22.564693: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:29:22.808575: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-22 16:29:22.975975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-22 16:29:23.012252: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-22 16:29:23.313184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-22 16:29:23.348768: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-22 16:29:23.888002: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-22 16:29:23.888264: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:23.889110: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:23.889816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
WARNING:tensorflow:From /content/automl/efficientdet/dataloader.py:329: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
W0522 16:29:23.953376 140217329719168 deprecation.py:323] From /content/automl/efficientdet/dataloader.py:329: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
WARNING:tensorflow:From /content/automl/efficientdet/utils.py:320: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
W0522 16:29:36.019176 140217329719168 deprecation.py:323] From /content/automl/efficientdet/utils.py:320: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.tensor_shape_from_node_def_name
W0522 16:29:39.917539 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.tensor_shape_from_node_def_name
Parsing Inputs...
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:138: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.Conv2D instead.
W0522 16:29:40.535063 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:138: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.Conv2D instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
W0522 16:29:40.538732 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:164: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0522 16:29:40.621974 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:164: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:682: separable_conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.SeparableConv2D instead.
W0522 16:29:40.652706 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:682: separable_conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.SeparableConv2D instead.
Parsing Inputs...
Parsing Inputs...
WARNING:tensorflow:From /content/automl/efficientdet/det_model_fn.py:645: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.

W0522 16:30:03.451806 140217329719168 module_wrapper.py:138] From /content/automl/efficientdet/det_model_fn.py:645: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.

2020-05-22 16:30:12.299059: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2300000000 Hz
2020-05-22 16:30:12.299335: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23e1b80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-22 16:30:12.299375: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-05-22 16:30:12.389253: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.390139: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23e19c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-22 16:30:12.390183: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla K80, Compute Capability 3.7
2020-05-22 16:30:12.391280: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.391961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2020-05-22 16:30:12.392049: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:30:12.392115: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-22 16:30:12.392167: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-22 16:30:12.392205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-22 16:30:12.392245: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-22 16:30:12.392284: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-22 16:30:12.392324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-22 16:30:12.392442: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.393222: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.393880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-22 16:30:12.398628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:30:18.007722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-22 16:30:18.007792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-05-22 16:30:18.007811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-05-22 16:30:18.008335: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:18.009186: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:18.009890: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2020-05-22 16:30:18.009957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10634 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
2020-05-22 16:31:05.522371: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.522384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.567831: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.572154: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.590671: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:05.604242: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:05.624042: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.633405: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.652860: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.658814: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:05.681913: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:05.686797: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:05.708839: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.729091: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:05.738880: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:05.759249: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:05.777218: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:05.797153: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:05.816734: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:05.838102: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.846522: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.866338: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.882193: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:05.893664: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:05.913585: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.932648: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:05.950592: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.956992: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:05.974346: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.995711: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.004369: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.022955: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.038539: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.057798: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.075323: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.093582: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.099683: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.126675: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.154887: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.191888: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.199997: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.221942: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[6] = [6] does not index into param shape [5,1]
2020-05-22 16:31:06.239163: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.245122: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.271038: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.275912: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.297768: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.309857: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:06.348741: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.350590: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[26] = [26] does not index into param shape [8,1]
2020-05-22 16:31:06.387384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.388206: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.412207: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.422883: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.452221: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.544299: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.567363: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:06.585986: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.612142: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.625769: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:06.652096: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.667538: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.680634: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.714303: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.737168: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.742907: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.763202: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.786136: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.804955: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.825781: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.834186: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.860635: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.874033: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.890233: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.901098: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.922681: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.932385: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.947384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.967061: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.976468: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.998789: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:07.001308: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.032170: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.052380: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:07.086655: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:07.094043: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:07.118464: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.127837: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.145598: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.160204: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.180313: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.186828: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.207349: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:07.215665: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.241168: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.243895: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.267392: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:07.283598: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:07.294371: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.316382: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:07.326330: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:07.339827: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.368194: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:07.371477: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:07.406615: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:07.411576: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.434634: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [4,1]
2020-05-22 16:31:07.445323: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.462931: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:07.465411: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:07.500710: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.501345: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:07.520868: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.541397: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.544464: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:07.567987: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.589679: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[6] = [6] does not index into param shape [5,1]
2020-05-22 16:31:07.621578: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:07.661384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[26] = [26] does not index into param shape [8,1]
2020-05-22 16:31:07.674124: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:07.681213: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.703896: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.709902: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
WARNING:tensorflow:Reraising captured error
W0522 16:31:08.010707 140217329719168 error_handling.py:149] Reraising captured error
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_11169]]
(1) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 397, in
app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "main.py", line 364, in main
steps=int(FLAGS.num_examples_per_epoch / FLAGS.train_batch_size))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3083, in train
rendezvous.raise_errors()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 150, in raise_errors
six.reraise(typ, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3078, in train
saving_listeners=saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 349, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1182, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1215, in _train_model_default
saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1518, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 778, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1283, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1384, in run
raise six.reraise(*original_exc_info)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1369, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1442, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1200, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 958, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1181, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_11169]]
(1) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.

i tried to train for 4 classes with following tfrecors in zip but i get the above error if my methos are wrong please guide me
vehicle detection and counting.zip

@adarsh0906
Copy link

2020-05-22 16:29:19.583405: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
WARNING:tensorflow:From main.py:229: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.

W0522 16:29:22.477168 140217329719168 module_wrapper.py:138] From main.py:229: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.

WARNING:tensorflow:From main.py:234: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.

W0522 16:29:22.477524 140217329719168 module_wrapper.py:138] From main.py:234: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.

WARNING:tensorflow:From main.py:237: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.

W0522 16:29:22.477794 140217329719168 module_wrapper.py:138] From main.py:237: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.

WARNING:tensorflow:From main.py:354: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.

W0522 16:29:22.478182 140217329719168 module_wrapper.py:138] From main.py:354: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.

WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
W0522 16:29:22.478967 140217329719168 tpu_context.py:218] eval_on_tpu ignored because use_tpu is False.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
W0522 16:29:22.497153 140217329719168 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0522 16:29:22.497616 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
2020-05-22 16:29:22.514062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-22 16:29:22.563861: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:22.564640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2020-05-22 16:29:22.564693: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:29:22.808575: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-22 16:29:22.975975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-22 16:29:23.012252: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-22 16:29:23.313184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-22 16:29:23.348768: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-22 16:29:23.888002: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-22 16:29:23.888264: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:23.889110: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:23.889816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
WARNING:tensorflow:From /content/automl/efficientdet/dataloader.py:329: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
W0522 16:29:23.953376 140217329719168 deprecation.py:323] From /content/automl/efficientdet/dataloader.py:329: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
WARNING:tensorflow:From /content/automl/efficientdet/utils.py:320: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
W0522 16:29:36.019176 140217329719168 deprecation.py:323] From /content/automl/efficientdet/utils.py:320: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.tensor_shape_from_node_def_name
W0522 16:29:39.917539 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.tensor_shape_from_node_def_name
Parsing Inputs...
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:138: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.Conv2D instead.
W0522 16:29:40.535063 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:138: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.Conv2D instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
W0522 16:29:40.538732 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:164: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0522 16:29:40.621974 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:164: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:682: separable_conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.SeparableConv2D instead.
W0522 16:29:40.652706 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:682: separable_conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.SeparableConv2D instead.
Parsing Inputs...
Parsing Inputs...
WARNING:tensorflow:From /content/automl/efficientdet/det_model_fn.py:645: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.

W0522 16:30:03.451806 140217329719168 module_wrapper.py:138] From /content/automl/efficientdet/det_model_fn.py:645: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.

2020-05-22 16:30:12.299059: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2300000000 Hz
2020-05-22 16:30:12.299335: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23e1b80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-22 16:30:12.299375: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-05-22 16:30:12.389253: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.390139: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23e19c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-22 16:30:12.390183: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla K80, Compute Capability 3.7
2020-05-22 16:30:12.391280: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.391961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2020-05-22 16:30:12.392049: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:30:12.392115: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-22 16:30:12.392167: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-22 16:30:12.392205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-22 16:30:12.392245: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-22 16:30:12.392284: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-22 16:30:12.392324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-22 16:30:12.392442: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.393222: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.393880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-22 16:30:12.398628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:30:18.007722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-22 16:30:18.007792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-05-22 16:30:18.007811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-05-22 16:30:18.008335: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:18.009186: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:18.009890: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2020-05-22 16:30:18.009957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10634 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
2020-05-22 16:31:05.522371: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.522384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.567831: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.572154: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.590671: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:05.604242: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:05.624042: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.633405: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.652860: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.658814: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:05.681913: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:05.686797: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:05.708839: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.729091: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:05.738880: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:05.759249: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:05.777218: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:05.797153: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:05.816734: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:05.838102: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.846522: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.866338: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.882193: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:05.893664: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:05.913585: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.932648: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:05.950592: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.956992: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:05.974346: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.995711: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.004369: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.022955: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.038539: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.057798: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.075323: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.093582: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.099683: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.126675: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.154887: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.191888: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.199997: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.221942: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[6] = [6] does not index into param shape [5,1]
2020-05-22 16:31:06.239163: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.245122: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.271038: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.275912: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.297768: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.309857: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:06.348741: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.350590: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[26] = [26] does not index into param shape [8,1]
2020-05-22 16:31:06.387384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.388206: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.412207: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.422883: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.452221: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.544299: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.567363: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:06.585986: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.612142: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.625769: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:06.652096: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.667538: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.680634: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.714303: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.737168: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.742907: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.763202: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.786136: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.804955: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.825781: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.834186: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.860635: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.874033: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.890233: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.901098: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.922681: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.932385: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.947384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.967061: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.976468: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.998789: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:07.001308: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.032170: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.052380: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:07.086655: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:07.094043: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:07.118464: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.127837: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.145598: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.160204: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.180313: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.186828: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.207349: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:07.215665: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.241168: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.243895: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.267392: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:07.283598: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:07.294371: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.316382: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:07.326330: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:07.339827: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.368194: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:07.371477: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:07.406615: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:07.411576: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.434634: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [4,1]
2020-05-22 16:31:07.445323: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.462931: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:07.465411: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:07.500710: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.501345: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:07.520868: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.541397: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.544464: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:07.567987: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.589679: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[6] = [6] does not index into param shape [5,1]
2020-05-22 16:31:07.621578: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:07.661384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[26] = [26] does not index into param shape [8,1]
2020-05-22 16:31:07.674124: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:07.681213: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.703896: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.709902: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
WARNING:tensorflow:Reraising captured error
W0522 16:31:08.010707 140217329719168 error_handling.py:149] Reraising captured error
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_11169]]
(1) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 397, in
app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "main.py", line 364, in main
steps=int(FLAGS.num_examples_per_epoch / FLAGS.train_batch_size))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3083, in train
rendezvous.raise_errors()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 150, in raise_errors
six.reraise(typ, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3078, in train
saving_listeners=saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 349, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1182, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1215, in _train_model_default
saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1518, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 778, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1283, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1384, in run
raise six.reraise(*original_exc_info)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1369, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1442, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1200, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 958, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1181, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_11169]]
(1) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.

i tried to train for 4 classes with following tfrecors in zip but i get the above error if my methos are wrong please guide me
vehicle detection and counting.zip

sorry i found the problem that was causing the error i rectified it the problem is with tfrecords file

@Li505358678
Copy link

2020-05-22 16:29:19.583405: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
WARNING:tensorflow:From main.py:229: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.
W0522 16:29:22.477168 140217329719168 module_wrapper.py:138] From main.py:229: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.
WARNING:tensorflow:From main.py:234: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.
W0522 16:29:22.477524 140217329719168 module_wrapper.py:138] From main.py:234: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.
WARNING:tensorflow:From main.py:237: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.
W0522 16:29:22.477794 140217329719168 module_wrapper.py:138] From main.py:237: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.
WARNING:tensorflow:From main.py:354: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.
W0522 16:29:22.478182 140217329719168 module_wrapper.py:138] From main.py:354: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
W0522 16:29:22.478967 140217329719168 tpu_context.py:218] eval_on_tpu ignored because use_tpu is False.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
W0522 16:29:22.497153 140217329719168 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0522 16:29:22.497616 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
2020-05-22 16:29:22.514062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-22 16:29:22.563861: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:22.564640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2020-05-22 16:29:22.564693: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:29:22.808575: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-22 16:29:22.975975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-22 16:29:23.012252: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-22 16:29:23.313184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-22 16:29:23.348768: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-22 16:29:23.888002: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-22 16:29:23.888264: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:23.889110: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:23.889816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
WARNING:tensorflow:From /content/automl/efficientdet/dataloader.py:329: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
W0522 16:29:23.953376 140217329719168 deprecation.py:323] From /content/automl/efficientdet/dataloader.py:329: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
WARNING:tensorflow:From /content/automl/efficientdet/utils.py:320: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
W0522 16:29:36.019176 140217329719168 deprecation.py:323] From /content/automl/efficientdet/utils.py:320: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.tensor_shape_from_node_def_name
W0522 16:29:39.917539 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.tensor_shape_from_node_def_name
Parsing Inputs...
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:138: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.Conv2D instead.
W0522 16:29:40.535063 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:138: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.Conv2D instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
W0522 16:29:40.538732 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:164: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0522 16:29:40.621974 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:164: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:682: separable_conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.SeparableConv2D instead.
W0522 16:29:40.652706 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:682: separable_conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.SeparableConv2D instead.
Parsing Inputs...
Parsing Inputs...
WARNING:tensorflow:From /content/automl/efficientdet/det_model_fn.py:645: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.
W0522 16:30:03.451806 140217329719168 module_wrapper.py:138] From /content/automl/efficientdet/det_model_fn.py:645: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.
2020-05-22 16:30:12.299059: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2300000000 Hz
2020-05-22 16:30:12.299335: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23e1b80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-22 16:30:12.299375: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-05-22 16:30:12.389253: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.390139: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23e19c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-22 16:30:12.390183: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla K80, Compute Capability 3.7
2020-05-22 16:30:12.391280: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.391961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2020-05-22 16:30:12.392049: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:30:12.392115: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-22 16:30:12.392167: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-22 16:30:12.392205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-22 16:30:12.392245: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-22 16:30:12.392284: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-22 16:30:12.392324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-22 16:30:12.392442: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.393222: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.393880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-22 16:30:12.398628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:30:18.007722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-22 16:30:18.007792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-05-22 16:30:18.007811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-05-22 16:30:18.008335: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:18.009186: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:18.009890: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2020-05-22 16:30:18.009957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10634 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
2020-05-22 16:31:05.522371: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.522384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.567831: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.572154: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.590671: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:05.604242: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:05.624042: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.633405: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.652860: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.658814: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:05.681913: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:05.686797: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:05.708839: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.729091: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:05.738880: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:05.759249: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:05.777218: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:05.797153: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:05.816734: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:05.838102: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.846522: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.866338: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.882193: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:05.893664: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:05.913585: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.932648: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:05.950592: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.956992: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:05.974346: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.995711: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.004369: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.022955: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.038539: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.057798: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.075323: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.093582: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.099683: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.126675: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.154887: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.191888: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.199997: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.221942: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[6] = [6] does not index into param shape [5,1]
2020-05-22 16:31:06.239163: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.245122: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.271038: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.275912: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.297768: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.309857: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:06.348741: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.350590: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[26] = [26] does not index into param shape [8,1]
2020-05-22 16:31:06.387384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.388206: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.412207: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.422883: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.452221: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.544299: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.567363: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:06.585986: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.612142: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.625769: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:06.652096: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.667538: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.680634: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.714303: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.737168: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.742907: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.763202: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.786136: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.804955: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.825781: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.834186: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.860635: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.874033: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.890233: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.901098: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.922681: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.932385: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.947384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.967061: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.976468: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.998789: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:07.001308: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.032170: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.052380: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:07.086655: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:07.094043: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:07.118464: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.127837: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.145598: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.160204: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.180313: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.186828: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.207349: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:07.215665: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.241168: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.243895: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.267392: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:07.283598: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:07.294371: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.316382: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:07.326330: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:07.339827: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.368194: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:07.371477: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:07.406615: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:07.411576: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.434634: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [4,1]
2020-05-22 16:31:07.445323: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.462931: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:07.465411: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:07.500710: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.501345: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:07.520868: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.541397: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.544464: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:07.567987: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.589679: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[6] = [6] does not index into param shape [5,1]
2020-05-22 16:31:07.621578: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:07.661384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[26] = [26] does not index into param shape [8,1]
2020-05-22 16:31:07.674124: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:07.681213: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.703896: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.709902: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
WARNING:tensorflow:Reraising captured error
W0522 16:31:08.010707 140217329719168 error_handling.py:149] Reraising captured error
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_11169]]
(1) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 397, in
app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "main.py", line 364, in main
steps=int(FLAGS.num_examples_per_epoch / FLAGS.train_batch_size))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3083, in train
rendezvous.raise_errors()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 150, in raise_errors
six.reraise(typ, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3078, in train
saving_listeners=saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 349, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1182, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1215, in _train_model_default
saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1518, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 778, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1283, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1384, in run
raise six.reraise(*original_exc_info)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1369, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1442, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1200, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 958, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1181, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_11169]]
(1) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.
i tried to train for 4 classes with following tfrecors in zip but i get the above error if my methos are wrong please guide me
vehicle detection and counting.zip

sorry i found the problem that was causing the error i rectified it the problem is with tfrecords file

what's wrong with your tfrecords file? I have questions just like yours

@hchauhan123
Copy link

@adarsh0906 @molyswu I have a similar error with my tfrecord SKU110K dataset on efficientdet model. Can I get more insight in to how you resolved it. So I can resolve my issue: #1173

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mingxingtan @molyswu @hchauhan123 @adarsh0906 @Li505358678