Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

It seems stop when I train my data #11

Open
frothmoon opened this issue Aug 31, 2018 · 1 comment
Open

It seems stop when I train my data #11

frothmoon opened this issue Aug 31, 2018 · 1 comment

Comments

@frothmoon
Copy link

Output is as follows :

+ set -e
+ export PYTHONUNBUFFERED=True
+ PYTHONUNBUFFERED=True
+ GPU_ID=0
+ DATASET=cow
+ array=($@)
+ len=2
+ EXTRA_ARGS=
+ EXTRA_ARGS_SLUG=
++ date +%Y_%m_%d_%H_%M_%S
+ LOG=logs/FPN_cow.txt.2018_08_31_14_51_58
+ exec
++ tee -a logs/FPN_cow.txt.2018_08_31_14_51_58
tee: logs/FPN_cow.txt.2018_08_31_14_51_58: No such file or directory
+ echo Logging output to logs/FPN_cow.txt.2018_08_31_14_51_58
Logging output to logs/FPN_cow.txt.2018_08_31_14_51_58
+ CUDA_VISIBLE_DEVICES=0
+ python ./tools/train.py
tfrecord path is --> /home/dongpeijie/FPN_TensorFlow/data/tfrecords/cow_train*
/home/dongpeijie/miniconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
model restore from pretrained mode, path is: data/pretrained_weights/resnet_v1_101.ckpt
2018-08-31 14:52:26.532747: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-08-31 14:52:26.532790: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2018-08-31 14:52:26.532798: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2018-08-31 14:52:26.532803: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2018-08-31 14:52:26.532808: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2018-08-31 14:52:28.340651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties:
name: Tesla P100-PCIE-16GB
major: 6 minor: 0 memoryClockRate (GHz) 1.3285
pciBusID 0000:06:00.0
Total memory: 15.89GiB
Free memory: 15.60GiB
2018-08-31 14:52:28.340709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0
2018-08-31 14:52:28.340716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   Y
2018-08-31 14:52:28.340727: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0)
restore model

I have waited for a long time without other output information.

Thank you for helping me solving the problem.

@cathy0522
Copy link

You need to check your file path is correct or not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants