Segmentation fault (core dumped) #198
Comments
|
Hi @Rayhane-mamah |
|
Hi @Rayhane-mamah, unfortunately a crash like this deep in AMD's driver doesn't give us a lot to go on! I've been investigating some potential reasons, but have nothing to report yet. Thanks for your patience. |
|
Hi @DuncanMcBain, thank you for your consideration. I understand, if there is anything I can help with, keep me informed. |
|
There is one thing you could try, actually: there was a bug in the ComputeCpp SDK recently that meant certain devices would crash while trying to build the kernel in one sample code, in the same way we see here. The sample "gaussian blur" at revision cf21955 exhibits the bug. Could you try building and running the gaussian blur sample at this revision? If it passes, then fair enough, we have no new information. However, if it fails, then we have a hint as to why this might be happening! |
|
Sure thing @DuncanMcBain, I will take care of it later tonight, or tomorrow at worst. |
|
@Rayhane-mamah ping? |
|
I found some problems building computecpp application (some gcc version problem or something like that) but I'm working on it, I'm sorry for the delay, I am a bit busy with work. |
|
@Rayhane-mamah Yes that would be the right thing to do, but specifically the gaussian blur version at the changelist Duncan mentioned. |
|
@Rayhane-mamah is that still a problem? |
Hello.
First of all, thank you for all the effort on working on tensorflow opencl support. I have looked in the following issue for a long time now without results, here it is:
After successfully compiling tensorflow from source using the following tutorial, testing the library on the mnist example (provided in the tutorial) throws the following segmentation fault (core dumped)
using python convolutional.py
2018-01-05 17:32:50.526588: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:70] Found following OpenCL devices:2018-01-05 17:32:50.526648: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:72] id: 0, type: GPU, name: Hainan, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILESegmentation fault (core dumped)using gdb i can get the following error message and back trace (if we can call it that):
Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffbb7fe700 (LWP 8436)] 0x00007fffe48087a6 in ?? () from /usr/lib/libamdocl64.so(gdb) bt#0 0x00007fffe48087a6 in ?? () from /usr/lib/libamdocl64.so#1 0x6f736e65545f5f6e in ?? ()#2 0x5f74736e6f635f72 in ?? ()#3 0x315f5f74616f6c66 in ?? ()#4 0x6e6f6c5f5f315f5f in ?? ()#5 0x5f5f36315f5f5f67 in ?? ()#6 0x6e696f50656b614d in ?? ()#7 0x6f635f5f5f726574 in ?? ()#8 0x73616c635f74736e in ?? ()#9 0x5f6e656769455f73 in ?? ()#10 0x42726f736e65545f in ?? ()#11 0x7473616364616f72 in ?? ()#12 0x6f635f704f676e69 in ?? ()#13 0x757274735f74736e in ?? ()#14 0x6e656769455f7463 in ?? ()#15 0x73657a6953445f5f in ?? ()#16 0x315f5f676e6f6c5f in ?? ()#17 0x74736e6f635f5f5f in ?? ()#18 0x455f7373616c635f in ?? ()#19 0x65545f5f6e656769 in ?? ()#20 0x68736552726f736e in ?? ()#21 0x5f704f676e697061 in ?? ()#22 0x74735f74736e6f63 in ?? ()`
My gpu is the AMD Radeon R5 M330 (also tried on the R5 M335) and i use the fglrx drivers provided here (As far as i checked, no other fglrx drivers work with my gpu).
Tensorflow (1.4.0) was built using dev/amd_gpu branch, bazel 0.9.0 (also tried 0.5 and later), computecpp 0.5.0 (also tried 0.3.0 and later), gcc/g++ 4.8.4 on a ubuntu x86_64 14.04.5 LTS fresh installation (also tried on 14.04.4, both kernels Linux 4.2.0-42-generic) with Anaconda 5.0.1 (python 3.6) installed (also tried without anaconda, python 3.4).
Following are the reports of clinfo command results:
`Number of platforms: 1
Platform Profile: FULL_PROFILE
Platform Version: OpenCL 2.0 AMD-APP (1912.5)
Platform Name: AMD Accelerated Parallel Processing
Platform Vendor: Advanced Micro Devices, Inc.
Platform Extensions: cl_khr_icd cl_amd_event_callback cl_amd_offline_devices
Platform Name: AMD Accelerated Parallel Processing
Number of devices: 2
Device Type: CL_DEVICE_TYPE_GPU
Vendor ID: 1002h
Board name: AMD Radeon (TM) R5 M330
Device Topology: PCI[ B#3, D#0, F#0 ]
Max compute units: 5
Max work items dimensions: 3
Max work items[0]: 256
Max work items[1]: 256
Max work items[2]: 256
Max work group size: 256
Preferred vector width char: 4
Preferred vector width short: 2
Preferred vector width int: 1
Preferred vector width long: 1
Preferred vector width float: 1
Preferred vector width double: 1
Native vector width char: 4
Native vector width short: 2
Native vector width int: 1
Native vector width long: 1
Native vector width float: 1
Native vector width double: 1
Max clock frequency: 800Mhz
Address bits: 32
Max memory allocation: 1173612288
Image support: Yes
Max number of images read arguments: 128
Max number of images write arguments: 8
Max image 2D width: 16384
Max image 2D height: 16384
Max image 3D width: 2048
Max image 3D height: 2048
Max image 3D depth: 2048
Max samplers within kernel: 16
Max size of kernel argument: 1024
Alignment (bits) of base address: 2048
Minimum alignment (bytes) for any datatype: 128
Single precision floating point capability
Denorms: No
Quiet NaNs: Yes
Round to nearest even: Yes
Round to zero: Yes
Round to +ve and infinity: Yes
IEEE754-2008 fused multiply-add: Yes
Cache type: Read/Write
Cache line size: 64
Cache size: 16384
Global memory size: 1797222336
Constant buffer size: 65536
Max number of constant args: 8
Local memory type: Scratchpad
Local memory size: 32768
Max pipe arguments: 0
Max pipe active reservations: 0
Max pipe packet size: 0
Max global variable size: 0
Max global variable preferred total size: 0
Max read/write image args: 0
Max on device events: 0
Queue on device max size: 0
Max on device queues: 0
Queue on device preferred size: 0
SVM capabilities:
Coarse grain buffer: No
Fine grain buffer: No
Fine grain system: No
Atomics: No
Preferred platform atomic alignment: 0
Preferred global atomic alignment: 0
Preferred local atomic alignment: 0
Kernel Preferred work group size multiple: 64
Error correction support: 0
Unified memory for Host and Device: 0
Profiling timer resolution: 1
Device endianess: Little
Available: Yes
Compiler available: Yes
Execution capabilities:
Execute OpenCL kernels: Yes
Execute native function: No
Queue on Host properties:
Out-of-Order: No
Profiling : Yes
Queue on Device properties:
Out-of-Order: No
Profiling : No
Platform ID: 0x7f8d27032a18
Name: Hainan
Vendor: Advanced Micro Devices, Inc.
Device OpenCL C version: OpenCL C 1.2
Driver version: 1912.5 (VM)
Profile: FULL_PROFILE
Version: OpenCL 1.2 AMD-APP (1912.5)
Extensions: cl_khr_fp64 cl_amd_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_gl_sharing cl_ext_atomic_counters_32 cl_amd_device_attribute_query cl_amd_vec3 cl_amd_printf cl_amd_media_ops cl_amd_media_ops2 cl_amd_popcnt cl_khr_image2d_from_buffer cl_khr_spir cl_khr_gl_event
Device Type: CL_DEVICE_TYPE_CPU
Vendor ID: 1002h
Board name:
Max compute units: 4
Max work items dimensions: 3
Max work items[0]: 1024
Max work items[1]: 1024
Max work items[2]: 1024
Max work group size: 1024
Preferred vector width char: 16
Preferred vector width short: 8
Preferred vector width int: 4
Preferred vector width long: 2
Preferred vector width float: 8
Preferred vector width double: 4
Native vector width char: 16
Native vector width short: 8
Native vector width int: 4
Native vector width long: 2
Native vector width float: 8
Native vector width double: 4
Max clock frequency: 2739Mhz
Address bits: 64
Max memory allocation: 2147483648
Image support: Yes
Max number of images read arguments: 128
Max number of images write arguments: 64
Max image 2D width: 8192
Max image 2D height: 8192
Max image 3D width: 2048
Max image 3D height: 2048
Max image 3D depth: 2048
Max samplers within kernel: 16
Max size of kernel argument: 4096
Alignment (bits) of base address: 1024
Minimum alignment (bytes) for any datatype: 128
Single precision floating point capability
Denorms: Yes
Quiet NaNs: Yes
Round to nearest even: Yes
Round to zero: Yes
Round to +ve and infinity: Yes
IEEE754-2008 fused multiply-add: Yes
Cache type: Read/Write
Cache line size: 64
Cache size: 32768
Global memory size: 5995737088
Constant buffer size: 65536
Max number of constant args: 8
Local memory type: Global
Local memory size: 32768
Max pipe arguments: 16
Max pipe active reservations: 16
Max pipe packet size: 2147483648
Max global variable size: 1879048192
Max global variable preferred total size: 1879048192
Max read/write image args: 64
Max on device events: 0
Queue on device max size: 0
Max on device queues: 0
Queue on device preferred size: 0
SVM capabilities:
Coarse grain buffer: No
Fine grain buffer: No
Fine grain system: No
Atomics: No
Preferred platform atomic alignment: 0
Preferred global atomic alignment: 0
Preferred local atomic alignment: 0
Kernel Preferred work group size multiple: 1
Error correction support: 0
Unified memory for Host and Device: 1
Profiling timer resolution: 1
Device endianess: Little
Available: Yes
Compiler available: Yes
Execution capabilities:
Execute OpenCL kernels: Yes
Execute native function: Yes
Queue on Host properties:
Out-of-Order: No
Profiling : Yes
Queue on Device properties:
Out-of-Order: No
Profiling : No
Platform ID: 0x7f8d27032a18
Name: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
Vendor: GenuineIntel
Device OpenCL C version: OpenCL C 1.2
Driver version: 1912.5 (sse2,avx)
Profile: FULL_PROFILE
Version: OpenCL 1.2 AMD-APP (1912.5)
Extensions: cl_khr_fp64 cl_amd_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_gl_sharing cl_ext_device_fission cl_amd_device_attribute_query cl_amd_vec3 cl_amd_printf cl_amd_media_ops cl_amd_media_ops2 cl_amd_popcnt cl_khr_spir cl_khr_gl_event
`
also the computecpp_info:
`********************************************************************************
ComputeCpp Info (CE 0.5.0)
Toolchain information:
GLIBC version: 2.19
GLIBCXX: 20150426
This version of libstdc++ is supported.
Device Info:
Discovered 2 devices matching:
platform :
device type :
Device 0:
Device is supported : UNTESTED - Device not tested on this OS
CL_DEVICE_NAME : Hainan
CL_DEVICE_VENDOR : Advanced Micro Devices, Inc.
CL_DRIVER_VERSION : 1912.5 (VM)
CL_DEVICE_TYPE : CL_DEVICE_TYPE_GPU
Device 1:
Device is supported : UNTESTED - Device running untested driver
CL_DEVICE_NAME : Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
CL_DEVICE_VENDOR : GenuineIntel
CL_DRIVER_VERSION : 1912.5 (sse2,avx)
CL_DEVICE_TYPE : CL_DEVICE_TYPE_CPU
If you encounter problems when using any of these OpenCL devices, please consult
this website for known issues:
https://computecpp.codeplay.com/releases/v0.5.0/platform-support-notes
`
for the ./configure part we tried different stuff with no results:
PS1:
Back in august, when your implementation was still at tensorflow 1.0, everything used to work fine ( under eigen_mehdi branch, this problem started once we moved to tensorflow 1.1 if i'm not mistaking. )
PS2:
while building tensorflow using dev/amd_gpu branch with Anaconda, we start by getting the following error:
ERROR: /home/silvershade/Downloads/tensorflow-master/tensorflow/tools/pip_package/BUILD:131:1 undeclared inclusion(s) in rule '@protobuf//:python/google/protobuf/pyext/_message.so':this rule is missing dependency declarations for the following files included by 'external/protobuf/python/google/protobuf/pyext/descriptor_containers.cc':'/home/silvershade/anaconda3/include/python3.6m/Python.h''/home/silvershade/anaconda3/include/python3.6m/patchlevel.h''/home/silvershade/anaconda3/include/python3.6m/pyconfig.h''/home/silvershade/anaconda3/include/python3.6m/pymacconfig.h''/home/silvershade/anaconda3/include/python3.6m/pyport.h''/home/silvershade/anaconda3/include/python3.6m/pymath.h'...
we fixed that by adding
cxx_builtin_include_directory: "/home/silvershade/anaconda3/include/python3.6m"into the following file /home/silvershade/tensorflow/third_party/sycl/crosstool/CROSSTOOL.tpl (in case this has anything to do with the issue at hand).I hope this is enough information to go with. Thanks in advance for your answers.
I'm a desperate man in a desperate situation. (ToT)
The text was updated successfully, but these errors were encountered: