Hough indexing and lazy loading of patterns: Cannot determine Numba type of <class 'dask.array.core.Array'>

[Erlendos12]

Hi! Following the tutorial for Hough indexing at https://kikuchipy.org/en/stable/tutorials/hough_indexing.html and loading patterns lazily using kikuchipy, a numba TypingError occurs (The traceback can be found at the bottom of this post). The loading of a dataset with path named pattern_path and indexing is done by:

try:
     s = kp.load(pattern_path, lazy=True)
except Exception as e:
     raise e

...

indexer = EBSDIndexer(
            phaselist=self.phase_proxys,  # FCC, BCC or both
            vendor="BRUKER",
            PC=self.pc,
            sampleTilt=sample_tilt,
            camElev=camera_tilt,
            rhoMaskFrac=self.rho_mask,
            nBands=self.number_bands,
            patDim=sig_shape,
        )
data, *_ = indexer.index_pats(s.data.reshape(-1, *sig_shape), verbose=1)

Setting lazy to false produces no errors and Hough indexing works. I know it is possible to save a lazy loaded s to a h5-format and then load the dataset by using:

file_h5 = File(path.join(directory_out, "pattern.h5"), mode="r")
dset_h5 = file_h5["Scan 1/EBSD/Data/patterns"]

This works, but saving an additional h5 file is waste of storage space, and I am guessing the pattern is not loaded lazy anymore anyway. I am aware that this is probably a problem with PyEBSDIndex rather than Kikuchipy, but I was wondering if anything can be done to do hough indexing on a lazy loaded dataset. The versions being used are Kikuchipy 0.7.0 and PyEBSDIndex 0.1.1.

This is maybe something coming to version 0.8.0?

The traceback of the error:

Traceback (most recent call last):

  File "c:\EBSD-GUI-erlenmos\scripts\hough_indexing.py", line 315, in hough_indexing_revised

data, *_ = indexer.index_pats(s.data.reshape(-1, *sig_shape), verbose=1)

  File "C:\EBSD-GUI-erlenmos\env\lib\site-packages\pyebsdindex\_ebsd_index_single.py", line 475, in index_pats

bandData = self.bandDetectPlan.find_bands(

  File "C:\EBSD-GUI-erlenmos\env\lib\site-packages\pyebsdindex\band_detect.py", line 322, in find_bands

rdnNorm = self.radonPlan.radon_faster(patterns[chnk[0]:chnk[1],:,:], self.padding, fixArtifacts=False, background=self.backgroundsub)

  File "C:\EBSD-GUI-erlenmos\env\lib\site-packages\pyebsdindex\radon_fast.py", line 178, in radon_faster

counter = self.rdn_loops(image,self.indexPlan,nIm,nPx,indxDim,radon, np.asarray(padding))

  File "C:\EBSD-GUI-erlenmos\env\lib\site-packages\numba\core\dispatcher.py", line 468, in _compile_for_args

error_rewrite(e, 'typing')

  File "C:\EBSD-GUI-erlenmos\env\lib\site-packages\numba\core\dispatcher.py", line 409, in error_rewrite

raise e.with_traceback(None)

numba.core.errors
.
TypingError
:
Failed in nopython mode pipeline (step: nopython frontend)
�[1m�[1mnon-precise type pyobject�[0m
�[0m�[1mDuring: typing of argument at C:\EBSD-GUI-erlenmos\env\lib\site-packages\pyebsdindex\radon_fast.py (193)�[0m
�[1m
File "env\lib\site-packages\pyebsdindex\radon_fast.py", line 193:�[0m
�[1m  def rdn_loops(images,index,nIm,nPx,indxdim,radon, padding):
�[1m    nRho = indxdim[0]
�[0m    �[1m^�[0m�[0m 

This error may have been caused by the following argument(s):
- argument 0: �[1mCannot determine Numba type of <class 'dask.array.core.Array'>�[0m

[hakonanes]

Hi @Erlendos12!

Passing Dask arrays to PyEBSDIndex only works if the GPU is used, i.e. if PyOpenCL is installed. The Numba kernels assume NumPy arrays, hence the error you get.

I've added a very light, experimental wrapper around PyEBSDIndex in EBSD.hough_indexing() in version 0.8. This method throws an error immediately if patterns are lazily loaded but PyOpenCL is not installed:

kikuchipy/kikuchipy/signals/ebsd.py

Lines 1742 to 1757 in fed8f05

	if not _pyebsdindex_installed: # pragma: no cover
	raise ValueError(
	"Hough indexing requires pyebsdindex to be installed. Install it with "
	"pip install pyebsdindex. See "
	"https://kikuchipy.org/en/stable/user/installation.html for details"
	)
	if self._lazy:
	from pyebsdindex import _pyopencl_installed
	if not _pyopencl_installed: # pragma: no cover
	raise ValueError(
	"Hough indexing of lazy signals must use the GPU, which requires "
	"pyopencl to be installed. See "
	"https://documen.tician.de/pyopencl/misc.html for installation "
	"instructions"
	)

Updates to PyEBSDIndex are required if indexing of Dask arrays on CPU is to be supported.

[Erlendos12]

Currently working on my laptop which only has a Intel HD Graphics 520 integrated onto the CPU, so I am thinking about trying it out.

[hakonanes]

The GPU implementation in PyEBSDIndex gives a massive speed-up compared to the CPU. I strongly suggest you install PyOpenCL whenever possible!

[Erlendos12]

Installed it with pip and it works right away! Saw a large speed-up, e.g. on my small test dataset from 20 s to just 3 s. Thank you!

Another problem occurs now when trying to save the crystal map with h5-format, using:

io.save(path.join(self.dir_out, "xmap_hi.h5"), self.xmap)

the application (or python kernel) simply freezes/ hangs. The h5 file is created, but I am unsure if it is finished generating before it freezes.

Saving to an ang-format and generating maps works, so it is not a big deal, but any idea why introducing pyopencl could break saving to h5-format?

[hakonanes]

No, I have not encountered such an issue before. I don’t see how PyOpenCL can affect h5py. h5py just released version 3.8.0 today, did you install that version? If so, writing HDF5 files in orix might not work properly with that version. I don’t know…

[Erlendos12]

I have not updated. My current versions are h5py 3.7.0 and orix 0.10.2. I'll look further into it though, thanks!