Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix error caused by proj4 6.0 #31

Closed
deeplycloudy opened this issue May 23, 2019 · 6 comments
Closed

Fix error caused by proj4 6.0 #31

deeplycloudy opened this issue May 23, 2019 · 6 comments

Comments

@deeplycloudy
Copy link
Owner

As discussed in #22, version 6.0 of proj4 causes an error. Need to identify what changed between 5.2 and 6.0 and fix upstream in lmatools.

  File "examples/grid/make_GLM_grids.py", line 301, in <module>
    gridder(glm_filenames, start_time, end_time, **grid_kwargs)
  File "/data/probsevere/ancillary/miniconda3/envs/glmval/lib/python3.6/site-packages/glmtools-0.1.dev0-py3.6.egg/glmtools/grid/make_grids.py", line 826, in grid_GLM_flashes
    outputs = list(map(this_proc_each_grid, subgrids))
  File "/data/probsevere/ancillary/miniconda3/envs/glmval/lib/python3.6/site-packages/glmtools-0.1.dev0-py3.6.egg/glmtools/grid/make_grids.py", line 890, in proc_each_grid
    gridder.process_flashes(glm, **process_flash_kwargs_ij)
  File "/data/probsevere/ancillary/miniconda3/envs/glmval/lib/python3.6/site-packages/glmtools-0.1.dev0-py3.6.egg/glmtools/grid/make_grids.py", line 394, in process_flashes
    nadir_lon=nadir_lon)
  File "/data/probsevere/ancillary/miniconda3/envs/glmval/lib/python3.6/site-packages/glmtools-0.1.dev0-py3.6.egg/glmtools/io/mimic_lma.py", line 109, in read_flashes
    results = list(map(chunk_func,flash_chunks))
  File "/data/probsevere/ancillary/miniconda3/envs/glmval/lib/python3.6/site-packages/glmtools-0.1.dev0-py3.6.egg/glmtools/io/mimic_lma.py", line 162, in fast_fixed_grid_read_chunk
    np.zeros_like(flash_data.flash_x.data)))
  File "/data/probsevere/ancillary/miniconda3/envs/glmval/lib/python3.6/site-packages/lmatools/coordinateSystems.py", line 210, in toECEF
    return proj4.transform(self.fixedgrid, self.ECEFxyz, X, Y, Z)
  File "/data/probsevere/ancillary/miniconda3/envs/glmval/lib/python3.6/site-packages/pyproj/transformer.py", line 367, in transform
    xx=x, yy=y, zz=z, tt=tt, radians=radians, errcheck=errcheck
  File "/data/probsevere/ancillary/miniconda3/envs/glmval/lib/python3.6/site-packages/pyproj/transformer.py", line 180, in transform
    inx, iny, inz=inz, intime=intime, radians=radians, errcheck=errcheck
  File "pyproj/_transformer.pyx", line 107, in pyproj._transformer._Transformer._transform
pyproj.exceptions.ProjError: x,y,z, and time must be same size
@deeplycloudy deeplycloudy mentioned this issue May 23, 2019
Closed
@deeplycloudy deeplycloudy mentioned this issue Jan 17, 2020
Open
@deeplycloudy
Copy link
Owner Author

deeplycloudy commented Feb 20, 2020 •
edited
Loading

There have been some additional changes in pyproj since this bug report in the section where the time size check was added. This commit seems to substantially change the logic.

pyproj 2.0.2 does not have _transformer.pyx; it first appeared in 2.1. The time check did not appear until 2.2.

This conda environment, with pyproj-2.2.2 and proj4-6.1.1 fails on OR_GLM-L2-LCFA_G16_s20200490529*.nc with the error above, using glmtools and lmatools on the isatss branch. pyproj <2.2 did not fail.

blosc                     1.17.1               h4a8c4bd_0    conda-forge
bzip2                     1.0.8                h0b31af3_2    conda-forge
ca-certificates           2019.11.28           hecc5488_0    conda-forge
certifi                   2019.11.28               py36_0    conda-forge
cftime                    1.0.4.2          py36h3b54f70_0    conda-forge
curl                      7.68.0               h8754def_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
freetype                  2.10.0               h24853df_1    conda-forge
glmtools                  0.1.dev0                  dev_0    <develop>
hdf4                      4.2.13            h84186c3_1003    conda-forge
hdf5                      1.10.5          nompi_h3e39495_1104    conda-forge
jpeg                      9c                h1de35cc_1001    conda-forge
kiwisolver                1.1.0            py36ha1b3eb9_0    conda-forge
krb5                      1.16.4               h1752a42_0    conda-forge
libblas                   3.8.0               14_openblas    conda-forge
libcblas                  3.8.0               14_openblas    conda-forge
libcurl                   7.68.0               h709d2b2_0    conda-forge
libcxx                    9.0.1                         1    conda-forge
libedit                   3.1.20170329      hcfe32e1_1001    conda-forge
libffi                    3.2.1             h6de7cb9_1006    conda-forge
libgfortran               4.0.0                         2    conda-forge
liblapack                 3.8.0               14_openblas    conda-forge
libnetcdf                 4.7.3           nompi_hda4e5f1_101    conda-forge
libopenblas               0.3.7                h3d69b6c_7    conda-forge
libpng                    1.6.37               h2573ce8_0    conda-forge
libssh2                   1.8.2                hcdc9a53_2    conda-forge
llvm-openmp               9.0.1                h28b9765_2    conda-forge
lmatools                  0.6a0                    pypi_0    pypi
matplotlib                3.1.3                    py36_0    conda-forge
matplotlib-base           3.1.3            py36h11da6c2_0    conda-forge
mock                      3.0.5                    py36_0    conda-forge
ncurses                   6.1               h0a44026_1002    conda-forge
netcdf4                   1.5.3           nompi_py36hcf2264a_102    conda-forge
numexpr                   2.7.1            py36h4f17bb1_0    conda-forge
numpy                     1.18.1           py36hde6bac1_0    conda-forge
openssl                   1.1.1d               h0b31af3_0    conda-forge
pandas                    1.0.1            py36h4f17bb1_0    conda-forge
pip                       20.0.2                     py_2    conda-forge
proj4                     6.1.1                hca663eb_1    conda-forge
pyclipper                 1.1.0.post1      py36h4a8c4bd_0    conda-forge
pyparsing                 2.4.6                      py_0    conda-forge
pyproj                    2.2.2            py36h9bb365a_0    conda-forge
pytables                  3.6.1            py36h6f8395a_1    conda-forge
python                    3.6.7             h4285619_1006    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
pytz                      2019.3                     py_0    conda-forge
readline                  8.0                  hcfe32e1_0    conda-forge
scipy                     1.4.1            py36h82752d6_0    conda-forge
setuptools                45.2.0                   py36_0    conda-forge
six                       1.14.0                   py36_0    conda-forge
sqlite                    3.30.1               h93121df_0    conda-forge
stormdrain                0.1                      pypi_0    pypi
tk                        8.6.10               hbbe82c9_0    conda-forge
tornado                   6.0.3            py36h0b31af3_4    conda-forge
wheel                     0.34.2                     py_1    conda-forge
xarray                    0.15.0                     py_0    conda-forge
xz                        5.2.4             h1de35cc_1001    conda-forge
zlib                      1.2.11            h0b31af3_1006    conda-forge

@deeplycloudy
Copy link
Owner Author

Updating to proj2.4.2 and proj 6.3.0 does not resolve the problem. Hmm...

@deeplycloudy
Copy link
Owner Author

deeplycloudy commented Feb 20, 2020 •
edited
Loading

glmtools@ugf-newgrid with lmatools@master (from today) fixes the problem, while glmtools@isatss and lmtools@master does have the problem.

Traceback (most recent call last):
  File "/Users/ebruning/code/glmtools/examples/grid/make_GLM_grids.py", line 301, in <module>
    gridder(glm_filenames, start_time, end_time, **grid_kwargs)
  File "/Users/ebruning/code/glmtools/glmtools/grid/make_grids.py", line 826, in grid_GLM_flashes
    outputs = list(map(this_proc_each_grid, subgrids))
  File "/Users/ebruning/code/glmtools/glmtools/grid/make_grids.py", line 890, in proc_each_grid
    gridder.process_flashes(glm, **process_flash_kwargs_ij)
  File "/Users/ebruning/code/glmtools/glmtools/grid/make_grids.py", line 394, in process_flashes
    nadir_lon=nadir_lon)
  File "/Users/ebruning/code/glmtools/glmtools/io/mimic_lma.py", line 112, in read_flashes
    results = list(map(chunk_func,flash_chunks))
  File "/Users/ebruning/code/glmtools/glmtools/io/mimic_lma.py", line 247, in fast_fixed_grid_read_chunk
    flash_target.send((fl_events, fl_flashes))
  File "/Users/ebruning/code/lmatools/lmatools/grid/density_to_files.py", line 215, in flashes_to_frames
    target.send((these_events, these_flashes))
  File "/Users/ebruning/code/lmatools/lmatools/stream/subset.py", line 83, in broadcast
    target.send(stuff)
  File "/Users/ebruning/code/lmatools/lmatools/grid/density_to_files.py", line 291, in project
    points[x_coord], points[y_coord], points[z_coord]))
  File "/Users/ebruning/code/lmatools/lmatools/coordinateSystems.py", line 74, in toECEF
    projectedData = array(proj4.transform(self.ERSlla, self.ERSxyz, lon, lat, alt ))
  File "/Users/ebruning/miniconda3/envs/glmvaloldproj/lib/python3.6/site-packages/pyproj/transformer.py", line 742, in transform
    ).transform(xx=x, yy=y, zz=z, tt=tt, radians=radians, errcheck=errcheck)
  File "/Users/ebruning/miniconda3/envs/glmvaloldproj/lib/python3.6/site-packages/pyproj/transformer.py", line 445, in transform
    errcheck=errcheck,
  File "pyproj/_transformer.pyx", line 432, in pyproj._transformer._Transformer._transform
pyproj.exceptions.ProjError: x,y,z, and time must be same size

Seems like something was fixed in ugf-newgrid that unknowingly resolved this issue? Hmm....

@deeplycloudy
Copy link
Owner Author

deeplycloudy commented Feb 20, 2020 •
edited
Loading

Looking at the traceback, the data structures in glmtools@ugf-newgrid that changed from glmtools@isatss had to have been (fl_events, fl_flashes) in mimic_lma.py. Those come from the _fake_lma_events_from_split_glm_lutevents and _fake_lma_from_glm_lutflashes functions, which definitely changed when comparing across branches. Something in the change to those data structures seems to have helped, though I confess to not knowing what that specific change was.

@deeplycloudy
Copy link
Owner Author

Given the eventual move to ugf-newgrid in #30 for all users, I'm inclined to consider this issue resolved.

@deeplycloudy deeplycloudy mentioned this issue Apr 28, 2020
Open
@deeplycloudy
Copy link
Owner Author

With the fix for lmatools#29 in lmatools#28, I think I now understand why the above changes to glmtools fixed the issue. The newgrid-faster branch removed a bunch of extraneous coord transforms, and so probably simply stopped hitting the code in lmatools.coordinateTransforms that was sensitive to zero-length or scalar arrays.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@deeplycloudy