WIP: Make some changes to build under the new module system

[scottwittenburg]

PLEASE DO NOT MERGE THIS! The purpose of this WIP PR is just to show changes against the vtk-opengl2-build branch.

[scottwittenburg]

@danlipsa Do you recognize the issue in the attached log file? It seems that clang compiler (which seems quite old to me) can't compile a simple test program, so the conda build vtk-cdat doesn't get very far.

conda-build-vtk-cdat-01.log

[danlipsa]

I'll take a look at my compilation environment tonight when I have access to my laptop.

[scottwittenburg]

@danlipsa Nevermind about that issue in the log I uploaded, @doutriaux1 pointed out that the issue might be related to a change he made in your branch here, and indeed changing that back fixed the problem for me.

Now, however, I'm curious about something else. I wonder why during the conda build process, cmake picked up my homebrew python instead of the one in the conda build environment? Do you know what's the right way to make sure it picks the right python? I could try setting the cmake variable PYTHON_EXECUTABLE to something that's based on the ${prefix}, but I'm wondering if you have run into this, and how you got around it.

[scottwittenburg]

So this is kind of promising, but maybe I'm still missing something. I have used the recipe as it exists in this branch to build vtk-cdat on OSX. As you can see from my change to the meta.yaml.in, the VTK I picked is one from my fork of VTK on gitlab, where I've been debugging the VTK_PYTHON_OPTIONAL_LINK feature.

Regarding my question above, I did figure out a way to make sure cmake found the python in my conda build environment, you can see it in the latest commit on this PR. I pushed the packages that were built to anaconda.org under my account scottwittenburg. Then I tested as follows (adding any more cdat packages to the conda create command resulted in the wrong vtk-cdat getting picked):

$ conda create -n cdat-test-scotts -c conda-forge "python=3.6"
$ source activate cdat-test-scotts
$ conda install -c scottwittenburg vtk-cdat
$ python
Python 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 02:16:08) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import vtk
>>>

No message related to ThreadState or anything weird. I also used the python from that environment to run some vtk python examples I have sitting around related to Context2D stuff without any issue. Do you guys (@danlipsa @doutriaux1 ) think this means we could be past this weird issue we've been having?

[scottwittenburg]

Before we get too excited, let's see what happens with the VTK buildbots. I have created a WIP MR on VTK for the change and triggered testing:

https://gitlab.kitware.com/vtk/vtk/merge_requests/5275

[danlipsa]

also try the test that produces the symbol pi. This is one of the vcs tests which I modified to run directly.

mpl.zip

[scottwittenburg]

When I try to install vcs and cdms2 into the testing environment described above, it wants to update the vtk-cdat version. I guess that's a problem with the label/name on the vtk-cdat package I created.

https://anaconda.org/scottwittenburg/vtk-cdat

I'm not sure how it picked that version it did (v6.2.0.13540.8.0.2019.01.30.12.10.gaf2ae7bf41).

I wonder if there's some other way to verify the matplotlib mathtext rendering?

[danlipsa]

That is strange. Did you installed from nightly? I don't know of another way to test the matplotlib stuff. Note that the VTK tests are different as they are C++. We could add a python VTK test - that would do it.

[scottwittenburg]

@danlipsa You probably saw the issue I made a bit ago on VTK regarding the python test of some matplotlib functionality? I ended up down that road as a result of your message above, and while there was that little sidetrack (segfault depending on how modules are imported), it did eventually validate that the build worked running a matplotlib example under python 3 on OSX.

So I figured out how the packages had been mislabeled, and built them again. This time I can install a test environment as follows:

conda create -n cdat-test-scotts -c scottwittenburg -c cdat/label/nightly -c conda-forge "python=3.6" vcs cdat cdms2 testsrunner

And once I got the sample data downloaded, I ran the test which I recall invokes the math text rendering capability:

nosetests -s test/test_vcs_1D_mintics.py

That test passed, and I'm currently running the entire test suite there. Do you you think you could give my OSX package a try too (just paste the command above to create the test env)? And @doutriaux1 if you want to validate the package too, that could be nice. If it's working for everyone, then we could go ahead and:

1. Review/accept my VTK MR to get that into master
2. Maybe make some changes to this PR until it looks how we want
3. Review/accept this PR
4. Build new packages for all os/arch and python versions

Or maybe 3 and 4 should be switched so we don't make any changes until we can validate all the packages on CircleCI.

[scottwittenburg]

Oh, and I forgot that with the latest VTK master, we picked up some fix which resulted in some of the isofill plots (plots which had extraneous triangles) getting fixed. So we'll need matching vcs and uvcdat-testdata topics for that reason. Although it's just baselines we really need, does that require a vcs topic in order to be tested?

[danlipsa]

@scottwittenburg Sure, I can install and try out your packages tonight when I have access to my laptop.

[danlipsa]

For a simple boxfill test I get:

(cdat-test-scotts) [~/projects/cdat/vcs (master $%=)]$ python t.py
Fatal error in MPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(474)..............:
MPID_Init(190).....................: channel initialization failed
MPIDI_CH3_Init(89).................:
MPID_nem_init(320).................:
MPID_nem_tcp_init(173).............:
MPID_nem_tcp_get_business_card(420):
MPID_nem_tcp_init(379).............: gethostbyname failed, brasov.local (errno 1)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=3191311
:
system msg for write_line failure : Bad file descriptor

[doutriaux1]

I have seen this when i'm under vpn. @dnadeau4 we should really investigate this.

[scottwittenburg]

That's interesting, I didn't know we used mpi in there anywhere. But I thought the problem before was that you would get the "invalid thread state" or something as soon as you tried to import vtk.

[dnadeau4]

@doutriaux1 I cannot reproduce the MPI error. Any scripts somewhere where it happens?

[danlipsa]

@dnadeau4 Attached is the file generated with conda list --explicit
cdat-test-scotts.txt

and the python script

t.txt

[scottwittenburg]

Ok @danlipsa as you've probably seen Utkarsh encountered an error when build VTK inside of ParaView, caused by the re-enabled VTK_PYTHON_OPTIONAL_LINK flag:

https://gitlab.kitware.com/vtk/vtk/merge_requests/5275#note_535542

I've made a new VTK branch that defaults that flag to OFF instead, and I've updated my branch here to reflect that. When you get a few minutes, do you think you could trigger the linux builds using that and see if you encounter a similar error to the one Utkarsh saw? If so, it might be fixable by changing this recipe to turn that flag OFF. Ben thought it could be a python2 vs python3 issue, so doing both linux build would be important. Thanks Dan!

By the way @doutriaux1 I've also pushed new osx packages based on the VTK branch I mentioned above (pushed them to my channel "scottwittenburg", see above for a command line to install). I've tested the py3 one locally, and it seems to pass the tests except for the isofill ones I showed you last week. If you get a chance to try one of them out, that could be good. Make sure the version you get looks like: vtk-cdat-8.2.0.1428.8.0.2019.03.12.17.12. Thanks!

[danlipsa]

@scottwittenburg I get:

-- Could NOT find Python3 (missing: Python3_LIBRARY Development) (found suitable version "3.6.7", minimum required is "3.2")
CMake Error at CMake/vtkModule.cmake:3524 (message):
Could not find the Python3 external dependency.
Call Stack (most recent call first):
Utilities/Python/CMakeLists.txt:44 (vtk_module_find_package)

@mathstuf Any ideas?

[mathstuf]

Seems that you're missing a libpython?

[scottwittenburg]

Thanks @danlipsa I pushed another commit that should fix that issue you mentioned above. I had meant to put an |so in that regular expression, but then forgot to do it before pushing. Just now I lost an hour trying to find a way to do it in a single line, see my commit message for the reason why I couldn't.

I wonder if you could trigger the build again now so we can find out if we have the same problem Utkarsh had when building VTK within ParaView?

[danlipsa]

Did you push anything to the VTK branch?
python-optional-link-default-off
I don't see anything new there. The last commit is:
066c58393beafcd32074341ef741bbaa62dbd332
from Mar 12

[scottwittenburg]

No I didn't, but all that branch did compared to master was make the default OFF, and this build script turns it on explicitly. I'm just wondering whether the problem Utkarsh faced affects us at all.

[danlipsa]

So, should I update the cdat-conda-recipes vtk-new-module-system and then rebuild?

[danlipsa]

I just did that.

[scottwittenburg]

That's what I was thinking. What happened this time?

[danlipsa]

Now it finished - it built two new packages for 3.6

[scottwittenburg]

Thanks @danlipsa, if you push them, I'm happy to give them a test drive to see whether they're working.

[danlipsa]

Just pushed them to my channel danlipsa. Look for 8.2.0.1428 in the file name.

[scottwittenburg]

Cool, thanks! I'll let you know what happens.

[danlipsa]

@scottwittenburg Why don't you test the packages for 2.7 as well?

[scottwittenburg]

@danlipsa FYI:

Just pushed them to my channel danlipsa. Look for 8.2.0.1428 in the file name.

I've tested the non-mesa version of this package you pushed for linux, and although I didn't run all the tests, the package seems to work fine. I ran test_vcs_basic_gms.py as well as test_vcs_1D_mintics.py (which exercises the matplotlib math text rendering). Both passed.

Why don't you test the packages for 2.7 as well?

I'm not sure I follow you here. I don't see any python 2.7 package for linux on your channel, except some old ones (vtk-cdat-8.2.0rc2.8.1-...) which won't be testing the VTK_PYTHON_OPTIONAL_LINK.