Parallel-safe mesh-to-mesh interpolation #128

ddundo · 2024-03-05T15:57:28Z

Closes #39.

Introduces parallel-compatible mesh-to-mesh interpolation (already wrapped up in firedrake as f_dest = Function(V_dest).interpolate(f_src) where two VertexOnlyMeshes are used as intermediaries).

ddundo · 2024-03-05T16:26:58Z

Hi @jwallwork23 and @stephankramer, after reading the whole day about VertexOnlyMeshes and petsc's star forest, I realised that the entire procedure has already been wrapped up super nicely within the usual interpolate and is in the main firedrake branch. So I am creating a super early draft pull request here to just demonstrate how simple it appears to be and to ask for a general strategy before doing more.

The biggest question is what to do with project - do we want to keep both options? I guess this PR should also address #123 and #44. We'd also need to make plotting parallel-compatible (could be addressed in #117). Anything else?

jwallwork23 · 2024-03-06T09:02:22Z

Thanks for this @ddundo. Hopefully you learnt something while looking into VertexOnlyMesh and PETSc star forest!

I noticed the test failures on this PR so I tried out the mesh-to-mesh interpolation functionality in Firedrake myself. It seems like there are still some teething problems. I tried interpolating between two UnitSquareMeshes and got DofNotDefinedErrors. Weirdly, these still persisted even when I tweaked the target mesh so that all of its DoFs lie within the source domain. I then tried a case where the target mesh is already fully embedded within the source mesh, but still got DofNotDefinedErrors for all DoFs in the target space.

Regarding your question, let's not try to fix too many things in one PR. I think we should have both interpolation and projection options available.

jwallwork23 · 2024-03-06T09:04:53Z

goalie/mesh_seq.py

@@ -324,7 +325,9 @@ def get_checkpoints(
                checkpoints.append(
                    AttrDict(
                        {
-                            field: project(sols[field], fs[i + 1])
+                            field: firedrake.Function(fs[i + 1]).interpolate(


Two comments:

I think we should still support project, with an option to switch. I guess interpolate should be the default, if we can get it to work better in parallel.

Do we expect the interpolate method to work for mixed spaces? (I haven't tested it.) The project method doesn't, which is why we had to write an interpolation module for Goalie.

ddundo · 2024-03-06T11:29:16Z

Thanks @jwallwork23! Regarding test failures and your review comment, actually when I run the tests locally only the gray_scott demo fails because interpolate raises the error NotImplementedError: UFL expressions for mixed functions are not yet supported..

So yes, potentially I'll overload interpolate - I still have to test more
About the other errors, I was surprised to see them here since they pass locally. Could you please try running the tests locally as well?

jwallwork23 · 2024-03-06T13:25:26Z

Hm no I get the same errors running locally:

====================================== short test summary info ======================================
FAILED test_demos.py::test_demos[burgers] - subprocess.CalledProcessError: Command '['/home/firedrake/firedrake/bin/python3', '/home/joe/sof...
FAILED test_demos.py::test_demos[burgers_time_integrated] - subprocess.CalledProcessError: Command '['/home/firedrake/firedrake/bin/python3', '/home/joe/sof...
FAILED test_demos.py::test_demos[gray_scott] - subprocess.CalledProcessError: Command '['/home/firedrake/firedrake/bin/python3', '/home/joe/sof...
FAILED test_demos.py::test_demos[burgers-hessian] - subprocess.CalledProcessError: Command '['/home/firedrake/firedrake/bin/python3', '/home/joe/sof...
FAILED test_demos.py::test_demos[burgers2] - subprocess.CalledProcessError: Command '['/home/firedrake/firedrake/bin/python3', '/home/joe/sof...
FAILED test_demos.py::test_demos[burgers_ee] - subprocess.CalledProcessError: Command '['/home/firedrake/firedrake/bin/python3', '/home/joe/sof...
FAILED test_demos.py::test_demos[burgers_oo] - subprocess.CalledProcessError: Command '['/home/firedrake/firedrake/bin/python3', '/home/joe/sof...
================ 7 failed, 121 passed, 3 skipped, 123 warnings in 286.62s (0:04:46) =================

ddundo · 2024-03-06T16:49:36Z

Hmm thanks @jwallwork23, that's odd. I guess you are using the same docker image as the workflow here? Here is the results of my firedrake-status and the output of goalie test suite out.txt

---------------------------------------------------------------------------
|Package             |Branch                        |Revision  |Modified  |
---------------------------------------------------------------------------
|FInAT               |master                        |e3a869d   |False     |
|PyOP2               |master                        |e0a4d3a9  |False     |
|animate             |main                          |b39ff5c   |False     |
|concave_hull        |master                        |bdca55e   |False     |
|fiat                |master                        |dbc1c5d   |False     |
|firedrake           |master                        |63104ed4f |False     |
|goalie              |39_vom_parallel_interpolation |6faac5d   |False     |
|h5py                |firedrake                     |6b512e5e  |False     |
|icepack             |master                        |28eed36   |True      |
|libsupermesh        |master                        |84becef   |False     |
|loopy               |main                          |8158afdb  |False     |
|petsc               |jwallwork23/parmmg-rebased    |30d0933c23|False     |
|pyadjoint           |master                        |2c6614d   |False     |
|pytest-mpi          |main                          |8241bdc   |False     |
|tsfc                |master                        |90c20c5   |False     |
|ufl                 |master                        |054b0617  |False     |
---------------------------------------------------------------------------

I noticed the test failures on this PR so I tried out the mesh-to-mesh interpolation functionality in Firedrake myself. It seems like there are still some teething problems. I tried interpolating between two UnitSquareMeshes and got DofNotDefinedErrors. Weirdly, these still persisted even when I tweaked the target mesh so that all of its DoFs lie within the source domain. I then tried a case where the target mesh is already fully embedded within the source mesh, but still got DofNotDefinedErrors for all DoFs in the target space.

Could you please send me this script?

jwallwork23 · 2024-03-06T17:56:38Z

Yeah, I'm using the docker image locally rather than installing.

Attached scripts here.
interp.tar.gz

ddundo · 2024-03-06T18:01:26Z

Thanks @jwallwork23, these both work for me locally. Could you please check if the docker image needs to be updated? I'm clueless for docker sadly :) but no rush! Sorry for the spam today

jwallwork23 · 2024-03-07T10:14:54Z

Thanks @jwallwork23, these both work for me locally. Could you please check if the docker image needs to be updated? I'm clueless for docker sadly :) but no rush! Sorry for the spam today

Yeah annoyingly the weekly automated Docker build hasn't worked recently for some reason (see mesh-adaptation/animate#79). I'll rebuild it now, push to DockerHub and then rerun the tests here.

ddundo · 2024-03-07T12:16:04Z

Yeah annoyingly the weekly automated Docker build hasn't worked recently for some reason (see pyroteus/animate#79). I'll rebuild it now, push to DockerHub and then rerun the tests here.

Thanks for doing that! Unfortunately there are still some failures here but everything passes on my local install.

jwallwork23 · 2024-03-07T13:47:02Z

Thanks for doing that! Unfortunately there are still some failures here but everything passes on my local install.

No worries. I notice that Gray-Scott is now fixed and all the errors are with Burgers. I wonder if the problem is with the demo...

ddundo · 2024-03-07T13:58:16Z

No worries. I notice that Gray-Scott is now fixed and all the errors are with Burgers. I wonder if the problem is with the demo...

Well let me get docker running locally and investigate - I don't want to waste your time (and I need to finally give docker a go anyway :) ). I will ping you with updates/questions. Thanks for the help!

Closes #105.

Closes #130. File has been renamed VTKFile on the Firedrake side

…polation

ddundo · 2024-03-16T03:19:55Z

No rush at all with this, but it's ready for review when you have time please :) I just wanted to get it out of the way.

A few comments:

I assigned interpolate/project to MeshSeq.transfer but there is already an existing function called transfer in indicate_errors. I don't think it's confusing but feedback is welcome
The methodology in interpolation._transfer_adjoint is unfamiliar to me so please check that it and the docstring make sense

jwallwork23

Really nice work @ddundo, thanks for this. Just a few requests.

goalie/mesh_seq.py

goalie/interpolation.py

ddundo · 2024-03-21T19:14:38Z

Thanks again @jwallwork23, ready for a re-review :)

jwallwork23

Good to go, thanks @ddundo!

ddundo · 2024-03-21T19:19:07Z

@acse-ej321 Just pinging you here to make you aware that interpolation is now used as the main transfer method :) so you might need to pass transfer_method="project" to MeshSeq in your code if you want to go back to projection

(Temporarily) replacing project with interpolate in solve_forward (#39)

aa719f4

ddundo added bug Something isn't working enhancement New feature or request testing Extensions and improvements to the testing infrastructure labels Mar 5, 2024

ddundo self-assigned this Mar 5, 2024

ddundo linked an issue Mar 6, 2024 that may be closed by this pull request

Make counting parallel safe #123

Closed

jwallwork23 reviewed Mar 6, 2024

View reviewed changes

jwallwork23 added this to the Version 1 milestone Mar 6, 2024

jwallwork23 mentioned this pull request Mar 7, 2024

VTK API update #131

Merged

This was referenced Mar 7, 2024

Docker build fails mesh-adaptation/animate#85

Closed

Issue building Docker container on GitHub Actions mesh-adaptation/animate#79

Closed

ddundo removed a link to an issue Mar 16, 2024

Make counting parallel safe #123

Closed

ddundo and others added 6 commits March 16, 2024 02:35

Avoid interpolation deprecation warning (#129)

80da600

Closes #105.

VTK API update (#131)

3b9b37b

Closes #130. File has been renamed VTKFile on the Firedrake side

Overload interpolate for mixed spaces (#39)

34ee5db

#39: Overload interpolate and add generic transfer method

91d0040

#39: Add interpolate as the default mesh-to-mesh transfer method

022936e

#39: Expand tests for interpolation

a00cad2

ddundo force-pushed the 39_vom_parallel_interpolation branch from 39c7313 to a00cad2 Compare March 16, 2024 02:45

Merge remote-tracking branch 'origin/main' into 39_vom_parallel_inter…

4fd86a7

…polation

ddundo marked this pull request as ready for review March 16, 2024 03:10

ddundo requested review from jwallwork23 and acse-ej321 March 16, 2024 03:10

ddundo removed the bug Something isn't working label Mar 16, 2024

jwallwork23 requested changes Mar 20, 2024

View reviewed changes

goalie/mesh_seq.py Outdated Show resolved Hide resolved

goalie/interpolation.py Outdated Show resolved Hide resolved

goalie/interpolation.py Show resolved Hide resolved

goalie/interpolation.py Outdated Show resolved Hide resolved

jwallwork23 mentioned this pull request Mar 21, 2024

Wrapper function/class for mesh-to-mesh interpolation mesh-adaptation/animate#56

Closed

ddundo added 4 commits March 21, 2024 16:09

#39: Remove confusing EventDecorator

76606e8

#39: Raise error for wrong transfer method

98b40fe

#39: Make transfer a private method. Fix docstring

78a2516

Merge branch 'main' into 39_vom_parallel_interpolation

3d0155c

ddundo requested a review from jwallwork23 March 21, 2024 19:14

jwallwork23 approved these changes Mar 21, 2024

View reviewed changes

ddundo merged commit ac3f431 into main Mar 21, 2024
1 check passed

ddundo deleted the 39_vom_parallel_interpolation branch March 21, 2024 19:16

ddundo mentioned this pull request Apr 14, 2024

Burgers demo hangs locally #160

Closed

This was referenced Apr 27, 2024

Enable mass lumping for projection operator mesh-adaptation/animate#115

Merged

Allow passing kwargs to MeshSeq._transfer #185

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallel-safe mesh-to-mesh interpolation #128

Parallel-safe mesh-to-mesh interpolation #128

ddundo commented Mar 5, 2024 •

edited

Loading

ddundo commented Mar 5, 2024

jwallwork23 commented Mar 6, 2024

jwallwork23 Mar 6, 2024

ddundo commented Mar 6, 2024

jwallwork23 commented Mar 6, 2024

ddundo commented Mar 6, 2024

jwallwork23 commented Mar 6, 2024

ddundo commented Mar 6, 2024

jwallwork23 commented Mar 7, 2024

ddundo commented Mar 7, 2024

jwallwork23 commented Mar 7, 2024

ddundo commented Mar 7, 2024

ddundo commented Mar 16, 2024

jwallwork23 left a comment

ddundo commented Mar 21, 2024

jwallwork23 left a comment

ddundo commented Mar 21, 2024

Parallel-safe mesh-to-mesh interpolation #128

Parallel-safe mesh-to-mesh interpolation #128

Conversation

ddundo commented Mar 5, 2024 • edited Loading

ddundo commented Mar 5, 2024

jwallwork23 commented Mar 6, 2024

jwallwork23 Mar 6, 2024

Choose a reason for hiding this comment

ddundo commented Mar 6, 2024

jwallwork23 commented Mar 6, 2024

ddundo commented Mar 6, 2024

jwallwork23 commented Mar 6, 2024

ddundo commented Mar 6, 2024

jwallwork23 commented Mar 7, 2024

ddundo commented Mar 7, 2024

jwallwork23 commented Mar 7, 2024

ddundo commented Mar 7, 2024

ddundo commented Mar 16, 2024

jwallwork23 left a comment

Choose a reason for hiding this comment

ddundo commented Mar 21, 2024

jwallwork23 left a comment

Choose a reason for hiding this comment

ddundo commented Mar 21, 2024

ddundo commented Mar 5, 2024 •

edited

Loading