Docs update: solvers and parallel multipoint analysis (#405)

* added solver and MPI docs * minor fix * added parallel multipoint test * changed linear solver in large mesh examples * black * trying testflo * yml debug * reinstalling vsp/pygeo * petsc * fixing coverage * cleanup * testflo time limit * trying OM3.25 with py3.8 * oldest is latest * old packages with py3.11 * py3.10 * py3.9 * skip MPI test on the oldest * fix * legacy installation * py3.11 docs build * readthedocs * petsc install * petsc install * petsc install * I cannot sudo * petsc install * petsc * install blas * docs build finally worked * stop running MPI code in docs build * return of the vsp and pygeo * cleanup * fixed typos and grammar * use literalinclude in parallel docs * petsc manual install * yaml cleanup * have to make check? * drop petsc make check * vsp back
mdolab · May 17, 2023 · 9f592ae · 9f592ae
1 parent f2c0526
commit 9f592ae
Show file tree

Hide file tree

Showing 13 changed files with 757 additions and 11 deletions.
diff --git a/.github/workflows/oas.yml b/.github/workflows/oas.yml
@@ -32,6 +32,7 @@ jobs:
         MPHYS_VERSION_OLDEST: ['1.0.0']  # latest is most recent on PyPI
         PYGEO_VERSION_OLDEST: ['1.6.0']  # latest is pulled from main branch, for some reason anything after 1.12.2 seg faults on the old build
         VSP_VERSION: ['3.27.1']  # used for both builds
+        PETSC_VERSION_LATEST: ['3.19.1']
 
     steps:
     - uses: actions/checkout@v2
@@ -99,12 +100,29 @@ jobs:
         python -m pip install --upgrade pip wheel
         pip install -e .[test,mphys]
 
+    - name: Install MPI
+      run: |
+        sudo apt-get install openmpi-bin libopenmpi-dev
+        pip install mpi4py
+
+    # install PETSc on the latest for MPI tests
+    - name: Install PETSc
+      if: ${{ matrix.dep-versions == 'latest' }}
+      run: |
+        cd ..
+        pip download petsc==${{ matrix.PETSC_VERSION_LATEST }}
+        tar -xf petsc-${{ matrix.PETSC_VERSION_LATEST }}.tar.gz
+        cd petsc-${{ matrix.PETSC_VERSION_LATEST }}
+        export PETSC_ARCH=real-debug
+        export PETSC_DIR=`pwd`
+        ./configure --PETSC_ARCH=$PETSC_ARCH --download-fblaslapack
+        make PETSC_DIR=$PETSC_DIR PETSC_ARCH=$PETSC_ARCH all
+        pip install petsc4py==${{ matrix.PETSC_VERSION_LATEST }}
+
     # We need pySpline/pyGeo to run FFD tests.
     - name: Install pySpline
       run: |
         cd ..
-        sudo apt-get install openmpi-bin libopenmpi-dev
-        pip install mpi4py
         git clone https://github.com/mdolab/pyspline.git
         cd pyspline
         cp config/defaults/config.LINUX_GFORTRAN.mk config/config.mk
@@ -124,17 +142,29 @@ jobs:
       run: |
         pip list -v
 
-    - name: Run tests
+    - name: Run tests (latest)
       env:
         OMPI_MCA_btl: ^openib   # prevent OpenMPI warning messages
+      if: ${{ matrix.dep-versions == 'latest' }}
+      run: |
+        testflo -n 2 -v openaerostruct --coverage --coverpkg openaerostruct
+        coverage xml
+    # skip MPI tests on the oldest
+    - name: Run tests (oldest)
+      env:
+        OMPI_MCA_btl: ^openib   # prevent OpenMPI warning messages
+      if: ${{ matrix.dep-versions == 'oldest' }}
       run: |
-        python -m pytest --cov-config=.coveragerc --cov=openaerostruct --cov-report=xml
+        testflo -n 2 -v --exclude \*MPI\* openaerostruct --coverage --coverpkg openaerostruct
+        coverage xml
 
     - name: Upload Coverage to Codecov
-      uses: codecov/codecov-action@v2
+      uses: codecov/codecov-action@v3
       with:
         fail_ci_if_error: true
         token: ${{ secrets.CODECOV_TOKEN }}
+        verbose: true
+
   # --- linting and formatting ---
   black:
     uses: mdolab/.github/.github/workflows/black.yaml@main

diff --git a/.readthedocs.yaml b/.readthedocs.yaml
@@ -2,6 +2,11 @@
 
 version: 2
 
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+
 python:
   install:
     - method: pip

diff --git a/openaerostruct/docs/advanced_features.rst b/openaerostruct/docs/advanced_features.rst
@@ -8,11 +8,13 @@ These examples show some advanced features in OpenAeroStruct.
 .. toctree::
    :maxdepth: 1
 
+   advanced_features/solver.rst
    advanced_features/geometry_manipulation.rst
    advanced_features/custom_mesh_example.rst
    advanced_features/openvsp_mesh_example.rst
    advanced_features/multiple_surfaces.rst
    advanced_features/multipoint.rst
+   advanced_features/multipoint_parallel.rst
    advanced_features/aerostruct_ffd.rst
    advanced_features/ground_effect.rst
    advanced_features/openconcept_coupling.rst

diff --git a/openaerostruct/docs/advanced_features/multipoint_parallel.rst b/openaerostruct/docs/advanced_features/multipoint_parallel.rst
@@ -0,0 +1,230 @@
+.. _Multipoint Optimization with MPI:
+
+.. note:: This feature requires MPI, which does not come with OpenAeroStruct by default.
+
+Parallel Multipoint Optimization using MPI
+==========================================
+
+Multipoint analysis or optimization can be parallelized to reduce the runtime.
+Because each flight condition (or point) is independent, it is `embarassingly parallel`, meaning that we can easily parallelize these analyses.
+
+Here, we will parallelize the :ref:`previous multipoint aerostructural example (Q400)<Multipoint Optimization>`.
+This requires a little modification to the original serial runscript.
+
+Runscript modifications
+-----------------------
+We first import MPI.
+If this line does not work, make sure that you have a working MPI installation.
+
+.. code-block::
+    
+    from mpi4py import MPI
+
+You may need to turn off the numpy multithreading.
+This can be done by adding the following lines before importing numpy.
+The name of environment variable may be different depending on the system.
+
+.. code-block::
+
+    import os
+    os.environ['OPENBLAS_NUM_THREADS'] = '1'
+
+Then, let's set up the problem in the same way as the serial runscript.
+
+.. code-block::
+    
+    prob = om.Problem()
+    
+    # Setup problem information in indep_var_comp
+    ...
+
+    # Add AerostructGeometry to the model
+    ...
+
+Next, we need to add AS_points under a ``ParallelGroup`` instead of directly under the ``prob.model``.
+
+.. literalinclude:: ../../tests/test_multipoint_parallel.py
+    :dedent: 8
+    :start-after: # [rst Setup ParallelGroup (beg)]
+    :end-before: # [rst Setup ParallelGroup (end)]
+
+After establishing variable connections and setting up the driver, we define the optimization objective and constraints.
+Here, we will setup the parallel derivative computations.
+In this example, we have 6 functions of interest (1 objective and 5 constraints), which would require 6 linear solves for reverse-mode derivatives in series.
+Among 6 functions, 4 depend only on AS_point_0, and 2 depend only on AS_point_1.
+Therefore, we can form 2 pairs and perform linear solves in parallel.
+We specify ``parallel_deriv_color`` to tell OpenMDAO which function's derivatives can be solved for in parallel.
+
+.. literalinclude:: ../../tests/test_multipoint_parallel.py
+    :dedent: 8
+    :start-after: # [rst Parallel deriv color setup 1 (beg)]
+    :end-before: # [rst Parallel deriv color setup 1 (end)]
+
+Furthermore, we will add another dummy (nonsense) constraint to explain how parallelization works for reverse-mode derivatives.
+This dummy constraint (sum of the fuel burns from AS_point_0 and AS_point_1) depends on both AS points.
+In this case, the linear solves of AS_point_0 and AS_point_1 will be parallelized.
+
+.. literalinclude:: ../../tests/test_multipoint_parallel.py
+    :dedent: 8
+    :start-after: # [rst Parallel deriv color setup 2 (beg)]
+    :end-before: # [rst Parallel deriv color setup 2 (end)]
+
+Finally, let's change the linear solver from default.
+This step is not necessary and not directly relevant to parallelization, but the ``LinearBlockGS`` solver works better on a fine mesh than the default ``DirectSolver``.
+
+.. literalinclude:: ../../tests/test_multipoint_parallel.py
+    :dedent: 8
+    :start-after: # [rst Change linear solver (beg)]
+    :end-before: # [rst Change linear solver (end)]
+
+
+Complete runscript
+------------------
+
+.. embed-code::
+    openaerostruct.tests.test_multipoint_parallel.Test.test_multipoint_MPI
+
+To run this example in parallel with two processors, use the following command:
+
+.. code-block:: bash
+
+    $ mpirun -n 2 python <name of script>.py
+
+Solver Outputs
+--------------
+The stdout of the above script would look like the following.
+The solver outputs help us understand how solvers are parallelized for analysis and total derivative computations.
+
+.. code-block:: bash
+
+    # Nonlinear solver in parallel
+    ===========================
+    parallel.AS_point_0.coupled
+    ===========================
+
+    ===========================
+    parallel.AS_point_1.coupled
+    ===========================
+    NL: NLBGS 1 ; 82168.4402 1
+    NL: NLBGS 1 ; 79704.5639 1
+    NL: NLBGS 2 ; 63696.5109 0.775194354
+    NL: NLBGS 2 ; 68552.4805 0.860082248
+    NL: NLBGS 3 ; 2269.83605 0.0276241832
+    NL: NLBGS 3 ; 2641.30776 0.0331387267
+    NL: NLBGS 4 ; 26.8901082 0.000327255917
+    NL: NLBGS 4 ; 33.4963389 0.000420256222
+    NL: NLBGS 5 ; 0.20014208 2.43575367e-06
+    NL: NLBGS 5 ; 0.273747809 3.43453117e-06
+    NL: NLBGS 6 ; 0.000203058798 2.47125048e-09
+    NL: NLBGS 6 ; 0.00033072442 4.14937871e-09
+    NL: NLBGS 7 ; 3.3285346e-06 4.05086745e-11
+    NL: NLBGS 7 ; 5.16564573e-06 6.48099115e-11
+    NL: NLBGS 8 ; 9.30405466e-08 1.13231487e-12
+    NL: NLBGS Converged
+    NL: NLBGS 8 ; 1.63279302e-07 2.04855649e-12
+    NL: NLBGS 9 ; 2.01457772e-09 2.5275563e-14
+    NL: NLBGS Converged
+
+    # Linear solver for "parcon1". Derivatives of AS_point_0.fuelburn and AS_point_1.L_equals_W in parallel.
+    ===========================
+    parallel.AS_point_0.coupled
+    ===========================
+
+    ===========================
+    parallel.AS_point_1.coupled
+    ===========================
+    LN: LNBGS 0 ; 180.248073 1
+    LN: LNBGS 0 ; 1.17638541e-05 1
+    LN: LNBGS 1 ; 0.00284457871 1.57814653e-05
+    LN: LNBGS 1 ; 1.124189e-06 0.0955629836
+    LN: LNBGS 2 ; 1.87700622e-08 0.00159557081
+    LN: LNBGS 2 ; 4.66688449e-05 2.58914529e-07
+    LN: LNBGS 3 ; 1.13549461e-11 9.65240308e-07
+    LN: LNBGS Converged
+    LN: LNBGS 3 ; 8.18485966e-08 4.54088609e-10
+    LN: LNBGS 4 ; 9.00103905e-10 4.99369503e-12
+    LN: LNBGS Converged
+
+    # Linear solver for "parcon2". Derivatives of AS_point_0.CL and AS_point_1.wing_perf.failure in parallel.
+    ===========================
+    parallel.AS_point_1.coupled
+    ===========================
+
+    ===========================
+    parallel.AS_point_0.coupled
+    ===========================
+    LN: LNBGS 0 ; 334.283603 1
+    LN: LNBGS 0 ; 0.00958374526 1
+    LN: LNBGS 1 ; 2.032696e-05 6.08075293e-08
+    LN: LNBGS 1 ; 2.02092209e-06 0.000210869762
+    LN: LNBGS 2 ; 2.3346978e-06 6.98418281e-09
+    LN: LNBGS 2 ; 2.90180431e-08 3.02783956e-06
+    LN: LNBGS 3 ; 4.98483883e-08 1.49120052e-10
+    LN: LNBGS 3 ; 8.63240127e-11 9.0073359e-09
+    LN: LNBGS Converged
+    LN: LNBGS 4 ; 5.58667374e-11 1.67123774e-13
+    LN: LNBGS Converged
+
+    # Linear solver for derivatives of fuel_vol_delta.fuel_vol_delta (not parallelized)
+    ===========================
+    parallel.AS_point_0.coupled
+    ===========================
+    LN: LNBGS 0 ; 0.224468335 1
+    LN: LNBGS 1 ; 3.54243924e-06 1.57814653e-05
+    LN: LNBGS 2 ; 5.81181131e-08 2.58914529e-07
+    LN: LNBGS 3 ; 1.01928513e-10 4.54088604e-10
+    LN: LNBGS 4 ; 1.12121714e-12 4.99499023e-12
+    LN: LNBGS Converged
+
+    # Linear solver for derivatives of fuel_diff (not parallelized)
+    ===========================
+    parallel.AS_point_0.coupled
+    ===========================
+    LN: LNBGS 0 ; 0.21403928 1
+    LN: LNBGS 1 ; 3.37785348e-06 1.57814653e-05
+    LN: LNBGS 2 ; 5.54178795e-08 2.58914529e-07
+    LN: LNBGS 3 ; 9.71927996e-11 4.54088612e-10
+    LN: LNBGS Converged
+
+    # Linear solver for derivatives of fuel_sum in parallel.
+    ===========================
+    parallel.AS_point_0.coupled
+    ===========================
+
+    ===========================
+    parallel.AS_point_1.coupled
+    ===========================
+    LN: LNBGS 0 ; 360.496145 1
+    LN: LNBGS 0 ; 511.274568 1
+    LN: LNBGS 1 ; 0.00568915741 1.57814653e-05
+    LN: LNBGS 1 ; 0.00838867553 1.64073788e-05
+    LN: LNBGS 2 ; 0.00013534629 2.64723299e-07
+    LN: LNBGS 2 ; 9.33376897e-05 2.58914529e-07
+    LN: LNBGS 3 ; 1.00754737e-07 1.97065811e-10
+    LN: LNBGS 3 ; 1.63697193e-07 4.54088609e-10
+    LN: LNBGS 4 ; 2.24690253e-09 4.39470819e-12
+    LN: LNBGS Converged
+    LN: LNBGS 4 ; 1.80020781e-09 4.99369503e-12
+    LN: LNBGS Converged
+
+
+Comparing Runtime
+-----------------
+How much speedup can we get by parallelization?
+Here, we compared the runtime for the example above (but with a finer mesh of `nx=3` and `ny=61`).
+In this case, we achieved decent speedup in nonlinear analysis, but not so much in derivative computation.
+The actual speedup you can get depends on your problem setups, such as number of points (flight conditions) and functions of interest.
+
+.. list-table:: Runtime for Q400 example
+   :widths: 30 35 35
+   :header-rows: 1
+
+   * - Case
+     - Analysis walltime [s]
+     - Derivatives walltime [s]
+   * - Serial
+     - 1.451
+     - 5.775
+   * - Parallel 
+     - 0.840
+     - 4.983
diff --git a/openaerostruct/docs/advanced_features/scripts/two_part_wing_custom_mesh.py b/openaerostruct/docs/advanced_features/scripts/two_part_wing_custom_mesh.py
@@ -351,6 +351,10 @@
 # Set up the problem
 prob.setup()
 
+# change linear solver for aerostructural coupled adjoint
+prob.model.AS_point_0.coupled.linear_solver = om.LinearBlockGS(iprint=0, maxiter=30, use_aitken=True)
+prob.model.AS_point_1.coupled.linear_solver = om.LinearBlockGS(iprint=0, maxiter=30, use_aitken=True)
+
 # om.view_model(prob)
 
 # prob.check_partials(form='central', compact_print=True)