Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP]Add diagnostic file output for the implicit solvers #5464

Open
wants to merge 3 commits into
base: development
Choose a base branch
from
Open

[WIP]Add diagnostic file output for the implicit solvers #5464

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ade1e55
Add diagnostic file output for the implicit solvers
dpgrote Nov 15, 2024
d629d82
Remove uses of older versions of gcc where std::filesystem is not ful…
dpgrote Nov 19, 2024
cafba61
Merge branch 'development' into add_implicit_solver_diagnostic_file
dpgrote Dec 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Docs/source/install/dependencies.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Dependencies
WarpX depends on the following popular third party software.
Please see installation instructions below.

- a mature `C++17 <https://en.wikipedia.org/wiki/C%2B%2B17>`__ compiler, e.g., GCC 8.4+, Clang 7, NVCC 11.0, MSVC 19.15 or newer
- a mature `C++17 <https://en.wikipedia.org/wiki/C%2B%2B17>`__ compiler, e.g., GCC 9.1+, Clang 7, NVCC 11.0, MSVC 19.15 or newer
- `CMake 3.24.0+ <https://cmake.org>`__
- `Git 2.18+ <https://git-scm.com>`__
- `AMReX <https://amrex-codes.github.io>`__: we automatically download and compile a copy of AMReX
Expand Down
2 changes: 2 additions & 0 deletions Examples/Tests/implicit/inputs_test_2d_theta_implicit_jfnk_vandb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,13 +52,15 @@ implicit_evolve.particle_tolerance = 1.0e-12
#picard.relative_tolerance = 0.0 #1.0e-12
#picard.absolute_tolerance = 0.0 #1.0e-24
#picard.require_convergence = false
#picard.diagnostic_file = "diags/picard_solver.txt"

implicit_evolve.nonlinear_solver = "newton"
newton.verbose = true
newton.max_iterations = 20
newton.relative_tolerance = 1.0e-12
newton.absolute_tolerance = 0.0
newton.require_convergence = false
newton.diagnostic_file = "diags/newton_solver.txt"

gmres.verbose_int = 2
gmres.max_iterations = 1000
Expand Down
2 changes: 1 addition & 1 deletion Source/FieldSolver/ImplicitSolvers/SemiImplicitEM.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ void SemiImplicitEM::OneStep ( amrex::Real a_time,
// Solve nonlinear system for Eg at t_{n+1/2}
// Particles will be advanced to t_{n+1/2}
m_E.Copy(m_Eold); // initial guess for Eg^{n+1/2}
m_nlsolver->Solve( m_E, m_Eold, half_time, 0.5_rt*m_dt );
m_nlsolver->Solve( m_E, m_Eold, half_time, 0.5_rt*m_dt, a_step );

// Update WarpX owned Efield_fp to t_{n+1/2}
m_WarpX->SetElectricFieldAndApplyBCs( m_E );
Expand Down
2 changes: 1 addition & 1 deletion Source/FieldSolver/ImplicitSolvers/StrangImplicitSpectralEM.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ void StrangImplicitSpectralEM::OneStep ( amrex::Real a_time,

// Solve nonlinear system for E at t_{n+1/2}
// Particles will be advanced to t_{n+1/2}
m_nlsolver->Solve( m_E, m_Eold, half_time, 0.5_rt*m_dt );
m_nlsolver->Solve( m_E, m_Eold, half_time, 0.5_rt*m_dt, a_step );

// Update WarpX owned Efield_fp and Bfield_fp to t_{n+1/2}
UpdateWarpXFields( m_E, half_time );
Expand Down
2 changes: 1 addition & 1 deletion Source/FieldSolver/ImplicitSolvers/ThetaImplicitEM.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ void ThetaImplicitEM::OneStep ( const amrex::Real a_time,
// Solve nonlinear system for Eg at t_{n+theta}
// Particles will be advanced to t_{n+1/2}
m_E.Copy(m_Eold); // initial guess for Eg^{n+theta}
m_nlsolver->Solve( m_E, m_Eold, theta_time, m_theta*m_dt );
m_nlsolver->Solve( m_E, m_Eold, theta_time, m_theta*m_dt, a_step );

// Update WarpX owned Efield_fp and Bfield_fp to t_{n+theta}
UpdateWarpXFields( m_E, theta_time );
Expand Down
59 changes: 57 additions & 2 deletions Source/NonlinearSolvers/NewtonSolver.H
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
#include <AMReX_ParmParse.H>

#include <vector>
#include <istream>
#include <filesystem>

/**
* \brief Newton method to solve nonlinear equation of form:
Expand Down Expand Up @@ -45,7 +47,8 @@ public:
void Solve ( Vec& a_U,
const Vec& a_b,
amrex::Real a_time,
amrex::Real a_dt ) const override;
amrex::Real a_dt,
int a_step) const override;

void GetSolverParams ( amrex::Real& a_rtol,
amrex::Real& a_atol,
Expand Down Expand Up @@ -200,6 +203,34 @@ void NewtonSolver<Vec,Ops>::Define ( const Vec& a_U,

this->m_is_defined = true;

// Create diagnostic file and write header
if (!this->m_diagnostic_file.empty() && amrex::ParallelDescriptor::IOProcessor()) {

std::filesystem::path diagnostic_path(this->m_diagnostic_file);
std::filesystem::path diagnostic_dir = diagnostic_path.parent_path();
if (!diagnostic_dir.empty()) {
std::filesystem::create_directories(diagnostic_dir);
}

std::ofstream diagnostic_file{this->m_diagnostic_file, std::ofstream::out | std::ofstream::trunc};
int c = 0;
diagnostic_file << "#";
diagnostic_file << "[" << c++ << "]step()";
diagnostic_file << " ";
diagnostic_file << "[" << c++ << "]time(s)";
diagnostic_file << " ";
diagnostic_file << "[" << c++ << "]iters";
diagnostic_file << " ";
diagnostic_file << "[" << c++ << "]norm_abs";
diagnostic_file << " ";
diagnostic_file << "[" << c++ << "]norm_rel";
diagnostic_file << " ";
diagnostic_file << "[" << c++ << "]gmres_iters";
diagnostic_file << " ";
diagnostic_file << "[" << c++ << "]gmres_last_res";
diagnostic_file << "\n";
diagnostic_file.close();
}
}

template <class Vec, class Ops>
Expand All @@ -211,6 +242,7 @@ void NewtonSolver<Vec,Ops>::ParseParameters ()
pp_newton.query("relative_tolerance", m_rtol);
pp_newton.query("max_iterations", m_maxits);
pp_newton.query("require_convergence", m_require_convergence);
pp_newton.query("diagnostic_file", this->m_diagnostic_file);

const amrex::ParmParse pp_gmres("gmres");
pp_gmres.query("verbose_int", m_gmres_verbose_int);
Expand All @@ -227,7 +259,8 @@ template <class Vec, class Ops>
void NewtonSolver<Vec,Ops>::Solve ( Vec& a_U,
const Vec& a_b,
amrex::Real a_time,
amrex::Real a_dt ) const
amrex::Real a_dt,
int a_step) const
{
BL_PROFILE("NewtonSolver::Solve()");
WARPX_ALWAYS_ASSERT_WITH_MESSAGE(
Expand All @@ -248,6 +281,7 @@ void NewtonSolver<Vec,Ops>::Solve ( Vec& a_U,
amrex::Real norm_rel = 0.;

int iter;
int linear_solver_iters = 0;
for (iter = 0; iter < m_maxits;) {

// Compute residual: F(U) = U - b - R(U)
Expand Down Expand Up @@ -293,6 +327,7 @@ void NewtonSolver<Vec,Ops>::Solve ( Vec& a_U,
// Solve linear system for Newton step [Jac]*dU = F
m_dU.zero();
m_linear_solver->solve( m_dU, m_F, m_gmres_rtol, m_gmres_atol );
linear_solver_iters += m_linear_solver->getNumIters();

// Update solution
a_U -= m_dU;
Expand Down Expand Up @@ -321,6 +356,26 @@ void NewtonSolver<Vec,Ops>::Solve ( Vec& a_U,
}
}

if (!this->m_diagnostic_file.empty() && amrex::ParallelDescriptor::IOProcessor()) {
std::ofstream diagnostic_file{this->m_diagnostic_file, std::ofstream::out | std::ofstream::app};
diagnostic_file << std::setprecision(14);
diagnostic_file << a_step;
diagnostic_file << " ";;
diagnostic_file << a_time;
diagnostic_file << " ";;
diagnostic_file << iter;
diagnostic_file << " ";;
diagnostic_file << norm_abs;
diagnostic_file << " ";;
diagnostic_file << norm_rel;
diagnostic_file << " ";;
diagnostic_file << linear_solver_iters;
diagnostic_file << " ";;
diagnostic_file << m_linear_solver->getResidualNorm();
diagnostic_file << "\n";
diagnostic_file.close();
}

}

template <class Vec, class Ops>
Expand Down
4 changes: 3 additions & 1 deletion Source/NonlinearSolvers/NonlinearSolver.H
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,8 @@ public:
virtual void Solve ( Vec&,
const Vec&,
amrex::Real,
amrex::Real ) const = 0;
amrex::Real,
int) const = 0;

/**
* \brief Print parameters used by the nonlinear solver.
Expand All @@ -81,6 +82,7 @@ protected:

bool m_is_defined = false;
mutable bool m_verbose = true;
std::string m_diagnostic_file;

};

Expand Down
51 changes: 48 additions & 3 deletions Source/NonlinearSolvers/PicardSolver.H
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@
#include "Utils/TextMsg.H"

#include <vector>
#include <istream>
#include <filesystem>

/**
* \brief Picard fixed-point iteration method to solve nonlinear
Expand Down Expand Up @@ -42,7 +44,8 @@ public:
void Solve ( Vec& a_U,
const Vec& a_b,
amrex::Real a_time,
amrex::Real a_dt ) const override;
amrex::Real a_dt,
int a_step) const override;

void GetSolverParams ( amrex::Real& a_rtol,
amrex::Real& a_atol,
Expand Down Expand Up @@ -114,6 +117,31 @@ void PicardSolver<Vec,Ops>::Define ( const Vec& a_U,

this->m_is_defined = true;

// Create diagnostic file and write header
if (!this->m_diagnostic_file.empty() && amrex::ParallelDescriptor::IOProcessor()) {

std::filesystem::path diagnostic_path(this->m_diagnostic_file);
std::filesystem::path diagnostic_dir = diagnostic_path.parent_path();
if (!diagnostic_dir.empty()) {
std::filesystem::create_directories(diagnostic_dir);
}

std::ofstream diagnostic_file{this->m_diagnostic_file, std::ofstream::out | std::ofstream::trunc};
int c = 0;
diagnostic_file << "#";
diagnostic_file << "[" << c++ << "]step()";
diagnostic_file << " ";
diagnostic_file << "[" << c++ << "]time(s)";
diagnostic_file << " ";
diagnostic_file << "[" << c++ << "]iters";
diagnostic_file << " ";
diagnostic_file << "[" << c++ << "]norm_abs";
diagnostic_file << " ";
diagnostic_file << "[" << c++ << "]norm_rel";
diagnostic_file << "\n";
diagnostic_file.close();
}

}

template <class Vec, class Ops>
Expand All @@ -125,14 +153,15 @@ void PicardSolver<Vec,Ops>::ParseParameters ()
pp_picard.query("relative_tolerance", m_rtol);
pp_picard.query("max_iterations", m_maxits);
pp_picard.query("require_convergence", m_require_convergence);

pp_picard.query("diagnostic_file", this->m_diagnostic_file);
}

template <class Vec, class Ops>
void PicardSolver<Vec,Ops>::Solve ( Vec& a_U,
const Vec& a_b,
amrex::Real a_time,
amrex::Real a_dt ) const
amrex::Real a_dt,
int a_step) const
{
BL_PROFILE("PicardSolver::Solve()");
WARPX_ALWAYS_ASSERT_WITH_MESSAGE(
Expand Down Expand Up @@ -213,6 +242,22 @@ void PicardSolver<Vec,Ops>::Solve ( Vec& a_U,
}
}

if (!this->m_diagnostic_file.empty() && amrex::ParallelDescriptor::IOProcessor()) {
std::ofstream diagnostic_file{this->m_diagnostic_file, std::ofstream::out | std::ofstream::app};
diagnostic_file << std::setprecision(14);
diagnostic_file << a_step;
diagnostic_file << " ";
diagnostic_file << a_time;
diagnostic_file << " ";
diagnostic_file << iter;
diagnostic_file << " ";
diagnostic_file << norm_abs;
diagnostic_file << " ";
diagnostic_file << norm_rel;
diagnostic_file << "\n";
diagnostic_file.close();
}

}

#endif
13 changes: 0 additions & 13 deletions Tools/machines/desktop/spack-ubuntu-cuda.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,19 +120,6 @@ spack:
modules: []
environment: {}
extra_rpaths: []
- compiler:
spec: gcc@8.3.0
paths:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
f77: /usr/bin/gfortran
fc: /usr/bin/gfortran
flags: {}
operating_system: debian10
target: x86_64
modules: []
environment: {}
extra_rpaths: []

# binary caches
mirrors:
Expand Down
13 changes: 0 additions & 13 deletions Tools/machines/desktop/spack-ubuntu-openmp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,19 +110,6 @@ spack:
modules: []
environment: {}
extra_rpaths: []
- compiler:
spec: gcc@8.3.0
paths:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
f77: /usr/bin/gfortran
fc: /usr/bin/gfortran
flags: {}
operating_system: debian10
target: x86_64
modules: []
environment: {}
extra_rpaths: []

# binary caches
mirrors:
Expand Down
13 changes: 0 additions & 13 deletions Tools/machines/desktop/spack-ubuntu-rocm.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,19 +114,6 @@ spack:
modules: []
environment: {}
extra_rpaths: []
- compiler:
spec: gcc@8.3.0
paths:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
f77: /usr/bin/gfortran
fc: /usr/bin/gfortran
flags: {}
operating_system: debian10
target: x86_64
modules: []
environment: {}
extra_rpaths: []

# binary caches
mirrors:
Expand Down
Loading