merge MBD additions

CHANGELOG.md

@@ -4,6 +4,7 @@
 
 ### Added (new features/APIs/variables/...)
 - [[PR 907]](https://github.com/parthenon-hpc-lab/parthenon/pull/907) PEP1: Allow subclassing StateDescriptor
+- [[PR 921]](https://github.com/parthenon-hpc-lab/parthenon/pull/921) Add more flexible ways of adding and using MeshData/MeshBlockData objects to DataCollections
 - [[PR 900]](https://github.com/parthenon-hpc-lab/parthenon/pull/900) Add Morton numbers and expand functionality of LogicalLocation
 - [[PR 902]](https://github.com/parthenon-hpc-lab/parthenon/pull/902) Add ability to output NaNs for de-allocated sparse fields
 - [[PR 887]](https://github.com/parthenon-hpc-lab/parthenon/pull/887) Add ability to dump more types of params and read them from restarts

doc/sphinx/src/interface/state.rst

@@ -66,6 +66,22 @@ several useful features and functions.
 - ``T *MutableParam(const std::string &key)`` returns a pointer to a
   parameter that has been marked mutable when it was added. Note this
   pointer is *not* marked ``const``.
+- ``MetadataFlag GetMetadataFlag()`` returns a ``MetadataFlag`` that is
+  automatically added to all fields, sparse pools, and swarms that are
+  added to the ``StateDescriptor``.
+- ``std::vector<std::string> GetVariableNames(...)`` provides a means of
+  getting a list of variables that satisfy specified requirements and that
+  are associated with the ``StateDescriptor``.  Optional arguments, in order,
+  include a vector of variable names, a ``Metadata::FlagCollection`` to select,
+  variables by flags, and a vector of sparse ids to allow selecting subsets of
+  sparse fields.  Selection of sparse fields by name requires providing the
+  base name of the sparse field.  Names of a sparse field tied to a specific
+  sparse index are not supported (and will throw).  The returned list is
+  appropriate for use in adding ``MeshData`` and/or ``MeshBlockData`` objects
+  with specified fields to the ``DataCollection`` objects in ``Mesh`` and
+  ``MeshBlock``.  For convenience, the ``Mesh`` class also provides this
+  function, which provides a list of variables gathered from all the package
+  ``StateDescriptor``s.
 - ``void FillDerivedBlock(MeshBlockData<Real>* rc)`` delgates to the
   ``std::function`` member ``FillDerivedBlock`` if set (defaults to
   ``nullptr`` and therefore a no-op) that allows an application to provide
@@ -293,6 +309,18 @@ downstream applications to allocate storage in a more targeted fashion,
 as might be desirable to hold source terms for particular equations, for
 example.
 
+Analogously, ``DataCollection`` provides ``AddShallow`` functions that
+differ from ``Add`` only in that ***all*** included variables, even
+non-``Metadata::OncCopy`` variables, are simply shallow copies.  For
+these functions, no new storage for variables is ever allocated.
+
+Finally, all of the functionality just described for ``MeshBlockData``
+objects is also provided for ``MeshData`` objects.  Adding a new
+``MeshData`` object to the ``Mesh``-level ``DataCollection`` automatically
+adds the corresponding ``MeshBlockData`` objects to each of the
+``MeshBlock``-level ``DataCollection``s.  Using this ``Mesh`` level
+functionality can be more convenient.
+
 Two simple examples of usage of these new containers are 1) to provide
 storage for multistage integration schemes and 2) to provide a mechanism
 to allocate storage for right hand sides, deltas, etc. Both of these

src/interface/data_collection.cpp

@@ -1,5 +1,5 @@
 //========================================================================================
-// (C) (or copyright) 2020. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2020-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC
@@ -35,7 +35,7 @@ DataCollection<T>::Add(const std::string &name, const std::shared_ptr<T> &src,
   }
 
   auto c = std::make_shared<T>();
-  c->Copy(src.get(), field_names, shallow);
+  c->Initialize(src.get(), field_names, shallow);
 
   Set(name, c);
 
@@ -49,26 +49,16 @@ DataCollection<T>::Add(const std::string &name, const std::shared_ptr<T> &src,
   return containers_[name];
 }
 template <typename T>
-std::shared_ptr<T> &DataCollection<T>::Add(const std::string &name,
+std::shared_ptr<T> &DataCollection<T>::Add(const std::string &label,
                                            const std::shared_ptr<T> &src,
-                                           const std::vector<std::string> &field_names) {
-  return Add(name, src, field_names, false);
+                                           const std::vector<std::string> &flags) {
+  return Add(label, src, flags, false);
 }
 template <typename T>
-std::shared_ptr<T> &
-DataCollection<T>::AddShallow(const std::string &name, const std::shared_ptr<T> &src,
-                              const std::vector<std::string> &field_names) {
-  return Add(name, src, field_names, true);
-}
-template <typename T>
-std::shared_ptr<T> &DataCollection<T>::Add(const std::string &name,
-                                           const std::shared_ptr<T> &src) {
-  return Add(name, src, {}, false);
-}
-template <typename T>
-std::shared_ptr<T> &DataCollection<T>::AddShallow(const std::string &name,
-                                                  const std::shared_ptr<T> &src) {
-  return Add(name, src, {}, true);
+std::shared_ptr<T> &DataCollection<T>::AddShallow(const std::string &label,
+                                                  const std::shared_ptr<T> &src,
+                                                  const std::vector<std::string> &flags) {
+  return Add(label, src, flags, true);
 }
 
 template <>

src/interface/data_collection.hpp

@@ -1,5 +1,5 @@
 //========================================================================================
-// (C) (or copyright) 2020. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2020-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC
@@ -41,11 +41,9 @@ class DataCollection {
   std::shared_ptr<T> &Add(const std::string &label, const std::shared_ptr<T> &src,
                           const std::vector<std::string> &flags, const bool shallow);
   std::shared_ptr<T> &Add(const std::string &label, const std::shared_ptr<T> &src,
-                          const std::vector<std::string> &flags);
+                          const std::vector<std::string> &flags = {});
   std::shared_ptr<T> &AddShallow(const std::string &label, const std::shared_ptr<T> &src,
-                                 const std::vector<std::string> &flags);
-  std::shared_ptr<T> &Add(const std::string &label, const std::shared_ptr<T> &src);
-  std::shared_ptr<T> &AddShallow(const std::string &label, const std::shared_ptr<T> &src);
+                                 const std::vector<std::string> &flags = {});
   std::shared_ptr<T> &Add(const std::string &label) {
     // error check for duplicate names
     auto it = containers_.find(label);

src/interface/mesh_data.hpp

@@ -1,5 +1,5 @@
 //========================================================================================
-// (C) (or copyright) 2020-2022. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2020-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC
@@ -237,15 +237,16 @@ class MeshData {
   }
 
   template <typename... Args>
-  void Copy(const MeshData<T> *src, Args &&...args) {
+  void Initialize(const MeshData<T> *src, Args &&...args) {
     if (src == nullptr) {
       PARTHENON_THROW("src points at null");
     }
+    pmy_mesh_ = src->GetParentPointer();
     const int nblocks = src->NumBlocks();
     block_data_.resize(nblocks);
     for (int i = 0; i < nblocks; i++) {
       block_data_[i] = std::make_shared<MeshBlockData<T>>();
-      block_data_[i]->Copy(src->GetBlockData(i).get(), std::forward<Args>(args)...);
+      block_data_[i]->Initialize(src->GetBlockData(i).get(), std::forward<Args>(args)...);
     }
   }
 
@@ -439,9 +440,8 @@ class MeshData {
 
 template <typename T, typename... Args>
 std::vector<Uid_t> UidIntersection(MeshData<T> *md1, MeshData<T> *md2, Args &&...args) {
-  auto u1 = md1->GetBlockData(0)->GetVariableUIDs(std::forward<Args>(args)...);
-  auto u2 = md2->GetBlockData(0)->GetVariableUIDs(std::forward<Args>(args)...);
-  return UidIntersection(u1, u2);
+  return UidIntersection(md1->GetBlockData(0).get(), md2->GetBlockData(0).get(),
+                         std::forward<Args>(args)...);
 }
 
 } // namespace parthenon

src/interface/meshblock_data.cpp

@@ -75,9 +75,9 @@ void MeshBlockData<T>::AddField(const std::string &base_name, const Metadata &me
 
 // TODO(JMM): Move to unique IDs at some point
 template <typename T>
-void MeshBlockData<T>::Copy(const MeshBlockData<T> *src,
-                            const std::vector<std::string> &names,
-                            const bool shallow_copy) {
+void MeshBlockData<T>::Initialize(const MeshBlockData<T> *src,
+                                  const std::vector<std::string> &names,
+                                  const bool shallow_copy) {
   assert(src != nullptr);
   SetBlockPointer(src);
   resolved_packages_ = src->resolved_packages_;

src/interface/meshblock_data.hpp

@@ -87,12 +87,6 @@ class MeshBlockData {
     return GetBlockPointer()->cellbounds.GetBoundsK(domain);
   }
 
-  /// Create copy of MeshBlockData, possibly with a subset of named fields,
-  /// and possibly shallow.  Note when shallow=false, new storage is allocated
-  /// for non-OneCopy vars, but the data from src is not actually deep copied
-  void Copy(const MeshBlockData<T> *src, const std::vector<std::string> &names,
-            const bool shallow);
-
   /// Set the pointer to the mesh block for this container
   void SetBlockPointer(std::weak_ptr<MeshBlock> pmb) { pmy_block = pmb.lock(); }
   void SetBlockPointer(const std::shared_ptr<MeshBlockData<T>> &other) {
@@ -108,6 +102,12 @@ class MeshBlockData {
   void Initialize(const std::shared_ptr<StateDescriptor> resolved_packages,
                   const std::shared_ptr<MeshBlock> pmb);
 
+  /// Create copy of MeshBlockData, possibly with a subset of named fields,
+  /// and possibly shallow.  Note when shallow=false, new storage is allocated
+  /// for non-OneCopy vars, but the data from src is not actually deep copied
+  void Initialize(const MeshBlockData<T> *src, const std::vector<std::string> &names,
+                  const bool shallow);
+
   //
   // Queries related to Variable objects
   //

src/interface/metadata.cpp

@@ -1,5 +1,5 @@
 //========================================================================================
-// (C) (or copyright) 2020-2022. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2020-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC

src/interface/metadata.hpp

@@ -491,8 +491,9 @@ class Metadata {
   /**
    * @brief Returns true if any flag is set
    */
-  template <template <class...> class Container_t, class... extra>
-  bool AnyFlagsSet(const Container_t<MetadataFlag, extra...> &flags) const {
+  template <class Container_t,
+            REQUIRES(std::is_same<typename Container_t::value_type, MetadataFlag>::value)>
+  bool AnyFlagsSet(const Container_t &flags) const {
     return std::any_of(flags.begin(), flags.end(),
                        [this](MetadataFlag const &f) { return IsSet(f); });
   }
@@ -501,24 +502,25 @@ class Metadata {
     return AnyFlagsSet(FlagVec{flag, std::forward<Args>(args)...});
   }
 
-  template <template <class...> class Container_t, class... extra>
-  bool AllFlagsSet(const Container_t<MetadataFlag, extra...> &flags) const {
+  template <class Container_t,
+            REQUIRES(std::is_same<typename Container_t::value_type, MetadataFlag>::value)>
+  bool AllFlagsSet(const Container_t &flags) const {
     return std::all_of(flags.begin(), flags.end(),
                        [this](MetadataFlag const &f) { return IsSet(f); });
   }
   template <typename... Args>
   bool AllFlagsSet(const MetadataFlag &flag, Args... args) const {
     return AllFlagsSet(FlagVec{flag, std::forward<Args>(args)...});
   }
-
-  template <template <class...> class Container_t, class... extra>
-  bool NoFlagsSet(const Container_t<MetadataFlag, extra...> &flags) const {
+  template <class Container_t,
+            REQUIRES(std::is_same<typename Container_t::value_type, MetadataFlag>::value)>
+  bool NoFlagsSet(const Container_t &flags) const {
     return std::none_of(flags.begin(), flags.end(),
                         [this](MetadataFlag const &f) { return IsSet(f); });
   }
   template <typename... Args>
   bool NoFlagsSet(const MetadataFlag &flag, Args... args) const {
-    return AllFlagsSet(FlagVec{flag, std::forward<Args>(args)...});
+    return NoFlagsSet(FlagVec{flag, std::forward<Args>(args)...});
   }
 
   template <template <class...> class Container_t, class... extra>

src/interface/state_descriptor.cpp

@@ -444,15 +444,23 @@ StateDescriptor::CreateResolvedStateDescriptor(Packages_t &packages) {
   return state;
 }
 
+// Build a list of variables in the following order
+// 1. fields requested by name (if present), in the order they're requested
+// 2. non-sparse fields picked up by the provided Metadata::FlagCollection
+// 3. sparse fields picked up by metadata
+// For sparse fields, only those ids in sparse_ids are included, unless
+// sparse_ids is empty, in which case all ids are included.
+// Variable names are only added to the list once the first time they
+// are encountered.
 std::vector<std::string>
-StateDescriptor::GetVariableNames(const std::vector<std::string> &req_names,
+StateDescriptor::GetVariableNames(const std::vector<std::string> &requested_names,
                                   const Metadata::FlagCollection &flags,
                                   const std::vector<int> &sparse_ids) {
   std::unordered_set<int> sparse_ids_set(sparse_ids.begin(), sparse_ids.end());
   std::unordered_set<std::string> names;
   std::vector<std::string> names_vec;
   // first add names that are present
-  for (const auto &name : req_names) {
+  for (const auto &name : requested_names) {
     if (FieldPresent(name)) {
       if (metadataMap_[VarID(name)].IsSet(Metadata::Sparse)) {
         PARTHENON_THROW("Cannot ask for variables by name with specific sparse ids");
@@ -504,18 +512,18 @@ StateDescriptor::GetVariableNames(const std::vector<std::string> &req_names,
   return names_vec;
 }
 std::vector<std::string>
-StateDescriptor::GetVariableNames(const std::vector<std::string> &req_names,
+StateDescriptor::GetVariableNames(const std::vector<std::string> &requested_names,
                                   const std::vector<int> &sparse_ids) {
-  return GetVariableNames(req_names, Metadata::FlagCollection(), sparse_ids);
+  return GetVariableNames(requested_names, Metadata::FlagCollection(), sparse_ids);
 }
 std::vector<std::string>
 StateDescriptor::GetVariableNames(const Metadata::FlagCollection &flags,
                                   const std::vector<int> &sparse_ids) {
   return GetVariableNames({}, flags, sparse_ids);
 }
 std::vector<std::string>
-StateDescriptor::GetVariableNames(const std::vector<std::string> &req_names) {
-  return GetVariableNames(req_names, Metadata::FlagCollection(), {});
+StateDescriptor::GetVariableNames(const std::vector<std::string> &requested_names) {
+  return GetVariableNames(requested_names, Metadata::FlagCollection(), {});
 }
 std::vector<std::string>
 StateDescriptor::GetVariableNames(const Metadata::FlagCollection &flags) {

src/interface/state_descriptor.hpp

@@ -102,9 +102,9 @@ class StateDescriptor {
   // Preferred constructor
   explicit StateDescriptor(std::string const &label) : label_(label) {
     if (Metadata::FlagNameExists(label)) {
-      AddParam("MetadataFlag", Metadata::GetUserFlag(label));
+      AddParam("PackageMetadataFlag_", Metadata::GetUserFlag(label));
     } else {
-      AddParam("MetadataFlag", Metadata::AddUserFlag(label));
+      AddParam("PackageMetadataFlag_", Metadata::AddUserFlag(label));
     }
   }
 
@@ -114,7 +114,9 @@ class StateDescriptor {
   static std::shared_ptr<StateDescriptor>
   CreateResolvedStateDescriptor(Packages_t &packages);
 
-  MetadataFlag GetMetadataFlag() { return params_.Get<MetadataFlag>("MetadataFlag"); }
+  MetadataFlag GetMetadataFlag() {
+    return params_.Get<MetadataFlag>("PackageMetadataFlag_");
+  }
 
   template <typename T>
   void AddParam(const std::string &key, T value, Params::Mutability mutability) {

src/mesh/mesh.cpp

@@ -3,7 +3,7 @@
 // Copyright(C) 2014 James M. Stone <jmstone@princeton.edu> and other code contributors
 // Licensed under the 3-clause BSD License, see LICENSE file for details
 //========================================================================================
-// (C) (or copyright) 2020-2022. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2020-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC
@@ -492,7 +492,6 @@ Mesh::Mesh(ParameterInput *pin, ApplicationInput *app_in, Packages_t &packages,
                         packages, resolved_packages, gflag);
     block_list[i - nbs]->SearchAndSetNeighbors(tree, ranklist.data(), nslist.data());
   }
-  mesh_data.Get()->Set(block_list, "base");
 
   ResetLoadBalanceVariables();
 
@@ -1201,6 +1200,9 @@ void Mesh::Initialize(bool init_problem, ParameterInput *pin, ApplicationInput *
     }
   } while (!init_done);
 
+  // Initialize the "base" MeshData object
+  mesh_data.Get()->Set(block_list, "base");
+
   Kokkos::Profiling::popRegion(); // Mesh::Initialize
 }
 

src/mesh/meshblock.cpp

@@ -3,7 +3,7 @@
 // Copyright(C) 2014 James M. Stone <jmstone@princeton.edu> and other code contributors
 // Licensed under the 3-clause BSD License, see LICENSE file for details
 //========================================================================================
-// (C) (or copyright) 2020-2022. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2020-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC

src/parthenon/driver.hpp

@@ -1,5 +1,5 @@
 //========================================================================================
-// (C) (or copyright) 2020-2021. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2020-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC

tst/unit/test_meshblock_data_iterator.cpp

@@ -3,7 +3,7 @@
 // Copyright(C) 2014 James M. Stone <jmstone@princeton.edu> and other code contributors
 // Licensed under the 3-clause BSD License, see LICENSE file for details
 //========================================================================================
-// (C) (or copyright) 2020-2022. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2020-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC

tst/unit/test_metadata.cpp

@@ -1,5 +1,5 @@
 //========================================================================================
-// (C) (or copyright) 2020. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2020-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC