Merge branch 'develop' into lroberts36/add-noncell-amr

CHANGELOG.md

@@ -3,6 +3,7 @@
 ## Current develop
 
 ### Added (new features/APIs/variables/...)
+- [[PR 921]](https://github.com/parthenon-hpc-lab/parthenon/pull/921) Add more flexible ways of adding and using MeshData/MeshBlockData objects to DataCollections
 - [[PR 900]](https://github.com/parthenon-hpc-lab/parthenon/pull/900) Add Morton numbers and expand functionality of LogicalLocation
 - [[PR 902]](https://github.com/parthenon-hpc-lab/parthenon/pull/902) Add ability to output NaNs for de-allocated sparse fields
 - [[PR 887]](https://github.com/parthenon-hpc-lab/parthenon/pull/887) Add ability to dump more types of params and read them from restarts
@@ -21,6 +22,7 @@
 - [[PR 885]](https://github.com/parthenon-hpc-lab/parthenon/pull/885) Expose PackDescriptor and use uids in SparsePacks
 
 ### Fixed (not changing behavior/API/variables/...)
+- [[PR 917]](https://github.com/parthenon-hpc-lab/parthenon/pull/917) Update Iterative Tasking Infrastructure
 - [[PR 890]](https://github.com/parthenon-hpc-lab/parthenon/pull/890) Fix bugs in sparse communication and prolongation
 
 ### Infrastructure (changes irrelevant to downstream codes)

doc/sphinx/src/interface/state.rst

@@ -66,6 +66,22 @@ several useful features and functions.
 - ``T *MutableParam(const std::string &key)`` returns a pointer to a
   parameter that has been marked mutable when it was added. Note this
   pointer is *not* marked ``const``.
+- ``MetadataFlag GetMetadataFlag()`` returns a ``MetadataFlag`` that is
+  automatically added to all fields, sparse pools, and swarms that are
+  added to the ``StateDescriptor``.
+- ``std::vector<std::string> GetVariableNames(...)`` provides a means of
+  getting a list of variables that satisfy specified requirements and that
+  are associated with the ``StateDescriptor``.  Optional arguments, in order,
+  include a vector of variable names, a ``Metadata::FlagCollection`` to select,
+  variables by flags, and a vector of sparse ids to allow selecting subsets of
+  sparse fields.  Selection of sparse fields by name requires providing the
+  base name of the sparse field.  Names of a sparse field tied to a specific
+  sparse index are not supported (and will throw).  The returned list is
+  appropriate for use in adding ``MeshData`` and/or ``MeshBlockData`` objects
+  with specified fields to the ``DataCollection`` objects in ``Mesh`` and
+  ``MeshBlock``.  For convenience, the ``Mesh`` class also provides this
+  function, which provides a list of variables gathered from all the package
+  ``StateDescriptor``s.
 - ``void FillDerivedBlock(MeshBlockData<Real>* rc)`` delgates to the
   ``std::function`` member ``FillDerivedBlock`` if set (defaults to
   ``nullptr`` and therefore a no-op) that allows an application to provide
@@ -293,6 +309,18 @@ downstream applications to allocate storage in a more targeted fashion,
 as might be desirable to hold source terms for particular equations, for
 example.
 
+Analogously, ``DataCollection`` provides ``AddShallow`` functions that
+differ from ``Add`` only in that ***all*** included variables, even
+non-``Metadata::OncCopy`` variables, are simply shallow copies.  For
+these functions, no new storage for variables is ever allocated.
+
+Finally, all of the functionality just described for ``MeshBlockData``
+objects is also provided for ``MeshData`` objects.  Adding a new
+``MeshData`` object to the ``Mesh``-level ``DataCollection`` automatically
+adds the corresponding ``MeshBlockData`` objects to each of the
+``MeshBlock``-level ``DataCollection``s.  Using this ``Mesh`` level
+functionality can be more convenient.
+
 Two simple examples of usage of these new containers are 1) to provide
 storage for multistage integration schemes and 2) to provide a mechanism
 to allocate storage for right hand sides, deltas, etc. Both of these

src/amr_criteria/amr_criteria.hpp

@@ -17,12 +17,13 @@
 #include <string>
 
 #include "defs.hpp"
-#include "interface/meshblock_data.hpp"
 #include "mesh/domain.hpp"
 
 namespace parthenon {
 
 class ParameterInput;
+template <class>
+class MeshBlockData;
 
 struct AMRBounds {
   AMRBounds(const IndexRange &ib, const IndexRange &jb, const IndexRange &kb)

src/basic_types.hpp

@@ -1,5 +1,5 @@
 //========================================================================================
-// (C) (or copyright) 2021-2022. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2021-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC
@@ -36,7 +36,7 @@ using Real = double;
 #endif
 #endif
 
-enum class TaskStatus { fail, complete, incomplete, iterate, skip };
+enum class TaskStatus { fail, complete, incomplete, iterate, skip, waiting };
 enum class AmrTag : int { derefine = -1, same = 0, refine = 1 };
 enum class RefinementOp_t { Prolongation, Restriction, None };
 

src/interface/data_collection.cpp

@@ -1,5 +1,5 @@
 //========================================================================================
-// (C) (or copyright) 2020. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2020-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC
@@ -22,43 +22,44 @@
 namespace parthenon {
 
 template <typename T>
-std::shared_ptr<T> DataCollection<T>::Add(const std::string &name,
-                                          const std::shared_ptr<T> &src,
-                                          const std::vector<std::string> &flags) {
+std::shared_ptr<T> &
+DataCollection<T>::Add(const std::string &name, const std::shared_ptr<T> &src,
+                       const std::vector<std::string> &field_names, const bool shallow) {
   auto it = containers_.find(name);
   if (it != containers_.end()) {
-    if (!(it->second)->Contains(flags)) {
-      PARTHENON_THROW(name + "already exists in collection but does not contain flags");
+    if (!(it->second)->Contains(field_names)) {
+      PARTHENON_THROW(name +
+                      "already exists in collection but does not contain field names");
     }
     return it->second;
   }
 
   auto c = std::make_shared<T>();
-  c->Copy(src, flags);
+  c->Initialize(src.get(), field_names, shallow);
 
-  containers_[name] = c;
-  return containers_[name];
-}
+  Set(name, c);
 
-template <typename T>
-std::shared_ptr<T> DataCollection<T>::Add(const std::string &name,
-                                          const std::shared_ptr<T> &src) {
-  // error check for duplicate names
-  auto it = containers_.find(name);
-  if (it != containers_.end()) {
-    // check to make sure they are the same
-    if (!(*src == *(it->second))) {
-      PARTHENON_THROW("Error attempting to add a Container to a Collection");
+  if constexpr (std::is_same<T, MeshData<Real>>::value) {
+    for (int b = 0; b < pmy_mesh_->block_list.size(); b++) {
+      auto &mbd = pmy_mesh_->block_list[b]->meshblock_data;
+      mbd.Set(name, c->GetBlockData(b));
     }
-    return it->second;
   }
 
-  auto c = std::make_shared<T>();
-  c->Copy(src);
-
-  containers_[name] = c;
   return containers_[name];
 }
+template <typename T>
+std::shared_ptr<T> &DataCollection<T>::Add(const std::string &label,
+                                           const std::shared_ptr<T> &src,
+                                           const std::vector<std::string> &flags) {
+  return Add(label, src, flags, false);
+}
+template <typename T>
+std::shared_ptr<T> &DataCollection<T>::AddShallow(const std::string &label,
+                                                  const std::shared_ptr<T> &src,
+                                                  const std::vector<std::string> &flags) {
+  return Add(label, src, flags, true);
+}
 
 template <>
 std::shared_ptr<MeshData<Real>> &

src/interface/data_collection.hpp

@@ -1,5 +1,5 @@
 //========================================================================================
-// (C) (or copyright) 2020. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2020-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC
@@ -28,6 +28,11 @@ class Mesh;
 /// Current usage includes (but is not limited to) storing MeshBlockData for different
 /// stages in multi-stage drivers or the corresponding MeshBlockPacks in a
 /// DataCollection of MeshData.
+///
+/// T must implement:
+///   bool Contains(std::vector<std::string>)
+///   Initialize(T*, std::vector<std::string>, bool)
+/// TODO: implement a concept
 template <typename T>
 class DataCollection {
  public:
@@ -38,10 +43,13 @@ class DataCollection {
 
   void SetMeshPointer(Mesh *pmesh) { pmy_mesh_ = pmesh; }
 
-  std::shared_ptr<T> Add(const std::string &label, const std::shared_ptr<T> &src,
-                         const std::vector<std::string> &flags);
-  std::shared_ptr<T> Add(const std::string &label, const std::shared_ptr<T> &src);
-  std::shared_ptr<T> Add(const std::string &label) {
+  std::shared_ptr<T> &Add(const std::string &label, const std::shared_ptr<T> &src,
+                          const std::vector<std::string> &flags, const bool shallow);
+  std::shared_ptr<T> &Add(const std::string &label, const std::shared_ptr<T> &src,
+                          const std::vector<std::string> &flags = {});
+  std::shared_ptr<T> &AddShallow(const std::string &label, const std::shared_ptr<T> &src,
+                                 const std::vector<std::string> &flags = {});
+  std::shared_ptr<T> &Add(const std::string &label) {
     // error check for duplicate names
     auto it = containers_.find(label);
     if (it != containers_.end()) {
@@ -64,6 +72,8 @@ class DataCollection {
     return it->second;
   }
 
+  void Set(const std::string &name, std::shared_ptr<T> &d) { containers_[name] = d; }
+
   std::shared_ptr<T> &GetOrAdd(const std::string &mbd_label, const int &partition_id);
 
   void PurgeNonBase() {

src/interface/make_pack_descriptor.hpp

@@ -31,10 +31,13 @@
 #include "interface/metadata.hpp"
 #include "interface/sparse_pack.hpp"
 #include "interface/state_descriptor.hpp"
+#include "interface/variable.hpp"
 #include "mesh/mesh.hpp"
 
 namespace parthenon {
 
+inline auto MakeDefaultPackDescriptor() { return typename SparsePack<>::Descriptor(); }
+
 inline auto MakePackDescriptor(StateDescriptor *psd, const std::vector<std::string> &vars,
                                const std::vector<bool> &use_regex,
                                const std::vector<MetadataFlag> &flags = {},
@@ -116,6 +119,23 @@ inline auto MakePackDescriptor(
   return MakePackDescriptor(psd, vars, use_regex, flags, options);
 }
 
+inline auto MakePackDescriptor(StateDescriptor *psd, const std::vector<Uid_t> &var_ids,
+                               const std::vector<MetadataFlag> &flags = {},
+                               const std::set<PDOpt> &options = {}) {
+  auto selector = [&](int vidx, const VarID &id, const Metadata &md) {
+    if (flags.size() > 0) {
+      for (const auto &flag : flags) {
+        if (!md.IsSet(flag)) return false;
+      }
+    }
+    if (Variable<Real>::GetUniqueID(id.label()) == var_ids[vidx]) return true;
+    return false;
+  };
+
+  impl::PackDescriptor base_desc(psd, var_ids, selector, options);
+  return typename SparsePack<>::Descriptor(base_desc);
+}
+
 } // namespace parthenon
 
 #endif // INTERFACE_MAKE_PACK_DESCRIPTOR_HPP_

src/interface/mesh_data.hpp

@@ -1,5 +1,5 @@
 //========================================================================================
-// (C) (or copyright) 2020-2022. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2020-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC
@@ -31,6 +31,7 @@
 #include "utils/communication_buffer.hpp"
 #include "utils/error_checking.hpp"
 #include "utils/object_pool.hpp"
+#include "utils/unique_id.hpp"
 #include "utils/utils.hpp"
 
 namespace parthenon {
@@ -236,14 +237,16 @@ class MeshData {
   }
 
   template <typename... Args>
-  void Copy(const std::shared_ptr<MeshData<T>> src, Args &&...args) {
-    if (src.get() == nullptr) {
+  void Initialize(const MeshData<T> *src, Args &&...args) {
+    if (src == nullptr) {
       PARTHENON_THROW("src points at null");
     }
+    pmy_mesh_ = src->GetParentPointer();
     const int nblocks = src->NumBlocks();
     block_data_.resize(nblocks);
     for (int i = 0; i < nblocks; i++) {
-      block_data_[i]->Copy(src->GetBlockData(i), std::forward<Args>(args)...);
+      block_data_[i] = std::make_shared<MeshBlockData<T>>();
+      block_data_[i]->Initialize(src->GetBlockData(i).get(), std::forward<Args>(args)...);
     }
   }
 
@@ -435,6 +438,12 @@ class MeshData {
   BvarsCache_t bvars_cache_;
 };
 
+template <typename T, typename... Args>
+std::vector<Uid_t> UidIntersection(MeshData<T> *md1, MeshData<T> *md2, Args &&...args) {
+  return UidIntersection(md1->GetBlockData(0).get(), md2->GetBlockData(0).get(),
+                         std::forward<Args>(args)...);
+}
+
 } // namespace parthenon
 
 #endif // INTERFACE_MESH_DATA_HPP_