Template Class DistributedFullGrid

Defined in File DistributedFullGrid.hpp

Inheritance Relationships

Derived Type

public combigrid::OwningDistributedFullGrid< FG_ELEMENT, DIM > (Template Class OwningDistributedFullGrid)

Class Documentation

template<typename FG_ELEMENT, DimType DIM> class DistributedFullGrid

DistributedFullGrid : a (non-owning) indexed full grid data structure.

The DistributedFullGrid accesses a component grid that lives in distributed memory, typically distributed on a cartesian grid within a DisCoTec process group.

Because of the distributed-memory character, there are four different ways of indexing the grid:

global vector index: d-dimensional index in the global grid, ranges from 0^d to this->getGlobalSizes()
local vector index: d-dimensional index in the local part of the global grid, ranges from 0^d to this->getLocalSizes(), which corresponds to this->getLowerBounds to this->getUpperBounds() in the global grid
global linear index: linear index in the global grid, ranges from 0 to this->getNrElements()
local linear index: linear index in the local grid, ranges from 0 to this->getNrLocalElements() ; these indices do not have to be contiguous in the global grid!

The conversion between vector and linear indices is by Fortran order, i.e. elements that differ by 1 in the vector index are the most contiguous in the linear index.

For efficient iteration over all the indices in this grid, you should prefer the local linear index.

Template Parameters:: FG_ELEMENT – the data type to be stored on the grid (e.g. double, complex, etc.)

Subclassed by combigrid::OwningDistributedFullGrid< FG_ELEMENT, DIM >

Public Types

using ElementType = FG_ELEMENT 

using LevelsType = LevelArray<DIM>

using BoundaryFlagsType = BoundaryArray<DIM>

using SpacingType = std::array<double, DIM>

using DimIndexType = IndexArray<DIM>

using IndexerType = TensorIndexer<DIM>

using TensorType = TensorDim<FG_ELEMENT, DIM>

using DecompositionType = std::array<IndexVector, DIM>

Public Functions

DistributedFullGrid(DimType dim, const LevelVector &levels, CommunicatorType const &comm, const std::vector<BoundaryType> &hasBdrPoints, FG_ELEMENT *dataPointer, const std::vector<int> &procs, bool forwardDecomposition = false, const std::vector<IndexVector> &decomposition = std::vector<IndexVector>())

DistributedFullGrid constructor

Parameters:

dim – the dimensionality of the full grid (may become a template parameter in future versions)
levels – the level vector describing the full grid
comm – the Cartesian communicator to be used for the distributed full grid (will not be duplicated; ownership is not transferred)
hasBdrPoints – a vector of flags to show if the dimension has boundary points (0 for no points, 1 for one-sided boundary, 2 for both sides)
dataPointer – a pointer to the beginning of the data array
procs – a vector of the number of processes in each dimension (must match the decomposition in comm)
forwardDecomposition – a flag to decide if the middle grid points on the process boundaries belong to the process on the right-hand side (true) of the process boundary, or to the one on the left-hand side (false).
decomposition – a vector of the lower bounds of the grid points in each dimension

virtual ~DistributedFullGrid()

inline void getCoordsGlobal(IndexType globalLinearIndex, std::vector<real> &coords) const

get the coordinates on the unit square corresponding to global linear index

Parameters:

globalIndex – [IN] global linear index of the element
coords – [OUT] coordinates on the unit square [0,1]^D

inline void getCoordsLocal(IndexType localLinearIndex, std::vector<real> &coords) const

get coordinates on the unit square corresponding to local linear index

Parameters:

localLinearIndex – [IN] local linear index of the element
coords – [OUT] coordinates

inline void getGlobalLI(IndexType elementIndex, LevelArray<DIM> &levels, IndexArray<DIM> &indices) const

inline void getGlobalVectorIndex(IndexType globLinIndex, IndexArray<DIM> &globAxisIndex) const

inline void getGlobalVectorIndex(const IndexArray<DIM> &locAxisIndex, IndexArray<DIM> &globAxisIndex) const

inline void getLocalVectorIndex(IndexType locLinIndex, IndexArray<DIM> &locAxisIndex) const

inline bool getLocalVectorIndex(const IndexArray<DIM> &globAxisIndex, IndexArray<DIM> &locAxisIndex) const

inline IndexType getGlobalLinearIndex(const IndexArray<DIM> &globAxisIndex) const

inline IndexType getGlobalLinearIndex(IndexType locLinIndex) const

inline IndexType getLocalLinearIndex(const IndexArray<DIM> &locAxisIndex) const

inline IndexType getLocalLinearIndex(IndexType globLinIndex) const

inline bool isGlobalIndexHere(const IndexArray<DIM> &globalVectorIndex) const

inline bool isGlobalIndexHere(IndexType globLinIndex) const

inline const IndexArray<DIM> &getOffsets() const: get the strides per dimension for global vector index conversion

inline const IndexArray<DIM> &getLocalOffsets() const: get the strides per dimension for local vector index conversion

inline std::tuple<std::array<int, DIM>, std::array<int, DIM>, std::array<int, DIM>> getSizesSubsizesStartsOfSubtensor() const

get the sizes, subsizes, and starts vectors of the local data

as required to create a MPI subarray datatype

inline const LevelsType &getLevels() const: get the level vector

inline const SpacingType &getGridSpacing() const

get the grid spacing (sometimes called h)

Returns:: the distance between points in each dimension, assuming equidistant grid on unit hypercube

inline double getInverseGridSpacingIn(DimType inDimension) const

get the inverse grid spacing (1/h) for a given dimension

Parameters:: inDimension – the dimension
Returns:: the inverse of getGridSpacing()[inDimension]

inline std::array<double, DIM> getInverseGridSpacing() const: get a vector of inverse grid spacings

double getInnerNodalBasisFunctionIntegral() const

get the integral of a “normal” nodal hat basis function on this grid

helper function for integration

Returns:: the integral of a nodal basis function

inline IndexType getNrElements() const: get the number of elements in the entire, global full grid

inline IndexType getNrLocalElements() const: get the number of elements in the local partition

inline const IndexArray<DIM> &getLocalSizes() const: get vector extents of the local grid

inline const IndexArray<DIM> &getGlobalSizes() const: get vector extents of the global grid

inline IndexType globalNumPointsInDimension(DimType i) const: number of points per dimension i

inline const BoundaryFlagsType &returnBoundaryFlags() const: get vector of flags to show how many boundary points this dimension has

inline FG_ELEMENT *getData(): get pointer to the beginning of the local data

inline const FG_ELEMENT *getData() const: get const pointer to the beginning of the local data

inline void setZero()

zero out all values in the local storage

uses memset for speed

inline CommunicatorType getCommunicator() const: get local Cartesian MPI_Communicator

inline int getCommunicatorSize() const: get the size of the local Cartesian MPI communicator

inline RankType getRank() const: get the rank in the local Cartesian MPI communicator

inline const DimIndexType &getLowerBounds() const: get lower global vector bounds of the local data in this process

inline DimIndexType getLowerBounds(RankType r) const: get lower global vector bounds of the local data in rank r

inline real getLowerBoundsCoord(DimType inDimension) const: get coordinates of this process’ lower bounds in some dimension

inline const DimIndexType &getUpperBounds() const: get upper global vector bounds of the local data in this process

inline DimIndexType getUpperBounds(RankType r) const: get upper global vector bounds of the local data in rank r

inline std::array<real, DIM> getUpperBoundsCoords() const: get coordinates of this process’ upper bounds

inline std::array<real, DIM> getUpperBoundsCoords(RankType r) const: get coordinates of rank r’s upper bounds

RankType getNeighbor1dFromAxisIndex(DimType dim, IndexType idx1d) const

get the neighboring process’ rank in a given dimension

in the sense that the neighbor has the same partion coordinates in all other dimensions than d; and in dimension d it contains the point with the one-dimensional index idx1d

@ param dim the dimension in which the neighbor is @ param idx1d the one-dimensional index of the point in the neighbor’s partition

inline const std::array<int, DIM> &getParallelization() const: get the number of cartesian ranks in every dimension

inline IndexType getFirstGlobal1dIndex(DimType d) const: get the 1d global index of the first point in the local domain

inline IndexType getLastGlobal1dIndex(DimType d) const: get the 1d global index of the last point in the local domain

inline LevelType getLevel(DimType d, IndexType idx1d) const

get the (lowest possible) hierarchical level of a global 1d index

Parameters:

d – the dimension
idx1d – the global 1d index

inline IndexType getLeftPredecessor(DimType d, IndexType idx1d) const

get 1d index of the left hierarchical predecessor of a point

if this point has level l, return the 1d index of the closest point with lower 1d index and level l-1

Parameters:

d – the dimension
idx1d – the global 1d index

Returns:

negative number if point has no left predecessor, else 1d index of the left predecessor

inline IndexType getRightPredecessor(DimType d, IndexType idx1d) const

get 1d index of the right hierarchical predecessor of a point

if this point has level l, return the 1d index of the closest point with higher 1d index and level l-1

Parameters:

d – the dimension
idx1d – the global 1d index

Returns:

negative number if point has no right predecessor, else 1d index of the right predecessor

inline void getPartitionCoords(const IndexVector &globalAxisIndex, std::vector<int> &partitionCoords) const

get process coordinates of the partition which contains a given point

Parameters:

globalAxisIndex – [IN] the global vector index of the point
partitionCoords – [OUT] the partition coordinates of the process containing the point

inline void print() const

print values on the grid

(convenience function up to 3d)

inline MPI_Datatype getMPIDatatype() const: get the MPI_Datatype corresponding to FG_ELEMENT

inline double getPointDistanceToCoordinate(IndexType oneDimensionalLocalIndex, double coord, DimType d) const

One-D distance between a grid point (by linear index) and a coordinate

used as a helper function for interpolation

Parameters:

oneDimensionalLocalIndex – the (local) linear index of the grid point
coord – ND coordinates within the support of the grid point (param 1)
d – the dimension along which the distance is to be calculated

Returns:

the distance between the grid point and the coordinate in the specified dimension

inline FG_ELEMENT evalIndexAndAllUpperNeighbors(const IndexArray<DIM> &localIndex, const std::array<real, DIM> &coords) const

Performs the interpolation of the full grid at the specified coordinates

assumes that the localIndex is the highest index whose coordinates on the unit cube are still <= coords

Parameters:

localIndex – the local linear index of the grid point
coords – ND coordinates on the unit square [0,1]^D

Returns:

the interpolated value assuming nodal hat basis functions

FG_ELEMENT evalLocal(const std::vector<real> &coords) const

void evalLocal(const std::vector<real> &coords, FG_ELEMENT &value) const

FG_ELEMENT evalLocal(const std::array<real, DIM> &coords) const

void evalLocal(const std::array<real, DIM> &coords, FG_ELEMENT &value) const

inline void getCoordsGlobal(IndexType globalLinearIndex, std::array<real, DIM> &coords) const

inline std::array<real, DIM> getCoordsGlobal(IndexType globalLinearIndex) const

inline void getCoordsLocal(IndexType localLinearIndex, std::array<real, DIM> &coords) const

inline std::array<real, DIM> getCoordsLocal(IndexType localLinearIndex) const

std::vector<FG_ELEMENT> getInterpolatedValues(const std::vector<std::vector<real>> &interpolationCoords) const

evaluates the full grid on a set of specified coordinates

Parameters:: interpolationCoords – vector of ND coordinates on the unit square [0,1]^D
Returns:: vector of interpolated values

void gatherFullGrid(FullGrid<FG_ELEMENT> &fg, RankType root)

Gather a full grid from all processes to the root process

The root process will have the full grid. Only possible for small grids!

Parameters:: fg – the full grid to gather to

inline std::vector<IndexType> getFGPointsOfSubspace(const LevelVector &l) const

Get the indices of the points of the subspace on this partition.

Parameters:: l – level of hierarchical subspace
Returns:: the indices of points on this partition

template<bool sparseGridFullyAllocated = true> size_t extractFromUniformSG(const DistributedSparseGridUniform<FG_ELEMENT> &dsg)

extracts the (hopefully) hierarchical coefficients from dsg to the full grid’s data structure

Parameters:: dsg – the DSG to extract from

inline IndexType getNumPointsOnThisPartition(LevelType l, DimType d) const

get the number of points of a given hiararchical 1d-level on this process’ grid partition

Parameters:

l – the level
d – the dimension

Returns:

the number of 1d points on this partition

inline void get1dIndicesLocal(DimType d, LevelType l, IndexVector &oneDIndices) const

get the local 1d indices that correspond to a given level

Parameters:

d – the dimension in which we want the indices for the level
l – the level
oneDIndices – [OUT] a list of the local vector indices, length is getNumPointsOnThisPartition(d,l)

real getLpNorm(int p) const

Get the Lp Norm of the data on the dfg: norm = (sum_i(abs(x_i)^p)*integral(basis_i))^1/p.

Parameters:: p – : the (polynomial) parameter, 0 is interpreted as maximum norm
Returns:: real : the norm

void writePlotFile(const char *filename) const: write data to dense binary file using MPI-IO

void writePlotFileVTK(const char *filename) const: write data to legacy-type VTK file using MPI-IO

inline const DecompositionType &getDecomposition() const

get the decomposition vector

gives the exact correspondence of 1d indices to cartesian ranks: a vector of the grid point indices at which a process boundary is assumed (for every dimension)

Returns:: the decomposition vector of length this->getDimension()

void getHighestAndLowestNeighbor(DimType d, int &highest, int &lowest) const: get the ranks of the highest and lowest “neighbor” rank in dimension d only sets highest and lowest if they actually are my neighbors

void writeLowerBoundaryToUpperBoundary(DimType d)

periodic boundary exchange from lower to upper boundary

Parameters:: d – the dimension which we want to exchange

std::vector<FG_ELEMENT> exchangeGhostLayerUpward(DimType d, std::vector<int> &subarrayExtents)

send the data on the highest-indexed layer in dimension d to the upward-neighboring process (and receive the data from the downward-neighboring process)

Parameters:

d – the dimension in which we want to exchange
subarrayExtents – [out] the extents of the sent subarray (1 in dimension d and local size in all other dimensions)

Returns:

the received data

void exchangeGhostLayerUpward(DimType d, std::vector<int> &subarrayExtents, std::vector<FG_ELEMENT> &recvbuffer, MPI_Request *recvRequest = nullptr): non-RVO dependent version of ghost layer exchange

std::array<bool, DIM> isGlobalLinearIndexOnBoundary(IndexType globalLinearIndex) const: check if given globalLinearIndex is on the boundary of this DistributedFullGrid

std::array<bool, DIM> isLocalLinearIndexOnBoundary(IndexType localLinearIndex) const: check if given localLinearIndex is on the boundary of this DistributedFullGrid

std::vector<IndexArray<DIM>> getCornersGlobalVectorIndices() const: get a vector containing the global vector indices of the 2^d corners of this dfg

std::vector<FG_ELEMENT> getCornersValues() const

get a vector of the values at the grid’s global corners

Returns:: a vector of the values at the 2^d corners

inline const MPICartesianUtils<DIM> &getCartesianUtils() const: get the MPICartesianUtils associated with this

Public Static Functions

static inline constexpr DimType getDimension(): get the dimension of the full grid

Protected Functions

inline void setData(FG_ELEMENT *newData): set the data pointer, data needs to be allocated outside