Template Class SparseGridWorker

• Defined in File SparseGridWorker.hpp

Class Documentation

template<typename CombiDataType = double>
class SparseGridWorker

The SparseGridWorker class is responsible for managing the distributed sparse grids.

Public Functions

inline explicit SparseGridWorker(TaskWorker<CombiDataType> &taskWorkerToReference)

Constructor.

Parameters:

taskWorkerToReference – the task worker that this sparse grid worker should reference

template<bool keepValuesInExtraSparseGrid = false>
inline void collectReduceDistribute(CombinationVariant combinationVariant, uint32_t maxMiBToSendPerThread, bool collectMinMaxCoefficients = false)

collect, reduce, and distribute the sparse grid data

Template Parameters:

keepValuesInExtraSparseGrid – if true, the values that would be useful to the extra sparse grid are also gathered there (necessary for widely-distributed chunked reduction)

Parameters:

• combinationVariant – the combination variant to use

• maxMiBToSendPerThread – the maximum amount of data to send per OpenMP thread at once

• collectMinMaxCoefficients – if true, the min and max hierarchical coefficients are collected

inline void copyFromExtraDsgToPartialDSG(size_t gridNumber = 0)

copy data from the extra sparse grid to the partially allocated combined sparse grid

necessary for widely-distributed chunked reduction; copies only the subspaces allocated in both

inline void copyFromPartialDsgToExtraDSG(size_t gridNumber = 0)

copy data from the partially allocated combined sparse grid to the extra sparse grid

copies only the subspaces allocated in both.

necessary for widely-distributed chunked reduction; ; only implemented for file-based setup

inline void deleteDsgsData()

free sparse grid data memory

inline void distributeChunkedBroadcasts(uint32_t maxMiBToSendPerThread)

chunked broadcast of the sparse grid data

necessary for widely-distributed chunked reduction; only implemented for file-based setup

Parameters:

maxMiBToSendPerThread – the maximum amount of data to send per OpenMP thread at once

inline void distributeCombinedSolutionToTasks()

update the tasks’ full grids from the hierarchical sparse grid data

inline void fillDFGFromDSGU(Task<CombiDataType> &t, const std::vector<bool> &hierarchizationDims, const std::vector<BasisFunctionType> &hierarchicalBases, const LevelVector &lmin) const

fill a task’s full grid from the distributed sparse grid and dehierarchize

can only be used for non-chunked reductions

Parameters:

• dfg – the distributed full grid to fill

• boundary – the boundary conditions used for hierarchization

• hierarchizationDims – the hierarchized dimensions

• hierarchicalBases – the hierarchical bases used for hierarchization

• lmin – the minimum level vector the grids are hierarchized to

inline std::vector<std::unique_ptr<DistributedSparseGridUniform<CombiDataType>>> &getCombinedUniDSGVector()

get a reference to the combined distributed sparse grids

inline const std::vector<std::unique_ptr<DistributedSparseGridUniform<CombiDataType>>> &getCombinedUniDSGVector() const

inline std::vector<std::unique_ptr<DistributedSparseGridUniform<CombiDataType>>> &getExtraUniDSGVector()

get a reference to the extra distributed sparse grids

inline const std::vector<std::unique_ptr<DistributedSparseGridUniform<CombiDataType>>> &getExtraUniDSGVector() const

inline size_t getNumberOfGrids() const

get the number of grids (in the tasks and combined sparse grids)

Returns:

the number of grids, may be more than one e.g. for multiple ion species

inline std::unique_ptr<DistributedSparseGridUniform<CombiDataType>> &getSparseGridToUseForThirdLevel(bool thirdLevelExtraSparseGrid)

get a reference to the sparse grid to use for the widely-distributed reduction

Parameters:

thirdLevelExtraSparseGrid – if true, the third level extra sparse grid is set and returned, otherwise the “normal” sparse grid is returned

inline void initCombinedUniDSGVector(const LevelVector &lmin, LevelVector lmax, const LevelVector &reduceLmaxByVector, size_t numGrids, CombinationVariant combinationVariant, bool clearLevels = false)

initialize the combined distributed sparse grids

to be called after the tasks have been initialized, such that the necessary sizes can be set from the tasks

Initializes the dsgu for each species by setting the subspace sizes of all dfgs in the global reduce comm. After calling, all workers which share the same spatial distribution of the dsgu (those who combine during global reduce) have the same sized subspaces and thus share the same sized dsg.

Attention: No data is created here, only subspace sizes are shared.

Parameters:

• lmin – the minimum level vector to use for sparse grid construction

• lmax – the maximum level vector to use for sparse grid construction

• reduceLmaxByVector – the vector to reduce the maximum level vector by

• numGrids – the number of grids to initialize

• combinationVariant – the combination variant to use

• clearLevels – if true, the subspace levels are cleared before initialization

inline void interpolateAndPlotOnLevel(const std::string &filename, const LevelVector &levelToEvaluate, const std::vector<BoundaryType> &boundary, const std::vector<bool> &hierarchizationDims, const std::vector<BasisFunctionType> &hierarchicalBases, const LevelVector &lmin, const std::vector<int> &parallelization, const std::vector<LevelVector> &decomposition) const

interpolate the current combined sparse grid solution at the given levelToEvaluate, get the results at the given interpolationCoords, and write them to a file

Parameters:

interpolationCoords – the coordinates to interpolate at

Returns:

the interpolated values

inline void maxReduceSubspaceSizesInOutputGroup()

have the output group resize their subspace sizes according to the maximum between the groups

necessary for widely-distributed chunked reduction; only implemented for file-based setup

inline int readDSGsFromDisk(const std::string &filenamePrefixToRead, bool addSpeciesNumberToFileName, bool alwaysReadFullDSG = false)

read the sparse grid data from disk into the getSparseGridToUseForThirdLevel() grids

necessary for file-based widely-distributed combination

reads either a single file across the output communicator or multiple files in case of partitioned sparse grid output

Parameters:

• filenamePrefixToRead – the prefix of the file to read

• addSpeciesNumberToFileName – whether to add the species number to the filename

• alwaysReadFullDSG – whether to always read the full DSG

Returns:

the number of values read in total

inline int readDSGsFromDiskAndReduce(const std::string &filenamePrefixToRead, uint32_t maxMiBToReadPerThread, bool addSpeciesNumberToFileName, bool alwaysReadFullDSG = false)

read the sparse grid data from disk and sum-reduce it into the getSparseGridToUseForThirdLevel() grids

necessary for file-based widely-distributed combination

reads either a single file across the output communicator or multiple files in case of partitioned sparse grid output

Parameters:

• filenamePrefixToRead – the prefix of the file to read

• maxMiBToReadPerThread – the maximum amount of data to read per OpenMP thread at once

• addSpeciesNumberToFileName – whether to add the species number to the filename

• alwaysReadFullDSG – whether to always read the full DSG

Returns:

the number of values read in total

inline int readReduce(const std::vector<std::string> &filenamePrefixesToRead, const std::vector<std::string> &startReadingTokenFileNames, uint32_t maxMiBToReadPerThread, bool overwrite)

read the sparse grid data from disk and reduce it with the local solution

waits for a token file to appear, and when it is there calls readDSGsFromDiskAndReduce or readDSGsFromDisk, depending on overwrite

necessary for file-based widely-distributed combination

Parameters:

• filenamePrefixesToRead – the prefixes of the files to read

• startReadingTokenFileNames – the names of the token files to wait for

• maxMiBToReadPerThread – the maximum amount of data to read per OpenMP thread at once

• overwrite – whether to overwrite the values in the extra sparse grid (only true for testing)

Returns:

the number of values read in total

inline int reduceExtraSubspaceSizes(const std::vector<std::string> &filenamesToRead, CombinationVariant combinationVariant, bool overwrite)

reduce the subspace sizes in the extra sparse grid from the sizes given in other files

necessary for file-based widely-distributed combination; should be called from all ranks

inline void reduceLocalAndGlobal(CombinationVariant combinationVariant, uint32_t maxMiBToSendPerThread, RankType globalReduceRankThatCollects = MPI_PROC_NULL)

reduction within and between process groups

Parameters:

• combinationVariant – the combination variant to use

• maxMiBToSendPerThread – the maximum amount of data to send per OpenMP thread at once

• globalReduceRankThatCollects – the rank that collects the global reduction, or MPI_PROC_NULL for allreduce

inline void reduceSubspaceSizesBetweenGroups(CombinationVariant combinationVariant)

reduce the subspace sizes between the groups

necessary for all combination variants

inline void removeReadingFiles(const std::vector<std::string> &filenamePrefixToRead, const std::vector<std::string> &startReadingTokenFileName, bool keepSparseGridFiles) const

remove the read sparse grid files

removes exactly the files that were read in readDSGsFromDisk(AndReduce)

inline void setExtraSparseGrid(bool initializeSizes = true)

creates an extra sparse grid on the ranks that call it

necessary for widely-distributed combination if the extra sparse grid should be used

inline void startSingleBroadcastDSGs(CombinationVariant combinationVariant, RankType broadcastSender, MPI_Request *request)

starts an asynchronous broadcast of the sparse grid data from broadcastSender to the other ranks in the global reduce communicator

necessary for all types of widely-distributed combination

inline int writeDSGsToDisk(const std::string &filenamePrefix, CombinationVariant combinationVariant)

write the sparse grid data to disk

writes a single file across the output communicator, or multiple files in case of partitioned output.

should only be called from the ranks in the output group

Parameters:

• filenamePrefix – the prefix of the file to write

• combinationVariant – the combination variant to use

Returns:

the number of values written in total

inline int writeExtraSubspaceSizesToFile(const std::string &filenamePrefixToWrite) const

write the subspace sizes of the extra sparse grid to a file

should only be called from the ranks in the output group

needed for file-based widely-distributed combination

inline void writeMinMaxCoefficients(const std::string &fileNamePrefix) const

write the minimum and maximum hierarchical coefficients to a file

this requires that they were collected during the reduction, see collectReduceDistribute

inline void writeTokenFiles(const std::string &writeCompleteTokenFileName) const

write the token file(s) to signal that the sparse grid file(s) are written

should only be called from the ranks in the output group, see writeDSGsToDisk

inline int writeSubspaceSizesToFile(const std::string &filenamePrefixToWrite) const

write the subspace sizes of the combined sparse grid to a file

should only be called from the ranks in the output group

needed for file-based widely-distributed combination

inline void zeroDsgsData(CombinationVariant combinationVariant)

zero-initialize or zero-overwrite the sparse grid data