Template Class ProcessGroupWorker

• Defined in File ProcessGroupWorker.hpp

Class Documentation

template<typename CombiDataType = double>
class ProcessGroupWorker

The ProcessGroupWorker comprises a TaskWorker and a SparseGridWorker; it is responsible for executing the time stepping in the combination technique as well as various I-O operations.

When using a manager-worker scheme, the ProcessGroupWorker is instantiated on each worker rank; else, it is instantiated on each rank.

Public Functions

explicit ProcessGroupWorker()

SignalType wait()

wait for command from manager

void runAllTasks()

call run on each Task once

void exit()

release all tasks

inline const std::vector<std::unique_ptr<Task<CombiDataType>>> &getTasks() const

get the vector of tasks

void initCombinedDSGVector()

initializes all subspace sizes in the distributed sparse grid data structure

void updateFullFromCombinedSparseGrids()

extracts data from sparse to component grids and dehierarchizes the component grids

void dehierarchizeAllTasks()

dehierarchize all component grids

void combineAtOnce(bool collectMinMaxCoefficients = false)

do a whole combination, assuming a single-system setup

Parameters:

collectMinMaxCoefficients – if true, the min and max coefficients per subspace will be collected as part of the combination

void combineSystemWide()

perform the first part of a system-wide combination, namely hierarchization and reduction (but not dehierarchization)

based on file-exchange mechanism (w/o third level manager)

void combineSystemWideAndWrite(const std::string &writeSparseGridFile, const std::string &writeSparseGridFileToken)

perform the first part of a system-wide combination, but reducing to output ranks only instead of all ranks, which then write data to file.

based on file-exchange mechanism (w/o third level manager)

void parallelEval()

interpolate the current solution from all component grids on this group

receives the target level and file name from the ProcessGroupManager

void parallelEvalUniform(const std::string &filename, const LevelVector &leval) const

called by parallelEval to interpolate at resolution level leval and write the results to a binary file readable with Paraview

void doDiagnostics()

do task-specific diagnostics / postprocessing

void doDiagnostics(size_t taskID)

std::vector<double> getLpNorms(int p) const

calculate the Lp Norm for each individual task

std::vector<CombiDataType> interpolateValues(const std::vector<std::vector<real>> &interpolationCoordinates) const

interpolate values on all tasks’ component grids

Parameters:

interpolationCoordinates – the coordinates to interpolate at

Returns:

the interpolated values, one vector of values per task

void writeInterpolatedValuesPerGrid(const std::vector<std::vector<real>> &interpolationCoords, const std::string &fileNamePrefix) const

interpolate values on all tasks’ component grids and write results to file

one file per component grid, values are not combined

void writeInterpolatedValuesSingleFile(const std::vector<std::vector<real>> &interpolationCoords, const std::string &filenamePrefix) const

interpolate values on all tasks’ component grids and write results to file

one file for all component grids; values are combined with the combination formula

void writeSparseGridMinMaxCoefficients(const std::string &fileNamePrefix) const

write the highest and smallest sparse grid coefficient per subspace

int writeDSGsToDisk(const std::string &filenamePrefix, const std::string &writeCompleteTokenFileName)

write extra sparse grids to disk (binary w/ MPI-IO)

this will only have an effect on the output ranks, as they are the only ones that have an extra sparse grid. used for widely-distributed simulations with files (no third level manager)

int readDSGsFromDisk(const std::string &filenamePrefix, bool alwaysReadFullDSG = false)

read extra SGs from disk (binary w/ MPI-IO)

this will only have an effect on the output ranks, as they are the only ones that have an extra sparse grid

used for widely-distributed simulations with files (no third level manager)

void setCombiParameters(CombiParameters &&combiParameters)

set the CombiParameters member

usually after signal from ProcessGroupManager (when in manager-worker setup) or directly (when in worker-only setup)

inline CombiParameters &getCombiParameters()

get a reference to combi parameters

void combineThirdLevel()

perform a whole widely-distributed combination

based on TCP/socket setup with third level manager.

int combineThirdLevelFileBasedWrite(const std::string &filenamePrefixToWrite, const std::string &writeCompleteTokenFileName)

write the contents of the extra sparse grids to disk

based on file-exchange mechanism (w/o third level manager); should only be called from the ranks in the output group.

Parameters:

• filenamePrefixToWrite – the prefixes of the file to write, will have the species and (if partitioned output) the partition number appended

• writeCompleteTokenFileName – either the name of the token file to write, or the prefix (in case the outputComm is set, for partitioned file output), in which case the partition number will be appended

void waitForTokenFile(const std::string &startReadingTokenFileName) const

wait for a token file to appear.

one rank polls for the token file to appear (every some microseconds), and signals to the other output ranks once it does.

based on file-exchange mechanism (w/o third level manager); should only be called from the ranks in the output group.

int readReduce(const std::vector<std::string> &filenamePrefixesToRead, const std::vector<std::string> &startReadingTokenFileNames, bool overwriteInMemory)

read the contents of the extra sparse grids from disk and reduce them

based on file-exchange mechanism (w/o third level manager); should only be called from the ranks in the output group.

Parameters:

• filenamePrefixToRead – the prefixes of the file to read, will have the species and (if partitioned output) the partition number appended

• startReadingTokenFileName – either the name of the token file to read, or the prefix (in case the outputComm is set, for partitioned file output), in which case the partition number will be appended

• overwriteInMemory – whether to overwrite the values in the extra sparse grid (only true for testing)

void combineThirdLevelFileBasedReadReduce(const std::vector<std::string> &filenamePrefixesToRead, const std::vector<std::string> &startReadingTokenFileNames, bool overwrite = false, bool keepSparseGridFiles = false)

perform the second part of the widely-distributed combination, namely reading the reduced sparse grids from disk and combining them with the local solution

based on file-exchange mechanism (w/o third level manager); this function should only be called from the ranks in the output group, cf. combineReadDistributeSystemWide

Parameters:

• filenamePrefixesToRead – the prefixes of the files to read

• startReadingTokenFileNames – the names or prefixes of the token files to wait for

• overwrite – whether to overwrite the values in the extra sparse grid (only true for testing)

• keepSparseGridFiles – whether to keep the sparse grid files after reading

void combineReadDistributeSystemWide(const std::vector<std::string> &filenamePrefixesToRead, const std::vector<std::string> &startReadingTokenFileNames, bool overwrite = false, bool keepSparseGridFiles = false)

perform the second part of the widely-distributed combination, namely reading the reduced sparse grids from disk and combining them with the local solution

based on file-exchange mechanism (w/o third level manager); this function should be called from all worker ranks

Parameters:

• filenamePrefixToRead – the prefixes of the files to read

• startReadingTokenFileName – the names or prefixes of the token files to wait for

• overwrite – whether to overwrite the values in the extra sparse grid (only true for testing)

• keepSparseGridFiles – whether to keep the sparse grid files after reading *

void combineThirdLevelFileBased(const std::string &filenamePrefixToWrite, const std::string &writeCompleteTokenFileName, const std::vector<std::string> &filenamePrefixToRead, const std::vector<std::string> &startReadingTokenFileName)

perform a whole widely-distributed combination

based on file-exchange mechanism (w/o third level manager); this function should be called from all worker ranks.

equivalent to calling combineSystemWideAndWrite and combineReadDistributeSystemWide in succession

void waitForThirdLevelCombiResult(bool fromOutputGroup = false)

waits until the third level pg or output group broadcasts the combined solution, then updates the component grids

used for both kinds of widely-distributed simulations (with and without third level manager)

Parameters:

fromOutputGroup – whether the signal comes from the output group or from the third level group

void reduceSubspaceSizesThirdLevel(bool thirdLevelExtraSparseGrid)

computes a max reduce on the dsg’s subspace sizes with the other systems

Reduces subspace sizes with remote.

int reduceExtraSubspaceSizes(const std::vector<std::string> &filenameToRead, bool overwrite = false)

reduce the subspaces sizes in the extra sparse grid from the sizes given in a file

Parameters:

• filenameToRead – the file to read the sizes from

• overwrite – whether to overwrite the sizes in the extra sparse grid (only true for testing)

int reduceExtraSubspaceSizesFileBased(const std::string &filenamePrefixToWrite, const std::string &writeCompleteTokenFileName, const std::vector<std::string> &filenamePrefixToRead, const std::vector<std::string> &startReadingTokenFileName)

reduce the subspaces sizes in the extra sparse grid with another system

Parameters:

• filenamePrefixToWrite – the prefix of the file to write the own sizes to

• writeCompleteTokenFileName – the name of the token file to write

• filenamePrefixToRead – the prefixes of the files to read the sizes from

• startReadingTokenFileName – the names or prefixes of the token files to wait for

void waitForThirdLevelSizeUpdate()

resize the sparse grid data structures according to sizes received from the third level process group

used for widely-distributed simulations based on TCP/socket setup with third level manager

inline std::vector<std::unique_ptr<DistributedSparseGridUniform<CombiDataType>>> &getCombinedDSGVector()

inline std::vector<std::unique_ptr<DistributedSparseGridUniform<CombiDataType>>> &getExtraDSGVector()

get the extra sparse grid data structures

template<typename TaskType, typename ...TaskArgs>
inline void initializeAllTasks(const std::vector<LevelVector> &levels, const std::vector<combigrid::real> &coeffs, const std::vector<size_t> &taskNumbers, TaskArgs&&... args)

initialize all tasks at once

used in worker-only setup

void initializeTask(std::unique_ptr<Task<CombiDataType>> t)

store task and run its init function with current combiParameters

Parameters:

t – pointer to a heap-allocated task, the function takes over ownership here

inline IndexType getCurrentNumberOfCombinations() const

get the current number of combinations

void zeroDsgsData()

sets all subspaces in all dsgs to zero; allocates them if necessary

inline const TaskWorker<CombiDataType> &getTaskWorker() const

get a reference to the task worker member

inline const SparseGridWorker<CombiDataType> &getSparseGridWorker() const

get a reference to the sparse grid worker member