Parallel computations

OGGM is designed to use the available resources as well as possible. For single nodes machines but with more than one processor (frequent case for personal comuters) OGGM ships with a multiprocessing approach which is fairly simple to use. For cluster environments, use MPI.


Most OGGM computations are embarrassingly parallel: they are standalone operations to be realized on one single glacier entity and therefore independent from each other (they are called entity tasks, as opposed to the non-parallelizable global tasks).

When given a list of Glacier working directories on which to apply a given task, the workflow.execute_entity_task() will distribute the operations on the available processors using Python’s multiprocessing module. You can control this behavior with the use_multiprocessing config parameter and the number of processors with mp_processes. The default in OGGM is:

In [1]: from oggm import cfg

In [2]: cfg.initialize()

In [3]: cfg.PARAMS['use_multiprocessing']  # whether to use multiprocessing
Out[3]: True

In [4]: cfg.PARAMS['mp_processes']  # number of processors to use
Out[4]: -1

-1 means that all available processors will be used.


OGGM can be run in a clustered environment, using standard mpi features. OGGM depends on mpi4py in that case, which can be installed via either conda:

conda install -c conda-forge mpi4py

or pip:

pip install mpi4py

mpi4py itself depends on a working mpi environment, which is usually supplied by the maintainers of your cluster. On conda, it comes with its own copy of mpich, which is nice and easy for quick testing, but likely undesireable for the performance of actual runs.

For an actual run, invoke any script using oggm via mpiexec, and pass the --mpi parameter to the script itself:

mpiexec -n 10 python ./ --mpi

Be aware that the first process with rank 0 is the manager process, that by itself does not do any calculations and is only used to distribute tasks. So the actual number of working processes is one lower than the number passed to mpiexec/your clusters scheduler.