YAC 3.21.0
Yet Another Coupler
Loading...
Searching...
No Matches
The Python bindings (yac.pyx)

Table of Contents

For a module reference see yac.

This package provides Python bindings for the YAC coupler. They are generated using cython and are automatically built when installing YAC via pip. The Python extension module (.so file) and tests are compiled during the installation process.

Dependencies

The bindings need only a handful of Python packages. python/requirements.txt lists more: it describes the complete environment for the YAC test suite and is not a prerequisite for building or using the bindings.

The following packages are required to build the bindings:

  • cython (version 3.0 or newer) to translate the .pyx sources into C; not needed at run time,
  • numpy for all array arguments; the only package that is also required at run time,
  • setuptools to run the generated setup.py; not needed at run time.

cython and setuptools can, therefore, be omitted from environments that only run YAC, such as a container image or a deployment venv.

setuptools only matters for a regular YAC build: Autotools runs the generated setup.py in the current environment and configure fails without it, while CMake does not use setuptools itself and only needs it when installing from the build directory, where pip normally provides it.

When installing with pip, none of them have to be present beforehand: pip resolves the build requirements declared in pyproject.toml in an isolated environment.

The following packages are optional. They are needed neither to build nor to import yac, but enable additional features and are used by the Python tests, examples and tools (tests are skipped if their dependencies are missing):

  • mpi4py to convert MPI communicators, e.g. in Component.comp_comm and get_comps_comm; it has to be built against the same MPI as YAC (see Using mpi4py with yac),
  • matplotlib for the plotting example component and the plotting tools,
  • netCDF4 for the NetCDF example components and yac.utils.read_grid,
  • scipy to process raw exchange data in the CSR sparse matrix format,
  • isodate or mtime to parse the ISO8601 date/time strings returned by YAC (e.g. YAC.start_datetime); mtime is the only package listed here that pip cannot install,
  • uxarray for yac.core.BasicGrid.from_uxgrid and the grid handling of several tools.

Installation

There are two ways to get the Python bindings:

Installation with pip

To install the Python bindings, use pip install with the path to the YAC source directory or the YAC GitLab repository (git@g.nosp@m.itla.nosp@m.b.dkr.nosp@m.z.de:YAC/YAC-dev.git). The following example shows how to set up a virtual environment and install YAC:

# setup venv
python -m venv <path_to_your_venv> # user-defined path where the venv will be created
source <path_to_your_venv>/bin/activate
# clone and install yac with all optional dependencies
git clone git@gitlab.dkrz.de:YAC/YAC-dev.git
pip install ./YAC-dev[full]

Or, without cloning the repository first:

pip install "yac[full] @ git+https://gitlab.dkrz.de/YAC/YAC-dev.git"

The [full] extra pulls in the optional packages. Drop it (i.e., only run pip install ./YAC-dev) if you only need the required ones.

By default, pip install will automatically detect available MPI compilers and build YAC with Python bindings. To customize the build configuration, set CMake options via the CMAKE_ARGS environment variable:

export CMAKE_ARGS="-DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DYAXT_ROOT=<yaxt_inst_dir>"
pip install ./YAC-dev

For finding yaxt users can also set the environment variable yaxt_ROOT. The automatic installation of dependencies (in particular yaxt) is currently in progress.

Installation from a YAC build directory

The Python bindings can also be built as part of a regular YAC build, i.e. alongside the C and Fortran libraries. With the autotools based build system, users can pass --enable-python-bindings together with --prefix to the configure to control where YAC and its Python module are installed:

./configure --enable-python-bindings --prefix=<install_dir>
make
make install

After installation the yac Python module will be placed under <install_dir>/lib/pythonX.Y/site-packages/.

With CMake (-DYAC_ENABLE_PYTHON=ON), the bindings are built but not installed by cmake --install, which only installs the YAC libraries. Instead, the complete package is assembled in the python subdirectory of the build directory together with a generated setup.py, which the users can employ to install the bindings into whichever Python environment they prefer:

python -m pip install <yac_build_dir>/python

Note that this only copies the prebuilt package, i.e. nothing is compiled or linked during the installation. Consequently, YAC must be built before this command is run.

Warning
The installed extension modules are not self-contained: when YAC is built with shared libraries (the default), they keep loading libyac_* from the build directory, which is baked into their runtime path. The build directory must, therefore, remain in place and must not be moved or deleted, or importing yac fails. Moreover, the installed package silently picks up whatever state the build directory is in, i.e. rebuilding YAC changes the behaviour of the installed bindings without reinstalling them, and the two can get out of sync (e.g. when the Python sources or the extension modules change but the package is not reinstalled). To avoid this, either build YAC with static libraries (-DBUILD_SHARED_LIBS=OFF), which links the YAC code into the extension modules and makes the installed package independent of the build directory, or use an editable installation (see below), which keeps the whole package in the build directory and thus consistent with the YAC libraries by construction.
This applies to the CMake build system only. The Autotools build system generates a different setup.py in its build directory, which builds the extension modules and is meant to be run by make install (it fails when called directly because it expects the linker flags in the environment). With Autotools, install the bindings with make install, as described above.
Note
The generated setup.py also supports an editable installation, which is convenient when developing YAC itself:
python -m pip install --editable <yac_build_dir>/python
The yac package then resolves to the build directory instead of being copied into site-packages, i.e. the bindings do not have to be reinstalled after every rebuild. Since nothing is rebuilt on import yac, YAC itself still has to be rebuilt manually (cmake --build), and files that are newly added to the package are only picked up after re-running the pip install --editable command.

Alternatively, with either build system, one can use the build directory directly by setting the PYTHONPATH environment variable:

export PYTHONPATH="<yac_build_dir>/python:$PYTHONPATH"

Note that PYTHONPATH must point to the python subdirectory of the build directory (which contains the assembled yac package), not to the python subdirectory of the source directory. See also Python bindings installation broken.

Note
If YAC is built with CMake and shared libraries (i.e. the default -DBUILD_SHARED_LIBS=ON), the extension modules link against libyac_*.so. Those libraries must remain findable at run time, either in the build directory or in the installation prefix that CMake was configured with.

Design of the bindings

The bindings are designed to build a thin layer between Python and YAC. I.e. methods are directly forwarded to the corresponding YAC functions with a few exceptions:

  • For all data structures for that YAC returns an id, Python creates an object that holds this id and provides member functions that use this id.
  • Whenever a c-pointer to memory is expected by the YAC function, the argument of the Python function is converted to a contiguous numpy array using numpy.ascontiguousarray (involving a copy if it is not already contiguous) and converted to a cython memory view to extract the size and pointer.
  • MPI communicators are converted into mpi4py.MPI.Comm and vice-versa.

Coroutines

The Python interface provides the coroutines yac::Field::get_coro and yac::Field::put_coro that can be used to write asynchronous code. This is in particular useful if multiple fields with different timesteps need to be handled. The following example shows how to use the coroutines assuming a source field field_source and target field field_target are defined.

import asyncio
async def recv_loop(field):
while True:
buf, info = await field.get_coro()
print(f"recv (field_id={field.field_id}):", buf, info)
if info == yac.Action.GET_FOR_RESTART:
break
async def send_loop(field):
while True:
info = await field.put_coro(np.random.rand(4))
print(f"send (field_id={field.field_id}):", info)
if info == yac.Action.PUT_FOR_RESTART:
break
# asyncio entry point
async def main():
# create task group
await asyncio.gather(
send_loop(field_source),
recv_loop(field_target),
)
# entry point
asyncio.run(main())

Note that the code does not need to know anything about the timestepping. Both fields are processed independently from each other. Effectively the event loop of asyncio spins between the two coroutines and proceeds where possible.

Note: The Python binding coroutines do not depend on asyncio and can also be used with other coroutine frameworks.

Logging

The Python logging package has been integrated into the Python bindings. Most log records are generated at the DEBUG level. Please note that a limitation in cython results in inaccurate stack information passed to the logger (specifically, incorrect filenames and module names).

To enable debug messages, use the following code:

import logging
logging.basicConfig(level=logging.WARNING, format=f'%(name)s rank={rank} - %(levelname)s - %(message)s')
logging.getLogger("yac").setLevel(logging.DEBUG)

This will print warnings from all packages and all logging messages from the yac module.

Using mpi4py with yac

If your Python code requires mpi4py (e.g., for internal communication), it's important to build mpi4py using the same MPI implementation as YAC. Within a virtual environment (venv), this can be accomplished by configuring the necessary compilers (e.g., loading a module or spack package) and then installing mpi4py from source with pip. For example, on Levante:

spack load openmpi/mnm
pip install --no-binary --no-cache-dir mpi4py

The --no-binary flag instructs pip to build the package from source.

See also the mpi4py documentation.

Using OpenMPI 5 with yac

OpenMPI version 5 requires that libmpi.so be loaded with the RTLD_GLOBAL flag when loaded dynamically using dlopen (as is the case when using Python). Currently, YAC does not handle this scenario, which can cause issues such as segmentation faults. The simplest solution is to use mpi4py (built against the compatible OpenMPI version) and import it before YAC. mpi4py automatically manages this requirement, ensuring that libmpi.so is properly loaded for YAC. We are actively working on a more robust solution for this situation.

See also OpenMPI documentation.

Python Examples

In the examples directory a framework of classes can be found that act as model component. To start a configuration with different components the driver.py can be used. It allows sequential coupling as well as parallel coupling.

The framework contains the following example components: