YAC 3.13.0
Yet Another Coupler
Loading...
Searching...
No Matches
Raw data exchange

Introduction

YAC is currently most commonly used in cases where a source component provides a field, which is redistributed and interpolated by YAC to be received by a target component.

Some applications may require only the redistribution of the source field to the processes of the target component without application of the interpolation. These applications can access the interpolation information and apply it themselves.

Description

If a couple is configured to use raw data exchange, target processes can query YAC for information on how to interpolate their local points of the respective target field after the End of Definition Phase.

During the Data Exchange nothing changes for source processes. However, target processes will not receive an interpolated target field, but instead a set of source points (and associated fractional mask values, in case the source field is configured for Dynamic fractional masking). This set of source points is configured such that each target process can compute their local target points based on the interpolation information mentioned above without requiring additonal communicaiton.

Flux coupling

When coupling fluxes, usage of an exchange grid (Balaji 2006) can improve results compared to applying a conservative interpolation to the source flux in order to get the target flux. This approach was successfully used among others by Bauer 2021 and Karsten 2024.

YAC is not designed to be able to provide an exchange grid to the user. However with the introduction of raw data exchange, similar results can be achieved. If raw data exchange is used for a couple together with an Interpolation stack that consists of 1st order conservative (partial_converage: false; normalisation: destarea) followed by Fixed value interpolation, then the weights generated by this interpolation stack represent the overlap between the source and target cells. The sum of the weights for a target cell, which is completely covered by valid source cells is 1.0 in this configuration. The processes of the target component can then compute the flux of the target field based on the fractional contribution of the received source flux. In addition, all target points that are supposed to receive a fixed value, based on the interpolation information provided by YAC, do not overlap with any unmasked source cell and can be handled accordingly.

Stochastic coupling

Rackow 2020 proposes to use a stochastic-based coupling approach, when coupling atmospheric and oceanic surface fields. This approach can produce improved coupling results in case grid resolutions of both components differ significantly.

Stochastic coupling is not supported by YAC. However, a target component should easily be able to implement it, if raw data exchange is used.

Usage in YAC

Enabling raw data exchange

The raw exchange is enabled by setting the appropriate parameter during the definition of the couple.

If the couple is defined via a configuration file, use the use_raw_exchange parameter.

definitions:
- &atm
component: dummy_atmosphere
grid: dummy_atmosphere_grid
- &ocn
component: dummy_ocean
grid: dummy_ocean_grid
- &time_config
src_lag: 1
tgt_lag: 1
coupling_period: 3600
- &config_model
<<: *time_config
time_reduction: none
interpolation:
- conservative:
order: 1
partial_coverage: true
normalisation: destarea
- fixed:
user_value: 0.0
coupling:
- source: *atm
target: *ocn
<<: *config_model
field: [surface_fresh_water_flux,
total_heat_flux]
use_raw_exchange: true # enables raw data exchange

In the case the respective interface routines are used see the following example:

Query interpolation information

After the end of the definitions, the target processes can query the interpolation information for their local target points. Per target field this routine can only be called once.

  • C
    double frac_mask_fallback_value;
    double scaling_factor;
    double scaling_summand;
    size_t num_fixed_values;
    double * fixed_values;
    size_t * num_tgt_per_fixed_value;
    size_t * tgt_idx_fixed;
    size_t num_wgt_tgt;
    size_t * wgt_tgt_idx;
    size_t * num_src_per_tgt;
    double * weights;
    size_t * src_field_idx;
    size_t * src_idx;
    size_t num_src_fields;
    size_t * src_field_buffer_sizes;
    tgt_field_id,
    &frac_mask_fallback_value, &scaling_factor, &scaling_summand,
    &num_fixed_values, &fixed_values,
    &num_tgt_per_fixed_value, &tgt_idx_fixed,
    &num_wgt_tgt, &wgt_tgt_idx, &num_src_per_tgt,
    &weights, &src_field_idx, &src_idx,
    &num_src_fields, &src_field_buffer_sizes);
    void yac_cget_raw_interp_weights_data(int const field_id, double *frac_mask_fallback_value, double *scaling_factor, double *scaling_summand, size_t *num_fixed_values, double **fixed_values, size_t **num_tgt_per_fixed_value, size_t **tgt_idx_fixed, size_t *num_wgt_tgt, size_t **wgt_tgt_idx, size_t **num_src_per_tgt, double **weights, size_t **src_field_idx, size_t **src_idx, size_t *num_src_fields, size_t **src_field_buffer_size)
    Definition yac.c:2193
  • Fortran
    DOUBLE PRECISION :: frac_mask_fallback_value
    DOUBLE PRECISION :: scaling_factor
    DOUBLE PRECISION :: scaling_summand
    INTEGER :: num_fixed_values
    DOUBLE PRECISION, ALLOCATABLE :: fixed_values(:)
    INTEGER, ALLOCATABLE :: num_tgt_per_fixed_value(:)
    INTEGER, ALLOCATABLE :: tgt_idx_fixed(:)
    INTEGER :: num_wgt_tgt
    INTEGER, ALLOCATABLE :: wgt_tgt_idx(:)
    INTEGER, ALLOCATABLE :: num_src_per_tgt(:)
    DOUBLE PRECISION, ALLOCATABLE :: weights(:)
    INTEGER, ALLOCATABLE :: src_field_idx(:)
    INTEGER, ALLOCATABLE :: src_idx(:)
    INTEGER :: num_src_fields
    INTEGER, ALLOCATABLE :: src_field_buffer_sizes(:)
    tgt_field_id,
    frac_mask_fallback_value, scaling_factor, scaling_summand, &
    num_fixed_values, fixed_values, &
    num_tgt_per_fixed_value, tgt_idx_fixed, &
    num_wgt_tgt, wgt_tgt_idx, num_src_per_tgt, &
    weights, src_field_idx, src_idx,
    num_src_fields, src_field_buffer_sizes)
    subroutine yac_fget_raw_interp_weights_data(field_id, frac_mask_fallback_value, scaling_factor, scaling_summand, num_fixed_values, fixed_values, num_tgt_per_fixed_value, tgt_idx_fixed, num_wgt_tgt, wgt_tgt_idx, num_src_per_tgt, weights, src_field_idx, src_idx, num_src_fields, src_field_buffer_size)
  • Python
    raw_data = tgt_field.get_raw_interp_weights_data()
    raw_data is a data class object, see RawInterpWeights

Alternatively, the exchange data can also be obtained in a format that is compatible with the CSR sparse matrix format. See for example yac_cget_raw_interp_weights_data_csr.

Allocate receive buffers

The interpolation information contains the size of the buffers required to receive the source points. These buffers have to be allocated by the target processes.

  • C
    double *** src_field_buffer =
    malloc(collection_size * sizeof(*src_field_buffer));
    for (size_t i = 0; i < collection_size; ++i) {
    src_field_buffer[i] =
    malloc(num_src_fields * sizeof(**src_field_buffer));
    for (size_t j = 0; j < num_src_fields; ++j) {
    src_field_buffer[i][j] =
    malloc(src_field_buffer_sizes[j] * sizeof(***src_field_buffer));
    }
    }
    int collection_size
  • Fortran

    REAL(KIND=wp), ALLOCATABLE, TARGET :: src_field_buffer(:,:)
    INTEGER :: src_field_buffer_size
    src_field_buffer_size = sum(src_field_buffer_sizes(:))
    ALLOCATE(src_field_buffer(src_field_buffer_size,collection_size))

    Alternatively:

    TYPE(yac_dble_ptr), ALLOCATABLE :: src_field_buffer_ptr(:,:)
    ALLOCATE(src_field_buffer_ptr(num_src_fields,collection_size)
    DO i = 1, num_src_fields
    DO j = 1, collection_size
    ALLOCATE(src_field_buffer_ptr(i,j)%p(src_field_buffer_sizes(i)))
    END DO
    END DO
  • Python
    src_field_buffer = np.empty(shape=(collection_size, np.sum(raw_data.src_field_buffer_sizes)))
    Note: In case of multiple source fields the src_field_buffers are concatenated.

In case the source processes provides a fractional mask along with the source field data, the target processes will have to allocate a buffer for it in the same way as for the source field buffer.

Receive source field data

The put-operation is not effected by the use of raw data exchange. However, in the get-operation the target processes have to provide the previously allocated source field buffers and fractional mask buffers (if required).

  • C
    yac_cget_raw(tgt_field_id, collection_size, src_field_buffer, &info, &ierror);
    int ierror
    void yac_cget_raw(int const field_id, int collection_size, double ***src_field_buffer, int *info, int *ierr)
    Definition yac.c:2963
  • Fortran

    CALL yac_fget_raw( &
    tgt_field_id, src_field_buffer_size, collection_size, src_field_buffer, info, ierror)

    Alternatively:

    CALL yac_fget_raw( &
    tgt_field_id, num_src_fields, collection_size, src_field_buffer_ptr, info, ierror)
  • Python
    src_field_buffer, frac_mask_buffer, info = tgt_field.get_raw(src_field_buffer)

With Dynamic fractional masking :

  • C
    tgt_field_id, collection_size, src_field_frac_buffer, src_frac_mask_buffer, &info, &ierror);
    void yac_cget_raw_frac(int const field_id, int collection_size, double ***src_field_buffer, double ***src_frac_mask_buffer, int *info, int *ierr)
    Definition yac.c:2990
  • Fortran

    CALL yac_fget_raw( &
    tgt_field_id, src_field_buffer_size, collection_size, &
    src_field_buffer, src_frac_mask_buffer, tgt_info, ierror)

    Alternatively:

    CALL yac_fget_raw( &
    tgt_field_id, num_src_fields, collection_size, &
    src_field_buffer_ptr, src_frac_mask_buffer_ptr, tgt_info, ierror)
  • Python
    src_field_buffer, frac_mask_buffer, info = tgt_field.get_raw(src_field_buffer, src_frac_mask_buffer )

Apply interpolation information

The following routines demonstrate how the interpolation information could be applied to the source field buffer in order to compute the target field.

  • C
    static void compute_tgt_field(
    double *** src_field_buffer, double *** src_frac_mask_buffer,
    double ** tgt_field, size_t collection_size,
    double frac_mask_fallback_value, double scaling_factor, double scaling_summand,
    size_t num_fixed_values, double * fixed_values, size_t * num_tgt_per_fixed_value,
    size_t * tgt_idx_fixed, size_t num_wgt_tgt, size_t * wgt_tgt_idx,
    size_t * num_src_per_tgt, double * weights, size_t * src_field_idx, size_t * src_idx) {
    // we have to use memcmp to compare against YAC_FRAC_MASK_NO_VALUE,
    // because nan is a valid value for frac_mask_fallback_value and depending
    // on compiler optimisation a comparision agains nan can produce varying
    // results
    int with_frac_mask =
    memcmp(
    &frac_mask_fallback_value, &YAC_FRAC_MASK_NO_VALUE, sizeof(YAC_FRAC_MASK_NO_VALUE));
    for (size_t collection_idx = 0; collection_idx < collection_size; ++collection_idx) {
    // set fixed targets
    for (size_t i = 0, k = 0; i < num_fixed_values; ++i) {
    double fixed_value = fixed_values[i];
    size_t num_fixed_tgt = num_tgt_per_fixed_value[i];
    for (size_t j = 0; j < num_fixed_tgt; ++j, ++k)
    tgt_field[collection_idx][tgt_idx_fixed[k]] = fixed_value;
    }
    if (with_frac_mask) {
    // set weighted targets
    for (size_t i = 0, k = 0; i < num_wgt_tgt; ++i) {
    double tgt_value = 0.0;
    double frac_weight_sum = 0.0;
    double weight_sum = 0.0;
    size_t num_src = num_src_per_tgt[i];
    for (size_t j = 0; j < num_src; ++j, ++k) {
    tgt_value +=
    weights[k] * src_field_buffer[collection_idx][src_field_idx[k]][src_idx[k]];
    frac_weight_sum +=
    weights[k] * src_frac_mask_buffer[collection_idx][src_field_idx[k]][src_idx[k]];
    weight_sum += weights[k];
    }
    tgt_field[collection_idx][wgt_tgt_idx[i]] =
    (fabs(frac_weight_sum) > FRAC_MASK_TOL)?
    (scaling_factor * (tgt_value / frac_weight_sum) * weight_sum + scaling_summand):
    frac_mask_fallback_value;
    }
    } else {
    // set weighted targets
    for (size_t i = 0, k = 0; i < num_wgt_tgt; ++i) {
    double tgt_value = 0.0;
    size_t num_src = num_src_per_tgt[i];
    for (size_t j = 0; j < num_src; ++j, ++k)
    tgt_value +=
    weights[k] * src_field_buffer[collection_idx][src_field_idx[k]][src_idx[k]];
    tgt_field[collection_idx][wgt_tgt_idx[i]] = scaling_factor * tgt_value + scaling_summand;
    }
    }
    }
    }
    double const YAC_FRAC_MASK_NO_VALUE
    static void compute_tgt_field(double const *restrict **src_fields, double const *restrict **src_frac_masks, double *restrict *tgt_field, size_t *restrict tgt_buffer_sizes, size_t num_src_fields, size_t collection_size, double frac_mask_fallback_value, double scale_factor, double scale_summand)
    #define FRAC_MASK_TOL
  • Fortran

    SUBROUTINE compute_tgt_field( &
    tgt_field, frac_mask_fallback_value, scaling_factor, scaling_summand, num_fixed_values, &
    fixed_values, num_tgt_per_fixed_value, tgt_idx_fixed, num_wgt_tgt, wgt_tgt_idx, num_src_per_tgt, &
    weights, src_field_idx, src_idx, num_src_fields, src_field_buffer_sizes, collection_size, &
    src_field_buffer, src_frac_mask_buffer)
    REAL(KIND=wp), INTENT(OUT) :: tgt_field(:,:)
    DOUBLE PRECISION, INTENT(IN) :: frac_mask_fallback_value
    DOUBLE PRECISION, INTENT(IN) :: scaling_factor
    DOUBLE PRECISION, INTENT(IN) :: scaling_summand
    INTEGER, INTENT(IN) :: num_fixed_values
    DOUBLE PRECISION, INTENT(IN) :: fixed_values(:)
    INTEGER, INTENT(IN) :: num_tgt_per_fixed_value(:)
    INTEGER, INTENT(IN) :: tgt_idx_fixed(:)
    INTEGER, INTENT(IN) :: num_wgt_tgt
    INTEGER, INTENT(IN) :: wgt_tgt_idx(:)
    INTEGER, INTENT(IN) :: num_src_per_tgt(:)
    DOUBLE PRECISION, INTENT(IN) :: weights(:)
    INTEGER, INTENT(IN) :: src_field_idx(:)
    INTEGER, INTENT(IN) :: src_idx(:)
    INTEGER, INTENT(IN) :: num_src_fields
    INTEGER, INTENT(IN) :: src_field_buffer_sizes(:)
    INTEGER, INTENT(IN) :: collection_size
    REAL(KIND=wp), INTENT(IN) :: src_field_buffer(:,:)
    REAL(KIND=wp), OPTIONAL, INTENT(IN) :: src_frac_mask_buffer(:,:)
    INTEGER :: collection_idx, i, j, k, accu
    DOUBLE PRECISION :: fixed_value, tgt_value, frac_weight_sum, weight_sum
    INTEGER :: src_field_buffer_offsets(num_src_fields)
    accu = 0
    DO i = 1, num_src_fields
    src_field_buffer_offsets(i) = accu
    accu = accu + src_field_buffer_sizes(i)
    END DO
    DO collection_idx = 1, collection_size
    ! set fixed targets
    k = 1
    DO i = 1, num_fixed_values
    fixed_value = fixed_values(i)
    DO j = 1, num_tgt_per_fixed_value(i)
    tgt_field(tgt_idx_fixed(k), collection_idx) = real(fixed_value, wp)
    k = k + 1
    END DO
    END DO
    ! set weighted targets
    IF (PRESENT(src_frac_mask_buffer)) THEN
    k = 1
    DO i = 1, num_wgt_tgt
    tgt_value = 0.0_wp
    frac_weight_sum = 0.0_wp
    weight_sum = 0.0_wp
    DO j = 1, num_src_per_tgt(i)
    tgt_value = tgt_value + weights(k) * &
    src_field_buffer( &
    src_idx(k) + src_field_buffer_offsets(src_field_idx(k)), collection_idx)
    frac_weight_sum = frac_weight_sum + weights(k) * &
    src_frac_mask_buffer( &
    src_idx(k) + src_field_buffer_offsets(src_field_idx(k)), collection_idx)
    weight_sum = weight_sum + weights(k)
    k = k + 1
    END DO
    IF (abs(frac_weight_sum) > frac_mask_tol) THEN
    tgt_field(wgt_tgt_idx(i),collection_idx) = &
    REAL(scaling_factor * (tgt_value / frac_weight_sum) * weight_sum + scaling_summand, wp)
    ELSE
    tgt_field(wgt_tgt_idx(i),collection_idx) = real(frac_mask_fallback_value, wp)
    END IF
    END DO
    ELSE
    k = 1
    DO i = 1, num_wgt_tgt
    tgt_value = 0.0_wp
    DO j = 1, num_src_per_tgt(i)
    tgt_value = tgt_value + weights(k) * &
    src_field_buffer(src_idx(k) + src_field_buffer_offsets(src_field_idx(k)), collection_idx)
    k = k + 1
    END DO
    tgt_field(wgt_tgt_idx(i),collection_idx) = &
    REAL(scaling_factor * tgt_value + scaling_summand, wp)
    END DO
    END IF
    END DO
    END SUBROUTINE compute_tgt_field

    Alternatively:

    SUBROUTINE compute_tgt_field( &
    tgt_field, frac_mask_fallback_value, scaling_factor, scaling_summand, num_fixed_values, &
    fixed_values, num_tgt_per_fixed_value, tgt_idx_fixed, num_wgt_tgt, wgt_tgt_idx, num_src_per_tgt, &
    weights, src_field_idx, src_idx, collection_size, src_field_buffer_ptr, src_frac_mask_buffer_ptr)
    REAL(KIND=wp), INTENT(OUT) :: tgt_field(:,:)
    DOUBLE PRECISION, INTENT(IN) :: frac_mask_fallback_value
    DOUBLE PRECISION, INTENT(IN) :: scaling_factor
    DOUBLE PRECISION, INTENT(IN) :: scaling_summand
    INTEGER, INTENT(IN) :: num_fixed_values
    DOUBLE PRECISION, INTENT(IN) :: fixed_values(:)
    INTEGER, INTENT(IN) :: num_tgt_per_fixed_value(:)
    INTEGER, INTENT(IN) :: tgt_idx_fixed(:)
    INTEGER, INTENT(IN) :: num_wgt_tgt
    INTEGER, INTENT(IN) :: wgt_tgt_idx(:)
    INTEGER, INTENT(IN) :: num_src_per_tgt(:)
    DOUBLE PRECISION, INTENT(IN) :: weights(:)
    INTEGER, INTENT(IN) :: src_field_idx(:)
    INTEGER, INTENT(IN) :: src_idx(:)
    INTEGER, INTENT(IN) :: collection_size
    TYPE(yac_dble_ptr), INTENT(IN) :: src_field_buffer_ptr(:,:)
    TYPE(yac_dble_ptr), OPTIONAL, INTENT(IN) :: src_frac_mask_buffer_ptr(:,:)
    INTEGER :: collection_idx, i, j, k
    DOUBLE PRECISION :: fixed_value, tgt_value, frac_weight_sum, weight_sum
    DO collection_idx = 1, collection_size
    ! set fixed targets
    k = 1
    DO i = 1, num_fixed_values
    fixed_value = fixed_values(i)
    DO j = 1, num_tgt_per_fixed_value(i)
    tgt_field(tgt_idx_fixed(k), collection_idx) = real(fixed_value, wp)
    k = k + 1
    END DO
    END DO
    ! set weighted targets
    IF (PRESENT(src_frac_mask_buffer_ptr)) THEN
    k = 1
    DO i = 1, num_wgt_tgt
    tgt_value = 0.0_wp
    frac_weight_sum = 0.0_wp
    weight_sum = 0.0_wp
    DO j = 1, num_src_per_tgt(i)
    tgt_value = tgt_value + &
    weights(k) * src_field_buffer_ptr(src_field_idx(k), collection_idx)%p(src_idx(k))
    frac_weight_sum = frac_weight_sum + &
    weights(k) * src_frac_mask_buffer_ptr(src_field_idx(k), collection_idx)%p(src_idx(k))
    weight_sum = weight_sum + weights(k)
    k = k + 1
    END DO
    IF (abs(frac_weight_sum) > frac_mask_tol) THEN
    tgt_field(wgt_tgt_idx(i),collection_idx) = &
    REAL(scaling_factor * (tgt_value / frac_weight_sum) * weight_sum + scaling_summand, wp)
    ELSE
    tgt_field(wgt_tgt_idx(i),collection_idx) = real(frac_mask_fallback_value, wp)
    END IF
    END DO
    ELSE
    k = 1
    DO i = 1, num_wgt_tgt
    tgt_value = 0.0_wp
    DO j = 1, num_src_per_tgt(i)
    tgt_value = tgt_value + &
    weights(k) * src_field_buffer_ptr(src_field_idx(k), collection_idx)%p(src_idx(k))
    k = k + 1
    END DO
    tgt_field(wgt_tgt_idx(i),collection_idx) = &
    REAL(scaling_factor * tgt_value + scaling_summand, wp)
    END DO
    END IF
    END DO
    END SUBROUTINE compute_tgt_field
  • Python

    tgt_buffer = np.empty(shape=(collection_size, tgt_field.size))
    # assuming only one fixed value
    tgt_buffer[:, raw_data.tgt_idx_fixed] = raw_data.fixed_values[0]
    indptr = np.insert(np.cumsum(raw_data.num_src_per_tgt), 0, 0)
    for c in range(collection_size):
    for tgt_idx, start_idx, end_idx in zip(raw_data.wgt_tgt_idx, indptr[:-1], indptr[1:]):
    tgt_buffer[c, tgt_idx] = raw_data.weights[start_idx:end_idx]@buf[c, raw_data.src_idx[start_idx: end_idx]]

    In the case you have the raw_exchange data in CSR format (see yac::Field::get_raw_interp_weights_data_csr), you can pass the data directly to scipy's csr_matrix.

    raw_data_csr = tgt_field.get_raw_interp_weights_data_csr()
    W = csr_matrix((raw_data_csr.weights, raw_data_csr.src_idx, raw_data_csr.src_indptr))
    tgt_field.get_raw(buf)
    for c in range(tgt_field.collection_size):
    tgt_buffer[c, :] = W@buf[c, :]
    tgt_buffer[c, raw_data_csr.tgt_idx_fixed] = raw_data_csr.fixed_values[0]

Freeing of interpolation information

The user is responsible for freeing the memory associated with the interpolation information.

  • C
    free(fixed_values);
    free(num_tgt_per_fixed_value);
    free(tgt_idx_fixed);
    free(wgt_tgt_idx);
    free(num_src_per_tgt);
    free(weights);
    free(src_field_idx);
    free(src_idx);
    free(src_field_buffer_sizes);
  • Fortran
    DEALLOCATE( &
    fixed_values, num_tgt_per_fixed_value, tgt_idx_fixed, &
    wgt_tgt_idx, num_src_per_tgt, weights, src_field_idx, src_idx, &
    interp_weights_data%src_field_buffer_sizes)
  • Python
    del raw_data

References

Balaji, V., Anderson, J., Held, I., Winton, M., Durachta, J., Malyshev, S., and Stouffer, R. J.: The Exchange Grid: a mechanism for data exchange between Earth System components on independent grids, in: Parallel computational fluid dynamics 2005, 179–186, Elsevier, https://doi.org/10.1016/B978-044452206-1/50021-5, 2006.

Bauer, T. P., Holtermann, P., Heinold, B., Radtke, H., Knoth, O., and Klingbeil, K.: ICONGETM v1.0 – flexible NUOPC-driven two-way coupling via ESMF exchange grids between the unstructured-grid atmosphere model ICON and the structured-grid coastal ocean model GETM, Geosci. Model Dev., 14, 4843–4863, https://doi.org/10.5194/gmd-14-4843-2021, 2021.

Karsten, S., Radtke, H., Gröger, M., Ho-Hagemann, H. T. M., Mashayekh, H., Neumann, T., and Meier, H. E. M.: Flux coupling approach on an exchange grid for the IOW Earth System Model (version 1.04.00) of the Baltic Sea region, Geosci. Model Dev., 17, 1689–1708, https://doi.org/10.5194/gmd-17-1689-2024, 2024.

Rackow T, Juricke S. Flow-dependent stochastic coupling for climate models with high ocean-to-atmosphere resolution ratio. Q J R Meteorol Soc. 2020; 146: 284–300. https://doi.org/10.1002/qj.3674