Interfaces

There are two interfaces to control the filter. One uses HDF5’s generic interface via an array of unsigned int cd_values as is used in H5Pset_filter(). The other uses HDF5 property lists added to the dataset creation property list used when the dataset to be compressed is being created. You can find examples of writing HDF5 data using both the generic and properties interfaces in test_write.c.

The filter itself supports either interface. The filter also supports all of the standard ZFP controls for affecting compression including rate, precision, accuracy, expert and reversible modes. For more information and details about these modes of controlling ZFP compression, please see the ZFP README.

Finally, you should not attempt to combine the ZFP filter with any other byte order altering filter such as, for example, HDF5’s shuffle filter. Space-performance will be ruined. This is in contrast to HDF5’s deflate filter which often performs better when used in conjunction with the shuffle filter. To understand why, see the description of endian issues.

Generic Interface

The generic interface is the only means of controlling the H5Z-ZFP filter when it is used as a dynamically loaded HDF5 plugin.

For the generic interface, the following CPP macros are defined in the H5Zzfp_plugin.h header file:

H5Pset_zfp_rate_cdata(double rate, size_t cd_nelmts, unsigned int *cd_vals);
H5Pset_zfp_precision_cdata(unsigned int prec, size_t cd_nelmts, unsigned int *cd_vals);
H5Pset_zfp_accuracy_cdata(double acc, size_t cd_nelmts, unsigned int *cd_vals);
H5Pset_zfp_expert_cdata(unsigned int minbits, unsigned int maxbits,
                        unsigned int maxprec, int minexp,
                        size_t cd_nelmts, unsigned int *cd_vals);
H5Pset_zfp_reversible_cdata(size_t cd_nelmts, unsigned int *cd_vals);

These macros utilize type punning to store the relevant ZFP parameters into a sufficiently large array (>=6) of unsigned int cd_values. It is up to the caller to then call H5Pset_filter() with the array of cd_values constructed by one of these macros.

Here is example code from test_write.c

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
    if (zfpmode == H5Z_ZFP_MODE_RATE)
        H5Pset_zfp_rate_cdata(rate, cd_nelmts, cd_values);
    else if (zfpmode == H5Z_ZFP_MODE_PRECISION)
        H5Pset_zfp_precision_cdata(prec, cd_nelmts, cd_values);
    else if (zfpmode == H5Z_ZFP_MODE_ACCURACY)
        H5Pset_zfp_accuracy_cdata(acc, cd_nelmts, cd_values);
    else if (zfpmode == H5Z_ZFP_MODE_EXPERT)
        H5Pset_zfp_expert_cdata(minbits, maxbits, maxprec, minexp, cd_nelmts, cd_values);
    else if (zfpmode == H5Z_ZFP_MODE_REVERSIBLE)
        H5Pset_zfp_reversible_cdata(cd_nelmts, cd_values);
    else
        cd_nelmts = 0; /* causes default behavior of ZFP library */

    /* print cd-values array used for filter */
    printf("\n%d cd_values=", (int) cd_nelmts);
    for (int i = 0; i < (int) cd_nelmts; i++)
        printf("%u,", cd_values[i]);
    printf("\n");

    /* Add filter to the pipeline via generic interface */
    if (0 > H5Pset_filter(cpid, H5Z_FILTER_ZFP, H5Z_FLAG_MANDATORY, cd_nelmts, cd_values)) SET_ERROR(H5Pset_filter);

However, these macros are only a convenience. You do not need the H5Zzfp_plugin.h header file if you want to avoid using it. But, you are then responsible for setting up the cd_values array correctly for the filter. For reference, the cd_values array for this ZFP filter is defined like so…

  cd_values index
ZFP mode 0 1 2 3 4 5
rate 1 unused rateA rateB unused unused
precision 2 unused prec unused unused unused
accuracy 3 unused accA accB unused unused
expert 4 unused minbits maxbits maxprec minexp
reversible 5 unused unused unused unused unused

A/B are high/low 32-bit words of a double.

Note that the cd_values used in the generic interface to H5Pset_filter() are not the same cd_values ultimately stored to the HDF5 dataset header for a compressed dataset. The values are transformed in the set_local method to use ZFP’s internal routines for ‘meta’ and ‘mode’ data. So, don’t make the mistake of examining the values you find in a file and think you can use those same values, for example, in an invocation of h5repack.

Properties Interface

For the properties interface, the following functions are defined in the H5Zzfp_props.h header file:

herr_t H5Pset_zfp_rate(hid_t dcpl_id, double rate);
herr_t H5Pset_zfp_precision(hid_t dcpl_id, unsigned int prec);
herr_t H5Pset_zfp_accuracy(hid_t dcpl_id, double acc);
herr_t H5Pset_zfp_expert(hid_t dcpl_id,
    unsigned int minbits, unsigned int maxbits,
    unsigned int maxprec, int minexp);
herr_t H5Pset_zfp_reversible(hid_t dcpl_id);

These functions take a dataset creation property list, hid_t dcp_lid and create temporary HDF5 property list entries to control the ZFP filter. Calling any of these functions removes the effects of any previous call to any one of these functions. In addition, calling any one of these functions also has the effect of adding the filter to the pipeline.

Here is example code from test_write.c

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
    H5Z_zfp_initialize();

    /* Setup the filter using properties interface. These calls also add
       the filter to the pipeline */
    if (zfpmode == H5Z_ZFP_MODE_RATE)
        H5Pset_zfp_rate(cpid, rate);
    else if (zfpmode == H5Z_ZFP_MODE_PRECISION)
        H5Pset_zfp_precision(cpid, prec);
    else if (zfpmode == H5Z_ZFP_MODE_ACCURACY)
        H5Pset_zfp_accuracy(cpid, acc);
    else if (zfpmode == H5Z_ZFP_MODE_EXPERT)
        H5Pset_zfp_expert(cpid, minbits, maxbits, maxprec, minexp);
    else if (zfpmode == H5Z_ZFP_MODE_REVERSIBLE)
        H5Pset_zfp_reversible(cpid);

The properties interface is more type-safe than the generic interface. However, there is no way for the implementation of the properties interface to reside within the filter plugin itself. The properties interface requires that the caller link with with the filter as a library, libh5zzfp.a. The generic interface does not require this.

Note that either interface can be used whether the filter is used as a plugin or as a library. The difference is whether the application calls H5Z_zfp_initialize() or not.

Fortran Interface

Fortran equivalents for both the properties and generic interfaces, described above, has been added by Scot Breitenfeld of the HDF5 group. The code that implements the Fortran interfaces is in the file H5Zzfp_props_f.F90. An example of its use is in test/test_rw_fortran.F90.

Plugin vs. Library Operation

The filter is designed to be compiled for use as both a standalone HDF5 dynamically loaded HDF5 plugin and as an explicitly linked library. When it is used as a plugin, it is a best practice to link the ZFP library into the plugin dynamic/shared object as a static library. Why? In so doing, we ensure that all ZFP public namespace symbols remain confined to the plugin so as not to interfere with any application that may be directly explicitly linking to the ZFP library for other reasons.

All HDF5 applications are required to find the plugin dynamic library (named lib*.{so,dylib}) in a directory specified by the environment variable, HDF5_PLUGIN_PATH. Currently, the HDF5 library offers no mechanism for applications themselves to have pre-programmed paths in which to search for a plugin. Applications are then always vulnerable to an incorrectly specified or unspecified HDF5_PLUGIN_PATH environment variable.

However, the plugin can also be used explicitly as a library. In this case, do not specify the HDF5_PLUGIN_PATH environment variable and instead have the application link to libH5Zzfp.a in the lib dir of the installation. Instead two initialization and finalization routines are defined:

int H5Z_zfp_initialize(void);
int H5Z_zfp_finalize(void);

These functions are defined in the H5Zzfp_lib.h header file. Any applications that wish to use the filter as a library are required to call the initialization routine, H5Z_zfp_initialize() before the filter can be referenced. In addition, to free up resources used by the filter, applications may call H5Z_zfp_finalize() when they are done using the filter.