Using H5Z-ZFP Plugin with H5Repack

A convenient way to use and play with the ZFP filter is as a plugin with the HDF5 h5repack utility using the -f command-line argument to apply ZFP to existing data in a file.

Patching h5repack

Warning

Versions of HDF5’s h5repack utility prior to 1.10.4 contain a bug that prevents proper parsing of the -f argument’s option. In order to use h5repack with -f argument as described here, you need to apply the patch from h5repack_parse.patch. To do so, after you’ve downloaded and untar’d HDF5 but before you’ve built it, do something like the following using HDF5-1.8.14 as an example:

gunzip < hdf5-1.8.14.tar.gz | tar xvf -
cd hdf5-1.8.14
patch ./tools/h5repack/h5repack_parse.c <path-to-H5Z-ZFP-test-dir>/h5repack_parse.patch

Constructing an HDF5 cd_values array

HDF5’s h5repack utility uses only the generic interface to HDF5 filters. Another challenge in using h5repack as described here is constructing the set unsigned int cd_values as is used in H5Pset_filter() required by the generic HDF5 filter interface, especially because of the type-punning (doubles as unsigned int) which may be involved.

Note

Querying an existing dataset using h5dump or h5ls to obtain the cd_values stored with a ZFP compressed dataset will not provide the correct cd_values needed to invoke the filter. This is because the cd_values stored in the file are different from those needed in the generic interface to invoke the ZFP filter.

To facilitate constructing a valid -f argument to h5repack, there is a utility program, print_h5repack_farg, which is presently in the test directory and is built when tests are built. This program was originally written simply to facilitate testing of H5Z-ZFP filter. It should eventually be made a first class tool installed with H5Z-ZFP. However, it is presently made only as part of testing H5Z-ZFP.

The print_h5repack_farg utility can be used to read a command-line consisting of ZFP filter parameters you wish to use and output part of the command-line needed for the -f argument to h5repack.

Examples

Note

The examples below assume H5Z-ZFP has been configured and built with -DBUILD_TESTING=ON and tests have been run via ctest.

In the examples below, we use h5repack with the example data file, mesh.h5 in the tests directory.

To use ZFP filter in rate mode with a rate of 4.5 bits per value, first, use the print_h5repack_farg:

% ./print_h5repack_farg zfpmode=1 rate=4.5

Print cdvals for set of ZFP compression parameters...
    zfpmode=1  set zfp mode (1=rate,2=prec,3=acc,4=expert,5=rev)
    rate=4.5                    set rate for rate mode of filter
    acc=0               set accuracy for accuracy mode of filter
    prec=0        set precision for precision mode of zfp filter
    minbits=0          set minbits for expert mode of zfp filter
    maxbits=0          set maxbits for expert mode of zfp filter
    maxprec=0          set maxprec for expert mode of zfp filter
    minexp=0            set minexp for expert mode of zfp filter
    help=0                                     this help message

h5repack -f argument...
    -f UD=32013,0,4,1,0,0,1074921472

Next, cut-n-paste the -f UD=32013,0,4,1,0,0,1074921472 in a command to h5repack like so:

env LD_LIBRARY_PATH=<path-to-dir-with-libhdf5.so>:$(LD_LIBRARY_PATH) \
    HDF5_PLUGIN_PATH=<path-to-dir-with-libh5zzfp.so> \
    $(HDF5_BIN)/h5repack -f UD=32013,0,4,1,0,0,1074921472 \
         -l Pressure,Pressure2,Pressure3:CHUNK=10x20x5 \
         -l Velocity,Velocity2,Velocity3,VelocityZ,VelocityZ2,VelocityZ3:CHUNK=11x21x1x1 \
         -l VelocityX_2D:CHUNK=21x31 \
         mesh.h5 mesh_repack.h5

where the -l arguments indicate the dataset(s) to be re-packed as well as their (new) chunking.

To use ZFP filter in accuracy mode with an accuracy of 0.075, first, use the print_h5repack_farg:

% ./print_h5repack_farg zfpmode=3 acc=0.075

Print cdvals for set of ZFP compression parameters...
    zfpmode=3  set zfp mode (1=rate,2=prec,3=acc,4=expert,5=rev)
    rate=3.5                    set rate for rate mode of filter
    acc=0.075           set accuracy for accuracy mode of filter
    prec=0        set precision for precision mode of zfp filter
    minbits=0          set minbits for expert mode of zfp filter
    maxbits=0          set maxbits for expert mode of zfp filter
    maxprec=0          set maxprec for expert mode of zfp filter
    minexp=0            set minexp for expert mode of zfp filter
    help=0                                     this help message

h5repack -f argument...
    -f UD=32013,0,4,3,0,858993459,1068708659

Next, cut-n-paste the -f UD=32013,0,4,3,0,858993459,1068708659 in a command to h5repack like so:

env LD_LIBRARY_PATH=<path-to-dir-with-libhdf5.so>:$(LD_LIBRARY_PATH) \
    HDF5_PLUGIN_PATH=<path-to-dir-with-libh5zzfp.so> \
    $(HDF5_BIN)/h5repack -f UD=32013,0,4,3,0,858993459,1068708659 \
         -l Pressure,Pressure2,Pressure3:CHUNK=10x20x5 \
         -l Velocity,Velocity2,Velocity3,VelocityZ,VelocityZ2,VelocityZ3:CHUNK=11x21x1x1 \
         -l VelocityX_2D:CHUNK=21x31 \
         mesh.h5 mesh_repack.h5