Using H5Z-ZFP Plugin with H5Repack¶
A convenient way to use and play with the ZFP filter is as a plugin with the HDF5 h5repack utility using the -f
command-line argument to apply ZFP to existing data in a file.
Patching h5repack¶
Warning
Versions of HDF5’s h5repack
utility prior to 1.10.4 contain a bug that prevents proper parsing of the -f
argument’s option.
In order to use h5repack
with -f
argument as described here, you need to apply the patch from h5repack_parse.patch.
To do so, after you’ve downloaded and untar’d HDF5 but before you’ve built it, do something like the following using HDF5-1.8.14 as an example:
gunzip < hdf5-1.8.14.tar.gz | tar xvf -
cd hdf5-1.8.14
patch ./tools/h5repack/h5repack_parse.c <path-to-H5Z-ZFP-test-dir>/h5repack_parse.patch
Constructing an HDF5 cd_values array¶
HDF5’s h5repack
utility uses only the generic interface to HDF5 filters.
Another challenge in using h5repack
as described here is constructing the set unsigned int cd_values
as is used in H5Pset_filter() required by the generic HDF5 filter interface, especially because of the type-punning (doubles as unsigned int) which may be involved.
Note
Querying an existing dataset using h5dump
or h5ls
to obtain the cd_values
stored with a ZFP compressed dataset will not provide the correct cd_values
needed to invoke the filter.
This is because the cd_values
stored in the file are different from those needed in the generic interface to invoke the ZFP filter.
To facilitate constructing a valid -f
argument to h5repack
, there is a utility program, print_h5repack_farg
, which is presently in the test
directory and is built when tests are built.
This program was originally written simply to facilitate testing of H5Z-ZFP filter.
It should eventually be made a first class tool installed with H5Z-ZFP.
However, it is presently made only as part of testing H5Z-ZFP.
The print_h5repack_farg
utility can be used to read a command-line consisting of ZFP filter parameters you wish to use and output part of the command-line needed for the -f
argument to h5repack
.
Examples¶
Note
The examples below assume H5Z-ZFP has been configured to run tests by either a preceding make check
(if using a vanilla gmake
build) or make test
(if using a cmake
build).
In the examples below, we use h5repack
with the example data file, mesh.h5
in the tests directory.
To use ZFP filter in rate mode with a rate of 4.5
bits per value, first, use the print_h5repack_farg
:
% ./print_h5repack_farg zfpmode=1 rate=4.5
Print cdvals for set of ZFP compression parameters...
zfpmode=1 set zfp mode (1=rate,2=prec,3=acc,4=expert,5=rev)
rate=4.5 set rate for rate mode of filter
acc=0 set accuracy for accuracy mode of filter
prec=0 set precision for precision mode of zfp filter
minbits=0 set minbits for expert mode of zfp filter
maxbits=0 set maxbits for expert mode of zfp filter
maxprec=0 set maxprec for expert mode of zfp filter
minexp=0 set minexp for expert mode of zfp filter
help=0 this help message
h5repack -f argument...
-f UD=32013,0,4,1,0,0,1074921472
Next, cut-n-paste the -f UD=32013,0,4,1,0,0,1074921472
in a command to h5repack
like so:
env LD_LIBRARY_PATH=<path-to-dir-with-libhdf5.so>:$(LD_LIBRARY_PATH) \
HDF5_PLUGIN_PATH=<path-to-dir-with-libh5zzfp.so> \
$(HDF5_BIN)/h5repack -f UD=32013,0,4,1,0,0,1074921472 \
-l Pressure,Pressure2,Pressure3:CHUNK=10x20x5 \
-l Velocity,Velocity2,Velocity3,VelocityZ,VelocityZ2,VelocityZ3:CHUNK=11x21x1x1 \
-l VelocityX_2D:CHUNK=21x31 \
mesh.h5 mesh_repack.h5
where the -l
arguments indicate the dataset(s) to be re-packed as well as their (new) chunking.
To use ZFP filter in accuracy mode with an accuracy of 0.075
, first, use the print_h5repack_farg
:
% ./print_h5repack_farg zfpmode=3 acc=0.075
Print cdvals for set of ZFP compression parameters...
zfpmode=3 set zfp mode (1=rate,2=prec,3=acc,4=expert,5=rev)
rate=3.5 set rate for rate mode of filter
acc=0.075 set accuracy for accuracy mode of filter
prec=0 set precision for precision mode of zfp filter
minbits=0 set minbits for expert mode of zfp filter
maxbits=0 set maxbits for expert mode of zfp filter
maxprec=0 set maxprec for expert mode of zfp filter
minexp=0 set minexp for expert mode of zfp filter
help=0 this help message
h5repack -f argument...
-f UD=32013,0,4,3,0,858993459,1068708659
Next, cut-n-paste the -f UD=32013,0,4,3,0,858993459,1068708659
in a command to h5repack
like so:
env LD_LIBRARY_PATH=<path-to-dir-with-libhdf5.so>:$(LD_LIBRARY_PATH) \
HDF5_PLUGIN_PATH=<path-to-dir-with-libh5zzfp.so> \
$(HDF5_BIN)/h5repack -f UD=32013,0,4,3,0,858993459,1068708659 \
-l Pressure,Pressure2,Pressure3:CHUNK=10x20x5 \
-l Velocity,Velocity2,Velocity3,VelocityZ,VelocityZ2,VelocityZ3:CHUNK=11x21x1x1 \
-l VelocityX_2D:CHUNK=21x31 \
mesh.h5 mesh_repack.h5