Support HDF5 in tensorflow-io

[yongtang]

The follow is from tensorflow/tensorflow#27510

I am currently using HDF5 files (.h5 or .hdf5) to store my data, which is a data type frequently used in scientific research. (See #2089 for a similar but different request which makes the case for a HDF5 interface nicely). It is very convenient and widely used. MATLAB for example uses HDF5 files for large files. Indeed, it is much more convenient to use than Tensorflow's TFRecord format.

However, in Tensorflow, there is no native support for HDF5 files in the tf.data.Dataset API, which is supposed to be the new API for all data loading. Currently, I am using tf.py_funtion to load my data for the simple reason that tf Dataset is always in graph mode and hence cannot give out the values of the files that I want it to read.

Moreover, I have found that reading an HDF5 file in this way DRAMATICALLY slows down data I/O for unknown reasons. When I used the tf.keras.utils.Sequence API to read HDF5 files without the supposed optimizations that tensorflow is making, an operation that previously took hours now took just a few seconds. (However, I suspect that using tf.defun somehow got tangled up with this. I am not sure why but when I removed some lines, the code sped up, but was still much slower than even a single threaded for loop)

Therefore, I would like to propose creating a new API in tf Dataset for HDF5 files. It could be called HDF5Dataset, similar to TFRecordDataset or CSVDataset.

Moreover, this would allow Tensorflow to make I/O optimizations for reading using the C++ API for HDF5 instead of h5py, which has many limitations and factors that newbie users might not be familiar with.

For example, most builds of h5py cannot do multiprocessing. Also, most people do not know how to chunk their data slices, though this can make a 5-fold difference in read/write speed.

I believe that adding this API would make Tensorflow much friendlier to scientific calculation.

Will this change the current api? How?
This would add a new Dataset type in tf.data.Dataset, or a new method/function for making a dataset from an HDF5 file of an arbitrary format. This may require some low-level integration with the HDF5 format.

Who will benefit with this feature?
People in medical imaging, video datasets, astronomy, or any other type of very large dataset, which is often stored in HDF5. See their website for information on its utility. Also people who don't want to go through the difficult process of making TFRecord files.

Any Other info.
Perhaps integrating aspects of h5py will make the process easier.

[yongtang]

/cc @veritas9872

[captain-pool]

Hey @yongtang, @veritas9872 is this issue still open? Can I work on this?

[yongtang]

@captain-pool Definitely! Let me know if you need any help. 👍

[captain-pool]

@captain-pool Definitely! Let me know if you need any help. +1

Thanks @yongtang. Can you point me some resources to write bazel BUILD files? I don't have much idea about writing BUILD Files.

[yongtang]

@captain-pool Bazel has a pretty steep learning curve... The Dataset implementation pattern is also not exactly very straightforward. I have been trying to simplify the pattern for adding new ops for tensorflow-io. At the moment I think TextDataset is the easily pattern to follow. The BUILD file is also easier to understand.

In TextDataset, you can take a look at only one function ReadRecord:
https://github.com/tensorflow/io/blob/master/tensorflow_io/text/kernels/text_input.cc#L24

That is pretty much all you need to implement if you want to add a new Dataset op.

The BUILD file for TextDataset is also simple enough I think.

[captain-pool]

Thanks for your response @yongtang , I've been looking through this.
https://www.tensorflow.org/guide/extend/formats
However, I've been wondering, how to include the HDF5 Library in the BUILD File. For HDF5, we need to download the Library, how can I include this external library in Bazel Build?

I'm downloading the Source code of HDF5 and building it.
https://www.hdfgroup.org/downloads/hdf5/source-code/
And I'm using the examples as reference.

[captain-pool]

Hey @yongtang , I did some digging, and I found these bazel BUILD files for HDF5 are being used by,

https://github.com/tensorflow/models/blob/58deb0599f10dc5b33570103339fb7fa5bb876c3/research/vid2depth/third_party/hdf5.BUILD#L1

I think reusing codes from these BUILD Files and the WORKSPACE File, would be good enough for this job.

[yongtang]

@captain-pool Yes let's just reuse the one if already exists. I think cc_library(name = "hdf5", and its dependency might be all we need to link with hdf5.

[veritas9872]

@captain-pool @yongtang I found some documentation from PyTables, which also uses the HDF5 library and has its own set of optimizations.
I hope that this might help.

[yongtang]

@veritas9872 With PR #236 merged in, the HDF5 support is almost done (partially supported with a couple of data types). I will take a look and see if #217 or a new PR could get the HDF5 support done.

[veritas9872]

@yongtang Hello. I was curious to know whether the current implementation of HDF5 for TF is compatible with common features of HDF5, such as compression filters, checksums, and chunking.

I am also curious about whether multi-processing or multi-threading would be implemented.

For example, I have found that chunking data for each slice makes reading data x5 faster as unnecessary data is not being read in.

I am not familiar with how #236 or #217 has been implemented and I was wondering whether the implementation was ironing out the complexities and optimizing HDF5 for people unfamiliar with the file system.