Towards information profiling: data lake content metadata management

A Alserafi, A Abelló, O Romero… - 2016 IEEE 16th …, 2016 - ieeexplore.ieee.org
2016 IEEE 16th International Conference on Data Mining Workshops …, 2016ieeexplore.ieee.org
There is currently a burst of Big Data (BD) processed and stored in huge raw data
repositories, commonly called Data Lakes (DL). These BD require new techniques of data
integration and schema alignment in order to make the data usable by its consumers and to
discover the relationships linking their content. This can be provided by metadata services
which discover and describe their content. However, there is currently a lack of a systematic
approach for such kind of metadata discovery and management. Thus, we propose a …
There is currently a burst of Big Data (BD) processed and stored in huge raw data repositories, commonly called Data Lakes (DL). These BD require new techniques of data integration and schema alignment in order to make the data usable by its consumers and to discover the relationships linking their content. This can be provided by metadata services which discover and describe their content. However, there is currently a lack of a systematic approach for such kind of metadata discovery and management. Thus, we propose a framework for the profiling of informational content stored in the DL, which we call information profiling. The profiles are stored as metadata to support data analysis. We formally define a metadata management process which identifies the key activities required to effectively handle this. We demonstrate the alternative techniques and performance of our process using a prototype implementation handling a real-life case-study from the OpenML DL, which showcases the value and feasibility of our approach.
ieeexplore.ieee.org
Showing the best result for this search. See all results