Towards a Hybrid Imputation Approach Using Web Tables
Research output: Contribution to book/Conference proceedings/Anthology/Report › Conference contribution › Contributed › peer-review
Contributors
Abstract
Data completeness is one of the most important data quality dimensions and an essential premise in data analytics. With new emerging Big Data trends such as the data lake concept, which provides a low cost data preparation repository instead of moving curated data into a data warehouse, the problem of data completeness is additionally reinforced. While traditionally the process of filling in missing values is addressed by the data imputation community using statistical techniques, we complement these approaches by using external data sources from the data lake or even the Web to lookup missing values. In this paper we propose a novel hybrid data imputation strategy that, takes into account the characteristics of an incomplete dataset and based on that chooses the best imputation approach, i.e. either a statistical approach such as regression analysis or a Web-based lookup or a combination of both. We formalize and implement both imputation approaches, including a Web table retrieval and matching system and evaluate them extensively using a corpus with 125M Web tables. We show that applying statistical techniques in conjunction with external data sources will lead to a imputation system which is robust, accurate, and has high coverage at the same time.
Details
| Original language | English |
|---|---|
| Title of host publication | 2015 IEEE/ACM 2nd International Symposium on Big Data Computing (BDC) |
| Editors | Rajkumar Buyya, Ioan Raicu, Omer Rana |
| Publisher | Institute of Electrical and Electronics Engineers (IEEE) |
| Pages | 21-30 |
| Number of pages | 10 |
| ISBN (electronic) | 978-0-7695-5696-3 |
| Publication status | Published - 11 Feb 2016 |
| Peer-reviewed | Yes |
Conference
| Title | 2nd IEEE/ACM International Symposium on Big Data Computing, BDC 2015 |
|---|---|
| Duration | 7 - 10 December 2015 |
| City | Limassol |
| Country | Cyprus |
External IDs
| ORCID | /0000-0001-8107-2775/work/198592309 |
|---|
Keywords
Research priority areas of TU Dresden
DFG Classification of Subject Areas according to Review Boards
Subject groups, research areas, subject areas according to Destatis
ASJC Scopus subject areas
Keywords
- Data preprocessing, Machine learning, Web mining