Data Integration
Data Integration
Data integration involves combining data residing in different sources and providing users with a unified
view of them.[1] This process becomes significant in a variety of situations, which include both commercial
(such as when two similar companies need to merge their databases) and scientific (combining research
results from different bioinformatics repositories, for example) domains. Data integration appears with
increasing frequency as the volume (that is, big data) and the need to share existing data explodes.[2] It has
become the focus of extensive theoretical work, and numerous open problems remain unsolved. Data
integration encourages collaboration between internal as well as external users. The data being integrated
must be received from a heterogeneous database system and transformed to a single coherent data store that
provides synchronous data across a network of files for clients.[3] A common use of data integration is in
data mining when analyzing and extracting information from existing databases that can be useful for
Business information.[4]
History
Issues with combining heterogeneous data sources are often
referred to as information silos, under a single query interface have
existed for some time. In the early 1980s, computer scientists began
designing systems for interoperability of heterogeneous
databases.[5] The first data integration system driven by structured
metadata was designed at the University of Minnesota in 1991, for
the Integrated Public Use Microdata Series (IPUMS). IPUMS used
a data warehousing approach, which extracts, transforms, and loads
data from heterogeneous sources into a unique view schema so data Figure 1: Simple schematic for a
data warehouse. The Extract,
from different sources become compatible.[6] By making thousands
transform, load (ETL) process
of population databases interoperable, IPUMS demonstrated the
extracts information from the source
feasibility of large-scale data integration. The data warehouse
databases, transforms it and then
approach offers a tightly coupled architecture because the data are
loads it into the data warehouse.
already physically reconciled in a single queryable repository, so it
usually takes little time to resolve queries.[7]
The data warehouse approach is less feasible for data sets that are
frequently updated, requiring the extract, transform, load (ETL)
process to be continuously re-executed for synchronization.
Difficulties also arise in constructing data warehouses when one has
only a query interface to summary data sources and no access to the
full data. This problem frequently emerges when integrating several
commercial query services like travel or classified advertisement
web applications.
Figure 2: Simple schematic for a
As of 2009 the trend in data integration favored the loose coupling
data-integration solution. A system
of data[8] and providing a unified query-interface to access real time designer constructs a mediated
data over a mediated schema (see Figure 2), which allows schema against which users can run
information to be retrieved directly from original databases. This is queries. The virtual database
consistent with the SOA approach popular in that era. This interfaces with the source databases
approach relies on mappings between the mediated schema and the via wrapper code if required.
schema of original sources, and translating a query into decomposed queries to match the schema of the
original databases. Such mappings can be specified in two ways: as a mapping from entities in the mediated
schema to entities in the original sources (the "Global-as-View"[9] (GAV) approach), or as a mapping from
entities in the original sources to the mediated schema (the "Local-as-View"[10] (LAV) approach). The
latter approach requires more sophisticated inferences to resolve a query on the mediated schema, but
makes it easier to add new data sources to a (stable) mediated schema.
As of 2010 some of the work in data integration research concerns the semantic integration problem. This
problem addresses not the structuring of the architecture of the integration, but how to resolve semantic
conflicts between heterogeneous data sources. For example, if two companies merge their databases,
certain concepts and definitions in their respective schemas like "earnings" inevitably have different
meanings. In one database it may mean profits in dollars (a floating-point number), while in the other it
might represent the number of sales (an integer). A common strategy for the resolution of such problems
involves the use of ontologies which explicitly define schema terms and thus help to resolve semantic
conflicts. This approach represents ontology-based data integration. On the other hand, the problem of
combining research results from different bioinformatics repositories requires bench-marking of the
similarities, computed from different data sources, on a single criterion such as positive predictive value.
This enables the data sources to be directly comparable and can be integrated even when the natures of
experiments are distinct.[11]
As of 2011 it was determined that current data modeling methods were imparting data isolation into every
data architecture in the form of islands of disparate data and information silos. This data isolation is an
unintended artifact of the data modeling methodology that results in the development of disparate data
models. Disparate data models, when instantiated as databases, form disparate databases. Enhanced data
model methodologies have been developed to eliminate the data isolation artifact and to promote the
development of integrated data models.[12] One enhanced data modeling method recasts data models by
augmenting them with structural metadata in the form of standardized data entities. As a result of recasting
multiple data models, the set of recast data models will now share one or more commonality relationships
that relate the structural metadata now common to these data models. Commonality relationships are a peer-
to-peer type of entity relationships that relate the standardized data entities of multiple data models. Multiple
data models that contain the same standard data entity may participate in the same commonality
relationship. When integrated data models are instantiated as databases and are properly populated from a
common set of master data, then these databases are integrated.
Since 2011, data hub approaches have been of greater interest than fully structured (typically relational)
Enterprise Data Warehouses. Since 2013, data lake approaches have risen to the level of Data Hubs. (See
all three search terms popularity on Google Trends.[13]) These approaches combine unstructured or varied
data into one location, but do not necessarily require an (often complex) master relational schema to
structure and define all data in the Hub.
Data integration plays a big role in business regarding data collection used for studying the market.
Converting the raw data retrieved from consumers into coherent data is something businesses try to do
when considering what steps they should take next.[14] Organizations are more frequently using data
mining for collecting information and patterns from their databases, and this process helps them develop
new business strategies to increase business performance and perform economic analyses more efficiently.
Compiling the large amount of data they collect to be stored in their system is a form of data integration
adapted for Business intelligence to improve their chances of success.[15]
Example
Consider a web application where a user can query a variety of information about cities (such as crime
statistics, weather, hotels, demographics, etc.). Traditionally, the information must be stored in a single
database with a single schema. But any single enterprise would find information of this breadth somewhat
difficult and expensive to collect. Even if the resources exist to gather the data, it would likely duplicate
data in existing crime databases, weather websites, and census data.
A data-integration solution may address this problem by considering these external resources as
materialized views over a virtual mediated schema, resulting in "virtual data integration". This means
application-developers construct a virtual schema—the mediated schema—to best model the kinds of
answers their users want. Next, they design "wrappers" or adapters for each data source, such as the crime
database and weather website. These adapters simply transform the local query results (those returned by
the respective websites or databases) into an easily processed form for the data integration solution (see
figure 2). When an application-user queries the mediated schema, the data-integration solution transforms
this query into appropriate queries over the respective data sources. Finally, the virtual database combines
the results of these queries into the answer to the user's query.
This solution offers the convenience of adding new sources by simply constructing an adapter or an
application software blade for them. It contrasts with ETL systems or with a single database solution, which
require manual integration of entire new data set into the system. The virtual ETL solutions leverage virtual
mediated schema to implement data harmonization; whereby the data are copied from the designated
"master" source to the defined targets, field by field. Advanced data virtualization is also built on the
concept of object-oriented modeling in order to construct virtual mediated schema or virtual metadata
repository, using hub and spoke architecture.
Each data source is disparate and as such is not designed to support reliable joins between data sources.
Therefore, data virtualization as well as data federation depends upon accidental data commonality to
support combining data and information from disparate data sets. Because of the lack of data value
commonality across data sources, the return set may be inaccurate, incomplete, and impossible to validate.
One solution is to recast disparate databases to integrate these databases without the need for ETL. The
recast databases support commonality constraints where referential integrity may be enforced between
databases. The recast databases provide designed data access paths with data value commonality across
databases.
Theory
The theory of data integration[1] forms a subset of database theory and formalizes the underlying concepts
of the problem in first-order logic. Applying the theories gives indications as to the feasibility and difficulty
of data integration. While its definitions may appear abstract, they have sufficient generality to
accommodate all manner of integration systems,[16] including those that include nested relational / XML
databases[17] and those that treat databases as programs.[18] Connections to particular databases systems
such as Oracle or DB2 are provided by implementation-level technologies such as JDBC and are not
studied at the theoretical level.
Definitions
Data integration systems are formally defined as a tuple where is the global (or mediated)
schema, is the heterogeneous set of source schemas, and is the mapping that maps queries between
the source and the global schemas. Both and are expressed in languages over alphabets composed of
symbols for each of their respective relations. The mapping consists of assertions between queries over
and queries over . When users pose queries over the data integration system, they pose queries over
and the mapping then asserts connections between the elements in the global schema and the source
schemas.
A database over a schema is defined as a set of sets, one for each relation (in a relational database). The
database corresponding to the source schema would comprise the set of sets of tuples for each of the
heterogeneous data sources and is called the source database. Note that this single source database may
actually represent a collection of disconnected databases. The database corresponding to the virtual
mediated schema is called the global database. The global database must satisfy the mapping with
respect to the source database. The legality of this mapping depends on the nature of the correspondence
between and . Two popular ways to model this correspondence exist: Global as View or GAV and
Local as View or LAV.
On the other hand, in LAV, the source database is modeled as a set of views over . In this case
associates to each element of a query over . Here the exact associations between and are no
longer well-defined. As is illustrated in the next section, the burden of determining how to retrieve elements
from the sources is placed on the query processor. The benefit of an LAV modeling is that new sources can
be added with far less work than in a GAV system, thus the LAV approach should be favored in cases
where the mediated schema is less stable or likely to change.[1]
In an LAV approach to the example data integration system above, the system designer designs the global
schema first and then simply inputs the schemas of the respective city information sources. Consider again
if one of the sources serves a weather website. The designer would add corresponding elements for weather
to the global schema only if none existed already. Then programmers write an adapter or wrapper for the
website and add a schema description of the website's results to the source schemas. The complexity of
adding the new source moves from the designer to the query processor.
Query processing
The theory of query processing in data integration systems is commonly expressed using conjunctive
queries and Datalog, a purely declarative logic programming language.[20] One can loosely think of a
conjunctive query as a logical function applied to the relations of a database such as " where
". If a tuple or set of tuples is substituted into the rule and satisfies it (makes it true), then we
consider that tuple as part of the set of answers in the query. While formal languages like Datalog express
these queries concisely and without ambiguity, common SQL queries count as conjunctive queries as well.
In terms of data integration, "query containment" represents an important property of conjunctive queries. A
query contains another query (denoted ) if the results of applying are a subset of the results
of applying for any database. The two queries are said to be equivalent if the resulting sets are equal for
any database. This is important because in both GAV and LAV systems, a user poses conjunctive queries
over a virtual schema represented by a set of views, or "materialized" conjunctive queries. Integration seeks
to rewrite the queries represented by the views to make their results equivalent or maximally contained by
our user's query. This corresponds to the problem of answering queries using views (AQUV).[21]
In GAV systems, a system designer writes mediator code to define the query-rewriting. Each element in the
user's query corresponds to a substitution rule just as each element in the global schema corresponds to a
query over the source. Query processing simply expands the subgoals of the user's query according to the
rule specified in the mediator and thus the resulting query is likely to be equivalent. While the designer does
the majority of the work beforehand, some GAV systems such as Tsimmis (http://www-db.stanford.edu/tsi
mmis/) involve simplifying the mediator description process.
In LAV systems, queries undergo a more radical process of rewriting because no mediator exists to align
the user's query with a simple expansion strategy. The integration system must execute a search over the
space of possible queries in order to find the best rewrite. The resulting rewrite may not be an equivalent
query but maximally contained, and the resulting tuples may be incomplete. As of 2011 the GQR
algorithm[22] is the leading query rewriting algorithm for LAV data integration systems.
In general, the complexity of query rewriting is NP-complete.[21] If the space of rewrites is relatively small,
this does not pose a problem — even for integration systems with hundreds of sources.
See also
Business semantics management Change data capture
Core data integration Information server
Customer data integration Information silo
Cyberinfrastructure Integration Competency Center
Data blending Integration Consortium
Data curation ISO 15926 (http://15926.org): Integration of
Data fusion life-cycle data for process plants including
oil and gas production facilities
Data mapping
JXTA
Data wrangling
Database model Master data management
Dataspaces Object-relational mapping
Edge data integration Open Text
Enterprise application integration Semantic integration
Enterprise architecture framework Schema matching
Enterprise information integration (EII) Three schema approach
Enterprise integration UDEF
Web data integration
Geodi: Geoscientific Data Integration
Web service
Information integration
References
1. Maurizio Lenzerini (2002). "Data Integration: A Theoretical Perspective" (http://www.dis.uniro
ma1.it/~lenzerin/homepagine/talks/TutorialPODS02.pdf) (PDF). PODS 2002. pp. 233–246.
2. Frederick Lane (2006). "IDC: World Created 161 Billion Gigs of Data in 2006" (https://web.ar
chive.org/web/20150715160327/http://www.toptechnews.com/article/index.php?story_id=01
300000E3D0). Archived from the original (http://www.toptechnews.com/article/index.php?sto
ry_id=01300000E3D0) on 2015-07-15.
3. mikben. "Data Coherency - Win32 apps" (https://docs.microsoft.com/en-us/windows/win32/fil
eio/data-coherency). docs.microsoft.com. Archived (https://web.archive.org/web/2020061204
5601/https://docs.microsoft.com/en-us/windows/win32/fileio/data-coherency) from the
original on 2020-06-12. Retrieved 2020-11-23.
4. Chung, P.; Chung, S. H. (2013-05). "On data integration and data mining for developing
business intelligence". 2013 IEEE Long Island Systems, Applications and Technology
Conference (LISAT): 1–6. doi:10.1109/LISAT.2013.6578235.
5. John Miles Smith; et al. (1982). "Multibase: integrating heterogeneous distributed database
systems" (http://dl.acm.org/citation.cfm?id=1500483). AFIPS '81 Proceedings of the May 4–
7, 1981, National Computer Conference. pp. 487–499.
6. Steven Ruggles, J. David Hacker, and Matthew Sobek (1995). "Order out of Chaos: The
Integrated Public Use Microdata Series". Historical Methods. Vol. 28. pp. 33–39.
7. Jennifer Widom (1995). "Research problems in data warehousing" (http://dl.acm.org/citation.
cfm?id=221319). CIKM '95 Proceedings of the Fourth International Conference on
Information and Knowledge Management. pp. 25–30.
8. Pautasso, Cesare; Wilde, Erik (2009-04-20). "Why is the web loosely coupled? a multi-
faceted metric for service design" (https://doi.org/10.1145/1526709.1526832). Proceedings
of the 18th International Conference on World Wide Web. WWW '09. Madrid, Spain:
Association for Computing Machinery: 911–920. doi:10.1145/1526709.1526832 (https://doi.
org/10.1145%2F1526709.1526832). ISBN 978-1-60558-487-4. S2CID 207172208 (https://a
pi.semanticscholar.org/CorpusID:207172208).
9. "What is GAV (Global as View)?" (https://www.geeksforgeeks.org/what-is-gav-global-as-vie
w/). GeeksforGeeks. 2020-04-18. Archived (https://web.archive.org/web/20201130194235/ht
tps://www.geeksforgeeks.org/what-is-gav-global-as-view/) from the original on 2020-11-30.
Retrieved 2020-11-23.
10. "Local-as-View" (https://de.wikipedia.org/w/index.php?title=Local-as-View&oldid=20217923
2), Wikipedia (in German), 2020-07-24, retrieved 2020-11-23
11. Shubhra S. Ray; et al. (2009). "Combining Multi-Source Information through Functional
Annotation based Weighting: Gene Function Prediction in Yeast" (http://shubhrasankar.tripo
d.com/cgi-bin/combiningMultisourceIEEE.pdf) (PDF). IEEE Transactions on Biomedical
Engineering. 56 (2): 229–236. CiteSeerX 10.1.1.150.7928 (https://citeseerx.ist.psu.edu/view
doc/summary?doi=10.1.1.150.7928). doi:10.1109/TBME.2008.2005955 (https://doi.org/10.11
09%2FTBME.2008.2005955). PMID 19272921 (https://pubmed.ncbi.nlm.nih.gov/19272921).
S2CID 10848834 (https://api.semanticscholar.org/CorpusID:10848834). Archived (https://we
b.archive.org/web/20100508164830/http://shubhrasankar.tripod.com/cgi-bin/combiningMulti
sourceIEEE.pdf) (PDF) from the original on 2010-05-08. Retrieved 2012-05-17.
12. Michael Mireku Kwakye (2011). "A Practical Approach To Merging Multidimensional Data
Models". hdl:10393/20457 (https://hdl.handle.net/10393%2F20457).
13. "Hub Lake and Warehouse search trends" (https://www.google.com/trends/explore#q=enterp
rise%20data%20warehouse%2C%20%22data%20hub%22%2C%20%22data%20lake%22
&cmpt=q&tz=Etc%2FGMT%2B5). Archived (https://web.archive.org/web/20170217030131/h
ttps://www.google.com/trends/explore#q=enterprise%20data%20warehouse%2C%20%22d
ata%20hub%22%2C%20%22data%20lake%22&cmpt=q&tz=Etc%2FGMT%2B5) from the
original on 2017-02-17. Retrieved 2016-01-12.
14. "Data mining in business analytics" (https://www.wgu.edu/blog/data-mining-business-analyti
cs2005.html#:~:text=Simply%20put%2C%20data%20mining%20is,raw%20data%20into%2
0useful%20information.&text=It%20pulls%20out%20information%20from,%2C%20market%
20effectively%2C%20and%20more.). Western Governors University. May 15, 2020.
Archived (https://web.archive.org/web/20201223004846/https://www.wgu.edu/blog/data-mini
ng-business-analytics2005.html#:~:text=Simply%20put%2C%20data%20mining%20is,ra
w%20data%20into%20useful%20information.&text=It%20pulls%20out%20information%20fr
om,%2C%20market%20effectively%2C%20and%20more.) from the original on December
23, 2020. Retrieved November 22, 2020.
15. Surani, Ibrahim (2020-03-30). "Data Integration for Business Intelligence: Best Practices" (htt
ps://www.dataversity.net/data-integration-for-business-intelligence-best-practices/).
DATAVERSITY. Archived (https://web.archive.org/web/20201130072535/https://www.datave
rsity.net/data-integration-for-business-intelligence-best-practices/) from the original on 2020-
11-30. Retrieved 2020-11-23.
16. Alagić, Suad; Bernstein, Philip A. (2002). Database Programming Languages. Lecture
Notes in Computer Science. Vol. 2397. pp. 228–246. doi:10.1007/3-540-46093-4_14 (https://
doi.org/10.1007%2F3-540-46093-4_14). ISBN 978-3-540-44080-2.
17. "Nested Mappings: Schema Mapping Reloaded" (http://www.vldb.org/conf/2006/p67-fuxma
n.pdf) (PDF). Archived (https://web.archive.org/web/20151028054747/http://www.vldb.org/co
nf/2006/p67-fuxman.pdf) (PDF) from the original on 2015-10-28. Retrieved 2015-09-10.
18. "The Common Framework Initiative for algebraic specification and development of software"
(http://homepages.inf.ed.ac.uk/dts/pub/psi.pdf) (PDF). Archived (https://web.archive.org/web/
20160304095226/http://homepages.inf.ed.ac.uk/dts/pub/psi.pdf) (PDF) from the original on
2016-03-04. Retrieved 2015-09-10.
19. Christoph Koch (2001). "Data Integration against Multiple Evolving Autonomous Schemata"
(https://web.archive.org/web/20070926211342/http://www.csd.uoc.gr/~hy562/Papers/thesis_
final.pdf) (PDF). Archived from the original (http://www.csd.uoc.gr/~hy562/Papers/thesis_fina
l.pdf) (PDF) on 2007-09-26.
20. Jeffrey D. Ullman (1997). "Information Integration Using Logical Views" (http://www-db.stanf
ord.edu/pub/papers/integration-using-views.ps). ICDT 1997. pp. 19–40.
21. Alon Y. Halevy (2001). "Answering queries using views: A survey" (http://www.cs.uwaterloo.c
a/~david/cs740/answering-queries-using-views.pdf) (PDF). The VLDB Journal. pp. 270–
294.
22. George Konstantinidis; et al. (2011). "Scalable Query Rewriting: A Graph-based Approach"
(http://www.southampton.ac.uk/~gk1e17/sigmod565konstantinidis.pdf) (PDF). in
Proceedings of the ACM SIGMOD International Conference on Management of Data,
SIGMOD'11, June 12–16, 2011, Athens, Greece.
23. William Michener; et al. "DataONE: Observation Network for Earth" (https://www.dataone.or
g/). www.dataone.org. Archived (https://web.archive.org/web/20130122055843/http://www.da
taone.org/) from the original on 2013-01-22. Retrieved 2013-01-19.
24. Sayeed Choudhury; et al. "Data Conservancy" (https://dataconservancy.org/).
dataconservancy.org. Archived (https://web.archive.org/web/20130113003316/http://datacon
servancy.org/) from the original on 2013-01-13. Retrieved 2013-01-19.
25. Margaret Hedstrom; et al. "SEAD Sustainable Environment - Actionable Data" (http://sead-d
ata.net/). sead-data.net. Archived (https://web.archive.org/web/20120920094243/http://sead-
data.net//) from the original on 2012-09-20. Retrieved 2013-01-19.
26. Reagan Moore; et al. "DataNet Federation Consortium" (http://datafed.org/). datafed.org.
Archived (https://web.archive.org/web/20130415161955/http://datafed.org/) from the original
on 2013-04-15. Retrieved 2013-01-19.
27. Steven Ruggles; et al. "Terra Populus: Integrated Data on Population and the Environment"
(http://www.terrapop.org/). terrapop.org. Archived (https://web.archive.org/web/20130518050
551/http://www.terrapop.org/) from the original on 2013-05-18. Retrieved 2013-01-19.
28. Bill Nichols. "Research Data Alliance" (http://rd-alliance.org/). rd-alliance.org. Archived (http
s://web.archive.org/web/20141118024001/https://www.rd-alliance.org/) from the original on
2014-11-18. Retrieved 2014-10-01.
External links