2anii2s, 1937 Demystifying Semantic Layers for Sel-Servce Analytics Gartner. Licensed for Distribution This research note is restricted to the personal use of Victor Alfonso Flores Cruz ([email protected]). Demystifying Semantic Layers for Self-Service Analytics 3 April 2023 - 1D G00783855 - 32 min read By Christopher Long, Joe Antelmi Initiatives: Analytics and Artificial Intelligence for Technical Professionals Data and analytics technical professionals struggle to deliver a universal semantic layer that balances agility and control in self-service analytics. This research helps technical professionals review options for deploying semantic layers to enable self-service analytics Overview Key Findings Implementing a universal semantic layer continues to be difficult due to lack of tool interoperability, poor usability, lagging data governance maturity and organizational inertia Meanwhile, the pressure from organizations to deliver metric consistency has only grown as consumption channels for analytics have expanded, * Widespread self-service analytics development fosters innovation and agility at the expense of the governance and consistency once provided through centralized semantic layer architectures. ‘* Modern A&Bi tools are expanding access to their modeling layers for more agnostic enterprise consumption. At the same time, stand-alone semantic layer platforms are extending capabilities to incorporate metrics store concepts. The expansion of these capabilities blurs the lines for organizations looking to implement a universal semantic layer. Recommendations As a data and analytics technical professional implementing semantic layers, you should: hntps:nww.gariner.comidocument!42402907ref=solrAllrefvale3854097068 11832423, 1937 Demysttving Semantic Layers fr Slt Service Anaytes * Evaluate and select an appropriate semantic layer for your use case by comparing the benefits, challenges and representative vendors of the three technical options: data layer semantics, stand-alone semantic layer, and A&B! tool semantic layer. * Design a federated analytics architecture. Include a combination of local and global semantic layer data models based on the use cases, users and desired governance models. ‘* Develop operationalization processes that leverage A&BI tool capabilities for prototype-to- production semantic layer development. This approach enables business measures, metrics and data models to originate as A&BI prototypes and go into production with appropriate enterprise- level support Comparison As organizations drive toward the goal of becoming data-driven, the demand for analytics increases significantly, Semantic layers are often one part of the solution to deliver the needed analytics across the enterprise. Historically, a centralized, governed semantic layer has been developed and maintained by IT to support BI and enterprise reporting use cases. However, the agility demanded by organizations and the capabilities delivered by modern analytics and BI tools decentralizes this responsibility. No matter how it is implemented, the semantic layer is still an important element of an analytics architecture. But the classic, centralized semantic layer is no longer the only option in this space. Semantic Layer Implementation Options Organizations taking a deliberate approach to implementing a semantic data layer will find several primary technology implementation options: © Data layer semantics * Stand-alone semantic layer * Analytics and business intelligence (A&B!) tool semantic layer These approaches intersect with a variety of locations in the data pipeline where you can place the semantic layer. This dynamic will be familiar to technical professionals who are deploying a logical data warehouse (LDW) or lakehouse architecture. Hence, similar guidance applies to the implementation decisions for semantic layers. The first two architectural options (below) describe a semantic layer as part of an organization's modem data architecture, such as an LDW or data fabric. The LDW is designed to satisfy the majority of analytics requirements. LDWs support a broad set of analytics engines that can serve a wide variety of users and applications. For that reason, placing the semantic layer in the LDW is. often optimal. The LDW itself is composed of multiple component parts, making a semantic layer a hntps:nww.gariner.comidocument!42402907ref=solrAllrefvale3854097068 21332423, 1937 Domysttving Semantic Layers for SeltServee Analytics logical construct of components that are applied based on the specific technologies implemented For additional information on the LDW construct, see Adopting a Logical Data Warehouse. Option 1: Data Layer Semantics This option describes a semantic layer that is built as an extension of data services within the data layer. Artifacts in this scenario may be made up of a variety of data marts, views (including materialized views), and online analytical processing (OLAP) models. They are presented as the connection point for analytics developers to link with source data, This manifestation of the semantic layer may not always appear to be a single, universal layer due to the potential for random asset development. Option 2: Stand-Alone Semantic Layer This option describes a semantic layer that exists as its own architectural element within the data and analytics stack. It is placed between the source data and consumption layers. In this architectural pattern, the tool used to build the semantic layer obscures the source data from analytics developers and consumers and becomes the primary connection point for analytical source data. Depending on the technology implemented, stand-alone semantic layers may manifest in a couple of ways: * As data virtualization platforms, where source data largely stays in place * As an abstraction layer, caching data for analytical usage separate from the source Option 3: A&B! Tool Semantic Layer This option describes a semantic layer with a local, optimized data store in the A&BI tools implemented. Because of its location inside an A&BI application, itis likely to be a siloed solution. Many organizations employ multiple A&B! tools for a variety of reasons, including * By design to take advantage of distinct capabilities within BI tools * Growth of the organization through acquisition where the acquired company uses a different A&B! solution * Business unit autonomy to purchase their own A&B! solutions While multiple tools can bring agility to business user development, they can also make analytical definitions more fragmented throughout the organization. Early reporting tools that implemented semantic layers were very siloed. And while modern A&BI tools allow for greater sharing between developer users, the application silo effect is still quite prevalent. hntps:nww.gariner.comidocument!42402907ref=solrAllrefvale3854097068 31332anii2s, 1937 Demystifying Semantic Layers for Sel-Servce Analytics Table 1 compares the characteristics of the different types of semantic layers Table 1: Common Semantic Layer Implementations PeTCn yg RU ee ead eae Cg ea ee ey Common Features Developer Roles Governance Localization Key ‘Strengths Is an extension of data store capabilities Sits at the edge of the data layer May include views (including materialized), data marts and graphs Data engineers Centralized Is highly governed Leverages in- house technology and skills Can deliver high volumes of data to many concurrent users Is an independent platform solution Sits between the data and consumption layers May include virtualization, abstraction and data lake enablement Date engineers Centralized Is well-governed Serves as a centralized source for analytical modeling and metrics Connects with diverse data formats across varying platforms Is a Bl tool capability Is part of the analytics consumption layer Includes cirect query and ingestiontype models Business analysts and data analysts Distributed Enables flexible and agile implementation and development Democratizes analytics development » Reduces time to. insight htps:nww.garner.comidocument#42402907ref=selAllrefval-3854097068 41332423, 1937 Demystifying Semantic Layers for Selt-Servce Analytics ETP EZ RCT PLA Teo Rd Ua Cd Reed Key « Ishighly reliant «Involves IT-heaw + Is less governed Challenges on central data development and + Offers inconsistent engineering implementation development and resources «Is traditionally not deployment of + Isslowto designed to support analytical models change data science and and metrics «Has limited mactine leering » Isoften isolate capabilities with (SML) sap within the vendor unstructured integrated apolication stack data use cases + Introduces another expensive layer to the data and analytics solution Example IBM, Oracle, SAP, Apache Druid, Apache MicroStrategy, Vendors Snowflake, Pinot, AtScale, Pyramid Analytics, Teradata ClickHouse, Data lik, Salesforce Virtuality, dbt, Denodo, (Tableau), Sisense, Dremio, Kyligence, Kyvos ThoughtSpot Insights, Zetaris Note: * Major vendors, including Amazon, Google, Microsoft, Oracle and SAP, offer capabilities to support ‘semantic layers in multiple architectural patterns. ‘Source: Gartner (April 2023) The remainder of this document will: * Provide background on semantic layer concepts ‘* Compare the three types of semantic layers © Provide guidance on choosing the right type of semantic layer for your needs Analysis hntps:nww.gariner.comidocument!42402907ref=solrAllrefvale3854097068 51332423, 1937 Demysttving Semantic Layers fr Slt Service Anaytes The efforts made by organizations to become data-driven often push the resource limits of what central data teams are able or willing to provide for A&BI reporting. Many data and analytics teams are overwhelmed by requests for related analytical deliverables, In response, many organizations deploy self-service A&B! tools to deliver the agility and flexibility that traditional (centralized) A&BI development does not provide for. The problem is that self-service A&BI tools (including their related semantic modeling capabilities) often turn a once-governed data environment into an uncontrolled, fragmented and inconsistent environment. This fragmented environment creates risk in decision making and breeds loss of trust by end users, The need for consistency in business language, analytics calculations, modeling and representation remains a key focus today as organizations strive to become data- driven. Defini n of “Self-Service” in This Research Self-service, generally provided through modern A&Bl tools, can be divided into two primary categories: * Self-service data prep: The capabilities and processes that enable data scientists and business users to shape and cleanse data for further analysis. Often, this process results in tables that are shared across multiple analyses. « Self-service analytics: The capabilities and features that allow analysts and business users to connect and model data for analysis, without relying on IT or a central developer resource. These models can become part of an organization's semantic layer. For the purposes of this research, we will discuss self-service analytics only and refer to it broadly as “self-service” Today's data and analytics technical professionals are challenged to deliver data and analytics solutions that adhere to the following general principles: * Outcome-oriented: Aligns to the broader organization's goals * Valuable: Provides a benefit to users (e.g,, is accessible to a larger population of users, is more flexible, provides more sophisticated analytics or is more affordable) + Easy to learn: Is intuitive to grasp, with training resources available hntps:nww.gariner.comidocument!42402907ref=solrAllrefvale3854097068 61332423, 1937 Domysttving Semantic Layers for SeltServee Analytics * Available and reusable: Is easily accessed, embedded in applications and workflows, and reusable by multiple systems and users * Safe and trusted: ‘© Supports governance policy * Includes sophisticated identity, access and security management « Is quality-controlled, regulatory-compliant, and delivered by a performant, reliable platform However, delivering on all of these principles is not easy. And historically, an IT-built semantic layer was a big part of the solution. But organizations struggle to balance analytics development agility with control in order to deliver valuable outcomes to a diverse set of users Business units are focused on delivering value in the form of fast, agile analytics in their silos. In the process, they often undermine broader organizational goals for data consistency and a shared set of key performance indicators (KPIs). By contrast, IT departments are occupied with furthering organizational goals for trusted, safe, IT-led data management. Therefore, they don't provide business units with the data or the analytics capabilities that they need in a timely fashion. These competing objectives have spurred a never-ending debate between control and freedom (see Figure 1). Download All Graphics in This Material + hntps:nww.gariner.comidocument!42402907ref=solrAllrefvale3854097068 71332423, 1937 Domysttving Semantic Layers for SeltServee Analytics Figure 1: The Never-Ending Debate Between Control and Freedom + Resolve the Never-Ending Debate Between Control and Freedom Consistency Autonomy. Share Best Practice Agility Consensus Innovation We must create a data architecture, organizational model, and governance framework that delivers the benefits of both. Gartner. Because modern tools and use cases offer diverse ways of implementing semantic layers, organizations’ views on the ideal semantic layer must evolve. Background on Semantic Layers What Is a Semantic Layer? Originally introduced as a term in 1991, a semantic layer is a business representation of data. It provides a consistent, unified view of — and access to — organizational data in common business terms. However, the concept of a semantic layer has existed as long as organizations have tried to model and deliver data in end-user terms. Regardless of how or where a semantic layer is implemented, it should be consumption-tool- agnostic and provide the following core functions: * A translation of the underlying database structures into business-user-oriented terms and constructs. ‘* Views of data elements that are intuitive to business users. hntps:nww.gariner.comidocument!42402907ref=solrAllrefvale3854097068 81332anii2s, 1937 Demystifying Semantic Layers for Sel-Servce Analytics «The opportunity to rename data elements so that they make sense to business users. * An interface to hold business descriptions of data elements. ‘* Amechanism to define and store calculations and business rules. ‘* The ability to apply rules and access privileges to KPIs and datasets. (The semantic layer is a junction for role-based access control and auditing.) With these functions in mind, semantic layers are built upon multiple foundational components, as shown in Figure 2. Figure 2: Semantic Layer Components + Semantic Layer Components Peay BES oSe B) vata Models a Creme 34+ Collection offConnection to Data Gartner Although the broad ambition of the universal semantic layer remains unfulfilled, it is important to understand where the semantic layer has come from and where it sits in today's modern D&A architecture. The following section outlines the history of the semantic layer and the emerging trends data and analytics technical professionals should be aware of. How Has the Semantic Layer Evolved? As noted, the concept of the semantic layer is not a new phenomenon and predates the implementation of the enterprise data warehouse commonly found today. Over time, the drive for agllity and the rise of self-service capabilities started to overshadow the use of centrally managed semantic layers. Figure 3 shows this movement. Today, we see a convergence of capabilities between independent semantic layer offerings and self-service BI tools. hntps:nww.gariner.comidocument!42402907ref=solrAllrefvale3854097068 91332423, 1937 Domysttving Semantic Layers for SeltServee Analytics Figure 3: Semantic Layer Journey + Semantic Layer Journey Illustrative a I z z i i £ e é z j i i ‘coma § | Covered : } SettSerion 5 | (Federated) 2) | Rrchectre |B 7 i 5 a i § Timeline 88 1980 2000 wwe 2020 6 Data Architectures Emergence of the LW Conceptual Architecture 792855,¢ Gartner The drive toward this convergence of capabilities has several prevailing themes * Organizations’ demands to balance governance and agility: Organizations that adopted self- service — whether organically or as a reaction to the inflexibility of IT-controlled semantic layers —find themselves straddled with mounting technical debt to maintain fractured views of metrics. They are looking for some enterprise level of governance to guide further development. * Organizations’ increased demands for integrated analytics: The growing demand for analytics across use cases, including data science and machine learning (DSML) and integrated applications, has caused many organizations to build dedicated pipelines to serve these needs. Both traditional and self-service semantic models generally have not supported these use cases. hntps:nww.gariner.comidocument!42402907ref=solrAllrefvale3854097068 10133