ASG DAMA Data Governance
Content
Metadata: The Critical Foundation for Data Governance
May 17, 2013
Ravi Chivukula Director, Business Solutions
Copyright © 2013 Allen Systems Group Inc. This document is unpublished and the foregoing notice is affixed to protect ASG in the event of inadvertent publication. All rights reserved. No part of this document may be reproduced in any form, including photocopying or transmission electronically to any computer, without prior written consent of ASG. The information contained in this document is confidential and proprietary to ASG and may not be used or disclosed except as expressly authorized in writing by ASG
www.asg.com
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Agenda
• What Metadata is
• Challenges with Data Governance
• Features of Metadata that help address Governance challenges
• Types of Metadata and where it can be found
• How to use Metadata to build an effective Governance program
1
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
What is Metadata?
Business Metadata Information Business Policies Rules Services Service Directory Service Users Interfaces
Business Policies
I Don’t Know What Metadata Is……
Information Usage Data Governance Information Data Quality Quality SLAs Entity Models Information Architecture Relational Master Object Tables Directory Data Elements Data Element Definitions
Access Control
But I Can Sure Tell When I See It !!
Data Formats Aliases & Synonyms
Critical Data Elements
Conceptual Domains
Value Domains
Reference Metadata Reference Tables Business Definitions Definitions
Mappings
Concepts
Business Terms
Semantics
2
Source: Master Data Management by David Loshin
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Challenges with Data Governance
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Highly Complex Data Architectures
Cloud
Visualization
Acct Mktg HR Sales CRM
Reports Dashboards
Analytics
E T L
Data Warehouse
Corporate
Operational Data store
Cross-Channel Analysis
Social Media
MDM
Multi-platform Usage
4
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Metadata: Roadmap for Data Governance
Source physical entities Analyze physical entities Develop Logical and Conceptual entities Map physical to Logical and then to Conceptual entities
Source Conceptual mappings from LOB Develop Enterprise entities Consolidate and map to Enterprise entities
Identify Enterprise entities for Governance Identify Attributes of governable entities Identify governanable attributes
LOB Repository
Enterprise Repository
Enterprise Attributes
5
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Devil maybe in the data, but sinful to ignore Rules!
Most Metadata is benign but Business rules are not! Business Rules manipulate your data when you are not looking!
Legal Corporate IT Finance Analysts
Rule Rule Rule
Rule Rule Rule
Rule Rule Rule
Rule Rule Rule
Rule Rule Rule
Business Rules represent the collective thinking of the enterprise
Governance reaches full potential when it manages Business Rules
Business rules is the metadata that glues your enterprise data
6
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Analyze and Troubleshoot Issues
Metadata Stack
Business Glossary /Conceptual layer
Enterprise Logical layer LOB/Application Logical layer
Upstream/Downstream Data traceability Enables robust impact analysis at the Enterprise level
Physical layer
Database schema
7
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Regulatory Compliance
Business Vocabulary
Root-cause Anaysis.
Data Transparency
Data Lineage
Enterprise Scale
Metadata
History / Versioning
8
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Deep Dive Into Metadata
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Versions and Configurations
Versioning allows storing different versions of an item, each displaying only those properties and details about the item that are relevant for a particular context.
A Configuration is a set of logically associated items (e.g., the items pertaining to a software application at different development stages) and thus define particular views of the items of a database.
10
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Simple Lineage
Traditional Lineage identifies that an item on a report is derived from information that originated in one or more Operational Systems Data Stores
Operational Systems Data Stores
ETL
Data Warehouse
Query
Analytics
Real Data Lineage is more complex, with multiple transformations … and governance demands to know the transformation details
11
© 2013 Allen Systems Group, Inc.
www.asg.com
“Extended” Data Lineage
A good lineage solution will allow this flow to be diagrammed, with user recognizable names in the boxes
Operational Transform Staging Transform Data System Database Warehouse Operational System Operational System
Data Mart
Data Mart
Data Mart
Report
Report
Report
12
© 2013 Allen Systems Group, Inc.
www.asg.com
“Deeper” Data Lineage
“Data Lineage” can be very complicated, and governance requires the ability to dive deep into lineage
Business Process Implemented By Process / Data Flow Application Code
Process / Data Flow
The “flow” may not just be data – it could be messages in a queuing system or even business processes
Implements Transform
The “flow” may be across many systems and from the highest level of abstraction to the most granular detail
13
© 2013 Allen Systems Group, Inc.
www.asg.com
Metadata Sources for Lineage
Business Terms Business Process Models
Operational Stores
ETL Technology
Data Warehouse / Data marts
Reporting Technologies
Data Models
Applications
Traditional Deep Lineage Lineage
14
© 2013 Allen Systems Group, Inc.
www.asg.com
Predominant Metadata Sources
Metadata Repository ASG-Rochade / Business Glossary
DBMS's
DB2/UDB Greenplum MS SQL Server Oracle Sybase Teradata
Transformation Modeling Tools Language
ARIS Casewise Erwin PowerDesigner Rational Tools System Architect C/C++ COBOL JAVA SQL / Stored Procedures
ETL Tools Data Stage Informatica SSIS
BI Tools Business Objects Cognos SSAS/SSRS
15
© 2013 Allen Systems Group, Inc.
www.asg.com
The Components Of A Comprehensive Lineage Approach
Data Model
Process Model
Display Model
Lineage
Solution
16
© 2013 Allen Systems Group, Inc.
www.asg.com
A View Of A Core Data Model
Contains Concepts for broad and deep lineage
Is technology agnostic
Connects naturally to other Repository Information Model constructs
17
© 2013 Allen Systems Group, Inc.
www.asg.com
Basic Process Flow
Build Knowledge Base
Discover Capture Rationalize
Analyze Lineage (Where) Impact (How)
Metadata Capture
18
© 2013 Allen Systems Group, Inc.
www.asg.com
Display Model
Visualization can be selected to Display only selected attributes
19
© 2013 Allen Systems Group, Inc.
www.asg.com
A Look at Business Metadata
Entity
Process (App/ETL)
Table
Attribute
Database
Column
Data Model
Relationship
Business Term
Concepts Definitions
Attributes
Contexts
Conceptual Model
Relationship
Metrics/Standards
20
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
How Does a Glossary help Governance
BUSINESS TERMS
Identifier Customer Authentication
Unique number assigned by the company
Password created by the Customer/user
Identifier Social Security Number Authentication
Unique number assigned by the Govt.
Password created by the Customer/user
Harmonization Through Collaboration The collaborative platform exposes variances that impact semantic consistency Governance Through Collaboration A collaborative platform allows interested individuals to track those business terms, data elements, concepts, and reference metadata that is relevant to your business 21
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Business Terms Help Lineage
Customer
Identifier
Unique number assigned by the company
Data Element Type First Name VARCHAR Last Name VARCHAR CustID VARCHAR Start Date DATE
Multiple definitions for same term can be captured and Shared through a central repository. Discrepancies can be resolved through collaboration
Data Element First Name Middle Last Name Customer_Id Telephone
Type VARCHAR CHAR VARCHAR VARCHAR VARCHAR
22
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
An Integrated Wide Ranging Management Environment
Business Terminology / Concepts ASG-metaGlossary
Higher level / Architectural metadata
ASG-Rochade
Detailed/ Application metadata
ASG-becubic
140 + metadata sources (databases, languages, models, DW/BI tools)
Operational and Decision Support (BI/DW) environments
Mainframe and distributed environments
23
© 2013 Allen Systems Group, Inc.
www.asg.com
Questions
Thank You For Attending!!
Ravi Chivukula Director, Business Solutions Email: [email protected] Tel: 817-652-6323
Carl Roberts Senior Account Executive Email: [email protected] Tel: 303-249-3607
24
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
May 17, 2013
Ravi Chivukula Director, Business Solutions
Copyright © 2013 Allen Systems Group Inc. This document is unpublished and the foregoing notice is affixed to protect ASG in the event of inadvertent publication. All rights reserved. No part of this document may be reproduced in any form, including photocopying or transmission electronically to any computer, without prior written consent of ASG. The information contained in this document is confidential and proprietary to ASG and may not be used or disclosed except as expressly authorized in writing by ASG
www.asg.com
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Agenda
• What Metadata is
• Challenges with Data Governance
• Features of Metadata that help address Governance challenges
• Types of Metadata and where it can be found
• How to use Metadata to build an effective Governance program
1
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
What is Metadata?
Business Metadata Information Business Policies Rules Services Service Directory Service Users Interfaces
Business Policies
I Don’t Know What Metadata Is……
Information Usage Data Governance Information Data Quality Quality SLAs Entity Models Information Architecture Relational Master Object Tables Directory Data Elements Data Element Definitions
Access Control
But I Can Sure Tell When I See It !!
Data Formats Aliases & Synonyms
Critical Data Elements
Conceptual Domains
Value Domains
Reference Metadata Reference Tables Business Definitions Definitions
Mappings
Concepts
Business Terms
Semantics
2
Source: Master Data Management by David Loshin
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Challenges with Data Governance
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Highly Complex Data Architectures
Cloud
Visualization
Acct Mktg HR Sales CRM
Reports Dashboards
Analytics
E T L
Data Warehouse
Corporate
Operational Data store
Cross-Channel Analysis
Social Media
MDM
Multi-platform Usage
4
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Metadata: Roadmap for Data Governance
Source physical entities Analyze physical entities Develop Logical and Conceptual entities Map physical to Logical and then to Conceptual entities
Source Conceptual mappings from LOB Develop Enterprise entities Consolidate and map to Enterprise entities
Identify Enterprise entities for Governance Identify Attributes of governable entities Identify governanable attributes
LOB Repository
Enterprise Repository
Enterprise Attributes
5
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Devil maybe in the data, but sinful to ignore Rules!
Most Metadata is benign but Business rules are not! Business Rules manipulate your data when you are not looking!
Legal Corporate IT Finance Analysts
Rule Rule Rule
Rule Rule Rule
Rule Rule Rule
Rule Rule Rule
Rule Rule Rule
Business Rules represent the collective thinking of the enterprise
Governance reaches full potential when it manages Business Rules
Business rules is the metadata that glues your enterprise data
6
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Analyze and Troubleshoot Issues
Metadata Stack
Business Glossary /Conceptual layer
Enterprise Logical layer LOB/Application Logical layer
Upstream/Downstream Data traceability Enables robust impact analysis at the Enterprise level
Physical layer
Database schema
7
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Regulatory Compliance
Business Vocabulary
Root-cause Anaysis.
Data Transparency
Data Lineage
Enterprise Scale
Metadata
History / Versioning
8
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Deep Dive Into Metadata
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Versions and Configurations
Versioning allows storing different versions of an item, each displaying only those properties and details about the item that are relevant for a particular context.
A Configuration is a set of logically associated items (e.g., the items pertaining to a software application at different development stages) and thus define particular views of the items of a database.
10
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Simple Lineage
Traditional Lineage identifies that an item on a report is derived from information that originated in one or more Operational Systems Data Stores
Operational Systems Data Stores
ETL
Data Warehouse
Query
Analytics
Real Data Lineage is more complex, with multiple transformations … and governance demands to know the transformation details
11
© 2013 Allen Systems Group, Inc.
www.asg.com
“Extended” Data Lineage
A good lineage solution will allow this flow to be diagrammed, with user recognizable names in the boxes
Operational Transform Staging Transform Data System Database Warehouse Operational System Operational System
Data Mart
Data Mart
Data Mart
Report
Report
Report
12
© 2013 Allen Systems Group, Inc.
www.asg.com
“Deeper” Data Lineage
“Data Lineage” can be very complicated, and governance requires the ability to dive deep into lineage
Business Process Implemented By Process / Data Flow Application Code
Process / Data Flow
The “flow” may not just be data – it could be messages in a queuing system or even business processes
Implements Transform
The “flow” may be across many systems and from the highest level of abstraction to the most granular detail
13
© 2013 Allen Systems Group, Inc.
www.asg.com
Metadata Sources for Lineage
Business Terms Business Process Models
Operational Stores
ETL Technology
Data Warehouse / Data marts
Reporting Technologies
Data Models
Applications
Traditional Deep Lineage Lineage
14
© 2013 Allen Systems Group, Inc.
www.asg.com
Predominant Metadata Sources
Metadata Repository ASG-Rochade / Business Glossary
DBMS's
DB2/UDB Greenplum MS SQL Server Oracle Sybase Teradata
Transformation Modeling Tools Language
ARIS Casewise Erwin PowerDesigner Rational Tools System Architect C/C++ COBOL JAVA SQL / Stored Procedures
ETL Tools Data Stage Informatica SSIS
BI Tools Business Objects Cognos SSAS/SSRS
15
© 2013 Allen Systems Group, Inc.
www.asg.com
The Components Of A Comprehensive Lineage Approach
Data Model
Process Model
Display Model
Lineage
Solution
16
© 2013 Allen Systems Group, Inc.
www.asg.com
A View Of A Core Data Model
Contains Concepts for broad and deep lineage
Is technology agnostic
Connects naturally to other Repository Information Model constructs
17
© 2013 Allen Systems Group, Inc.
www.asg.com
Basic Process Flow
Build Knowledge Base
Discover Capture Rationalize
Analyze Lineage (Where) Impact (How)
Metadata Capture
18
© 2013 Allen Systems Group, Inc.
www.asg.com
Display Model
Visualization can be selected to Display only selected attributes
19
© 2013 Allen Systems Group, Inc.
www.asg.com
A Look at Business Metadata
Entity
Process (App/ETL)
Table
Attribute
Database
Column
Data Model
Relationship
Business Term
Concepts Definitions
Attributes
Contexts
Conceptual Model
Relationship
Metrics/Standards
20
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
How Does a Glossary help Governance
BUSINESS TERMS
Identifier Customer Authentication
Unique number assigned by the company
Password created by the Customer/user
Identifier Social Security Number Authentication
Unique number assigned by the Govt.
Password created by the Customer/user
Harmonization Through Collaboration The collaborative platform exposes variances that impact semantic consistency Governance Through Collaboration A collaborative platform allows interested individuals to track those business terms, data elements, concepts, and reference metadata that is relevant to your business 21
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
Business Terms Help Lineage
Customer
Identifier
Unique number assigned by the company
Data Element Type First Name VARCHAR Last Name VARCHAR CustID VARCHAR Start Date DATE
Multiple definitions for same term can be captured and Shared through a central repository. Discrepancies can be resolved through collaboration
Data Element First Name Middle Last Name Customer_Id Telephone
Type VARCHAR CHAR VARCHAR VARCHAR VARCHAR
22
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
An Integrated Wide Ranging Management Environment
Business Terminology / Concepts ASG-metaGlossary
Higher level / Architectural metadata
ASG-Rochade
Detailed/ Application metadata
ASG-becubic
140 + metadata sources (databases, languages, models, DW/BI tools)
Operational and Decision Support (BI/DW) environments
Mainframe and distributed environments
23
© 2013 Allen Systems Group, Inc.
www.asg.com
Questions
Thank You For Attending!!
Ravi Chivukula Director, Business Solutions Email: [email protected] Tel: 817-652-6323
Carl Roberts Senior Account Executive Email: [email protected] Tel: 303-249-3607
24
www.asg.com
Copyright®2010 Allen Systems Group, Inc.
