DATA WAREHOUSING AND DATA MINING
ABSTRACT:
Fast, accurate and scalable data analysis techniques are needed to extract useful
information from huge pile of data. Data warehouse is a single, integrated source of decision
support information formed by collecting data from multiple sources, internal to the
organization as well as external, and transforming and summarizing this information to enable
improved decision making. Data warehouse is designed for easy access by users to large
amounts of information, and data access is typically supported by specialized analytical tools
and applications. Typical applications include decision support systems and execution
information system.
Data mining is the exploration and analysis of large quantities of data in order to
discover valid, novel, potentially useful, and ultimately understandable patterns in data. It is
“An information extraction activity whose goal is to discover hidden facts contained in
databases”.
The process of extracting valid, previously unknown, comprehensible and actionable
information from large databases and using it to make crucial business decisions.
The project entitled “Website Data Mining” is an application of data mining
which is built for the website developers for their effective creation of websites in
internet.
Data mining finds patterns and subtle relationships in data and infers rules that allow
the prediction of future results. It produces output values for an assigned set of input values.
Typical applications include market segmentation, customer profiling, fraud detection,
evaluation of retail promotions, and credit risk analysis.
DATA WAREHOUSING
Everyday increasingly, organizations are analyzing current and historical data to
identify useful patterns and support business strategies.
A large amount of the right information is the key to survival in today’s competitive
environment. And this kind of information can be made available only if there’s totally
integrated enterprise data warehouse.
What is data warehousing?
A data warehouse is a subject-oriented, integrated, non-volatile & time-variant
collection of data in support of management’s decisions.
Need for Data Warehousing:
• IT or business staff spending a lot of time developing special reports for decision-makers.
• Lots of PC-based or small server systems obtaining extracts of data incapable of presenting a
holistic view of the entire gamut of information.
• Same data present on different systems, in different department and users may be unaware of
this fact.
• Difficulty in getting meaningful information in a timely manner.
• Multiple systems giving different answer to the business questions.
• Less analysis by decision makers and policy planners due to non-availability of sophisticated
tools and easily decipherable, timely and comprehensive information
Purpose of Data Warehousing:
Better business intelligence for end users.
• Reduction in time to access and analyze information.
• Consolidation of disparate information sources.
• Replacement of older, less-responsive decision support systems
• Faster time to market for products and services
Data Warehouse Characteristics:
1. Subject-orientedàWH is organized around the major subjects of the enterprise rather
than the major application areas. This is reflected in the need to store decision-support
data rather than application-oriented data.
2. Integratedàbecause the source data come together from different enterprise-wide
applications systems. The source data is often inconsistent using..The integrated data
source must be made consistent to present a unified view of the data to the users.
3. Time-variantàthe source data in the WH is only accurate and valid at some point in
time or over some time interval. The time-variance of the data warehouse is also
shown in the extended time that the data is held, the implicit or explicit association of
time with all data, and the fact that the data represents a series of snapshots.
4. Non-volatileàdata is not update in real time but is refresh from OS on a regular basis.
New data is always added as a supplement to DB, rather than replacement. the DB
continually absorbs this new data, incrementally integrating it with previous data
DATA WAREHOUSE LIFE CYCLE:
Data warehousing is a concept. It is not a product that can be purchased off the shelf. It
is a set of hardware and software components integrated together which can be used to analyze
the massive amount of data stored in an efficient manner. It is a process through which one can
build a successful data warehouse. Following are the five steps towards building a successful
data warehouse.
1) JUSTIFICATION
2) REQUIREMENT ANALYSIS
3) DESIGN
4) DEVELOPMENT & IMPLEMENTATION
5) DEPLOYMENT
DATA WAREHOUSE ARCHITECTURE :
Operational
data source1
Meta-data
Operational
data source 2
Lightly
Manage
Reporting,
query,application
development, and
EIS(executive
information
system) tools
High
Query
summarized data
summarized
Load Manager
data
Operational
Detailed data
data source n
DBMS
OLAP(online
analytical
processing) tools
Operational
data store (ods)
Operational data store (ODS)
Data mining
Archive/backup
End-useraccess tools
data
Typical architecture
Warehouse Manager
of a data warehouse
Main Components:
•
Operational data sourcesàfor the DW is supplied from mainframe operational data held
in first generation hierarchical and network databases, departmental data held in
proprietary file systems, private data held on workstaions and private serves and external
systems such as the Internet, commercially available DB, or DB assoicated with and
organization’s suppliers or customers
•
Operational datastore(ODS)àis a repository of current and integrated operational data
used for analysis. It is often structured and supplied with data in the same way as the data
warehouse, but may in fact simply act as a staging area for data to be moved into the
warehouse
•
Load manageràalso called the frontend component, it performance all the operations
associated with the extraction and loading of data into the warehouse. These operations
include simple transformations of the data to prepare the data for entry into the warehouse
•
Warehouse manageràperforms all the operations associated with the management of the
data in the warehouse. The operations performed by this component include analysis of
data to ensure consistency, transformation and merging of source data, creation of indexes
and views, generation of denormalizations and aggregations, and archiving and backing-up
data
•
Query manageràalso called backend component, it performs all the operations
associated with the management of user queries. The operations performed by this
component include directing queries to the appropriate tables and scheduling the execution
of queries
•
End-user access toolsàcan be categorized into five main groups: data reporting and
query tools, application development tools, executive information system (EIS) tools,
online analytical processing (OLAP) tools, and data mining tools.
Tools and Technologies:
•
The critical steps in the construction of a data warehouse:
a. Extraction
•
b. Cleansing
c. Transformation
After the critical steps, loading the results into target system can be carried out either
by separate products, or by a single, categories:
•
Code generators
•
Database data replication tools
•
Dynamic transformation engine
Applications:
• Online Transaction Processing:
OLTP systems are the major kinds of enterprise applications:
Examples:
Order entry systems, Inventory control systems, Reservation systems, Point-of-sale
systems, Tracking systems, etc.
•
Executive information system (EIS) :
Present information at the highest level of summarization using corporate business
measures. They are designed for extreme ease-of-use and, in many cases, only a mouse
is required. Graphics are usually generously incorporated to provide at-a-glance
•
indications of performance
Decision Support Systems (DSS) :
They ideally present information in graphical and tabular form, providing the user
with the ability to drill down on selected information. Note the increased detail and
data manipulation options presented.
DATA MINING
What is data mining?
Data Mining refers to the process of analyzing the data from different perspectives and
summarizing it into useful information. Data mining software is one of the numbers of tools
used for analyzing data from many different dimensions or angles, categorize it, and
summarize the relationship identified.
Definition:
Data mining is the process of finding correlation or patterns among fields
in large
relational databases. “The process of extracting valid, previously unknown, comprehensible,
and actionable information from large databases and using it to make crucial business
decision”
Different Types of Data Mining: Business, Scientific and Internet Data Mining
Five major elements of Data Mining:
1. Extract, transform, & load transaction data on to the data warehouse system.
2. Store and manage data in multidimensional database system.
3. Provide access to business analysts and IT Professionals.
4. Analyze the data by application software.
5. Present the data in useful format such as graph or table..
Methods of Data Mining:
1. Classification
2.Regression
CONCLUSION
3.Clustering 4.Associative rules 5.Visualization
Data Warehousing provides the means to change the raw data into information for
making effective business decisions-the emphasis on information, not data. The Data
warehouse is the hub for decision support data.
Data mining is a useful tool with multiple algorithms that can be tuned for specific
tasks. It can benefit business, medicine, and science. It needs more efficient algorithms to
speed up data mining process.