HCLT Research Paper: Mainframe Performance Management
Custom Application Services: http://www.hcltech.com/custom-application-services/overviewThe mainframe story would be incomplete without its customers. Although some customers are on the look out to reduce dependency on mainframes by moving to systems powered by newer echnologies, many still continue to use them for historical reasons. Mainframe customers are big companies in data‐ intensive industries such as distribution, financial services, general business, and public sector that invested in mainframes several years ago
Content
Mainframe Performance Management
Can Make the Dinosaur Dance
AUTHORS
Srinivas Murthy Potharaju, ETS Mainframe CoE
HCL Technologies, Hyderabad
Dr Usha Thakur, ATS Technical Research
HCL Technologies, Chennai
Mainframe Performance Management: Can Make the Dinosaur Dance
© 2010, HCL Technologies Ltd.
October, 2010
Mainframe Performance Management
2
Contents
Introduction .................................................................................................................................................. 4
Purpose ......................................................................................................................................................... 4
Significance of Mainframes ........................................................................................................................... 4
Drivers for Mainframe Performance............................................................................................................. 5
Mainframe Performance Services from HCL................................................................................................. 6
HCL Approach ............................................................................................................................................ 7
Benefits ....................................................................................................................................................... 10
Endnotes ..................................................................................................................................................... 12
Mainframe Performance Management
3
Introduction
There was a time when IT folks could show value by focusing only on operational performance. Today,
however, the focus has shifted more to showing the extent to which IT products and services create
value and contribute to their organization’s business objectives. Businesses want to know how IT can
help in expanding their company’s customer base, increasing products and services portfolios,
increasing the profit margin, tapping into new markets, identifying activities for outsourcing, and much
more. The alignment of IT activities and deliverables to business goals gains even more significance in
the world of mainframes, given the high investments associated with them.
It is no longer enough to show mainframe performance metrics only in terms of operational efficiency;
we need metrics that can reflect the extent to which mainframes help their organization fulfil business
goals. That this is not just a fad is very clear from a study undertaken recently by Gartner and Financial
Executives International (FEI). According to that study a fairly high number of CIOs are already reporting
to CFOs, and the latter have a significant say in all IT investment matters. 1
Purpose
The audience of this paper are CIOs, CEOs, CFOs, and other executives whose business operations are
heavily dependent upon the performance of mainframes. It examines the main drivers for mainframe
performance management by organizations that have decided to continue using them. This paper also
highlights key features of the performance management service from HCL. 2
Significance of Mainframes
Throughout the history of the mainframe, IBM has retained its dominance, 3 though not without its
share of controversy. 4 It is estimated that IBM’s mainframe sales will reach $3.3 billion in 2010, which
would represent about three‐quarters of the overall mainframe market of $4.5 billion. 5
For sometime now, there has been a popular belief that mainframes are living on borrowed time and
may not be around in the near future. Writing in 2003, John R. Phelps of Gartner pointed out that a
number of developments over the past 15 years 6 led to stories about the death of the IBM mainframe,
but the reality has been to the contrary. Currently, a great deal of attention is being given to the
declining revenue from mainframes and servers in general, 7 and once again we are seeing stories about
the end of mainframes in the very near future. What is interesting, however, is that IBM has continued
to modernize and invest in a range of mainframes, improve pricing, and reach out to new customers,
especially in newly industrializing economies, 8 which indicates that IBM still has confidence in continued
business from mainframes. In fact, IBM is solidifying its position in niche accounts. For instance, the
Mainframe Performance Management
4
zEnterprise system (z196)‐‐latest mainframe model from IBM‐‐is targeted at the top 10% to 20% of its
customers where the volume of Millions of Instructions Per Second (MIPS) is very high. 9 MIPS is one of
the most important criterion that IBM uses for calculating the cost of mainframe license. 10
The mainframe story would be incomplete without its customers. Although some customers are on the
look out to reduce dependency on mainframes by moving to systems powered by newer technologies,
many still continue to use them for historical reasons. Mainframe customers are big companies in data‐
intensive industries such as distribution, financial services, general business, and public sector that
invested in mainframes several years ago. For these customers, reliability, security, and scalability are
top priorities. 11 There is a general consensus that any improvement in the end‐to‐end performance of
mainframes is a very important step in bringing down the cost of owning and running mainframes.
Drivers for Mainframe Performance
In general, the main business drivers for improving the performance of mainframes are as follows:
Reduce the Total Cost of Ownership (TCO) of mainframe servers
Increase productivity
Make the online applications available 24/7 to an increasing number of customers without any
delay 12
Improve end‐user experience e.g., comparing auto insurance quotes 13 and the like transactions
Increase customer retention 14
Eliminate SLA violation
Assuming that IT is a means for achieving the business objectives then it is imperative for IT
management and staff to be on the same page as the business management and staff in their company.
This will not only give all IT activities and decisions a business context but also reveal their significance
and impact on business.
The changing business environment today is compelling IT management to look at ways of obtaining
maximum performance from the core mainframe applications and of aligning applications directly to the
overall business objectives of their organization. The bulk of the mainframe applications were developed
years ago, which have since then undergone enhancements or patchwork. Consequently, they have lost
maintainability and traceability to business requirements.
Today, more than ever before, there is an ever changing workload being executed on the mainframe.
Starting with traditional batch and online workloads, there is an increasing number of transactions
Mainframe Performance Management
5
running on the mainframe which are invoked from the Web. Additionally, there are queries that are
used for complex reporting and analytics in Data Warehouse & Business Intelligence environments.
Poor mainframe performance may be caused by all or a combination of the following problems:
Environment, system and sub‐system parameters are not set correctly
Aging and monolithic applications are large and complex, and therefore difficult to integrate
with modern systems, which are distributed and agile
Database parameters are not tuned for optimal performance
Applications and databases are poorly designed
Application code and SQLs are not tuned for performance
Online applications may not be available due to the scheduling of jobs in critical path and end‐
of‐day batch jobs
File block size and buffer parameters are not set correctly, leading to prolonged and high usage
of CPU and I/O
The above‐mentioned technical problems need to be addressed by IT teams, given that if they are left
unattended they will most likely lead to the following undesirable consequences:
Higher licensing cost of mainframes resulting from a high consumption of MIPS by applications
Increasing customer dissatisfaction and decline in customer because of poor response time of
online applications
Lower productivity of day time employees caused by extended batch windows cutting into
regular business hours
SLA violations due to the inability of applications to scale up to increasing data volumes
Mainframe Performance Services from HCL
This is a comprehensive service from HCL. Using proven methodologies and frameworks, it identifies
poorly designed processes, programs, databases, and environment on the mainframe. On the basis of its
findings, HCL proposes a solution and remedial strategy. During implementation, HCL enhances the
configuration of the mainframe processes, programs, databases, and services such that the overall
response and execution time is faster than before. Customers who have availed of this service have
been able to reduce MIPS usage by their applications, lower TCO, and utilize the available processing
capacity for newer workloads.
Mainframe Performance Management
6
HCL Approach
As illustrated in Figure 1, HCL uses its proprietary framework for workload optimization to help
customers analyze their workload and identify areas of application tuning. This approach takes into
account inputs on the architecture of business applications and databases, the design of online and
batch components, and also usage/runtime statistics (from SMF & RMF, performance management
tools, history reports, and environment as well system/ subsystem parameters).
Figure 1: HCL Proprietary Framework
As highlighted below, the objective of the assessment approach adopted by HCL is to identify poorly
performing workloads on the mainframe so that appropriate recommendations could be made and a
remedial strategy developed.
PERFORMANCE OPTIMIZATION ASSESSMENT APPROACH
As shown in Figure 2, HCL first collects data on various parameters from the client systems and
subsystems. It analyzes that data and then proposes a solution, together with a remedial strategy.
Mainframe Performance Management
7
Analysis
Data
Collection
Analyze usage
and runtime
data from SMF
and RMF to
identify the
most expensive
workloads
Identify areas of
performance
improvement
using
performance
tools, history
reports, and
application
architecture
diagrams
Propose
Solutions
Make
recommendations to
optimize performance
of the workload.
Estimate cost, effort
and duration required
for implementing
proposed
recommendations
aimed at optimizing
performance.
Prepare Gain Vs Effort
matrix to prioritize
each recommendation.
Remediate
Construct high level designs
for each approved
recommendation.
Conduct baseline
performance of the code
(production version) in
controlled test region.
Carry out functional and
performance tests of
impacted components in
controlled test environment.
Compare results from
performance test,
implement agreed upon
recommendations, and
demonstrate benefits.
Provide warranty support
for impacted components.
Figure 2: Performance Optimization Execution Approach
HCL follows a systematic approach in identifying areas of improvement in mainframe systems and
processes. Here is a glimpse of that approach.
In batch jobs, HCL
Examines the critical path in the batch cycle in order to identify undue waiting times
Identifies the most expensive batch jobs, programs, utilities, and recurring job failures
Analyzes performance data reports for selected jobs
In online transactions, HCL
Identifies the most expensive online transactions and recurring failure events
Analyzes performance data reports for selected transactions
Mainframe Performance Management
8
For identifying design and architecture problems, HCL
Understands the pain points in the application architecture
Locates the precise problems in the design of databases and external interfaces
Analyzes performance data reports for selected jobs and transactions
For identifying specific problems in the database, HCL
Examines contentious issues
Considers or reconsider data compression
Considers if appropriate Indexes are being used by the query
Takes into account the possibility of reorganizing table spaces
Explores if partitioning and segmentation of table spaces would be appropriate
For isolating problems at the system and subsystem levels, HCL
Analyzes performance data reports for z/OS, CICS, DB2 and IMS systems
Analyzes all system and subsystem parameters
Examines Buffer Pool, Sort Pool, EDM Pool, and DB2 Catalog
PERFORMANCE MANAGEMENT BEST PRACTICES
HCL uses a set of performance management best practices that enable it to:
Identify early signs of performance problems by using performance tools and system
related performance reports.
Automate real‐time problem detection and resolution by putting in place performance
indicators and thresholds.
Prevent the occurrence of problems by using the SMF and RMF data reports to allocate
workload resources based on performance. Historically, the stellar reputation earned by
mainframes (for executing high performance operations continuously) has come, in part,
from the fact that engineers have been able to allocate system resources proactively. This
proactive allocation was not a problem when mainframe systems were siloed, software
changes were relatively infrequent, and workloads were predictable.
Mainframe Performance Management
9
Benefits
The overall business benefits that clients have enjoyed from an improvement in the performance of
mainframes are lower TCO and increased productivity. However, as illustrated in Table 1 and Table 2,
there are other specific advantages as well.
US‐based Auto & Home Insurance Company
Area of Engagement
Solution
Customer Benefits
Performance Optimization of
applications in scope
SQL query tuning ‐ overall
60% gain in performance
Business Objectives
Response time
improvement ‐ overall
73% faster views
Improve the performance of the
product in production and
recommend improvements in
consumption, response time,
database improvements,
architecture, WLM, etc before its
Phase 2 development.
New indexes to improve
performance
Technology
Mainframe Technology (REXX,
DB2 V7, ISPF Services, JCL,
OMEGAMON for DB2 & other
technologies)
Engagement Model/Size
Onsite (30%) – Offshore (70%)
Reduced TCO
Enhanced cost
effectiveness
Reduced technology
risk
Data model redesign
improvements
Optimized business
functions and their
mapping to system
landscape
WLM goal mode policy
recommendations ‐
performance gain of
around 20%
Improved global
visibility of
applications and
projects
All small jobs now run
under a minute as
compared to 5 – 60
minutes earlier
Reduced time to take
decisions for new
initiatives
Project Duration
4 months
Table 1: Performance Optimization – US Client
Mainframe Performance Management
10
German Financial Services Provider with Global Operations
Area of Engagement
Performance Optimization of
applications in scope
Solution
Customer Benefits
Expected Optimization
Business Objectives
Provide performance
optimization recommendations
for applications, SQL tuning,
related sub system and related
areas
Technology
Mainframe Technology (COBOL,
DB2, CICS, JCL, DB2 PM,
OMEGAMON for DB2 & other
technologies)
Addressed CPU
consumption – ~ 30,000
CPU Hours
Expected optimization for
HCLTs recommendations
– Average ~ 15%
Optimization Recommendations
SQL query tuning (100+
recommendations)
New and modified
Indexes to improve
performance
Engagement Model/Size
Onsite (10%) – Offshore (90%)
Project Duration
4 months
Application code
performance
improvements
Architecture &
application redesign
Reduced TCO
Easily manageable
Define a formal IT
governance
Enhanced cost
effectiveness
Reduced Technology
risk
Optimized business
functions and their
mapping to system
landscape
Improved global
visibility of
applications and
projects
Reduced time to take
decisions for new
initiatives
Table 2: Performance Optimization – German Client
Mainframe Performance Management
11
Endnotes
1
One of the 50 questions that the research team asked 482 senior financial executives was: who did IT report to in
their organization? The response was as follows: 42% reported to the CFO, 33% to the CEO, 16% to the COO, 2% to
a chief administrative officer and 7% to other officers. For full details of the survey results and analysis, see John E.
Van Decker, “2010 Gartner FEI Technology Study: The CFO as Technology Influencer" (9 April 2010 ID Number:
G00175773) p. 2
http://my.gartner.com/portal/server.pt?open=512&objID=260&mode=2&PageID=3460702&id=1338722&ref=clie
ntFriendlyUrl [October 2010]‐>represents when an URL was accessed.
2
Note: a discussion on the pros and cons of mainframe lies outside the scope of this paper.
3
IBM’s share of the market for mainframe is believed to be around 90%. Other mainframe vendors include
Computer Associates, BEA, Microsoft, Hewlett‐Packard, Fujitsu Siemens, Unisys, Sun, and Micro Focus.
4
The latest being antitrust complaints by TurboHercules, a French maker of open‐source software for mainframe
computers, and T3 Technologies, an American distributor of Flex software that runs mainframes files. On the basis
of those complaints, the European Commission has opened an investigation against IBM. For details, see Kevin J.
O'Brien, "Europe to Investigate Antitrust Complaints over I.B.M. Mainframes" (July 26, 2010)
http://www.nytimes.com/2010/07/27/business/27blue.html [October 2010]. See also "Antitrust: Commission
Initiates Formal Investigations against IBM in Two Cases of Suspected Abuse of Dominant Market Position" (July
26, 2010; Ref.: IP/10/1006)
http://europa.eu/rapid/pressReleasesAction.do?reference=IP/10/1006&format=HTML&aged=0&language=EN&gu
iLanguage=en [October 2010]. See also "NEON Files Motion for Partial Summary Judgment in NEON v. IBM"
(September 15, 2010) http://www.marketwatch.com/story/neon‐files‐motion‐for‐partial‐summary‐judgment‐in‐
neon‐v‐ibm‐2010‐09‐15?reflink=MW_news_stmp [October 2010]. Similar concerns have been expressed in a
comprehensive study by the Indian Council for Research and International Economic Relations. For details, see
Rajat Kathuria, et al., "The Issues of Competition in Mainframe and Associated Services in India" (March 11, 2010)
http://openmainframe.org/featured‐articles/the‐issues‐of‐competition‐in‐mainframe‐and‐associated‐servic.html
[October 2010].
5
Kevin J. O'Brien, Ibid. According to Toni Sacconaghi of Bernstein Research, nearly 40% of IBM’s profits are
mainframe‐related. Cited in “The Return of the mainframe Back in Fashion” (January 14, 2010)
http://www.economist.com/node/15276714?story_id=15276714&source=hptextfeature [October 2010].
6
He is referring in particular to the movement toward client/server platforms and complementary metal‐oxide
semiconductor (CMOS) technology combined with the availability, reliability, security, scalability, and
manageability of mainframes, the growth of the Internet and e‐business, and the Y2K problem, all of which led
many to believe that the mainframe was living on borrowed time. For details, see John R. Phelps, “The Future of
Mainframes Looks Surprisingly Good,” (August/September 2003 issue of zJournal, now a part of mainframezone)
http://www.mainframezone.com/it‐management/gartner‐cio‐update‐future‐of‐theibm‐mainframe‐the‐looks‐
surprisingly‐good [October 2010].
Mainframe Performance Management
12
7
According to IDC, the worldwide server revenue in 2009 was said to have declined 18.9% to $43.2 billion when
compared to 2008, and worldwide unit shipments declined 18.6% to 6.6 million units over the same period. For
details, see IDC, "Worldwide Server Market Rebounds Sharply in Fourth Quarter as Demand for Blades and x86
Systems Leads the Way" (24 Feb 2010) http://www.idc.com/getdoc.jsp?containerId=prUS22224510 [October
2010]. According to one estimate, IBM’s mainframe revenue took a plunge of 39% in the second quarter of 2009.
See Steve Hamm, "IBM Defends Its Big Iron" (August 4, 2009)
http://www.businessweek.com/technology/content/aug2009/tc2009084_001429.htm [October 2010].
8
There are several articles on this subjects. See, for instance, Zacks Investment Research, "IBM’s Mainframe Wins
Customer" (June 30, 2010)
http://www.zacks.com/stock/news/36260/Zacks+Analyst+Blog+Highlights%3A+Union+Pacific%2C+IBM%2C+Hewle
tt+Packard%2C+Vornado+Realty+and+KKR+Financial [October 2010]. See also Robert L. Mitchell, ""Morphing the
mainframe" (February 6, 2006) http://features.techworld.com/operating‐systems/2229/morphing‐the‐mainframe/
[October 2010].
9
The zEnterprise systems is said to start at about $1 million and according to Lou Miscioscia, an analyst with Collins
Stewart, they are expected to generate a profit margin of about 70 percent, vs. a 46 percent margin for the
company as a whole. Cited in By Katie Hoffman, "IBM Mainframes: Boring but Profitable" (July 22, 2010)
http://www.businessweek.com/magazine/content/10_31/b4189041885991.htm [October 2010]. See also IBM,
"IBM zEnterprise System," http://www‐03.ibm.com/systems/z/hardware/zenterprise/ and also "Success Stories,"
http://www‐03.ibm.com/systems/z/success/index.html [October 2010]. According to Martin Kennedy, the
managing director of Citigroup's enterprise systems infrastructure, "we will be able to collapse multiple existing
large systems into the new water‐cooled z196, which is a huge step in the company's attempt to shift more of its IT
dollars away from internal operations and maintenance and toward customer‐facing efforts. For more details, see
Bob Evans, "Global CIO: IBM's Blazing New Mainframe Wins Raves from Citigroup," (September 2, 2010)
http://www.informationweek.com/news/global‐
cio/interviews/showArticle.jhtml?articleID=227200199&pgno=2&queryText=&isPrev= [October 2010]. See also
John R. Phelps and Mike Chuba, "IBM Adds Integrated Management Role for Mainframe," July 27, 2010; ID:
G00205553)
http://my.gartner.com/portal/server.pt?open=512&objID=260&mode=2&PageID=3460702&resId=1411013&ref=
QuickSearch&sthkw=mainframe [October 2010].
10
MIPS reflect the mainframe’s footprint, and over the past decade it has witnessed a steady rise with installed
capacity reaching a high of 11.1 million in 2007. For details, see Timothy Prickett Morgan, "The IBM Mainframe
Base: Alive and Kicking" (July 10, 2007) http://www.itjungle.com/big/big071007‐story01.html [October 2007].
11
For a quick overview on this aspect, see IBM, "The IBM Enterprise System Advantage ‐ Running the World's Most
Sophisticated Business Transactions," http://www‐03.ibm.com/systems/migratetoibm/getthefacts/market.html
[October 2010]. See also Anne Rawland Gabriel, "Debate over the Future of Mainframe Computing Rages On"
(September 16, 2008) http://www.wallstreetandtech.com/it‐infrastructure/210601591?pgno=1 [October 2010].
12
For instance, in the future banks need to accommodate for the increasing number of customers accessing
mobile banking application. According to a study by Berg Insight, "the number of active users of mobile banking
and related financial services worldwide is forecasted to increase from 20 million in 2008 to 913 million in 2014."
Mainframe Performance Management
13
Berg Insight, "Mobile Banking and Payments," p. 3, http://www.berginsight.com/ReportPDF/ProductSheet/bi‐
mbp‐ps.pdf [September 2010]. Anticipating such high volume, IBM has developed a 5.2GHz chip ‐ considered to be
the highest speed rating to date ‐ for its fastest mainframe computers. See Brooke Crothers, "IBM Ships 5.2GHz
Chip, Its Fastest Yet" September 1, 2010) http://news.cnet.com/8301‐13924_3‐20015297‐64.html#ixzz10uOqdeX8
[September 2010].
13
Ryan Arsenault’s observation captures the views of many experts thus: “[t]he problem is not the mainframe
platform, but the fact that many insurance companies are making use of legacy mainframe application code which
was originally designed to provide quotations to real people.” What is needed is an improvement in the
performance of hardware and software applications that use computer resources for running a transaction such as
generating insurance quotes. For details, see Ryan Arsenault, "Mainframes Run into Performance Problems with
Online Insurance Comparison Shopping," http://itknowledgeexchange.techtarget.com/mainframe‐
blog/mainframes‐run‐into‐performance‐problems‐with‐online‐insurance‐comparison‐shopping/ [September 2010.
14
Improving performance of business applications is a key to delivering business value to customers and their end‐
users. A recent survey by the Aberdeen Group found that companies that monitored and measured application
performance at the point of consumption by end users had few end user complaints. For full details of the survey,
see Jeffrey Hill, “End‐User Experience Monitoring and Management” (August 2010)
http://v1.aberdeen.com/launch/report/benchmark/6778‐RA‐end‐user‐experience‐monitoring.asp [October 2010].
Writing about the excessive significance given to operational performance metrics in the post 1990s era, Michael
Bitterman noted that "IT continues to focus on operations and...explain ‘how the clock works’ when users are
asking, ‘what time is it?’ The availability of tools for monitoring hardware and network performance [has given] IT
the ability to overwhelm users with statistics regarding IT performance. Users [become] baffled by the metrics that
show that mainframe availability was 99.99% yet they couldn’t enter data or access systems.” For the complete
report, see Michael Bitterman "IT Metrics for the Information Age" (July 25, 2004) http://www.performance‐
measurement.net/news‐detail.asp?nID=198 [October 2010].
Mainframe Performance Management
14
