Database 1
Content
Oracle Database Monitors and Tools
White Paper: Oracle Database Administration
February 2012
Oracle Database Monitors and Tools
February 2012
TABLE OF CONTENTS
OVERVIEW ......................................................................................................................... 3 SCHEDULING THE DATABASE SYSTEM CHECK ........................................................... 4 CONFIGURING THE DATABASE SYSTEM CHECK ......................................................... 5 VIEWING THE DATABASE SYSTEM CHECK ................................................................... 7 USING THE ALERT MONITOR .......................................................................................... 9
Database Alerts ........................................................................................................................... 14
Space Management......................................................................................................................................... 14 Performance .................................................................................................................................................... 14 Backup and Restore ........................................................................................................................................ 15 SAP Consistency ............................................................................................................................................. 15 Health............................................................................................................................................................... 16
CASE STUDY: ADDING A NEW CONDITION TO THE DATABASE SYSTEM CHECK 23 BRCONNECT .................................................................................................................... 27
Default Conditions for Database Administration ...................................................................... 27 Default Conditions for Database Operations ............................................................................ 29 Critical Database Messages in the Oracle Alert File ................................................................ 29 BRCONNECT Default Conditions for Database Profile Parameters ........................................ 30
ADDITIONAL DOCUMENTATION .................................................................................... 31
SAP Library ................................................................................................................................. 31 SAP Notes ................................................................................................................................... 31
2
Oracle Database Monitors and Tools
February 2012
OVERVIEW
To help you quickly recognize and analyze Oracle database problems, SAP provides comprehensive and fully integrated database system checking. By running regular checks and looking at the results, you can rapidly identify potential problems and take the required action before the database crashes. The following graphic shows how the SAP system supports you in monitoring your Oracle database system:
1 Schedule DBA Planning Calendar DB13
2 Run daily Configure Database System Check DB17
5 BRCONNECT -f check
Alerts
3 Display alert messages DB16
Alerts – mainly health Other alerts Non-BRCONNECT Methods 4 Alert Monitor RZ20
In this paper we discuss the main functions in the figure above: 1. You schedule the database system check to run daily using the Database Planning Calendar [page 4], transactions DBCOCKPIT and DB13, in the Computing Center Management System (CCMS). 2. If required, you can configure the database system check [page 5] using transactions DBCOCKPIT and DB17. 3. When BRCONNECT has run, you can view the results [page 7] using transactions DBACOCKPIT and DB16. 4. You can also view the results of the BRCONNECT and other checks using the database alert monitor [page 9] in transaction RZ20. Transactions DBACOCKPIT and DB13 deliver the results immediately to the alert monitor. 5. BRCONNECT [page 27] is the program that performs a series of database system checks, including those for database administration, database operations, critical database messages in the Oracle alert file, and incorrectly set database parameters. It is fully integrated with the alert monitor. Using a case study [page 23], we show you how to add a new condition to the database system check and view this in the alert monitor. If you want to find more detailed information, look at the additional documentation [page 31] provided by SAP.
3
Oracle Database Monitors and Tools
February 2012
SCHEDULING THE DATABASE SYSTEM CHECK
You can schedule the Oracle database system check in the Database planning Calendar in transaction DBACOCKPITof the Computing Center Management System (CCMS).
We strongly recommend you to schedule the check daily. Otherwise, you might miss important warnings and errors, possibly resulting in expensive system downtime. The best way to do this is by selecting an action pattern, as shown in the following graphic:
Every action pattern includes a daily database system check. If required, you can run a database system check immediately using the DBA Planning Calendar. You can also run the database system check from the command line of your operating system using brconnect –u / -c –f check.
For more information on scheduling the database system check, see DBA Planning Calendar (Oracle) in the documentation CCMS Oracle [page 31].
4
Oracle Database Monitors and Tools
February 2012
CONFIGURING THE DATABASE SYSTEM CHECK
You can configure the Oracle database system check in the Computing Center Management System (CCMS) to: Add new conditions of type ORA or PROF Exclude individual conditions from the check Specify threshold values for the conditions Create object-specific conditions to exclude them from the check Create object-specific conditions to set individual threshold values Specify corresponding corrective actions Maintain the condition description For an example of how to add a new condition of type ORA, see the case study [page 23]. The configuration data is stored in the DBCHECKORA table. Changed or new conditions take effect the next time that BRCONNECT runs. You can view the results in transaction DBACOCKPIT or DB16 [page 7] or the alert monitor [page 9]. To change the configuration, you use transaction DBACOCKPIT or DB17, as shown in the following graphic:
For example, if you want to change the TABLESPACE FULL parameter above, you can do so as follows:
5
Oracle Database Monitors and Tools
February 2012
In the above example, we have changed the threshold at which an alert is raised for a tablespace to 95%. You might also decide to change the Condition Level from Error to Warning, deactivate the check by setting the Active Flag to No, or change the text of the Description. In the following example, we have changed the threshold for a single tablespace, the SYSTEM tablespace, from 95% to 80%, by specifying SYSTEM in the Check Object field. To do this, you first need to copy the condition and then specify Check Object for the new condition.
For more information on configuring the database system check, including changing or creating check conditions, see the following path in the documentation CCMS: Oracle [page 31]: Database System Check Configuring Database System Check (Oracle)
6
Oracle Database Monitors and Tools
February 2012
VIEWING THE DATABASE SYSTEM CHECK
To check the results of a database check you use transaction DBACOCKPIT or DB16:
At the top of the screen, the system displays: Check Results: the number of checks that have finished with an alert. An alert is an error, a warning, or an exception (that is, a deviation from the SAP standard value). Settings: the period of results displayed, the refresh interval, and the deletion period You can configure the display to set the: Number of days for which alerts are displayed Refresh interval after which the display is updated Deletion period after which old alerts are deleted You can also drill down to look at the details of an individual check or Message.
For example, to look in detail at the line TABLESPACE_FULL above:
7
Oracle Database Monitors and Tools
February 2012
You can choose Check log to view the log produced by this check. An example of the log is shown in the case study [page 23]. For more information on viewing the database system check, see the following path in the documentation CCMS: Oracle [page 31]: Database System Check Displaying Alert Messages from Database System Check
8
Oracle Database Monitors and Tools
February 2012
USING THE ALERT MONITOR
You use the alert monitor in the Computing Center Management System (CCMS) to check the following Oracle database functions: Space management – tablespaces and segments Performance – optimizer statistics, buffers, logs, and checkpoints Backup or restore – database and redo log backup Consistency – between database objects in the ABAP and Oracle dictionaries Health – database system checks from BRCONNECT By configuring data collection tools, or methods, to run periodically, alerts are automatically updated and fed to the monitoring architecture. The main tool feeding alerts is BRCONNECT [page 27], especially for Health alerts. Analysis tools provide additional information about the alert conditions, and you can configure autoreact tools to automatically respond when an alert occurs. The alerts are hierarchically arranged and grouped. This is what you see when you open the alert database monitor tree in transaction RZ20:
9
Oracle Database Monitors and Tools
February 2012
If you choose Space Management Tablespaces and open some of the nodes, you see the following:
10
Oracle Database Monitors and Tools
February 2012
This level displays individual database checks, some derived from BRCONNECT, some not. Color-coding and the message text indicate whether there is an alert. If an alert has occurred, the message text gives more information. You can look at the details of an alert, such as Space management Segments Most allocated extents in any segment:
11
Oracle Database Monitors and Tools
February 2012
You can also look at the properties and methods of a node, which determine how the alert is displayed, as follows:
12
Oracle Database Monitors and Tools
February 2012
You can alter data collection methods and other attributes such as the threshold values in the monitor.
13
Oracle Database Monitors and Tools
February 2012
Database Alerts
This section lists the individual database alerts in the alert monitor by branch of the monitoring tree. The tables below include all possible alerts. You can tailor the tree to suit your own requirements by adding, changing, or deleting alerts.
Space Management
Type Tablespace Description Freespace for each tablespace Used space for each tablespace Status (online or offline) for each tablespace Segments Segments with too few allocatable extents * Fewest allocatable extents for a segment * Segments approaching MAX_EXTENTS * Fewest extents left before MAX_EXTENTS * Most allocated extents in any segment * Segments with non-zero PCTINCREASE * Segments in wrong tablespace Rollback segment extension failed * These alerts are only relevant for locally managed tablespaces.
Performance
Type Optimizer Description Last successful update statistics Last brconnect -f stats run Harmful statistics Missing Statistics Buffers Buffer cache Library buffer Redo log buffer
14
Oracle Database Monitors and Tools
February 2012
Locks
Age of oldest exclusive transaction lock Deadlocked resource
Checkpoints
Checkpoint not complete
Backup and Restore
Type Archiving Description Last successful archive log backup Last brarchive run Archiver destination full Archiving off Backup Status Last successful complete database backup Last brbackup run Tablespace in backup mode
SAP Consistency
Type SAP consistency check Objects missing in the database Description Last SAP consistency check run
Primary indexes Secondary indexes Tables Views
Unknown objects in ABAP Dictionary
Database tables Database indexes Database views Database tables without unique index
Inconsistent objects
Primary indexes Secondary indexes
15
Oracle Database Monitors and Tools
February 2012
Tables Views Other checks Primary indexes Secondary indexes Tables Views Primary index not unique Optional indexes Too many indexes created Indexes not created
Health
Type Database health check Description Last successful database check Last brconnect -f check run Last evaluation of check results Start of database check Database administration Archiver destination full Control file not mirrored Missing control file Critical tablespace Critical autoextendable file Critical segment Data file mismatch Missing data file Database file offline File system full Harmful statistics
16
Oracle Database Monitors and Tools
February 2012
Invalid file type Segment in wrong tablespace Missing index Missing statistics Database in NOARCHIVELOG mode Segment with non-zero PCTINCREASE Redo log file not mirrored Missing redo log file Tablespace full Tablespace in backup mode Tablespace offline Segment with too many extents DBA operations Last successful archive log backup too old Last successful complete database backup too old Last archive log backup failed Last complete database backup failed Last operation failed Last update statistics failed Last successful operation too old Last successful update statistics too old
17
Oracle Database Monitors and Tools
February 2012
Type Oracle messages
Description ORA-00060 Deadlock while waiting for resource ORA-00255 Error archiving log file
ORA-00257 Archiver error ORA-00270 Error creating archive log
ORA-00272 Error writing archive log ORA-00376 File cannot be read ORA-00447 ORA-00470 ORA-00471 ORA-00472 ORA-00473 ORA-00474 Fatal error in background process LGWR process terminated with error DBWR process terminated with error PMON process terminated with error ARCH process terminated with error SMON process terminated with error
ORA-00600 Oracle internal error ORA-00603 Oracle server session terminated by fatal error
ORA-01114 I/O error writing database file ORA-01115 I/O error reading database file ORA-01122 File verification check failed ORA-01149 Cannot shut down – data file in backup mode ORA-01555 Snapshot too old ORA-01562 Failed to extend rollback segment ORA-01578 Database block corrupted ORA-01628 Max. extents reached for rollback segment ORA-01629 ORA-01630 tablespace ORA-01631 Max. extents reached saving undo for tablespace Max. extents reached in temp segment in
Max. extents reached in table
18
Oracle Database Monitors and Tools
February 2012
ORA-01632 Max. extents reached in index ORA-01650 ORA-01651 tablespace ORA-01652 ORA-01157 ORA-01654 ORA-01655 Unable to extend rollback segment in tablespace Unable to extend save undo segment for
Unable to extend temp segment in tablespace Unable to extend table in tablespace Unable to extend index in tablespace Unable to extend cluster in tablespace
ORA-01656 Max. extents reached in cluster ORA-01680 ORA-01681 tablespace ORA-01683 ORA-01684 ORA-01685 ORA-01688 ORA-01691 ORA-01692 tablespace ORA-01693 ORA-01694 ORA-07445 ORA-16014 ORA-16038 ORA-19502 ORA-19504 ORA-19510 ORA-27044 Unable to extend LOB segment in tablespace Max. extents reached in LOB segment in
Unable to extend index partition Max. extents reached in table partition Max. extents reached in index partition Unable to extend table partition Unable to extend lob segment in tablespace Unable to extend lob segment partition in
Max. extents reached in lob segment Max. extents reached in lob segment partition Exception encountered: core dump Destination not available for archiving log file Log file cannot be archived Write error on log file Failed to create log file Failed to set size of blocks for file Unable to write the header block of file
19
Oracle Database Monitors and Tools
February 2012
ORA-27072 ORA-30036
File I/O error Unable to extend segment in undo tablespace
Checkpoint not complete – Cannot switch to the next redo log file due to pending checkpoint Corrupt block – Database block is corrupted Profile parameters – Oracle 10g compatible control_file_record_keep_time cursor_space_for_time db_block_checksum db_block_size db_files db_file_multiblock_read_count filesystemio_options log_archive_start log_buffer log_checkpoints_to_alert max_dump_file_size open_cursors optimizer_mode parallel_execution_message_size parallel_threads_per_cpu recyclebin remote_os_authent replication_dependency_tracking shared_pool_size star_transformation_enabled statistics_level
20
Oracle Database Monitors and Tools
February 2012
timed_statistics trace_enabled undo_management undo_tablespace Profile parameters – Oracle 11g background_dump_dest commit_logging commit_wait commit_write compatible control_file_record_keep_time core_dump_dest db_block_size db_files db_file_multiblock_read_count db_writer_processes filesystemio_options log_archive_start log_buffer log_checkpoints_to_alert max_dump_file_size nls_length_semantics open_cursors optimizer_dynamic_sampling optimizer_features_enable optimizer_index_caching optimizer_index_cost_adj
21
Oracle Database Monitors and Tools
February 2012
optimizer_mode parallel_execution_message_size parallel_threads_per_CPU query_rewrite_eabled recyclebin remote_os_authent replication_dependency_tracking shared_pool_size star_transformation_enabled statistics_level timed_statistics trace_enabled undo_management undo_tablespace user_dump_dest
22
Oracle Database Monitors and Tools
February 2012
CASE STUDY: ADDING A NEW CONDITION TO THE DATABASE SYSTEM CHECK
This section shows how you can add a new condition to the database system check and then display the new check. The new condition is a database message – type ORA – but the principle is the same if you want to add a new condition of type PROF. 1. You enter the new condition in transaction DBACOCKPIT or DB17:
The Condition Name is 01157, which generates an error with the Cannot identify / lock data file. The recommendation for this error is to Check the Oracle alert log / trace file. 2. Using transaction RZ20 to call up the alert monitor [page 9], you delete and then rebuild the monitoring tree on the host running the SAP central instance: a) Activate maintenance functions in View: Current system status. b) Reset all alerts in the Oracle monitoring tree. c) Delete the Oracle monitoring tree. d) Run ABAP program RSDBMON0 on the main application server to rebuild the monitoring tree. 3. You call up the alert monitor and see the new check condition, ORA-01157:
23
Oracle Database Monitors and Tools
February 2012
4. You run a database system check in the DBA Planning Calendar [page 4], either immediately or at the next scheduled time. In this example, we assume that the new database check causes an alert. 5. You view the results of the database system check [page 7] using transaction DBACOCKPIT or DB16 to see the alert raised by the new condition:
24
Oracle Database Monitors and Tools
February 2012
The error 01157 appears in the Name column 6. You look at the detail log. a) You can see the new condition ORA-01157:
b) You can also see the alert ORA-1157 that was raised in this example:
25
Oracle Database Monitors and Tools
February 2012
7. You can also see the alert ORA-01157 displayed in the alert monitor, transaction RZ20 [page 9]:
26
Oracle Database Monitors and Tools
February 2012
BRCONNECT
BRCONNECT checks the following types of condition: Database administration – configuration, space management, database state, consistency Database operations – backup and archive results, failed operations Critical database messages in the Oracle alert file – for example, ORA-00600 Database profile parameters in the Oracle initialization file When it finds a critical situation, BRCONNECT writes an alert message to the detail log and to the table DBMSGORA. You can view the results with transaction DBACOCKPIT or DB16 [page 7]. Many alerts are also passed to the alert monitor [page 9]. You can tailor the conditions to suit your own requirements [page 5]. For critical database messages or database profile parameters, you can also add new conditions.
Default Conditions for Database Administration
Condition NOARCHIVELOG_MODE Severity Error Description Checks whether the database is in NOARCHIVELOG mode, which is not allowed for production databases. Checks the highest fill level of the archiving directory (by default, oraarch). Checks the fill level of file systems on the database host. Checks whether there are tablespaces that are offline. Checks whether there are tablespaces that have the BACKUP status although BRBACKUP is not active Checks the fill level of tablespaces in the database.
ARCHIVER_STUCK
Warning
FILE_SYSTEM_FULL
Warning
TABLESPACE_OFFLINE
Error
TABLESPACE_IN_BACKUP
Warning
TABLESPACE_FULL
Warning
DATA_FILE_MISSING
Warning
Checks whether there are data files that no longer exist in the file system. Checks whether there are online redo log files that no longer exist in the file system. Checks whether there are control files that no longer exist in the file system. Checks whether there are data files that are flagged as MISSING in Oracle control file. Checks whether there are database files that have an illegal operating system type, for example, block raw files on Unix or compressed files on Windows.
REDOLOG_FILE_MISSING
Error
CONTROL_FILE_MISSING
Error
DATA_FILE_MISMATCH
Error
INVALID_FILE_TYPE
Error
27
Oracle Database Monitors and Tools
February 2012
Condition REDOLOG_FILE_MIRROR
Severity Error
Description Checks whether there are online redo log files that are not mirrored on the Oracle side. Checks whether there are control files that are not mirrored on the Oracle side. Checks whether there are data files or online redo log files that are OFFLINE. Examines the data files with an activated auto extend feature to see whether the file system can be brought to overflow, due to the existing parameter setting (NEXT and MAXSIZE), during the automatic file extension. Checks whether there are tables or indices, for which the number of allocated extents exceeds the specified threshold value. Checks whether there are tables or indexes that can bring the tablespace to overflow when up to 5 next extents are allocated. Examines the data of a tablespace to see whether the file system can be brought to overflow, due to the existing parameter setting (NEXT and MAXSIZE), during the automatic file extension. Checks whether there are tables that not in a table tablespace or indices, which are not in an index tablespace. Checks whether there are tables that do not have any indices and are not specified in the DBDIFF table exception. Checks whether there are tables or indices that do not have any statistics, although they should have these. Checks whether there are tables or indices that have statistics, although they should not have these (for example, pool and cluster tables). Checks whether there are tables or indexes for which the PCTINCREASE storage parameter is not equal to zero. This can lead to storage fragmentation and is not suitable for the SAP System.
CONTROL_FILE_MIRROR
Error
FILE_OFFLINE
Error
CRITICAL_FILE
Warning
TOO_MANY_EXTENTS *
Warning
CRITICAL_SEGMENT *
Warning
CRITICAL_TABLESPACE
Warning
IN_WRONG_TABLESPACE
Error
MISSING_INDEX
Error
MISSING_STATISTICS
Error
HARMFUL_STATISTICS
Error
PCTINCREASE_NOT_ZERO *
Error
* These conditions are only relevant for locally managed tablespaces.
28
Oracle Database Monitors and Tools
February 2012
The above check conditions are specified in the control table DBCHECKORA.
You cannot add new check conditions but you can change some of the above checks as follows: You can exclude objects – that is, individual tables and indexes or even complete tablespaces – from certain checks that run at table or index level. You do this by creating a new check condition with the relevant object in transaction DBCOCKPIT or DB17 [page 5], setting its ACTIVE FLAG to NO or by specifying the object in the check_exclude BRCONNECT profile parameter.
You can change the threshold values for individual tables and indexes You can change the following check conditions in this way: TOO_MANY_EXTENTS, CRITICAL_SEGMENT, IN_WRONG_TABLESPACE, MISSING_INDEX, MISSING_STATISTICS, HARMFUL_STATISTICS, PCTINCREASE_NOT_ZERO
Default Conditions for Database Operations
Condition LAST_ARCHIVE_FAILED Severity Warning Description Checks whether the last backup of the offline redo log files with BRARCHIVE failed. Checks whether the last complete backup of the database with BRBACKUP failed. Checks whether the last update of the optimizer statistics with BRCONNECT failed. Checks whether the last DBA operation failed. Checks whether the last successful backup of the offline redo log files with BRARCHIVE is too old. Checks whether the last successful complete backup of the database with BRBACKUP is too old. Checks whether the last successful update of the optimizer statistics with BRCONNECT is too old. Checks whether the last successful DBA operation is too old.
LAST_BACKUP_FAILED
Warning
LAST_STATS_FAILED
Warning
LAST_OPERATION_FAILED ARCHIVE_TOO_OLD
Warning Warning
BACKUP_TOO_OLD
Warning
STATS_TOO_OLD
Warning
OPERATION_TOO_OLD
Warning
Since the test conditions for database operations are programmed in a specific way in BRCONNECT (known as built-in test conditions), no new check conditions can be added to the DBCHECKORA table. However, this is generally not necessary because other operations can be monitored by the LAST_OPERATION_FAILED and OPERATION_TOO_OLD check conditions, or by specifying function IDs in the PARAM field.
Critical Database Messages in the Oracle Alert File
See “Oracle Messages” in the table “Health” [Page 16].
29
Oracle Database Monitors and Tools
February 2012
You can enter any Oracle error code or error text as a condition name for this condition type. BRCONNECT searches the Oracle Alert log for corresponding Oracle error messages and might then generate alert messages.
BRCONNECT Default Conditions for Database Profile Parameters
These test conditions check the values of Oracle parameters. The standard test conditions for the database profile correspond to the current SAP recommendations described in SAP Note 830576 for Oracle 10g and in SAP Note 1431798.
You can easily adjust the test conditions for the database profile parameters, depending on the changed recommendations and for new Oracle releases.
30
Oracle Database Monitors and Tools
February 2012
ADDITIONAL DOCUMENTATION
SAP Library
You can find more information on Oracle database administration and the contents of this document in the SAP Library as follows:
All paths refer to SAP NetWeaver 7.3. 1. Call up the SAP Help Portal at help.sap.com/nw73 Application Help SAP Library: English. 2. Choose SAP NetWeaver Library: Function-Oriented View Database Administration Database Administration for Oracle. 3. Choose one of the following: o SAP Database Guide: Oracle o CCMS: Oracle You can also find these plus selected extracts from the SAP Library at: www.sdn.sap.com/irj/sdn/ora SAP on Oracle Knowledge Center SAP Documentation in Help Portal
SAP Notes
You can find more information on the contents of this document in the following SAP Notes: 483856 Description of the Alerts for Oracle Database Monitoring 426781 Corrections in the Oracle Database Monitoring 483659 BRCONNECT Support for Oracle Monitoring in RZ20 You can find SAP Notes at: service.sap.com/notes
31
www.sap.com
© Copyright 2012 SAP AG. All rights reserved
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Excel, Outlook, PowerPoint, Silverlight, and Visual Studio are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, z10, z/VM, z/OS, OS/390, zEnterprise, PowerVM, Power Architecture, Power Systems, POWER7, POWER6+, POWER6, POWER, PowerHA, pureScale, PowerPC, BladeCenter, System Storage, Storwize, XIV, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, AIX, Intelligent Miner, WebSphere, Tivoli, Informix, and Smarter Planet are trademarks or registered trademarks of IBM Corporation. Linux is the registered trademark of Linus Torvalds in the United States and other countries.
Adobe, the Adobe logo, Acrobat, PostScript, and Reader are trademarks or registered trademarks of Adobe Systems Incorporated in the United States and other countries.
Oracle and Java are registered trademarks of Oracle and its affiliates. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems Inc. HTML, XML, XHTML, and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology. Apple, App Store, iBooks, iPad, iPhone, iPhoto, iPod, iTunes, Multi-Touch, Objective-C, Retina, Safari, Siri, and Xcode are trademarks or registered trademarks of Apple Inc.
IOS is a registered trademark of Cisco Systems Inc.
RIM, BlackBerry, BBM, BlackBerry Curve, BlackBerry Bold, BlackBerry Pearl, BlackBerry Torch, BlackBerry Storm, BlackBerry Storm2, BlackBerry PlayBook, and BlackBerry App World are trademarks or registered trademarks of Research in Motion Limited. Google App Engine, Google Apps, Google Checkout, Google Data API, Google Maps, Google Mobile Ads, Google Mobile Updater, Google Mobile, Google Store, Google Sync, Google Updater, Google Voice, Google Mail, Gmail, YouTube, Dalvik and Android are trademarks or registered trademarks of Google Inc. INTERMEC is a registered trademark of Intermec Technologies Corporation. Wi-Fi is a registered trademark of Wi-Fi Alliance. Bluetooth is a registered trademark of Bluetooth SIG Inc. Motorola is a registered trademark of Motorola Trademark Holdings LLC. Computop is a registered trademark of Computop Wirtschaftsinformatik GmbH.
White Paper: Oracle Database Administration
February 2012
Oracle Database Monitors and Tools
February 2012
TABLE OF CONTENTS
OVERVIEW ......................................................................................................................... 3 SCHEDULING THE DATABASE SYSTEM CHECK ........................................................... 4 CONFIGURING THE DATABASE SYSTEM CHECK ......................................................... 5 VIEWING THE DATABASE SYSTEM CHECK ................................................................... 7 USING THE ALERT MONITOR .......................................................................................... 9
Database Alerts ........................................................................................................................... 14
Space Management......................................................................................................................................... 14 Performance .................................................................................................................................................... 14 Backup and Restore ........................................................................................................................................ 15 SAP Consistency ............................................................................................................................................. 15 Health............................................................................................................................................................... 16
CASE STUDY: ADDING A NEW CONDITION TO THE DATABASE SYSTEM CHECK 23 BRCONNECT .................................................................................................................... 27
Default Conditions for Database Administration ...................................................................... 27 Default Conditions for Database Operations ............................................................................ 29 Critical Database Messages in the Oracle Alert File ................................................................ 29 BRCONNECT Default Conditions for Database Profile Parameters ........................................ 30
ADDITIONAL DOCUMENTATION .................................................................................... 31
SAP Library ................................................................................................................................. 31 SAP Notes ................................................................................................................................... 31
2
Oracle Database Monitors and Tools
February 2012
OVERVIEW
To help you quickly recognize and analyze Oracle database problems, SAP provides comprehensive and fully integrated database system checking. By running regular checks and looking at the results, you can rapidly identify potential problems and take the required action before the database crashes. The following graphic shows how the SAP system supports you in monitoring your Oracle database system:
1 Schedule DBA Planning Calendar DB13
2 Run daily Configure Database System Check DB17
5 BRCONNECT -f check
Alerts
3 Display alert messages DB16
Alerts – mainly health Other alerts Non-BRCONNECT Methods 4 Alert Monitor RZ20
In this paper we discuss the main functions in the figure above: 1. You schedule the database system check to run daily using the Database Planning Calendar [page 4], transactions DBCOCKPIT and DB13, in the Computing Center Management System (CCMS). 2. If required, you can configure the database system check [page 5] using transactions DBCOCKPIT and DB17. 3. When BRCONNECT has run, you can view the results [page 7] using transactions DBACOCKPIT and DB16. 4. You can also view the results of the BRCONNECT and other checks using the database alert monitor [page 9] in transaction RZ20. Transactions DBACOCKPIT and DB13 deliver the results immediately to the alert monitor. 5. BRCONNECT [page 27] is the program that performs a series of database system checks, including those for database administration, database operations, critical database messages in the Oracle alert file, and incorrectly set database parameters. It is fully integrated with the alert monitor. Using a case study [page 23], we show you how to add a new condition to the database system check and view this in the alert monitor. If you want to find more detailed information, look at the additional documentation [page 31] provided by SAP.
3
Oracle Database Monitors and Tools
February 2012
SCHEDULING THE DATABASE SYSTEM CHECK
You can schedule the Oracle database system check in the Database planning Calendar in transaction DBACOCKPITof the Computing Center Management System (CCMS).
We strongly recommend you to schedule the check daily. Otherwise, you might miss important warnings and errors, possibly resulting in expensive system downtime. The best way to do this is by selecting an action pattern, as shown in the following graphic:
Every action pattern includes a daily database system check. If required, you can run a database system check immediately using the DBA Planning Calendar. You can also run the database system check from the command line of your operating system using brconnect –u / -c –f check.
For more information on scheduling the database system check, see DBA Planning Calendar (Oracle) in the documentation CCMS Oracle [page 31].
4
Oracle Database Monitors and Tools
February 2012
CONFIGURING THE DATABASE SYSTEM CHECK
You can configure the Oracle database system check in the Computing Center Management System (CCMS) to: Add new conditions of type ORA or PROF Exclude individual conditions from the check Specify threshold values for the conditions Create object-specific conditions to exclude them from the check Create object-specific conditions to set individual threshold values Specify corresponding corrective actions Maintain the condition description For an example of how to add a new condition of type ORA, see the case study [page 23]. The configuration data is stored in the DBCHECKORA table. Changed or new conditions take effect the next time that BRCONNECT runs. You can view the results in transaction DBACOCKPIT or DB16 [page 7] or the alert monitor [page 9]. To change the configuration, you use transaction DBACOCKPIT or DB17, as shown in the following graphic:
For example, if you want to change the TABLESPACE FULL parameter above, you can do so as follows:
5
Oracle Database Monitors and Tools
February 2012
In the above example, we have changed the threshold at which an alert is raised for a tablespace to 95%. You might also decide to change the Condition Level from Error to Warning, deactivate the check by setting the Active Flag to No, or change the text of the Description. In the following example, we have changed the threshold for a single tablespace, the SYSTEM tablespace, from 95% to 80%, by specifying SYSTEM in the Check Object field. To do this, you first need to copy the condition and then specify Check Object for the new condition.
For more information on configuring the database system check, including changing or creating check conditions, see the following path in the documentation CCMS: Oracle [page 31]: Database System Check Configuring Database System Check (Oracle)
6
Oracle Database Monitors and Tools
February 2012
VIEWING THE DATABASE SYSTEM CHECK
To check the results of a database check you use transaction DBACOCKPIT or DB16:
At the top of the screen, the system displays: Check Results: the number of checks that have finished with an alert. An alert is an error, a warning, or an exception (that is, a deviation from the SAP standard value). Settings: the period of results displayed, the refresh interval, and the deletion period You can configure the display to set the: Number of days for which alerts are displayed Refresh interval after which the display is updated Deletion period after which old alerts are deleted You can also drill down to look at the details of an individual check or Message.
For example, to look in detail at the line TABLESPACE_FULL above:
7
Oracle Database Monitors and Tools
February 2012
You can choose Check log to view the log produced by this check. An example of the log is shown in the case study [page 23]. For more information on viewing the database system check, see the following path in the documentation CCMS: Oracle [page 31]: Database System Check Displaying Alert Messages from Database System Check
8
Oracle Database Monitors and Tools
February 2012
USING THE ALERT MONITOR
You use the alert monitor in the Computing Center Management System (CCMS) to check the following Oracle database functions: Space management – tablespaces and segments Performance – optimizer statistics, buffers, logs, and checkpoints Backup or restore – database and redo log backup Consistency – between database objects in the ABAP and Oracle dictionaries Health – database system checks from BRCONNECT By configuring data collection tools, or methods, to run periodically, alerts are automatically updated and fed to the monitoring architecture. The main tool feeding alerts is BRCONNECT [page 27], especially for Health alerts. Analysis tools provide additional information about the alert conditions, and you can configure autoreact tools to automatically respond when an alert occurs. The alerts are hierarchically arranged and grouped. This is what you see when you open the alert database monitor tree in transaction RZ20:
9
Oracle Database Monitors and Tools
February 2012
If you choose Space Management Tablespaces and open some of the nodes, you see the following:
10
Oracle Database Monitors and Tools
February 2012
This level displays individual database checks, some derived from BRCONNECT, some not. Color-coding and the message text indicate whether there is an alert. If an alert has occurred, the message text gives more information. You can look at the details of an alert, such as Space management Segments Most allocated extents in any segment:
11
Oracle Database Monitors and Tools
February 2012
You can also look at the properties and methods of a node, which determine how the alert is displayed, as follows:
12
Oracle Database Monitors and Tools
February 2012
You can alter data collection methods and other attributes such as the threshold values in the monitor.
13
Oracle Database Monitors and Tools
February 2012
Database Alerts
This section lists the individual database alerts in the alert monitor by branch of the monitoring tree. The tables below include all possible alerts. You can tailor the tree to suit your own requirements by adding, changing, or deleting alerts.
Space Management
Type Tablespace Description Freespace for each tablespace Used space for each tablespace Status (online or offline) for each tablespace Segments Segments with too few allocatable extents * Fewest allocatable extents for a segment * Segments approaching MAX_EXTENTS * Fewest extents left before MAX_EXTENTS * Most allocated extents in any segment * Segments with non-zero PCTINCREASE * Segments in wrong tablespace Rollback segment extension failed * These alerts are only relevant for locally managed tablespaces.
Performance
Type Optimizer Description Last successful update statistics Last brconnect -f stats run Harmful statistics Missing Statistics Buffers Buffer cache Library buffer Redo log buffer
14
Oracle Database Monitors and Tools
February 2012
Locks
Age of oldest exclusive transaction lock Deadlocked resource
Checkpoints
Checkpoint not complete
Backup and Restore
Type Archiving Description Last successful archive log backup Last brarchive run Archiver destination full Archiving off Backup Status Last successful complete database backup Last brbackup run Tablespace in backup mode
SAP Consistency
Type SAP consistency check Objects missing in the database Description Last SAP consistency check run
Primary indexes Secondary indexes Tables Views
Unknown objects in ABAP Dictionary
Database tables Database indexes Database views Database tables without unique index
Inconsistent objects
Primary indexes Secondary indexes
15
Oracle Database Monitors and Tools
February 2012
Tables Views Other checks Primary indexes Secondary indexes Tables Views Primary index not unique Optional indexes Too many indexes created Indexes not created
Health
Type Database health check Description Last successful database check Last brconnect -f check run Last evaluation of check results Start of database check Database administration Archiver destination full Control file not mirrored Missing control file Critical tablespace Critical autoextendable file Critical segment Data file mismatch Missing data file Database file offline File system full Harmful statistics
16
Oracle Database Monitors and Tools
February 2012
Invalid file type Segment in wrong tablespace Missing index Missing statistics Database in NOARCHIVELOG mode Segment with non-zero PCTINCREASE Redo log file not mirrored Missing redo log file Tablespace full Tablespace in backup mode Tablespace offline Segment with too many extents DBA operations Last successful archive log backup too old Last successful complete database backup too old Last archive log backup failed Last complete database backup failed Last operation failed Last update statistics failed Last successful operation too old Last successful update statistics too old
17
Oracle Database Monitors and Tools
February 2012
Type Oracle messages
Description ORA-00060 Deadlock while waiting for resource ORA-00255 Error archiving log file
ORA-00257 Archiver error ORA-00270 Error creating archive log
ORA-00272 Error writing archive log ORA-00376 File cannot be read ORA-00447 ORA-00470 ORA-00471 ORA-00472 ORA-00473 ORA-00474 Fatal error in background process LGWR process terminated with error DBWR process terminated with error PMON process terminated with error ARCH process terminated with error SMON process terminated with error
ORA-00600 Oracle internal error ORA-00603 Oracle server session terminated by fatal error
ORA-01114 I/O error writing database file ORA-01115 I/O error reading database file ORA-01122 File verification check failed ORA-01149 Cannot shut down – data file in backup mode ORA-01555 Snapshot too old ORA-01562 Failed to extend rollback segment ORA-01578 Database block corrupted ORA-01628 Max. extents reached for rollback segment ORA-01629 ORA-01630 tablespace ORA-01631 Max. extents reached saving undo for tablespace Max. extents reached in temp segment in
Max. extents reached in table
18
Oracle Database Monitors and Tools
February 2012
ORA-01632 Max. extents reached in index ORA-01650 ORA-01651 tablespace ORA-01652 ORA-01157 ORA-01654 ORA-01655 Unable to extend rollback segment in tablespace Unable to extend save undo segment for
Unable to extend temp segment in tablespace Unable to extend table in tablespace Unable to extend index in tablespace Unable to extend cluster in tablespace
ORA-01656 Max. extents reached in cluster ORA-01680 ORA-01681 tablespace ORA-01683 ORA-01684 ORA-01685 ORA-01688 ORA-01691 ORA-01692 tablespace ORA-01693 ORA-01694 ORA-07445 ORA-16014 ORA-16038 ORA-19502 ORA-19504 ORA-19510 ORA-27044 Unable to extend LOB segment in tablespace Max. extents reached in LOB segment in
Unable to extend index partition Max. extents reached in table partition Max. extents reached in index partition Unable to extend table partition Unable to extend lob segment in tablespace Unable to extend lob segment partition in
Max. extents reached in lob segment Max. extents reached in lob segment partition Exception encountered: core dump Destination not available for archiving log file Log file cannot be archived Write error on log file Failed to create log file Failed to set size of blocks for file Unable to write the header block of file
19
Oracle Database Monitors and Tools
February 2012
ORA-27072 ORA-30036
File I/O error Unable to extend segment in undo tablespace
Checkpoint not complete – Cannot switch to the next redo log file due to pending checkpoint Corrupt block – Database block is corrupted Profile parameters – Oracle 10g compatible control_file_record_keep_time cursor_space_for_time db_block_checksum db_block_size db_files db_file_multiblock_read_count filesystemio_options log_archive_start log_buffer log_checkpoints_to_alert max_dump_file_size open_cursors optimizer_mode parallel_execution_message_size parallel_threads_per_cpu recyclebin remote_os_authent replication_dependency_tracking shared_pool_size star_transformation_enabled statistics_level
20
Oracle Database Monitors and Tools
February 2012
timed_statistics trace_enabled undo_management undo_tablespace Profile parameters – Oracle 11g background_dump_dest commit_logging commit_wait commit_write compatible control_file_record_keep_time core_dump_dest db_block_size db_files db_file_multiblock_read_count db_writer_processes filesystemio_options log_archive_start log_buffer log_checkpoints_to_alert max_dump_file_size nls_length_semantics open_cursors optimizer_dynamic_sampling optimizer_features_enable optimizer_index_caching optimizer_index_cost_adj
21
Oracle Database Monitors and Tools
February 2012
optimizer_mode parallel_execution_message_size parallel_threads_per_CPU query_rewrite_eabled recyclebin remote_os_authent replication_dependency_tracking shared_pool_size star_transformation_enabled statistics_level timed_statistics trace_enabled undo_management undo_tablespace user_dump_dest
22
Oracle Database Monitors and Tools
February 2012
CASE STUDY: ADDING A NEW CONDITION TO THE DATABASE SYSTEM CHECK
This section shows how you can add a new condition to the database system check and then display the new check. The new condition is a database message – type ORA – but the principle is the same if you want to add a new condition of type PROF. 1. You enter the new condition in transaction DBACOCKPIT or DB17:
The Condition Name is 01157, which generates an error with the Cannot identify / lock data file. The recommendation for this error is to Check the Oracle alert log / trace file. 2. Using transaction RZ20 to call up the alert monitor [page 9], you delete and then rebuild the monitoring tree on the host running the SAP central instance: a) Activate maintenance functions in View: Current system status. b) Reset all alerts in the Oracle monitoring tree. c) Delete the Oracle monitoring tree. d) Run ABAP program RSDBMON0 on the main application server to rebuild the monitoring tree. 3. You call up the alert monitor and see the new check condition, ORA-01157:
23
Oracle Database Monitors and Tools
February 2012
4. You run a database system check in the DBA Planning Calendar [page 4], either immediately or at the next scheduled time. In this example, we assume that the new database check causes an alert. 5. You view the results of the database system check [page 7] using transaction DBACOCKPIT or DB16 to see the alert raised by the new condition:
24
Oracle Database Monitors and Tools
February 2012
The error 01157 appears in the Name column 6. You look at the detail log. a) You can see the new condition ORA-01157:
b) You can also see the alert ORA-1157 that was raised in this example:
25
Oracle Database Monitors and Tools
February 2012
7. You can also see the alert ORA-01157 displayed in the alert monitor, transaction RZ20 [page 9]:
26
Oracle Database Monitors and Tools
February 2012
BRCONNECT
BRCONNECT checks the following types of condition: Database administration – configuration, space management, database state, consistency Database operations – backup and archive results, failed operations Critical database messages in the Oracle alert file – for example, ORA-00600 Database profile parameters in the Oracle initialization file When it finds a critical situation, BRCONNECT writes an alert message to the detail log and to the table DBMSGORA. You can view the results with transaction DBACOCKPIT or DB16 [page 7]. Many alerts are also passed to the alert monitor [page 9]. You can tailor the conditions to suit your own requirements [page 5]. For critical database messages or database profile parameters, you can also add new conditions.
Default Conditions for Database Administration
Condition NOARCHIVELOG_MODE Severity Error Description Checks whether the database is in NOARCHIVELOG mode, which is not allowed for production databases. Checks the highest fill level of the archiving directory (by default, oraarch). Checks the fill level of file systems on the database host. Checks whether there are tablespaces that are offline. Checks whether there are tablespaces that have the BACKUP status although BRBACKUP is not active Checks the fill level of tablespaces in the database.
ARCHIVER_STUCK
Warning
FILE_SYSTEM_FULL
Warning
TABLESPACE_OFFLINE
Error
TABLESPACE_IN_BACKUP
Warning
TABLESPACE_FULL
Warning
DATA_FILE_MISSING
Warning
Checks whether there are data files that no longer exist in the file system. Checks whether there are online redo log files that no longer exist in the file system. Checks whether there are control files that no longer exist in the file system. Checks whether there are data files that are flagged as MISSING in Oracle control file. Checks whether there are database files that have an illegal operating system type, for example, block raw files on Unix or compressed files on Windows.
REDOLOG_FILE_MISSING
Error
CONTROL_FILE_MISSING
Error
DATA_FILE_MISMATCH
Error
INVALID_FILE_TYPE
Error
27
Oracle Database Monitors and Tools
February 2012
Condition REDOLOG_FILE_MIRROR
Severity Error
Description Checks whether there are online redo log files that are not mirrored on the Oracle side. Checks whether there are control files that are not mirrored on the Oracle side. Checks whether there are data files or online redo log files that are OFFLINE. Examines the data files with an activated auto extend feature to see whether the file system can be brought to overflow, due to the existing parameter setting (NEXT and MAXSIZE), during the automatic file extension. Checks whether there are tables or indices, for which the number of allocated extents exceeds the specified threshold value. Checks whether there are tables or indexes that can bring the tablespace to overflow when up to 5 next extents are allocated. Examines the data of a tablespace to see whether the file system can be brought to overflow, due to the existing parameter setting (NEXT and MAXSIZE), during the automatic file extension. Checks whether there are tables that not in a table tablespace or indices, which are not in an index tablespace. Checks whether there are tables that do not have any indices and are not specified in the DBDIFF table exception. Checks whether there are tables or indices that do not have any statistics, although they should have these. Checks whether there are tables or indices that have statistics, although they should not have these (for example, pool and cluster tables). Checks whether there are tables or indexes for which the PCTINCREASE storage parameter is not equal to zero. This can lead to storage fragmentation and is not suitable for the SAP System.
CONTROL_FILE_MIRROR
Error
FILE_OFFLINE
Error
CRITICAL_FILE
Warning
TOO_MANY_EXTENTS *
Warning
CRITICAL_SEGMENT *
Warning
CRITICAL_TABLESPACE
Warning
IN_WRONG_TABLESPACE
Error
MISSING_INDEX
Error
MISSING_STATISTICS
Error
HARMFUL_STATISTICS
Error
PCTINCREASE_NOT_ZERO *
Error
* These conditions are only relevant for locally managed tablespaces.
28
Oracle Database Monitors and Tools
February 2012
The above check conditions are specified in the control table DBCHECKORA.
You cannot add new check conditions but you can change some of the above checks as follows: You can exclude objects – that is, individual tables and indexes or even complete tablespaces – from certain checks that run at table or index level. You do this by creating a new check condition with the relevant object in transaction DBCOCKPIT or DB17 [page 5], setting its ACTIVE FLAG to NO or by specifying the object in the check_exclude BRCONNECT profile parameter.
You can change the threshold values for individual tables and indexes You can change the following check conditions in this way: TOO_MANY_EXTENTS, CRITICAL_SEGMENT, IN_WRONG_TABLESPACE, MISSING_INDEX, MISSING_STATISTICS, HARMFUL_STATISTICS, PCTINCREASE_NOT_ZERO
Default Conditions for Database Operations
Condition LAST_ARCHIVE_FAILED Severity Warning Description Checks whether the last backup of the offline redo log files with BRARCHIVE failed. Checks whether the last complete backup of the database with BRBACKUP failed. Checks whether the last update of the optimizer statistics with BRCONNECT failed. Checks whether the last DBA operation failed. Checks whether the last successful backup of the offline redo log files with BRARCHIVE is too old. Checks whether the last successful complete backup of the database with BRBACKUP is too old. Checks whether the last successful update of the optimizer statistics with BRCONNECT is too old. Checks whether the last successful DBA operation is too old.
LAST_BACKUP_FAILED
Warning
LAST_STATS_FAILED
Warning
LAST_OPERATION_FAILED ARCHIVE_TOO_OLD
Warning Warning
BACKUP_TOO_OLD
Warning
STATS_TOO_OLD
Warning
OPERATION_TOO_OLD
Warning
Since the test conditions for database operations are programmed in a specific way in BRCONNECT (known as built-in test conditions), no new check conditions can be added to the DBCHECKORA table. However, this is generally not necessary because other operations can be monitored by the LAST_OPERATION_FAILED and OPERATION_TOO_OLD check conditions, or by specifying function IDs in the PARAM field.
Critical Database Messages in the Oracle Alert File
See “Oracle Messages” in the table “Health” [Page 16].
29
Oracle Database Monitors and Tools
February 2012
You can enter any Oracle error code or error text as a condition name for this condition type. BRCONNECT searches the Oracle Alert log for corresponding Oracle error messages and might then generate alert messages.
BRCONNECT Default Conditions for Database Profile Parameters
These test conditions check the values of Oracle parameters. The standard test conditions for the database profile correspond to the current SAP recommendations described in SAP Note 830576 for Oracle 10g and in SAP Note 1431798.
You can easily adjust the test conditions for the database profile parameters, depending on the changed recommendations and for new Oracle releases.
30
Oracle Database Monitors and Tools
February 2012
ADDITIONAL DOCUMENTATION
SAP Library
You can find more information on Oracle database administration and the contents of this document in the SAP Library as follows:
All paths refer to SAP NetWeaver 7.3. 1. Call up the SAP Help Portal at help.sap.com/nw73 Application Help SAP Library: English. 2. Choose SAP NetWeaver Library: Function-Oriented View Database Administration Database Administration for Oracle. 3. Choose one of the following: o SAP Database Guide: Oracle o CCMS: Oracle You can also find these plus selected extracts from the SAP Library at: www.sdn.sap.com/irj/sdn/ora SAP on Oracle Knowledge Center SAP Documentation in Help Portal
SAP Notes
You can find more information on the contents of this document in the following SAP Notes: 483856 Description of the Alerts for Oracle Database Monitoring 426781 Corrections in the Oracle Database Monitoring 483659 BRCONNECT Support for Oracle Monitoring in RZ20 You can find SAP Notes at: service.sap.com/notes
31
www.sap.com
© Copyright 2012 SAP AG. All rights reserved
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Excel, Outlook, PowerPoint, Silverlight, and Visual Studio are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, z10, z/VM, z/OS, OS/390, zEnterprise, PowerVM, Power Architecture, Power Systems, POWER7, POWER6+, POWER6, POWER, PowerHA, pureScale, PowerPC, BladeCenter, System Storage, Storwize, XIV, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, AIX, Intelligent Miner, WebSphere, Tivoli, Informix, and Smarter Planet are trademarks or registered trademarks of IBM Corporation. Linux is the registered trademark of Linus Torvalds in the United States and other countries.
Adobe, the Adobe logo, Acrobat, PostScript, and Reader are trademarks or registered trademarks of Adobe Systems Incorporated in the United States and other countries.
Oracle and Java are registered trademarks of Oracle and its affiliates. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems Inc. HTML, XML, XHTML, and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology. Apple, App Store, iBooks, iPad, iPhone, iPhoto, iPod, iTunes, Multi-Touch, Objective-C, Retina, Safari, Siri, and Xcode are trademarks or registered trademarks of Apple Inc.
IOS is a registered trademark of Cisco Systems Inc.
RIM, BlackBerry, BBM, BlackBerry Curve, BlackBerry Bold, BlackBerry Pearl, BlackBerry Torch, BlackBerry Storm, BlackBerry Storm2, BlackBerry PlayBook, and BlackBerry App World are trademarks or registered trademarks of Research in Motion Limited. Google App Engine, Google Apps, Google Checkout, Google Data API, Google Maps, Google Mobile Ads, Google Mobile Updater, Google Mobile, Google Store, Google Sync, Google Updater, Google Voice, Google Mail, Gmail, YouTube, Dalvik and Android are trademarks or registered trademarks of Google Inc. INTERMEC is a registered trademark of Intermec Technologies Corporation. Wi-Fi is a registered trademark of Wi-Fi Alliance. Bluetooth is a registered trademark of Bluetooth SIG Inc. Motorola is a registered trademark of Motorola Trademark Holdings LLC. Computop is a registered trademark of Computop Wirtschaftsinformatik GmbH.
