OTech Magazine - Winter 2014

#5
winter
2014

OTECH
MAGAZINE
Michael Rosenblum

Challenging Dynamic
SQL Myths
Lonneke Dikmans

Lucas Jellema

Choosing the
right Mobile
Architecture

The Rapid Reaction
Force – real time
business monitoring

Foreword

New Years’
Resolutions
The holiday season is upon us. This is the season
to start reflecting on the past year and look towards the future. Most people start thinking of
brilliant new years’ resolutions to make amends
on the faults discover during their reflections.
2014 has been a wonderful year in many aspects.
Globally the world economy has begun to climb
out of recession and technology innovation has
come in a rapid change. As is our field of work.
There are more and more new technologies
added to the stack of tools that are the ITdepartments’. But that also means that there
are more and more technologies to choose
from to solve business problems.
As I see it, 2014 is a grand year of change.
Finally, topics like Big Data and The Internet of
Things, have shifted from tech-savvy interest2 – OTech Magazine #3 – May 2014

groups towards business driven problem
solvers. Plus: (maybe the most important
change in the game) privacy has become a
competitive advantage for most of the highend technology companies. All major news
items around the globe in the tech-industry
have been around the topic of privacy.
Consumers are more aware than ever.
This relatively new insight of ‘data control’ also
offers us a new promise: professional integrity.
As we learned this past year, there’s a lot to say
on this topic. Some of the larger consultancy
firms in our line of business didn’t always take
professional integrity as strict as they should
have. And this reflects on all professionals in
the field.

Therefore I would love to address all of you,
let’s make 2015 the year where professional
integrity becomes the standard again. Let’s
make making business of IT knowledge fun
and trustworthy again. It’s up to all of us.
We can make this the biggest New Years’
Resolution in our career. Let’s make sure
that our customers can entrust us with their
information once and again!
Douwe Pieter van den Bos
[email protected]

want your ad in
otech magazine?
click here to
download the
media kit

3 – OTech Magazine #6 – winter 2014

Content

“dear patrick”

5

DevOps and Continuous Delivery
for Oracle

Patrick Barel

49

Goran Stankovski
Challenging Dynamic SQL Myths

8
Choosing the right

Michael Rosenblum

Mobile Architecture
How to protect your

55

Lonneke Dikmans

sensitive data using Oracle
Database Vault / Creating and
Testing Realms Part II

Flashback - Empowering
14

Power Users!

Anar Godjaev

Biju Thomas

12c Partitioning for

The Rapid Reaction Force –

Data Warehouses

20

real time business monitoring

Michelle Kolbe

Lucas Jellema

Defining Custom Compliance

Oracle Database In-Memory

Rules using 12c OEM Lifecycle

(Part I)

Management Pack

32

64

73

90

Mahir M Quluzade

Phillip Brown
Time Series Forecasting in SQL
Starting WebLogic

39

Cato Aune

4 – OTech Magazine #6 – winter 2014

Kim Berg Hansen

98

dear patrick...

Patrick Barel

Dear Lillian,

Dear Patrick,
What is an ANTI-JOIN?
And what is the difference
between the SEMI-JOIN
and the ANTI-JOIN?
Lillian Sturdey

First of all, both SEMI-JOIN and ANTI-JOIN are not in the SQL syntax but
they are more a pattern. You might expect to be able to write something
like:
[PATRICK]SQL>SELECT d.deptno, d.dname, d.loc
2
FROM dept d
3
SEMI JOIN emp e ON (e.deptno = d.deptno)
4 /

to get all the departments that have at least one employee.
Or:
[PATRICK]SQL>SELECT d.deptno, d.dname, d.loc
2
FROM dept d
3
ANTI JOIN emp e ON (e.deptno = d.deptno)
4 /

to get the departments with no employees. But all you get is an error
saying your command is not properly ended, which can be read as a
syntax error.
ERROR at line 3:
ORA-00933: ORA-00933 SQL command not properly ended.

Maybe your first idea would be to use a normal join to get all the
departments with at least one employee:

5 – OTech Magazine #6 – winter 2014

Dear patrick
2/3

Patrick Barel

[PATRICK]SQL>SELECT d.deptno, d.dname, d.loc
2
FROM dept d
3
JOIN emp e ON (e.deptno = d.deptno)

3
JOIN emp e ON (e.deptno = d.deptno)
4 /

Exactly the result we are looking for:

4 /

But this results in a record for every row in the EMP table. And we only
wanted every unique department.
DEPTNO
---------20
30
30
20
30
30
10
20
10
30
20

DNAME
-------------RESEARCH
SALES
SALES
RESEARCH
SALES
SALES
ACCOUNTING
RESEARCH
ACCOUNTING
SALES
RESEARCH

LOC
------------DALLAS
CHICAGO
CHICAGO
DALLAS
CHICAGO
CHICAGO
NEW YORK
DALLAS
NEW YORK
CHICAGO
DALLAS

DEPTNO
---------30
20
10

DNAME
-------------SALES
RESEARCH
ACCOUNTING

LOC
------------CHICAGO
DALLAS
NEW YORK

14 rows selected.

Well, that’s easy enough, you think, just add a DISTINCT to the statement:
[PATRICK]SQL>SELECT DISTINCT d.deptno, d.dname, d.loc
2
FROM dept d

6 – OTech Magazine #6 – winter 2014

DEPTNO
---------20
10
30

DNAME
-------------RESEARCH
ACCOUNTING
SALES

LOC
------------DALLAS
NEW YORK
CHICAGO

But what if the EMP table contains hundreds, thousands or maybe
millions of rows. That would mean the database has to do a lot of work
to filter out the distinct values.
A different, and probably better, approach would be to use the
SEMI-JOIN pattern. You can use the IN operator like this:
[PATRICK]SQL>SELECT d.deptno, d.dname, d.loc
2
FROM dept d
3 WHERE d.deptno IN (SELECT e.deptno
4
FROM emp e)
5 /
DEPTNO
---------20
30
10

DNAME
-------------RESEARCH
SALES
ACCOUNTING

LOC
------------DALLAS
CHICAGO
NEW YORK

This is exactly what we want to see but for big tables this is not the
correct way to go. For every record in the dept table all the records in

Dear patrick
3/3

Patrick Barel

the EMP table are checked. Again, if we have a lot of employees,
this means a lot of work for the database.
A better SEMI-JOIN to use is the EXISTS operator:

DEPTNO DNAME
LOC
---------- -------------- ------------40 OPERATIONS
BOSTON

[PATRICK]SQL>SELECT d.deptno, d.dname, d.loc
2
FROM dept d
3 WHERE EXISTS (SELECT 1
4
FROM emp e
5
WHERE e.deptno = d.deptno)
6 /

Hope this gives you a bit more insight in this subject and gives you a
better understanding of the wonders of the SQL language. Notice there
are many ways to reach the same result, but one approach might be
more economical than the other.

Please note that with the current optimizer in the database Oracle will
rewrite your query to use the best approach for the task. If the inner
table (in our example EMP) is rather small, then the IN approach might
be the best, in other cases it might be better to use the EXISTS approach.
Where in earlier versions you had to think about which way to go (IN is
better for small tables, EXISTS is better for big ones), you can now rely
on the optimizer to make the correct choice.
If you would want to see exactly the opposite of this query, i.e. all departments with no employees, you use an ANTI-JOIN pattern, which is pretty
much the same but in this case you use NOT IN or NOT EXISTS. A different
approach, which I think is pretty nice is to use an OUTER JOIN and check
for the non-existence of values in column for the OUTER JOINED table.
[PATRICK]SQL>SELECT
2
FROM
3
LEFT
4 WHERE
5 /

d.deptno, d.dname, d.loc
dept d
OUTER JOIN emp e ON (e.deptno = d.deptno)
e.empno IS NULL

7 – OTech Magazine #6 – winter 2014

Happy Oracle’ing,
Patrick Barel

If you have any comments on this subject or you have a question you
want answered, please send an email to [email protected].
If I know the answer, or can find it for you, maybe I can help.

Michael Rosenblum
www.dulcian.com

Challenging
Dynamic SQL
Myths

8 – OTech Magazine #6 – winter 2014

Challenging
dynamic SQL myths
2/6
Michael Rosenblum
For all of the years that Dynamic SQL was a part of the Oracle technology
stack, it developed its own mythology. The reason for these myths is the
magnitude of changes that Oracle has applied to Dynamic SQL over the
last decade. The problem is that some of these myths are simply wrong
and they have always been wrong. Dynamic SQL needs to be used with
care but most “conventional wisdom” about dynamic SQL is in full, or in
part, incorrect.
It is fair to say that limitations existed in previous versions of Dynamic
SQL. Many of these limitations have been removed or reduced in the
current release of the Oracle DBMS. As a result, most problems that you
hear about should always be met with the question: “In which version did
you experience this issue?”
Unfortunately, people often listen to the detractors and thereby fail
to use Dynamic SQL where it could save them a great deal of time and
effort. This section challenges the most common myths and provides
evidence to show why they are wrong.

Myth #1: Dynamic SQL is Always a Security Risk
When using Dynamic SQL, it is very important to keep in mind the fact
that the dreaded code injection can happen only if you do not follow
security best practices. It is not difficult to keep your code safe by
adhering to the following guidelines:

9 – OTech Magazine #6 – winter 2014

1. Always use bind variables to pass data inside of dynamically constructed code modules. Never concatenate. Since bind variables are processed after the code is parsed, unexpected values (such as ‘ OR 1=1’)
will not be able to impact what is being executed.
2. The datatypes of these variables should match the datatypes of the
defined parameters. This eliminates the risk of somebody manipulating NLS-settings and sneaking malicious code into an implicit datatype
conversion mechanism.
3. If you need to concatenate any structural elements (columns, tables,
and so on), always use the DBMS_ASSERT package. This package has
a number of APIs to ensure that input values are what they should be.
For example:
a. SQL_OBJECT_NAME (string) – checks whether the string is a valid
object.
b. SIMPLE _SQL_NAME(string) – checks whether the string is a valid
simple SQL name.
c. SCHEMA_NAME(string) – validates that the passed string is a valid
schema.
It is also important to remember that technologies similar to Dynamic
SQL exist in other environments, and not only in PL/SQL. For example,
it is reasonably common for middle-tier developers to build their SQL
statements on the fly and even open that functionality to end-users using
ad-hoc query tools. You need to be aware that providing users with this
capability gives them far more power than you might expect and opens

Challenging
dynamic SQL myths
3/6
Michael Rosenblum
new holes faster than you can programmatically close. Any environment
that allows end-users to enter real code (even in the form of customized
WHERE-clauses) should be considered a major security threat, no matter
how carefully it is constructed.

can be somewhat resolved by using the DYNAMIC_SAMPLING hint, it is
still valid to say that since Oracle samples the data set, there is still a
probability of a mistake. Since you want to be 100% certain that Oracle
uses the correct cardinality, you can convert the entire module to
Dynamic SQL and adjust the CARDINALITY hint on the spot:

Myth #2: Dynamic SQL is Always Slower than Regular SQL
The biggest misconception that prevents people from using Dynamic SQL
is the idea that it will always be slower than regular SQL. Nobody questions
the fact that if you compare the execution of a plain query with exactly
the same query wrapped with EXECUTE IMMEDIATE, you will see some
performance degradation. However, this is not a fair test. Instead of
comparing syntactically equivalent cases, you should be comparing functionally equivalent ones. If your tests are properly set up, you will find that,
in some cases, Dynamic SQL provides a very viable alternative and can
sometimes provide better performance than traditionally written code.

SQL>
2
3
4
5
6
7
8
9
10
11
12
SQL>

CREATE OR REPLACE PROCEDURE p_processObjects (i_tt id_tt) IS
v_sql_tx VARCHAR2(32767);
v_out_dt DATE;
BEGIN
v_sql_tx:=’SELECT min(created) inlist_tab’||
‘ WHERE object_id IN ‘||
‘(SELECT /*+ cardinality(t ‘||i_tt.count||’)*/ column_value ‘||
‘ FROM table(:1) t)’;
EXECUTE IMMEDIATE v_sql_tx INTO v_out_dt USING i_tt;
dbms_output.put_line(‘MinDate:’||to_char(v_out_dt,’YYYYMMDD’));
END;
/
exec p_processObjects (id_tt(106,25,543));

MinDate:20111103

The main advantage of Dynamic SQL is that you can fine-tune your code
at runtime. By adding a small extra layer of complexity, you can utilize the
knowledge that was not available at compilation time. This extra knowledge is what can make Dynamic SQL-based solutions more efficient than
their hard-coded counterparts.
In general, Dynamic SQL thrives on the unknown. The less information
that is available now, the more valuable its usage. For example, even
though the classic problem of object IN-list default cardinality of 8168

10 – OTech Magazine #6 – winter 2014

Myth #3: Dynamic SQL Always Causes Parsing
One of the most widespread myths about Dynamic SQL is that any
EXECUTE IMMEDIATE call causes parsing, even when using bind variables.
This myth is very dangerous because people might think that if you
cannot avoid parsing, why bother about properly configured binds?
The reality was true in the past. Up to and including Oracle 10g, if you
executed N statements with bind variables, you would get 1 hard parse
and N-1 soft parses. Starting from version 11g onwards, Oracle is efficient

Challenging
dynamic SQL myths
4/6
Michael Rosenblum
enough to completely skip soft parses. The proof can be demonstrated
by running the following script and checking the trace:
SQL>
SQL>
SQL>
2
3
4
5
6
7
8

CREATE TABLE dynamic_sql_q1 (a NUMBER);
CREATE TABLE dynamic_sql_q2 (a NUMBER);
begin
dbms_monitor.session_trace_enable(waits=>true, binds=>true);
FOR i IN 1..50 LOOP
EXECUTE IMMEDIATE ‘INSERT INTO dynamic_sql_q1(a) VALUES (:1)’ USING i;
EXECUTE IMMEDIATE ‘INSERT INTO dynamic_sql_q2(a) VALUES (:1)’ USING i;
END LOOP;
dbms_monitor.session_trace_disable;
END;

9 /

Note that both DYNAMIC_SQL_Q1 and DYNAMIC_SQL_Q2 tables are brand
new and there could not be any cached information available to Oracle.
This is visible from the trace file aggregated using the TKPROF utility:
...
SQL ID: 63rua82qvjn8j Plan Hash: 0
INSERT INTO dynamic_sql_q1(a) VALUES (:1)
call
count
cpu
elapsed
disk
query
current
rows
------- ------ -------- ---------- -------- ---------- ---------- ---------Parse
1
0.00
0.00
0
0
0
0
Execute
50
0.00
0.01
0
4
94
50
Fetch
0
0.00
0.00
0
0
0
0
------- ------ -------- ---------- -------- ---------- ---------- ---------total
51
0.00
0.01
0
4
94
50
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 83
(recursive depth: 1)

11 – OTech Magazine #6 – winter 2014

Number of plan statistics captured: 1
********************************************************************************
SQL ID: 1x6bn2s5p36gf Plan Hash: 0
INSERT INTO dynamic_sql_q2(a) VALUES (:1)
call
count
cpu
elapsed
disk
query
current
rows
------- ------ -------- ---------- -------- ---------- ---------- ---------Parse
1
0.00
0.00
0
0
0
0
Execute
50
0.03
0.01
0
4
99
50
Fetch
0
0.00
0.00
0
0
0
0
------- ------ -------- ---------- -------- ---------- ---------- ---------total
51
0.03
0.01
0
4
99
50
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 83
(recursive depth: 1)
Number of plan statistics captured: 1

Obviously, both queries were parsed only once. Exactly the same
optimization is applicable to a FORALL statement in addition to firing
all INSERTs as a single roundtrip:
SQL>
SQL>
2
3
4
5
6
7
8
9
10
11
12
13

CREATE TABLE dynamic_sql_q3 (a NUMBER);
DECLARE
v_tt dbms_sql.Number_Table;
BEGIN
FOR i IN 1..50 LOOP
v_tt(i):=i;
END LOOP;
dbms_monitor.session_trace_enable(waits=>true, binds=>true);
FORALL i IN INDICES OF v_tt
EXECUTE IMMEDIATE ‘insert into dynamic_sql_q3(a) values (:1)’
USING v_tt(i);
dbms_monitor.session_trace_disable;
END;
/

Challenging
dynamic SQL myths
5/6
Michael Rosenblum
<<<<<<<<<<<<<< Extract from the TKRPROF report >>>>>>>>>>>>>>>>>>
SQL ID: 7uawqxwvd81jc Plan Hash: 0
INSERT INTO dynamic_sql_q3(a) VALUES (:1)
call
count
cpu
elapsed
disk
query
current
rows
------- ------ -------- ---------- -------- ---------- ---------- ---------Parse
1
0.00
0.00
0
0
0
0
Execute
1
0.01
0.01
0
4
45
50
Fetch
0
0.00
0.00
0
0
0
0
------- ------ -------- ---------- -------- ---------- ---------- ---------total
2
0.01
0.01
0
4
45
50
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 83
(recursive depth: 1)
Number of plan statistics captured: 1

The report above shows perfect results. For 50 processed rows, there
is only one PARSE step and one EXECUTE step. This is solid proof that
Dynamic SQL is no longer guilty of expending too many additional
resources while preparing statements on the fly.

Myth #4: DDL Statements are Only for DBAs
Too many DBAs insist that no DDL commands should ever be fired in a
production environment. Therefore, they often try to prevent DDLs from
happening by any means, whether technical or organizational. In addition
to very strict internal policies, the authors have encountered databaselevel triggers that limit the entire system to basic DML operations.
Of course, tightening security is not such a bad idea, but it should not stop
you from providing the required functionality. Because of Dynamic SQL,
DDL statements are no longer the exclusive prerogative of DBA’s SQL*Plus

12 – OTech Magazine #6 – winter 2014

scripts. They can be directly integrated into PL/SQL code and become a
critical part of the overall solution. Although, creating real database objects on the fly in many formally regulated environments can indeed “cross
the line,” there are legitimate cases that will still pass the strictest checks.
One such case comes from my experience maintaining a multi-tier system
where the communication between the middle tier and the database was
done using a connection pool. Eventually, the following problem was
observed. At the end of the day, each session from the pool locked a significant number of segments from the TEMP tablespace. The reason was
that one part of the system used global temporary tables (GTT), defined
as “ON COMMIT PRESERVE.” Once they acquire a segment, these tables
do not release it even if you delete all rows. Obviously, the only way to
reset the high-water mark is to use the TRUNCATE command. However,
TRUNCATE is a DDL statement and cannot be used in straight PL/SQL!
That’s where Dynamic SQL comes to the rescue. The following module
resets all GTTs in the defined session if any one of them was touched:
PROCEDURE p_truncate IS
v_exist_yn VARCHAR2(1);
BEGIN
SELECT ‘Y’
INTO v_exist_yn
FROM v$session s,
v$tempseg_usage u
WHERE s.audsid = SYS_CONTEXT(‘USERENV’,’SESSIONID’)
AND s.saddr = u.session_addr
AND u.segtype = ‘DATA’
AND rownum = 1;

Challenging
dynamic SQL myths
6/6
Michael Rosenblum
FOR c IN (SELECT table_name FROM user_tables
WHERE TEMPORARY = ‘Y’ and DURATION = ‘SYS$SESSION’) LOOP
EXECUTE IMMEDIATE ‘truncate table ‘||c.table_name;
END LOOP;
END;

Unfortunately, there is no simple way to find out exactly which table
was touched, so TRUNCATE is fired for all of them. Therefore, if the total
number of GTTs is reasonably small, the overhead can be tolerated.
TIP: You should never forget that all DDL statements (even unsuccessful
ones) always fire COMMIT before doing anything else. This implicit
COMMIT should be taken into consideration if you add DDLs using
Dynamic SQL to your procedural logic. Depending upon the requirements, you may consider using autonomous transactions to protect
uncommitted changes.

Summary
The biggest challenge in learning Dynamic SQL is to get past your initial
fear of this feature. Of course, with any advanced technology comes risk
with its misuse. In the case of Dynamic SQL, the chances of encountering
security or performance issues often outweigh the potential benefits.

13 – OTech Magazine #6 – winter 2014

Anar Godjaev
www.yapikredi.com.tr
ACE Associate

Blog: anargodjaev.wordpress.com/

How to protect
your sensitive data
using Oracle Database
Vault / Creating and
Testing Realms Part II

14 – OTech Magazine #6 – winter 2014

how to protect your sensitive data
using oracle database vailt / creating and testing realms part II
2/6
Anar Godjaev
Any measures that should be taken for data security purposes should
also be considered at the database level, similar to hardware, network
and operation system levels. Generally, companies buy a firewall product
and think that they have already solved the problems related to security.
Researches show that despite it is possible to take measures against
external attacks by the firewall products, no sufficient measures may be
taken against internal attacks. In particular, no action related to protection of the data is executed on the server where the database operates.
Taking into account that a user having DBA authority will have all type
authority at the database and may perform the same operations even
when s/he gets connected from other computers, possible security gaps
should be considered. Access of a database admin to every data is as disadvantageous as him/her to connect from other computers and perform
the same operations, and is even a security gap. Oracle Database Vault,
which is one of the security solutions of Oracle Database, may be recommended as an application which may assist in solving of the abovementioned problems.
With the increased sophistication of attacks on data, the need to put
more operational controls on the database is greater than ever. Given the
fact that most customers have a small number of DBAs to manage their
databases, it is very important to keep database security related tasks
separate in their own dedicated database accounts. Creating dedicated
database accounts to manage database security helps customers prevent
privileged DBA accounts from accessing application data, restricts ad hoc

15 – OTech Magazine #6 – winter 2014

database changes, and enforces controls over how, when and where
application data can be accessed. Oracle Database Vault provides security
benefits to customers even when they have a single DBA by:
• Preventing hackers from using privileged users’ accounts to steal
application data
• Protecting database structures from unauthorized and/or harmful
changes
• Enforcing controls over how, when and where application data can be
accessed
• Securing existing database environments transparently and without
any application changes
Among the more common audit findings are unauthorized changes to
database entitlements, including grants of the DBA role, as well as new
accounts and database objects. Preventing unauthorized changes to
production environments is important not only for security, but also for
compliance as such changes can weaken security and open doors to hackers, violating privacy and compliance regulations. Oracle Database Vault
SQL Command Controls allow customers to control operations inside the
database, including commands such as create table, truncate table, and
create user. Various out-of-the-box factors such as IP address, authentication method, and program name help implement multi-factor authorization to deter attacks leveraging stolen passwords. These controls prevent
accidental configuration changes and also prevent hackers and malicious
insiders from tampering with applications. The Duty Separation feature

how to protect your sensitive data
using oracle database vailt / creating and testing realms part II
3/6
Anar Godjaev
of Oracle Database Vault will create three different responsibilities such
as the security administration on the database, the account management
and the database administration.
The Security Administrator (Security Administration), the responsible
person for Security is also the manager of the Oracle Database Vault.
S/he is responsible for all security operations at the database. S/he may
manage Realms, command rules and factors and may operate Database
Vault report, while s/he may not get access to the application data.
The Account Manager (Account Management) may create, delete and
change user accounts.
And the Database Administrator (Database Administration) has dba
functions such as backup/restoration, patch application and performance
management.

16 – OTech Magazine #6 – winter 2014

Oracle customers today still have hundreds and even thousands of databases distributed throughout the enterprise and around the world. However, database consolidation will continue as a cost-saving strategy in the
coming years. The physical security provided by the distributed database
architecture must be available in the consolidated environment. Oracle
Database Vault addresses the primary security concerns of database
consolidation.

how to protect your sensitive data
using oracle database vailt / creating and testing realms part II
4/6
Anar Godjaev
First, it’s important to understand the basic architecture of the application you wish to protect. For example, are the objects associated with
the application spread across multiple database schemas or are they
contained in a single database schema? This analysis should include all
objects related to application data including tables, views, materialized
views, and stored procedures. Identify the programs, processes, middle
tier connections, database users, and application administrators that
interact with the application objects. Once this information is obtained,
the Oracle Database Realm definitions can be created and you can authorize who should be able to access application data. Application end users
typically access application data through the middle tier. Some legacy
applications may still use the client server architecture where end users
have their own account in the database. More sophisticated applications
may have application specific processes that run on the server hosting
the Oracle Database.
Along this review document, we used the database Vault Administrator
(DVA) console to administrate Oracle Database Vault. Using DVA, we
created an HR Data Realm to protect human resources data. In order to
set up this realm with DVA we should firstly click Realms, then click
Create, and then naming and defining the realm HR Data Realm.

17 – OTech Magazine #6 – winter 2014

During the setup procedure, one of the main objective was to ensure that
the users with the high privileges was not able to access HR data but
could still administer the database containing the HR Data Realm. . Once
the realm was named and enabled, we selected Audit on failure in order
to send a notification if rules are violated.These are referred to as Realm
Secured Objects. For each object in realm owner, object type and name
need to be specified. In this case, we used the wildcard (%) option to
identify all objects owned by the HR user.

how to protect your sensitive data
using oracle database vailt / creating and testing realms part II
5/6
Anar Godjaev
We also run queries as the HR user to define what owner of the object
could do when a Secured Realm existed for the object they owned. And
also we ensure that no specific privileges had been granted within Oracle
Database Vault to HR at this point. By default, the owner of the object
could only be apply Data Manipulation language (DML) queries. Data
Definition Language (DDL) could not be issued at this point.
In this point of setup procedure, the next step was to determine controls
of privileged user, such as System, when the user accesses objects in the
realm. In this case, the goal is whenever System user or other privileged
user attempt to query HR object result in message should be like System
had insufficient privileges or this object. Similarly, SYSTEM could not be
able to create objects in the HR Data Realm, and Oracle Database Vault
returned a violation notification.

Some employees will need authorization to modify the database as business needs dictate. After running the test above, the user, HR, was added
to HR Data Realm using realm authorizations.

18 – OTech Magazine #6 – winter 2014

how to protect your sensitive data
using oracle database vailt / creating and testing realms part II
6/6
Anar Godjaev
Thus privileged database accounts are one of the most commonly
used pathways for gaining access to sensitive applications data in the
database. While their broad and unrestricted access facilitates database
maintenance, the same access also creates a point of attack for gaining
access to large amounts of data. Oracle Database Vault Realms around
application schemas, sensitive tables and stored procedures provide
controls to prevent privileged accounts from being exploited by hackers
and insiders to access sensitive application data.
Once authorized, this user could issue any statement chosen, including
DDL and DML statements.

19 – OTech Magazine #6 – winter 2014

Source :
Oracle® Database Vault Administrator’s Guide 11g Release 2 (11.2)

Michelle Kolbe
www.backcountry.com

12c Partitioning
for Data
Warehouses

20 – OTech Magazine #6 – winter 2014

12c partitioning
for data warehouses
2/11
Michelle Kolbe
A feature of the Oracle database since Oracle 8.0 in 1997, Oracle Partitioning enables large tables and indexes to be split into smaller pieces to
improve performance, availability, and manageability. Queries are sped
up by orders of magnitudes and a key feature for data warehouse environment implementations. These smaller pieces that the table is split
into are called partitions. Each partition has its own name and optionally
its own storage characteristics. Partitions can be used to spread out and
balance IO across different storage devices. The DBA is able to manage
the partitioned table as a whole or as individual pieces. The cost based
optimizer can use partition pruning to read only the blocks from specific
partitions into memory based on the where clause filters in the query.

Partitioning Strategies
Using a partitioning key, a set of columns used to determine which partition a row will exist in, tables and indexes can be partitioned using one or
many of multiple strategies.
• Range: Data is split based on a range of values of the partitioning key,
most commonly a date field. Partitions are defined by their upper limit
with the lower limit being defined by the upper limit of the preceding
partition. The last partition can optionally be open-ended with no limit
to avoid errors when inserting new data.
• List: A discrete list of values is used for distributing the data for example, a list of states may be used for the partitioning key. A ‘DEFAULT’
partition is used to catch any values that may not fall into one of the
defined list values.

21 – OTech Magazine #6 – winter 2014

• Hash: Data is divided into equal balanced partitions using an internal
hash algorithm.
• Interval: This is a new feature in 11g. With this strategy, the Oracle database will automatically create new partitions when new data is inserted. As an example, on a table that is partitioned by year, when data for
2015 is inserted, a new partition for the year 2015 will be created. This
type of table makes the manageability of partitions much easier.
• Reference: Leverages a parent-child relationship through the primary
key – foreign key relationship so that the child table inherits the parent
table’s partitioning strategy. For every partition in the parent table,
there is a partition in the child table. Later in this paper, there will be an
example of Reference partitioning.
• Virtual Column-Based Partitioning: A partitioning key defined by an
expression is used to distribute the data into partitions. This expression, or virtual column, is stored as metadata only.
• System: Similar to Hash partitioning except does not require a key
column to be specified as the hash key column.
Composite partitioning is a combination of two of the partitioning strategies Range, List and Hash. A couple of examples of using this strategy
would be an Order table that is range partitioned by order_date and
sub-partitioned by hash on customer_id or sub-partitioned by range on
ship_date.

12c partitioning
for data warehouses
3/11
Michelle Kolbe
New Features
Interval-Reference Partitioning
New in 12c is the ability to composite partition first by interval then reference. With this method, new partitions will be created when the data
arrives and the child tables will be automatically maintained with the new
data. The partition names will be inherited from already existing partitions.

An example

In 12c the above script gives this result:
table ORDERS created.
table ORDERLINES created.

Let’s check what partitions we have. There is one partition in each table.
select table_name, partition_name, high_value, interval
from user_tab_partitions
where lower(table_name) in (‘orders’, ‘orderlines’);

If I try to create the following two tables in Oracle 11g, the first statement
will succeed but the second will give me an error.
create table orders
(
order_number number,
order_date_id number,
constraint orders_pk primary key(order_number)
)
partition by range(order_date_id) INTERVAL(7)
(
partition p1 values less than (20140101)
);
table ORDERS created.
create table orderlines
(
orderline_id number,
order_number number not null,
constraint orderlines_pk primary key(orderline_id),
constraint orderlines_fk foreign key (order_number) references orders
)
partition by reference(orderlines_fk);
ORA-14659: Partitioning method of the parent table is not supported

22 – OTech Magazine #6 – winter 2014

Now let’s insert some data in the the orders table and check the
partitions.
insert into orders values (1, 20131231);
insert into orders values (2, 20140102);
insert into orders values (3, 20140113);
commit;
1 rows inserted.
1 rows inserted.
1 rows inserted.
committed.
where lower(table_name) in (‘orders’, ‘orderlines’);

12c partitioning
for data warehouses
4/11
Michelle Kolbe
We can see that the orders table has 3 partitions now and 2 were created
as INTERVAL partitions. Let’s add data to the orderlines table now and
check the partitions.
insert into orderlines values (1, 2);
commit;
1 rows inserted.
committed.

Indexing

The child table now has one INTERVAL partition that is named the same
as the parent table.
Oracle gives us the functionality to manually split partitions into smaller
subsets. If we do this on an interval partition, then some of the partitions
will be converted to conventional partitions instead of interval partitions. In
12c, this split will also convert the child partitions to conventional partitions.
alter table orders
split partition for (20140104) at (20140104)
into (partition p20140101, partition p20140104);
table ORDERS altered.
select table_name, partition_name, high_value, interval
from user_tab_partitions
where lower(table_name) in (‘orders’, ‘orderlines’);

23 – OTech Magazine #6 – winter 2014

Regardless of the partitioning strategy, three types of indexes are
available for use.
• Global Non-Partitioned Index: Identical to an index on a non-partitioned table. This type of index is typically used on a primary key field
in a data warehousing environment.
• Global Partitioned Index: Index on a partitioned or non-partitioned
table that has a partitioning strategy different than the table. This is
not common in a data warehousing environment because the index
maintenance is more difficult.
• Local Index: Index coupled with the underlying partitioned table.
Each index corresponds only one partition. These are most common
in data warehousing environments because it optimizes partition maintenance in that when one partition is dropped, the index partition is
also dropped.

Partial Index
Oracle 12c has a new feature called a Partial Index which allows the user

12c partitioning
for data warehouses
5/11
Michelle Kolbe
to create indexes that only span certain partitions, not all. This feature
works on Local and Global indexes and can complement the full indexing
strategy. The policy can be overwritten at any time. To explain this concept, I think that the Oracle chart below describes the situation the best.

create table orders
(
order_number number
, col2 number
, col3 number
, col4 number
)
indexing off
partition by range(order_number)
(
partition p1 values less than (100) indexing on,
partition p2 values less than (200) indexing on,
partition p3 values less than (300) indexing on,
partition p4 values less than (400) indexing on,
partition p5 values less than (500) indexing on,
partition p_max values less than (MAXVALUE) indexing off
);
table ORDERS created.

Let’s query user_tab_partitions to see the partition definitions.
select table_name, partition_name, high_value, indexing
from user_tab_partitions
where table_name = ‘ORDERS’;

As you can see from this chart’s example, the table was defined with 3
of the partitions having partial local and global indexes. The last partition
does not have any indexes.
Here’s how this is implemented: First, create a partitioned table with
INDEXING OFF set on the whole table definition but then INDEXING ON
for each individual partition that you want indexes on.

24 – OTech Magazine #6 – winter 2014

12c partitioning
for data warehouses
6/11
Michelle Kolbe
Now create two LOCAL indexes; the first one is a partial index.

Now let’s create two GLOBAL indexes, the first one being a partial index.

create index orders_idx1 on orders(order_number) local indexing partial;
create index orders_idx2 on orders(col2) local;
index ORDERS_IDX1 created.
index ORDERS_IDX2 created.

create index orders_idx3 on orders(col3) indexing partial;
create index orders_idx4 on orders(col4);
index ORDERS_IDX3 created.
index ORDERS_IDX4 created.

And let’s check out how these are defined in the Index Partitions table.

And now let’s query the indexes table for these indexes.

select index_name, partition_name, status
from user_ind_partitions
where index_name in (‘ORDERS_IDX1’, ‘ORDERS_IDX2’);

select index_name, status, indexing
from user_indexes
where index_name in (‘ORDERS_IDX3’, ‘ORDERS_IDX4’);

Both indexes are VALID and the first one is defined as Partial.
If we look at the number of segments created for these indexes,
we can see that the local indexes have differing amounts of segments.
select segment_name, segment_type, count(*)
from user_segments
where segment_name in (‘ORDERS_IDX1’, ‘ORDERS_IDX2’, ‘ORDERS_IDX3’, ‘ORDERS_IDX4’)
group by segment_name, segment_type
order by 1;

As you can see from the results, an index partition was created on
the partial index for P_MAX even though we specified INDEXING OFF
however, it’s marked as UNUSABLE.

25 – OTech Magazine #6 – winter 2014

12c partitioning
for data warehouses
7/11
Michelle Kolbe
So by now you are probably asking, what does this mean when someone queries the whole dataset? Let’s look at the explain plan for a query
against the global partial index: orders_idx3.
explain plan for select count(*) from orders where col3 = 3;
select * from table(dbms_xplan.display);

Adding or Dropping Multiple New Partitions
Prior to the 12c release, only one new partition could be added a time and
multiple new partitions required multiple alter table statements. This was
the same with dropping partitions. In 12c, this feature is now available.
ALTER TABLE
PARTITION
PARTITION
PARTITION

orders_range_part
2014 VALUES LESS
2015 VALUES LESS
2016 VALUES LESS

ADD
THAN to_date(‘01-01-2015’, ‘MM-DD-YYYY’),
THAN to_date(‘01-01-2016’, ‘MM-DD-YYYY’),
THAN to_date(‘01-01-2017’, ‘MM-DD-YYYY’);

Splitting or Merging Partitions
Previously, the operation of merging two partitions could only be done
on two partitions into one at a time. In 12c, we can now merge multiple
partitions into one all in one statement.
ALTER TABLE orders_range_part
MERGE PARTITIONS year_2010, year_2011, year_2012, year_2013
INTO PARTITION historical_data_partition;

Notice that the index is used and also the orders table is queried with
TABLE ACCESS FULL for the one partition without the index.

Partition Maintenance
The syntax ALTER TABLE can be used for maintaining partitioned tables
similar to non-partitioned tables. 12c has added functionality to make this
table maintenance easier.

26 – OTech Magazine #6 – winter 2014

12c also added functionality to split a partition into multiple partitions
versus only into two partitions in previous database releases. For instance, if we had a partition for a whole year but we wanted to split it
into quarters, this previously would have taken 3 alter table statements.
Now it can be done in one statement.
ALTER TABLE orders_range_part
SPLIT PARTITION year_2013 INTO
(year_2013_q1 VALUES LESS THAN to_date(‘04-01-2013’, ‘MM-DD-YYYY’),
year_2013_q2 VALUES LESS THAN to_date(‘07-01-2013’, ‘MM-DD-YYYY’),
year_2013_q3 VALUES LESS THAN to_date(‘10-01-2013’, ‘MM-DD-YYYY’),
year_2013_q4);

12c partitioning
for data warehouses
8/11
Michelle Kolbe
The specification of the partitions can also be done as a range or as a list
of values in that partition.
ALTER TABLE orders_range_part
MERGE PARTITIONS year_2010 to year_2013
INTO PARTITION historical_data_partition;
ALTER TABLE orders_range_part
MERGE PARTITIONS for (to_date(‘01-01-2010’, ‘MM-DD-YYYY’)),
for (to_date(‘01-01-2011’, ‘MM-DD-YYYY’)),
for (to_date(‘01-01-2012’, ‘MM-DD-YYYY’)),
for (to_date(‘01-01-2013’, ‘MM-DD-YYYY’)),
INTO PARTITION historical_data_partition;

ALTER TABLE orders TRUNCATE PARTITION 2011_q1 CASCADE;

Similar to the TRUNCATE functionality, EXCHANGE PARTITION also now
be completed in 12c in one operation instead of a bottom-up truncate
then top-down insert. The reference-partitioned hierarchy structure must
match for the target table and the new table being exchanged.

Cascading Truncate or Exchange Partition
In Reference partitioning, performing maintenance can be time consuming to make sure the integrity of the entire hierarchy is maintained. New
with 12c, is a cascading function for TRUNCATE or EXCHANGE PARTITION.
CASCADE applies to the whole reference tree so that it happens in a
single, atomic transaction, all or nothing. This functionality is available for
non-partitioned tables also. To make this work, ON DELETE CASCADE for
all foreign keys is required.

Figure 7 - 11g Truncate / 12c Truncate

27 – OTech Magazine #6 – winter 2014

Figure 8 - 11g Exchange / 12c Exchange

Asynchronous Global Index Maintenance
Every TRUNCATE or DROP command automatically triggers asynchronous global index maintenance, meaning that there’s no waiting for a
global index maintenance to take place. What this means is that if a partition is truncated, the database will keep track of what records have been
orphaned in the index and keep the index as usable. It will filter these out
when queries are run against the index. The DROP and TRUNCATE operations are fast, metadata-only operations which use minimal resources.
These orphaned records will be synchronized automatically through the
Oracle job SYS.PMO_DEFERRED_GIDX_MAINT_JOB which is scheduled

12c partitioning
for data warehouses
9/11
Michelle Kolbe
to run daily at 2 am by default. The index can also be manually cleaned up
by running the above job, running ALTER INDEX REBUILD [PARTITION] or
ALTER INDEX [PARTITION] COALESCE CLEANUP.

An Example
Create a range partitioned table with 500 records in 5 partitions.
create table orders
(
order_number number
)
partition by range(order_number)
(
partition p1 values less than (100),
partition p2 values less than (200),
partition p3 values less than (300),
partition p4 values less than (400),
partition p5 values less than (500),
partition p_max values less than (MAXVALUE)
);
table ORDERS created.
insert /*+ APPEND*/ into orders
select level from dual
connect by level < 501;
commit;
500 rows inserted.
committed.
select count(*)
from orders;
COUNT(*)
---------500

28 – OTech Magazine #6 – winter 2014

Now create an index on this table. When the index is first created, it will
not have any orphaned records.
create index orders_idx on orders(order_number);
index ORDERS_IDX created.
select index_name, orphaned_entries
from user_indexes
where index_name = ‘ORDERS_IDX’;

INDEX_NAME
ORPHANED_ENTRIES
---------------------------------------------------------------------------------------------------------------------ORDERS_IDX
NO
Now we are going to truncate the partition. This statement runs super
fast and the index is still valid.
alter table orders truncate partition p1 update indexes;
table ORDERS altered.
select index_name, status, orphaned_entries
from user_indexes
where index_name = ‘ORDERS_IDX’;

INDEX_NAME
STATUS
ORPHANED_ENTRIES
---------------------------------------------------------------------------------------------------------------------ORDERS_IDX
VALID

YES
The index has orphans which can either be cleaned manually or cleaned
automatically with the SYS.PMO_DEFERRED_GIDX_MAINT_JOB that runs

12c partitioning
for data warehouses
10/11
Michelle Kolbe
by default at 2 AM daily. Let’s manually clean it now.
exec dbms_part.cleanup_gidx();
anonymous block completed
select index_name, status, orphaned_entries
from user_indexes
where index_name = ‘ORDERS_IDX’;

INDEX_NAME
STATUS ORPHANED_ENTRIES
----------------------------------------------------------------------------------ORDERS_IDX
VALID
NO
Note that the cleanup_gidx procedure has some optional parameters.
Without the parameters, it runs on the entire database cleaning up the indexes. If we include the schema name, it will run on the schema level and
if we give it a schema and a table name, it will run for only the indexes on
that table.
An ALTER INDEX <index name> REBUILD PARTITION <partition name>
or ALTER INDEX <index name> COALESCE CLEANUP will also cleanup
orphaned rows.

Online Partition Move
Data warehouses are facing even more pressure to be available 24/7. In
11g and prior versions, running certain maintenance commands would
cause DML commands to be locked until the maintenance completed.

29 – OTech Magazine #6 – winter 2014

This locking often puts a strain on the data warehouse environment.
The new MOVE PARTITION ONLINE functionality in 12c helps meet this
requirement to be available for data manipulation while doing partition
maintenance. ALTER TABLE MOVE is used to move partitions with older
data to slower, cheaper storage devices since it is not being frequently
accessed. By adding ONLINE to this statement, the table will not be
locked for DML operations while the move is taking place. This can also
be used for compressing a table as shown in the code examples below.
ALTER TABLE orders
MOVE PARTITION year2010
TABLESPACE old_storage
UPDATE INDEXES ONLINE;
ALTER TABLE orders
MOVE PARTITION year2010
COMPRESS
UPDATE INDEXES ONLINE;

There are a couple of best practices to note when using Online Partition
Move. Compression while DML is being performed is that the DML will
have an impact on compression efficiency. The best compression ratio will
come with the initial bulk move. Also, you should seek to reduce the number of concurrent DML operations that occur while doing a partition move
because it requires additional disk space and resources for journaling.

Reference Partitioning
When the Oracle 11g release included a new feature called Reference

12c partitioning
for data warehouses
11/11
Michelle Kolbe
Partitioning, it was a bit limited in its scope of operations. With 12c, three
new features have been added to help with creation and maintenance of
this partitioning scheme.
TRUNCATE CASCADE is now an option on the parent table that will automatically truncate the child partitions also. In 11g to achieve this functionality, separate TRUNCATE statements would need to be run on the
parent and the child tables.
Similarly to the TRUNCATE, CASCADE has been added to the partition
EXCHANGE feature. This will modify the child table when modifications
are made to the parent table.
Lastly, reference partitioning can now be used on parent tables that are
interval partitioned as mentioned above in Partitioning Strategies.

A few tips
If you want to change an existing partitioned table to interval partitioning, you can execute this command:
ALTER TABLE <table name> SET INTERVAL (numtoyminterval(1,’MONTH’));

With interval partitioning, to change the tablespace that new partitions
are stored in use:
ALTER TABLE <table name> SET STORE IN (<tablespace name>);

30 – OTech Magazine #6 – winter 2014

References
Oracle Database 12c: What’s New in Partitioning?
Tutorial.
http://www.oracle.com/webfolder/technetwork/tutorials/obe/db/12c/
Partitioning/12c_parti.html
Oracle Database 12c: Partitioning Improvements with Tom Kyte Video.
https://www.youtube.com/watch?v=W71G0H91n_k





 
  





 
 
  





 

 

            













  

  
  
   

31 – OTech Magazine #6 – winter 2014

Phillip Brown
www.e-dba.com

Defining Custom
Compliance
Rules using 12c
OEM Lifecycle
Management
Pack

32 – OTech Magazine #6 – winter 2014

defining custiom compliance rules
using 12c oem lifecycle management pack
2/7
Phillip Brown
Security should be a key area of concern for any DBA and there are many
aspect to it. The starting point is a database security standard. Here
you will define different aspects of the database security, be it access
controls, data management, hardening parameters and determine how
various compliance checks will be made. A compliance check could be
anything from user access to certain DBA_ views or if a particular privileged account which should be locked. To enable the evaluation of these
compliance checks you can use the Oracle Lifecycle Management Pack.
This pack gives you the functionality to be able to define your own security standard within 12c OEM but more importantly the ability to create
custom compliance rules. However, this can be time-consuming and
complicated. This article looks to demonstrate that with some technical
guidance, planning and consideration defining your own custom compliance rules in 12c OEM can be as easy as locking the SYSTEM account.

The key to deploying custom compliance rules
within 12c are the three amber sections, Compliance Rules, Configuration Extensions and Database
SQL. Within the LCMP all compliance rules will go
to define a compliance standard, but this effectively
is just grouping your rules together that you have
defined. You can also define a compliance standard
(collection of rules) for different target types, be it
host, database, application server etc. Ultimately
all you compliance standards go to make a compliance framework.

The Lifecycle Management Pack in OEM has been around for some time
now and it covers a wealth of functionality. It’s formation started at
around 10g but at that time the functionality was split across a number of
management packs and didn’t go by the LCMP name. From 12c the packs
(Configuration Pack, Change Management, Provisioning) were amalgamated into the Lifecycle Management Pack and the functionality was
refined and significantly improved. The Lifecycle Management Pack can
take you from bare-metal provisioning to data change management within a schema, however the functionality I have been working with almost
exclusively for 12 months now is the area of security and compliance.

The compliance rules are as follow:

33 – OTech Magazine #6 – winter 2014

I am going to create a compliance standard in 12c
OEM with three compliance rules.

• The SYSTEM account must be locked
• ‘DROP ANY TABLE’ must be audited
• SYSOPER accounts should be used for
segregation of duties
The next section will demonstrate some of the key
issues when defining SQL for your compliance rules.

defining custiom compliance rules
using 12c oem lifecycle management pack
3/7
Phillip Brown

So this is a quick first attempt at getting the data into 12c OEM from
our targets, it looks ok, but there are some key problems. If we select a
target and have a look at the preview data brought back I can show you a
couple of issues.

Defining your Configuration Extensions
When defining configuration extensions we need to think a little bit
about the data being brought back into the OEM. The key is to answer
the compliance question within the most efficient way possible (I guess
that is the same for any query). On the configuration extension page
there are a couple of key points to note:
• It’s most efficient to create a single configuration with multiple SQL
statements in (it will become clear later why)
• Use a standard incrementing naming convention for your Alias
• Use the select target and preview to test you configuration extensions
against a 12c OEM target

34 – OTech Magazine #6 – winter 2014

With the first query, we are just bringing back too much information, we
need to be selective. We only really need a single column single value returned. This is important when we start to write the compliance checks,
they are much less flexible than configuration extensions so you want to
make your job as easy as possible here.

defining custiom compliance rules
using 12c oem lifecycle management pack
4/7
Phillip Brown
The next query brings back less data, but raises another issue. Here we
want to know if we have enabled audit on DROP ANY TABLE. The query
is simple, asking have we got auditing enabled for that system privilege,
but the two rows returned is an unexpected value, it should be one
shouldn’t it? If we look at the table DBA_STMT_AUDIT_OPTS we can see
the problem. We have auditing enabled for specific users which although
is good would actually give us a false positive in this case. Thinking we
have the auditing option enabled but in reality it is not auditing exactly
what we want to know. We want it enabled globally, i.e. where USER_
NAME is NULL.

The final configuration extensions brings back nothing, which you may
think is ok, but it’s actually an issue. If your configuration extension is
bringing back nothing it means it cannot be evaluated, therefore it is
compliant. Regardless of what the configuration extension is, it ALWAYS
need to bring back a value.

So this is really why it is very very important to clarify the ‘questions’ you
are asking prior to developing configuration extensions and subsequent
compliance rules. With that in mind our queries are updated to this:

Deploying Configuration Extensions
With that now completed we can deploy this configuration extensions
and start to look at some of the data stored within the 12c OEM repository. Deployment of a configuration extensions is completed via the Actions > Manage Deployments menu item on the configuration extensions
main page. If you are undertaking a mass deployment across you estate
then be mindful of targets which are under blackout as those configuration extensions will not be deployed.

So now that configuration extensions are deployed we should be able
to see not only the ‘collection’ of this information but the information
stored as well. Once a configuration extensions is deployed to a target
you can see it in the list of agent scheduled jobs. The name of the con-

35 – OTech Magazine #6 – winter 2014

defining custiom compliance rules
using 12c oem lifecycle management pack
5/7
Phillip Brown
figuration extension is translated into an internal name however it keeps
the OTECH.. section making it easy to locate.

select cm_target_name, cm_snapshot_type, data_source_name, attr, value
from mgmt$ccs_data where cm_target_name=’DBTEST’

Executing this command on the agent:
emctl status agent scheduler | grep TECH
2014-11-18 21:01:07.983 : oracle_database:DBTEST:ccs_c_TECH_EXTENS_08100983F47B00C0E0
531EDFF56893FA

All configuration extension data is stored in the MGMT$CCS_DATA table.
The name of the scheduler items is the same as the SNAPSHOT_TYPE
in MGMT$CCS_DATA table. Another key thing to note is that the ATTR
stores the COLUMN_NAME of the configuration extension query and
the VALUE is data returned by configuration extension. Therefore each
COLUMN of you configuration extension will be stored as a ROW within
the database. This is why configuration extensions need to be kept as
simple as possible and really just return single column values as you can
tie yourself up in knots without even trying.
Finally it is worth nothing that the configuration extensions CM_SNAPSHOT_TYPE is unique per configuration extension and all the subsequent
ALIAS’s are stored as the under the column DATA_SOURCE_NAME.
When you are trying to pull lots of compliance data into 12c OEM it is
worth trying to use as few configuration extensions as possible as it will
ease the creation of compliance rules and also keep the number of collections on the agent down to a minimum.

36 – OTech Magazine #6 – winter 2014

Mapping Configuration Extensions to Compliance Rules

Now we have the configuration extensions defined and deployed we can
now create the compliance rules. When you start to generate the compliance rules there are some key things to note. This is really around the
section ‘Define Query’ and ‘Define Validation’.
When you come to define query the best place to start using the data
modeller. From here you can select the target type and you will be able
to define the configuration extensions you have already defined. Be sure
to select the ‘Parsed Data’ as this is the actual data which you need to

defining custiom compliance rules
using 12c oem lifecycle management pack
6/7
Phillip Brown
validate. The key columns from the parsed data are ‘Data Source’ and
‘Value’. Once you have selected these items you don’t need to specify
the contents at this point. Click on OK and when you come out of the
data modeller you will see the query generated.

SELECT
s1.target_guid
,
‘attrvalue’ AS info ,
s2.DATA_SOURCE_NAME ,
s2.VALUE
FROM
MGMT$CCS_DATA s2
,
MGMT$ECM_CURRENT_SNAPSHOTS s1gen1,
MGMT$TARGET s1
WHERE
(
s1gen1.TARGET_GUID
= s1.TARGET_GUID
AND s1gen1.ECM_SNAPSHOT_ID = s2.ECM_SNAPSHOT_ID (+)
AND s1.TARGET_TYPE
= ‘oracle_database’
AND s1gen1.SNAPSHOT_TYPE = ‘ccs_c_OTECH_EXTENS_08100983F47B00C0E0531EDF
F56893FA’
)

This query can now be used as the basis for ALL your compliance rules.
If you execute this query in SQLPlus you will see it will bring back all your
configuration extensions. What you can now do is for each compliance
rule add in the relevant data source name (ALIAS) which you defined in
each configuration extension. If you have a standard naming convention
and a single configuration extension this becomes a very quick process.
SELECT
s1.target_guid
,
‘attrvalue’ AS info ,
s2.DATA_SOURCE_NAME ,
s2.VALUE
FROM
MGMT$CCS_DATA s2
,
MGMT$ECM_CURRENT_SNAPSHOTS s1gen1,
MGMT$TARGET s1

37 – OTech Magazine #6 – winter 2014

defining custiom compliance rules
using 12c oem lifecycle management pack
7/7
Phillip Brown
WHERE
(
AND
AND
AND
AND

s1gen1.TARGET_GUID
= s1.TARGET_GUID
s1gen1.ECM_SNAPSHOT_ID = s2.ECM_SNAPSHOT_ID (+)
s1.TARGET_TYPE
= ‘oracle_database’
s1gen1.SNAPSHOT_TYPE = ‘ccs_c_QTECH_EXTENS_08100983F47B00C0E0531EDFF56893FA’
s2.DATA_SOURCE_NAME
= ‘OTECH_1’
)

When defining the validation of that data, i.e. if it is compliant or not
then, all you need to do is select the VALUE column and define the value
it would return when there is a violation. For this rule, is the SYSTEM
locked, you specify the condition in which it is seen as a VIOLATION, i.e.
when it is OPEN. The only other thing to note is with numeric checks
which are based on a COUNT(*); in some cases zero will define a violation
(i.e. when auditing on an object isn’t enabled) and it others it will be compliant (i.e. when a user doesn’t have a grant to a SYSTEM PRIVILEGE).
When it comes to defining custom compliance rules in 12c OEM you need
to consider the following:
• Translate all your compliance rules into SQL checks
• Determine for each check what the value of the violation would be
• Group SQLs together in configuration extensions and use a pre-defined
naming convention for the aliases
• Prior to generating any compliance rules ensure the configuration
extensions are returning the ‘expected’ results from a selection of
database targets

38 – OTech Magazine #6 – winter 2014

Cato Aune
www.sysco.no/en/stikkord/
middleware

Starting
WebLogic

39 – OTech Magazine #6 – winter 2014

Starting web logic
2/10
Cato Aune
WebLogic is an advanced and feature rich application server. Unfortunately that also makes WebLogic a bit complex, and often it is possible to
achieve the same task in different ways.
Starting WebLogic is one of the tasks that can be done in several different
ways. Each ending up with a running WebLogic instance, but the result
could still be a bit different, and not exactly what you really wanted.
Unlike some other application servers, there are no out-of-the-box scripts
for automatically start of WebLogic when the server boots. But the documentation and scripts are getting better for each release, so maybe in a
few years’ time, you might get everything set up by the installer.
This article will guide you thru different options on how to start WebLogic, along with a recommended method and a set of sample scripts,
a bit simplified, but that works. For the purpose of demonstrating the
concepts, some more advanced items are not part of the scripts, like error handling, possibility to start several servers at once (start the WLST
engine only once, limits overhead) and starting the whole cluster in one
operation.

Why use bootscripts?
There are several reasons for using bootscripts. One very important
reason is that the server is started the same way every time. Why is that
important? Well, if you start WebLogic with different users, at some point

40 – OTech Magazine #6 – winter 2014

WebLogic won’t start anymore. One common mistake is that someone
starts WebLogic as user root. That will work, it is not recommended, but
everything should run fine. Until you next time try to start WebLogic as a
regular user (i.e. oracle or weblogic). Some important files have changed
ownership to root, and the user you try to start WebLogic with, doesn’t
have access to those files anymore.
Another issue you could run into, is that depending on how you start
WebLogic, actual runtime configuration could be different, more about
that later on.
And if you like to avoid the call at 3 am from the operation center, setting
up automatic boot or give the operations team just one command to run
after server reboot, increases your chances for a good night sleep.

Components
There are three important components in booting a WebLogic server
• Node Manager
• WebLogic Scripting Tool (WLST)
• Shell scripts
Node Manager is a WebLogic Server utility that let you
• Start
• Shut down
• Restart
Administration Server and Managed Server instances.

Starting web logic
3/10
Cato Aune
Before WebLogic 12.1.2 there were one Node Manager per server (host),
while from 12.1.2 and onwards the default is one Node Manager per domain, and the configuration is kept under each $DOMAIN_HOME

When you have decided on how you want to start WebLogic, stick with
that decision. Using different methods each time will give you some interesting mysteries to solve.

WebLogic Scripting Tool (WLST) is a Jython (Python) scripting tool that
let you do almost everything that could be done in WebLogic. WLST could
be run with a script or interactive. It is even possible to do a recording in
Admin Console and get the result as a WLST file.
Be sure to have the correct path before running WLST, you could set this
by sourcing setDomainEnv.sh (i.e. . $DOMAIN_HOME/bin/setDomainEnv.
sh, note the extra . before $DOMAIN_HOME)

Auto generated start scripts
Some scripts are autogenerated during the domain creation, most are
put in $DOMAIN_HOME/bin like setDomainEnv.sh and startManagedWebLogic.sh, while startWebLogic.sh is placed in $DOMAIN_HOME.

Shell scripts could be regular bash shell scripts for Linux or cmd/bat files
on Windows. Shell scripts will be used for init.d/xinit.d scripts in Linux to
start Node Manager and WebLogic on server boot or to create Windows
services in Windows. When running the configuration wizard, default
start scripts are generated and placed in $DOMAIN_HOME/bin
These scripts will be used later on.

Different methods to start WebLogic
• Auto generated start scripts
• Using WLST and Admin Server
• Using WLST and Node Manager

41 – OTech Magazine #6 – winter 2014

To start the Admin Server, you could run
$DOMAIN_HOME/startWebLogic.sh and to start the managed server you could run $DOMAIN_
HOME/bin/startManagedWebLogic.sh ms1

This works well, but make sure to use nohup and put the process in the
background, or the server instance would stop when you log out.
$ nohup startWebLogic.sh &

Using WLST and Admin Server
It is possible to start the Admin Server from WLST, but it is not recommended with Fusion Middleware suite products (according to the documentation).
To use WLST and Admin Server to start managed server, there are some
requirements

Starting web logic
4/10
Cato Aune
• Admin Server must be running
• Node Manager must be running
• Admin Server must be able to communicate with Node Manager
Connect to AdminServer to start managed servers

Starting directly from Node Manager does not set the environment
variables mentioned above. It is possible to provide the same information
manually along with nmStart.

connect(userConfigFile=userFile, userKeyFile=keyFile, url=adminUrl)
start(...)

Custom WebLogic config

When starting a managed server from Admin Server via Node Manager,
several environment variables are defined:
JAVA_VENDOR, JAVA_HOME, JAVA_OPTIONS, SECURITY_POLICY, CLASSPATH, ADMIN_URL
Using WLST and Node Manager
To use WLST and Node Manager, the requirements are
• Node Manager must be up and running
Connect to Node Manager using nmConnect
nmConnect(userConfigFile=nmUserFile, userKeyFile=nmKeyFile,
host=nmHost, port=nmPort, domainName=domain,
domainDir=domainPath, nmType=nmType)
Start Admin Server and Managed Servers using nmStart
nmStart(‘AdminServer’)
nmStart(‘ms1’)

42 – OTech Magazine #6 – winter 2014

Often you want some custom config for each server, like heap size (-Xms,
-Xmx) or more advanced like configure where to find the coherence
cache in a high availability setup.
Where to put you custom config depends on how WebLogic is started.
If you start from Admin Server via Node Manager, you could place the
config in Arguments on the Server Start tab in the Configuration for each
server. The config will be stored in config.xml and is visible from the Admin Console.

Starting web logic
5/10
Cato Aune
If you start directly from Node Manager this config will not be used
(unless you provides it along with nmStart)
Never put custom config in setDomainEnv.sh, startWebLogic.sh or startManagedWebLogic.sh - those files will we overwritten during upgrade.
If you want to put the config in a script file, create setUserOverrides.sh in
$DOMAIN_HOME/bin. In 12c that is enough, while in 11g, you have to add
the following lines in setDomainEnv.sh (this is how it is done in the 12c
version of setDomainEnv.sh).
if [ -f ${DOMAIN_HOME}/bin/setUserOverrides.sh ] ; then
. ${DOMAIN_HOME}/bin/setUserOverrides.sh
fi

• It is recommended to let Node Manager use start script
(StartScriptEnabled=true)
• It is recommended to start from AdminServer to give server start arguments and SSL arguments to Node Manager
Our approach
• Enable start script in Node Manager (StartScriptEnabled=true in nodemanager.properties)
• Connect to Node Manager and start AdminServer
• Connect to AdminServer and start managed servers

Scripts
wls.py - the actual WLST script that starts and stops WebLogic instances

Make sure you set StartScriptEnabled=true in nodemanager.properties,
so the default start scripts are used when starting a WebLogic instance.
If you want to use your own start scripts and not the auto generated ones, you have to put the name of the start script in
StartScriptName=myStartupScript.sh in nodemanager.properties
Make sure you call the other necessary scrips in $DOMAIN_HOME/bin like
setDomainEnv.sh and use JAVA_OPTIONS that are set if you start from
Admin Server.

Recommendations
• It is recommended to always use Node Manager to start Admin Server
and managed servers

43 – OTech Magazine #6 – winter 2014

import sys
def startAdmin():
print ‘Starting AdminServer’
nmConnect(userConfigFile=nmUserFile, userKeyFile=nmKeyFile, host=nmHost,
port=nmPort, domainName=domain, domainDir=domainPath, nmType=nmType)
nmStart(‘AdminServer’)
nmDisconnect()
return
def stopAdmin():
print ‘Stopping AdminServer’
connect(userConfigFile=wlsUserFile, userKeyFile=wlsKeyFile, url=adminUrl)
shutdown(‘AdminServer’, force=’true’)
disconnect()
return

Starting web logic
6/10
Cato Aune
def startManaged(managed):
print ‘Starting ‘, managed
connect(userConfigFile=wlsUserFile, userKeyFile=wlsKeyFile, url=adminUrl)
start(managed)
disconnect()
return
def stopManaged(managed):
print ‘Stopping ‘, managed
connect(userConfigFile=wlsUserFile, userKeyFile=wlsKeyFile, url=adminUrl)
shutdown(managed, force=’true’)
disconnect()
return
if ((len(sys.argv) < 2) | (len(sys.argv) > 3)):
print ‘ Wrong number of arguments’
elif (sys.argv[1] == ‘startadmin’):
startAdmin()
elif (sys.argv[1] == ‘stopadmin’):
stopAdmin()
elif (sys.argv[1] == ‘start’):
startManaged(sys.argv[2])
elif (sys.argv[1] == ‘stop’):
stopManaged(sys.argv[2])

startall.sh - calls the wls.py script to start each WebLogic instance
Full path for wlst.sh (under wlserver/common/bin), config.properties and
wls.py must be given (not necessary for wlst.sh if wlserver/common/bin is
in $PATH)
wlst.sh -loadProperties config.properties - skipWLSModuleScanning wls.py startadmin
wlst.sh -loadProperties config.properties - skipWLSModuleScanning wls.py start ms1

44 – OTech Magazine #6 – winter 2014

stopall.sh - calls the wls.py script to stop each WebLogic instance
Full path for wlst.sh (under wlserver/common/bin), config.properties and
wls.py must be given (not necessary for wlst.sh if wlserver/common/bin is
in $PATH)
wlst.sh -loadProperties config.properties - skipWLSModuleScanning wls.py stop ms1
wlst.sh -loadProperties config.properties - skipWLSModuleScanning wls.py stopadmin

config.properties - adminUrl must point to the AdminServer (often a VIP
address) while nmHost points to the local Node Manager serving this
domain
adminUrl=t3://wls12c.dev.sysco.no:7001
nmHost=wls12c.dev.sysco.no
nmPort=5556
nmUserFile=/u01/app/oracle/config/nmUserFile
nmKeyFile=/u01/app/oracle/config/nmKeyFile
nmType=plain
wlsUserFile=/u01/app/oracle/config/wlsUserFile
wlsKeyFile=/u01/app/oracle/config/wlsKeyFile
domain=mydomain
domainPath=/u01/app/oracle/user_projects/domains/mydomain

Encrypting username and password, so they can be referenced from wls.
py as nmUserFile/wlsUserFile and nmKeyFile/wlsKeyFile, is different in 12c
and 11g. The 11g method still works as of now in 12c, but is deprecated and
it is better to use the 12c method.

Starting web logic
7/10
Cato Aune
11g method - source setDomainEnv.sh before running this command to
get the paths correctly (everything on one line)
java weblogic.Admin -username nodemanager -userconfigfile /u01/app/oracle/config/nmUserFile -userkeyfile /u01/app/oracle/config/nmKeyFile STOREUSERCONFIG

12c method - source setDomainEnv.sh before running this command to
get the paths correctly
java weblogic.WLST

Encrypt username and password for Node Manager
wls:/offline> nmConnect( ‘nodemanager’,’welcome1’,’localhost’,5556,’mydomain’, ‘/u01/
app/oracle/user_projects/domains/mydomain’, ‘plain’)
wls:/mydomain/serverConfig> storeUserConfig(
‘/u01/app/oracle/config/nmUserFile’,’/u01/app/oracle/config/nmKeyFile’, ‘true’)

Encrypt username and password for WebLogic admin user
wls:/mydomain/serverConfig> connect(‘weblogic’, ‘welcome1’, ‘t3://wls12c.dev.sysco.
no:7001’)
wls:/mydomain/serverConfig> storeUserConfig(
‘/u01/app/oracle/config/wlsUserFile’,’/u01/app/oracle/config/wlsKeyFile’,’false’)

/etc/init.d/nodemanager - init.d script that starts Node Manager during
server boot.
This is not based on the script from the 12c documentation ( http://docs.
oracle.com/middleware/1213/wls/NODEM/java_nodemgr.htm#BABJIDFD
), but is pretty similar anyway. One important difference is that this script

45 – OTech Magazine #6 – winter 2014

starts Node Manager as user oracle, while the script in the documentation starts Node Manager as root, which is not recommended.
Make sure to adjust the paths in the script to fit your environment.
The script must be made runnable (chmod 0755) and activated (chkconfig --add) before it will be used next time the server starts or stops.
#!/bin/sh
#
# nodemanager Oracle Weblogic NodeManager service
#
# chkconfig: 345 85 15
# description: Oracle Weblogic NodeManager service
### BEGIN INIT INFO
# Provides: nodemanager
# Required-Start: $network $local_fs
# Required-Stop:
# Should-Start:
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Short-Description: Oracle Weblogic NodeManager service.
# Description: Starts stops Oracle Weblogic NodeManager.
### END INIT INFO
. /etc/rc.d/init.d/functions

Starting web logic
8/10
Cato Aune
# Your WLS home directory (where wlserver is)
export MW_HOME=”/u01/app/oracle/middleware/Oracle_Home”
export JAVA_HOME=”/usr/java/latest”
DAEMON_USER=”oracle”
PROCESS_STRING=”^.*weblogic.NodeManager.*”
source $MW_HOME/wlserver/server/bin/setWLSEnv.sh > /dev/null
export NodeManagerHome=”$MW_HOME/user_projects/domains/mydomain/nodemanager”
NodeManagerLockFile=”$NodeManagerHome/nodemanager.log.lck”
PROGRAM=”$MW_HOME/user_projects/domains/mydomain/bin/startNodeManager.sh”
SERVICE_NAME=`/bin/basename $0`
LOCKFILE=”/var/lock/subsys/$SERVICE_NAME”
RETVAL=0
start() {
OLDPID=`/usr/bin/pgrep -f $PROCESS_STRING`
if [ ! -z “$OLDPID” ]; then
echo “$SERVICE_NAME is already running (pid $OLDPID) !”
exit
fi
echo -n $”Starting $SERVICE_NAME: “
/bin/su $DAEMON_USER -c “$PROGRAM &”
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && touch $LOCKFILE

/bin/echo “$SERVICE_NAME is stopped”
fi
echo
/bin/rm -f $NodeManagerLockFile
[ $RETVAL -eq 0 ] && rm -f $LOCKFILE
}
restart() {
stop
sleep 10
start
}
case “$1” in
start)
start
;;
stop)
stop
;;
restart|force-reload|reload)
restart
;;
*)
echo $”Usage: $0 {start|stop|restart}”
exit 1
esac

}

exit $RETVAL

stop() {
echo -n $”Stopping $SERVICE_NAME: “
OLDPID=`/usr/bin/pgrep -f $PROCESS_STRING`
if [ “$OLDPID” != “” ]; then
/bin/kill -TERM $OLDPID
else

/etc/init.d/weblogic - init.d script that calls startall.sh during server boot,
and stopall.sh when the server shuts down.
Make sure to adjust the paths in the script to fit your environment.

46 – OTech Magazine #6 – winter 2014

Starting web logic
9/10
Cato Aune
The script must be made runnable (chmod 0755) and activated (chkconfig --add) before it will be used next time the server starts or stops.
#!/bin/sh
#
# weblogic Oracle Weblogic start
#
# chkconfig: 345 85 15
# description: Oracle Weblogic service
### BEGIN INIT INFO
# Provides:
# Required-Start: $nodemanager
# Required-Stop:
# Should-Start:
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Short-Description: Oracle Weblogic service.
# Description: Starts and stops Oracle Weblogic.
### END INIT INFO
. /etc/rc.d/init.d/functions
# Your WLS home directory (where wlserver is)
export MW_HOME=”/u01/app/oracle/middleware/Oracle_Home”
export BOOT_HOME=”/u01/app/oracle/bootscripts”
export JAVA_HOME=”/usr/java/latest”
DAEMON_USER=”oracle”
source $MW_HOME/wlserver/server/bin/setWLSEnv.sh > /dev/null
PROGRAM_START=”$BOOT_HOME/startall.sh”
PROGRAM_STOP=”$BOOT_HOME/stopall.sh”
SERVICE_NAME=`/bin/basename $0`
LOCKFILE=”/var/lock/subsys/$SERVICE_NAME”
RETVAL=0

47 – OTech Magazine #6 – winter 2014

start() {
echo -n $”Starting $SERVICE_NAME: “
/bin/su $DAEMON_USER -c “$PROGRAM_START &” RETVAL=$?
echo [ $RETVAL -eq 0 ] && touch $LOCKFILE
}
stop() {
echo -n $”Stopping $SERVICE_NAME: “
/bin/su $DAEMON_USER -c “$PROGRAM_STOP &” RETVAL=$?
[ $RETVAL -eq 0 ] && rm -f $LOCKFILE
}
restart() {
stop
sleep 10
start
}
case “$1” in
start)
start
;;
stop)
stop
;;
restart|force-reload|reload)
restart
;;
*)
echo $”Usage: $0 {start|stop|restart}”
esac
exit 1

Starting web logic
10/10
Cato Aune
Summary
The scripts shown in this article just covers the minimum of functionality
needed to start WebLogic. Feel free to use the scripts “as is” or as a basis
for your own enhancements to fit your requirements.
Although it is very convenient to start everything automatically, you
should consider if this is the best solution for you. If a database server is
not up yet, the managed server that needs to connect to that database
server will end up in ADMIN state. A possible solution (but not good for
all) is to set Initial Capacity to 0, then the managed server should start
(but it might be a bit grumpy if someone tries to use the database connection while it is not there yet).
To be on the safe side, you could create a startadmin.sh script, which just
starts the Admin Server, and then use startall.sh for the managed servers. Remember to edit the /etc/init.d/weblogic script and replace startall.
sh with startadmin.sh. The Admin Server will now start when the server
boots, and you will have to log in and run startall.sh manually. You should
still be allowed to sleep thru the night, just tell the operation team to be
sure all other servers are up and running, before they can run startall.sh.

48 – OTech Magazine #6 – winter 2014

Goran Stankovski
www.limepoint.com

DevOps and
Continuous
Delivery for
Oracle

49 – OTech Magazine #6 – winter 2014

Devops and continuing delivery
for oracle
2/5
Goran Stankovski
This article introduces the DevOps and Continuous Delivery for Oracle
article series.
There is a common thread amongst Oracle implementations - from the
smallest implementation to the largest mission-critical implementation:
those that have had an active engagement from the Operations team
during implementation have been far more successful with the transition
to BAU. Reflecting on this, the DevOps and Continuous Delivery approaches of today are being applied to harness the power in the collaborative and cooperative efforts of multi-team environments to deliver value
tomorrow. Over the next 4 issues of this magazine, we will be talking
about DevOps and Continuous Delivery; in particular the challenges, pitfalls, and key considerations in embarking on such initiatives in an Oracle
context.
Traditionally development and project streams deliver capabilities to an
organisation via fulfilment of business initiatives through project delivery, before handing over the capability to Operations teams to run and
manage. These can range anywhere from small projects and initiatives, to
large transformational programmes. No matter how small or large, there
are always key benefits to be realised by the business in delivery of such
initiatives, or so a majority of business cases will suggest.
In traditional enterprise project delivery, a project team will work on a
system for many months (often years), polishing, honing, and perfect-

50 – OTech Magazine #6 – winter 2014

ing it until it is deemed ready to be: deployed into production; handed
over to the Operations team; and then kept in place as-is until the next
release. From this point onward any enhancements or remediation to
these systems must be coordinated through a change management function, whose procedures are often centred on actively discouraging these
activities.

Devops and continuing delivery
for oracle
3/5
Goran Stankovski
Operations of the delivered capability often is not on the priority list of
project teams. The project teams objective is to deliver the solution in
the timelines communicated by the project management team. Timelines
that are often very tight, and usually defined with little technical input
and/or made with large assumptions that remain invalidated until so late
in the life of the project that remediation is impractical, if not impossible.
This puts enormous pressure on project teams to deliver and transition
the system to Operations for the on-going management. Meanwhile
Operations are often left with an unmanageable solution, with which
they have had little to no involvement. From the Operations perspective
they’ve been thrown a “ticking time bomb” in that while it delivers an essential capability to the business, Operations cannot effectively manage,
modify, remediate, or enhance it.
As a side effect in a traditional enterprise delivery model, Operations
teams are incentivised to dissuade and even actively impede delivery of
new, improved or streamlined business function. After all, completely
remaking “the world” will always carry significant operational and procedural risks associated with any system that is untried and untested in the
production environment. Additionally as any new system or capability
can often represent a year or more of change, it is virtually impossible to
roll back if things go wrong – if not technically, often politically.
DevOps (as the name suggests) is the union of Development and Operations teams, typically demonstrated through their increased collaboration

51 – OTech Magazine #6 – winter 2014

and cooperation. Implementing DevOps and Continuous Delivery methodologies in a traditional enterprise environment has many challenges.
One such challenge is that DevOps can mean many different things to
different people - there is no DevOps governing or certifying body, nor is
there one clear and concise definition. In many respects DevOps is about
attitudes and relationships, more than tooling and procedures - it is the
bringing closer of Development and Operations capabilities within an organisation. One thing is clear; the greater the cohesion of Development
and Operations, the greater the business value of delivered solutions and
capabilities.
Common objectives of DevOps include delivering capabilities with a
faster time to market (or speed-to-value), reducing the failure rate of new
releases, minimising the lead time between fixes, improving system quality, and achieving a faster mean time to recovery in the event of a new
release impacting the current system. Continuous Delivery is the method
by which such objectives can be met.
Continuous Delivery is the practice of software delivery through automation, providing the ability to rapidly, reliably and repeatedly deploy software capabilities to customers with minimal manual interaction, minimal
risk, and typically much higher quality.
With Continuous Delivery, we continuously deliver small, understandable
and reversible changes over time. Of course there is an overhead to this

Devops and continuing delivery
for oracle
4/5
Goran Stankovski
– you can no longer delay all testing until the end of the project; you are
now doing a deployment every week (even every day) instead of every
year. Therefore automated testing and deployment systems tend to be
inextricably linked to these methodologies.
There are a number of key DevOps and Continuous Delivery principles
that we believe are imperative in ensuring that the benefits of DevOps
and Continuous Delivery can be realised.
Embrace change. Change is good. Delivering capability to the business
increases business value. Being able to do this faster and better than your
competition provides a business differentiator that can be used to capitalise on opportunity.
Automate everything! Automation is a foundational building block in
DevOps and Continuous Delivery, providing a repeatable, reliable, and
higher quality outcome. The more manual interaction you have in your
system development and operation, the slower you will be able to adapt
to change.
Do not avoid the “too hard” basket. If something is “difficult” or “too
hard”, then do it more often – it will become easier over time. If something is hard to automate, this only increases the necessity to automate
it.

52 – OTech Magazine #6 – winter 2014

Keep everything in Source Control. Adopting an “Infrastructure-ascode” approach will allow you to define your infrastructure and platform
through code constructs and deliver it through automation. Storing your
entire platform and application configuration in source control systems,
such as GIT, will allow you to deliver consistent platforms and applications to support your business initiatives.
Large numbers of small changes are superior to small numbers of large
changes. Smaller, digestible changes are easier managed, and easily
rolled back.
Adopt the Continuous Improvement approach. Improving the Continuous Delivery and Automation capability over time will allow you to incorporate feedback into the process to deliver prolonged value.
So what about DevOps and Continuous Delivery in an Oracle context?
Oracle is considered an “Enterprise” class software. In Continuous Delivery and DevOps circles, enterprise software is viewed by many to be diametrically opposed to its key principles and tenants. It is inflexible, “hard
to work with”, delivered on a long software release cycle, and as a result
is slow to catch up with industry trends and emerging standards.
This misalignment in worldview tends to have many DevOps and Continuous Delivery initiatives place enterprise software, such as Oracle, in the
“too hard” basket and leave it for a later project to worry about. Many

Devops and continuing delivery
for oracle
5/5
Goran Stankovski
enterprises and businesses, however, rely on such enterprise software
for a majority of their business capability, and as a result, the business
value of DevOps or Continuous Delivery initiatives delivered excluding
such enterprise software is significantly diminished.

take into account in order to deliver this capability? What does this mean
with respect to Oracle Infrastructure, Virtualisation, Database, Middleware, and Applications technologies?
Subsequent articles in the series will focus on this aspect.

Adopting Continuous Delivery and DevOps within your enterprise requires a fundamental change to your technical project delivery capability, which will require maturity to progress it within the organisation. In
our opinion, DevOps and Continuous Delivery capability maturity can be
viewed in the following three distinct phases:

1.

Destructible Development Environments

The foundational capability in DevOps and Continuous Delivery initiatives
is the ability to deliver destructible development environments. How
do you deliver consistent environments through a repeatable and predictable process? How do you deliver this across Oracle Infrastructure,
Virtualisation, Database, Middleware, and Applications?
This will be the topic of our next article in the series.

2.

DevOps Lifecycle Management

As your maturity progresses, the next step in the evolution is bringing
lifecycle and release management capability through your DevOps and
Continuous Delivery framework. How do you adopt DevOps principles in
your organisation’s Dev/Test lifecycle? What key considerations must you

53 – OTech Magazine #6 – winter 2014

3.

DevOps and BAU

The final phase in the DevOps maturity is bringing the capability into
business as usual (BAU). Considerations such as configuration change
management and drift detection are critical in this phase. This is the most
difficult phase to adapt, as there is typically a huge separation and disconnect between the Development and Operations BAU teams in many
organisations. Most of these issues are typically organisational and not
technical in nature. Trust plays a key part. How can BAU trust what is
delivered? How can we bridge this divide?
Over the next 4 issues of this magazine, we will be delivering articles that
focus on each of the phases, and share our experience and recommendations in delivering such DevOps and Continuous Delivery capabilities to
organisations that leverage Oracle technology.

Deliver your environments
10X faster
Holistic approach for
public & private clouds
End-to-end provisioning from
infrastructure to applications
Configure it once and
build it many

Continuous delivery
automation for Oracle
Infrastructure. Databases. Middleware. Applications.

Design, configure, manage and deploy your Oracle
environments with speed, quality and consistency.
find out more @
www.environmint.com

54 – OTech Magazine #6 – winter 2014

Lonneke Dikmans
www.eproseed.com

Choosing the
right Mobile
Architecture

55 – OTech Magazine #6 – winter 2014

choosing
the right mobile architecture
2/9
Lonneke Dikmans
Deciding you need to create a mobile app is a no brainer these days. However, once this decision is made the next one pops up immediately: what
architecture should I chose? Native? Web? Hybrid? In this article you will
learn about the differences between these three architectures and the
impact the mobile architecture has on your service architecture, on security and last but not least on scalability. Three use cases are described to
guide you through the decision process.

Mobile Architecture types
There are three types mobile architecture:

do that, you have access to device specific capabilities like the camera.
Web applications are the opposite of that; they reside on the server and
are accessed using a browser. This makes them independent of device
type and OS version. You are limited with respect to the features you can
use by the capabilities of the browser and HTML5, but can use any tool
that supports web applications.
Hybrid apps are a mix between Native apps and Web apps. You can
download them from the App store, or Google Play etc. Some parts are
programmed in the native app language; some parts are residing on the
server and are shown using embedded web views. You can access both
Native apps and Hybrid apps online and offline, as long as everything you
need is available offline. Web apps you can access online only.
The table below summarizes the most important features of the three
architectures.
Native
 App
 
Resides
 on
 the
 device
 
Download
 from
 ‘App
 store’
 
Native
 views,
 depending
 on
 
device
 and
 OS
 version
 
Access
 to
 all
 device
 features
 

Illustration 1. Mobile architecture types

Native apps are specifically built for a specific device and depend on the
specific operation system (OS) version. For example, you can build an
iPhone app for iOS 8.2 using XCode to program Objective C. When you

56 – OTech Magazine #6 – winter 2014

Online
 and
 offline
 

 

Web
 App
 
Resides
 on
 the
 server
 
URL
 
Browser,
 depending
 
on
 Browser
 version
 
Limited
 functionality,
 
depending
 on
 HTML5
 
Online
 

Hybrid
 App
 
Device
 and
 server
 
Download
 from
 ‘App
 store’
 
Native
 views
 and
 Embedded
 
Browser
 
Access
 to
 all
 device
 features
 
Online
 and
 Offline
 

Oracle Mobile Application Framework offers capabilities to build native
apps, hybrid apps and web apps. Apart from that you can easily access
SOAP or REST services and you can send and receive push messages.

choosing
the right mobile architecture
3/9
Lonneke Dikmans

Illustration 2. Overview of Oracle Mobile Application Framework
(source: Oracle Mobile Application Framework datasheet)

As you can imagine, maintaining native apps for different devices and
different versions of the OS is expensive and cumbersome. That is why
Oracle Mobile Application Framework is very powerful: you can build
your application once and deploy it to different devices. Apart from the
device type, it allows you to choose the size of the device and the OS version. You can declaratively define icons and other device specific features
for the entire app. At deploy time these will be generated into your native
code.

Illustration 3. Creating native apps from one code base

Of course this does not take away the burden of deployment or releasing
for different platforms, but at least you can work from the same codebase.

Impact on your services
Mobile architecture is not just important for the app itself; it impacts your
service landscape as well.

57 – OTech Magazine #6 – winter 2014

choosing
the right mobile architecture
4/9
Lonneke Dikmans
Protocol

Granularity

Traditionally organizations have created services using SOAP and WSDL
for integration purposes between systems. This can be cumbersome
when creating graphical user interfaces for the web. This is even more
the case when creating mobile applications. REST is a more lightweight,
natural fit for mobile apps. There are two flavors that are used in REST:
REST/ XML and REST/JSON. The table below shows the most important
differences between the different styles. More and more tools support
JSON: you can transform JSON objects to XML using SOA Suite or OSB,
you can store native JSON objects in the database, etc. This means that
JSON slowly becomes language agnostic, like XML.

When you think about the granularity of your services, you have to take
the form factor (how big is the screen on my device and how do I access
it or input data) into account.
When you design for an iPad, you have almost as much ‘real estate’ as on
a laptop.

SOAP
 
Based
 on
 operations
 
Described
 by
 WSDL
 
Coarse
 grained
 
Message
 security
 plus
 
‘regular’
 web
 
application
 security
 (in
 
case
 of
 HTTP)
 
Platform
 and
 language
 
agnostic
 

 

REST
 /
 XML
 
Based
 on
 objects
 
Described
 by
 WADL
 
Fine
 grained
 (micro-­‐services)
 
‘regular’
 web
 application
 
security
 

REST/JSON
 
Based
 on
 objects
 
Described
 by
 WADL
 
Fine
 grained
 (micro-­‐services)
 
‘regular’
 web
 application
 
security
 

Platform
 and
 language
 
agnostic
 

Platform
 agnostic,
 
programming
 language
 
specific
 (JavaScript)
 

It depends on the framework you are using how easy it is to consume
SOAP or REST services. In general REST is more flexible and easier to use
in mobile applications.

58 – OTech Magazine #6 – winter 2014

Illustration 4. Difference in available ‘real estate’, or room, for data on a screen

However, when you are designing an app for a mobile phone or smart
watch, this changes rapidly. Typically, a mobile phone is chatty in terms of
service communication: it calls small services often, because there is less
room for data on the screen and because data is often paid by volume.

choosing
the right mobile architecture
5/9
Lonneke Dikmans
The following items determine the granularity of the services you need:
• Form factor. For smaller screens (both in device-size and resolution)
you want small services to avoid using to much bandwidth or memory
• On route versus stationary. When you are on route (walking to your
car) you spend less time on a transaction compared to when you are
stationary (sitting at home on the couch)
• Hands free or not. When you are driving, you need voice input. When
you are on the couch you can type
• Online versus Offline. If data needs to be cached you need bigger
services to make sure you fetch as much as you can before you lose the
connection.

Reuse versus specific services
As you have seen in the previous two paragraphs, the need of mobile app
as service consumer differs per type of mobile app, based on the form
factor (size/device) and context.
You can solve that in two ways: reuse your current services and filter the
data in your app, or create specific services for the different contexts.

Illustration 5. Reuse versus specific mobile services

Because in a mobile web app the code is residing on the server, it is relatively easy to reuse the services you already built in the backend and filter
the data in the web application, before they get sent over the wire. You
can create a responsive application that behaves differently based on the
device that the browser is running on.
When you are building a native app you probably want to filter the data
on the server, building a specific layer of services in the backend (‘presentation’ services). This avoids unnecessary delays and sending over too
much data to your device. The exception to this guideline is when you
need to cache a lot of data on your mobile device; in that case you probably want to call coarser grained services.
Note that in both cases you can reuse the original services in your backend and you should make sure they contain the business logic, to prevent
copying logic to multiple places

59 – OTech Magazine #6 – winter 2014

choosing
the right mobile architecture
6/9
Lonneke Dikmans
Security considerations
When you decide to create a mobile app using native or hybrid
technology, there are a number of things you need to take into account.
This includes:
• Network security: encryption;
• Mobile Device Management (MDM) versus Mobile Application
Management (MAM);
• Service side management: authentication and authorization of your
services in the back end and management of the services or API.
Let’s take a look at mobile device management, mobile application
management and service management:
MDM
 
Secure
 device
 password
 
VPN
 between
 device
 and
 
server
 
Wipe
 entire
 device
 
Track
 device
 
Native
 and
 hybrid
 
Dedicated
 devices
 

 

MAM
 
Secure
 container
 password
 
Secure
 tunnel
 between
 app
 
container
 and
 server
 
Wipe
 application
 
Track
 application
 
Native
 and
 hybrid
 
BYOD
 

Service
 management
 
Secure
 service
 
Transport
 level
 security
 
using
 SSL
 (for
 example)
 
Protection
 against
 DOS
 
Track
 service
 use
 
Web
 App
 
BYOD
 

Oracle offers mobile security suite to take care of your security considerations for Enterprise grade mobile apps by ‘containerizing’ them. On top
of that you can secure and manage your services (using API gateway)
and Oracle Identity and Access Management feature. This is shown in the
figure below.

60 – OTech Magazine #6 – winter 2014

Illustration 6. Oracle Mobile Security Suite
(source: Extending Enterprise Access and Governance with Oracle Mobile Security)

Scalability
The introduction of mobile devices has lead to a huge increase in load in
services that used to be very stable. Examples of this are:
• Bank apps to check your balance. In the past people would get a bank
statement periodically; the bank was in control of the information.
Then Internet banking and ATMs entered the arena, allowing people to
check their balance when getting cash or when behind the computer at
home. With the introduction of mobile banking, customers can check
their balance before they make any major or minor purchase. This has
increased the load on the backend banking systems greatly.
• Travel apps to check flight times. A similar effect has happened with
train and plane schedules. People always carry their mobile device and
check times frequently when traveling. This has increased the load on
the backend systems.

choosing
the right mobile architecture
7/9
Lonneke Dikmans
There are several solutions to handle the increase in load:
• Cache the response. Data can be cached on the device or on the service bus or API manager. This decreases the load if the same customer
checks the same data repetitively.
• Load balance the service in the backend. The servers in the backend
can be scaled out, so the additional load can be balanced over multiple
servers.
• Smaller services (less data) for mobile use. Depending on the underlying backend architecture, creating smaller micro-services might decrease the load on the system because there is no excess of data being
sent to devices.

Field engineer fixing a utility meter

Use cases

In this case, the main reason to choose a native app, was the need to be
able to work offline and cache data. Field engineers work in areas where
there is sometimes no connection to the Internet. Because the device is a
special purpose device, mobile device management was sufficient:
• If a field engineer leaves the job, or looses the device the entire device
is wiped clean.
• There are no other apps on the device than the ones installed on their
by the utility company

Now that you have seen some considerations you have to take into account when you decide to build a mobile app, let’s take a look at some
practical examples.

61 – OTech Magazine #6 – winter 2014

Illustration 7. Choosing the right mobile architecture: Field engineer fixing a meter

choosing
the right mobile architecture
8/9
Lonneke Dikmans
Judge reading court files

Engineer looking for a temp job

Illustration 8. Choosing the right mobile architecture: Judge reading case files

Illustration 9. Choosing the right mobile architecture: Engineer looking for a temp job

When people work from home, they are mostly working online. For
judges, security demands are high and they’d like to read files on a tablet. The solution is an hybrid app that works the similar to the browser
version for the desktop. The functionality is residing on the server. The
organization has chosen Mobile Device Management, because they hand
out the devices. There is no BYOD device policy in place. If there had been
a BYOD policy, MAM would have been better.
Specific presentation services are created for specific channels. There are
special services for mobile (smart phones). For the tablet, the services for
the Browser are reused.

In another case, field engineers who travel the world are looking for
their next job. They have time when at the airport, in a cab or on a train.
Usually they are connected (by Wi-Fi) and security demands for the application are relatively low. Jobs are posted and people can post resumes
on the site. In this case a web-app is chosen, using OAuth. Presentation
services were created to cater for the different form factors and to make
sure that the app is fast enough.

62 – OTech Magazine #6 – winter 2014

Summary and conclusion
Everybody is using his or her device all the time nowadays. When building
enterprise grade mobile apps, you need to take into account what type of
architecture you are going to use: native, hybrid or webapp. The biggest
difference between these three architectures is where the logic or app

choosing
the right mobile architecture
9/9
Lonneke Dikmans
resides: on the device or on the server. Depending on the device features
you need access to and the connectivity demands, one of the three is the
best fit.
Because we are talking about enterprise grade apps, security is an important feature. When choosing native or hybrid apps, you need to think
about whether you want to use Mobile Application Management or
Mobile Device Management. In general Mobile Application Management
is more fine-grained and more user friendly. It is very suitable for BYOD
situations. The mobile security suite integrates with Oracle Identity and
Access Management, so that your mobile app can truly be a part of your
enterprise solution. Apart from Mobile Application or Device Management, you need to manage the APIs that your mobile app calls; often the
APIs are used by multiple (even third party) apps and web applications
and you don’t want to rely on the diligence of the app programmer for
the security of the services and APIs.
When building mobile applications, the services in your organization are
impacted too. There are a number of design choices you have to make:
• Stick with SOAP services or move to REST services;
• Building an extra layer of services to cater for specific mobile app
needs (caching, smaller micro-services etc.) or solve this in the app
itself;
• Scale out or cache to cater for the increase in load.

63 – OTech Magazine #6 – winter 2014

Using tools to help solve these issues is one part of the equation. However, making the right architectural decisions is the critical success factor that determines whether your company will be successful in taking
advantage of the possibilities that arise from modern technologies.

Biju Thomas
www.oneneck.com

Flashback Empowering
Power Users!

64 – OTech Magazine #6 – winter 2014

flashback empowering power users!
2/9
Biju Thomas
Most of the time when a developer or application administrator comes
to the database administrator with a table restore request, the standard
answer is “No!” We cannot blame the DBA for the answer either, because the answer is based on the time it requires to restore a table from
the full backup. If the database is few terabytes big and if there are not
many tablespaces in the database, the DBA has to restore a subset of
the database (system, undo tablespaces and the tablespace holding the
table) to a temporary location, export the table, and import to the target database. It is not usual for the DBA to have the space sitting around
to create the temporary database for export. So, the process gets even
longer, starting with identifying the tablespaces to restore, requesting
space, identifying the backups (full, incremental, archive log), performing the restore and recovery, and exporting the table. Let us learn what
alternatives are available for power users themselves, without bothering
the DBA… Sometimes we get lucky, especially if the damage was done in
the last few hours or minutes.

Introduction to Flashback
Oracle introduced the automatic undo management in version 9i of the
database. Ever since the flashback operations are available to retrieve
the pre-modification state of the data in the database. Figure 1 shows the
database version and the flashback feature introduced.

65 – OTech Magazine #6 – winter 2014

Figure 1: Flashback Features by Release

As you can see in Figure 1, there are many flashback features in Oracle Database. Few require DBA privileges, few have configuration requirement
before you can start using, and few are enabled by default and available
for non-administrators. The flashback features that help to query and
restore data as it existed at a prior timestamp on individual tables are
available (or should be made available) to the developers and application

flashback empowering power users!
3/9
Biju Thomas
administrators. Having the knowledge and privilege helps developers
and power users to act quickly to resolve the issue before the undo data
disappears.

SQL> select * from hr.employees where employee_id = 106;
no rows selected
SQL> SELECT SYSTIMESTAMP FROM DUAL;

Let us review the flashback features for power users (and developers)
and how to use them.

SYSTIMESTAMP
---------------------------------------------------07-DEC-14 03.31.22.982933 PM -05:00

The Time Machine API!

SQL> EXECUTE DBMS_FLASHBACK.ENABLE_AT_TIME(TO_TIMESTAMP(‘07-DEC-14 11:30’,’DD-MON-YY
HH24:MI’));

The DBMS_FLASHBACK pl/sql package was introduced in Oracle9i Release 1 (yes, almost 15 years ago!!), when the automatic undo management was introduced in the Oracle Database. This is one of the powerful
and easy to use package to go back in time, and still available in Oracle
Database 12c.
The “back to the future” time point in the database can be enabled by
using the DBMS_FLASHBACK.ENABLE_AT_TIME or DBMS_FLASHBACK.
ENABLE_AT_SYSTEM_CHANGE_NUMBER procedure. Both procedures
enable flashback for the entire session. Once enabled, the queries in the
session will be as of the SCN or timestamp enabled. One big advantage of
using this package is to run pl/sql programs on data in the past.
The following code shows setting the flashback 4 hours back, and running
queries to retrieve some data accidentally deleted from the EMP table.

PL/SQL procedure successfully completed.
SQL> select count(*) from hr.employees where employee_id = 106;
COUNT(*)
---------1
SQL>

When a session is running in flashback mode, no DDL or DML is allowed.
Only queries can execute. Following code shows the error when I try to
run DML or DDL while in flashback mode.
SQL> insert into hr.employees
2 select * from hr.employees where employee_id = 106;
insert into hr.employees
*
ERROR at line 1:
ORA-08182: operation not supported while in Flashback mode
SQL> create table hr.emp_restore as select * from hr.employees where employee_id =
106;

66 – OTech Magazine #6 – winter 2014

flashback empowering power users!
4/9
Biju Thomas
create table hr.emp_restore as select * from hr.employees where employee_id = 106
*
ERROR at line 1:
ORA-08182: operation not supported while in Flashback mode

SQL> exec dbms_flashback.disable;

LOOP
FETCH frompast INTO pastrow;
EXIT WHEN frompast%NOTFOUND;
insert into hr.employees values pastrow;
END LOOP;
END;
/
PL/SQL procedure successfully completed.

PL/SQL procedure successfully completed.

SQL> select count(*) from hr.employees where employee_id = 106;

SQL> select * from hr.employees where employee_id = 106;
no rows selected

COUNT(*)
---------1

SQL>

SQL> commit;

To end flashback mode, you can use the DISABLE procedure.

So, how do you retreive data using this method? The answer is PL/SQL.
Let me show you with an example how to retrieve the deleted employee
row using pl/sql. In the code, we open the cursor after enabling flashback, but disable flashback before start fetching. COMMIT is not included
in the pl/sql code. So, after execution, we confirm if the row is inserted,
and commit the record.
DECLARE
cursor frompast is
SELECT * FROM hr.employees
where employee_id =106;
pastrow hr.employees%rowtype;
BEGIN
DBMS_FLASHBACK.ENABLE_AT_TIME(TO_TIMESTAMP(‘07-DEC-14 11:30’,’DD-MON-YY HH24:MI’));
OPEN frompast;
DBMS_FLASHBACK.DISABLE;

67 – OTech Magazine #6 – winter 2014

Commit complete.
SQL>

This is too much work to retrieve data. Enabling flashback for the session
is good to find information and run various queries. To actually restore
back the changes we should look into other flashback methods.

Flashback Query
Flashback query was introduced in Oracle9i Release 2. This is the best and
fastest way to save or retrieve information as it existed in the past. Flashback query is enabled by using the “AS OF TIMESTAMP” or “AS OF SCN”
clause in the SELECT statement.

flashback empowering power users!
5/9
Biju Thomas
SQL> select count(*) from hr.employees where employee_id = 106;
COUNT(*)
---------0
SQL>
SQL> select count(*) from hr.employees
as of timestamp
TO_TIMESTAMP(‘07-DEC-14 11:30’,’DD-MON-YY HH24:MI’)
where employee_id = 106;
COUNT(*)
---------1

If you do not want to restore the row directly into the table, you can save
the data in a table. In fact, it is a good practice to save the data in a table,
as soon as you know that some unintended data update or delete performed.
SQL> create table hr.employees_restore pctfree 0 nologging as
2 select * from hr.employees
3 as of timestamp
TO_TIMESTAMP(‘07-DEC-14 11:30’,’DD-MON-YY HH24:MI’)
4 where employee_id = 106;
Table created.
SQL>

SQL>

The required row is still available, to get it back is easier, because the
session is not running in flashback mode, just the subquery is in flashback
mode.
SQL> insert into hr.employees
2 select * from hr.employees
3 as of timestamp
TO_TIMESTAMP(‘07-DEC-14 11:30’,’DD-MON-YY HH24:MI’)
4 where employee_id = 106;

And remember, if you found the data still available for retrieve and want
to replace the entire table (by doing a truncate and insert) or want to
disable a constraint, do not attempt to do that before saving the data
to a temporary location. If you perform any DDL on the table, flashback
queries will not work anymore on that table.
Here is an experience. You can see here the DBA being proactive, tried to
disable the primary key before doing the insert, and…
SQL> select count(*) from bt1;

1 row inserted.
COUNT(*)
---------70

SQL>

SQL> select count(*) from bt1 as of timestamp to_timestamp(‘07-DEC-14 15:15’,’DDMON-YY HH24:MI’);

68 – OTech Magazine #6 – winter 2014

flashback empowering power users!
6/9
Biju Thomas
COUNT(*)
---------98
SQL> alter table bt1 disable primary key;
Table altered.
SQL> insert into bt1 select count(*) from bt1 a as of timestamp to_timestamp(‘07DEC-14 15:15’,’DD-MON-YY HH24:MI’) where a.cid not in (select b.cid from bt1 b);
ERROR at line 1:
ORA-01466: unable to read data - table definition has changed
SQL>

In such situations, always save the data in a temporary table, do the DDL
on the table, and insert from the temporary table. The following operations on the table prevent flashback operations.
• moving or truncating
• adding or dropping a constraint
• adding a table to a cluster
• modifying or dropping a column
• adding, dropping, merging, splitting, coalescing, or truncating a
partition or subpartition (except adding a range partition).
To use Flashback operations, you require FLASHBACK and SELECT
(or READ) privilege on the table.

When there are many tables or if the table is big, it might be easier for
you to save the data as export dump file, as soon as you hear about an
unintended data manipulation. You can use of the expdp parameters
FLASHBACK_SCN or FLASHBACK_TIME to perform the export as of a
specific timestamp.
These parameters along with QUERY parameter helps you to further refine and filter the data exported. For example, to export the table to save
the deleted employee 106, we can use the following parameters.
dumpfile=exprestore.dmp
logfile=emprestore.log
tables=hr.employees
flashback_time=”to_timestamp(‘07-DEC-14 16:45’,’DD-MON-YY HH24:MI’)”
query=”where employee_id = 106”

Flashback Table
If you are very certain that the entire table must be reverted back to a
state in the past, you can use the Flashback Table feature. For the flashback table to work, the row movement must be enabled. The following
example shows how to rollback a table to prior time.
SQL> FLASHBACK TABLE bt1 TO TIMESTAMP to_timestamp(‘07-DEC-14 16:45’,’DD-MON-YY
HH24:MI’);
ERROR at line 1:
ORA-08189: cannot flashback the table because row movement is not enabled
SQL> alter table bt1 enable row movement;

Flashback in Export
69 – OTech Magazine #6 – winter 2014

Table altered.

flashback empowering power users!
7/9
Biju Thomas
SQL> FLASHBACK TABLE bt1 TO TIMESTAMP to_timestamp(‘07-DEC-14 16:45’,’DD-MON-YY
HH24:MI’);
Flashback complete.
SQL>

Where is the Flashback Data Source
So, where does the flashback data come from? Remember, in the beginning I mentioned about automatic undo management. It is a requirement
for flashback operations to have automatic undo management. Automatic undo management is enabled by setting the parameters:
UNDO_MANAGEMENT=AUTO → Since Oracle Database 11g, AUTO is the
default setting.
UNDO_TABLESPACE=<tablespace_name> → specify the undo tablespace
for the instance. The default is the first available undo tablespace when
the instance starts.

is relevant here, the UNDO_RETENTION. This parameter provides a guidance to the database on how long the committed undo should be kept in
the undo segments.
Since flashback operations are very useful feature of the database, the
database tries to keep the undo records as long as possible, till there is
not enough room in the undo tablespace, even past the value specified in
seconds for UNDO_RETENTION (default is 900 seconds). The retention
behavior depends on the tablespace data file AUTOEXTENSIBLE property.
If the undo tablespace is autoextend, database tries to honor the time
specified in UNDO_RETENTION. If this duration is smaller than the longest running query in the database, undo is tuned to accommodate the
longest running query. When free space is low, instead of overwriting
unexpired undo (committed and within the retention period) record, the
tablespace auto-extends.

When any DML is run to insert, update, delete data from table, the undo
record is written to the rollback (undo) segment. This record is mainly
used by Oracle database to provide a read consistent view of the data. This
data is also used to rollback the transaction and to run flashback queries.

If the undo tablespace is not autoextensible, the database ignores the
UNDO_RETENTION parameter. Oracle tunes the undo retention to keep
transactions as old as possible, to keep the undo tablespace usage below
the warning alert threshold. When there is space pressure, unexpired
undo is overwritten.

The duration of undo data available for the database depends on the size
of the undo tablespace. Once the transaction is committed, the undo
data is not required for the database operation, but is used by long running queries and flashback operations. There is one other parameter that

Oracle Undo Advisor provides the size required for undo tablespace
to keep specified amount of undo retained. Undo advisor is available
in DBMS_ADVISOR pl/sql package, but I prefer using Oracle Enterprise

70 – OTech Magazine #6 – winter 2014

flashback empowering power users!
8/9
Biju Thomas
manager, which is much easier. Figure 2 shows the Undo Advisor screen
of OEM. We can tell OEM to analyze the workload of the database for the
past so many days and specify a desired undo retention, then the advisor
will give the required size of the tablespace.

Other Flashback Features
There are other flashback features that depend on undo data. They will
come handy in some situations.
Flashback Versions Query: This feature was added in Oracle Database
10gR1. The VERSION BETWEEN [TIMESTAMP | SCN} … AND … clause in
the SELECT statement provides the all changes to the record within the
timeframe. This is useful to audit changes.

Figure 2: Undo Advisor

It is best to configure the undo tablespace appropriately sized and fixed
size.
Note: If flashback operations and queries are important to you, you may
ask the DBA to enable guaranteed retention. This is not a recommended
configuration for production databases, as DML operations might fail if
there is not enough free space, because oracle will not overwrite unexpired undo, until the time specified in UNDO_RETENTION is reached. Guaranteed undo retention is enabled for the undo tablespace, by setting the
RETENTION GUARANTEE clause using the ALTER TABLESPACE statement.

71 – OTech Magazine #6 – winter 2014

Flashback Transaction Query: This feature was also added in Oracle
Database 10gR1. Shows the changes made to data at a transaction level,
without using LogMiner. You also get to see the SQL to undo the change.
For this feature to work, the database must have supplemental logging
enabled using “ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;”
Flashback Transaction: This feature was added in Oracle Database 11gR1.
As the name suggests, helps you to rollback a transaction and its child
transactions. This is primarily a DBA feature, also requires setting up database with supplemental log data on primary key columns. Using OEM
interface for transaction backout is much easier and error free.

flashback empowering power users!
9/9
Biju Thomas
Accidentally dropped a table? Don’t worry! The table is available in the
recycle bin, you can get it from there… (provided your DBA has not
changed the default behavior of the database by adjusting the parameter
RECYCLEBIN=OFF). This operation does not use undo, thus the dropped
table will be available in the recycle bin until there is space pressure on
the tablespace where the table used to reside.
SQL> DROP TABLE job_history;
Table dropped.
SQL> SHOW RECYCLEBIN
ORIGINAL NAME RECYCLEBIN NAME OBJECT TYPE DROP TIME
-----------------------------------------------------------------------JOB_HISTORY BIN$ZcGwtQ/sKCbgQAB/AQBl2g==$0 TABLE 2014-1207:23:27:54
SQL> FLASHBACK TABLE job_history TO BEFORE DROP;
Flashback complete.

72 – OTech Magazine #6 – winter 2014

Conclusion
As a developer or Power user, Oracle gives you the opportunity to recover from unintended mistakes in data changes. The requirement is
automatic undo management, which is the default since Oracle 11g. It is
also important to size the undo tablespace appropriately to retain undo
data for certain period of time. Since Oracle 10g, when a table is dropped,
it is kept in the recyclebin, and can be restored. Recyclebin feature is also
default since 10g. Discuss with your DBA, what you can do and the undo
retention settings. If you are a DBA reading this article, it is important for
you to show the developer or power user what they can do to query and
save data from the past… might save you an unnecessary restore of the
database or table!

Lucas Jellema
www.amis.nl

The Rapid
Reaction Force
– real time
business
monitoring

73 – OTech Magazine #6 – winter 2014

the rapid reaction force real time business monitoring
2/16
Lucas Jellema
Oracle offers a product that is almost like a hidden gem. Not because it is
hidden – because if it is, it is hidden in plain view – but because we have
collectively failed to grasp the true potential of this product. Oracle Event
Processor (OEP) is typically associated with real time processing of low
level events aka Fast Data. Typical examples of use cases involve stock
tick processing, Tweet-analysis and Internet of Things signal evaluation.
These are valid use cases – but perhaps a little exotic. There many examples closer to home where OEP can deliver excellent value.
This article will introduce the product OEP, the concepts and the underlying technology. It will introduce the new StreamExplorer tool that was
announced at Oracle OpenWorld 2014, soon to be released. The integration of OEP with companion components in SOA Suite 12c and the WebLogic platform is described. A simple example of using OEP for interpreting a real time stream of messages is discussed. And the article will
discuss how OEP can be useful in very ordinary circumstances.
Note that you may very well already have the right to make use of OEP:
OEP is included in the SOA Suite 12c (and 11g). Any organization with a
license for SOA Suite therefore can make use of OEP without additional
investment. Additionally, for use in IoT cases, Oracle offers an embedded
edition of OEP (Oracle Event Processing for Oracle Java Embedded) for a
very low per-device fee and the assumption that this edition will run on a
large number of small devices. OEP can also be purchased as stand-alone
product.

74 – OTech Magazine #6 – winter 2014

Introducing Oracle Event Processor
Before getting into technical details, we should first take a look at what
events and event processing entail and what the OEP can do in fairly conceptual terms.
Event is a somewhat vague term used to indicate a fairly small piece of
data associated with a timestamp that is published in a fire-and-forget
style. The publisher of the event volunteers the data and wants nothing
in return. In fact, the publisher does not even want to know who will consume the data. In general, the data published is not apparently valuable
and may only have some value by association with other – previous or
future – events. An event can represent the reading of a sensor, a tweet
or the latest value of a stock but it can also represent a step in a business
process, a user action on a web site, an update of a database record or
the execution of a business service. Note that the absence of an event
can also be [made] meaningful: the fact that something that was expected, did not materialize [in time] may represent value.

Key characteristics
OEP is a light weight component a with a fairly small footprint (one of the
reasons it can be used in the embedded use case for example on a device
the size of a Raspberry Pi). It holds relevant data in memory (or on grid);
most of the processing does not require I/O operation, allowing OEP to
respond very fast.

the rapid reaction force real time business monitoring
3/16
Lucas Jellema
OEP is not weighed down by transactions and synchronous interactions.
It operates in a very decoupled fashion – fire and forget - , both on the
receiving as well as on the publishing end.
The product uses smart algorithms for enrichment, filtering, partitioning,
aggregation and pattern detection. Note that most operations will take
place across collections of events, very much like SQL operations operate
on sets of relational database records.
OEP is triggered into action either by the reception of an event or by
internal timers. The latter are used to monitor deadlines before which an
event should have been received or to drive the periodic publication of
aggregates and current status updates.

Interactions
Messages are typically read from JMS—for example, a queue in WebLogic – or the SOA Suite Event Delivery Network (EDN). Tables can also
be used as an event source, and through custom adapters, we can consume messages from virtually any source, including files, sockets, NoSQL
database and JMX, Oracle Messaging Cloud Service, Web Sockets, RSS
feeds and other HTTP streams. The OEP server provides an HTTP Pub/Sub
event channel based on the Bayeux protocol; this allows OEP to consume
messages pushed from a Bayeux-compliant server.

75 – OTech Magazine #6 – winter 2014

The outcomes from event processing are published to similar, decoupled
destinations: JMS, EDN, tables, HTTP Pub/Sub Channels and Web Socket
channels. A common companion for OEP is BAM (Business Activity Monitoring) for final processing and dashboard style visualization of event
based findings, notifications and recommendations.

Stages and operations in Event processing
The objective of an OEP application is to get from fine grained data
events to relevant information packages. This is typically done in several
steps:
• Filter - Finding the needles in the haystack, those rare events that
indicate exceptions

the rapid reaction force real time business monitoring
4/16
Lucas Jellema
• Enrich - Adding context and thereby (business) meaning to events,
such as associating an event with GPS coordinates to a shop, bus stop,
restaurant, service vehicle or another meaningful location
• Aggregate - Counting events or determining a sum, an upper or lower
limit, or an average over a metric that the events carry in their payload.
Note: An aggregate is frequently calculated over a window that consists of all events in the specific timeframe or a specified number of the
latest events.
• Pattern detection - Spotting a trend by comparing correlated events
over a period of time, finding a special sequence of values or value
changes that is known to herald a certain incident, and discovering the
absence of an event that should have occurred are all examples of the
detection of meaningful patterns among a string of correlated events.
• Promote or propagate - The result of the preceding operations can
be to propagate the event, either as is or in an enriched or promoted
state, or to publish entirely new events that carry an aggregated value,
some indicator, or an alert.

76 – OTech Magazine #6 – winter 2014

It is quite common that the output from an OEP application is fed into
another OEP application to perform the next iteration of refinement,
filtering and upgrading to an even coarser grained event with even more
business value. An example of this is the combination of an embedded
OEP that processes high frequency local sensor signals into less frequent
aggregate values covering larger areas that are sent to centralized OEP
processors for pattern analysis and aggregation across locations.

Usage Scenario
The core strengths of OEP seem to be:
• aggregation of low level signals
• cross instance (or conversation) monitor
• real time interpreter
in a fast, lean, continuous and decoupled fashion that can easily be integrated with existing applications and engines.
In addition to all obvious fast data use cases – high volume of messages
that requires real time processing, such as social media, IoT, network
packets and stock tick feeds – there are use cases within closer reach for
most of us. Consider OEP the monitor that can look and analyze across
threads and sessions, at transactional or non-transactional actions, at
middle tier (JMS, HTTP) and database (HTTP push/pull or JDBC/SQL pull)
in a decoupled way that hardly impacts the operations you would like to
keep track of.

the rapid reaction force real time business monitoring
5/16
Lucas Jellema
• the execution of business process is orchestrated by engines such
as BPEL and BPM with contributions from various systems and components; the start and completion of each step in a process can be
reported in an event. OEP can easily be used to spot process instances
that are interrupted or that skip over a step (missing event). When
there is an order dependency between process instances or “a limit on
the number parallel instances of a certain process”, OEP can be used
to monitor these. A decrease in performance for a certain resource and
the build-up of a waiting queue can be detected as well
• the navigation and actions of users through a web application can be
reported; through OEP, we can easily get insight in the numbers of
current sessions and the activities in each of them. We can learn about
search terms that result in desired follow-up steps and outcomes and
frequently used terms that do not end in success. We can learn about
the step at which users typically abort their actions (by detecting the
missing event) as well as the steps that take long or short.
• when you consider certain combinations of activities suspicious, then
OEP is a perfect tool to detect instances of suspicious behavior and
allow immediate intervention; all it takes is publish an event for each
activity that can be part of a chain of actions that indicates something
fishy could be going on and – the hardest part – describe the patterns
to look out for. Examples are multiple concurrent logins from the same
IP address, activities at times or from locations that are quite different
from a user’s normal behavior, logins from the same user from entirely
different geographical locations (within a short period of time).

77 – OTech Magazine #6 – winter 2014

• OEP is straightforward to setup for detecting sudden peaks or gradual
increases – for example in web traffic, system usage, response times,
error rates and it can easily partition by specific characteristics.
Any type of behavior that can be represented by simple messages can be
presented to OEP for processing. Large volumes and high urgency may
be involved, but that is not necessarily the case. Looking for patterns,
keeping track, maintaining current status and doing so across systems
and instances is the core of OEPs offering. And for this, there are bound
to be many more use cases.

CQL – The Continuous Query Language
The core of OEP is formed by the CQL engine. CQL is an acronym for Continuous Query Language, one of most prominent Event Processing Languages. Event processors need to be programmed. They are like query
engines that need to query streams of events—not static sets of records,
but incessantly changing collections of data granules. Event queries are
not executed just once. They are instead attached to an event stream and
continuously executed. Every new arriving event may cause the event
query to produce a new result. Results are constantly being published in
the form of events.

the rapid reaction force real time business monitoring
6/16
Lucas Jellema
in a single CQL query.
Here is a simple example of a CQL query that produces an output event
whenever the number of input events in the someEventStream with a
payload value larger than 15, changes, considering a time window that
ranges over the last five seconds:
select count(payloadValue) as countEvents
from someEventStream [range 5]
where payloadValue > 15

CQL was inspired by, derived from and overlaps with SQL. The widespread knowledge of SQL can be leveraged when programming event
processing queries. CQL is allows us to combine event streams and relational data sources in a single query (for example, to join historical and
reference data with the live event feed).
As an aside: The Match_Recognize operator introduced in Oracle Database 12c to perform pattern matching on relational data sets makes use
of the CQL syntax to construct the pattern query.
CQL queries select from an event channel, often with some range or time
window applied, using a where clause to filter on the events that are returned by the query. The select clause refers to event properties and uses
functions and operators to derive results. Note that multiple channels
and other data sources can be joined – yes: even outer joined – together

78 – OTech Magazine #6 – winter 2014

The number of results produced by this query cannot be predicted: it
depends on the frequency and irregularity with which events appear in
the event stream.

Technology Overview
The Oracle Event Processor is part of the SOA Suite license. However, it
is not part of the SOA Suite SCA container—it does not even run in the
same WebLogic Server as the SOA Suite does. It runs on its own streamlined, lightweight server—which is POJO based, founded on Spring DM
and an OSGi-based framework to manage services. This server comes
with Jetty, an HTTP container for running servlets, and support for JMS
and JDBC. It has caching and clustering facilities—optionally backed by an
Oracle Coherence grid. OEP can handle thousands of concurrent queries
and process hundreds of thousands of events per second. The average
message latency can be under 1 ms.

the rapid reaction force real time business monitoring
7/16
Lucas Jellema
The OEP server provides an HTTP Pub/Sub event channel based on the
Bayeux protocol; this allows the OEP application to consume messages
pushed from a Bayeux-compliant server as well as allows clients to subscribe to messages published by the OEP server. Clients can also publish
to an HTTP channel without having to be explicitly connected to the HTTP
Pub/Sub server.
Event messages are typically processed by CQL processors—nodes in
the event processing network (EPN) that receive messages from stream
sources and act on them to produce new event messages. The outcome
of a CQL query is typically a stream of events.
OEP provides additional facilities, such as the ability to record events
flowing through an event processing network and store them. They can
later be played back from the event repository – very useful for testing.
The OEP Visualizer is a web based administration console that enables
you to view, develop, configure, and monitor aspects of Oracle Event
Processing applications. OEP applications can be deployed, paused and
resumed, and uninstalled from the Visualizer. Current operations can be
monitored, including all console output. It also has support for visual and
run-time CQL Query editing. Events published on an HTTP Pub/Sub channel can be monitored in the Visualizer, asynchronously pushed by the
server to the browser.

79 – OTech Magazine #6 – winter 2014

Airport Car Park Management using Event Processor
Let us take a look at a concrete example. We are borrowing here from
the case of Saibot Airport in Lexville that is used throughout the Oracle
SOA Suite 12c Handbook (Oracle Press). This airport makes a lot of money
from the car parks around the airport. Close monitoring and careful management of the parking process is important for that reason and because
it is an important aspect in the evaluation by travelers in airport surveys.
Indicating at the right time that a car park is full, prevents people from
needlessly driving in circles to find a free spot. However, too early warnings mean that people are unnecessarily forced to park their cars on the
more remote car parks – not necessarily improving their experience.
We will create an OEP application to process all events signaling a car entering or leaving a car park and thus deriving the number of cars currently
inside the car park. The application will publish a special event when the
car park is declared full. and it will publish a similar event when spaces
become available again.
OEP 12c applications are developed in JDeveloper (unlike OEP 11g which
used its own IDE plugin for Eclipse). These applications are deployed to
and run on an instance of the OEP server. This server is not part of the
integrated WebLogic Server and it is separate from the SOA Suite 12c run
time environment based on WebLogic. The OEP runtime is an OSGi server
that needs to be installed separately.

the rapid reaction force real time business monitoring
8/16
Lucas Jellema
Getting Started
A new JDeveloper application is created based on the template OEP Application, called SaibotParking. An OEP project CarParking is created inside
the application – based on the Empty OEP project template. The application is created with an empty Spring context file called CarParking.context.xml – which will contain definitions of beans, events, channels and
processors - and a file called processor.xml that is used to store the CQL
queries. The EPN Diagram visualizes the Event Processing Network that
is defined in the context file. The MANIFEST.MF file finally contains OSGi
definitions – for the application itself and all its bundle dependencies.
Events in OEP can be defined entirely declaratively or based on a Java
bean (a POJO). We will adopt the latter approach, using a Java Class CarMovementEvent which is a POJO with the properties carparkIdentifier,
entryOrExit, licencePlate and arrivalTime. Based on this POJO, an OEP
event type is defined. This represents the input to the application: cars
arriving at or departing from one of the car parks.
A second POJO represents the results from OEP: conclusions about the
status of the car parks. This POJO has properties carparkIdentifier, carCount, percentageFull and averageStayDuration.
The application contains a Java Class that generates the CarMovement
events. This class generates at a somewhat random rate car arrival events
with randomly generated licence plates. Later, more advanced incarna-

80 – OTech Magazine #6 – winter 2014

tions of this class will derive car exit events as well and will generate
events covering five different car parks. This generator is used during
development. When the event processing has been developed and verified, the generator can be replaced by an inbound adapter that reads the
events from JMS, an HTTP channel or some other source.
The output from the OEP application can be reported through outbound
adapters to external receivers, such as a JMS destination, an RMI client, a
Socket endpoint or the Event Delivery Network. During development and
for testing purposes, it is frequently convenient to work with an internal
event receiver. A simple Java Class that implements the StreamSink is be
used to receive the outcomes from the event processor and writes them
to the console. Java Class CarParkReporter implements that interface and
writes simple logging lines based on the events it receives

The Event Processing Network
The Event Processing Network or EPN ties together the event sources
– in this case the event generator -, the CQL event processors and the
StreamSinks that handle the outbound operations. In between the components are channels that carry events of a specific type.

the rapid reaction force real time business monitoring
9/16
Lucas Jellema
The flow from the input channel directly to the stream sink carParkReporter means that each and every car movement is channeled to the Java
Class, without any processing at all. The real intelligence of this network
is implemented in the carMovementProcessor that processes car movement messages using the following CQL statement:
IStream(select
`

sum(car.entryOrExit) as carCount, car.carparkIdentifier
from
carMovementInputChannel as car

Correlation and Missing Event Detection
OEP can do much more than just count and aggregate. It can for example
correlate events, detect ‘missing’ events and derive business intelligence
from them. Chaining OEP processors – within or across applications – is
commonly done to derive varying conclusions from a certain base set of
data signals at different levels of granularity. We will use several processors inside our application.

group by carparkIdentifier)

This query produces an update on the current number of cars per car
park. CQL will publish a result in this case whenever the previous result is
superseded by a new one – i.e. with every new car that enters the parking
lot and every car that leaves. Note: the value of property entryOrExit is -1
for an exit and +1 for an entry of a car.

In this section, we determine for cars that leave how long they have
stayed in the car park; we could use that to automatically calculate the
parking fees. We next extend our summary with the average stay duration of cars. We look for cars that have out-stayed their welcome: our car
parks are short stay only, intended for parking up to 36 hours. Cars that
have been in the car park for more than 48 hours become candidate for
towing.

Such an update of the car count is not really required for every car that
enters. We may want to settle for a summary update every five seconds.
This is easily accomplished by rewriting the from clause as:

Correlate events

from

carMovementInputChannel [range 1 DAY slide 5 second] as car

Here we have specified that the results from the query are to be calculated over one day worth of [input] events and should be produced once
every five seconds.

81 – OTech Magazine #6 – winter 2014

One thing OEP excels at is correlating events. It can find patterns across
events that arrive at various points in time. In this example, we will have
OEP associate the arrival and exit events for the same car. These events
together constitute the ‘car stay’ for which we could calculate the parking fee. By taking all these car stay events into consideration, we can
have OEP calculate the average stay duration – per car park.
The key CQL operator we leverage to correlate events into meaningful
patterns is called MATCH_RECOGNIZE. The query used in the carStay-

the rapid reaction force real time business monitoring
10/16
Lucas Jellema
DoneProcessor uses the following clause:
... from carInputChannel
match_recognize (
partition by carparkIdentifier, licencePlate
measures A.licencePlate as licencePlate
,
A.carparkIdentifier as carparkIdentifier
,
A.arrivalTime as arrivalTime
,
B.element_time - A.element_time as stayduration
pattern (A B)
define
A as entryOrExit= 1
, B AS entryOrExit = -1
) as stayed

The crucial element in a MATCH_RECOGNIZE expression is the pattern
that should be detected in the event stream. In this case, the pattern
consists of events A and B that both apply to the same carpark and car
(through the licenceplate) as defined by the partition by clause. Additionally, for this one car, A is the arrival event (entryOrExit equals 1) and B
the subsequent exit event. Whenever these two events have been found
– with no other events in between with different values for the entryOrExit attribute – the query will produce an event of type CarStayDone. The
stayduration measure is calculated using the pseudo column element_
time . Every event is timestamped by OEP in such a way that the difference between the element_time value for two events is equal to the time
lapse (in nano seconds) between the two events.

82 – OTech Magazine #6 – winter 2014

Using the CarStayDone events published to the carstayDoneChannel, it
becomes quite simple to derive the CarParkEvents that report the average stay duration for each car park – with an update published whenever
a car leaves a car park. The CQL snippet below takes care of this. Note
that the average duration in this case is calculated over the last 100 cars
that have left the car park.
<query id=”CarparkAvgStayQuery”><![CDATA[
IStream(select -1 as carCount
,
carparkIdentifier as carparkIdentifier
,
‘’ as description
,
avg(carstayDuration as averageStayDuration
from carstayDoneChannel [rows 100]
group by carparkIdentifier
)
]]></query>

The EPN is a little bit more complex at this point.

the rapid reaction force real time business monitoring
11/16
Lucas Jellema
Detect missing events and find abandoned cars
As discussed before, frequently the absence of an expected event is what
we should detect and what should result in a new business event to be
published. In our case of airport parking, the missing event is the car that
does not leave our short stay car park in time.
The event generator has been extended to also come up with cars that
will not depart any time soon. Our OEP application should be able to detect such cars – that do not leave within 120 seconds (on the time scale of
our sample application). See the sources in project CarParking_step3 for
this step.
A new event was defined – AbandonedCarEvent – based on a POJO with
that name. Java Class AbandonedCarsMonitor implementing the StreamSink interface was created to receive these events and write a log message
for them. The abandonedCarsProcessor was added to process the events
from the carInputChannel, detect the abandoned cars and report on them
to the abandonedCarsOutputChannel, as shown in the next figure.
The next CQL snippet is at the heart of the processor.
from carInputChannel
match_recognize (
partition by carparkIdentifier, licencePlate
measures A.licencePlate as licencePlate
,
A.carparkIdentifier as carparkIdentifier
,
A.arrivalTime as arrivalTime

83 – OTech Magazine #6 – winter 2014

INCLUDE TIMER EVENTS
pattern (A NotExit*)
duration 120 seconds
define
A as entryOrExit= 1
, NotExit AS entryOrExit!= -1
) as abandoned

Again, we inspect the CarMovementEvents on the carInputChannel. We
are looking for the pattern A – entry of a certain car – followed by the
NotExit event – which has been defined as an event for which the entryOrExit value is not equal to -1. A car that has entered the car park can do
nothing than at some point leave that car park again generating an event
with entryOrExit = -1 at that point. And if it does, it is not abandoned.
OEP allows us to mix in system generated timer events with the regular
event stream – using the INCLUDE TIMER EVENTS instruction, combined
with the clause DURATION 120 SCONDS. These tell OEP that in every partition of the event stream – for every licence plate and car park combination for which we have seen a first CarMovementEvent – a system event
should be added every two minutes if and when in there has not been
a real event in that two minute timeframe . Subsequently, when such a
timer event is added for a certain car – because for 120 seconds there has
not been a real event, which should have been the ‘car exit’ event – the
pattern will match for that car because the system generated timer event
does not have entryOrExit equals -1. In fact, timer events do not have attribute values at all.

the rapid reaction force real time business monitoring
12/16
Lucas Jellema
Credit Card Theft Detection
Another interesting use case for OEP concerns the shopping area at
Saibot Airport, where a series of credit card thefts have taken place.
Upon investigation, it became clear that on each occasion, the perpetrator stole the credit card in the crowded main terminal area, made his way
towards the exit and in between, within 15 minutes, made three or more
purchases with the credit card in the range of $200-$500. He sometimes
back traced a little, entering a shop a little further from the main exit than
the previous one.
OEP is set up to process a stream of credit card purchase events, that
contain the amount, the credit card number and the shop identifier. With
OEP, the pattern described above and shown in the next figure is to be
detected. When it is found, it is to be reported in the form of an EDN
event to be taken care of by a SOA composite.

The salient aspects of this CreditCard processing application:
• Use of the listAgg operator to aggregate multiple events into a Java
Collection
• Use of custom Java Classes from within CQL queries
• Use of XMLParse to add XML content inside XML Element with child
elements, based on a string [prepared in a custom Java class from a
List of shop identifiers and purchase amounts] to produce richer, more
complex EDN event payloads
• Use of keywords within and subset in the CQL query to respectively
find a pattern only if it occurs within a certain period of time (15 minutes) and to allow aggregation over all or some of the events that
constitute the pattern.
This next snippet shows some crucial aspects of the CQL query that finds
fishy transactions:
select its.creditCardNumber as creditCardNumber,totalSum as sumAmounts
, delta as shoppingDuration, shops as shops, purchaseAmounts as purchaseAmounts
from creditCardMidRangePurchaseChannel
MATCH_RECOGNIZE (
PARTITION BY creditCardNumber
MEASURES C.creditCardNumber as creditCardNumber
, sum(allPurchases.amount) as totalSum
, C.element_time - A.element_time as delta
, listAgg(allPurchases.shopIdentifier) as shops
, listAgg(allPurchases.amount) as purchaseAmounts
PATTERN ( A+ B? A+ C) within 30000 milliseconds
SUBSET allPurchases = (A,B,C)
DEFINE
A as A.shopIdentifier >= A.shopIdentifier

84 – OTech Magazine #6 – winter 2014

the rapid reaction force real time business monitoring
13/16
Lucas Jellema
, B as B.shopIdentifier < A.shopIdentifier and A.shopIdentifier B.shopIdentifier < 7
,C as A.shopIdentifier - C.shopIdentifier < 7
) as its

The messages produced by the OEP application are published to a JMS
queue. An ADF web application was developed to demonstrate how
these findings could be visualized to notify human actors. The next figure
shows how the ADF DVT component custom thematic map was used to
visualize the locations and relative sizes of all transactions in a suspicious
train of events.

85 – OTech Magazine #6 – winter 2014

StreamExplorer
Stream Explorer is a tool – first showcased at Oracle OpenWorld 2014 targeted at the Line of Business User (the non-technical IT consumer).
This tool provides a visual, declarative, browser based wrapper around
Oracle Event Processor and Oracle BAM. With Stream Explorer (assumed
to be available close to the turn of the year) it is every easy to create explorations and dashboard on live [streams of]data – reporting in real time
on patterns, correlations, aggregations and deviations. As Oracle puts it:

the rapid reaction force real time business monitoring
14/16
Lucas Jellema
“The Oracle Stream Explorer empowers Business Users, anywhere, for RealTime, Instant Insight of Fast Data Streaming Data Analysis. It is designed
for SaaS on the Oracle Cloud. Requiring NO knowledge of Real Time Event
Driven Architectures, the Standards-based Continuous Query Language
(CQL), the high performance Java Event Server or the semantics of Event
Stream Processing Application Models.”
The following figure shows a screenshot of the Stream Explorer user
interface. The business user can make use of many predefined templates
that are categorized by industry. These templates represent specific ways
of processing events.

86 – OTech Magazine #6 – winter 2014

The business user defines Streams for the input. Such a Stream can be a
(CSV) file, a HTTP Subscriber, EDN event, a JMS destination, a Database
Table or a REST service. These streams carry shape instances. A shape is
a record (event) definition that consists of the properties and their types.
The Streams can perhaps be preconfigured for the LoB user by the tech
savvy IT colleague. The business user can then define Explorations on the
Streams, that are based on predefined templates – as shown in the next
figure.,

the rapid reaction force real time business monitoring
15/16
Lucas Jellema
The business user has to provide values for template parameters such as
the aggregation interval, the aggregation function or the event property
to group by. Multiple explorations can be chained; this means that the
outcome of one exploration can provide the input for a next.
Downstream destination to which the outcome of an exploration can
be sent are a CSV file, an HTTP Publisher, EDN, a JMS destination, a REST
service or BAM – the Business Activity Monitor.

Summary

StreamExplorer is a web application that is installed in the Jetty Server
that also runs the OEP engine and the OEP Visualizer Admin Console.
OEP applications configured using StreamExplorer can be exported to
be imported and refined |extended | augmented in JDeveloper by the IT
specialists. The StreamExplorer applications contain an Event Processing
Network and CQL based event processors that were constructed using
templates.

Event processing applications use CQL – the continuous query language
that is similar in syntax to SQL and that can join event channels, caches
and relational data sources to produce coarser grained, higher value
output events. OEP connects on both inbound and outbound edges with
SOA Suite and Java applications through JMS, Coherence and the Event
Delivery Network. By focusing on the core competencies of OEP – light
weight, multi-channel interaction, cross instances|sessions|transactions|
conversations – it is easy to see how for example OEP can play an important role for monitoring of and across business processes. Note that the
core of the OEP engine has been embedded inside Oracle BAM to help
with precisely that.

The near future of StreamExplorer holds interaction with NoSQL sources
and destinations, BAM visualizations, support for machine learning algorithms, data redaction on sensitive data, Coherence cache integration and
location based features. This will make StreamExplorer – and OEP – more
powerful and more accessible for use in every-day situations instead of
only in the corner cases Complex Event Processing was typically considered for.

87 – OTech Magazine #6 – winter 2014

Real time findings – insight, alert, action - based on vast and possibly fast
data from a variety of sources – published to various types of channels
for onwards processing is in a nutshell what Oracle Event Processor can
do for us. By constantly scanning streams of events arriving at unpredictable moments, OEP is able to look out for abnormal conditions, meaningful patterns, relevant aggregates and even important event absences.

Compared to the 11g stack, development for OEP 12c is much easier
thanks to the integration in JDeveloper. The support for the Event Delivery Network in the SOA Suite means that it is easier still to implement
EDA (Event Driven Architecture) with SOA Suite or at least make the
service architecture more event-enabled. Finally, the new StreamExplorer

the rapid reaction force real time business monitoring
16/16
Lucas Jellema
product will most likely further lower the threshold for starting to leverage event processing.

Resources
The source code for the Saibot Airport Car Park Management example
discussed in this article can be downloaded from GitHub.
Also in GitHub are the sources for the CreditCard Theft case.
A presentation with a quick overview of event processing and how OEP
can be used (presented at Oracle OpenWorld 2014) is available here.
Example of using the Oracle Database 12c Match_Recognize clause to do
CQL-like pattern matching on relational data.

88 – OTech Magazine #6 – winter 2014

otech partner:

For over 22 years, AMIS has been the leading Oracle technology partner in the Netherlands. AMIS consultants are involved in major Oracle implementations in the Netherlands and in various ground breaking projects worldwide. Based on a solid business
case, we bring the latest technology into actual production for customers .
AMIS’s reputation for technological excellence is illustrated by the AMIS technology
blog. This blog services over 5.,000 visitors daily and is in the top 3 of the world’s most
visited Oracle technology blogs. There are 3 Oracle ACE’s in the AMIS team, including 1
ACE Director, who make regular contributions to the worldwide Oracle community.
AMIS is an Oracle Platinum Partner and was selected as the EMEA Oracle Middleware
partner of the year award in 2014, and in Holland in 2013 and 2011.
AMIS delivers expertise worldwide. Our experts are often asked to:
- Advise on fundamental architectural decisions
- Advise on license-upgrade paths
- Share our knowledge with your Oracle team
- Give you a headstart when you start deploying Oracle
- Optimize Oracle infrastructures for performance
Click here for
- Migrate mission-critical Oracle databases to cloud based
our
review on
infrastructures
OOW 2014
- Bring crashed Oracle production systems back on-line
- Deliver a masterclass

89 – OTech Magazine #6 – winter 2014

Patrick Barel

Lucas Jellema

www.amis.nl

www.amis.nl

Amis
Edisonbaan 15
3439 MN Nieuwegein
+31 (0) 30 601 6000
[email protected]
www.amis.nl
www.twitter.com/AMIS_Services
https://www.facebook.com/AMIS.Services?ref=hl
specializations: Oracle ADF 11g, Oracle Application Grid 11g
Oracle BPM 11g, Oracle Database 11g, Oracle Enterprise
Manager 12c, Oracle SOA Suite 11g, Oracle RAC 11g, Oracle
Weblogic Server 12c, Exalogic

Mahir M Quluzade
www.mahir-quluzade.com

Oracle
Database
In-Memory
(Part I)

90 – OTech Magazine #6 – winter 2014

oracle database in-memory
part i
2/8
Mahir M Quzulade
Database In-Memory is new and powerful feature of Oracle Database
coming with Oracle Database 12 Release 1 (12.1.0.2) which is published
July 2014. This article series covers this amazing feature of Oracle Database.

Introduction
Usually we are using analytics on Data Warehouses (DWH) which is
stored Online Transactional Processing (OLTP) systems data. Analytics is
complex queries running on very large tables of DWH. But DWHs is not
running in real-time as OLTP. Oracle Database In-Memory (IM) can help
run analytics and reports in real-time OLTP databases. Also Oracle Database In-Memory supports both DWH and mixed workload OLTP databases.
Figure 1: Data Block and Row Piece format

Database Block
Oracle Database has stored data as row format in other words as multicolumn records in data blocks on the disk (Figure 1). In a row format database, each new transaction or record stored as a new row in the table.
A row format is ideal format for OLTP databases, as it allows quick access
to all of the columns in a record since all of the data for a given record are
kept together in memory and on disk, also incredibly efficient for processing DML.

91 – OTech Magazine #6 – winter 2014

It is not changed in latest release of Oracle Database, changes only in
memory architecture it means changes only in Instance.

Instance Changes
An Oracle Instance contains Memory and set of Background processes.
Memory divided in two different areas: System Global Area (SGA) and
Program Global Area. Oracle creates server processes to handle the
requests of user processes connected to the instance. One of the important tasks of Sever Processes: read data blocks of objects from data files
into the database buffer cache. By default, Oracle store data in the data-

oracle database in-memory
part i
3/8
Mahir M Quzulade
base buffer cache as row format. Starting in Oracle Database 12c Release
1 (12.1.0.2) added new optional static pool into SGA called In-Memory area
and objects stored as new format In-Memory Column Store (IM column
store) in this area. IM Column store is an optional that stores copies of
tables, partitions, columns of tables, materialized views (objects specified
as INMEMORY using DDL) a special columnar format optimized for rapid
scans. The database uses special techniques, including SIMD vector processing, to scan columnar data rapidly. The IM column store is a supplement to rather than a replacement for the database buffer cache. The
IM column store does not replace the buffer cache. But both memory
areas can store the same data in different formats (Figure 2) and it is not
required for objects populated into the IM column store to be loaded into
the database buffer cache, in other words objects can populate only into
IM column store.

Figure 2: Dual format store in memory

92 – OTech Magazine #6 – winter 2014

Enable In-Memory Column Store
Starting in Oracle Database 12c Release 1 (12.1.0.2) added new six initialization parameters starting with INMEMORY_ prefix controlling InMemory functionality (Code Listing 1). Additionally new OPTIMIZER_INMEMORY_AWARE initialization parameter enables or disables all of the
optimizer cost model enhancements for in-memory. The default value is
TRUE. Setting the parameter to FALSE causes the optimizer to ignore the
in-memory property of tables during the optimization of SQL statements.
Code Listing 1: Initialization Parameters for In-Memory
SQL> show parameter inmemory
NAME

TYPE
VALUE
------------------------------------------------ ----------------- -------inmemory_clause_default string
inmemory_force string DEFAULT
inmemory_max_populate_servers integer 1
inmemory_query string ENABLE
inmemory_size big integer
0
inmemory_trickle_repopulate_servers_ integer 1
percent
optimizer_inmemory_aware boolean TRUE

INMEMORY_SIZE sets the size of the In-Memory Column Store (IM column
store) on a database instance and the default value is 0, which means
that the IM column store is not used, so In-Memory feature is not automatically enabled. We need change INMEMORY_SIZE initialization parameter to non-zero amount for enable IM Column store. If you want this

oracle database in-memory
part i
4/8
Mahir M Quzulade
amount not take from current SGA size, then you must extend SGA size
to account for the INMEMORY_SIZE parameter value. So if you are using
Automatic Memory Management (AMM), then you will need to extend
MEMORY_TARGET parameter value or if you are using Automatic Shared
Memory Management (ASMM), then you will need to extend SGA_TARGET parameter value.
In my case using AMM and memory maximum target is equal 1024MB and
setting the INMEMORY_SIZE parameter to 300MB (Code Listing 2).
Code Listing 2: Enable In-Memory functionality

In-Memory Area
Database mounted.
Database opened.

318767104 bytes

SQL> alter system set memory_target=1324M;
System altered.
SQL> select * from v$sga;
NAME
VALUE
CON_ID
-------------------- ---------- ---------Fixed Size
2924304
0
Variable Size
939524336 0
Database Buffers
117440512
0
Redo Buffers
13852672 0
In-Memory Area 318767104
0

SQL> alter system set memory_max_target=1324M scope=spfile;
System altered.

In-Memory Population

SQL> alter system set inmemory_size=300M scope=spfile;

Only objects specified INMEMORY attribute in DDL populate into
IM Column Store.

System altered.
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 1392508928 bytes
Fixed Size
2924304 bytes
Variable Size
1040187632 bytes
Database Buffers
16777216 bytes
Redo Buffers
13852672 bytes

93 – OTech Magazine #6 – winter 2014

Figure 3: INMEMORY clause

oracle database in-memory
part i
5/8
Mahir M Quzulade
You can define INMEMORY clause to specific groups of columns, whole
tables, materialized views or table partitions (Code Listing 3).
Code Listing 3: IM Column Store for tables and Materialized Views

Code Listing 4: IM Column Store for tablespaces
SQL> create tablespace TBSIM1 datafile ‘/u01/app/oracle/oradata/prmdb/tbsim1.dbf’ size
10M default inmemory;
Tablespace created.

SQL> conn mahir/mahir
Connected.
SQL> create table TBIM1 (id number, d date) INMEMORY;
Table created.
SQL> create table TB1 as select level as id from dual connect by level<=1e6;
Table created.
SQL> create materialized view MVIM_TB1 INMEMORY as
2 select * from tb1;
Materialized view created.
SQL> select table_name, inmemory from user_tables;
TABLE_NAME
INMEMORY
------------------ ---------------TBIM1
ENABLED
TB1
DISABLED
MVIM_TB1
ENABLED

SQL> select tablespace_name, def_inmemory from user_tablespaces;
TABLESPACE_NAME
DEF_INME
------------------------------ -------SYSTEM
DISABLED
SYSAUX
DISABLED
UNDOTBS1
DISABLED
TEMP
DISABLED
USERS
DISABLED
CMDATA
DISABLED
TBSIM1
ENABLED
SQL> create table TBIM2 as select * from dba_objects;
Table created.
SQL> alter table TBIM2 INMEMORY;
Table altered.
SQL> create tablespace TBSIM2
2 datafile ‘/u01/app/oracle/oradata/prmdb/tbsim2.dbf’ size 10M ;
Tablespace created.

However, you can enable IM Column Store for tablespaces. So, all tables
and materialized views in the tablespace are automatically enabled for the
IM column store. Also you can enable IM Column store for all objects and
tablespace after creation with ALTER … DDL command (Code Listing 4).

94 – OTech Magazine #6 – winter 2014

SQL> alter tablespace TBSIM2 default INMEMORY;
Tablespace altered.

oracle database in-memory
part i
6/8
Mahir M Quzulade
IM Column Store enabled by default only new created objects in tablespace, if IM Column Store enabled for tablespace. In other words it is
not effective for old objects of tablespace.
Populated objects to In-Memory stored into two pools: a pool size of 1MB
used to store the actual column formatted data and a pool size of 64KB
used for store metadata about populated objects. We can check amount
of available memory for each pool from V$INMEMORY_AREA view (Code
Listing 5).
Code Listing 5: Amount of available memory in In-Memory area

Memory Compression (inmemory_memcompress) clause specified compression method for data stored in the IM Column store. The default is
NOMEMCOMPRESS, means no compresses in-memory data. To instruct
the database to compress in-memory data, specify MEMCOMPRESS FOR
followed by one of the following methods (Table 1):
Table 1: Compression methods
Compression
(inmemory_
memcompress)

Description

DML

This method is optimized for DML operations and
performs little or no data compression

QUERY LOW 

This method compresses in-memory data the least
(except for DML) and results in the best query
performance. This is the default

QUERY HIGH 

This method compress in-memory data more than 
QUERY LOW, but less than CAPACITY LOW

CAPACITY LOW 

This method compresses in-memory data more
than QUERY HIGH, but less than CAPACITY HIGH,
and results in excellent query performance

CAPACITY HIGH 

This method compresses in-memory data the most
and results in good query performance.

SQL> select * from v$inmemory_area;
POOL
ALLOC_BYTES USED_BYTES POPULATE_STATUS
-------------------------- ----------- ---------- -------------------------1MB POOL
250609664 6291456
DONE
64KB POOL
46137344
262144
DONE
INMEMORY Parameters

We can specify how table data is stored in IM Column store using IM
parameters. Parameters in four types: Memory Compression, Priority,
Distribute and Duplicate parameters.

Recommendation: You can use Oracle Compression Advisor - DMBS_
COMPRESSION built-in PL/SQL package for an estimate of the compression ratio that can be use for MEMCOMPRESS.

95 – OTech Magazine #6 – winter 2014

oracle database in-memory
part i
7/8
Mahir M Quzulade
Code Listing 6: Using MEMCOMPRESS clause
SQL> alter table TBIM1 inmemory memcompress for capacity high;

Table 2: In-Memory priorities
Priority (inmemory_priority)

Description

NONE 

Oracle Database controls when table data is populated
in the IM column store. Population of data might be
delayed if the memory required for the data is needed
for other tasks. Also, Oracle Database might populate
some table columns, but not others, in the IM column
store. This is the default

LOW 

Table data is populated in the IM column store before
data for database objects with priority NONE and after
data for database objects with priority MEDIUM, HIGH,
or CRITICAL

MEDIUM 

TABLE_NAME INMEMORY
INMEMORY_COMPRESS
------------- ------------ -------------------TBIM1 ENABLED
FOR CAPACITY HIGH
TB1 DISABLED
MVIM_TB1 ENABLED FOR CAPACITY LOW
TBIM2 ENABLED FOR QUERY LOW
TBIM3 ENABLED FOR QUERY HIGH NONE

Table data is populated in the IM column store
before data for database objects with priority NONE or LOW and after data for database objects
with priority HIGH or CRITICAL

HIGH 

Table data is populated in the IM column store before
data for database objects with priority NONE, LOW,
or MEDIUM and after data for database objects with
priority CRITICAL

5 rows selected.

CRITICAL 

Table data is populated in the IM column store
before data for database objects with priority NONE, LOW, MEDIUM, or HIGH

Table altered.
SQL> alter table MVIM_TB1 inmemory memcompress for capacity low;
Table altered.
SQL> alter table TBIM2 inmemory memcompress for query low;
Table altered.
SQL> create table TBIM3 (n number,v varchar2(1))
2 inmemory memcompress for query high;
Table created.
SQL> select table_name, inmemory, inmemory_compression from user_tables;

Priority (inmemory_priority) clause specified the data population priority
for data in the IM column store. You can specify the priorities data in IM
column store with following PRIORITY clause (Table 2):

96 – OTech Magazine #6 – winter 2014

I showed sample examples for using PRIORITY clause as below:
Code Listing 6

oracle database in-memory
part i
8/8
Mahir M Quzulade
Code Listing 6: Using PRIORITY clause

• Objects that are smaller than 64KB are not populated into memory

SQL> alter table TBIM1 inmemory priority low;
Table altered.
SQL> alter table TBIM2 inmemory priority high;
Table altered.
SQL> create table TBIM4 (n number, v varchar2(1), d date)
2 inmemory memcompress for dml priority low;
Table created.
SQL> select table_name, inmemory, inmemory_compression,inmemory_priority
2 from user_tables;
TABLE_NAME INMEMORY
INMEMORY_COMPRESS
INMEMORY_PRIORITY
----------- -------------
----------------- ---------------TBIM1 ENABLED
FOR QUERY
LOW

LOW
TB1 DISABLED
MVIM_TB1 ENABLED
FOR CAPACITY
LOW
NONE
TBIM2 ENABLED
FOR QUERY
LOW

HIGH
TBIM3 ENABLED
FOR QUERY
HIGH

NONE
TBIM4 ENABLED
FOR DML

LOW

Also you can use IM column store on Logical Standby database but the
IM Column Store cannot be used on an Active Data Guard standby instance in the current release.

6 rows selected.

DISTRIBUTE (inmemory_distribute) and DUPLICATE (inmemory_duplicate) clauses using only with Oracle Real Application Clusers (RAC). You
can read about this clause in next part of this article series.
Restrictions on IM
IM Column Store has some restrictions:
• Index Organized Tables (IOTs) and Clustered Tables cannot be populated into IM Column Store.
• LONGS (deprecated since Oracle Database 8i) and Out of line LOBs
data types are also not supported in the IM Column store

97 – OTech Magazine #6 – winter 2014

Next Part
In next part of this article series you will read, In-Memory scans, joins and
In-Memory with RAC also Oracle SQL Optimizer with In-Memory.

Kim Berg Hansen
www.thansen.dk
Blog: http://dspsd.blogsport.com

Time Series
Forecasting
in SQL

98 – OTech Magazine #6 – winter 2014

time series forecasting
in sql
2/10
Kim Berg Hansen
One day our resident data analyst showed me an Excel spreadsheet and
said: “Look, Kim, I’ve taken monthly sales data of one of our items for
three years and created a Time Series Analysis model on it to forecast the
sales of the fourth year. Can you do it in SQL to apply the model on all
hundred thousand items?”
Well, I listened to her explanations about centered moving average,
seasonality, deseasonalizing, regression trends and reseasonalizing. And
I read this link http://people.duke.edu/~rnau/411outbd.htm that explains
almost the same model as she used. And yes, I could transform her Excel
to SQL. Here’s how…

insert into sales values (‘Snowchain’, date ‘2013-10-01’,
1);
insert into sales values (‘Snowchain’, date ‘2013-11-01’, 73);
insert into sales values (‘Snowchain’, date ‘2013-12-01’, 160);
insert into sales values (‘Sunshade’ , date ‘2011-01-01’,
4);
insert into sales values (‘Sunshade’ , date ‘2011-02-01’,
6);
insert into sales values (‘Sunshade’ , date ‘2011-03-01’, 32);
...
insert into sales values (‘Sunshade’ , date ‘2013-10-01’, 11);
insert into sales values (‘Sunshade’ , date ‘2013-11-01’,
3);
insert into sales values (‘Sunshade’ , date ‘2013-12-01’,
5);
The snowchain sells well every winter but hardly anything in the summertime and the
trend is that it sells more and more every year.
The sunshade on the other hand sells well in summer with a downward trend less and
less every year.

A couple of representative products
I’ll make a table containing montly sales for each item (product):
create table sales (
item varchar2(10)
, mth date
, qty number
);

For demonstration purposes I insert data for two items with seasonal
variations:
insert into sales values (‘Snowchain’, date ‘2011-01-01’, 79);
insert into sales values (‘Snowchain’, date ‘2011-02-01’, 133);
insert into sales values (‘Snowchain’, date ‘2011-03-01’, 24);
...

99 – OTech Magazine #6 – winter 2014

Figure 1: 2011-2013 sales for the two items

time series forecasting
in sql
3/10
Kim Berg Hansen
Time series
I start by creating a 48 month time series for each item:
select sales.item
, mths.ts
, mths.mth
, extract(year from mths.mth) yr
, extract(month from mths.mth) mthno
, sales.qty
from (
select add_months(date ‘2011-01-01’, level-1) mth
, level ts --time serie
from dual
connect by level <= 48
) mths
left outer join sales
partition by (sales.item)
on sales.mth = mths.mth
order by sales.item, mths.mth;

The inline view mths creates 48 rows with the months from 2011-01-01 to
2014-12-01 – the 3 years I have sales data for plus the 4th year I want to
forecast.
By doing a partition outer join on the sales table, I get 48 rows numbered
with ts=1..48 for each item having QTY NULL for the last 12 months:
ITEM
TS MTH
YR MTHNO QTY
---------- --- ------- ----- ----- ---Snowchain
1 2011-01 2011
1 79
Snowchain
2 2011-02 2011
2 133
Snowchain
3 2011-03 2011
3 24
...

100 – OTech Magazine #6 – winter 2014

Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
...
Snowchain
Snowchain
Sunshade
Sunshade
Sunshade
...

34
35
36
37
38
39

2013-10
2013-11
2013-12
2014-01
2014-02
2014-03

2013
2013
2013
2014
2014
2014

10
1
11 73
12 160
1
2
3

47
48
1
2
3

2014-11
2014-12
2011-01
2011-02
2011-03

2014
2014
2011
2011
2011

11
12
1
2
3

4
6
32

Centered moving average
I place the entire previous SQL in a with clause I call s1 and query that.
(This I will continue to do adding a new with clause for each step):
with s1 as (...)
select s1.*
, case when ts between 7 and 30
then
(nvl(avg(qty) over (
partition by item
order by ts
rows between 5 preceding and 6 following
),0) + nvl(avg(qty) over (
partition by item
order by ts
rows between 6 preceding and 5 following
),0)) / 2
else
null
end cma -- centered moving average
from s1
order by item, ts;

time series forecasting
in sql
4/10
Kim Berg Hansen
For moving average over 12 months I can use either from -5 to +6 months or
from -6 to +5 months (as seen in the two rows between clauses.) Centered
moving average then means to take the average of those two averages.
I only do that for those 24 months (ts=7..30=2011-07..2013-06) where 12
months of data is available:
ITEM
---------Snowchain
Snowchain
...
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
...
Snowchain
Snowchain
Snowchain
Snowchain
...

TS
--1
2

MTH
YR MTHNO QTY
CMA
------- ----- ----- ---- ------2011-01 2011
1 79
2011-02 2011
2 133

6
7
8
9
10

2011-06
2011-07
2011-08
2011-09
2011-10

2011
2011
2011
2011
2011

6
7
8
9
10

29
30
31
32

2013-05
2013-06
2013-07
2013-08

2013
2013
2013
2013

5
6
7
8

0
0
0
1
4

30.458
36.500
39.917
40.208

0 56.250
0 58.083
0
1

Seasonality
with s1 as (...), s2 as (...)
select s2.*
, nvl(avg(
case qty when 0 then 0.0001 else qty end / nullif(cma,0)
) over (
partition by item, mthno
),0) s -- seasonality
from s2
order by item, ts;

101 – OTech Magazine #6 – winter 2014

The qty divided by the cma gives the factor for how much the particular month sells compared to the average month. The model will fail for
months with qty=0 as that will mean some of the next steps get division
by zero or get wrong results due to multiplying with zero. Therefore any
0 sale I change to a small number.
Seasonality is then the average of this factor for each month. That is calculated by an analytic avg partitioned by item and month number. What
this means is, that the case expression finds the factor for the months
of July 2011 and July 2012 for a given item, and the portioning by month
number then finds the average of those two, and that average (column
alias s) will be in all 4 July rows for that item. Or in other words, seasonality will be calculated for the 12 months and be repeated in all 4 years:
ITEM
---------...
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain

TS MTH
YR MTHNO QTY
CMA
S
--- ------- ----- ----- ---- ------- ------9
10
11
12
13
14
15
16
17
18
19
20
21
22

2011-09
2011-10
2011-11
2011-12
2012-01
2012-02
2012-03
2012-04
2012-05
2012-06
2012-07
2012-08
2012-09
2012-10

2011
2011
2011
2011
2012
2012
2012
2012
2012
2012
2012
2012
2012
2012

9
1 39.917 .0125
10
4 40.208 .0774
11 15 40.250 .3435
12 74 40.250 2.5094
1 148 40.250 3.3824
2 209 40.292 4.8771
3 30 40.292 .7606
4
2 40.208 .0249
5
0 40.250 .0000
6
0 44.417 .0000
7
0 49.292 .0000
8
1 51.667 .0097
9
0 53.750 .0125
10
3 54.167 .0774

time series forecasting
in sql
5/10
Kim Berg Hansen
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
...

23
24
25
26
27
28

2012-11
2012-12
2013-01
2013-02
2013-03
2013-04

2012
2012
2013
2013
2013
2013

11 17 54.083 .3435
12 172 54.083 2.5094
1 167 54.083 3.3824
2 247 54.083 4.8771
3 42 54.083 .7606
4
0 54.000 .0249

Deseasonalized quantity
with s1 as (...), s2 as (...), s3 as (...)
select s3.*
, case when ts <= 36 then
nvl(case qty when 0 then 0.0001 else qty end / nullif(s,0), 0)
end des -- deseasonalized
from s3
order by item, ts;

Dividing the quantity by the seasonality factor gives us a deseasonalized
quantity that better shows the trend of the item sales with less disturbing
factors from seasonal variations in sales. Again I need to trick zero sales
into 0.0001.
ITEM
---------Snowchain
Snowchain
Snowchain
...
Snowchain
Snowchain
Snowchain
Snowchain

TS
--1
2
3
22
23
24
25

MTH
QTY
CMA
S
DES
------- ---- ------- ------- -------2011-01 79
3.3824 23.356
2011-02 133
4.8771 27.270
2011-03 24
.7606 31.555
2012-10
3 54.167 .0774
2012-11 17 54.083 .3435
2012-12 172 54.083 2.5094
2013-01 167 54.083 3.3824

38.743
49.490
68.542
49.373

102 – OTech Magazine #6 – winter 2014

Snowchain
...

26 2013-02 247 54.083 4.8771

50.645

Trend (regression)
Having the deseasonalized quantity I use regression to calculate a trend:
with s1 as (...), s2 as (...), s3 as (...), s4 as (...)
select s4.*
, regr_intercept(des,ts) over (partition by item)
+ ts*regr_slope(des,ts) over (partition by item) t -- trend
from s4
order by item, ts;

The two analytic functions calculate a linear regression of a graph line
with des on the Y axis and ts on the X axis. regr_intercept gives the point
where the regression line intersects the Y axis and regr_slope gives the
slope of the line. By multiplying the slope with ts (= months) and adding it
the interception point, I get the points of the trend line:
ITEM
---------Snowchain
Snowchain
Snowchain
...
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
...

TS
--1
2
3
34
35
36
37
38
39

MTH
QTY
CMA
S
DES
T
------- ---- ------- ------- -------- ------2011-01 79
3.3824 23.356 32.163
2011-02 133
4.8771 27.270 33.096
2011-03 24
.7606 31.555 34.030
2013-10
1
2013-11 73
2013-12 160
2014-01
2014-02
2014-03

.0774 12.914 62.976
.3435 212.518 63.910
2.5094 63.760 64.844
3.3824
65.777
4.8771
66.711
.7606
67.645

time series forecasting
in sql
6/10
Kim Berg Hansen
Viewed on a graph I can get a feeling about how perfect the seasonal
variations for the item is. The closer the deseasonalized graph is to a
straight line, the closer the item is to having the exact same seasonal pattern every year.
Small numbers near zero can give some hickups on the deseasonlized
graph (like late summer every year for the snowchain), but they tend to
switch to either side and even out, so the regression trend line still shows
the trend pretty well. If I had not used the 0.0001 trick in the code to
remove zero sales, there would have been more points on the red graph
going to zero and the trend line slope would have been lower and my
forecast too conservative.

Figure 2: Deseasonalization and trend of Snowchain

103 – OTech Magazine #6 – winter 2014

Reseasonalize (forecast)
Having calculated the trend line, I can re-apply the seasonality factor:
with s1 as (...), s2 as (...), s3 as (...), s4 as (...), s5 as (...)
select s5.*
, t * s forecast --reseasonalized
from s5
order by item, ts;
ITEM
---------Snowchain
Snowchain
...
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
...
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain
Snowchain

TS
--1
2

MTH
QTY
S
DES
T FORECAST
------- ---- ------- -------- ------- -------2011-01 79 3.3824 23.356 32.163 108.788
2011-02 133 4.8771 27.270 33.096 161.414

34
35
36
37
38

2013-10
1 .0774 12.914 62.976
4.876
2013-11 73 .3435 212.518 63.910 21.953
2013-12 160 2.5094 63.760 64.844 162.718
2014-01
3.3824
65.777 222.487
2014-02
4.8771
66.711 325.357

43
44
45
46
47
48

2014-07
2014-08
2014-09
2014-10
2014-11
2014-12

.0000
.0097
.0125
.0774
.3435
2.5094

71.380
.000
72.314
.700
73.247
.918
74.181
5.744
75.115 25.802
76.049 190.836

I got deseasonalized qty by dividing qty with seasonality factor. Then I
straightened out the deseasonalized qty into a trend line. And then in this
final step I multiply with the seasonality factor to get back a line shaped
like the original sales quantity:

time series forecasting
in sql
7/10
Kim Berg Hansen
The green reseasonalized graph matches the blue actual sales pretty well
for 2011-2013, and since it is calculated from the trend line, we can continue the green graph into 2014 (or further if we like) – it will keep having
the correct shape, just a bit “larger” or “smaller” as it follows the increasing or decreasing trend line.

Model describes reality?
I get a quick and dirty overview of model fit by analytic sum per year:
with s1 as (...), s2 as (...), s3 as (...), s4 as (...), s5 as (...)
select item
, mth
, qty
, t * s forecast --reseasonalized
, sum(qty) over (partition by item, yr) qty_yr
, sum(t * s) over (partition by item, yr) fc_yr
from s5
order by item, ts;
Figure 3: Reseasonalization (forecast) of Snowchain

For sunshade we can get a similar resulting graph:

Figure 4: Reseasonalization (forecast) of Sunshade

104 – OTech Magazine #6 – winter 2014

And I spot that model might not be perfect but definitely close:
ITEM
---------Snowchain
...
Snowchain
...
Snowchain
Snowchain
...
Snowchain
Snowchain
Snowchain

MTH
QTY FORECAST QTY_YR FC_YR
------- ---- -------- ------ ------2011-01 79 108.788
331 421.70
2012-01 148 146.687

582 556.14

2013-01 167 184.587
2013-02 247 270.710

691 690.57
691 690.57

2013-11 73 21.953
2013-12 160 162.718
2014-01
222.487

691 690.57
691 690.57
825.00

time series forecasting
in sql
8/10
Kim Berg Hansen
Snowchain
...
Sunshade
...
Sunshade
...
Sunshade
...
Sunshade
Sunshade
Sunshade
...
Sunshade
Sunshade
...

2014-02

325.357

825.00

2011-01

4

2.885

377 390.59

2012-01

2

2.418

321 322.75

2013-01

2

1.951

263 254.91

2013-05
2013-06
2013-07

23
46
73

23.645
54.579
51.299

263 254.91
263 254.91
263 254.91

2013-12
2014-01

5

3.764
1.484

263 254.91
187.07

Differences happen where the seasonal variations are not perfect or
where trend is not really linear or where numbers are too low. But it is
“good enough”.

Sales and forecast together
Having satisfied myself that the model is reasonably good, my end users
do not want to see a lot of superfluous information – they just need the
sales for 2011 to 2013 followed by the 2014 forecast:
with s1 as (...), s2 as (...), s3 as (...), s4 as (...), s5 as (...)
select item, mth
, case
when ts <= 36 then qty
else round(t * s)
end qty
, case
when ts <= 36 then ‘Actual’

105 – OTech Magazine #6 – winter 2014

else ‘Forecast’
end type
, sum(
case
when ts <= 36 then qty
else round(t * s)
end
) over (
partition by item, extract(year from mth)
) qty_yr
from s5
order by item, ts;

Which gives a nice and simple output:
ITEM
---------Snowchain
...
Snowchain
Snowchain
...
Snowchain
Snowchain
Snowchain
...
Snowchain
Snowchain
Sunshade
...
Sunshade
Sunshade
...
Sunshade
Sunshade
Sunshade
...

MTH
QTY TYPE
QTY_YR
------- ---- -------- -----2011-01 79 Actual
331
2012-12 172 Actual
2013-01 167 Actual

582
691

2013-12 160 Actual
2014-01 222 Forecast
2014-02 325 Forecast

691
825
825

2014-11 26 Forecast
2014-12 191 Forecast
2011-01
4 Actual

825
825
377

2012-12
2013-01

3 Actual
2 Actual

321
263

2013-12
2014-01
2014-02

5 Actual
1 Forecast
7 Forecast

263
187
187

time series forecasting
in sql
9/10
Kim Berg Hansen
Sunshade
Sunshade

2014-11
2014-12

3 Forecast
3 Forecast

187
187

That can be represented in an easily readable graph:

many items. But she also wanted the results put into a table so that the
forecast data could be used and reused for graphs and application and all
sorts of things for the purchasing department.
No problem, said I, I’ll create the table:
create table forecast (
item varchar2(10)
, mth date
, qty number
);

And then with one insert statement, I created the 2014 forecast of hundred thousand items in 1½ minute:
insert into forecast
with s1 as (...), s2 as (...), s3 as (...), s4 as (...), s5 as (...)
select item
, mth
, t * s qty -- forecast
from s5
where s5.ts >= 37; -- just 2014 data

She was happy that day.
Figure 5: 2011-2013 sales and 2014 forecast for the two items

The massive apply
But actually what my data analyst asked was, if I could apply this model
to a hundred thousand items. Yes, the SQL above works nicely for that

106 – OTech Magazine #6 – winter 2014

Conclusion
The original Time Series Analysis model was here built in Excel with a
series of successive columns with formulas. Using successive with clauses
and analytic functions to calculate centered moving average and regression, I could step by step transform the same model into a SQL statement

time series forecasting
in sql
10/10
Kim Berg Hansen
for very efficient forecasting of hundred thousand time series in one go.
But this is just a single statistical model. There are many, many other variants of models. Surely I can write them in SQL for efficiency – but handwriting hundreds of models will be very tedious. In tools like R there are
such models built in – next issue of OTech Magazine I’ll write about how
to use R packages and the free ROracle driver to do forecasting using
many statistical models.
If you see the presentation I gave at ODTUG Kscope14 in Seattle:
http://www.slideshare.net/KimBergHansen/analytic-functions-advancedcases, you can find a comparison of this method with another simple and
crude model I built with regr_slope. The scripts can be found here: http://
goo.gl/q1YJRL

107 – OTech Magazine #6 – winter 2014

call for content
OTech Magazine’s goal is to offer an insight
into Oracle technologies and the way they are
put into action. OTech Magazine publishes
news stories, credible rumors and how-to’s
covering a variety of topics. As a trustworthy technology magazine, OTech Magazine
provides opinion and analysis on the news in
addition to the facts.

cover topics relevant to professions such as
software architects, developers, designers
and other.
OTech Magazine‘s authors are considered to
be the best Oracle professionals in the world.
Only selected, high-quality articles will make
the magazine. Our editors are known worldwide for their knowledge in the Oracle field.

Our readership is made up of professionals
who work with Oracle and Oracle related
technologies on a daily basis. In addition we

Do you want to share your – Oracle – story to the world? Please fill in our
Call-for-Content or contact Editor-in-Chief Douwe Pieter van den Bos.

108 – OTech Magazine #6 – winter 2014

Douwe Pieter van den Bos
+31 6 149 143 43
[email protected]

OTech Magazine

OTech Magazine is an independent magazine for Oracle professionals. OTech Magazine‘s goal is to offer a clear perspective on Oracle
technologies and the way they are put into action. OTech Magazine
publishes news stories, credible rumors and how-to’s covering a
variety of topics. As a trusted technology magazine, OTech Magazine
provides opinion and analysis on the news in addition to the facts.
OTech Magazine is a trusted source for news, information and
analysis about Oracle and its products. Our readership is made up of
professionals who work with Oracle and Oracle related technologies
on a daily basis, in addition we cover topics relevant to niches like
software architects, developers, designers and others.
OTech Magazine‘s writers are considered the top of the Oracle
professionals in the world. Only selected and high-quality articles
will make the magazine. Our editors are trusted worldwide for their
knowledge in the Oracle field.
OTech Magazine will be published four times a year, every season
once. In the fast, internet driven world it’s hard to keep track of
what’s important and what’s not. OTech Magazine will help the
Oracle professional keep focus.
OTech Magazine will always be available free of charge. Therefore
the digital edition of the magazine will be published on the web.
OTech Magazine is an initiative of Douwe Pieter van den Bos. Please
note our terms and our privacy policy at www.otechmag.com.

Independence

OTech Magazine is an independent magazine. We are not affiliated, associated, authorized, endorsed by, or in any way officially
connected with The Oracle Corporation or any of its subsidiaries or
its affiliates. The official Oracle web site is available at www.oracle.
com. All Oracle software, logo’s etc. are registered trademarks of the
Oracle Corporation. All other company and product names are trademarks or registered trademarks of their respective companies.

In other words: we are not Oracle, Oracle
is Oracle. We are OTech Magazine.

109 – OTech Magazine #3 – May 2014

Authors

Why would you like to be published in OTech Magazine?
- Credibility. OTech Magazine only publishes stories of the best-ofthebest of the Oracle technology professionals. Therefore, if you
publish with us, you are the best-of-the-best.
- Quality. Only selected articles make it to OTech Magazine. Therefore, your article must be of high quality.
- Reach. Our readers are highly interested in in the opinion of the
best Oracle professionals in the world. And all around the world
are our readers. They will appreciate your views.
OTech Magazine is always looking for the best of the best of the Oracle technology professionals to write articles. Because we only want
to offer high-quality information, background stories, best-practices
or how-to’s to our readers we also need the best of the best. Do you
want to be part of the select few who write for OTech Magazine?
Review our ‘writers guidelines’ and submit a proposal today at www.
otechmag.com.

Advertisement

In this first issue of OTech Magazine there are no advertisements
placed. For now, this was solely a hobby-project. In the future, to
make sure the digital edition of OTech Magazine will still be available
free of charge, we will add advertisements. Are you willing to participate with us? Contact us on www.otechmag.com or +31614914343.

Intellectual Property

OTech Magazine and otechmag.com are trademarks that you may
not use without written permission of OTech Magazine.
The contents of otechmag.com and each issue of OTech Magazine,
including all text and photography, are the intellectual property of
OTech Magazine.
You may retrieve and display content from this website on a computer screen, print individual pages on paper (but not photocopy
them), and store such pages in electronic form on disk (but not on
any server or other storage device connected to a network) for your
own personal, non-commercial use. You may not make commercial
or other unauthorized use, by publication, distribution, or performance without the permission of OTech Magazine. To do so without
permission is a violation of copyright law.

All content is the sole responsibility of the authors. This includes all
text and images. Although OTech Magazine does it’s best to prevent
copyright violations, we cannot be held responsible for infringement
of any rights whatsoever. The opinions stated by authors are their
own and cannot be related in any way to OTech Magazine.

Programs and Code Samples

OTech Magazine and otechmag.com could contain technical inaccuracies or typographical errors. Also, illustrations contained herein
may show prototype equipment. Your system configuration may differ slightly. The website and magazine contains small programs and
code samples that are furnished as simple examples to provide an
illustration. These examples have not been thoroughly tested under
all conditions. otechmag.com, therefore, cannot guarantee or imply
reliability, serviceability or function of these programs and code samples. All programs and code samples contained herein are provided
to you AS IS. IMPLIED WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT AND FITNESS FOR A PARTICULAR PURPOSE ARE
EXPRESSLY DISCLAIMED.

OTECH MAGAZINE

Have a great Christmas.
See you in 2015.