Informatica Best Practices
Content
Tip 1: Bulk Binding In row mode, use bulk binding for reads and writes. Bulk binds improve the performance of PL/SQL statements. This feature significantly increases loading performance. Tip 2:ď€ Foreign Key Constraint Manipulation In set mode, manipulate foreign key constraints so that rows are inserted faster while still preserving referential integrity. Tip 3: Faster loading from Remote Tables Using the Inline View Technique OWB automatically detects the common scenario where one or more custom transformations are applied to multiple, joined tables that are located in a remote database. First join the tables in the source database and then apply the transformation functions on the (much reduced) result set in the target database. OWB forces the join on the remote database by means of an inline view. On average this feature brings a fivefold performance increase. Tip 4: Partition Exchange Loading Partition Exchange Loading enables parallel loading of data into partitioned tables, and avoids implicit index maintenance by loading a temporary table and then switching it with a partition. It is particularly efficient at loading data into a target table that already holds a large
amount of data. This technique can be up to ten times faster than conventional partition loading. Tip 5: Automated Bitmap index creation Bitmap indexing increases performance by enabling star query transformation. Warehouse Builder automates the process of creating bitmap indexes to speed up the design of a performance index scheme. Tip 6: ltrim and rtrim Use ltrim & rtrim while loading the records from the flat file to the staging table itself. This will avoid adding them in the Joins while going for subsequent Mappings Tip 7: Set Based Inserts Wherever set-based inserts are involved, direct path insert should be invoked using the /*+ APPEND */ optimizer hint. Direct-path insert is one of the most effective tools the Oracle ETL developer has available. By avoiding logging of the inserts, transformation runtimes can be reduced. Tip 8: Trailing Null cols Use trailing null cols while invoking the SQL*Loader otherwise it will give an error” Record Not Found” till the end of the file.
Tip 9: Set based Implementation In the Type 2 SCD approach ,where you need to look for the existence of data in the target before inserting, if you use set-based, and run the package you will get an ORA04091 error stating "Table ‘XXX’ is mutating, trigger/function may not see it." This error occurs because you are inserting into, and the function is selecting from, the target table in the same set-based statement. (fig1) Though ,the mapping will function properly when deployed as a row-based package, the performance of the row-based implementation may be unacceptable when considering the volume of rows being processed and the size of the ETL window. So, an alternative approach is available that will allow you to deploy the mapping using a set-based implementation. First, you will need to create a staging table that looks exactly like the target table; then, substitute the staging table for the target table in the mapping. The Type 2 SCD function will still do the lookup on the target table, but the inserts will be redirected to the staging table. A second mapping will then be required that inserts the new rows in the staging table into the target table.
FIGURE 1 BAD PRACTICE
SOURCE
Function Accessing TARGET
Other transfor mations
TARGET (set based insert)
GOOD PRACTICE Mapping 1
SOURCE
Function Accessing TARGET
Other transfor mations
STAGING TABLE (set based insert)
Mapping 2
STAGING
TARGET
Tip10: Repository installation It’s always a good practice to install the OWB repository and Run Time Audit tables in different schemas for better administration. Tip 11: Hardware The server hosting INFOMATICA gives good performance when minimum 128 MB RAM is used. The same is the case for the client side also. Tip 12: init.ora configuration In init.ora file on the server, the value of max open cursor should be more than 300. Tip 13: Projects If the staging and target tables are residing in two different instances, then it’s better to have two different projects to handle this logic.
Tip 14: Mappings Configuration Setup: The following should be the configuration parameters for PL/SQL mappings and SQL*Loader mappings to obtain the best performance: For PL/SQL mappings Option Name Value Deployable True Step type PL/SQL Default operating Set based fail over to row based For mode Update statements this doesn’t works. So it need to be “Row based (Target only)” Generate bulk False for the mappings processing processing less no. Of records, true for mappings processing large number of rows (sales fact) Bulk size 100 This value has to be matched with Commit Frequency if not it gives a warning. So for facts this value will be 1000. Default audit level Complete Commit Frequency 100 for dimensions, 1000 for Sales Fact Generate analyze False table statement Max no. Of errors 50 Bulk processing True Analyze table Default value
statistics
For SQL*Loader mappings Option Name Value Deployable False Step type SQL*Loader Bad file name, Keep blanks discard file name, log file name Discard max 0 Continue load False (Applicable only for direct mode) NLS character set Default value Direct Mode False Operation True Recoverable Parallel load False (applicable only for mode) Error Allowed 99999 (means no limit) Records to skip 0 (for sales fact it will have some value as the initial rows have row count, total sales, total qty etc. It will be 1 for all the dims, lookups and facts as Column names exist in the file which needs to be skipped. Records to load 0 (means load all) Rows per commit 100 (No. Of rows in bind array)
Read size Bind size (maximum size in bytes for bind array) Read buffers Preserve blanks Database file name Delimited record file termination Control file name Audit Default purge group
0 40000 0 False Leave empty Leave blank Leave blank True wb
Note: Some of the points mentioned above have been taken from secondary sources like OracleMetalink.
amount of data. This technique can be up to ten times faster than conventional partition loading. Tip 5: Automated Bitmap index creation Bitmap indexing increases performance by enabling star query transformation. Warehouse Builder automates the process of creating bitmap indexes to speed up the design of a performance index scheme. Tip 6: ltrim and rtrim Use ltrim & rtrim while loading the records from the flat file to the staging table itself. This will avoid adding them in the Joins while going for subsequent Mappings Tip 7: Set Based Inserts Wherever set-based inserts are involved, direct path insert should be invoked using the /*+ APPEND */ optimizer hint. Direct-path insert is one of the most effective tools the Oracle ETL developer has available. By avoiding logging of the inserts, transformation runtimes can be reduced. Tip 8: Trailing Null cols Use trailing null cols while invoking the SQL*Loader otherwise it will give an error” Record Not Found” till the end of the file.
Tip 9: Set based Implementation In the Type 2 SCD approach ,where you need to look for the existence of data in the target before inserting, if you use set-based, and run the package you will get an ORA04091 error stating "Table ‘XXX’ is mutating, trigger/function may not see it." This error occurs because you are inserting into, and the function is selecting from, the target table in the same set-based statement. (fig1) Though ,the mapping will function properly when deployed as a row-based package, the performance of the row-based implementation may be unacceptable when considering the volume of rows being processed and the size of the ETL window. So, an alternative approach is available that will allow you to deploy the mapping using a set-based implementation. First, you will need to create a staging table that looks exactly like the target table; then, substitute the staging table for the target table in the mapping. The Type 2 SCD function will still do the lookup on the target table, but the inserts will be redirected to the staging table. A second mapping will then be required that inserts the new rows in the staging table into the target table.
FIGURE 1 BAD PRACTICE
SOURCE
Function Accessing TARGET
Other transfor mations
TARGET (set based insert)
GOOD PRACTICE Mapping 1
SOURCE
Function Accessing TARGET
Other transfor mations
STAGING TABLE (set based insert)
Mapping 2
STAGING
TARGET
Tip10: Repository installation It’s always a good practice to install the OWB repository and Run Time Audit tables in different schemas for better administration. Tip 11: Hardware The server hosting INFOMATICA gives good performance when minimum 128 MB RAM is used. The same is the case for the client side also. Tip 12: init.ora configuration In init.ora file on the server, the value of max open cursor should be more than 300. Tip 13: Projects If the staging and target tables are residing in two different instances, then it’s better to have two different projects to handle this logic.
Tip 14: Mappings Configuration Setup: The following should be the configuration parameters for PL/SQL mappings and SQL*Loader mappings to obtain the best performance: For PL/SQL mappings Option Name Value Deployable True Step type PL/SQL Default operating Set based fail over to row based For mode Update statements this doesn’t works. So it need to be “Row based (Target only)” Generate bulk False for the mappings processing processing less no. Of records, true for mappings processing large number of rows (sales fact) Bulk size 100 This value has to be matched with Commit Frequency if not it gives a warning. So for facts this value will be 1000. Default audit level Complete Commit Frequency 100 for dimensions, 1000 for Sales Fact Generate analyze False table statement Max no. Of errors 50 Bulk processing True Analyze table Default value
statistics
For SQL*Loader mappings Option Name Value Deployable False Step type SQL*Loader Bad file name, Keep blanks discard file name, log file name Discard max 0 Continue load False (Applicable only for direct mode) NLS character set Default value Direct Mode False Operation True Recoverable Parallel load False (applicable only for mode) Error Allowed 99999 (means no limit) Records to skip 0 (for sales fact it will have some value as the initial rows have row count, total sales, total qty etc. It will be 1 for all the dims, lookups and facts as Column names exist in the file which needs to be skipped. Records to load 0 (means load all) Rows per commit 100 (No. Of rows in bind array)
Read size Bind size (maximum size in bytes for bind array) Read buffers Preserve blanks Database file name Delimited record file termination Control file name Audit Default purge group
0 40000 0 False Leave empty Leave blank Leave blank True wb
Note: Some of the points mentioned above have been taken from secondary sources like OracleMetalink.
