Batch Best Practices

Oracle Utilities Application Framework

September 2020 | Version 4.4.0.2.3

Copyright © 2020, Oracle and/or its affiliates
Public
PURPOSE STATEMENT
This document provides common practices for the batch processing in the Oracle Utilities Application Framework up to an
including version 4.4.0.2.3. It is intended solely to help you assess and plan your Oracle Utilities Application Framework
based product implementation.

DISCLAIMER
This document in any form, software or printed matter, contains proprietary information that is the exclusive property of
Oracle. Your access to and use of this confidential material is subject to the terms and conditions of your Oracle software
license and service agreement, which has been executed and with which you agree to comply. This document and
information contained herein may not be disclosed, copied, reproduced or distributed to anyone outside Oracle without
prior written consent of Oracle. This document is not part of your license agreement nor can it be incorporated into any
contractual agreement with Oracle or its subsidiaries or affiliates.
This document is for informational purposes only and is intended solely to assist you in planning for the implementation
and upgrade of the product features described. It is not a commitment to deliver any material, code, or functionality, and
should not be relied upon in making purchasing decisions. The development, release, and timing of any features or
functionality described in this document remains at the sole discretion of Oracle.
Due to the nature of the product architecture, it may not be possible to safely include all features described in this document
without risking significant destabilization of the code.

2 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
TABLE OF CONTENTS
Purpose Statement 2
Disclaimer 2
Introduction 5
Batch Concepts 5
Batch Processing At A Glance 5
Batch Program 6
Batch Controls 6
Batch Run Tree 7
Batch Parameters 8
Multi-Threaded Processes 11
Extract Process 13

Batch Architecture 13
Batch Technology 13
Batch Initiation 14
Batch Interface 14
Batch Objects 16
Batch Execution 16

Configuring Batch Architecture 18

Determining Cluster Configuration 18
Determining Threadpool Configuration 20
Setting Submitter Architecture (optional) 24
Designing Your Threadpool Architecture 25
Implementing Cache Threadpools 27
Implementing Administration Threadpools 28
Administrating Threadpools 29
Advanced Threadpools using Templates 30
Determining Threads 30

Batch Best Practices 33

Use of Oracle Coherence 33
Using the Batch Daemon 33
Using Timed Batch 34
Level 2 Caching in Batch 35
Using Queries to Monitor Batch 35
Calculating Thread End Tolerances 37
Calculating Thread Start Tolerances 38
To check this the following calculation must be performed on the above query: 38
Parameter Overrides 39
Parameter Guidelines 40
Setting The Error Tolerance 40
Altering Commit Interval and Timeout 40

Plug In Batch 42
Generic Templates 43
F1-PDBEX - Plug In Driven Extract Template 43
F1-PDBG - Plug In Driven Generic Template 44
F1-PDUPL - Plug In Driven File Upload Template 45
Using Templates 46

Batch Real Time Monitoring 46

Batch Level Of Service 47
Recommended Prerequisite Setup 47

3 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 Concepts 48
Metrics available 49
Status Applicability 49
Setting Thresholds 50
Building Your Own Algorithm Types 50
Setting Up Batch Level of Service Algorithms 50
Allocating Batch Level Of Service to Batch Controls 51
Allocating Batch Level Of Service to Job Stream Definitions 52
Assessing Batch Level Of Service 53
Using Health Check to Monitor Batch Level Of Service 54
Using Batch Control Maintenance to Monitor Batch Level Of Service 54
Invoking Batch Level Of Service from Extensions 55
Invoking Health Check from an external system 56
Implementation Guidelines for Batch Level Of Service 56

Batch Schedule Optimization 56

Concepts 56
Guidelines for Optimizing Schedules 57
Implementing Stream Conditions for Oracle Utilities SaaS 58

4 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
INTRODUCTION
Typically, a foreground process (a.k.a. online or Web Service transaction), performs operations on a single instance of an
object within the product. Maintaining a person's contact details or processing a single payment is two such examples.
Background processing, by contrast, performs operations on multiple instances of an object. This coins the term batch
where the background process processes batches of objects. The term background processing implies that the processing is
performed in the background with little or no user interaction.
The product ships with a preset number of background processes that may be used and configured to perform the
necessary business functions for your site. These background processes can be extended (just like the rest of the product
functionality) and custom background processes can be added.
This technical brief outlines the common and best practices used for the background processing (a.k.a. batch) component
of the Oracle Utilities Application Framework. The advice in this page is based upon Oracle internal studies and customer
feedback around the world. This information is provided to guide other sites in implementing or maintaining the product in
production.

BATCH CONCEPTS
The following section outlines some basic concepts when configuring and executing the batch component of the Oracle
Utilities Application Framework.

Batch Processing At A Glance

A batch program is a program that processes work in bulk to some sort of object for processing and then completes
successfully or fails (due to some error or reaching an error tolerance). Fundamentally, the batch processing can be
summarized as follows:
Batch Driver. This is a statement that decides the subset (if any) of records to process in the batch process. The
driver can be an SQL statement, or a file read operation to drive the batch process. Optionally, the subset can
support threading natively or rely on the Oracle Utilities Application Framework to perform this function.
Process Records. Once the subset of records is determined, something needs to process each record. This logic
can use various objects within the product and/or extract the information to a file (in the case of a file export batch
process). This can be one or more objects to process the records (typically a single entity that can call other objects)
to perform the necessary logic dictated by the batch process.
Optionally, perform logic on success or failure. When a batch process completes, it is either successful or has
failed. In some cases, additional logic is required to address this situation.
The batch program relies on the Oracle Utilities Application Framework to provide the infrastructure around the processing,
to reduce complexity and costs, including the following:
Batch Instance Management. Management of the technical aspects of executing logic in the Batch Architecture.
Thread Management. Managing the subdivision of workload if it is not covered in the Batch Driver and managing
the contention across those threads.
Check Pointing. As batch potentially processes large amounts of data, a check point should be taken on a regular
basis to save progress in case of a major failure.
Restart Logic. In concert with check pointing, the batch program, in conjunction with the Oracle Utilities
Application Framework, implements logic to start at an appropriate place on restart.

5 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 Interfacing. The Oracle Utilities Application Framework provides the basis for interfacing to objects, tables and files
to hide the technical aspects of the solution regardless of the platform.
Batch Run Tree Integration. One of the key monitoring and metrics capabilities of the Oracle Utilities Application
Framework is the Batch Run Tree. The Oracle Utilities Application Framework is responsible for populating the
appropriate information in the Batch Run Tree for any execution of a Batch program.
The processing is summarized below:

Figure 1. Summary of Batch Processing

Batch Program
The main component of the background process is the batch program. This program contains the logic to select the batch
of records that the process will perform actions upon. The batch of records is progressively passed to the relevant objects to
complete processing. The program acts as a driver to push individual records to the relevant objects. Objects in the product
are shared across all modes of access to maximize reuse.
Apart from the logic to decide the subset of records, the batch program contains the following additional information:
The batch program contains the code necessary to interface to the framework to automatically manage its
individual execution and restart (if necessary).
Some of the batch programs contain the code necessary to multi-thread the processing. The program determines
which slice or subset of the overall data it needs to process. Not all programs support multi-threading (most do,
except the extract to file type processes).
The product ships with a preset number of batch programs associated with background processes that may be used and
configured to perform the necessary business functions for your site. These background processes can be extended (just
like the rest of the product functionality) and custom batch programs can be added.

Batch Controls
A batch program and its parameters must be defined in metadata prior to initial execution. This data is stored in a Batch
Control object within the meta-data component of the framework.
The Batch Control contains the definition of the batch program and the following additional information:
Identifier. A Batch Code used as an identifier. This is used by the framework to identify the process internally and
used to denote output.
Technology. The technology used to build the batch program is configured so that the Oracle Utilities Application
Framework understands which technology to use to initiate and track the process.
Security. In Oracle Utilities Application Framework V4.3.0.0.0 and above, each batch control can be controlled via
an Application Service to provide additional levels of security.
Execution Metrics. Basic execution statistics for the last execution, including level of service (if configured), last
execution date and time and the latest run number which is primarily used for extract processing.
Fixed Parameters. Each Batch Control has several fixed parameters that control tracing, thread counts, commit
interval and other aspects of behavior.

6 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 Dynamic Parameters. Batch processes can have process specific additional parameters that control process
specific behavior. The batch control defines these as well as default values.
The product ships with a preset number of batch controls associated with background processes that may be used and
configured to perform the necessary business functions for your site. Custom batch controls can be added as needed.
An example of a Batch Control is shown below:

Figure 2. Sample Batch Control

Batch Run Tree

One of the main features of the Batch Architecture in the Oracle Utilities Application Framework is the Batch Run Tree. It is
an object that collates all the runtime information from the execution of a batch process and is available from the user
interface. The batch run tree captures the following information:
Batch Execution Metrics. The metrics of start and end times for the process are recorded as well as some state
information at the process level.
Note: The execution start and end times are determined from the longest running thread.

Thread Metrics. Each thread contains metrics for individual start and end times as well as additional information
such as records processed and in error.
Exceptions. If there are any exceptions, they are listed in the tree. If they are linked to transactions, they may be
hyperlinked to the record causing the error (process dependent).
Note: Some processes create To Do records to highlight errors within the process or within a companion process. Refer
to the product documentation for clarification.

An example of a Batch Run Tree is below:

7 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Figure 3. Sample Batch Run Tree

It is also possible using the Batch Run Tree to force a complete restart. By default, the Oracle Utilities Application Framework
will restart a process/thread in error at the last checkpoint (As indicated by Restart Data). To override this behavior and force
the checkpoint to be ignored then set Do Not Attempt Restart on the Run Control Tab. For example:

Figure 4. Sample Restart Override

Note: This functionality only applies to executions in an Error state not completed processes. Setting this value does not
guarantee the process will process the same set of records as originally as the individual process may ignore already processed
records

Batch Parameters
Controlling the behavior of the execution of the batch program are several parameters that are specified in the Batch
Control and on configuration files (depending on the method of execution). These parameters are technical, and business
related and can be explicit or implied. Setting these parameters can change the run-time and processing behavior of the
execution:

8 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Batch Parameters

PARAMETER TYPE SCOPE COMMENTS

Batch Control Explicit Technical/Business This is the identifier of the batch process in the
product. This identifier is used to initiate the
execution and used by some of the submission
methods for logging.

Batch Control Type Explicit Technical This is the type of batch execution used by the
Oracle Utilities Application Framework. This
supports Timed and Non-Timed process. This is
recommended to be set and not changed.

Batch Category Explicit Business This is used for reporting purposes on portals and
zones and categorizes the batch control into its
relevant business area

Accumulate All Instances Explicit Technical Refer to the online help for an explanation of this
setting.

Thread Count Explicit Technical This is the number of threads to execute for a multi-
threaded process. The minimum is 1 and the
maximum is the number of threads supported
concurrently on the architecture. Refer to Multi-
Threaded processes for more information.

Thread Number Explicit Technical This is the individual thread number to submit. This
is used at submission time to submit individual
threads for micromanagement and the value must
be less or equal to the Thread Count. Omitting this
value or setting it to 0 will spawn the number of
threads up to the thread count. Refer to Multi-
Threaded processes for more information.

Override Number of Records Explicit Technical The number of objects to process before a commit
to Commit performed for checkpointing purposes. If no value is
supplied, it is defaulted within the batch process
itself.

Override Maximum Timeout Explicit Technical This is the number of minutes the set of records is
in Minutes processed before a commit is performed. This
prevents sets of data becoming stale within the
database. This parameter works in conjunction with
the Override Number of Records to Commit to
control checkpoints.

Note: This parameter only applies to legacy versions

of Oracle Utilities Application Framework that
supported COBOL based extensions. It does not
apply for Java or ConfigTools based batch controls.

Trace Program Start Explicit Technical This setting enables or disables debug messages to
be added to batch logs to record program start. This
is used by developers to debug their code.

9 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 PARAMETER TYPE SCOPE COMMENTS

Note: This setting should not be enabled in

Production, unless otherwise instructed by Oracle
Support or your developers

Trace Program Exit Explicit Technical This setting enables or disables debug messages to
be added to batch logs to record program ending.
This is used by developers to debug their code.

Note: This setting should not be enabled in

Production, unless otherwise instructed by Oracle
Support or your developers

Trace SQL Explicit Technical This setting enables or disables debug messages to
be added to batch logs to record execution of any
SQL statements in the program. This is used by
developers to debug their code.

Note: This setting should not be enabled in

Production, unless otherwise instructed by Oracle
Support or your developers

Trace Output Explicit Technical This setting enables or disables debug messages to
be added to batch logs to record outcomes of calls
to other objects in the program. This is used by
developers to debug their code.

Note: This setting should not be enabled in

Production, unless otherwise instructed by Oracle
Support or your developers

Batch Rerun Number Explicit Technical/Business This setting is used at submission time to rerun
specific extracts. In normal operation this number is
not provided. Refer to Extract processes for a
discussion of this parameter.

Batch Business Date Explicit Technical/Business This is an important submission parameter that
specifies the date (in ISO format YYYY-MM-DD) that
tells the batch program which records are to be
considered for processing. If not provided, the
current system date is used. It is recommended that
a relevant common date be supplied for each
execution to ensure consistency in set of records
processed.

Batch User Explicit Technical/Business This parameter is required at submission for use in
the batch execution in following ways:
Security. The user supplied is checked within the
security system for appropriate access for the objects
used in the batch program including the Batch Control
itself.
Ownership. All records created or modified in the batch
process will be marked with this user in the appropriate
objects. For example, if To Do records are created then
the To Do creator will be this user.

10 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 PARAMETER TYPE SCOPE COMMENTS

Email Address Explicit Technical This is an optional parameter used by some

submission methods to send an email summarizing
the result of the process to this user or group of
users (the group must be setup in the mail system
for this to work).

Note: The use of this facility requires an appropriate

SMTP server to be setup as per the Installation Guide
and Server Administration Guide.

DIST-THD-POOL Implied Technical This is a hidden parameter in the collection of

parameters to set the threadpool for execution.

Note: This parameter is only used for online

submission and Oracle Scheduler Integration. The
command line-based submission should not use this
capability. The command line submission method
uses an alternative way of specifying the threadpool.

MAX-ERRORS Implied Business This is a hidden parameter which when used can set
the error tolerance for the process. This sets the
number of errors tolerated before the process is
considered a failure. When this tolerance is reached,
the process is stopped with an error status. By
default, it is omitted which disables the capability.
Refer to Setting the Error Tolerance for more
information.

Multi-Threaded Processes
One of the major features of the batch framework is the ability to support multi-threading. The multi-threading support
allows a site to increase throughput on an individual batch process by splitting the total workload across multiple individual
threads. This means each thread has fine level control over a segment of the total data volume at any time.
The idea behind the threading is based upon the notion that many hands make light work. Each thread takes a segment of
data in parallel and operates on that smaller set. The object identifier allocation algorithm built into the product randomly
assigns keys to help ensure an even distribution of the numbers of records across the threads and to minimize resource and
lock contention.
In Oracle Utilities Application Framework 4.3.0.4.0 and above, it is possible to separate the logic for threading using the
Process SQL algorithm in Plug In Batch and within the Oracle Utilities SDK built batch processes.
The capability can be described as follows:
Thread Limit is Set. A maximum number of Threads is set at execution time to break a large processing task into
smaller parallel tasks. The thread limits need to be set appropriately for the process as well the architecture. The
number of threads specified should not exceed the total number of threads in all threadpool instances allocated to
the batch process. At the execution time this limit and the name of the threadpool are specified, if the thread limit
exceeds the total number of threads available across all instances of the targeted threadpool then the excess
threads will wait till a thread being available. This will delay the total elapsed time of the process.
Partitioning Data Sets. Each Thread is allocated a slice or partition of data to process as dictated by the process.
By default, the Process SQL/selection component of the process will allocate roughly the same number of objects
to process. For example, if the thread limit is set to six (6) then each thread will get approximately 1/6 of the records
to process. The best way to visualize the concept of threading is to use a pie analogy. Imagine the total work set for
a batch process is a pie. If you split that pie into equal sized segments, each segment would represent an individual
thread. For example:

11 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Figure 5. Sample Threading

Note: The allocation of records in a thread does not guarantee that the threads will finish at the same time. If there
are some workload skew, some threads will finish later than others.

Each thread will execute independently. Each thread will work through the individual sets of records allocated to
each thread and finish updating the Batch Run Tree appropriately. When the last thread of the process is complete
then the process is complete.
The concept of threading has advantages and disadvantages:
Smaller elapsed run-times. Processes that are multi-threaded finish earlier than processes that are single
threaded. With smaller amounts of work to do, processes with threading will finish earlier.
Note: The elapsed runtime of the threads is rarely proportional to the number of threads executed. Even though
contention is minimized, some contention does exist for resources which can adversely affect run-time.

Threads can be managed individually. Each thread can be started individually and can also be restarted
individually in case of failure. If you need to rerun thread X then that is the only thread that needs to be
resubmitted.
Threading can be somewhat dynamic. The number of threads that are run on any instance can be varied as the
thread number and thread limit are parameters passed to the process at run-time. They can also be configured
using the configuration files outlined in the product documentation.
Note: Threading is not dynamic after the process has been submitted.

Failure due to data issues with threading has reduced impact. As mentioned earlier individual threads can be
restarted in case of failure. This limits the risk to the total process if there is a data issue with an individual thread or
a group of threads.
Number of threads is not infinite. As with any resource there is a theoretical limit. While the thread limit can be up
to 1000's of threads, the number of threads you can physically execute will be limited by the CPU and IO resources
available to the process at execution time.
Theoretically with the objects identifiers evenly spread across the threads the elapsed run-time for the threads should all be
the same. In other words, when executing in multiple threads theoretically all the threads should finish at the same time.
Whilst this is possible, it is also possible that individual threads may take longer than other threads for the following reasons:

Workloads within the threads are not always the same. Whilst each thread is operating on the roughly the same
amounts of objects, the amount of processing for each object is not always the same. For example, an account may
have a more complex rate which requires more processing, or a meter has a complex amount of configuration to
process. If a thread has a higher proportion of objects with complex processing it will take longer than a thread with
simple processing. The amount of processing is dependent on the configuration of the individual data for the
process.
Data may be skewed. Even though the object identifier generation algorithm attempts to spread the object
identifiers across threads there are some processes that use additional factors to select records for processing. If
any of those factors exhibit any data skew, then certain threads may finish later. For example, if more accounts are
allocated to a part of a schedule then threads in that schedule may finish later than other threads executed.

12 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Extract Process
A special type of process in the Batch Architecture are extracts of data from the database to a file. These are considered
special as the Batch Run Number pays a clear part of the process. When records are identified and extracted for processing,
the current batch number is attached to those objects (the field used will vary according to that object) as part of the
extraction. This batch number is obtained from the current batch number on the batch control. To rerun a previously
executed extract, if supported by the process, specify the desired batch number as the Batch Rerun Number.

Note: When using the Configuration Migration Assistant to migrate Batch Controls across environments, by default the current
run number in the target environment is retained and not overridden by the value from the source.

BATCH ARCHITECTURE
The Batch architecture provided with the Oracle Utilities Application Framework is a flexible and powerful way of defining
and executing batch programs. The architecture has several layers:
Batch Technology. This layer comprises the technology used to initiate the batch process and/or manage its
relationship with other batch processes. This can range from technology provided with the Oracle Utilities
Application Framework or the provision of third-party technology (in an enterprise solution). An implementation
will use one of the methods supported as their mechanism for managing and executing batch processes.
Batch Initiation. This layer describes the different methods of initiating batch processes including monitoring and
managing their progress or restart.
Batch Interface. This layer is the dedicated interfaces provided to manage and execute batch processes. This
includes providing information to and back to the initiating technology.
Batch Objects. These are a set of internal objects and functions within the Oracle Utilities Application Framework
to record and manage batch processes.
Batch Execution. This layer is responsible for executing the batch process as instructed and providing a
mechanism for feedback to the initiating technology as necessary.
The figure below summarizes the above layers as well their components which will be discussed in the subsequent sections:

Figure 6. Batch Architecture Summary

Batch Technology
In implementing batch architectures, you need technology to initiate and manage individual batch process or sets of related
batch processes (such as in a schedule). In most cases, this is one of the following technologies:

13 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 Third Party Scheduler. Some enterprises already have invested in an enterprise wide scheduler that needs to be
used to manage all batch processes across the portfolio of products they own via a set of command line-based
scripts. This is typically an enterprise scheduler that provides an ability to define schedules and groups of batch
processes to execute including processes external to the product. For Oracle Utilities SaaS Cloud implementations,
it is possible to use the Scheduler API to schedule and execute batch processes on your on-premise third-party
enterprise schedule to retain your on-premise investment, if desired.
Oracle Utilities Cloud Service Foundation (Oracle Utilities SaaS Cloud Only). The Oracle Utilities Cloud Service
Foundation includes a capability to define, manage and execute batch schedules from the provided interface. This
interface uses the Scheduler API to maintain, manage and execute batch processes and schedules.
Oracle Enterprise Manager or Oracle SQL Developer. For on-premise or hybrid customers who want to use the
Oracle Scheduler to manage and execute batch processes, it is possible to use the Oracle Scheduler directly via
Oracle Enterprise Manager (Scheduler Central) and/or Oracle SQL Developer.
Online Submission (Development/Testing Only). For developers and testers, an online submission capability,
with a dedicated submission screens, is provided to allow for individual batch processes to be executed. No inter
process dependency is supported in this interface. This interface is not supported on Production systems.
Note: For legacy customers of Oracle Utilities Customer Care And Billing, a basic workflow based scheduler was
provided for development and testing purposes. This interface is not supported for production use and one of the
alternatives outlined above should be used instead.

Batch Initiation
To initiate the execution of batch processes there are three interfaces:
Command Line Utilities. There are several scripts to submit and management processes including submission of
processes, establishment of a runtime environment for execution and a set of related configuration files that define
the behavior of the batch process and architecture.
Oracle Scheduler. A set of API's based upon Oracle's DBMS_SCHEDULER capability. This capability is ideal for most
implementations as it is included as part of the base license for all editions of the Oracle Database, so it tends not to
have a license impact. The solution can be implemented locally or remotely (for an enterprise implementation).
Note: If the DBMS_SCHEDULER objects are placed in a remote database for an enterprise implementation, ensure that
database is licensed appropriately.

Online Submission. This is reserved for use for developers and testers on non-production environments and is
available via the online of the product.

Batch Interface
Note: For details of all the scripts and API's shown in this section refer to the Server Administration Guide.

The product provides several flexible interfaces to allow technologies to initiate, manage and monitor batch processes at the
batch control level or the individual thread level.
Command Line Interface. The Oracle Utilities Application Framework includes a series of scripts that can be run by
an authorized user to configure, manage and execute batch processes. The following is a summary of the
commands available:
Command Line Utilities

UTILITY COMMENTS

bedit.sh This is a utility to create and maintain configuration files relating to the batch
execution architecture. The utility is designed with a wizard like interface to simplify
the building the architecture and reduce common configuration issues.

jmxbatchclient.sh It is possible to monitor and manage executing batch processes (and threads) using a
JMX interface. This utility provides an interface into the JMX capability to allow
management of executing processes.

starttpw.sh This is a utility to start a group of threadpools from the command line for schedulers.

14 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 UTILITY COMMENTS

submitjob.sh This is the main utility to start or restart execution of a batch process (or individual
thread) and wait for the process (or thread). This interface is used by third party
schedulers to execute individual batch processes.

threadpoolworker.sh This utility can work alone or in tandem with starttpw.sh to start threadpools
individually. This interface is used by third party schedulers to start or stop individual
threadpools.

Note: Refer to the Server Administration Guide for details of the command line options and the execution of these commands.

Note: As with all commands in the Oracle Utilities Application Framework the splenviron.sh command MUST be executed
prior to any command to set the appropriate environment variables.

Oracle Scheduler Interface. The OUAF_BATCH package interface is provided so that Oracle DBMS_SCHEDULER can be
used to schedule batch processes.
Note: Refer to the Batch Scheduler Integration for Oracle Utilities Application Framework (Doc Id: 2196486.1) and
Server Administration Guide for more information about each API.

OUAF_BATCH Procedures

UTILITY COMMENTS

Submit_process Submits a batch process (or thread) from the scheduler and waits for the process to
complete. This is the main procedure in the API to execute processes.

Restart_Thread Manually restarts threads for restart. This is used manually rather than from the
scheduler.

Cancel_process Cancels an executing process. This will cancel on the next checkpoint or poll interval,
whichever comes first. This is used by manually to cancel processes.

Set_Option Sets GLOBAL or batch control specific options for the scheduler. This sets interface
specific options and some key parameters.

Unset_Option Unsets GLOBAL or batch control specific options for the scheduler that have been set
by the Set_Option interface.

Raise_Thread_Error This is an internal option which allows for specific thread error processing. It can be
used by the scheduler to force error conditions for testing purposes if necessary.

Online Submission. The online submission screen is available for use for testing purposes. This allows a user to
queue an execution of a batch process. Refer to the online documentation for details of the operation of this
screen. An example of the submission is shown below:

15 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Figure 7. Sample Batch Submission

Batch Objects
To support the interfaces a set of objects are provided by the Oracle Utilities Application Framework to record and manage
batch processes. The table below lists the objects used provided for use by the interface and runtime:
Batch Objects

OBJECT PRIME TABLE USAGE

Batch Control CI_BATCH_CTRL Batch Controls that define the batch process to the Oracle Utilities Application
Framework

Batch Job CI_BATCH_JOB Submission initiation object used by Online Submission and Oracle Scheduler
Interface. This is not used by the command line interface.

Batch Run CI_BATCH_RUN This is the working object used for initiation, execution and monitoring by the
Oracle Utilities Application Framework batch architecture. It is used by the
execution capability as well as the basis for information in the Batch Run Tree.
An object per execution is persisted in this object along with related
information in child objects.

These objects are manipulated by the execution engine as well as the submission methods shown above. These objects are
not to be manipulated manually under any circumstances.

Batch Execution
The key component of the batch architecture is the execution architecture to execute multiple threads of multiple batch
processes effectively. The architecture is based upon the following principles:
Coherence Based. The architecture is based upon clustering batch resources using the Named Cache capability
within the Oracle Coherence product. Oracle Coherence provides the ability to cluster resources across machines
and effectively scale vertically as well as horizontally needed for performing batch effectively and efficiently.

16 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 Note: A restricted use license of Oracle Coherence is provided with the Oracle Utilities Application Framework for the
exclusive use as the runtime for the batch architecture. The libraries provided with the Oracle Utilities Application
Framework are not a full installation of Oracle Coherence and the license does not allow usage outside the runtime of
the batch architecture. If the additional use is needed for Oracle Coherence an additional license needs to be
purchased. Use of the Management Pack for Oracle Coherence for Oracle Enterprise Manager or Java VisualVM
Coherence Plugin is not recommended to be used with the batch architecture as the installation provides a limited set
of libraries which are not compatible with the Management Pack for Oracle Coherence.

Flexibility in architecture. The architecture supports simple and complex architectures across single or multiple
machines using a variety of networking methods.
Robust. The architecture detects issues in execution and keeps the Oracle Utilities Application Framework
appropriately informed in real time and as the process completes to ensure consistency.
In built batch daemon. For the online submission and Batch Scheduler integration a daemon is available in the
architecture to detect the initiation of activities. This daemon detects pending requests and submits them to the
architecture.
The batch architecture is based upon the following concepts:
Threadpools. Long running Java Virtual Machines (JVM's) are waiting for threads to process. These are known as
threadpools. These are the JVM's that perform the work.
− Clustered Threadpools. Each threadpool is named (for individual or groups of JVMs). This allows clusters
of similar named threadpools to form a larger virtual threadpool to allow for greater numbers of
concurrent threads.
− Thread Limits on Threadpools. Each threadpool has and have individual thread limits to avoid JVM
overload. The thread limits on similarly named threadpools accumulate to support greater numbers of
concurrent threads.
Configurable Threadpool Behavior. The configuration of threadpools allows for JVM behavior, caching and node
management within a cluster to be configured.
Submitters. When a Batch Interface requests to execute a batch process or thread a submitter is created within the
cluster to act as the client to the cluster. The submitter tells the cluster what to run and where and then waits
around for the threads to finish.
Batch Daemon. A running batch daemon that looks for pending work to submit to the cluster. This is used
exclusively with the Batch Scheduler Integration and Online Submission methods.
These execution concepts are shown below:

Figure 8. Batch Execution Architectures

17 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
CONFIGURING BATCH ARCHITECTURE
One of the first tasks to consider in the batch architecture is the composition of the architecture in terms of the cluster and
the resulting threadpools. It is highly recommended to use the bedit.sh utility to perform this task to reduce errors. To use
this facility, ensure that the Enable Batch Edit Functionality is set to true using the configureEnv.sh -a command. For
example:
$ configureEnv.sh -a
*************************************************
* Environment Configuration demo *
*************************************************
50. Advanced Environment Miscellaneous Configuration
...
Enable Batch Edit Functionality: true
...
To confirm this setting check for the BATCHEDIT_ENABLED setting in the $SPLEBASE/etc.ENVIRON.INI file:
$ cat $SPLEBASE/etc/ENVIRON.INI | grep BATCHEDIT
BATCHEDIT_ENABLED=true
Once enabled the following commands can be used to set up the architecture:
Command Options

COMMAND USAGE REFERENCES

bedit.sh -c <options> Sets up the cluster configuration including Cluster Architecture

networking

bedit.sh -w <options> Sets up threadpool definitions across the Threadpool Architecture

cluster Cache Threadpools
Administration Threadpools

bedit.sh -s <options> Sets up submitter definitions (optional) Submitter Architecture

bedit.sh <filename> Edit the configuration file name directly. This Refer to each section on the file names
file must be generated by bedit.sh to be edited that are generated by bedit.sh.
by that utility.

Note: Use the bedit.sh -h command for a full list of options available

Note: Refer to the Server Administration Guide shipped with your product for details of the bedit.sh utility and how to use its
facilities.

Determining Cluster Configuration

The first step in a batch architecture is to decide the cluster architecture and networking used for cluster communication.
This configuration is used by Oracle Coherence Clustering to define the communications across the batch architecture which
creates a tangosol-coherence-override.xml configuration file located in the $SPLEBASE/splapp/standalone/config
directory.

Note: If the online includes the daemon or batch server, then changes to this file will need to be included in the build of the
online deployment files and a redeployment is required to update the deployment files.

Note: If you are using the bedit.sh functionality, do not manually edit the generated configuration files as this may cause the
utility to overwrite manual changes.

The product supports several cluster types using the command:

18 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
$ bedit.sh -c -t <template>
The cluster types support are as follows:
Cluster Types

CLUSTER TYPE (TEMPLATE) USE CASES COMMENTS

Single Server (ss) The cluster is restricted to the This is useful for non-production environments with
current host. the simplest networking using Single Host Protocol

Uni-cast (wka) The cluster uses unicast protocol This is recommended for sites wanting to lock down a
with the hosts explicitly named cluster to specific hosts and does not want to use
within the cluster that are part of the multi-cast protocols. Administrators will have to name
cluster. the list of hosts, known as Well Known Addresses, that
are part of the cluster as part of this configuration

Multi-cast (mc) The cluster uses the multi-cast This is recommended for sites who want a dynamic
protocol with a valid multi-cast IP configuration where threadpools and submitters are
address and port. accepted on demand. This is the lowest amount of
configuration for product clusters as the threadpools
can join a cluster from any server with the right
configuration dynamically. It is not recommended for
sites that do not use the multi-cast protocol.

When using these configurations, the following advice should be considered:

Ensure the Cluster Name is unique per environment. A cluster name is a user-defined name that uniquely
identifies a cluster from other clusters that run on the network. When building batch cluster definitions ensure each
environment has their own unique cluster name and ports to avoid conflicts and incorrect networking.
Use Single Server for Non-Production. To save costs and risks, it is recommended to use Single Server Clustering
in all non-production environments. The only exception is when you are using a non-production environment with
multiple hosts.
Use Multi-cast if possible. If your site can use multi-cast, use that setup in preference to uni-cast to reduce
maintenance costs. Multi-cast implementations are more dynamic and reduce the amount of configuration.
Ensure all WKA's are defined in Uni-cast. Ensure all your hosts and associated ports are defined in your uni-cast
cluster configuration. Ensure each of those hosts has the same configuration. Hosts and their associated ports will
not be able to use the cluster if they are not defined explicitly.
Note: You may need to alter the configuration per host to ensure that the configuration is valid.

Use the same port per machine in WKA. The uni-cast configuration uses host names and port number
combinations to define the cluster. Avoid different port numbers on individual machines to reduce maintenance
complexity.
Avoid using prod mode in non-Production. The cluster may be configured to operate in either dev (development)
or prod (production mode). These modes do not limit access to features, but instead alter some default
configuration settings. For instance, development mode allows for faster cluster startup to ease the development
process. It is recommended to use the development mode for all pre-production activities, such as development
and testing. This is an important safety feature, because the cluster configuration automatically prevents these
nodes from joining a production cluster. The production mode must be explicitly specified when using the cluster in
a production environment.
Note: Oracle Utilities Application Framework has additional built-in membership validation to prevent unintentional
non-production nodes joining production, regardless of the mode setting.

The figure below summarizes the type of information to configure for each template:

19 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Figure 9. Cluster Configuration

The cluster is created, operationally, when the first threadpool is started. Conversely a cluster is closed when the no
threadpools are executing for that cluster. No additional operations are needed explicitly to start or stop a cluster.

Determining Threadpool Configuration

After the cluster is configured the next important step is to design and configure the threadpools needed for the
architecture. Before discussing advice on design, several concepts need to be understood:
Threadpools are Java Virtual Machines. Each instance of a threadpool is a separate Java Virtual Machine which
consumes memory and CPU, even when idle and waiting for work.
Batch Threads are long running and use more resources than usual threads. Individual Java Virtual Machines
can potentially process hundreds of threads in normal operation. Batch threads are long running and tend to use
far more resources per Java thread that normal threads, which tend to be more transient and shorter lived.
Therefore, the number of batch threads that each Java Virtual Machine can handle is greatly reduced. By default,
the number of batch threads can be as small as five (5) which means more than one threadpool instance may be
needed to cover all threads.
Threadpool Instances have a flexible architecture. A cluster of threadpool instances can exist within a single host
or across hosts and multiple instances of the same named threadpool can exist within the same machine or across
machine. The figure below illustrates some of the most common configuration scenarios.
The figure below illustrates a set of example configurations:

Figure 10. Example Scenarios

20 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
In the above example, there are two hosts (host1 and host2) with mixed number of thread pools with unique names:
Example Threadpool Setup

THREADPOOL COMMENTS

POOL1 This represents the simplest scenario (A) with a single threadpool instance on a single host. The
number of threads is limited by the limit on the individual instance.

POOL2 This represents a scenario (B) where multiple instances of the same named threadpool exist on a single
machine. The number of threads available in that named pool is the sum of the instances.

POOL3 This represents a scenario (C) where multiple instances across machines. This is like the scenario with
POOL2 in that the number of threads available in that named pool is the sum of the instances but
illustrates that the pool is not constrained within machines.

Note: The scenarios shown above are not exhaustive as variations within each of the scenarios having the ability to be
combined to enable support for complex scenarios. For example, the scenarios of POOL2 and POOL3 (B and C) can be combined
if necessary.

To setup threadpools use one of the following commands:

$ bedit.sh -w
$ bedit.sh -l <arg>
There are two commands to create threadpool definitions:
Default option (w). This is the first option used to create a generic set of threadpools. This generates a
threadpool.properties file used by the threadpoolworker.sh script.
Using Templates (l). This will be examined in Advanced Threadpools using Templates. This generates a template
specific threadpool.properties file that can be used by the threadpoolworker.sh script.

Figure 11. Threadpool Configuration

The following applies when configuring the threadpools:

Java Virtual Machine settings are within the template. The memory settings for the JVM are at the template
level not at the threadpool level. Unless otherwise stated, the default settings are enough for most needs.
There are several thread related settings. Threads are used internally to manage the cluster and within the
threadpool to execute the work.
Thread Settings

SETTING OUAF SETTING COMMENTS

distthds tangosol.coherence.distributed.threads This setting controls the number of

internal daemon threads allocated to

21 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 SETTING OUAF SETTING COMMENTS

Oracle Coherence to manage the

members in the cluster for the
Distributed Cache Service. If this
value is set to 0, then this will force
Coherence to use its service thread
to perform all relevant tasks, which
can cause issues in the Oracle
Utilities Application Framework. It is
recommended to set this value to
the default provided in the
templates, unless otherwise
instructed by Oracle Support.

invocthds tangosol.coherence.invocation.threads This setting controls the number of

internal daemon threads allocated to
Oracle Coherence to manage the
execution of members in the cluster
for the Invocation Service. If this
value is set to 0, then this will force
Coherence to use its service thread
to perform all relevant tasks, which
can cause issues in the Oracle
Utilities Application Framework. It is
recommended to set this value to
the default provided in the
templates, unless otherwise
instructed by Oracle Support.

threads com.splwg.grid.distThreadPool.threads.<threadpoolname> This is a setting provided by the

Oracle Utilities Application
Framework to set the number of
concurrent batch threads are
allocated to each instance of a
named threadpool. These are the
long running heavy threads
performing the work. Advice on this
setting will be later in this section.

Other Settings. There are several settings in threadpools that should be considered when configuring the
threadpools:
Thread Settings

SETTING OUAF SETTING COMMENTS

dkidisabled spl.runtime.batch.disableDeferredKeyInserts This is a setting that controls the use of

Deferred Key Insert logic that are needed
in some Oracle Utilities Application
Framework. Refer to the product
documentation to see if this capability is
required.

rmiport spl.runtime.management.rmi.port Port used for JMX communication for

monitoring purposes. This port is used for
cluster wide monitoring from the Oracle
Utilities Application Framework, Oracle

22 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 Utilities Cloud Service Foundation and the
Application Management Pack for Oracle
Utilities. This setting is overridden by the
jmxstartport parameter at runtime.

storage tangosol.coherence.distributed.localstorage This enables or disabled local storage in

the cache. Depending on the architecture
the value of this parameter will vary. For a
full discussion refer to Implementing
Cache Threadpools and Implementing
Administration Threadpools. At least one
threadpool per cluster must have this set
to true.

daemon com.splwg.batch.scheduler.daemon This disables or enables the inbuilt

processing daemon. This is set to true for
environments using online submission or
the Oracle Scheduler integration. Sites
using the command line submission
method from a third-party scheduler
should set this to false. Refer to Using the
Batch Daemon for further advice.

role tangosol.coherence.role This is a generic tag used for monitoring

purposes only. This allows groups of
threadpools to be defined for monitoring
purposes.

jmxstartport oracle.ouaf.jmx.remote.startPort This is a newer JMX port number

parameter which dynamically sets the
port number based upon availability of a
port. This setting is ideal for
implementations that have multiple
threadpools on the same host. Each
threadpool will have its own port from this
port. Set the value the same as the
rmiport parameter.

l2 spl.runtime.batch.L2CacheMode The batch architecture can use level 2

caching to improve performance. This
setting controls the caching with various
setting. Refer to Level 2 Caching in Batch
for additional advice.

devmode spl.runtime.options.isDevelopmentMode This setting is used for Development

environments (using Oracle Utilities SDK)
to turn off advanced features to allow
developers to perform unit testing
appropriately. This is recommended to be
set to true for Oracle Utilities SDK
environments only.

Threadpool Collection. Create a collection of threadpools to suit your needs. For each threadpool specify a
threadpool name and thread limit per instance of that threadpool. Refer to Designing Your Threadpool Architecture
for detailed design advice.
Note: Remember to save any changes at the end of the session to create the configuration files.

23 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Setting Submitter Architecture (optional)
Note: Customers using the online submission and Batch Scheduler integration should not generate configurations for
submitters as they use a combination of the Batch Control and command line overrides to process submitters.

Note: At present, there is no product supplied utility to automatically generate the parameter files from the Batch Control.

The final step in the batch architecture is to build and maintain a configuration file (if required) for individual batch controls
which are used by the submitjob.sh utility.
To setup submitters use one of the following commands:
$ bedit.sh -s
$ bedit.sh -b <batch_control>
There are two commands to create submitter definitions:
Default option (-s). This sets up a default configuration file used for any batch control where no specific batch
properties file exists. This creates a submitbatch.properties file located in the
$SPLEBASE/splapp/standalone/config subdirectory.
Batch Control Specific configuration (-b). This will create a batch control specific configuration file named
job.<batchcontrol>.properties where <batchcontrol> is the Batch Control identifier. This file is in the
$SPLEBASE/splapp/standalone/config/cm subdirectory. In this option, the soft parameters on the batch control
can be configured as well.
Note: The Batch control identifier used in the -b option should be in the same case as the Batch Control record in the
Oracle Utilities Application Framework.

Figure 12. Submitter Configuration

When using these commands, it is recommended to:

Use the -s option where possible. Setup a global configuration to cover as many processes as possible and then
create specific process parameter files for batch controls that require soft parameters.
Minimize the use command Line overrides. The advantage of setting up submitter configuration files is to reduce
your maintenance costs. Whilst it is possible to use command line overrides to replace all the settings in the
configuration, avoid overuse of overrides to stabilize your configuration.
Set common batch parameters. Using the -s option allows for the following common parameters to be setup:

Common Submitter Settings

SETTING COMMENTS

poolname This is the default threadpool used for the batch execution if none is specified on the command line or in
a process specific configuration. This should be set to an appropriate threadpool.

24 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 SETTING COMMENTS

threads This is the number of concurrent threads that are executed when no thread limit is specified on the
command line or in a process specific configuration. It is recommended to set this to 1 to prevent
unnecessary overloading of threadpools.

commit This is the default commit interval (in object counts) if no commit interval is specified on the command
line or in a process specific configuration.

maxtimeout This is the timeout limit (in minutes) for holding the set of records processed. This is a legacy setting for
older versions of the Oracle Utilities Application Framework that supported COBOL extensions to prevent
stale record sets.

user This is the default application user used for security purposes if no user is specified on the command line
or in a process specific configuration.

Note: The default SYSUSER account should NOT be used for this setting. The AUSER setting suggested in
the template, needs to be changed to an appropriate value. It is intentionally invalid to force a change in
the configuration.

lang This the default language code used for messages used in this execution if no language is specified on
the command line or in a process specific configuration.

storage This is the default for configuration storage within the cluster. This setting should be set to false unless
otherwise instructed in the product documentation.

role This is a tag used for monitoring purposes. By default, the batch code used at runtime is used to set this
value.

Setup soft parameters in process specific configurations. For batch controls that have parameters, these need
to be set in the configuration file or as overrides on the command line. To minimize maintenance costs and avoid
potential command line issues, it is recommended to set the values in a process specific configuration file, using
the add soft command in bedit.sh with the following recommendations:
− The parameter name in the parm setting must match the name and case of the Parameter Name on the
Batch Control.
− The value set in the value setting must be the same or valid for the Parameter Value on the Batch Control.
− Optional parameters do not not need to be specified unless used.
Note: Do not add a parameter for the DIST-THD-POOL soft parameter as that is configured via the poolname parameter.

Designing Your Threadpool Architecture

One of the most important aspects of the architecture is to decide and design the threadpools you need to create for your
implementation. There are two important aspects to consider:
Naming Threadpools. The key feature of the architecture is to name threadpools and use that name to target
batch processes to. The only limit for the threapools is the number of instances supported on a machine as the
physical memory allocated to a machine will dictate the number of JVM's supported on a host.
Note: Remember each threadool instance is a running JVM with memory requirements (by default, each instance
consumes 1GB of memory - This is customizable).

Thread Limits per Instance. Each threadpool instances supports a certain number of long running threads threads
set up to the limit.
The table below outlines common threadpool architecture patterns:

25 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Common Threadpool Patterns

PATTERN SUMMARY COMMENTS

Single Named Create a single threadpool name (such as This is the simplest architecture and tends
Threadpool DEFAULT) and then use that to run ALL processes to be used for non-Production
within the architecture using instances of that environments where you start with a single
threadpool . The number of instances of this threadpool instance either standalone or
threadpool will dictate the number of concurrent within the online JVM (the latter scenario is
threads supported. used for demonstrations and training).

Dual Threadpools Create two threadpools (one with L2 caching on and This is a variation on the Single Name
one with L2 caching off). There are internal Threadpool with an additional named
processes, such as Configuration Migration threadpool for L2 Caching on for the small
Assistant and LDAP Synchronization, that benefit number of processes that require it. Most
from the Level 2 cache being turned off. Treat the sites restrict themselves to a single L2
L2 Cache On threadpool the same way you would Cache Off instance per cluster with enough
treat Single Named Threadpool capacity to cater for the small number of
processes that require it.

Note: The threadpool with L2 Cache Off

only needs to be available when needed by
these processes.
For more guidelines around Level 2
Caching refer to Level 2 Caching in Batch.

Targetted This is a wide set of variations where you set up This is varied with the following
Threadpools several named threadpools with specific global considerations:
settings optimized for use with specific processes
Keep Default Configuration. Setup a generic
and some for generic processes.
default pool for non-specific processes. For
example, small processes or single threaded
processes can be targetted to this default
setup.
Specific Java Configuration. For processes
that require special java setups that are not
covered by the default, consider setting up
specific threadpools which are configured for
processing.
Consider Segregating Critical Processes.
While sharing threadpools across processes is
efficient, some sites have considered setting
up a specialist set of critical threadpools to
isolate and optimize those processes
accordingly. For example, it is easier to scale
those specialist threadpools in isolation to
other processes. This is more
micromanagement but allows
implementations to manage and target their
resources appropriately for their business
needs.

The following figure summarizes some typical patterns:

26 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Figure 13. Submitter Configuration

The decision on which patterns are best suited for you:

Use Single Threadpool or Dual Threadpools for non-production. Keep your maintenance costs low and setup
your non-production using the simplest architectures.
Threadpools Only need to be available when needed. One common misconception about threadpools is that
they need to stay running all the time. Whilst it is possible to keep them running all the time, they only need to be
available when the processes that use them require them. Most sites shut down threadpools when they are not
being actively used to reduce wastage of resources.
If using Single Threadpool pattern, use the online threadpool. For demonstrations, it is recommended to keep
the installation simple. In this case, the online can run some batch threads for demonstration and training
purposes. Refer to the Installation Guide for details of how to enable this functionality.
Flexible Production Setup. In production, a more complex architecture it required to satisfy the needs of the batch
processes to execute. When designing the production batch architecture consider any of the following patterns:
− If JVM requires additional parameters, consider a separate template. JVM settings are at the template
level, use the bedit.sh -l <template> command to create a new template with the settings. Within that
template specify your settings. Refer to Advanced Threadpools using Templates for more information.
− Create Dedicated Threadpools for critical processes. One technique that is popular is to create a
threadpool or a group of threadpools for critical processes. This allows these threadpools to scale
independently and allows processes using the threadpool to be isolated from other processes. If you do
not have a dedicated threadpool available, then there is a risk that any other process executing at the
same time will prevent efficient execution of the critical processes.
− Create L2 Cache Threadpools for utilities. As in production the use of certain utilities will be necessary
from time to time (for example using Configuration Migration Assistant) then a threadpool with L2
Caching disabled is recommended to be available whenever these processes are executed. Refer to Level 2
Caching in Batch for more discussion.
− Create Cache Threadpools. If the number of instances is high, it is recommended to consider using cache
threadpools. These threadpools will reduce the amount of communication in the cluster for many
threadpools. Refer to the Implementing Cache Threadpools for more advice.
− Create an Administration Threadpool (if desired). If you are actively monitoring the execution of
processes using JMX directly or via the Application Management Pack for Oracle Utilities, it is
recommended that you consider specifying a threadpool instance for strictly administration purposes.
Refer to Implementing Administration Threadpools for more advice.

Implementing Cache Threadpools

When implementing many threadpools in the cluster, the architecture establishes communications between each of the
threadpools across a cluster to determine state and workload. If there are n threadpool instances, then there are n * (n -1)

27 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
connections in the cluster. As the number of threadpool instances increases this potentially increases network overhead on
the cluster. To alleviate this issue, it is recommended to implement the concept of a cache threadpool. This special type of
threadpool does not run any threads but centralizes the communication across the threadpool instances on that host. This
reduces internal traffic between any threadpool instances on that host. For example:

Figure 14. Cache Threadpools

The concept is implemented using the local-storage option in Oracle Coherence. The threadpools are configured so that
the local-storage is disabled and, on the cache threadpool it is enabled. This tells Oracle Coherence to distribute the state
of any process to the cache threadpool to save networking. If the cache threadpool is active the cluster will use that
threadpool to centralize communications.

Note: When committing to use cache threadpools you MUST configure the threadpools that use the cache to storage set to
false.

To setup the cache threadpools use the recommended command:

$ bedit.sh -l cache
File /u01/demo/splapp/standalone/config/cm/threadpoolworker.cache.properties does not exist. Create?
(y/n) y
Editing file /u01/demo/splapp/standalone/config/cm/threadpoolworker.cache.properties using template
/u01/demo/etc/threadpoolworker.cache.be
Batch Configuration Editor [threadpoolworker.cache.properties]

Current Settings
poolname (cache)
minheap (768m)
maxheap (768m)
distthds (4)
invocthds (4)
role (OUAF_Cache_TPW)
jmxstartport (7540)
>
Edit the settings to suit your needs.

The product ships with a prebuilt cache template named threadpoolworker.cache.be.

Implementing Administration Threadpools

28 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Note: Use of Administration Threadpools is only recommended for sites wanting to monitor batch in real time via JMX or via
the Application Management Pack for Oracle Utilities. Customers on the Oracle Utilities SaaS Cloud will have an Administration
threadpool for use with Oracle Utilities Cloud Service Foundation.

By default, every active threadpool instance provides a JMX interface for monitoring purposes using different port numbers
for interaction. If you want to monitor the cluster in real time using JMX directly or via the Application Management Pack for
Oracle Utilities, then you need a fixed port and address to connect to the cluster each time. In these situations, it is
recommended, in complex production environment, to setup a dedicated Administration threadpool. An Administration
threadpool is similar in nature to a cache threadpool in that it does not run batch threads and is only available to act as a
conduit into the cluster for the JMX interface.

Note: To use the Administration threadpool ensure it is the first threadpool to start to reserve the JMX port number
appropriately.

Note: There is no Administration threadpool template shipped with the product. Use the default template to create the
administration threadpool.

To create the Administration threadpool use the command:

$ bedit.sh -l admin
Batch Configuration Editor [threadpoolworker.admin.properties]

Current Settings
minheap (768m)
maxheap (768m)
daemon (false)
rmiport (7540)
dkidisabled (false)
storage (true)
distthds (4)
invocthds (4)
role (OUAF_Admin_TPW)
jmxstartport (7540)
l2 (OFF)
devmode (false)
ollogdir (/u02/sploutput/demo)
ollogretain ()
thdlogretain ()
timed_thdlog_dis (false)
pool.1
poolname (ADMIN)
threads (1)
>

Note: The above settings are suggestions for the threadpool using the appropriate set command within bedit.sh.

Administrating Threadpools
Now that the threadpools are created the next step is to start and stop them using the threadpoolworker.sh command on
the hosts you want to run the threadpools:
$ threadpoolworker.sh -l <template> [-p <threadpool>]
Where:
<template> Configuration file to use threadpoolworker.<template>.properties
<threadpool> Threadpool within template to start. Omitting this will start an instance of all
threadpools in the file.
Note: Use the UNIX kill command or use the jmxbatchclient.sh interface to stop the instances. Refer to the Server
Administration Guide for more details.

29 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Note: Run the command more than once to create multiple instances of threadpools on a host. Ensure you have enough
memory to run the total number of instances running at any time.

Advanced Threadpools using Templates

The product ships with several templates that can be used to build threadpool definition using bedit.sh. These templates
exist in the $SPLEBASE/etc directory of the installation:
Threadpool Templates

TEMPLATE TEMPLATE NAME COMMENTS

Default threadpoolworker.be Used for default settings and administration threadpools.

This is the main configuration file.

Cache threadpoolworker.cache.be Used for cache threadpools only

Job threadpoolworker.job.be Used as a basis for any custom threadpools or specialist

threadpools (use the bedit.sh -l job command to use this
template)

it is recommended to use the templates as follows:

Use Default Template for Most Threadpools. Use the default template for non-product threadpools and use for
any general purpose threadpools in your architecture. Keep the defaults unless they are not appropriate for your
requirements.
Use Cache for a cache threadpool per host. For complex setups with lots of threadpools, create a cache
threadpool using this template. Avoid this for non-production to save on maintenance costs.
Use the Job Template for Specialist Threadpools. If you need to alter the threadpools for specialization
If a template does not exist for use in the -l option, then the default template will be used as a template, but the
configuration is stored in its own configuration file. Refer to the Implementing Administration Threadpools for an example
of this.

Determining Threads
One of the most important decisions to make on a day to day basis is to determine the number of threads to run for an
individual process. Before discussing the threading recommendations a few concepts need to be reinforced:
Thread Limit is Set At Submission Time. When you execute any process, using any submission method, one of
the parameters is the thread limit. This is the fixed limit set at submission time and cannot be altered, even for a
restart, after the process has started.
Individual Threads can be submitted. Even though most submissions will submit all the threads, it is also possible
to submit individual threads using the various submission methods. This allows sites to micromanage threads from
schedulers or support advanced threading models (for example, submitted different threads to different
threadpools).
Thread numbers are cumulative across named instances. When you submit a process, using any submission
method, the threadpool name is also a parameter. This is the set of threadpool instances that will execute the
threads of the batch process. The number of threads available for the process is the total cumulative number of
threads of active instances of that named threadpool. For example, if you have one instance and it was 5 threads
then the total is 5, if you have two instances of 5 threads the total is now 10 etc. For instance:

30 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Figure 15. Cumulation Of Threads

Respect Availability Of Threads. When a batch process is submitted to a threadpool, the total number of threads
in all running instances should be the same or more than the thread limit for all processes submitted at that time to
that threadpool. If there are not enough available threads across the instances some threads will wait and be
delayed from execution. This will delay the overall execution of the process. In the example below, the first
execution is within the thread limit ( 4 < 5 ), the second execution is not within the limit ( 6 > 5 ) and therefore a
threads waits for any other thread to complete before starting. The last example, shows that with two running
instances of the threadpool that there is enough capacity ( 6 < 10 ):

Figure 16. Thread Scenarios

Shared Threadpools Need Careful Planning. If more than one process is executing at the same time in the same
named threadpool, then there needs to enough capacity across all executions at any time. For example, if two
processes are executing at the same time and both have 3 threads each and the threadpool has a capacity of 5 then
a thread will wait ( 6 > 5 ):

31 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 Figure 17. Thread Scenarios

Consider Metrics. The number of objects processed per minute/second and the elapsed time are key metrics to
use for planning purposes. The number of objects processed per minute/second are determined by calculating the
total minutes or seconds in the elapsed time per thread and dividing the number of records processed to determine
the rate of processing. For error prone processes, also consider the number of errors as well.
Now these concepts are understood let us discuss the determination of threads:
Extracts and Imports are Single Threaded. Before you consider any threading, any exports and imports are only
single threads, so the thread limit should always be set to 1. This helps ensure the correct processing of the import
and export files involved in these processes.
Note: No attempt should be made to run single threaded processes in a multi-threaded mode as this may make the process
fail or cause inconsistencies. Refer to the product documentation for guidelines on individual processes (or review the
description on the batch control).

Some Processes are Single Threaded. Review the product documentation for thread guidelines for each product
process.
Do Not Overload Your Threadpools. It is tempting to flood the threadpools with large number of threads, but it is
recommended not to overload the threadpools with large number of threads. Increasing number of threads can
resolve data skew and reduce elapsed time but can introduce inefficiencies due to contention. The rule of thumb is
to increase the number of threads, monitoring the number of objects processed per second/minute till this value
decreases. That would be the natural limit of your configuration to the threading.
Only Increase Threading When Needed. It is tempting to change the thread limits on the processes on a regular
basis but stick to a limit and only change it when the number of objects processed per second/minute or level of
service is not reached.
Threadpool Limits are Configurable. The default in the configuration is 5 batch threads per instance of a
threadpool. This number was chosen to represent the heaviest processes in the product. It can be increased for
lighter processes.
Start Simple and Adapt. In your testing, determine a good starting point for the number of threads to submit for
any multi-threaded process. Monitoring the elapsed and number of objects processed per minute/second should
determine if you need to increase or decrease that number.
Consider Special Cases. In most implementations the data processed each execution of a process might yield the
same elapsed time overall but if the business has a special event that changes the amount to process then the
amount of threads for the duration of that event may need to increase or decrease. For example, if you are
expecting double the volume then you might need to increase the number of threads accordingly.

32 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
BATCH BEST PRACTICES
The batch architecture is both flexible and powerful so configuring, managing and operating batch is important to the
success of an implementation. There are techniques and practices that help ensure success.

Use of Oracle Coherence

As part of the batch architecture, key Oracle Coherence libraries are shipped with the product to create a cluster at runtime
to house submitter and threadpools. The libraries are part of a Restricted Use License provided with the Oracle Utilities
Application Framework with the following conditions:
Limited Implementation of Coherence. The libraries shipped are a subset of the Oracle Coherence stack and
implement the Distributed Cache and Named Cache functionality only. As the provided libraries represent a subset
only, it is not recommended to use the Oracle Coherence Pack available for Oracle Enterprise Manager with the
cache or cluster as that will yield inconsistent results.
Restricted Use. The libraries are provided using a Restricted Use License for exclusive use with the runtime of the
batch architecture and cannot be used in any extensions to extend the product (including any batch processes).
Patching. The Oracle Coherence libraries shipped with the Oracle Utilities Application Framework and are patched
with that product. Unless otherwise directed by Oracle Support, do not manually manipulate the versions of the
libraries as this may cause inconsistencies in execution.

Using the Batch Daemon

Note: Customers using the command line submission method (via the submitjob.sh utility) should not enable the daemon as
it is not used by that submission methods.

For the online submission and Oracle Scheduler Integration at least one threadpool must have the daemon running. This
daemon looks for pending work that has been submitted by these methods and submits it to the cluster on behalf of the
user. To use the daemon there are two places to set the value:
Daemon Settings By Submission Method

SUBMISSION METHOD SETTING

Online Submission (non- The following setting should be set to true using the configureEnv.sh -a command:
Production use only)
CONFIGUREENV SETTING ENVIRON.INI SETTING VALUE COMMENT

Online JVM Batch BATCHDAEMON true

Scheduler Daemon Enabled

Online JVM Batch Server BATCHENABLED true Only set this value if you want the
Enabled online JVM to include the
threadpool. This is recommended
for demonstration environments.

Batch Scheduler Integration In the default threadpool template, using the bedit.sh -w command, ensure that the
daemon setting is set to true. For example:
...
daemon (true)
...

The daemon should exist in at least one threadpool in the architecture and once active will actively look for jobs that are in
pending status (with other appropriate attributes) and submits them to the architecture. Once the job has completed the
daemon will communicate the state back to the product. The future below summarizes this processing:

33 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Figure 18. Batch Daemon Processing

Using Timed Batch

Note: This feature only applies to Oracle Utilities Application Framework V4.2.x and above only.

Note: This feature only applies to selected product supplied processes only. Batch Processes must be configured or written to
use this technique.

One of the features of the Oracle Utilities Application Framework is the ability to create timed batch processes. Typically,
most batch processes are executed as a single execution on a regular time period such as once a day, once an hour etc. In
some implementation scenarios, it is required to run a batch process continuously (such as a daemon) to process data as it
is found. The Oracle Utilities Application Framework introduced a feature to allow implementations to create processes that
are run continuously. The figure below illustrates the difference in the processing styles:

Figure 19. Concept of Timed Processes

The concept is as follows:

The batch program must be designed to run continuously. Existing processes in the product cannot be converted
from non-timed to timed just using configuration. The Oracle Utilities SDK contains information about writing
continuous batch programs.
An instance of the timed batch program runs for a timer interval and then completes on next commit after the
interval is reached. After completion a new instance is automatically started. This simulates a continuous execution
whilst minimizing performance impact on the overall system.
The batch control for timed processes must be configured with additional information including:
− Timer Interval. The duration each instance of the batch process will execute.

34 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 − Timer Active. This controls whether the process is executing or not. The process will only execute when
the timer active is set to true/Yes.
− UserId/Batch Language. Userid used for authorization and language for batch messages
− Email Address. Optional, email address used for monitoring for error executions.
The process must be started manually initially to start the process executing.
To stop a timed process, set the Timer Active to false.

Level 2 Caching in Batch

As with the rest of the architecture for Oracle Utilities Application Framework based products, a level 2 cache is created in
the cluster, within each theadpool instance, to cache commonly used configuration data for the processes in that
threadpool. The Oracle Utilities Application Framework will manage the population of this cache automatically. Some
processes that operate on this configuration data requires that this cache be disabled to update the information in that
cache correctly. This technique is only applicable to specific processes used to manage configuration migration and security.
For these processes it is recommended to create a special threadpool with L2 Caching turned off. To set up a threadpool
with caching disabled the following process should be used:

Use the bedit.sh -l l2cache command which will create a new template for use in the settings.
Set the following settings (settings not mentioned should remain as defaulted):
Level 2 Cache Settings

SETTING RECOMMENDED SETTING

daemon false

l2 OFF

Create a threadpool named appropriately, for example L2OFF, with a minimum number of threads to process the
utilities. Remove all other pool settings. Save the settings.
When needed start the threadpool using the threadpoolworker.sh -l l2cache command.
Ensure the processes have the above threadpool as the recommended one to execute in.

Using Queries to Monitor Batch

By default, all the monitoring information is available from the Batch Run Tree or via realtime using the JMX interface
(directly or via Application Management Pack for Oracle Utilities). Whilst this information is available via the online, it can
also be available via an appropriate query. The query below will reconcile all the executions of the product in one query. This
query is a sample only:

35 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
select t1.BCD BATCH,t1.BNR,t1.ID SCHID,SUBSTR(t1.TIME,6,14) STARTTIME,T1.BATCH_THREAD_NBR,
DECODE(t2.ID,null,'Running',SUBSTR(t2.TIME,6,14)) ENDTIME,
DECODE(T3.THREAD_STATUS,10,'Pending',20,'In Progress',30,'Error',40,'Completed',50,'Ready','??')
STATUS,
t1.BATCH_BUS_DT,
T1.BATCH_RERUN_NBR,
DECODE(SIGN(t2.TOTAL - t1.TOTAL),-1,(t2.TOTAL - t1.TOTAL0),(t2.TOTAL - t1.TOTAL)) TIMESEC,
t1.REC_PROC_CNT, t1.REC_ERR_CNT
from
(select b.batch_cd BCD,b.BATCH_NBR BNR,b.SCHEDULER_ID ID,b.BATCH_THREAD_NBR,b.BATCH_RERUN_NBR,
l.MESSAGE_PARM time,a.BATCH_BUS_DT,b.REC_PROC_CNT,b.REC_ERR_CNT,
SUBSTR(l.MESSAGE_PARM,9,2)*3600*24 + SUBSTR(l.MESSAGE_PARM,12,2)*3600 +
SUBSTR(l.MESSAGE_PARM,15,2)*60 + SUBSTR(l.MESSAGE_PARM,18,2) total,
SUBSTR(l.MESSAGE_PARM,12,2)*3600 +
SUBSTR(l.MESSAGE_PARM,15,2)*60 + SUBSTR(l.MESSAGE_PARM,18,2) total0,
l.MESSAGE_SEQ SEQ
from CI_BATCH_RUN a,
CI_BATCH_INST b,
CI_MSG_LOGPARM l
where b.SCHEDULER_ID=l.SCHEDULER_ID
and a.BATCH_CD=b.BATCH_CD
and a.BATCH_NBR=b.BATCH_NBR
and a.BATCH_RERUN_NBR=b.BATCH_RERUN_NBR
and l.MESSAGE_SEQ=(select min(x.MESSAGE_SEQ)
from CI_MSG_LOGPARM X
where X.scheduler_id=l.scheduler_id)) T1,
(select b.batch_cd BCD,b.BATCH_NBR BNR,b.SCHEDULER_ID ID,b.BATCH_THREAD_NBR,b.BATCH_RERUN_NBR,
l.MESSAGE_PARM time,a.BATCH_BUS_DT,b.REC_PROC_CNT,b.REC_ERR_CNT,
SUBSTR(l.MESSAGE_PARM,9,2)*3600*24 + SUBSTR(l.MESSAGE_PARM,12,2)*3600 +
SUBSTR(l.MESSAGE_PARM,15,2)*60 + SUBSTR(l.MESSAGE_PARM,18,2) total,
l.MESSAGE_SEQ SEQ
from CI_BATCH_RUN a,
CI_BATCH_INST b,
CI_MSG_LOGPARM l
where b.SCHEDULER_ID=l.SCHEDULER_ID
and a.BATCH_CD=b.BATCH_CD
and a.BATCH_NBR=b.BATCH_NBR
and a.BATCH_RERUN_NBR=b.BATCH_RERUN_NBR
and l.MESSAGE_SEQ=(select max(x.MESSAGE_SEQ)
from CI_MSG_LOGPARM X
where X.scheduler_id=l.scheduler_id)) T2,
CI_BATCH_THD T3

36 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
where T1.ID=T2.ID(+)
and T1.BCD=T3.BATCH_CD
and T1.BNR=T3.BATCH_NBR
and T1.BATCH_THREAD_NBR=T3.BATCH_THREAD_NBR
and T1.BATCH_RERUN_NBR=T3.BATCH_RERUN_NBR
and DECODE(t2.ID,null,'Running',SUBSTR(t2.TIME,6,14)) is not null
order by STARTTIME, T1.BATCH_THREAD_NBR
The following information is extracted:
Query Columns

COLUMN USAGE COMMENTS

BATCH Batch Control Identifier This is used to identify the process.

BNR Batch Run Number This is the system generated run number

SCHID Scheduler Identifier This is an internal generated number to identify an individual

execution of a process or restart

STARTTIME Start Date and Time This is the date and time of the start of the processing in the
format of the session

BATCH_THREAD_NBR Thread Number for execution This is the thread number. The BATCH, BNR, SCHID and
BATCH_THREAD_NBR are used as the key for the query.

ENDTIME End Date and Time This is the date and time of the end of the processing in the
format of the session

STATUS Status of process Status of the process. Either Completed (successful) or Error
(unsuccessful)

BATCH_BUS_DT Business Date The Business Date used to process these records. This can be
used to group executions across processes if necessary

BATCH_RERUN_NBR Batch Rerun Number If the process was a restart of an extract, then this will be
populated with the rerun number. A value of zero (0) is used to
denote processes that do not use this capability, or this is not a
rerun.

TIMESEC Seconds difference between This is the difference between STARTTIME and ENDTIME in
start and end date times seconds. Useful for calculations.

REC_PROC_CNT Record Count The number of records processed in the thread.

REC_ERR_CNT Error Record Count The number of records that errored in the thread.

Calculating Thread End Tolerances

In most cases the individual threads within a multi-thread process will finish together or within a reasonable time. But in
some cases, individual threads may finish later than other outside a tolerance. For example:

37 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Figure 20. Example Thread End Tolerance

To check this the following calculation must be performed on the above query:
Thread End Tolerance (TET) = Max(ENDTIME) - Min(ENDTIME) for a Batch Number (or SchId) for a Batch Control.
If this tolerance is within an acceptable limit, then there are no threads that are taking a much longer time to finish. If the
tolerance is above an acceptable limit, then there are outliers to examine. This may be caused by data skew.

Calculating Thread Start Tolerances

In most cases all threads of a multi-threaded job start at the same time, but in some cases, due to capacity issues or
scheduling issues, some threads may start significantly later (i.e. outside an acceptable tolerance). For example:

Figure 21. Example Thread Start Tolerance

To check this the following calculation must be performed on the above query:
Thread Start Tolerance (TST) = Max(STARTTIME) - Min(STARTTIME) for a Batch Number (or SchId) for a Batch
Control.
If this tolerance is within an acceptable limit, then there are no threads that are starting too late. If the tolerance is above an
acceptable limit, then there are outliers to examine. This may be caused by capacity or scheduler issues. These threads need
to be examined alongside what was happening at the same time.

38 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Parameter Overrides
The various submission methods allow for parameters for the processes to be set at various levels and overridden at lower
levels in the submission method. This allows implementations to be flexible in how they are defined and how they are used
in the execution of processes. The figure below illustrates the order of precedence for parameters by submission method:

Figure 22. Parameter Override Levels

Parameter Override Levels

COMMAND LINE SCHEDULER INTERFACE ONLINE SUBMISSION

Batch Interface defaults in

OUAF_BATCH interface

GLOBAL scoped defaults in

OUAF_BATCH interface

Parameters On Batch Control Parameters On Batch Control

Batch Process specific scoped

defaults in OUAF_BATCH interface

Parameters are expressed in

submitter properties files

Submission Record

Command Line Overrides Command Line Overrides

The lower levels can override the higher levels when configured to do or at the appropriate time based upon submission
method.

39 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Parameter Guidelines
When configuring and using parameters the following best practices apply to reduce maintenance costs:
Set Parameters at higher levels. Setting your parameters at higher levels reduces maintenance costs and
simplifies submission.
Only Override a parameter when necessary. Only override the parameters at lower levels if it is necessary. This
reduces maintenance costs.
Internal defaults should not be relied upon for non-development use. In some submission methods internal
defaults are used as root level parameters. They are provided for developers to unit test their code for various
testing techniques.
The command line options should only be used for reruns or off schedule (a.k.a. special runs). This avoids
updating the configuration files for one-off or special processing. The only exception to this rule is that the
Business Date parameter should be specified on the command line to ensure consistency across processes.
Batch Control is not used for Command Line Submission Method. Whilst it is possible to define parameters on
the Batch Control for use in some submission methods the command line submission method does not use the
Batch Control. The main reason for this is that the scheduler that executes the process needs control and may be
remote from the product.
It is not recommended to set MAX-ERRORS at a global level. This should only be specified at an individual thread
of a background process level.
Setup Security At The Highest Level. Avoid security issues and reduce maintenance costs by settings global users
as necessary.
It is not recommended that the default SYSUSER be used as the userid specified in background processes.
SYSUSER is the initial default user for the Oracle Utilities Application Framework and is only used to input
additional user in the initial stages of configuration. Most implementations create a dedicated user record to
delineate background processes from online processes.

Setting The Error Tolerance

By default, there are two circumstances where a background process will not complete successfully. First of all, there is some
technical issue with the environment or program and the other is if all records processed fail some business rule. The latter
condition may not be appropriate for your site. For example, if all but one payment fails in the payment background
processes then the process is still considered complete.
To support a more realistic tolerance, it is possible to set a limit on the number of errors tolerated before the process should
be cancelled. At the thread level, the MAX-ERRORS parameter can be used to specify a thread level error tolerance where the
thread will be cancelled when the tolerance is reached or exceeded.
The default value of MAX-ERRORS is 0 (zero) which turns off the facility. Any appropriate non-zero value will then become the
error tolerance limit for the thread.
Setting the appropriate value for your site will require business approval in line with your sites organization business
practices.

Note: The MAX-ERRORS parameter is applicable to business errors only. System errors (SEVERE errors) will terminate the process
immediately.

Altering Commit Interval and Timeout

Note: Timeout does not apply to Java based processes and is not used for any batch process in Oracle Utilities Application
Framework V4.3 and above.

There are two parameters that can be altered to control the amount of resources a background process uses when it
executes. While most implementations reuse the default values supplied with the product, it is possible to alter the values to
tune the performance of background processes and allow background processes to be executed during peak application
usage periods.
The Commit Interval parameter controls the Unit of work on the background process. It controls after how many primary
objects are processed before a commit point is taken, also known as commit frequency. It controls the amount of work the
database must do and when a checkpoint is taken for restart purposes. At the commit point in a background process, all the

40 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
outstanding work, since the last commit, is reflected on the database and a checkpoint is written on the batch control record
for the process. You usually take the default for the process, but it can be adjusted according to the circumstances.
The higher Commit Interval the value specified the less commit points are taken. Typically commit points can be expensive
transactions in a database so most implementations try and minimize them. The lower value the more commit points. The
latter is desirable when you are running background during online hours as it reduces the resource hit the background
process has on online processes.
Specifying a large value for Commit Interval can cause larger than normal rollbacks to be performed by the database. This
can cause a strain on the database and hardware. The longer the unit of work the more work the database must do to roll it
back in case of a failure.
The Commit Interval can have advantages and disadvantages depending on the situation and the value:
Comparison of Approaches

HIGH VALUE FOR COMMIT INTERVAL LOW VALUE FOR COMMIT INTERVAL

Larger Unit of Work Smaller Unit of Work

Longer rollback in case of failure Shorter rollbacks in case of failure

Can increase throughput on lightly loaded system Can allow background to work harmoniously with online

Less Commits in process (Less checkpoints) More Commits in process (More frequent checkpoints)

Lower Concurrency (higher impact on other users) Higher Concurrency (lower impact on other users)

The second parameter Timeout (in minutes) controls how long a solution set is held by the process. When Oracle database
processes a set of records it typically holds a snap shot of these records to save processing time. If the set is held too long,
the records may not reflect the state of the database, so it is a good idea for the Oracle database to maintain the currency of
this data regularly. Within the background process this is controlled by setting this value to prevent the snapshot being
discarded by Oracle database and causing an abort. If the records are held too long an ORA-1555 Snaphot too old error is
generated and the process aborts.
The Timeout and Commit Interval parameters are tunable parameters to affect the impact of the background processes on
the other processes running and prevent internal database errors. It is also important to understand their impact to ascertain
whether any change is required. The following rules of thumb apply to setting the values:
It is recommended that the Commit Interval should not be set to one (1), unless otherwise instructed, as this value
may causes excessive database I/O and therefore performance degradation.
For light jobs (short duration, single threaded, small numbers of records etc), the default value for Commit Interval
may satisfy your site performance requirements.
For heavy jobs (long duration, multi-threaded, large number of records etc), then a value for Commit Interval of
between 5 (five) to 200 (two hundred) is recommended.
The value of the Commit Interval directly affects the size of the redo logs allocated to the database. The higher the
commit interval the larger the redo logs need to be to hold the in-process objects. Work with your sites DBA groups
to come up with a compromise between redo logs and Commit Interval.
During processing of any background process, a main object is used to drive the process. For example, in BILLING the main
object is Account. The BILLING process loops through the accounts objects as it processes. For other processes it is other
objects that are considered the main object. This main object type is used to determine when a transaction is complete.
For both Timeout and Commit Interval this is important as:
At any time in a process a commit for objects processed may be caused by the reaching the Commit Interval or the
time limit set on Timeout, whichever comes first.
The setting of Commit Interval and Timeout has impact on the amount of memory the JVM memory space
allocated to the individual threads. Higher values of both require more memory to hold the data.

41 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
PLUG IN BATCH
Note: This capability is available for Oracle Utilities Application Framework V4.3.0.4.0 and above only.

One of the new features of the Oracle Utilities Application Framework is the ability to define the processing necessary for a
batch process without the need for creating a java-based extension in the Oracle Utilities SDK. This is known as Plug In Batch
and is designed to allow implementations to quickly build batch processes without the need for complex coding.
The idea behind Plug In Batch is that you can develop in a series of algorithms in Java, Scripting or Groovy (the latter two are
supported for Cloud implementations) to break a batch process into its constituent parts for processing. The Oracle Utilities
Application Framework will take care of any data access, restarting, checkpointing, multi-threading etc.
The following algorithms are available:
Plug In Batch Algorithms

ALGORITHM TYPE COMMENTS

Select Records This algorithm typically contains an SQL statement to filter the records to process
including threading

File Upload This algorithm is used for processing XML, Fixed or CSV formatted files

Process Record This algorithm takes the information from either Select Records or File Upload
algorithm to process the records appropriately. For file extracts, this algorithm
contains the logic to format the output

Post Processing This algorithm provides the ability to provide extra logic if the Process Record
processing is successful for the record set.

Error Post Processing This algorithm provides the ability to provide extra logic if the Process Record
processing is not successful for the record set.

The first algorithms (Select Records and File Upload) determine the set of records to process for the batch process. Each
record is passed to a Process Record algorithm to process the object (for file extracts that means formatting and writing
records). Optionally, if required, if the process is successful then extra logic can be executed in the Post Processing
algorithm. Conversely, if required, if the process is not successful then extra logic can be executed in the Error Post
Processing algorithm. The figure below illustrates this processing:

Figure 23. Plug In Batch Algorithms

42 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Note: Refer to the ConfigTools Best Practices for more advice on building Plug In Batch.

When building an algorithm for Plug In Batch consider the following:

SQL Statement in Select Records should consider Threading. In the SQL statement, consider added support for
f1.lowId and f1.highId in your SQL for automatic thread handling. There are numerous examples shipped with
the product to illustrate this.
Consider using Bind Variables in your SQL. Several internal bind variables are available to use for your SQL
statement if necessary, to determine the subset of records to process.
Post Processing and Error Post Processing for Process Level Control. These algorithms are typically used to link
processes in the Oracle Scheduler, process chaining, file footer processing or process cleanup activities after the
last thread of the process has completed.

Generic Templates
The Oracle Utilities Application Framework provides a set of template batch controls, based upon base classes, that can be
cloned and used as a basis for custom plug in batch processes. This section will outline the features of the templates and
how they can be used to satisfy requirements.

F1-PDBEX - Plug In Driven Extract Template

One of the common requirements is to extract data in various formats from the database to a file (on local storage for on-
premise or to Object Storage on the Cloud). The base class,
com.splwg.base.domain.batch.pluginDriven.PluginDrivenExtractProcess, represented by the F1-PDBEX template
features the following:
Supports Single or Multiple Threads. The extract process can be single or multiple threaded for performance.
Multiple Extract Formats. The product supports the delimited, fixed or XML format that is configurable at runtime.
The delimited format allows for specification of the delimiter in either character or Unicode format.
Character Set Support. The base class supports java character sets to support character encoding. By default, files
are encoded in UTF-8.
Flexible File Naming. The name of the file can be fixed or use variables to form the name. In a multi-threaded
extract, thread number will be inserted into the name if not specified.
Flexible file Location Support. The base class supports the file adapter supplied with the Oracle Utilities
Application Framework to support named local storage or named Object Storage buckets for Oracle Cloud
implementations. In Oracle Utilities Application Framework 4.4.0.2.2 and above, storage bucket prefixes are
supported to segregate storage locations.
Concatenation Support. If multiple threads are used, it is possible to concatenate files across threads to produce a
single extract file.
Date Time Formatting. Global formats of dates and times in the extract can be configured to support XSD or
Framework formats.
Specification of Root Name. For XML based extracts it is possible to specify the root node for the extract. In Oracle
Utilities Application Framework 4.4.0.2.2 and above, it is possible to suppress the root node.
Manifest File Support. In Oracle Utilities Application Framework 4.4.0.2.2 and above, it is possible to generate a
JSON based manifest file that contains a summary of the processing of the extract including file name and hash
values for file comparison. An example of a manifest file is shown below:

Figure 24. Sample Manifest File

43 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 Compression Support. In Oracle Utilities Application Framework 4.4.0.2.2 and above, it is possible to have the
export file automatically compressed using the zip or gzip format using the appropriate file suffix on the file name.
If the multiple threads are used with concatenation, then gzip is only supported.
Override Thread Values. By default, the engine will process all records returned by the Select Records algorithm, it
is also possible to filter these values at execution time using thread level limits. Additionally, for on-premise
implementations, it is possible to implement custom java classes for additional filtering.

The processing of an extract can be summarized as follows:

Select Records Algorithm. This algorithm contains the SELECT SQL statement to select the records to process for
the extract. The SQL should be as precise as possible to avoid additional filtering as much as practical within the
SQL statement.
Process Record Algorithm. For each record returned from the Select Records algorithm, the Process Record
algorithm is invoked. The focus of this algorithm is to assemble the data for the extract, via relevant calls, and then
call the relevant routine to write the record. Additional filtering that is programmatic and not expressed in the SQL
can be performed in this routine to determine record eligibility. There are special internal variables within the
algorithm to determine if the record is the first and last in the set for header and footer processing. Code for this
algorithm may be in scripting language or Groovy.
Post Processing Algorithm (Optional). Additionally, if the outcome of the extract is successful, it is possible to
perform additional processing such as specialist footer processing (that is not included in the Process Record
algorithm).

A summary of the processing as well as generic example algorithms available with the product, is shown in the flow below:

Figure 25. Export Processing Flow

Note: Additional examples are supplied with products under the Batch Control – Select Records Algorithm Entity.

When using this template, the following guidelines apply:

Ensure the SQL SELECT statement used in the Select Records algorithm returns the set of records to process. Try to
avoid additional filtering later in the process unless it is unavoidable.
Use Multi-threading for large extracts. Start with a single thread process to avoid the need for concatenation unless
the volumes require multiple threads to be efficient.
Setup a schema with the record elements in the Process Record algorithm and move the data into the elements as
expected prior to any writing activity.
If the extract requires no header or footer processing, then you do not need to explicitly write the record in the
Process Record algorithm, it is implied at the exit of the algorithm for each record.

F1-PDBG - Plug In Driven Generic Template

One of the common use cases is to simply process a set of data for a business process. In this use case an SQL statement is
used to identify the data to process and then some code to process the record. The base class,
com.splwg.base.domain.batch.pluginDriven.PluginDrivenGenericProcess, provides the generic processing
capability.

44 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
The key algorithms for this use case are:

Select Records Algorithm. This is the algorithm that contains the SELECT SQL statement to filter the records to be
processed.
Process Record Algorithm. This the algorithm that takes each record passed from the Select Records algorithm
and performs the process as required on that record. This algorithm can also contain additional filtering that is
programmatic in nature that cannot be expressed in the SQL statement.

A summary of the processing as well as generic example algorithms available with the product, is shown in the flow below:

Figure 26. Generic Processing Flow

When using this template, the following guidelines apply:

Ensure the SQL SELECT statement used in the Select Records algorithm returns the set of records to process. Try to
avoid additional filtering later in the process unless it is unavoidable.
When using multi-threading build thread limits into the Select Records algorithm into the selection criteria.
The Select Records algorithm supports override values for Low and High Values to further limit the record criteria.
For non-SaaS implementations, a custom java class for record selection is also supported.

F1-PDUPL - Plug In Driven File Upload Template

Importing data from files from other systems is a use-case for batch processing. To simplify this process the plug-in batch
supports a specialist algorithm File Upload. This algorithm contains the format and logic necessary to read the file and send
each record to the Process Record algorithm. The base class,
com.splwg.base.domain.batch.pluginDriven.PluginDrivenUploadProcess, provides the generic import capability.
The template provides support for the following features:

File Adapter Support. The template can use the File Adapter capability to parameterize storage locations for reuse
across environments as an input File Path. Non-SaaS implementations this File Path can be located within local
storage and SaaS implementations can utilize Object Storage.
Multiple Formats. As with the export plug in batch template, the upload process supports delimited files, fixed files
and XML based files. Unlike the export process, the format chosen is explicit within the Process Record algorithm.
File Pattern Support. The specification of the filename can use wildcards to implement file patterns. In this case
the process the File Path location looking for files that match the pattern configured. The file pattern should use
glob syntax.
Configurable Character Encoding. As with the export capability, the import can support a configurable java
character sets that represents the encoding used on the files to be imported. By default, this is set to UTF-8.
Configurable Processed File Extension. Once a file is successfully processed it is renamed to include an extension
so that it is ignored for further processing. By default, this extension is PROCESSED.
Files Not Found Conditions. If no files are found with the file pattern or file name configured, you can configure
the process to ignore the fact that no files were found or return an error condition. The default is to return an error
if no files are found.
Header and Footer Support. As with the export capability, the Process Records algorithm has capabilities to detect
when the record is the first (header) or last (footer) for specialist processing, if required.
Decompression Support. In Oracle Utilities Application Framework 4.4.0.2.2 and above, it is possible to have the
import file automatically decompressed using the zip or gzip format using the appropriate file suffix on the file
name.

To help with the use of the template, several sample plug in scripts are available to process different formats:

45 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Plug In Upload Sample Scripts

SCRIPT USAGE

F1UplSmplDlm Sample Delimited File Upload

F1UplSmplFlt Sample Fixed Flat File Upload

F1UplSmplXML Sample XML File Upload

A summary of the processing as well as generic example algorithms available with the product, is shown in the flow below:

Figure 27. Import Processing Flow

When using this template, the following guidelines apply:

Decide the format of the import file and use the appropriate sample as a basis for custom File Upload algorithms.
Ensure the schema has the elements contained in the records you are processing.
Remember to include data areas and other schemas of objects you want to load.
If loading into staging objects, remember to set the status to the initial status of the related object.

Using Templates
Note: Whilst the templates are fully featured, examine algorithms shipped with similar batch controls for additional examples.

The provision of template is designed to streamline the development process for plug in batch. To use the templates,
perform the following:
Copy the template batch control to a custom batch control.
Configure the appropriate algorithms for the use cases for the batch process. These are configured on the batch
control. In the past sections, several samples can be copied, and those copies modified to suit your needs. Remove
any capabilities in those algorithms not necessary.
Configure the parameters for the custom batch control as you require.

BATCH REAL TIME MONITORING

By default, the Batch Run Tree records the performance information of a batch process that has executed or whenever a
commit has been issued, to record a checkpoint for restart. This is useful for post analysis but has limited use if you want to
actively monitor the process as it executes and need more timely information.
The batch framework includes a Java Management Extensions (JMX) based interface that permits the real time monitoring
of batch processes as they execute. The interface has several advantages:
It can be used to monitor active threads of a job. Once the job is not active it is not available via the API.
It exposes more up to date metrics for record counts and other metrics available on the Batch Run Tree.
Exposes additional metrics applicable at runtime like work units etc that are not recorded on the Batch Run Tree.

To use the interface the following must be configured:

46 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 The Batch JMX interface must be enabled as outlined in the Server Administration Guide.
A JSR-160 supported client must be used to access the interface, via the JMX RMI protocol, after it is enabled.
Products like jconsole, Java Mission Control etc, can be used to access the API in real time. Customers using the
Application Management Pack for Oracle Utilities have access to the API via the Utilities Batch target.
For more information about the interface refer to the Server Administration Guide and the Batch Troubleshooting Guide in
the Performance Troubleshooting Guideline Series (Doc Id: 560382.1) available from My Oracle Support.

BATCH LEVEL OF SERVICE

Note: This feature is only available in Oracle Utilities Application Framework V4.2.0.2.0 and above.

Note: For additional advice for Batch Level of Service refer to the Calling Batch Level of Service, Configuring Batch Level Of
Service, Building a custom Batch Level Of Service Algorithm and New Batch Level Of Service Algorithms articles.

Note: This capability is available on Oracle Utilities SaaS Cloud Services and on-premise implementations (including PaaS and
IaaS implementations). There are exclusive capabilities for Oracle Utilities SaaS implementations which will be indicated
appropriately.

In the Oracle Utilities products there is an inbuilt batch service level monitoring capability called Batch Level Of Service. This
capability, using configuration, assesses level of service using various metrics and batch instance run-time information to
produce the level of service and a message describing the service level. Once configured this capability can be used to
assess the service level associated with batch at any time using various avenues within the cloud service.
The Batch Level Of Service features the following capabilities:
Ability to define pre-built metrics to assess level of service. The product provides a library of metrics that can be
used to define how to assess the level of service.
Ability to define both warning and error target values. The Batch Level Of Service allows for a configuration of a
warning target to provide proactive detection of potential issues as well as an error target to detect unacceptable
performance.
Support for Individual batch jobs or job streams. The Batch Level Of Service can be configured to apply to
individual batch controls and/or job streams. Job Stream support is only available on the Oracle Utilities SaaS Cloud
if the inbuilt provided scheduler is used.
Support for multiple metrics and targets. The Batch Level Of Service supports multiple metrics and targets
allocated to individual batch controls and/or job streams. If multiple batch level of service configurations is
allocated, then the capability will assess each configuration and return the worst case of all assessments as the
overall assessment.
Flexibility on what run-time statistics are used. The library of pre-built metrics can assess the targets against
currently running executions, latest completed execution or both. The latter allows for flexibility to assess metrics
against running execution during the execution and then use completed execution for completed jobs or streams
(Oracle Utilities SaaS Cloud Services only). This means level of service can be assessed at any time for flexible
monitoring.
Multiple assessment interfaces. The invocation of the assessment is available automatically via the Health Check
portal, Health Check REST API and Batch Control maintenance functions. It can be also programmatically invoked
at any time using a provided business service.
Transparent to Execution. The Batch Level Of Service does not impact the execution of the batch processes that
they track as they read batch run statistics from batch instance tables maintained by the Oracle Utilities Application
Framework. The assessment is only performed when requested via the relevant API's or interfaces provided.

Recommended Prerequisite Setup

Before using the capability, the base algorithms for the health check must be configured on the Installation Options -
Framework under the Health Check System Event:
Health Check System Event Algorithms

SCRIPT USAGE

F1-HCBLOS Batch Level Of Service for Batch Controls

47 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 SCRIPT USAGE

K1-HCBJSLOS Batch Level Of Service for Batch Job Stream. This should only be configured if you are using the inbuilt
batch scheduling capability provided with the cloud service, directly or indirectly (via an external
scheduler). Customers using external schedulers to micromanage individual jobs cannot not use this
algorithm as the internal scheduler would not expose those metrics to the capability. For those
customers, it is recommended to use the SLA functionality provided by the external scheduler, if
available.

Note: This capability is exclusive to the Oracle Utilities SaaS Cloud Services and only if the inbuilt
scheduler is used on that cloud service.

Concepts
The Batch Level Of Service is an algorithm based assessment process which takes run metrics from batch instance tables to
assess performance against defined targets to produce a state and a message describing the level of service assessment.
This algorithm can be used with individual batch controls and/or Job Streams.
The concepts related to the Batch Level Of Service are as follows:
Batch Controls and Job Streams. The assessment of level of service is targeted at Batch Controls and Job
Streams. Job Streams are only supported if the inbuilt scheduling capability of the service is used for scheduling
jobs. Customers using external schedulers to schedule jobs cannot use this capability for external job streams. Job
Stream support is only available on the Oracle Utilities SaaS Cloud if the inbuilt provided scheduler is used.
Metrics are used to assess level of service. The capability includes a library of metrics, delivered as pre-
configured Algorithm Types, that are used to assess level of service. The metrics can be extended using
ConfigTools to add custom metrics as necessary.
Batch Instance Statistics are used to assess level of service using Status Applicability. The Batch Level of
Service can use statistics from executing processes or the statistics from the latest completed process to allow level
of service to be assessed at any phase of execution. Additionally, the provided level of service assessment process
can use both executing or completed for maximum flexibility in assessments.
Multiple Tolerances are supported. Whilst the service can be assessed for unacceptable behavior, the base Batch
Level Of Service capability supports a warning level as well. This promotes proactive monitoring.
Batch Instance Records are Used for Assessment. The batch architectures create batch instance records no
matter how a batch job is submitted collecting information on submission, execution and completion.
Returned Level Of Service And Message. Using the calculation of statistics batch instance records and the
tolerances, the Batch Level Of service returns the level of service and a message describing the level of service.

The process is summarized in the figure below:

Figure 28. Batch Level Of Service architecture

48 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Metrics available
The service includes a preset number of metrics that can be used to assess level of service. These represent the most
common metrics used for assessing level of service and become the basis for setting targets by creating algorithms based
upon the appropriate algorithm type. The below summarizes the metrics provided:
Batch Level Of Service Metrics

METRIC DESCRIPTION ALGORITHM SAAS NON- SCOPE

TYPE SAAS

Error Threshold Compare Records in Error to F1-BAT-ERLOS Batch Control

Threshold

Time Between Compare Error Count and Time F1-BAT-LVSVC Batch Control
Executions Since Completion

Elapsed Time Compare Total Batch Run Time to F1-BAT-RTLOS Batch Control
Threshold Threshold

Throughput Compare Throughput to Threshold F1-BAT-TPLOS Batch Control

Not Started On Time Batch Job Not Started in X Minutes K1BATJNSXM Batch Control

Low Number Of Not Processed Expected Number Of K1BATJPLNR Batch Control

Records Processed Records

Job Ran Too Long Batch Job Ran Longer Than Average K1BATLOSRTL Batch Control
(Relative)

Throughput Too Low Batch Job Throughput Lower Than K1BATLOSTTL Batch Control
(Relative) Average

Batch Job Stream Not Compare Batch Job Stream run date K1BJSDJNSXM Batch Job Stream
Started in X Minutes plus threshold to current time

Job Stream Ran Too Batch Job Stream Ran Longer Than K1BJSDJRTL Batch Job Stream
Long (Relative Run) Average

Batch Job Stream Has Batch Job Stream Has Failed K1-BJSD-LVSV Batch Job Stream
Failed

Status Applicability
Note: These settings are applicable to the base provided algorithm types. They may not apply to custom algorithm types.

The base level of service algorithm type support configuring the latest batch instance records to be used as part of the
assessment. These can be configured on the individual Algorithms configured to be used on Batch Controls or Job Streams.
The Status Applicability settings are as follows:
Status Applicability

APPLICABILITY DESCRIPTION SETTING WHERE TO USE

In Progress Assess against the current batch runs that I This is recommended only to be used when
are in progress. If this value is set and there you have monitor jobs that are run
is no batch run currently in progress, the continuously or in timed mode.

49 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 APPLICABILITY DESCRIPTION SETTING WHERE TO USE

algorithm returns disabled as the Level of

Service.

Complete Assess against the most recent completed C This is recommended to be used for jobs that
batch run. are not actively monitored during execution.
This is used when after the fact monitoring is
exclusively performed.

Both Assess against current executing batch run if B This is recommended for most assessments to
the process is executing. If there is no cover both executing and completed processes
running process, then the most recent effectively. This allows you to assess in context
complete batch run for assessment. to when the assessment is requested.

Setting Thresholds
The last piece of the algorithm configuration is to set two thresholds to be used in assessment.
Thresholds

THRESHOLD COMMENTS

Warning This tolerance is used to proactively detect the threshold is close to the error level but sufficiently below
that it should be addressed before it exceeds the error threshold. This should be set to a value that is
enough to generate a warning appropriately but not so close to the error threshold that it does not give
you enough time to address the situation appropriately.

Error This tolerance is used to detect potentially unacceptable behavior in relation to the level of service.

Building Your Own Algorithm Types

The provided metrics as supplied in pre-built algorithm types represent the most common metrics used for assessment. It is
possible to build your own algorithm creating your own metrics to assess. When building custom algorithm types the
following should be considered:
Mirror the features of the pre-built algorithm types, if possible. Ensure your custom algorithm type supports
Status Applicability and implements both warning and error thresholds to keep the experience consistent with the
provided content.
Keep the logic simple. It is tempting to combine a large complex set of metrics, additional parameters and
features to assess level of service. Balance functionality and performance.
Remember your custom code may be used with base code. It is possible to implement more than one algorithm
per batch control and job stream (Oracle Utilities SaaS Cloud only). Avoid duplication of functionality as you may
duplicate logic and confuse configuration.
Refer to Building A Batch Level Of Service Algorithm for an example of a simple Batch Level Of Service algorithm.

Note: The base algorithm types supplied are provided with source code that can be copied and used as a basis for your own
algorithms.

Setting Up Batch Level of Service Algorithms

To use Batch Level Service the first step is to create algorithm entries with the relevant algorithm type and applicable
tolerances. The technique to do this is the same process used for all other algorithms with the following steps:
Navigate to Algorithm Maintenance menu option. Navigate to the Algorithm maintenance capability using the
Administration menu or use the Search Menu function.

50 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 Check for existing algorithms first. Before adding new algorithms, use the Algorithm Search with the Batch
Control - Level Of Service or Batch Job Stream - Level Of Service (Oracle Utilities SaaS Cloud only) in the Algorithm
Entity filter to ascertain whether algorithms for use already exist. For example:

Figure 29. Batch Level Of Service search

Create an algorithm. Use Add (or copy) to create a new algorithm with the following details:
− Choose the appropriate Algorithm Type. From the library of available algorithm types select the
algorithm type that will use the appropriate metric to use in the calculation.
− Choose the appropriate Status Applicability. If using the provided algorithm types, decide whether you
want to use In Progress, Completed or both statistics in your calculation.
− Set your Warning and Error Threshold. Set the warning and error tolerances to appropriate values.
− Name the Algorithm appropriately. To ensure you find the algorithm easily to attach to the batch
controls and/or batch streams (Oracle Utilities SaaS Cloud only), name it appropriately.

Allocating Batch Level Of Service to Batch Controls

Once the Batch Control - Level Of Service algorithm has been created it needs to be assigned to a Batch Control for use. This
can be done using the following process:
Navigate to Batch Control Maintenance. Open the desired batch control within the Batch Control maintenance
menu option either from the Administration menu or using the Search Menu function.
Navigate to the Algorithms Tab. To add the algorithm, navigate to the Algorithms tab.
Specify the Level Of Service Algorithm. Add the Level Of Service algorithm with the appropriate sequence
number. Remember it is possible to add more than one Level Of Service algorithm to support complex
requirements.

For example:

51 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Figure 30. Batch Level Of Service algorithm on Batch Control

Save the Changes. Save the changes to reflect the algorithm onto the Batch Control.
Confirmation. You can confirm the change by looking at the Level Of Service assessment on the Main tab of the
Batch Control maintenance.

Allocating Batch Level Of Service to Job Stream Definitions

For Oracle Utilities SaaS Cloud Service implementation using the inbuilt batch scheduler capability, it is possible to attach a
level of service to individual Batch Job Stream Definitions. To do this the following must be done:
Navigate to Batch Job Stream Definition Maintenance. Open the desired batch job stream, within the Batch Job
Stream Definition maintenance menu option either from the Administration menu or using the Search Menu
function.
Navigate to the Level Of Service section. Navigate to the last section of the maintenance page to the Level Of
Service section.
Attach the Level Of Service Algorithm. Link the appropriate Level Of Service algorithm with an appropriate
sequence number.

For example:

52 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Figure 31. Batch Level Of Service algorithm on Batch Job Stream

Assessing Batch Level Of Service

The Batch Level of Service is assessed upon request through various features. The features that initiate and report Batch
Level Of Service include the following:
Health Check. The primary method of access the Batch Level Of Service for all configured batch controls and batch
job streams is to use the Health Check function. This is available via several ways:
− Health Check Screen. A screen is available in the product that initiates assessment and returns the
assessment including the global assessment and the detailed assessment for each configured Batch Level
Of Service. This screen can be used by any authorized user to perform assessment at any time.
− Health Check REST API. The Health Check is available as a REST based service to allow external
monitoring products and notification systems to initiate assessment and receive results. This can be
performed from any REST based client including curl.
Batch Control Maintenance. After configuring Batch Level of Service, the assessment will be performed and
displayed on the Batch Control Maintenance.
Batch Job Stream Maintenance (coming soon). For Oracle Utilities SaaS Cloud customers using the inbuilt
scheduler, the Batch Level Of Service for the Batch Job Stream, if configured, will appear on the Batch Job Stream
Maintenance.
Programmatically via Business Service. For implementations who wish to invoke the assessment as part of an
extension, the product provides a Business Service, F1-BatchLevelOfService, to be used to initiate an assessment
on a Batch Control or Batch Job Stream (SaaS Customers only).

The figure below summarizes these features:

53 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
Figure 32. Features initiating Batch Level Of Service.

Using Health Check to Monitor Batch Level Of Service

Once the level of service algorithms is attached to Batch Controls and/or Batch Job Streams (Oracle Utilities SaaS Cloud
only), the assessment of level of service is available without the need for redeployment. The main capability to invoke the
level of service for all configured batch controls and batch job streams is the Health Check. This will perform the following:
For each batch control and batch job stream (Oracle Utilities SaaS Cloud only) configured with at least one level of
service algorithm it will invoke the algorithm and return the assessed level of service, the status reason message
describing that level of service and response.
If the batch control or batch job stream (Oracle Utilities SaaS Cloud only) has more than one algorithm defined, the
health check will invoke each algorithm individually. The overall level of service is the worst status of all the
assessments for that batch control or batch job stream.
The overall response will be the worst status of all assessments with the appropriate HTTP status code.

For example:

Figure 33. Example Health Check

Note: The Health Check is also available as a REST API for external monitoring tool use. Refer to the documentation for
additional information about the API.

Using Batch Control Maintenance to Monitor Batch Level Of Service

54 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
One of the easiest ways of verifying batch level of service is to use the Batch Control maintenance function. After finding the
batch control and loading it into the maintenance screen the current level of service is displayed. For example:

Figure 34. Example Batch Level Of Service within Batch Control Maintenance

This is useful for verifying after making algorithm changes and saving those changes.

Invoking Batch Level Of Service from Extensions

Whilst the Health Check and Batch Control maintenance are the common ways of invoking the batch level of service, it is
also possible to assess batch level of service in extensions using the F1-BatchLevelOfService business service. This
service will return the level of service and message parameters describing that level.
The schema for the Batch Level Of Service is as follows:
<schema name="schema">
<input type="group">
<batchControlId mapField="BATCH_CD"/>
</input>
<output type="group">
<levelOfService dataType="lookup" mapField="F1_BATCH_LEVEL_OF_SERVICE_FLG"
lookup="F1_BATCH_LEVEL_OF_SERVICE_FLG"/>
<levelOfServiceReason mapField="LEVEL_OF_SERVICE_REASON"/>
<messageCategory dataType="number" mapField="MESSAGE_CAT_NBR"/>
<messageNumber dataType="number" mapField="MESSAGE_NBR"/>
<messageParameters type="group">
<parameters type="list" mapList="MSG_PARMS">
<parameterType dataType="lookup" mapField="MSG_PARM_TYP_FLG"
lookup="MSG_PARM_TYP_FLG"/>
<parameterValue mapField="MSG_PARM_VAL"/>
</parameters>
</messageParameters>
</output>
</schema>

Note: The schema above is the schema at the time of publication and may differ from the schema delivered.

To use this service, you need to populate the batchControlId input parameter when calling the service for the service to
return the message and levelOfService.
This business service can be invoked from an extension in the following ways:
Service Script. Include the F1-BatchLevelOfService service as a Data Area attached to the script and use
invokeBS to call the business service. For example:

move "parm/batchControlId" to "F1-BatchLevelOfService/input/batchControlId";

invokeBS 'F1-BatchLevelOfService' using "F1-BatchLevelOfService";
Query Portal. Use the source=bs tag in your column with a call to the F1-BatchLevelOfService service passing
the column number that contains the Batch Control Id. For example:

source=BS bs='F1-BatchLevelOfService' input=[input/batchControlId=C1]

output=output/levelOfService

55 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
As standard the status message can be formatted using the F1-ReturnMessage business service using the
messageCatgeory, messageNumber and messageParemeters elements.
Refer to Calling Batch Level Of Service for additional information.

Invoking Health Check from an external system

Note: curl is used as an example in this section. Any external system with REST support can call the API natively, if possible.

One of the common use cases is being able to poll the Health Check on a regular basis within a script executed from an on-
premise (or another cloud) to pull the information into another system (such as a management console) or trigger
notifications via a company notification system. The focus is to call the curl command shown in the previous section,
interpret the results (either the overall status or individual status values) and take the appropriate action using an
appropriate channel.
To do this the curl command is incorporated into an appropriate scripting language to form the basis of processing.
Refer to Oracle Utilities System Health Check Probe (Doc Id: 2711546.1) available from My Oracle Support for a sample script
that illustrates a common processing structure for formatting the call via curl (to avoid hardcoding), interpreting the results
and then taking a configured action.

Implementation Guidelines for Batch Level Of Service

The Batch Level Of Service can be a powerful monitoring mechanism for bath processing but there are a few guidelines
which will optimize the experience:
Agree Tolerances. The most important aspect of using this capability is to agree to the tolerances balancing
business expectation and technical considerations to set both the warning and error tolerance. Avoid just setting
random values.
Avoid False Negatives. This capability is designed around service levels and expressing those services levels
appropriately will avoid the capability reporting false negatives.
Use Warnings to Be Proactive. Ensure you set the warning tolerance to give you sufficient warning of an impeding
error condition. Set it far enough away from the error tolerance to generate the warning appropriately but not too
far away.
Target Job Streams Initially. If your implementation is on the Oracle Utilities SaaS Cloud Service using the inbuilt
job scheduler to schedule processes, then consider using the Batch Job Stream level of service initially to reduce
initial costs of implementation. Setting batch window level tracking is an easier way to start using the capability.
Target Critical Jobs First. It is easy to assume that you will use this facility for every single batch process but in
most cases batch processes run in a very short time. Avoid too much micromanagement by at first targeting the
most business critical processes first.
Only Use Where Necessary. As complementary to the last advice, only set the level of service on batch processes
and/or batch job streams (Oracle Utilities SaaS Cloud Services only) you want to track. Micromanagement can add
unnecessary costs to an implementation.

BATCH SCHEDULE OPTIMIZATION

One of the most common implementation tasks for Oracle Utilities products, whether you are on-premise or enjoying the
benefits of the Oracle Utilities SaaS Cloud, batch jobs need to be managed and scheduled appropriately. This applies
regardless of which scheduling product you are using, whether this is the inbuilt scheduler in the Oracle Utilities SaaS Cloud
Service, DBMS Scheduler or an external third-party scheduler.
Managing batch processes is relegated to the responsibility of that batch scheduler. The batch scheduler is responsible for
executing individual jobs, managing the dependencies and to a lesser extent, providing resource management for
execution. There are techniques and processes for optimizing the schedule to take advantage of resources and
dependencies.

Concepts
Before outlining the techniques and processes for optimizing there are several concepts that must be understood:

56 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 Schedulers are responsible for execution of individual jobs. When a job is to be executed the scheduler would
initiate the execution of this job to the Oracle Utilities product. Schedulers can use a variety of methods to initiate
execution via command line, REST API or scheduler integrations.
Schedulers are responsible for managing dependencies. This is one of the important reasons why schedulers
exist. Certain Jobs depend on the success (or failure) of other jobs for several key reasons:
− Data Dependency. The data from one job feeds into another job, so therefore there is a dependency,
known as a hard dependency, between the jobs.
− Status Dependency. The state of one job directly impacts another jobs. For example, a failure of one job
might initiate a job to perform cleanup processes.
− Resource Dependency. A resource such as a shared tables or shared resources (CPU, memory, disk, etc)
mean that there is a dependency, known as a soft dependency, between the jobs. This is usually
associated with avoiding contention of any kind. For example, a series of jobs that use the same set of
tables may have a soft dependency to ensure each individual job has enough level of access to those
tables common to those processes to reduce contention.
For example:

Figure 35. Example Job dependency

Schedulers are responsible for calendaring. The scheduler is responsible for running the jobs and their
dependencies where and when it is necessary. This is associated with a calendar which governs applicable times
and dates for execution.
Schedulers interface to resource management capabilities. One of the other important characteristics of a
scheduler is the ability to take capacity into account when scheduling processes. This helps the execution adhere to
any physical or implied resource constraints.
Jobs are grouped into Job Streams. Regardless of the scheduler and its individual terminology, jobs (instances of
a batch control to be executed) are grouped in logical streams (a.k.a. job streams) with the expression of
dependencies and a calendar. Terminology for these concepts may vary but for the purposes of this document the
terms job and job streams will be used.
Serial and parallel processing is supported. When using a scheduler, job streams are not restricted to a serial
sequence of individual jobs as this is not efficient unless dependencies restrict the stream to be serial. In any
collection of dependencies, the concept of parallel jobs running is typically recommended within resource
constraints.
Critical Jobs need to be run as early as practical. In any depiction of a job stream, based upon execution times
and business direction, there will be a subset of the stream that are considered critical jobs. These are business
important jobs or jobs that can influence the overall elapsed time of a job stream. The goal of any optimized
schedule is to try and schedule these types of processes as early as practical and with efficient access to resources
to maximize throughput.

These concepts will be used across each of the schedule optimization techniques.

Guidelines for Optimizing Schedules

When optimizing a stream of jobs (a.k.a. a schedule), consider the following techniques:
Remove unused or unnecessary jobs. If you are using a provided schedule from your partner or via an
accelerator, not all the jobs will apply to your implementation. Not all functionality is used by all implementations
expressed in the jobs. Therefore, remove jobs not needed and adjust dependencies accordingly. For example:

57 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
 Figure 36. Example job removal and adjusted dependencies.

Identify business critical processes. In every schedule there are a handful of jobs that are considered business
critical. These are the main processing that must be performed during the schedule. They will become the focus of
optimization as their efficient execution will help ensure success, therefore quickly identifying them is important.
Adjust dependencies for critical processes. The goal of scheduling is not only to run each job but run them
efficiently. The key to success is therefore to ensure that the critical jobs are run with enough resources and as early
as possible to improve overall efficiency. This means adjust less critical jobs to be "out of the way" of the critical
jobs, either after the critical jobs or in parallel to other less critical jobs to help get them complete before the critical
job. Dependencies that are necessary still need to be respected. For example, dedicated To Do batch processes in
an overnight process are typically scheduled alongside jobs that pertain to. Whilst, this seems logical, as To Do
processes are manually processed by humans during business hours, so therefor may be shifted to later in a
schedule safely whilst freeing up time for more critical processes.
Examine each dependency using several dimensions. Examine each individual dependency to ascertain whether
the relationship is necessary for data, status or resource purposes. Unnecessary dependencies may artificially
lengthen the overall duration of a schedule. If the dependency is not needed, then adjust the dependencies
accordingly including running the jobs in parallel or in a different location in the schedule.
Use Parallel Processing effectively. In adjusting dependencies one of the techniques is to consider running more
jobs in parallel. Whilst, this is valid, running too much at the same time can be inefficient from a resource
perspective. Balance running jobs in parallel with the resource usage for each job to ensure the jobs run within your
resource constraints. Remember at any point in time, the cumulative number of active threads needs to be
considered against the thread limits to avoid delayed threads (this may be detected where individual thread start
times vary across a range). For example:

Figure 37. Example thread capacity issue.

Implementing Stream Conditions for Oracle Utilities SaaS

Note: The technique described in this section only applies to Oracle Utilities SaaS Cloud customers using the inbuilt scheduler
functionality. Customer on alternative platforms and/or alternative schedulers can use the capabilities provided by the external
scheduler used. Refer to the documentation provided by that external scheduler for advice in this area.

Note: This technique is best used with schedules are run multiple times or in an interval to skip individual executions.

A scenario that arises from time to time is to prevent an execution of a stream due to a condition or the stream will only
proceed if a condition is met. The conditions for this can vary from simply detecting another stream is executing to complex

58 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
data relationships that need to be checked. This type of requirement can be implemented using a custom job that checks
the condition and returns an appropriate value and depending on the return value continues processing the stream or
jumps to the end of the stream to bypass. For example:

Figure 38. Example job stream with stream condition

To implement this logic in your schedule the following process should be undertaken:
Create a Service Script with the Logic. Create a service script containing the logic you want to implement passing
in any parameters in the Batch Control configuration, to avoid hardcoding. For example, to check other streams are
active in the inbuilt scheduler implementation for SaaS customers, use the F1-DBMSGetJobDetails business
service to check the execution status of another flow with the name element containing the name of the Job
Stream and the isInProgress boolean being used to ascertain the running state of the job and the appropriate
termination statement (optionally with message code). If your script requires custom messages, ensure the
messages used are configured prior to implementing the script.
Note: For customers on Oracle Utilities SaaS Cloud implementations, the Cloud Service Foundation includes several
sample scripts such as K1InvokePAS provide ways to process parameters using advanced techniques.

Create a Batch Control. Create a Batch Control to house the logic for the job. For Oracle Utilities SaaS Cloud
customers clone the provided Batch Control K1-IPAIS (Post Activation Installation Scripts). This batch control has
several parameters including a script name and parameter, with up to five (5) scripts supported. The parameter can
be used to pass in a value to each script to compare against.
Configure the Batch Control to use the script. Configure the batch control with the script configured and any
parameter on the job you want to pass in.
Alter the Job Stream. All the new job to the job stream in the appropriate place. Alter the dependencies according
to the logic. To bypass the stream, the end event should make this new job a prerequisite with the appropriate
conditions.

Anytime the stream is now executed, the new job will determine the conditions and depending on the result take the
appropriate action.
This technique can be used for a variety of use cases including cross stream checks.

59 TECHNICAL BRIEF | Batch Best Practices | Version

4.4.0.2.3 Copyright © 2020, Oracle and/or its
affiliates | Public
CONNECT WITH US
Call +1.800.ORACLE1 or visit oracle.com.
Outside North America, find your local office at oracle.com/contact.

blogs.oracle.com/utilities linkedin.com/company/oracle-utilities twitter.com/oracleutilities

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the contents hereof are subject to change without
notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties
and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document, and no contractual obligations are formed
either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without
our prior written permission.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of
SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered
trademark of The Open Group. 0120

Batch Best Practices - Oracle Utilities Application Framework

September 2020
Author: Anthony Shorten, Senior Principal Product Manager, Utilities Global Business Unit