Scribd
Upload
2 views

Efficient Data Retrieval using in SAP ISCI with Adapter

The document discusses the implementation of efficient data retrieval using pagination in SAP Cloud Integration (SAP CI) with the HTTP adapter. It explains the concept of pagination, its importance in managing large datasets, and provides a step-by-step guide on how to set up pagination, including Groovy scripts for dynamic calculations. The approach allows for incremental data retrieval while addressing specific database constraints, ultimately enhancing system performance and integration efficiency.

Uploaded by

jaydata2018
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
2 views

Efficient Data Retrieval using in SAP ISCI with Adapter

The document discusses the implementation of efficient data retrieval using pagination in SAP Cloud Integration (SAP CI) with the HTTP adapter. It explains the concept of pagination, its importance in managing large datasets, and provides a step-by-step guide on how to set up pagination, including Groovy scripts for dynamic calculations. The approach allows for incremental data retrieval while addressing specific database constraints, ultimately enhancing system performance and integration efficiency.

Uploaded by

jaydata2018
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 6

Efficient Data Retrieval using Pagination in SAP BTP IS/CI with

HTTP Adapter

Efficient Data Retrieval using Pagination in SAP B... - SAP Community

Introduction:

Efficiently handling large datasets is a critical aspect of modern enterprise integration.


When retrieving data from databases using SAP Cloud Integration (SAP CI), leveraging
pagination can significantly enhance performance and prevent overloading systems. In
this blog, we’ll explore how to implement pagination using the HTTP adapter in SAP CI,
ensuring efficient and seamless data processing.

What is Pagination?

Pagination is the process of dividing large data sets into smaller chunks (pages) to be
retrieved incrementally. This approach prevents excessive memory usage and reduces
the risk of timeouts or errors when processing large volumes of data.

Use Case:

In many cases, SAP CI provides a standard OData adapter that supports pagination.
However, there are scenarios where the source application, such as a database,
imposes restrictions and only allows HTTP calls from outside the network. Due to this
limitation, the standard OData adapter cannot be used directly. Instead, a customized
pagination approach is implemented in SAP CI to handle data retrieval efficiently. This
blog demonstrates how to achieve this using the HTTP adapter.

Steps to Implement Pagination in SAP CI:

1. Design the Integration Flow

o Use the HTTP adapter to connect to the database API or service.

o Configure the integration flow to handle paginated requests and


responses.
2. Start the Pagination Process

o Begin by making an initial HTTP call to retrieve the total count of records
from the database.

o Use the count query provided by the database team to fetch the total
count records information.

Example response: {
"TotalRecCount": 50000
}

3. Calculate Page Count and Loop Parameters

 Initially set the Pagenumber to 0 so that will be used to increment the


pagenumber in the data retrieving process, and Recordsperpage is 2000 in my
case as the database allowed only a maximum of 2000 records per one page
that we can retrieve.

 Use a Groovy script to calculate the number of pages required to retrieve all
records based on the total count and records per page.
 In this example, let’s assume the total count of records is 50,000 and the source
application can process 2,000 records per page. The script calculates the page
count by dividing the total count by the records per page and then rounding up to
ensure all records are retrieved.

import com.sap.gateway.ip.core.customdev.util.Message;
import java.lang.*;

def Message processData(Message message) {


def body = message.getBody();
def map = message.getHeaders();
def value1 = map.get("TotalRecCount");
def value2 = map.get("RecordsPerPage");
if (value1 == null) {
def A = 0;
} else {
A = Integer.parseInt(value1);
B = Integer.parseInt(value2);
C = Math.ceil(A / B);
D = Math.round(C);
}
message.setHeader("PagingLoopCount", D);
message.setBody(body);
return message;
}

In this scenario:

 Total Records = 50,000


 Records Per Page = 2,000

 Pages Required = ceil(50000 / 2000) = 25

The script ensures all records are retrieved by calculating and rounding up the page
count.

4. Looping Process Call Configuration

 Use the "Looping Process Call" to retrieve data page by page.

 Configure the loop condition to continue until the Pagenumber is less than or
equal to the PagingLoopCount

Example configuration:

This ensures that the process continues to loop and retrieve data until all pages are
processed.
Local Integration process:

 This local integration process works to retrieve the actual data from the database
in a looping mode. First, it will retrieve the page 1 data using the data query.
Once the data is retrieved, as per the given script, it will increment the
Pagenumber by +1 for every looping call.

Groovy Script to Increment Page Number:

import com.sap.gateway.ip.core.customdev.util.Message;

def Message processData(Message message) {


def body = message.getBody();
def pageNumber;
def headers = message.getHeaders();
def value = headers.get("Pagenumber");
def value1 = value.toInteger()
value1 = value1 + 1;
message.setHeader("Pagenumber", value1);
message.setBody(body);
return message;
}
Sample HTTP query:

{(PageNumber:${header.Pagenumber},pageSize: ${header.RecordsPerPage}}

The Pagenumber and RecordsPerPage headers will be parsed into the actual data
query to pull the managed data from the database, ensuring compatibility with the
source application.

Conclusion:

Pagination is a powerful and essential technique to manage and process large data sets
efficiently in SAP CPI. This blog demonstrated how to implement customized pagination
using the HTTP adapter, addressing the limitations of standard OData pagination when
direct HTTP calls are required by the database. By leveraging Groovy scripts for
dynamic page calculations and loop management, you can:

 Retrieve data incrementally without overloading the database or SAP CI


resources.

 Adapt to database-specific constraints, such as the maximum number of records


per page.

 Ensure all data is retrieved accurately and efficiently.

With this approach, enterprises can streamline data retrieval processes, optimize
system performance, and scale their integration solutions.

You might also like