Pagination
Pagination
Can you please make a video of an odata query that returns thousands of records ( e.g 20.000 ) and
write it to an ftp server as a single csv file? I heard of odata pagination, page size and looping process call
can be used to retrieve very large records, but I don't know how it works. I would be very happy if you
can solve this.
----------------------------------------------------------------------------------------------------------------------------------------
The OData API provides several pagination options for query results. Some key concepts about
pagination must be understood before choosing the integration design.
🔺 A single OData HTTP GET request can return at most 1,000 records.
🔺 The OData standard provides a '__next' link in your query response if there are more results in the
database.
🔺 Pagination only works for entity sets. Function imports do not support pagination.
🔺Client-side pagination uses query options on the client side to create an offset that restricts the
amount of data returned from the server.
For example, a query with $skip=2000&$top=500 returns the fifth page of data where the page size is
500.
🔺Cursor-based pagination maintains a database "cursor" on the server throughout pagination HTTP
requests.
🔺Snapshot-based pagination is a server-side pagination that keeps a list of all business keys of the data
set on the server.
💡 How Do I Choose?
As a developer, you can choose among the different types of pagination based on their advantages and
disadvantages. Here are the recommendations by SAP:
Always use client offset pagination for UI consumptions. This is the only supported method on UI.
Cursor-based pagination and snapshot-based pagination are recommended for integration use cases.
Only use snapshot-based pagination for complex queries with large data volume, for example, a query
that has complex sorting, filtering, and expanding options.
It is recommended to tune your queries when you use snapshot-based pagination.
Efficient Data Retrieval using Pagination in SAP BTP IS/CI with HTTP Adapter
Efficiently handling large datasets is a critical aspect of modern enterprise integration. When retrieving
data from databases using SAP Cloud Integration (SAP CI), leveraging pagination can significantly
enhance performance and prevent overloading systems. In this blog, we’ll explore how to implement
pagination using the HTTP adapter in SAP CI, ensuring efficient and seamless data processing.
What is Pagination?
Pagination is the process of dividing large data sets into smaller chunks (pages) to be retrieved
incrementally. This approach prevents excessive memory usage and reduces the risk of timeouts or
errors when processing large volumes of data.
Use Case:
In many cases, SAP CI provides a standard OData adapter that supports pagination. However, there are
scenarios where the source application, such as a database, imposes restrictions and only allows HTTP
calls from outside the network. Due to this limitation, the standard OData adapter cannot be used
directly. Instead, a customized pagination approach is implemented in SAP CI to handle data retrieval
efficiently. This blog demonstrates how to achieve this using the HTTP adapter.
o Begin by making an initial HTTP call to retrieve the total count of records from the
database.
o Use the count query provided by the database team to fetch the total count records
information.
Example response: {
"TotalRecCount": 50000
}
Use a Groovy script to calculate the number of pages required to retrieve all records based on
the total count and records per page.
In this example, let’s assume the total count of records is 50,000 and the source application can
process 2,000 records per page. The script calculates the page count by dividing the total count
by the records per page and then rounding up to ensure all records are retrieved.
import com.sap.gateway.ip.core.customdev.util.Message;
import java.lang.*;
In this scenario:
The script ensures all records are retrieved by calculating and rounding up the page count.
Configure the loop condition to continue until the Pagenumber is less than or equal to the
PagingLoopCount
Example configuration:
This ensures that the process continues to loop and retrieve data until all pages are processed.
Local Integration process:
This local integration process works to retrieve the actual data from the database in a looping
mode. First, it will retrieve the page 1 data using the data query. Once the data is retrieved, as
per the given script, it will increment the Pagenumber by +1 for every looping call.
import com.sap.gateway.ip.core.customdev.util.Message;
{(PageNumber:${header.Pagenumber},pageSize: ${header.RecordsPerPage}}
The Pagenumber and RecordsPerPage headers will be parsed into the actual data query to pull the
managed data from the database, ensuring compatibility with the source application.
Conclusion:
Pagination is a powerful and essential technique to manage and process large data sets efficiently in SAP
CPI. This blog demonstrated how to implement customized pagination using the HTTP adapter,
addressing the limitations of standard OData pagination when direct HTTP calls are required by the
database. By leveraging Groovy scripts for dynamic page calculations and loop management, you can:
Adapt to database-specific constraints, such as the maximum number of records per page.
With this approach, enterprises can streamline data retrieval processes, optimize system performance,
and scale their integration solutions.
Question: how does one combine all the payloads received from each call in loop. E.g. I have to create
file a single xml file on receiver end with all the pagination calls.
Answer: At the end of the local integration process, we need to store the XML records in the datastore.
In the next Iflow When retrieving the collected data from the datastore, we can set the maximum
number of polling messages to ensure all data is retrieved in a single file, which can then be sent to the
receiver.
Looping Process Call refers to a way of repeating processes in an SAP program using loops, calling one or
more processes within a specific cycle. This can be useful, especially when working with large data sets
or when automating a series of tasks. f we are working with large datasets, fetching them all at once
increases RAM consumption.
With this method, we retrieve our data in fragments based on a certain condition, and after the loop
ends, our data is merged as a whole. From a performance perspective, we alleviate the strain on
memory. It reduces processing time and enhances performance. It simplifies our overall data analysis.
Now, we will design a scenario in cloud integration. We will fetch large data from OData and loop it
through a specific condition (looping process call). Then, we will observe the results together.
Step 1. First, we will create a CPI link to be able to make calls to the service.
Figure 2. Sender HTTPS Adapter
Step 2. We specify that the looping process call will work according to the condition expression specified
in the "condition expression" field. By stating ".hasMoreRecords contains 'true', we indicate that the
loop will continue to run as long as there are multiple records. You can take a look at (hasMoreRecords).
Step 4.We use the "select" clause to choose which fields we want to retrieve from the Orders entity.
We need to mark "Process in Pages". If we don't mark it, the system will send all the data at once after
entering the loop once.
Figure 5.Odata Adapter Processing Information
Step 5.After passing through the filter, the data will no longer include "Orders" but will start with
"Order." This is because we need the information of "Orders/Order" due to sending the data in
fragments. After completing the process of sending fragmented data, we will merge it in the "Message
Body" of the Content Modifier.
Figure 6.Filter
Step 7.We add the "Orders" tag, which we ignored with the filter, to this content modifier. Once the
loop is completely finished, we add the merged data as a property.
Figure 8.Content Modifier-Message Body
Step 8.To indicate that the last data will come in XML format, we add the "Content-Type" header.
Step 10.We determine who the email will come from and who it will go to.
Step 11. Save and Deploy. Then once we have created a CPI link, we need to call it using the GET method
in Postman after the deployment.
Step 12. We are making a call to the CPI service using the CPI username and password.
Figure 12.Postman
Step 13. It entered the loop a total of 6 times, but on the 6th request, since there was no data left inside,
it combined the data sent in fragments, exited the loop, and continued to 'End'.
Figure 12.Monitoring
When we look at our first loop, it indicates that in the first request, it fetched the first 200 records from
the entire data set and provided the information that the next loop would start with OrderID 10448
using the expression "$skiptoken=10447".
In each loop, as it adds data, it indicates that there were 400 records in the 2nd request, and when it
enters the 3rd loop, it won't fetch the same initial 400 records again. Similarly, it shows that in the next
loop, the data will start with OrderID 10648.
The important point to note is that it continues to loop as long as the condition we set is met, meaning it
enters the loop as long as it evaluates to true.
When we check the final step, we understand that this condition returns false, indicating that it has
fetched all the data inside.
Due to the condition, since the Loop process has ended, we receive information that the last data has
the OrderID number 11047.
Finally, I wanted to add an email adapter. It notifies all the information via email.
Fetch data in chunks using pagination from S/4 Hana Cloud's OData API
Introduction: This document describes how to fetch data in chunks using pagination from S/4 Hana
cloud's OData API.
In the context of OData (Open Data Protocol), pagination refers to the practice of dividing a large set of
data into smaller, more manageable chunks or pages. This is done to improve the performance of data
retrieval and to reduce the amount of data transferred over the network. Pagination is a common
technique in APIs and web services to handle large result sets efficiently.
In OData, pagination is typically achieved through the use of query parameters, specifically the $skip and
$top parameters. Here's a brief explanation of these parameters:
1. $skip: This parameter is used to specify the number of items that should be skipped
from the beginning of the result set. It is often used in conjunction with $top to
implement paging. For example, if you want to retrieve results 11 to 20, you would set
$skip=10.
1. $top: This parameter is used to specify the maximum number of items to be returned in
the result set. It works in conjunction with $skip to define the size of each page. For
example, if you want to retrieve the first 10 items, you would set $top=10.
1. $inlinecount: This parameter is used to get total count of records by passing value
"allpages" to this parameter.
By using $skip,$top and $inlinecount together, you can navigate through the result set in chunks,
effectively implementing pagination.
For instance, to retrieve the second page of results (items 501-600), you might use $top=100 and
$skip=500.
Now, you have a scenario where you need to fetch the full load from the Business Partner API or any
other API of S/4 Hana public cloud with 500 records per page.
in response, you will get 500 record if records are equal to 500 or more else full load will come in first
page it self.
Along with these records, you will also get count of total records in "count" element
if (count>500)
then calculate the number of API call you need to make based on total count
To do all this, you need to write a program in language supported by you application or middleware
system.
As you can see in the below picture, fetching data from S/4 Hana, transform the data in Salesforce Soap
API format and transferring to Salesforce system is happing in "Local Integration Process 1"
And Integration Process is only use to call "Local Integration Process 1" in loop.
In OData adapter, just tick "Process in Pages" and enter the total number or records should be called in a
single call in "Page size".
Every time when API call fetches the data, a property "$
{property.<RECEIVER>.<Channel>.hasMoreRecords} " is set with true if there are still records to fetch
from S/4 Hana or set with false if no records are left to fetch
This property we need to set in Lopping Process Call to give the condition for stopping loop
Receiver: S4Hana
Channel:ODataS4
Handling large data sets efficiently is a common challenge in integration scenarios. In SAP Cloud
Integration (SAP CI), pagination plays a vital role in retrieving data in manageable chunks, ensuring
seamless performance and data consistency. In this blog series, I will explore how to implement
pagination across four widely-used adapters: Salesforce, Microsoft Dynamics CRM, Workday, and
Coupa.
Check out the Second blog in this series on Microsoft Dynamics CRM Adapter, check it out
here Seamless Integration with SAP CI: Mastering Pagination for Microsoft Dynamics CRM Adapter
We’ll begin with Salesforce, diving into practical examples and best practices that streamline data
synchronization. Whether you're dealing with customer records, transactional data, or employee
information, mastering pagination will enhance your integration flows. Stay tuned as we navigate
through each adapter, tackling real-world challenges and solutions.
To optimize performance and ensure seamless data retrieval, a custom pagination strategy is the
recommended approach. By fetching records page by page in batch mode, we can prevent system
overload, improve response times, and maintain control over data flow. In this blog, we will explore how
to implement custom pagination in SAP CI for the Salesforce adapter, ensuring scalability and efficiency
in your integrations.
Design Approach:
The scenario involves sending Account entity data from Salesforce to a target system that has a batch
limit of 2,000 records per request. To accommodate this constraint, we implement a custom
pagination approach using two Salesforce receiver adapter calls.
The first call fetches the initial page of records, while the second call operates within a looping
process to retrieve subsequent pages. Each fetched page is passed through message mapping, ensuring
seamless transformation before being delivered to the target system. This cycle continues—fetching a
page, processing it, and sending it to the target—until all records are transferred, ensuring an efficient
full end to end Batching processing and controlled data flow from Salesforce to the target system.
Design components elaboration:
Timer : This is a timer based flow which be a scheduled as per the user's input
Content Modifier : "set properties" - This step is used to accept inputs such as EntityName, fieldnames,
StartDate from the user, all these fields are externalized to make it dynamic.
Process Call : "Initiate Salesforce Call" - Once all the values are set, this process call is used to initiate
the salesforce call
Request Reply : "Salesforce Receiver Adapter" - This step initiates the first call to salesforce to fetch the
first page using SOQL query, By default, the SOQL query returns up to 2000 records on a single response
page.
We are not digging into the connection tab in this blog, this blog focuses only on pagination concept.
Output:
In the response, we retrieve 2,000 records from the initial Salesforce call. The total record count
is 24,446, and the response contains a <done>false</done> flag, indicating that more pages are
available. Additionally, a <nextRecordsUrl> tag is provided, which serves as a reference for fetching the
next batch of records.
A router is used to determine whether the fetched page contains any records. If records are present, the
integration flow proceeds to process to next steps. However, if the page is empty, the flow terminates
immediately, preventing unnecessary processing.
This check ensures optimized execution by stopping the pagination loop when there are no more
records to fetch, improving efficiency and reducing unnecessary API calls.
Content Modifier : "Read nextURL" - This is a crucial step where are we reading two key fields—
<done> and <nextRecordsUrl> using xpaths.
Process Call : "Message Mapping" - After this step, the page goes to message mapping and it is
processed to the target, this step is target specific and this blog does not cover this part.
Once the first batch is successfully processed and sent to the target system, the design ensures that the
integration flow moves to the next page. To achieve this, a router is used to check whether another
page is available.
This check is performed using the "LoadDone" property, which was set in the previous content
modifier. If more pages exist, the flow loops to fetch the next batch using the <nextRecordsUrl>. If no
further pages are available, the process terminates, ensuring efficient data retrieval without
unnecessary API calls.
Properties generated at the runtime for this run:
Once it is confirmed that more pages are available, the flow enters a looping process call to retrieve
subsequent pages. This looping mechanism operates based on the condition:
${property.LoadDone} = "false"
As long as LoadDone remains "false", the process continues fetching the next batch of records using
the <nextRecordsUrl>. This ensures that each page is processed and sent to the target system
sequentially.
Once all pages have been retrieved and processed, LoadDone is set to "true"**, signaling that no further
pages exist. At this point, the loop terminates, marking the successful completion of data transfer.
In this looping process call, there is the second request-reply step with Salesforce Adapter which will
help us to fetch the next pages.
For this call, we will make use of Operation SOQL - Execute Query for More Results and
nextRecoredsURL which was set in the previous content modifier will be passed dynamically to the next
"Next Records URL"
Output :
Page 2: This page has 2000 records as well, and done = false which means there are more pages.
The same Content Modifier : "Read nextURL" is used after this step to save the key
properties LoadDone and nextRecordsURL.
The same Process Call : "Message Mapping" is used here to send this page to the Message Mapping and
Send to Target LIP.
At this point, the looping process terminates, ensuring that all records have been successfully retrieved
and processed.
Page 13 Output:
Full Run in TRACE:
Conclusion
Implementing custom pagination in SAP CI for the Salesforce adapter ensures efficient data retrieval
while preventing performance bottlenecks. By leveraging looping mechanisms, routers, and conditional
checks, we can seamlessly fetch and process large datasets in manageable batches. This approach not
only optimizes API calls but also ensures smooth end-to-end data transfer. Stay tuned for the next part
of this series, where we explore pagination for next adapters.
Seamless Integration with SAP CI: Mastering Pagination for Microsoft Dynamics CRM Adapter
When integrating with Microsoft Dynamics CRM, handling large datasets efficiently is key to ensuring
smooth data synchronization. Unlike Salesforce, which relies on SOQL and nextRecordsUrl, Dynamics
CRM follows an OData-based pagination approach using <odata.nextLink> to fetch records in
manageable batches.
If you missed the first part on Salesforce Adapter, check it out here Seamless Integration with SAP CI:
Mastering Pagination for Salesforce Adapter
Microsoft Dynamics CRM Adapter - Handling Large Data Sets with Pagination
In this use case, we focus on retrieving records from the accounts object in Microsoft Dynamics CRM,
which contains 10K+ records. To ensure efficient data retrieval, we utilize Advanced OData
Queries along with the pagination feature in the receiver adapter.
Fetching data page by page, processing each batch, and then moving to the next page is the best
approach for handling large datasets. In this blog, I will explain the end-to-end batching process,
ensuring seamless data synchronization while optimizing API performance.
Design Approach:
The scenario involves sending accounts object data from Microsoft Dynamics CRM to a target system
that has a batch limit of 500 records per request. To accommodate this constraint, we implement
a pagination approach using two Microsoft Dynamics CRM receiver adapter call.
The call fetches the page of records within a looping process to retrieve the pages one after the other.
Each fetched page is passed through message mapping, ensuring seamless transformation before being
delivered to the target system. This cycle continues—fetching a page, processing it, and sending it to the
target—until all records are transferred, ensuring an efficient full end to end Batching processing and
controlled data flow from Microsoft Dynamics CRM to the target system.
Design components elaboration:
Timer : This is a timer based flow which be a scheduled as per the user's input
Content Modifier : "set properties" - This step is used to accept inputs such as ObjectName, fieldnames,
StartDate from the user, all these fields are externalized to make it dynamic.
Looping Process Call : "Initiate MS Dynamics Call" - This Looping Process Call continuously runs until
the <odata.nextLink> field is absent from the response payload. The condition for looping is:
${property.NextPageLink} != ''
This ensures efficient batch processing, fetching all records page by page while maintaining seamless
data flow.
The Microsoft Dynamics CRM receiver adapter is placed inside the Looping Process Call to fetch
records page by page.
Processing Tab
We are not digging into the connection tab in this blog, this blog focuses only on pagination concept.
Output:
Page 1 : Response from the Microsoft Dynamics CRM system contains 500 records and "odata.nextLink"
which has the URL to the next page.
Page 2: Response from the Microsoft Dynamics CRM system contains 500 records and "odata.nextLink"
which has the URL to the next page.
Content Modifier : "Read nextURL" - This is a crucial step where are we reading key field —
<odata.nextLink> using xpath.
After fetching each page, the data is passed to Message Mapping, where it is processed before being
sent to the target system. This step is target-specific and is not covered in this blog.
For demonstration purposes, I have included this step to mimic the "send to target" operation after
message mapping. The key focus of this blog remains on pagination handling in the Microsoft Dynamics
CRM adapter and ensuring seamless data retrieval.
Final Page Handling: Completing the Pagination Process
At this point, the looping process terminates, ensuring that all records have been successfully retrieved
and processed.
Page 22
From the TRACE logs below, its evident that looping process call "Initiate MS Dynamics Call" executed 23
times indicating 23 pages.
Full Run in TRACE
Conclusion
Implementing pagination in SAP CI for the Microsoft Dynamics CRM Adapter ensures efficient data
retrieval while preventing performance bottlenecks. By leveraging looping mechanisms, we can
seamlessly fetch and process large datasets in manageable batches. This approach not only optimizes
API calls but also ensures smooth end-to-end data transfer. Stay tuned for the next part of this series,
where we explore pagination for next adapters.
Looping Process Call and Poll Enrich to read multiple files from the SFTP Server.
Hi, I am using Looping Process Call and Poll Enrich to read multiple files from the SFTP Server.
In the SFTP adapter, in Post-Processing , i have selected Keep File and Mark as Processed in Idempotent
Repository because if i choose Keep file and Process Again then it is reading only one file and the loop is
running infinitely. Also, I can't choose Move file and Delete file because of restrictions.
My question is if the file is read and at some point in my iFlow if i get any error then i have to reprocess
that file. But due to using "Keep File and Mark as Processed in Idempotent Repository", I am not able to
read the file again.
Below document is based on delta sync configuration in SAP CPI system. Many times we need to fetch
data at regular intervals from the source system. In such cases we don't want to retrieve all records
during every run. Whenever there is new record or data change only that data needs to be fetched from
the source system. Using timestamp we can design query of delta data retrieval.
The trigger of Iflow is timer based. I have used CPI Odata (MessageProcessingLogs) entity to retrieve
failed message details and send this as a daily report via mail.
Now first run will be based on the default value provided while deploying Iflow. However, for next
subsequent runs I should be able to get only new failed records report on my mail. For this purpose we
need to perform delta sync.
Later this will fetch value from the write variable which is timestamp_DS_DateNow.
Below variable values are coming from the content modifier (getLastTimeStamp)
datetime'${property.lastSync}'
'${property.dateNow}'
The response from the OData is moving into XSLT mapping to generate XML to Table format and
followed by Write Variable where we need to store the last execution time which will be used in
subsequent calls.
Write Variable :-
Use exact same name as what you want to pass to the property variable in content modifier. This value
will be used in OData query.
Once the interface is triggered, this will send message failure report to configured mail ID -
[SAP CPI] – OPERATION BATCH IN ODATA V2 RECEIVER ADAPTER
WITH CLOUD PLATFORM INTEGRATION
Scenario 01 – Get data
Payload request
<batchParts>
<batchQueryPart>
<method>GET</method>
<uri>A_EnterpriseProject</uri>
</batchQueryPart>
<batchQueryPart>
<method>GET</method>
<uri>A_EnterpriseProjectElement</uri>
</batchQueryPart>
</batchParts>
Scenario 02 – POST DATA
We can configure method POST with ODATA Receiver Adapter. This is simple so I do not take demo.
We can create and update data in one call to backend with BATCH
Structure below illustrate for this one
With this structure, we can create 2 entities for Projects and update 2 entities Project Element.
SUMMARY
In this article I shared way to send request ODATA to backend with BATCH operation. Instead call one by one to backend,
we can use this way to improve performance.