6/7/22, 1:17 AM How to use a Base_UserDefined Transform in Data Services 4.

Data Services 4.x to return JSON Web Service data | SAP Blogs

Community Topics Groups Answers Blogs Events Programs

Ask a Question Write a Blog Post Login

Tom Fallwell
July 28, 2015 | 4 minute read

How to use a Base_UserDefined

Transform in Data Services 4.x to
return JSON Web Service data
Follow
 14  6  13,346

 Like

Overview
 RSS Feed
The proliferation of JSON data available from web service based sources is
enormous these days. So I wanted to capture some JSON web service data for a
proof of concept via SAP Data Services, but was discouraged to find there was no
simple way to do this.

Searching for a solution, there were a couple of posts that vaguely refer to using
the Data Service Adapter SDK to build a special adapter. But this seemed like a
complicated route for something so simple as JSON. However then I found out
about Base_UserDefined transform.

In this article, I will use Base_UserDefined transform to return web service data
using the “Per Collection” option. I initially learned about Base_UserDefined
from Kamal’s excellent article, http://scn.sap.com/community/data-
services/blog/2013/01/31/how-to-use-userdefined-transform-in-sap-
bodsbusiness-object-data-services. But in that article it uses Base_UserDefined
in a “Per record” mode. My json request will return many records as a result so I
need to use the “Per Collection” mode and it requires a different way to work with

https://blogs.sap.com/2015/07/28/how-to-use-a-baseuserdefined-transform-in-data-services-4x-to-return-json-web-service-data/ 1/11
6/7/22, 1:17 AM How to use a Base_UserDefined Transform in Data Services 4.x to return JSON Web Service data | SAP Blogs

the Collection object of Base_UserDefined. I could not find any articles or

documentation on the “Per Collection” mode so I will help fill that gap with this
article.

The Base_UserDefined transform simply provides an interface to the Python

engine and language. There is also a strong community of Python code one can
get assistance with.

By the way, Python is an object-oriented programing language that is interpreted

making it a very quick development tool. But what is most helpful are the module
and packages that are supported by an open source community. It is rare that I
do not find a module already out there already is written for the job I need to do.

How it’s done

In this example, I want to return data from a json data source maintained by the
Consumer Financial Protection Bureau (my daughter is just starting college and
the number of credit card offers she is already receiving has caused me to think
“financial protection”). Hense the choice of web service.

The url I will use is this: http://data.consumerfinance.gov/api/views.json

1. Prerequisite – First you will need to download and copy the urllib2.py
module to your \Program Files (x86)\SAP BusinessObjects\Data
Services\DataQuality\python\lib\ folder. This should be on the server the
job server is running on.
urllib2.py is a module that handles web service requests. You can find it here
and then save it as a file called urllib2.py.
http://svn.python.org/projects/python/branches/release22-

https://blogs.sap.com/2015/07/28/how-to-use-a-baseuserdefined-transform-in-data-services-4x-to-return-json-web-service-data/ 2/11
6/7/22, 1:17 AM How to use a Base_UserDefined Transform in Data Services 4.x to return JSON Web Service data | SAP Blogs

maint/Lib/urllib2.py

2. Now to the design work. We need to create a source that provides the URL
for the web service. In my example I simply added a SQL Transform and
wrote a SQL select statement with hardcoded url.
Select statement I used is
select ‘http://data.consumerfinance.gov/api/views.json‘ as JSONuri
3. Next drag in to the Data Flow a Base_UserDefined transform from the Data
Quality\User Defined folder in Transforms, add a Temp Table, and connect
all three.

4. Open the Base_UserDefined transform and in the Input tab of the transform,
drag and drop Input column of the input schema to the Input Schema
Column Name so as to map with the Transform Input Field Name.

5. Now click on the Options tab and then click on Edit Options button.

https://blogs.sap.com/2015/07/28/how-to-use-a-baseuserdefined-transform-in-data-services-4x-to-return-json-web-service-data/ 3/11
6/7/22, 1:17 AM How to use a Base_UserDefined Transform in Data Services 4.x to return JSON Web Service data | SAP Blogs

6. Select the Mode as Per Collection (this is key, one input record will create
multiple output records) and then click on Python Expression Editor to
display editor on the right. You can paste the code directly in this window
for now but we need to setup output fields via the “Launch Python Editor”
button on the right.

See get_json.py.zip attachment for the python code.

7. Notice there are thee tabs on left panel. Click on the I/O Fields and then
right click on Output Fields to select Insert. Do this to create two output
fields called complaint_id and complaint_name both as varchar 50.

8. Click Ok twice to close the editor and then click on the Output tab. From
here you simply click on the check box next to each output field which will
result in their mapping on the right panel. (I set complaint_id as a primary
field but that is not required. Just me being picky.)

https://blogs.sap.com/2015/07/28/how-to-use-a-baseuserdefined-transform-in-data-services-4x-to-return-json-web-service-data/ 4/11
6/7/22, 1:17 AM How to use a Base_UserDefined Transform in Data Services 4.x to return JSON Web Service data | SAP Blogs

9. Now launch the job and you should see a result in template table like this:

Conclusion
Alert Moderator
Using Base_UserDefined transform opens up a whole new world of flexibility and
productivity but leveraging Python. In this case I did not need to write any code
to handle the web service request or to parse json and now I can load a database
Assigned Tags of all US Consumer Complaint departments.

SAP Data Services

SAP Data Integrator

Tip: It is tricky to debug a Python script within a Base_UserDefined transform, so
data integrator I will often create a plain version of the script and run it from my own Python
engine and then convert it to one that will work in Data Services. I will often use
data services
a Python Print statement to display variables values which you will be able to see
data services 4.1 in the DS job log.

json

python

View more... 

Similar Blog Posts 

https://blogs.sap.com/2015/07/28/how-to-use-a-baseuserdefined-transform-in-data-services-4x-to-return-json-web-service-data/ 5/11
6/7/22, 1:17 AM How to use a Base_UserDefined Transform in Data Services 4.x to return JSON Web Service data | SAP Blogs

Consuming ODATA service in BODS

By Vinaya HG Jul 21, 2018

How to use User_Defined Transform in Data Services 4.x using Python Part-1
By Kamal Kumar Jan 31, 2013

Use of User-Defined Transform in Data Service 4.X for Accessing Web Service Part 1
By Titto Antony May 10, 2013

Related Questions 
How to load JSON data in SAP BO Data Services
By Ronite Bachu Feb 19, 2018

Help, How to develop JSON Adapter?

By Former Member Mar 03, 2017

How to create a data store for JSON files In SAP BODS?

By Murali Krishna Mar 06, 2017

Join the Conversation 

SAP TechEd
Tune in for tech talk. Stay for inspiration. Upskill your future.

SAP BTP Learning Group

SAP Business Technology Platform Learning Journeys.

Coffee Corner
Join the new Coffee Corner Discussion Group.

14 Comments

You must be Logged on to comment or reply to a post.

https://blogs.sap.com/2015/07/28/how-to-use-a-baseuserdefined-transform-in-data-services-4x-to-return-json-web-service-data/ 6/11
6/7/22, 1:17 AM How to use a Base_UserDefined Transform in Data Services 4.x to return JSON Web Service data | SAP Blogs

Arun Sasi
July 28, 2015 at 7:13 am

Hi Tom,

This is simply brilliant. Appreciate your efforts to discover such a useful feature of base_userdefined
transform.

I read in one of the SCN posts about the usage of JSON web service to read Twitter tweet data. I am
working on a POC where I need to access user's tweet data in to staging tables through JSON. For this I
have created an application in Twitter and got the security tokens for securely accessing twitter data but I
dont know how to use these tokens in the python code such that the JSON URL reads the data in BODS.

Can you help me

Regards

Arun Sasi

Like 0 | Share

Tom Fallwell | Blog Post Author

July 28, 2015 at 1:27 pm

Thanks for the comment Arun.

That is what I like about Python. It islikely someone has already written a module to access Twitter
securely. Googling the words twitter and python returns a perfect hit for this. This module
appears to do exactly what you want..

bear/python-twitter · GitHub

Like 0 | Share

Arun Sasi
July 29, 2015 at 7:44 am

Thanks Tom!!

It gets little bit tricky when you write the code inside the Base_UserDefined transform to
access Twitter data.

Regards
Arun Sasi

Like 0 | Share

Former Member
https://blogs.sap.com/2015/07/28/how-to-use-a-baseuserdefined-transform-in-data-services-4x-to-return-json-web-service-data/ 7/11
6/7/22, 1:17 AM How to use a Base_UserDefined Transform in Data Services 4.x to return JSON Web Service data | SAP Blogs

September 14, 2015 at 2:47 pm

Hi,

I am working on building a user defined transform that lits files of a certain type in a directory specified by
the user and the results to be put in a temporary table. I am using the per collection option in the python
editor, however I am having issues getting the input values. Have done this before when I was doing per
record by using record.GetField, however this does not seem to work when using per collection. Would you
be able to help me with this?

Like 0 | Share

Former Member
August 30, 2016 at 3:36 pm

Hello Tom-

Must thank you for a very elegant solution you presented. I've been able to tweak it somehow to consume a
GET service which returns a JSON file as response. However, my main issue is to get POST service
working. Can you guide me somehow to extend the solution you presented to work for a POST operation (
to send relational data as a JSON input to a API)?

Thanks in advance,

Abhishek

Like 0 | Share

Joshua Blythe
August 31, 2016 at 1:19 pm

The Python requests module makes short work of making rest api calls.

import requests,json

payload = {'logonkey' : auth.logonkey,

'tokenkey' : auth.tokenkey,

'fileid' : fileId,

'newstatus' : newStatus}

response = requests.post('https://epfws.usps.gov/ws/resources/download/status',

data={'obj' : json.dumps(payload)},

verify=False)

https://blogs.sap.com/2015/07/28/how-to-use-a-baseuserdefined-transform-in-data-services-4x-to-return-json-web-service-data/ 8/11
6/7/22, 1:17 AM How to use a Base_UserDefined Transform in Data Services 4.x to return JSON Web Service data | SAP Blogs

This endpoint updates the usps file server after downloading postal directories used by the
address cleanse. (I wish SAP would provide a rest api to download their directories) 🙁

Like 0 | Share

Former Member
September 8, 2016 at 9:24 am

Thanks Joshua. I tried using the requests module but it seems the DS instance of
Python(which is pre-installed) doesn't support requests module. Do you know how we can
install this module in DS to make it importable in python scripts(within BODS)?

Like 0 | Share

Joshua Blythe
September 8, 2016 at 1:11 pm

Do a google search and follow the installation instructions.

Also the latest DS patch we installed looks to have finally updated the Python
version to 2.7, so you may need to take that into consideration.

Like 0 | Share

Former Member
August 31, 2016 at 1:57 pm

Outstanding work! Significant Contribution...

Like 0 | Share

Former Member
November 3, 2017 at 2:18 pm

Could you please share get_json.py.zip file again? It is not available at the link. Thank you.

Like 2 | Share

Steve Yochum
July 19, 2018 at 7:09 pm

Following these instructions I'm capturing only the last record in the URL. Is there something missing to
capture all of the records in the URL?

https://blogs.sap.com/2015/07/28/how-to-use-a-baseuserdefined-transform-in-data-services-4x-to-return-json-web-service-data/ 9/11
6/7/22, 1:17 AM How to use a Base_UserDefined Transform in Data Services 4.x to return JSON Web Service data | SAP Blogs

Like 0 | Share

Pedro Soares
August 3, 2018 at 6:19 pm

How can I add basic authentication (user and password) to the request?

Like 0 | Share

Dhanraj Andhale
November 8, 2019 at 5:59 am

Ouststanding..!! Really the above blog is very helpful to consuming JSON API but for large API structure it
will taking time to develop the same.

Thank you very much Tom..!!

Like 0 | Share

Karel Turek
February 26, 2020 at 7:01 am

I had no success with using the OData adapter. This article helped me to solve the problem in real-time.
Thanks a lot for sharing, Tom!

Like 0 | Share

Find us on

Privacy Terms of Use

Legal Disclosure Copyright

Trademark Cookie Preferences

Newsletter Support
https://blogs.sap.com/2015/07/28/how-to-use-a-baseuserdefined-transform-in-data-services-4x-to-return-json-web-service-data/ 10/11
6/7/22, 1:17 AM How to use a Base_UserDefined Transform in Data Services 4.x to return JSON Web Service data | SAP Blogs

https://blogs.sap.com/2015/07/28/how-to-use-a-baseuserdefined-transform-in-data-services-4x-to-return-json-web-service-data/ 11/11