Scribd
Upload
52 views5 pages

PEN Efine Utorial: Module 1.2 Video - Instructions

This document provides instructions for using Open Refine to clean up and prepare customer, stock, and other types of data. It demonstrates how to perform basic cleanup tasks like trimming whitespace, combining columns, removing brackets from text. More advanced techniques shown include faceting to merge duplicate values, numeric faceting, filtering rows, removing NA values, splitting columns, making web service calls, and performing multi-column operations. Useful links are also provided for GREL string functions, clustering, and other Open Refine recipes.

Uploaded by

sivalakshmi mavilla
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
52 views5 pages

PEN Efine Utorial: Module 1.2 Video - Instructions

This document provides instructions for using Open Refine to clean up and prepare customer, stock, and other types of data. It demonstrates how to perform basic cleanup tasks like trimming whitespace, combining columns, removing brackets from text. More advanced techniques shown include faceting to merge duplicate values, numeric faceting, filtering rows, removing NA values, splitting columns, making web service calls, and performing multi-column operations. Useful links are also provided for GREL string functions, clustering, and other Open Refine recipes.

Uploaded by

sivalakshmi mavilla
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 5

Data Understanding and Preparation

OPEN REFINE TUTORIAL

Module 1.2 Video – Instructions

1. Basic customer data clean up


a. Download and unzip Customers-messy.zip and get familiar with dataset
b. Create Project > Browse > Customers-messy.zip
c. Character encoding ( browse through the many options available) > select UTF-8
d. Save Project Name > create Project
e. Combine contact firstName and contact lastName to contact fullName
f. Contact firstName > Edit Cells > Common transforms > trim Leading and trailing whitespace
g. Contact lastName > Edit Cells > Common transforms > trim Leading and trailing whitespace
h. Contact firstName > Edit Column > Add Column based on this column
i. Value + " " + cells["contactLastName"].value
j. Enter Column name to contactFullName
k. Undo / Redo to go back and forth
l. Contact FullName > Edit Columns > Remove this column
2. Faceting ( text)
a. City > Edit Cells > Common transforms > To titlecase
b. Country > Facet > Text Facet
c. On left panel scroll down to see multiple vaues : “United states”, “US”, etc..
d. Click on cluster to get the overall view
e. Select multiple method options such as Key Collision and Keying functions such as fingerprint
f. Select merge function between clusters and select merge Selected and Recluster
g. Method > nearest Neighbour & Keying function > PPM select merge and click on merge Selected
and Recluster
3. Faceting (number)
a. CreditLimit > Edit Cells > common transforms > To number
b. CreditLimit > facet > Numeric Facet
c. Uncheck the numeric box on the left facet window and click on change
d. toNumber(value.replace("USD","").replace("$","").replace(",",""))
e. goto creditLimit > Edit Cells > Transform
f. toNumber(value.replace("USD","").replace("$","").replace(",",""))
g. Export > comma seperated value

4. Stocks data cleanup


a. Data (Stocks.Zip JSON)
b. Language Setting to English
c. Create Project > Browse > Stocks.zip
d. Click on the top level JSON enclosures to get to tabular format
Open Refine Tutorial

e. Once preview is available, enable the Trim leading and trailing whitespace from strings ( options
below the table)
f. Enable the Parse cell text into numbers/dates
g. Save Project Name > create Project
h. ALL ( left corner) > Edit Columns > reorder / remove columns ( enables to discard irrelevant
columns )
i. Click on any cell and edit the cell with any information
j. Click on dropdowns for any column ( say Volume ) > sort ( choose options as applicable )

5. Stocks data augmentation


a. Data ( stocks_demo.csv Webservice calls [subset of the original zip file])
b. Create Project > Browse > stocks_demo.csv
c. Save Project Name > create Project
d. Ticker > Edit Column > Add Columns by fetching URLs
e. Test the below URLs on a browser to make sure it works

http://dev.markitondemand.com/MODApis/

http://dev.markitondemand.com/MODApis/Api/v2/Quote?symbol=AAPL

f. Invoke Web API from Open Refine. Use following expression:


'http://dev.markitondemand.com/MODApis/Api/v2/Quote?symbol=' + escape(value,'url')
value.partition("<LastPrice>")[2].partition("</LastPrice>")[0]

Other Examples

6. Remove rows with NA


a. Facet > Text Facet > Include/Exclude the value in the Facet/Filter
b. All > Edit Rows > Remove all Matching Rows

7. Filter rows with NA


a. Column > Text Filter > Include/Exclude the value in the Facet/Filter>
b. All> Edit Rows > Remove all Matching Rows
Open Refine Tutorial

8. Remove brackets around text


Edit Cells > Column Transform > Expression box :
value.split("(")[0].split(")")[0]
Open Refine Tutorial

9. Split a column into multiple columns


Edit Column > Split into Several Columns

10. Multi-column operations


All columns in the project
forEach(row.columnNames,cn,cells[cn].value)

Merge all columns in the project into single column


forEach(row.columnNames,cn,cells[cn].value).join("|")

Count rows with some blank cells


filter(row.columnNames,cn,isBlank(cells[cn].value)).length()
Open Refine Tutorial

Useful Links

https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions

https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-Depth

http://schoolofdata.org/handbook/recipes/cleaning-spending-data-open-refine/

http://blogs.worldbank.org/opendata/unpivoting-data-excel-open-refine-and-python

You might also like