0% found this document useful (0 votes)
14 views2 pages

Presentation Codes

This document contains examples of using closures, accumulators, and broadcast variables in Spark. It shows how closures allow sharing state across distributed functions, accumulators can be used to aggregate values across partitions, and broadcast variables efficiently distribute large read-only datasets to worker nodes.

Uploaded by

Balu Bonam
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
14 views2 pages

Presentation Codes

This document contains examples of using closures, accumulators, and broadcast variables in Spark. It shows how closures allow sharing state across distributed functions, accumulators can be used to aggregate values across partitions, and broadcast variables efficiently distribute large read-only datasets to worker nodes.

Uploaded by

Balu Bonam
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
You are on page 1/ 2

#closure

c=0
def f1(n):
global c
c = c +1

rdd = sc.parallelize(range(1,41),4).foreach(f1)
print(c)
------------------------------------
#savefile
c=0
def f1(n):
global c
c = c +1
rdd = sc.parallelize(range(1,41),4).map(f1)
rdd.saveAsTextFile('save2')
print(c)
------------------------------------------------

#Accumulator
c=sc.accumulator(0)
def f1(n):
global c
c.add(1)

sc.parallelize(range(1,41),4).foreach(f1)
print(c)

------------------------------------------------

#python
c=0
def f1():
global c
c = c + 1
return c

res = f1()
print(res)
--------------------
l= sc.broadcast(['Afghanistan', 'Aland Islands', 'Albania', 'Algeria', 'American
Samoa', 'Andorra', 'Angola', 'Anguilla', 'Antarctica', 'Antigua and Barbuda',
'Argentina', 'Armenia', 'Aruba', 'Australia', 'Austria', 'Azerbaijan', 'Bahamas',
'Bahrain', 'Bangladesh', 'Barbados', 'Belarus', 'Belgium', 'Belize', 'Benin',
'Bermuda', 'Bhutan', 'Bolivia, Plurinational State of', 'Bonaire, Sint Eustatius
and Saba', 'Bosnia and Herzegovina', 'Botswana', 'Bouvet Island', 'Brazil',
'British Indian Ocean Territory', 'Brunei Darussalam', 'Bulgaria', 'Burkina Faso',
'Burundi', 'Cambodia', 'Cameroon', 'Canada', 'Cape Verde', 'Cayman Islands',
'Central African Republic', 'Chad', 'Chile', 'China', 'Christmas Island', 'Cocos
(Keeling) Islands', 'Colombia', 'Comoros', 'Congo', 'Congo, The Democratic Republic
of the', 'Cook Islands', 'Costa Rica', "Côte d'Ivoire", 'Croatia', 'Cuba',
'Curaçao', 'Cyprus', 'Czech Republic', 'Denmark', 'Djibouti', 'Dominica',
'Dominican Republic', 'Ecuador', 'Egypt', 'El Salvador', 'Equatorial Guinea',
'Eritrea', 'Estonia', 'Ethiopia', 'Falkland Islands (Malvinas)', 'Faroe Islands',
'Fiji', 'Finland', 'France', 'French Guiana', 'French Polynesia', 'French Southern
Territories', 'Gabon', 'Gambia', 'Georgia', 'Germany', 'Ghana', 'Gibraltar',
'Greece', 'Greenland', 'Grenada', 'Guadeloupe', 'Guam', 'Guatemala', 'Guernsey',
'Guinea', 'Guinea-Bissau', 'Guyana', 'Haiti', 'Heard Island and McDonald Islands',
'Holy See (Vatican City State)', 'Honduras', 'Hong Kong', 'Hungary', 'Iceland',
'India', 'Indonesia', 'Iran, Islamic Republic of', 'Iraq', 'Ireland', 'Isle of
Man', 'Israel', 'Italy', 'Jamaica', 'Japan', 'Jersey', 'Jordan', 'Kazakhstan',
'Kenya', 'Kiribati', "Korea, Democratic People's Republic of", 'Korea, Republic
of', 'Kuwait', 'Kyrgyzstan', "Lao People's Democratic Republic", 'Latvia',
'Lebanon', 'Lesotho', 'Liberia', 'Libya', 'Liechtenstein', 'Lithuania',
'Luxembourg', 'Macao', 'Macedonia, Republic of', 'Madagascar', 'Malawi',
'Malaysia', 'Maldives', 'Mali', 'Malta', 'Marshall Islands', 'Martinique',
'Mauritania', 'Mauritius', 'Mayotte', 'Mexico', 'Micronesia, Federated States of',
'Moldova, Republic of', 'Monaco', 'Mongolia', 'Montenegro', 'Montserrat',
'Morocco', 'Mozambique', 'Myanmar', 'Namibia', 'Nauru', 'Nepal', 'Netherlands',
'New Caledonia', 'New Zealand', 'Nicaragua', 'Niger', 'Nigeria', 'Niue', 'Norfolk
Island', 'Northern Mariana Islands', 'Norway', 'Oman', 'Pakistan', 'Palau',
'Palestinian Territory, Occupied', 'Panama', 'Papua New Guinea', 'Paraguay',
'Peru', 'Philippines', 'Pitcairn', 'Poland', 'Portugal', 'Puerto Rico', 'Qatar',
'Réunion', 'Romania', 'Russian Federation', 'Rwanda', 'Saint Barthélemy', 'Saint
Helena, Ascension and Tristan da Cunha', 'Saint Kitts and Nevis', 'Saint Lucia',
'Saint Martin (French part)', 'Saint Pierre and Miquelon', 'Saint Vincent and the
Grenadines', 'Samoa', 'San Marino', 'Sao Tome and Principe', 'Saudi Arabia',
'Senegal', 'Serbia', 'Seychelles', 'Sierra Leone', 'Singapore', 'Sint Maarten
(Dutch part)', 'Slovakia', 'Slovenia', 'Solomon Islands', 'Somalia', 'South
Africa', 'South Georgia and the South Sandwich Islands', 'Spain', 'Sri Lanka',
'Sudan', 'Suriname', 'South Sudan', 'Svalbard and Jan Mayen', 'Swaziland',
'Sweden', 'Switzerland', 'Syrian Arab Republic', 'Taiwan, Province of China',
'Tajikistan', 'Tanzania, United Republic of', 'Thailand', 'Timor-Leste', 'Togo',
'Tokelau', 'Tonga', 'Trinidad and Tobago', 'Tunisia', 'Turkey', 'Turkmenistan',
'Turks and Caicos Islands', 'Tuvalu', 'Uganda', 'Ukraine', 'United Arab Emirates',
'United Kingdom', 'United States', 'United States Minor Outlying Islands',
'Uruguay', 'Uzbekistan', 'Vanuatu', 'Venezuela, Bolivarian Republic of', 'Viet
Nam', 'Virgin Islands, British', 'Virgin Islands, U.S.', 'Wallis and Futuna',
'Yemen', 'Zambia', 'Zimbabwe'])
l.value

You might also like