Scribd
Upload
37 views

Parcs Python

This document describes how to set up a distributed computing platform using PARCS-python. It discusses node types including a master node that dispatches work and worker nodes that perform work. It provides instructions for SDK setup in Google Cloud including creating a project, enabling APIs, and configuring zones and regions. It also provides instructions for creating firewall rules and instances including creating a master instance and multiple worker instances. Finally, it provides an example problem that maps work across workers and reduces the results.

Uploaded by

Max The human
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
37 views

Parcs Python

This document describes how to set up a distributed computing platform using PARCS-python. It discusses node types including a master node that dispatches work and worker nodes that perform work. It provides instructions for SDK setup in Google Cloud including creating a project, enabling APIs, and configuring zones and regions. It also provides instructions for creating firewall rules and instances including creating a master instance and multiple worker instances. Finally, it provides an example problem that maps work across workers and reduces the results.

Uploaded by

Max The human
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 10

PARCS-python

Distributed computations platform


Node types

• Master - dispatches work

• Worker - performs work


SDK setup

• Install gcloud sdk, login with gcloud auth login

• gcloud projects create parcs-python


• gcloud config set project parcs-python
• # enable billing and compute APIs for project
• gcloud config set compute/zone europe-north1-a
• gcloud config set compute/region europe-north1
Firewall setup

• gcloud compute

firewall-rules create allow-all

--direction=INGRESS --priority=1000

--network=default --action=ALLOW

--rules=all --source-ranges=0.0.0.0/0
• 🎉🎉🎉
Instances

• gcloud compute instances



create-with-container master

--container-image=registry.hub.docker.com/
hummer12007/parcs-node

--container-env PARCS_ARGS="master"
• Record master’s IP, e.g. 10.166.0.2
• gcloud compute instances

create-with-container worker1 worker2 worker3…

--container-image=registry.hub.docker.com/
hummer12007/parcs-node

--container-env PARCS_ARGS="worker 10.166.0.2"
🎉🎉🎉🎉🎉🎉🎉🎉🎉
• Control panel accessible at http://$MASTER_IP:8080
Example problem
https://git.sr.ht/~hummer12007/parcs-python/tree/master/examples/scripts

from Pyro4 import expose


@staticmethod
class Solver:
@expose
def __init__(self, workers=None, input_file_name=None, output_file_name=None):
def mymap(a, b):
self.input_file_name = input_file_name
print (a, b)
self.output_file_name = output_file_name
res = 0
self.workers = workers
for i in xrange(a, b):
print("Inited")
res += i
return res
def solve(self):
print("Job Started")
@staticmethod
print("Workers %d" % len(self.workers))
@expose
n = self.read_input()
def myreduce(mapped):
step = n / len(self.workers)
output = 0
for x in mapped:
# map
output += x.value
mapped = []
return output
for i in xrange(0, len(self.workers)):
mapped.append(self.workers[i].mymap(i * step, i * step + step))
def read_input(self):
f = open(self.input_file_name, 'r')
print('Map finished: ', mapped)
line = f.readline()
f.close()
# reduce
return int(line)
reduced = self.myreduce(mapped)
print("Reduce finished: " + str(reduced))
def write_output(self, output):
f = open(self.output_file_name, 'w')
# output
f.write(str(output))
self.write_output(reduced)
f.write('\n')
f.close()
print("Job Finished")
Example problem
https://git.sr.ht/~hummer12007/parcs-python/tree/master/examples/scripts
mapped = []
for i in xrange(0, len(self.workers)):
mapped.append(

self.workers[i].mymap(i * step, i * step + step))
print('Map finished: ', mapped)
# reduce
reduced = self.myreduce(mapped)
print("Reduce finished: " + str(reduced))

@staticmethod @staticmethod
@expose @expose
def mymap(a, b): def myreduce(mapped):
print (a, b) output = 0
res = 0 for x in mapped:
for i in xrange(a, b): output += x.value
res += i return output
return res
Done!

🎉
https://git.sr.ht/~hummer12007/parcs-python

You might also like