Cloud Computing Lab Manual
Cloud Computing Lab Manual
Experiment 1
Installing VirtualBox/VMware with Different OS Flavors
Aim:
To install VirtualBox or VMware Workstation and set up different flavors of Linux/Windows
OS on top of Windows 7/8.
Software/Tools Required:
Oracle VirtualBox / VMware Workstation
ISO file of Ubuntu or other Linux/Windows OS
Windows Host Machine
Procedure:
1. Install VirtualBox:
o Download Oracle VirtualBox from the official website:
https://www.virtualbox.org/
o Run the installer and follow the installation steps.
Beginning with the Installation:
1
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Getting Started:
2
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Ready to Install:
3
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Installing Certificates:
Finished Installation:
4
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
When you will open virtualbox it will look like as shown below:
5
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
6
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 3: Change the Base Memory Size by at least 3000 MB to adjust any Virtual
Machine on Linux.
Step 4: In the same way, increase the Disk Size by 30 GB for the same purpose.
7
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
8
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 3: Change the Graphics Controller to VBoxVGA & click on the OK option.
9
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
10
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 3: After a long time & more configuration, it will ask to click on the Continue to
Reboot the Virtual Machine. And hence, we have completed our job.
11
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
12
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Enter your Virtual Machine name, and choose your guest OS and architecture (32- vs.
64-bit) from the dropdown menu and click Next button.
A 64-bit guest needs the CPU virtualization technology (VT-x AMD/V) to be enabled
in BIOS.
Enter memory (RAM) to reserve for your virtual machine and click Next button.
Leave enough memory to the host OS.
13
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Tick at Startup Disk and Create New Hard disk and click at Next button.
Choose the type of file that you want to use for virtual disk and click Next button.
14
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Enter the size of your virtual disk (in MB) and click Next button.
A dynamically growing virtual disk will only use the amount of physical hard drive
space it needs. It is better to be rather generous to avoid running out of guest hard
drive space.
15
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
You will see the detail of your input here. Click Create button to continue.
16
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
The "New Virtual Machine Wizard" will close and back to VirtualBox Manager.
Select your Virtual Machine and click Start button.
17
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
18
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
19
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
After a successful installation we have to remove our installation .iso image from the
virtual optical drive before we reboot. This can be done from the "Devices" menu or
by removing the .iso from the VM settings:
20
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 1: Before we begin with the installation process, we need to Download ISO
for Ubuntu. For that, all the versions of Ubuntu are available on the Official Site.
21
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 3: Give a Name to your Virtual Machine and select the Location for it to install.
22
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 5: Create a Virtual Hard Disk for the machine to store files.
Step 6: Select the type of Hard disk. Using VDI type is recommended.
23
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 7: Either of the Physical Storage types can be selected. Using a Dynamically
Allocated Disk is by default recommended.
Step 8: Select Disk Size and provide the Destination Folder to install.
24
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 9: After the Disk creation is done, boot the Virtual Machine and begin
installing Ubuntu.
Step 10: If the installation disk is not automatically detected. Browse the file location and
select the ISO file for Ubuntu.
25
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 11: Proceed with the installation file and wait for further options.
Step 12: Click on the Install Ubuntu option, this might look different for other Ubuntu
versions.
26
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 13: Select Keyboard Layout, if the defaults are compatible, just click on
the Continue button and proceed.
Step 14: Select Installation Type. By default, it is set to Normal Installation, which is
recommended, but it can also be changed to Minimal Installation if there is no need for all
Ubuntu features.
27
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 15: Click on the Install Now button and carry on with the installation. Do not get
worried about the Erase disk option, it will only be effective inside the virtual machine,
and other system files outside the VirtualBox remain intact.
Step 16: Click on the Continue button, and proceed with writing changes on the disk.
28
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 18: Choose a Name for your computer and set a Password to secure login info.
29
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 20: Once the installation process is over, reboot your Virtual Machine.
30
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 21: Voila!! You’re finished with the installation process. Now you can use Ubuntu
along with Windows, without creating a dual boot.
31
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 1: To overcome this and make Ubuntu fullscreen in Virtual Box, click
on Devices then on Insert Guest Additions CD Image.
Step 2: Now you will be able to see a Disk icon on the Taskbar, click on that and it will
open the disk
32
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 3: Write right-click inside the folder select Open in the Terminal, and run the
following commands. It will ask for the user’s password, enter the password, and wait till
completion.
sudo apt update
33
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 5: Now it will ask for the password and after checking the password a new tab will
open inside the terminal. Enter Yes there.
Step 6: After installation restart Ubuntu and now you can use Ubuntu in Full-Screen Mode.
So, these are some long steps needed to Download & Install Ubuntu on VirtualBox. You
should go through the entire steps for successfully Launching Ubuntu on VirtualBox in
Full- Screen Mode. If you miss any step or perform any error, the VirtualBox Installation
of Ubuntu will not be successful.
34
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Experiment 2
Install a C Compiler in the Virtual Machine and Execute
Simple Programs
Aim:
Install a C compiler in the virtual machine created using a virtual box and execute
Simple Programs.
Hands-on Procedure:
STEP 1: Install the VMware player and host the Virtual Machine(CorePlus)
STEP 2: Open the VMware workstation and power on the virtual machine
Step2: After loading the Guest Operating Open the terminal and type the command
tc@box:~$
tce-load -wicompiletc
35
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step3: After downloading and installing all the library and gcc it return to the prompt.
Step4: Open the Terminal and type the Command sudo editor then the text window will
appear
type your c program and save it with .c extension.
36
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
c@box:~$cc demo.c
Step7 : For output use the command ./a.out.
tc@box:~$./a.out
37
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Result:
Thus the Install a C compiler in the virtual machine created using a virtual box and
execute Simple Programs.
38
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Experiment 3
Create an Amazon EC2 Instance and Set Up a Web Server
What Is EC2?
Think of EC2 (Elastic Compute Cloud) as a hotel room in the cloud. Instead of booking a
physical server to store your website, you’re renting one from Amazon’s magical cloud
infrastructure. This room (or instance) comes with all the amenities you need to host a
website. Today, we’ll install HTTPD (a web server software) in our “room” to make our
website live. 🏨✨
What is HTTPD?
At its core, HTTPD stands for Hypertext Transfer Protocol Daemon. Let’s break that
down:
Hypertext Transfer Protocol (HTTP): This is the standard protocol used on the web.
When you type a URL into your browser or click a link, you’re using HTTP to tell the
server, “Hey, send me this web page!”
Daemon (D): A daemon is just a fancy term for a background process that runs
continuously on a server. In this case, the daemon is responsible for responding to
requests from web browsers (like Chrome or Firefox) and sending back the
appropriate content.
So, HTTPD is a program that listens for incoming HTTP requests (like when you
visit a webpage) and serves back the data (HTML, CSS, images, and so on) needed to
display that page.
HTTPD vs. Apache2: Different Names, Same Game
Depending on your Linux distribution, you may encounter different names for the same basic
software:
On RPM-based distributions (like Red Hat, CentOS, or Fedora), it’s called httpd.
On Debian-based distributions (like Ubuntu or Debian itself), it’s referred to as
apache2.
Let’s look at the steps you can use to launch your EC2 instance, and how to set up a web
server using HTTPD.
Step 1: How to Launch Your EC2 Instance
First things first, let’s launch our EC2 instance. You’ll need an AWS account—signing up is
free, and AWS offers a free tier, so this won’t cost you a dime for small-scale experiments.
Head over to the AWS Management Console and log in. From the search bar, type “EC2” and
click on EC2 Dashboard.
39
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Next, choose the Amazon Machine Image (AMI) by selecting the Amazon Linux AMI, which
is free-tier eligible and super reliable. Don’t forget to give your instance a unique name!
40
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Adding a "Name" tag with a value like "MyFirstInstance" or "ProductionServer" helps you
keep track of multiple instances while adding a personal touch to your cloud workspace.
Also, remember to check the default username for the AMI you select. Since you’ve chosen
Amazon Linux, the default username is ec2-user. Keep this in mind for connecting to your
instance later!
41
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Select an Instance Type: The t2.micro is your best buddy here again, free-tier eligible and
perfect for our needs.
42
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Key Pair for SSH Access: Here’s where it gets important to have a .pem file to securely
connect to your instance. This file, also known as a key pair, acts like the secret key to your
cloud “hotel room,” allowing you to log in via SSH.
If you already have a .pem file for a previously created key pair, go ahead and choose that
from the dropdown menu.
If you don’t have a .pem file, no worries! Create a new key pair by clicking Create New Key
Pair, and download the .pem file to your computer. Make sure to store this file safely—
you’ll need it to log in, and if you lose it, you won’t be able to access your EC2 instance!
Why is this file important? The .pem file is your private key, and AWS uses it to verify that
you are the rightful owner trying to connect to the instance. You won’t get access without it,
just like how you can’t get into a hotel room without the key.
Configure Security Group: AWS EC2 security groups are like virtual firewalls that control
traffic in and out of your instance, ensuring only specific types of access. To allow web
visitors, set up an HTTP rule on port 80, and for secure server logins, enable SSH on port 22
with restricted IPs.
You can reuse security groups across instances, making configuration easier and more
consistent. Regularly review these settings to keep your instance secure and organized.
43
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Launch the instance: Boom! You’ve just launched your very own server in the cloud.
Wait a minute or two for your instance to come online. Now that we have our EC2 instance
running, let’s move to the next step of `setting up our web server.
Step 2: How to Connect to Your EC2 Instance
44
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
To connect, we’ll use the .pem file (key pair) we created earlier. If you’re on a Mac or Linux
machine, this is super simple with SSH. For Windows folks, I recommend using MobaXterm
—it’s a user-friendly terminal with SSH built-in.
If you’re new to connecting EC2 instances using MobaXterm, I’ve written a detailed guide in
my previous blog post. You can check it out here, where I show how to set up and connect to
an EC2 instance using MobaXterm.
For now, here’s a quick overview of the connection process using SSH:
ssh -i "your-key.pem" ec2-user@your-ec2-public-ip
Replace "your-key.pem" with the name of your key pair and "your-ec2-public-ip" with the
public IP of your instance (you can find this in the EC2 dashboard).
If you’ve connected successfully, congratulations! 🎉 You’re inside your cloud server.
Step 3: How to Install and Start HTTPD (Apache Web Server)
Alright, time to install our web server software (HTTPD)! We’ll be using Apache, one of the
most popular web servers around. Don’t worry, you don’t need a degree in IT to get this
working.
After you successfully connect to your EC2 instance from MobaXterm, you should be all set
to start the installation. You’re just a few commands away from having your web server up
and running!
It’s always good practice to make sure your server is up to date. To update your server, run:
sudodnf update -y
Next, we’ll install HTTPD (Apache):
sudodnf install httpd -y
Then start the HTTPD service. Run this command to get the server running.
sudosystemctl start httpd
Next, enable it to start on boot so that every time your EC2 instance reboots, your web server
comes back to life automatically.
sudosystemctl enable httpd
Time to test it out! Open your browser and type in your instance’s public IP. If you see the
Apache test page, give yourself a high-five. 🖐️You’ve just launched a web server!
Step 4: How to Host Your Custom Web Page
Now, let’s get creative! Instead of the default web server message, let’s host your very own
custom web page in just one step. This will allow you to display a unique message on your
site in no time.
Run the following command in your EC2 instance to create and display a simple,
45
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
46
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Experiment 4
Install Google App Engine. Create a hello world app and
other simple web applications using python/java.
Aim:
To Install Google App Engine. Create hello world app and other simple web
applications using python/java.
Procedure:
1. Install Google Plugin for Eclipse
Read this guide – how to install Google Plugin for Eclipse. If you install the Google App
Engine
Java SDK together with “Google Plugin for Eclipse“, then go to step 2, Otherwise, get the
Google
App Engine Java SDK and extract it.
2. Create New Web Application Project
In Eclipse toolbar, click on the Google icon, and select “New Web Application Project…”
Figure – New Web Application Project
Figure – Deselect the “Google Web ToolKit“, and link your GAE Java SDK via the
“configure
SDK” link.
47
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Click finished, Google Plugin for Eclipse will generate a sample project automatically.
3. Hello World
Review the generated project directory
48
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Copy
The extra is this file “appengine-web.xml“, Google App Engine need this to run and deploy
the
application.
File : appengine-web.xml
<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<application></application>
<version>1</version>
</appengine-web-app>
Copy
4. Run it local
Right click on the project and run as “Web Application“.
Eclipse console :
//...
INFO: The server is running at http://localhost:8888/
49
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
</appengine-web-app>
50
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Figure 1.2 – Sign in with your Google account and click on the Deploy button.
Figure 1.3 – If everything is fine, the hello world web application will be deployed to this
URL –
http://mkyong123.appspot.com/
51
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Result:
52
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Experiment 5
Simulate a cloud scenario using CloudSim and run a
scheduling algorithm that is not present in CloudSim.
Aim:
To Simulate a cloud scenario using CloudSim and run a scheduling algorithm
that is not present in CloudSim.
Steps:
53
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
54
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Steps:
1. You can copy few (or more) lines with copy & paste mechanism.
For this you need to share clipboard between host OS and guest OS, installing
Guest Addition on both the virtual machines (probably setting bidirectional
and restarting them). You copy from guest OS in the clipboard that is shared
with the host OS.
Then you paste from the host OS to the second guest OS.
2. You can enable drag and drop too with the same method (Click on the
machine, settings, general, advanced, drag and drop: set to bidirectional )
3. You can have common Shared Folders on both virtual machines and
use one of the directory shared as buffer to copy.
Installing Guest Additions you have the possibility to set Shared Folders too.
As you put a file in a shared folder from host OS or from guest OS, is
immediately visible to the other. (Keep in mind that can arise some problems
for date/time of the files when there are different clock settings on the
different virtual machines).
If you use the same folder shared on more machines you can exchange files
directly copying them in this folder.
4. You can use usual method to copy files between 2 different computer with
client-server application. (e.g. scp with sshd active for linux, winscp... you
can get some info about SSH servers e.g. here)
You need an active server (sshd) on the receiving machine and a client on
the sending machine. Of course you need to have the authorization setted
(via password or, better, via an automatic authentication method).
Note: many Linux/Ubuntu distribution install sshd by default: you can see if
it is running with pgrep sshd from a shell. You can install with sudo apt-get
install openssh-server.
5. You can mount part of the file system of a virtual machine via NFS or
SSHFS on the other, or you can share file and directory with Samba.
You may find interesting the article Sharing files between guest and
host without VirtualBox shared folders with detailed step by step
instructions.
You should remember that you are dialling with a little network of machines
with different operative systems, and in particular:
Each virtual machine has its own operative system running on and acts
as a physical machine.
55
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
PROCEDURE:
Steps:
1. Open Browser, type localhost:9869
2. Login using username: oneadmin, password: opennebula
3. Then follow the steps to migrate VMs
a. Click on infrastructure
b. Select clusters and enter the cluster name
c. Then select host tab, and select all host
d. Then select Vnets tab, and select all vnet
e. Then select datastores tab, and select all datastores
f. And then choose host under infrastructure tab
g. Click on + symbol to add new host, name the host then click on create.
4. on instances, select VMs to migrate then follow the stpes
th
a. Click on 8 icon ,the drop down list display
b. Select migrate on that ,the popup window display
c. On that select the target host to migrate then click on migrate.
56
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Before migration
Host:SACET
Host:one-sandbox
57
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
After Migration:
58
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Host:one-sandbox
Host:SACET
59
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
APPLICATIONS:
Easily migrate your virtual machine from one pc to another.
Result:
Thus the file transfer between VM was successfully completed…..
Experiment 7
Find a procedure to launch virtual machine using trystack
(Online Openstack Demo Version)
Aim:
To Find a procedure to launch virtual machine using trystack.
Steps:
60
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
TryStack.org Homepage
I assume that you already join to the Facebook Group and login to the dashboard.
After you log in to the TryStack, you will see the Compute Dashboard like:
61
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
62
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Result:
Thus the openstack demo worked successfully.
Experiment 8
Install Hadoop single node cluster and run simple
applications like word count.
Aim:
To Install Hadoop single node cluster and run simple
applications like wordcount.
Steps:
Install Hadoop
Step 1: download the Java 8 Package. Save this file in your home
directory.
63
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
For applying all these changes to the current Terminal, execute the source command.
Command: source .bashrc
To make sure that Java and Hadoop have been properly installed on your system and can
be
64
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
accessed through the Terminal, execute the java -version and hadoop version commands.
Command: cd hadoop-2.7.3/etc/hadoop/
Command: ls
All the Hadoop configuration files are located in hadoop-2.7.3/etc/hadoop directory as you
can
see in the snapshot below:
65
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 7: Open core-site.xml and edit the property mentioned below inside
configuration tag:
Command: vi core-site.xml
Step 8: Edit hdfs-site.xml and edit the property mentioned below inside
configuration tag:
Command: vi hdfs-site.xm
66
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Step 9: Edit the mapred-site.xml file and edit the property mentioned below
inside configuration tag:
In some cases, mapred-site.xml file is not available. So, we have to create the mapred-
site.xml
file using mapred-site.xml template.
Command: vi mapred-site.xml.
Step 9: Edit the mapred-site.xml file and edit the property mentioned below
inside configuration tag:
67
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
In some cases, mapred-site.xml file is not available. So, we have to create the mapred-
site.xml
file using mapred-site.xml template.
Command: vi mapred-site.xml.
Step 11: Edit hadoop-env.sh and add the Java Path as mentioned below:
hadoop-env.sh contains the environment variables that are used in the script to run Hadoop
like Java home path, etc.
Command: vi hadoop–env.sh
68
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
This formats the HDFS via NameNode. This command is only executed for the first time.
Formatting the file system means initializing the directory specified by the dfs.name.dir
variable.
Never format, up and running Hadoop filesystem. You will lose all your data stored in the
HDFS.
Step 13: Once the NameNode is formatted, go to hadoop-2.7.3/sbin directory and start all the
daemons.
Command: cd hadoop-2.7.3/sbin
Either you can start all daemons with a single command or do it individually.
Command: ./start-all.sh
Start NameNode:
The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all
files
stored in the HDFS and tracks all the file stored across the cluster.
69
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Start NameNode:
The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all
files
stored in the HDFS and tracks all the file stored across the cluster.
Start DataNode:
On startup, a DataNode connects to the Namenode and it responds to the requests from
the Namenode for different operations.
Start ResourceManager:
ResourceManager is the master that arbitrates all the available cluster resources and
thus helps in managing the distributed applications running on the YARN system.
Its work is to manage each NodeManagers and the each application’s
ApplicationMaster.
JobHistoryServer is responsible for servicing all job history related requests from client.
Step 14: To check that all the Hadoop services are up and running, run the below
command.
Command: jps
71
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Result:
Thus the Hadoop one cluster was installed and simple applications executed
72
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Experiment 9
Create a database instance in the cloud using Amazon
RDS.
Amazon Relational Database Service (Amazon RDS) is a managed database service that
simplifies the process of setting up, operating, and scaling relational databases in the
cloud. It provides resizable capacity at a cost-effective price while handling routine
database administration tasks.
A DB instance is the fundamental building block of Amazon RDS. It serves as an isolated
database environment within the AWS Cloud, allowing users to create and manage
multiple databases in a single instance. You can create and modify a DB instance using:
AWS Management Console
AWS CLI (Command Line Interface)
Amazon RDS API
Supported Database Engines
Amazon RDS supports five major database engines:
1. MySQL
2. MariaDB
3. PostgreSQL
4. Oracle
5. Microsoft SQL Server
Each database engine comes with unique features and parameters managed through DB
parameter groups that control its behavior.
Steps to Create a Amazon RDS DB instance
Follow these steps to create an RDS DB instance using the AWS Management Console.
Step 1: Access the RDS Dashboard
Log in to your AWS Management Console .
Navigate to the Amazon RDS dashboard.
Click on Create Database to start the setup process.
73
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Creation Method
Select the MySQL database engine (or any other engine based on your requirements).
74
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Version of DB Engine
Template Types
75
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Settings
DB Instance
76
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
In the above image, we see that a MySQL database named geeksdemo has been created.
77
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
Experiment 10
Create a database instance in the cloud using Google
Cloud SQL
Google Cloud SQL is a completely managed relational database service. It provides high
obtainability and automatic failover, which confirms that our database never fails and is
available for application. If a server administrator is not available, then, with the help of
Cloud SQL, users can easily deploy, maintain, and manage databases. When Google Cloud
SQL integrates with Google cloud platform services like GCE (Google Compute Engine),
App Engine, and Kubernetes Engine, it is easier to create and manage applications
requiring databases.
Instance: In order to run the database, we need to use a virtual machine called
instance created in the Google Cloud Platform. We can create multiple database
instances.
Database: A database is a collection of data that is organized in a structured way. A
database is in the form of a table, which consists of more than one row and column.
Table: A table is an arrangement of information or data, usually in rows and columns
or in a more complex structure. Tables are commonly used in reporting, research, and
data analysis.
Field: A field is a single piece of data that is stored in a record in a table.
Primary Key: It’s a unique identifier like a driver’s license number, area code, or
vehicle identification number. A relational database should have only one primary key.
Each row of data must have a primary key value and none of the rows can be NULL.
Replication: Replication is the ability to create a copy of a Cloud SQL instance or a
local database and transfer your work to the copies.
Backups: Backups restore lost data to Cloud SQL instances. If something goes wrong,
we can also restore it to its previous state by overwriting it with a backup. Enable
automatic backup for each instance that contains the necessary data. Backups protect
data from loss.
What is Google Cloud SQL?
Fully-Managed:
In a fully managed setup, the business pays fees to the provider and gains access to the
database infrastructure.
The provider manages tasks such as migration, backup & recovery, and patching.
This setup allows businesses to focus on productivity and collaboration while scaling
their applications.
Relational Database:
A relational database is a type of database management system that stores related data.
Data is organized into rows and tables, which hold interrelated data items.
Key characteristics include:
o Structured Query Language (SQL): Primary interface for communication
with relational databases.
o Data Integrity: Maintains accuracy, consistency, and completeness of data.
Three relational Database Engines Supported by Google Cloud SQL
Each of these database engines supported by Google Cloud SQL has its own strengths and
use cases, allowing developers to choose the one that best fits their application
requirements and preferences.
MySQL
78
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
PostgreSQL
SQL Server
Steps to Create a Cloud SQL instance on GCP
Here are the detailed steps to create a Cloud SQL instance on Google Cloud Platform
(GCP).
Step 1: First, go to the navigation menu and click SQL.
79
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
80
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
81
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
82
DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
graphical tools to create and manage database structures according to your application
requirements.
Connecting to Instances: Establish secure connections to your Google Cloud SQL
instances using standard database connection methods such as JDBC, ODBC, or
language-specific APIs. Ensure proper authentication and access control mechanisms
to protect sensitive data and resources.
Executing Queries and Transactions: Execute SQL queries to retrieve, update, or
delete data stored in your databases. Implement transactions to maintain data
consistency and integrity, especially in multi-user environments or critical
applications.
What is Cloud SQL Application Programming Interface (API)?
The Google Cloud SQL API allows for programmatic management of instances, providing
various functionalities such as:
Retrieving Information: Obtaining details about instances, including their
configurations and status. Listing all SSL certificates associated with an instance.
Enumerating instances within a project. Listing available service tiers for Cloud SQL
instances.
Managing Instances: Creating new instances with specified configurations. Deleting
existing instances. Restarting instances to apply configuration changes or resolve
issues.
Backup and Restore Operations: Querying information about backup runs, including
their status and metadata. Exporting databases from Cloud SQL instances to Google
Cloud Storage. Importing databases from Google Cloud Storage into Cloud SQL
instances. Restoring instances from backups to recover data or configurations.
Google Cloud SQL Advantages and Disadvantages
Advantages of Google Cloud SQL
Fully Managed Service: Google Cloud SQL is a fully managed service, meaning
Google takes care of database management tasks such as patching, backups, and
replication, allowing developers to focus on building applications.
Scalability: Google Cloud SQL offers both vertical and horizontal scalability,
allowing you to easily scale your database resources up or down based on your
application’s needs.
High Availability: Google Cloud SQL provides built-in high availability with
automatic failover, ensuring your databases remain accessible even in the event of
hardware failures or maintenance events.
Disadvantages of Google Cloud SQL
Limited Database Options: Google Cloud SQL currently supports only a limited set
of database engines, including MySQL, PostgreSQL, and SQL Server. If you require
support for other databases such as Oracle or MongoDB, you may need to consider
alternative solutions.
Vendor Lock-in: Using Google Cloud SQL may lock you into the Google Cloud
Platform ecosystem, making it difficult to migrate to other cloud providers in the
future.
Cost: While Google Cloud SQL offers a range of pricing options, including pay-as-
you-go and committed use discounts, running databases in the cloud can still incur
significant costs, especially for large-scale deployments or resource-intensive
workloads.
83