EMC Avamar Installation and Configuration

Lab Guide

Lab Exercise 4-1: Node Replacement

Purpose:

Objectives:

In this lab, you will incorporate a Spare storage node as an

active member of the Avamar server to replace a node that
failed.

In this lab you will

References:

Fail a node
Perform data recovery
Decommission the failed node
Add a replacement data node

EMC Avamar Installation and Configuration Student Guide

DISCLAIMER: The node replacement

procedures in this lab are provisional and
should not be used at a customer site.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 1 of 26

EMC Avamar Installation and Configuration

Lab Guide

Desired Architecture
Simulate storage4 failure.
Add storage5 as a replacement node.
Avamar Utility
Node

Avamar Data
Node

Avamar Data
Node

Avamar Data
Node

Avamar Data
Node

Avamar Data
Node
(Spare)

utilitynode

storage0

storage1

storage2

storage3

storage4

192.168.100.2

192.168.100.3

192.168.100.4

192.168.100.5

192.168.100.6

192.168.100.7

Avamar 1X4 Server Configuration

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 2 of 26

EMC Avamar Installation and Configuration

Lab Guide

Lab Exercise 4-1 Part 1: Fail a node

Step
Action
1
Open a PuTTY session to your utility node by clicking the PuTTY icon.

Login as user root with password: changeme.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 3 of 26

EMC Avamar Installation and Configuration

Step
2
Type: nodenumbers
Make note of the output

Lab Guide

Action

Start anAvamar Administrator in Windows, click the Avamar Administrator

shortcut on the desktop or from the Windows Start menu.
Enter the following information:
User Name: MCUser
Password: MCUser1
Domain Name: /
Administrator Server: utilitynode
Then, click Log On. Accept any JRE warnings.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 4 of 26

EMC Avamar Installation and Configuration

Step
4
Click on Server.

Lab Guide

Action

Observe that all nodes are online.

Close all Avamar Administrator windows

Open a PuTTY session to storage3. Accept any warnings and login as user root
password: changeme.

Kill the storage nodes gsan process in order to cause the node to fail. Type:
killall gsan

Close the PuTTY session to the storage node.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 5 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
9
To start Avamar Administrator in Windows, click the Avamar Administrator
shortcut on the desktop or from the Windows Start menu.
Enter the following information:
User Name: MCUser
Password: MCUser1
Domain Name: /
Administrator Server: utilitynode
Then, click Log On. Accept any JRE warnings.

Note: It may take several minutes for the log on; please be patient.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 6 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
10 You will see an Avamar Alert popup that informs you that a node has gone offline.
Click View Details.

Study the information in the Event Detail window. Then, click Close.

Click OK on the Avamar Alert popup to close the window.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 7 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
11 From the Avamar Administrator menu, click Server.
The following screen appears. Note that Node 0.3 is red (offline).

Close the Avamar Administrator Server window.

Close the Avamar Administrator menu.

End of Lab 4-1 Part 1.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 8 of 26

EMC Avamar Installation and Configuration

Lab Guide

Lab Exercise 4-1 Part 2: Perform Recovery When Avamar

Server is Running in Degraded Mode
Step
Action
1
On your client machine, open Windows Explorer and navigate to
C:\WINDOWS\Fonts.

Delete the ariblk.TTF file.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 9 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
2
Launch Avamar Administrator. Click Backup & Restore.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 10 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
3
Expand the tree in the left pane and select your client.
Click the Select for Restore tab in the right pane.
Browse through your backup and select the C:\WINDOWS\Fonts\ariblk.ttf file.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 11 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
4
Select Actions > Restore Now from the menu bar. Then, click OK to begin the
restore.

Click OK to confirm.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 12 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
5
Monitor the progress of the restore by clicking Navigation > Activity. Verify that the
restore completes successfully.

Verify that the ariblk.ttf file was restored by checking the contents of the
C:\WINDOWS\Fonts folder on your client.

End of Lab 4-1 Part 2.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 13 of 26

EMC Avamar Installation and Configuration

Lab Guide

Lab Exercise 4-1 Part 3: Perform Node Replacement

Step
Action
1
Open a PuTTY session to your utility node and login as user root password
changeme.
Type: nodenumbers and study the output.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 14 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
2
In Avamar Administrator, select Navigation Server, and click the Checkpoint
Management tab.

Ensure that a checkpoint that is acceptable as a suitable rollback point (if

required) exists and has been validated (hfs checked).
3

Back in the PuTTy session for the utilitynode, change users to admin. Type:
su - admin
Load the admin SSH keys. Type:
ssh-agent bash
ssh-add ~/.ssh/admin_key
passphrase: P3t3rPan

Suspend all Avamar client activities. Type: avmaint suspend

Suspend all maintenance jobs. Type: avmaint sched suspend --avamaronly

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 15 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
4
Type the following to display information about the spare node:
nodedb print --nodes=0.spare.0

Update the /usr/local/avamar/var/probe.xml file using the nodedb

command as shown below.
Verify that the info on node 0.3 in the probe.xml file matches the node that has
failed:
nodedb print --nodes=0.3
Remove the entry for the spare node:
nodedb delete node --node=0.spare.0
Replace the failed node with a storage node using the same information as the
spare node:
nodedb add node --addr=192.168.100.7 --replace
--node=0.3 --type=storage

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 16 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
6
Verify that the probe.xml file contains information reflecting the replaced node:
nodedb print

Exit out of the admin user shell (twice).

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 17 of 26

EMC Avamar Installation and Configuration

Step
8
Change users to user dpn. Type:

Lab Guide

Action

su - dpn
Load the dpn SSH keys, type:
ssh-agent bash
ssh-add ~/.ssh/dpnid
export SYSPROBEUSER=root

To synchronize time on the new storage node, type:

asktime

Enter y to use the timeservers specified.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 18 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
10 Enter n to the question Do you have a dedicated Avamar Subnet?

11

Enter y to use the previously specified time zone.

12

Enter y to configure the default set of nodes.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 19 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
13 Enter y to proceed with the installation.

14

Check the time displayed and confirm by entering y.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 20 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
15 Enter y to watch for time synchronization.

16

This next step takes about 10 minutes. After completion, enter y to see the results.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 21 of 26

EMC Avamar Installation and Configuration

Step
17 Results:

Lab Guide

Action

(0.3) ssh -x [email protected] '/usr/sbin/ntpq -pn 2>&1'

remote
refid
st t when poll reach
delay
offset jitter
==============================================================================
*10.127.175.1
10.127.210.201
5 u
22
64 377
99.884
30.536 41.489
+10.127.175.2
10.127.175.1
6 u
13
64 377
99.964
21.053 36.499
192.168.100.2
.INIT.
16 u
16
64
0
0.000
0.000 4000.00
192.168.100.3
.INIT.
16 u
16
64
0
0.000
0.000 4000.00
Using /usr/local/avamar/var/probe.xml
(0.s) ssh -x [email protected] '/bin/date'
(0.0) ssh -x [email protected] '/bin/date'
(0.1) ssh -x [email protected] '/bin/date'
(0.2) ssh -x [email protected] '/bin/date'
(0.3) ssh -x [email protected] '/bin/date'
Thu Mar 18 14:41:46 EDT 2010
Thu Mar 18 18:41:46 UTC 2010
Thu Mar 18 18:41:46 UTC 2010
Thu Mar 18 18:41:46 UTC 2010
Thu Mar 18 18:41:46 UTC 2010
-------------------------------------------------------Done.
NOTES:
* Time on this host is: Thu Mar 18 14:41:46 EDT 2010

18

Exit out of the dpn user shell (twice).

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 22 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
19 Change user to admin and load the admin SSH keys. Type:
su - admin
ssh-agent bash
ssh-add ~/.ssh/admin_key
passphrase: P3t3rPan

20

Note: This task must be performed as the admin user.

Rebuild the new node using the rebuild.node command. Type:
rebuild.node --nodes=0.3 --clean --tag=0.3

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 23 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
21 Launch a new PuTTY session to your utility node and login as user root password
changeme.
Monitor the status of the new node by typing:
status.dpn <Number of seconds to refresh>

Ensure that the new node comes online.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 24 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
22 Go back to the SSH session running as the admin user and type: nodenumbers

23

In the PuTTY session to your utility node (host 1) logged on as root, monitor the
status of your nodes by typing:
status.dpn

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 25 of 26

EMC Avamar Installation and Configuration

Lab Guide

Step
Action
24 Restart server operations and scheduled cron jobs. In the PuTTY session running as the
admin user, type:
avmaint resume
avmaint sched resume --avamaronly
25

Exit all PuTTY sessions.

26

View the Server Monitor in Avamar Administrator to see that the new node is online.

End of Lab 4-1 Part 3

End of Lab 4-1.

Copyright 2010 EMC Corporation. All Rights Reserved.

Version E

Page 26 of 26