FAS3050/3070/60xx CONVERGED INSTALL FOR NEW SYSTEMS

1. Prework: CFE Environment Variables, DNS, Cabling, Extract Kernel Source Files
and setup on Admin Server (Http & TFTP enabled),
disk ownership
2. Initial Deployment: Setup Netboot Port & Netboot kernel file
3. Boot phase: mount path from admin server - rootfs.img file once Ontap comes u
p
4. Install phase: provide http path for image.tgz file from Boot Menu (Install)
5. Setup phase: after
BR - Change admin password
security login create -username administrator -role admin -application http -aut
hmethod password
Hardware requirements for FAS3050/3040/3070/60xx
NOTE: The Hardware Configuration Guide for the release is the final authority on
configurations. For the Boilermaker release,
see the Boilermaker Hardware Configuration Guide for up-to-date configuration op
tions.
The Data ONTAP GX/Converged nodes requires the following components in fixed slot
locations:
1GB CF Card X1418A-R6
RLM Card X3304
Dual GbE Copper card in slot 1 or slot 4 (PCI-X1) X1037C (FAS3050 only)
Dual GBE copper card in slot 4 (PCI-E) X1039A-R6 (FAS3040/3070/60xx only)
NVRAM5 / IB card in slot 3 (PCI-X3) X3145A ( For FAS60xx CFO use slot1 or slot 2
for non CFO)
Network/Subnet requirements
You need to have 3 different subnets to configure GX/Converged. It is not necess
ary to have 3 different networks. Therefore,
a /22 IP address which is what we use in our SVL Java 4 labs is sufficient witho
ut the need to install additional switches.
However, the benefits of seperate networks are clear and should be obliged when
possible.
For Clustered ONTAP, if you are going the route of separate networks (usually le
ss painful), then plan on getting 6 IP addresses.
2 For the cluster IPs, 2 for the mangament IPs, 1 for 7G and 1 for the RLM. Once
full 7 mode is supported, 2 IP addresses should
suffice - 1 for the filer and 1 for the RLM.

Cabling for FAS30xx

If you need more than one data port:
Ports
1 or
Ports
Ports

e1a
pci
e0a
e0c

and RLM --> Management Network (install the mangement card in pci slot
slot4)
and e0b --> Cluster Network
and e0d --> Data Network

If you do not need more than one data port:

port e0d and RLM --> Management Network
Ports e0a and e0b --> Cluster Network
Port e0c -->Data Network
Cabling for FAS31xx
FAS3140, 3160, and 3170
If you need more than one data port:
Ports e0M and
Ports e0a and
Ports e1a and
pci slot4)
If you do not

RLM --> Management Network

e0b --> Cluster Network
e1b --> Data Network (install the mangement card in pci slot 1 or
need more than one data port:

port e0M and RLM --> Management Network

Ports e0a and e0b --> Cluster Network
Port e1a -->Data Network
Cabling for FAS60xx
Ports e0f and RLM --> Management Network
Ports e0a and e0b --> Cluster Network
Ports e0c and e0d --> Data Network
If additional client access port are desired, you can add/use expansion ports e1
b, e2a, e2b, e2c, e2d, e4a, etc.
Cabling for FAS2040: ShuarXL
Shuar Cabling, official guide
Check your firmware
CFE> version
The minimum required firmware/bios versions to run ONTAP GX/Converged are
FAS3050 (Deux)
CFE: 3.0.0 or higher
RLM: 1.2 or higher
FAS3070 (Drwho)
BIOS_VERSION
2.2X1
LOADER_VERSION
1.4X1
FAS60xx (Excelsior)
1.5X2 or greater
Get the latest Excelsior version from Excelsior_BIOS
If your firmware/Bios is different, update it using these steps
Grab the Bios from here:
1.5X2 for Excelsior/Excalibur platform (/x/eng/rlse/bios/1.5X2/OEM/netapp/excel/
excelsior.flash)
2.2X1 for Drwho/Wahoo platform (/x/eng/rlse/bios/2.2X1/OEM/netapp/DrWho/drwho.fl
ash) (the newest version is 2.2X2)
For FAS3050 CFE firmware:
http://now.netapp.com/NOW/download/tools/serviceimage/30800486/ (You will need t
o extract the firmware.img)
Copy the firmware/Bios to your ~/web directory, (or to a TFTP server)
Update the filers firmware via the network
ifconfig e0a -addr=172.17.xxx.xxx -mask=255.255.xxx.xxx -gw=172.17.xxx.xxx -dns
=172.19.2.30

ping the ~/web or tftp server to verify connectivity. Next we will flash the new
image into our filer.
flash http://10.57.83.10/~eliott/drwho.flash flash0a
If you're using a tftp server:
flash tftp://172.17.130.90/firmware.img flash0a
AFter the firmware is finished programing you need to reload it by issuing the B
YE command.
CFE> bye
Verify and upgrade RLM if needed
CFE> priv set advanced
CFE*> rlm status
If the RLM does not respond, the firmware on the RLM card is most likely downrev
, and cannot be accessed via CFE and needs to be upgraded.
First verify the RLM module is installed on the 30xx systems.
CFE*> show devices
If you see the RLM module then you're good to proceed to the next step. If you d
on't see the RLM then you need to order one and install it. Part# X3304-R5 If yo
u don't know how to install the rlm then refer to for FAS30xx: http://now.netapp
.com/NOW/knowledge/docs/hardware/filer/rep_rlm_ro.pdf
Download the latest RLM version from here: http://web.netapp.com/engineering/pro
jects/rlm/sw/RLM_Firmware_Nightly_Builds.html
NOTE: The RLM firmware is the same for FAS30xx and FAS60xx filers
Make sure you have IP's assigned to one of your ethernet ports if you need to do
wnload/update the rlm module
ifconfig e0a -addr=172.17.xxx.xxx -mask=255.255.xxx.xxx -gw=172.17.xxx.xxx -dns
=172.19.2.30
CFE*> rlm update http://web.netapp.com/~eliott/RLM_FW.tar.gz
Reboot the filer
If your filer has an older version of the RLM module that does not support netwo
rk boot, you will have to update it from Ontap by doing the following:
software install http://web.netapp.com/~eliott/RLM_FW.zip -f
rlm update
Then reboot
Configure the RLM
CFE> priv set advanced
CFE*> rlm setup
Follow the wizard to assign your RLM IP's
At this point the CFE and RLM should be upgraded and running the recommended fir
mware, and all hardware components installed and cabled as per GX/Converged requ
irements
Setup the boot environment
Data ONTAP GX/Converged requires a few changes to the CFE/LOADER environment to
properly boot.
With change 892911 for burt 300201, boot arguments were renamed. As a result, wh
en setting up the boot environment using a Boilermaker build that includes that
change, you should replace "ntap" with "bootarg" in the boot arguments described
below.
From the loader prompt run the following environment commands
OUTDATED on 06/05/2009-- should be bootarg.<foo> not ntap.<foo> Please confirm a
nd edit. . as well as here

set-defaults
setenv ONTAP_NG true
setenv bootarg.init.usebootp false
setenv bootarg.mgwd.autoconf.disable true
setenv bootarg.init.dhcp.disable false (if you are using dhcp then don't use th
is command)
setenv bootarg.bsdportname e1a (on 60x0: e0f; on 30x0: e1a or e0M on Fas3170, o
r e0c)
For 10 mode only
setenv bootarg.init.boot_clustered true
For transition from 10M or mixed-mode to 7-mode
If you have a 10M or mixed mode system and you wish to clear the existing 10-Mod
e config prior to booting in pure 7-Mode:
There is a TASK in gx_boot called clear_var that will blow away the /var on the
cfcard and will wack the NVRAM:
setenv bootarg.init.wipeclean true
setenv bootarg.init.clearvarfsnvram true
boot_primary
It will then shutdown to the CFE/LOADER again and the next time you boot you sho
uld go to the bootmenu automatically. Then you should run init or fake_init.
Make sure that you do this to all nodes in the cluster, or the remaining active
nodes will attempt to restore the 10M config. To be safe, you should bring all t
he nodes to the CFE prompt at the same time, so that no 10-mode instances are ac
tive.
For transition from 7M or mixed-mode ? 10-mode
Needs to be filled in.
Netboot the GX/Converged kernel
For FAS3050/FAS3070 e1a is supposed to be the management port but sometimes has
timeout issues when netbooting converged builds. You should use e4a or e0c inste
ad for netboots. We will use e0c until we get the software installed/netbooted a
nd then go back to e4a or e1a (depending on where you have the pci nic installed
) once The Gx/Converged is loaded. (This does not apply to FAS60xx filers, you a
lways use the e0f for Excelsior/Excalibur systems)
For FAS3040/3050/FAS3070
LOADER> setenv ntap.bsdportname e0c
Now we can assign the IP address belonging to the port we're using.
Note: If you're using different networks/switches use the IP that belongs to por
t 0c (aka data port), not the management IP.
ifconfig e0c -addr=172.17.xxx.xxx -mask=255.255.xxx.xxx -gw=172.17.xxx.xxx -dns=
172.19.2.30
For FAS60xx (port e0f is the management port and should always be used to netboo
t)
ifconfig e0f -addr=172.17.xxx.xxx -mask=255.255.xxx.xxx -gw=172.17.xxx.xxx -dns
=172.19.2.30
You're almost ready to netboot the filer but first you need to grab the latest c
onverged build.
/x/eng/nrlse31/ontapng/nightly/br-mainngN_070713_2200/final/bedrock/export/x86_
64/tarball/image.tgz

(you can replace br-mainngN_070713_2200 with the date you desire)

Build path Update
Paloma is not dumping nightly so you will have to use the date instead, here is
the path:
/rlse/ontapng/dualN_080125_1200/final/bedrock/export
BR latest builds can be found here:
/rlse/ontapng/br-mainngN/final/bedrock/export
You need to extract the kernel and the rootfs.img from image.tgz
From a linux host run:
tar -zvxf image.tgz
You now have all the files you need to netboot your filer:
BUILD
CHECKSUM
COMPAT.TXT
INSTALL
README.TXT
VERSION
diags.tgz
kernel
perl.tgz
platform.ko
rootfs.img
You need to copy the kernel and the platform.ko to your tftp server or your ~/we
b directory and copy the rootfs.img to a NFS share. Place the original image.tgz
in your web directory or a http server of your choice.
Ping the TFTP/HTTP server to insure connectivity.
For FAS3050 (Deux)
netboot tftp://yourtftpip/kernel
For FAS60xx (Excelsior)
netboot http://web.netapp.com/~eliott/kernel
If the rootfs.img file does not exist in the same directory as the kernel file,
you are prompted for the complete NFS path of
the untarred rootfs.img file.
Enter the NFS path for rootfs.img:
10.10.20.52:/vol/users/eliott/web/rootfs.img
If you dont know the filer name, use df -k (after you have cd'd to the path)
martink@dpg-36a /x/eng/kc-users/martink/image $cd /x/eng/kc-users/martink
martink@dpg-36a /x/eng/kc-users/martink $df -k .
Filesystem 1K-blocks Used Available Use% Mounted on
kcpine.rtp.netapp.com:/vol/users
If you get the following error
Skipped mounting the CFCARD.
ERROR: /nfsroot/platfs.img not found.
You will have to copy the platfs.img into the same NFS share where you put the r
ootfs.img
After you provide the NFS path hit control-c when prompted for the boot menu
How would you like to continue booting?
(normal)
Normally
(install)
Install new software first
(setup)
Run setup first

(init)
Initialize disks and create flexvol
(maint)
Boot into maintenance mode
(syncflash) Update flash from backup config
(reboot)
Reboot device
Please make a selection:
Choose install
install
You will hit Y to confirm the install and you will be prompted to provide the ht
tp path to the image.tgz
What is the URL for the package?:http://web.netapp.com/~eliott/image.tgz
Hit enter for username and password prompts
Once install is complete, answer "y" to the reboot now question
The system will then boot to the loader prompt
Reset the ntap.bsdportname to e1a or e4a depending on what pci slot you have the
management card installed in. (This does not
apply to FAS60xx filers)
NOTE: Don't forget to set the set the ntap.bsdportname back to e1a or e4a if you
changed it to e0c from the previous Netboot
section. Follow the example below:
LOADER> setenv ntap.bsdportname e1a
Once you set the management port back to e1a or e4a you can continue to boot the
filer (only applies to 30xx filers)
LOADER> autoboot
Hit control-C when prompted for boot menu
We will now make sure the filer owns all the shelfs/disk it has connected
From the boot menu, select maint (maintenance mode) to assign/re-assign disk own
ership
Usr: root
Passwd: admin
Type "dbladecli" to access GB mode (aka 7 mode)
disk show -v
disk remove_ownership all
disk assign all
disk show -v (to confirm ownership)
Hit cntrl-d to exit GB shell
Type reboot
If your filer is not autobooting you can change that by issues the following fro
m the CFE/LOADER prompt:
CFE> setenv AUTOBOOT true
CFE> autoboot
Hit control-C when prompted for boot menu
From the boot menu select setup
setup
Follow the setup wizard like so:
Please enter the node hostname []: dpg-gx3
Where is the node located? []: J4.2.511
Do you want to configure a management interface? (no, yes) [yes]: yes
Which port do you want to use? [e4a]:
Please enter the IP Address for this interface: 172.17.131.82
Please enter the Netmask for this interface: 255.255.252.0
Please enter the IP Address of the default gateway: 172.17.128.1
Do you want to run DNS resolver? (no, yes) [no]: yes
Please enter DNS domain name(s): lab.netapp.com
Please enter the IP address for the first nameserver: 172.19.2.30

Please enter the IP address for the second nameserver: 172.19.3.32

Do you want to enable remote management? (no, yes) [yes]:
Please specify the remote management username [admin]: root
Which access protocol(s) can this user use? (console, http, ssh, telnet)
[console, http, ssh]: console, http, ssh, telnet
The node has now been configured.
Now you need to initialize the disk
At the boot menu choose init
Zero disks and install a new file system? (y or n): y
After the disk zero you will be done with the Converged install and should be at
the login prompt
username:root
password:admin
GB shell: dbladecli
GX shell: ngsh
Your filer is now ready to create a cluster, configuring the data ports, create
vservers, etc.
Cluster configuration
To create a cluster you will do the following on the first node
For instance, on filer wqa-gx1 (First one in) The cluster command syntax is as f
ollows:
cluster create -license NWUZZJPJYBFDAA -clustername cluster -mgmt-ip <cserver li
f ip> -mgmt-netmask <cserver lif mask>
-mgmt-gateway <cserver lif gw> -mgmt-port <cserver lif port> -ipaddr1 <ipad
dress 1> -netmask <netmask>
-mtu <mtu size> -ipaddr2 <ipaddress 2>
For an explanation on what the extra 'mgmt' arguments mean and how they are conf
igured, please refer to the developer setup wiki.
Example:
cluster create -license NWUZZJPJYBFDAA -clustername wqa-cluster1 -mgmt-ip 17.19
2.1.5
-mgmt-netmask 255.255.255.0 -mgmt-gateway 17.192.1.1 -mgmt-port e0c -ipaddr
1 172.17.241.20
-netmask 255.255.255.0 -mtu 9000 -ipaddr2 172.17.241.21
wqa-gx2 (second node)
cluster join -clusteripaddr <ip address of first node's cluster network> -ipad
dr1 <ipaddress 1>
-netmask <netmask> -mtu <mtu size> -ipaddr2 <ipaddress 2>
Example:
cluster join -clusteripaddr 172.17.241.20 -ipaddr1 172.17.241.22 -netmask 255.2
55.255.0 -mtu 9000
-ipaddr2 172.17.241.23
Verify your network
network interface show
License
You can now install the licenses you might need ( note: you cannot add any licen
ses until the cluster is created)
GX_Base NWUZZJPJYBFDAA 666
GX_Base VKMHHVUWGFAAAA 1
GX_Base RCXQNZCDNIAAAA 2
GX_Mirror LLQDBKPJYBFDAA 666

GX_FlexVol_HPO XFOSBKPJYBFDAA 666

BR_NFS TJFAEKPJYBFDAA 666
GX_CIFS JAMHCKPJYBFDAA 666
GX_SnapRestore VUJWCKPJYBFDAA 666
License Commands
license add NWUZZJPJYBFDAA
license add LLQDBKPJYBFDAA
license add XFOSBKPJYBFDAA
license add TJFAEKPJYBFDAA
license add JAMHCKPJYBFDAA
license add VUJWCKPJYBFDAA
Verify the licenses were installed
license show
LICENSE UPDATE
NOTE: the use of licenses with serial number 1-80-123456 should be discontinued
as that serial number could potentially
overlap with a serial number in the field. Those license keys have now been remo
ved from this wiki and no further
feature keys for those serial numbers will be provided. You will have to re-inst
all the Base license with one of
the following to change the serial number of your cluster as well as all of the
existing feature licenses before you
can add any new feature licenses from one of the tables below We are now providi
ng three (3) sets of internal licenses
to make it a bit easier to setup multiple clusters that may need to communicate
with each other. The designated serial numbers
for these will be 1-80-000011, 1-80-000013, 1-80-000017 these are special, non-M
FG serial numbers for easier tracking
Serial Number: 1-80-000011
Service
License Code
GX_Base
VMZIVZGKWVMADF
GX_Mirror
TBVMWZGKWVMADF
GX_FlexVol_HPO FWSBXZGKWVMADF
GX_CIFS
RQQQXZGKWVMADF
GX_SnapRestore DLOFYZGKWVMADF
NFS
BAKJZZGKWVMADF 666
SnapMirror_DP NUHYZZGKWVMADF
Striped_Volume LJDCBAHKWVMADF
iSCSI ZOFNAAHKWVMADF 666
FCP
XDBRBAHKWVMADF 666
FlexClone
PFMUYZGKWVMADF
SAN limits are enforced
independent of node count
Serial Number: 1-80-000013
Service
License Code
GX_Base
XBOLYLDPMZSEBG
GX_Mirror
VQJPZLDPMZSEBG
GX_FlexVol_HPO HLHEAMDPMZSEBG
GX_CIFS
TFFTAMDPMZSEBG
GX_SnapRestore FADIBMDPMZSEBG
NFS
DPYLCMDPMZSEBG 666
SnapMirror_DP PJWADMDPMZSEBG
Striped_Volume NYREEMDPMZSEBG
iSCSI BEUPDMDPMZSEBG 666
FCP
ZSPTEMDPMZSEBG 666
FlexClone
RUAXBMDPMZSEBG

Node Count
666
666
666
666
666
666
666
666

Node Count
666
666
666
666
666
666
666
666

SAN limits are enforced

independent of node count
Serial Number: 1-80-000017
Service
License Code
GX_Base
ZYSOQSYLTDIFMA
GX_Mirror
XNOSRSYLTDIFMA
GX_FlexVol_HPO JIMHSSYLTDIFMA
GX_CIFS
VCKWSSYLTDIFMA
GX_SnapRestore HXHLTSYLTDIFMA
NFS
FMDPUSYLTDIFMA 666
SnapMirror_DP RGBEVSYLTDIFMA
Striped_Volume PVWHWSYLTDIFMA
iSCSI DBZSVSYLTDIFMA 666
FCP
BQUWWSYLTDIFMA 666
FlexClone
TRFAUSYLTDIFMA
SAN limits are enforced
independent of node count

Node Count
666
666
666
666
666
666
666
666

Virtual server configuration

Create Virtual server by issuing below command. (replace vserver parameters with
your own)
vserver create -vserver vs0 -rootvolume root_vs0 -aggregate aggr1 -ns-switch fi
le, nis -rootvolume-security-style mixed
Verify vservers you create by issuing the following command
vserver show
Data network configuration
Create data network interface by using below command
network interface create -server (wafl_vf) -vif (dataline) -role data -home-nod
e wqa-gx1 -home-port e0c
-address xxx.xxx.xxx.xxx -netmask 255.255.xxx.0 -status-admin up -failover-pol
icy priority
Add a route for the newly created interface and confirm cluster network
network routing-groups route create -server wqa-gx1 -routing-group <hit tab to
autofill>
-lif dataline -destination 0.0.0.0/0 -gateway 172.17.128.1 -metric 10
Note You must create a route for every data vif's you create, otherwise it will
not be accessible. Leave ip,mask at 0.0.0.0,
just include the GW of the data port.
Example:
dpg-drwho7::> network routing-groups route create -server vs0 -routing-group d10
.225.164.0/22 -destination 0.0.0.0/0 -gateway 10.225.164.1
Verify route table with
network routing-groups show

dpg-drwho7::> network routing-groups show

Routing
Server Group Subnet Role Metric
--------- --------- --------------- ------------ ------cluster
c10.224.164.0/22

10.224.164.0/22 cluster-mgmt 20
c10.225.164.0/22
10.225.164.0/22 cluster-mgmt 20
dpg-drwho7
c10.225.204.0/23
10.225.204.0/23 cluster 30
n10.225.160.0/22
10.225.160.0/22 node-mgmt 10
dpg-drwho8
c10.225.204.0/23
10.225.204.0/23 cluster 30
n10.225.160.0/22
10.225.160.0/22 node-mgmt 10
vs0
d10.225.164.0/22
10.225.164.0/22 data 20
7 entries were displayed.
The NIS, AD, CIFS configuration is out of the scope of this document and should
be configured by the user, not eng support. For more commands and customization
of Ontap GX, take a look at msaberi command page: http://web.netapp.com/~msaberi
/gx.html
To confirm cluster networking - try the following cmd:
ping cluster
--Eliott 08:35, 21 August 2007 (PDT)
Developer Fastpath To 10-Mode Installation
If your work requires that you have usable filers but you find that you need to
re-install them frequently, another option is to use one of the available automa
ted approaches. These tend to be a lot less error-prone than manual installation
s and can be a quick way of verifying your hardware and installation image.
--------------------------------------------------------------------------------------L

Automatically booting into 7/10 mode

Default boot variables
Useful Links
scripted install of gx
cold and warm start
cifs in 10 mode
cluster setup guide
Tools and debugging
10mode cifs setup
This page was last modified on 7 December 2010, at 00:47.This page has been acce

ssed 6,275 times.Privacy policyAbout WikidNetApp Confidential