Facebook Network Appliance

Deployment
USB Based Software Installation Guide

Version 3.0
 FNA: USB Based Software Installation Guide | 1

Copyrights and Trademarks

© 2017 Facebook, Inc. All rights reserved.
Facebook is a trademark of Facebook, Inc. in the U.S. and other countries.
Linux® is a registered trademark of Linus Torvalds in the United States.
Mac® and Mac OS® are registered trademarks of Apple Inc.
Windows® is either a registered trademark or a trademark of Microsoft Corporation in the United States and/or other
countries.
 FNA: USB Based Software Installation Guide | 2

Contents
Chapter 1: Introduction..........................................................3
Chapter 2: USB Boot Disk Creation......................................4
Chapter 3: IP Address Configuration.....................................7
3.1 Basic Configuration...................................................7
3.2 Manual Configuration................................................8
3.3 Next Steps................................................................9
Chapter 4: Configure the FNA Cluster Switch.....................10
4.1 Switch Configuration...............................................10
4.2 Next Step................................................................12
Chapter 5: Re-Image the Servers to BSD...........................13
Chapter 6: Troubleshooting.................................................14
 FNA: USB Based Software Installation Guide | 3

Chapter 1: Introduction
Thank you for choosing to install the Facebook Network Appliance (FNA)!
Facebook Network Appliance is Facebook’s content caching program. FNA provides Internet Ser-
vice Providers (ISPs) with an efficient means of delivering static Facebook content from within their
existing network. After deploying an FNA kit, an ISP will offload a significant amount of Facebook
content from its backbone network and vastly improve the Facebook user experience.
If the FNA kit contents (switch and servers) were not configured with the IP configuration specified
in the order form, the IP reconfiguration procedures described in this document need to be
performed. Also, at any time, network changes can necessitate reconfiguration.
Note: For purposes of following these procedures, Facebook assumes you have already
completed the hardware installation of the supplied kit, including racking and
connection of the servers and switch.
 FNA: USB Based Software Installation Guide | 4

Chapter 2: USB Boot Disk Creation

The USB boot disk creation process can be performed using a Windows, Mac OS, or Linux
computer and a blank USB flash drive.
1. Go to the Network Partners Portal, URL: https://partners.facebook.com/network/.
2. Locate the cluster that needs to be reconfigured: Caching > Manage FNA > your cluster name
(see Figure 1).
3. In the area labeled USB Boot Disk (see Figure 1), click to download the following to your
computer:
●● Disk Image - USB boot disk image
●● Switch Config - Configuration files
●● Hosts Config - Configuration files

Figure 1: USB Disk Image Download Location

4. Connect a blank USB flash drive to your computer.

Warning: Make sure that no important data exists on the USB flash drive before continuing. These procedures
will completely overwrite any current contents of the flash drive.
5. Create the USB boot disk on the flash drive by following the instructions for the appropriate operating system:
●● Table 1: Instructions for Windows Operating Systems
●● Table 2: Instructions for Mac Operating System
●● Table 3: Instructions for Linux Operating Systems
 FNA: USB Based Software Installation Guide | 5

Table 1: Instructions for Windows Operating Systems

Step Instruction Illustration
a. Download and install Win32 Disk Imager and 7-Zip.

b. Use 7-Zip to extract the disk image.

c. Open Win32 Disk Imager. Navigate to the image in

Image File and select the blank USB flash drive in
Device. Make sure the correct device is selected, then
click Write (Warning: This action will overwrite any
disk specified. Ensure you have selected the correct
USB disk before continuing).

Go to step 6 on page 6.

Table 2: Instructions for Mac Operating System

Step Instruction Detail
a. Run diskutil list to find the mount point of the diskutil list
USB drive. /dev/disk0 (internal, physical):
/dev/disk1 (internal, virtual):
/dev/disk2 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: FDisk_partition_scheme *15.5 GB disk2
1: DOS_FAT_32 UNTITLED 15.5 GB disk2s1
Unlocked Encrypted
b. Confirm you have selected the correct mounting
point by running diskutil info <disk#> and
verifying the drive name, size, and protocol.

c. Run diskutil unmountDisk <disk#> to diskutil unmountDisk /dev/disk2

unmount the USB drive. Unmount of all volumes on disk2 was successful

d. Run gunzip to decompress the disk image. gunzip –d ~/Downloads/caelus.img.gz

 FNA: USB Based Software Installation Guide | 6

Step Instruction Detail

e. Run dd to write the disk image to the USB flash drive sudo dd if=~/Downloads/caelus.img of=/dev/rdisk2
(Warning: This action will overwrite any disk specified. bs=1m && sync
Ensure you have selected the correct USB disk before
continuing). Note: to increase the speed of imaging Example:
process in MacOS, add an “r” before the disk’s name sudo dd if=./caelus.img of=/dev/rdisk2 bs=1m
(in our example the disk was “disk2”, but we are using 256+1 records in
“rdisk2” as an output device). 256+1 records out
268435456 bytes transferred in 24.221118 secs
Go to step 6 below.
(11082703 bytes/sec)

Table 3: Instructions for Linux Operating Systems

Step Instruction Detail
a. Run mount or blkid to find the mount point of the
USB drive.
b. Run umount to unmount the USB drive.
c. Run gunzip to unzip the disk image. gunzip –d ~/Downloads/caelus.img.gz
d. Run dd to write the disk image to the USB flash drive In this example USB flash has been mounted as /dev/sdb.
(Warning: This action will overwrite any disk specified. sudo dd if=~/Downloads/caelus.img of=/dev/sdb bs=1M
Ensure you have selected the correct USB disk before && sync
continuing).
Example:
Go to step 6 below. sudo dd if=./caelus.img of=/dev/sdb bs=1M
256+1 records in
256+1 records out
268435456 bytes transferred in 24.221118 secs
(11082703 bytes/sec)

6. After creating the USB boot disk, make sure it is connected it to the computer where the configuration files (step 3)
were saved.
7. Verify that you see the CONFIG partition of the USB drive as a normal USB partition.
8. Move the saved Switch Config and Hosts Config archive files onto the CONFIG partition of the USB drive.
 FNA: USB Based Software Installation Guide | 7

Chapter 3: IP Address Configuration

This section describes how to configure the IP settings of the servers in the FNA cluster. Most
servers can be configured using the Basic Configuration steps. If needed, Manual Configuration
steps are provided.

3.1 Basic Configuration

1. Connect the flash drive containing the USB boot disk (see Chapter 2: USB Boot Disk Creation)
to the FNA server (see Figure 2). If this is the first server being reconfigured, connect it to the
top-most server fna001 (see Figure 3).
Note: You need to perform configuration for all servers in the FNA cluster.

Figure 2: USB Port Locations on FNA Servers

Figure 3: High-Level FNA System Overview

2. Reboot the server while the USB flash drive is connected. If the server does not automatically boot from the USB
flash drive, access the BIOS settings and set the USB drive as the first boot option, then reboot.
3. Connect a VGA monitor to the server’s VGA port and a keyboard to a USB port.
Note: When the operating system has loaded, you will automatically be logged in as root.
4. Verify the settings displayed on the VGA monitor:
a. Listing is correct. If the listing is correct (see Figure 4), go to 3.3 Next Steps. If the listing is incorrect, go to 3.2
Manual Configuration.
 FNA: USB Based Software Installation Guide | 8

Figure 4: Configuration Confirmation

b. Configuration file was not found. If you receive a message that the configuration file was not found (see Figure
5), repeat the steps in creating the CONFIG partition on the USB drive (see steps 7 and 8 in Chapter 2: USB Boot
Disk Creation). Restart the server with the USB flash drive connected. If the listing is still incorrect, use the steps
in 3.2 Manual Configuration.

Figure 5: Server Configuration File Not Found

3.2 Manual Configuration

1. Begin manual configuration by following either step a or b:
a. Type ./config_ipaddress.sh
b. Type ./c and press [TAB]
2. Enter the new IPv4 address as prompted, for example, 192.168.1.1 (see Figure 6) and select < OK > and press
[ENTER]. (This information is available from the CONFIG partition of the USB flash drive. The CONFIG partition
contains n server configuration files (where n is the number of servers in the FNA cluster - each filename is the
server’s serial number).

Figure 6: IPv4 Address Configuration Prompt

3. Enter the new IPv4 netmask as prompted (see Figure 7) and select < OK > and press [ENTER]. The netmask needs
to be in a dot separated format. Most commonly used values are:
/24 – 255.255.255.0
/25 – 255.255.255.128
/26 – 255.255.255.192
/27 – 255.255.255.224
 FNA: USB Based Software Installation Guide | 9

Figure 7: IPv4 Netmask Configuration Prompt

4. Configure the IPv4 address for the default gateway as prompted, for example, 10.10.10.100 (see Figure 8) and
select < OK > and press [ENTER].

Figure 8: IPv4 Default Route Configuration Prompt

5. If you are prompted to confirm that the configuration is correct (see Figure 9) and the address is acceptable, select
< OK > and press [ENTER]. You then receive the configuration confirmation (see Figure 4) and can go to 3.3 Next
Steps. If the address needs to change, select < No > and press [ENTER], then repeat the steps in 3.2 Manual
Configuration.

Figure 9: Confirm Configuration

6. If you are prompted that the address is already configured (see Figure 10) it may be due to configuring the address
through a different method, such as DHCP. If the address is acceptable, select < No > and press [ENTER]. You then
receive the configuration confirmation (see Figure 4) and can go to 3.3 Next Steps. If the address needs to change,
select < No > and press [ENTER], then repeat the steps in 3.2 Manual Configuration.

Figure 10: Address Already Configured

3.3 Next Steps

1. Repeat the basic or manual configuration steps (see 3.1 Basic Configuration or 3.2 Manual Configuration) for all
cluster servers.
2. When all servers are configured, configure the switch as described in Chapter 4: Configure the FNA Cluster Switch.
 FNA: USB Based Software Installation Guide | 10

Chapter 4: Configure the FNA Cluster Switch

This section describes how to use the USB boot disk created as described in Chapter 2: USB Boot
Disk Creation to configure the FNA cluster switch.

4.1 Switch Configuration

1. Select any of the servers with successfully reconfigured IP settings (as described in Chapter 3:
IP Address Configuration).
2. Ensure that the server is powered on and connected to the FNA cluster switch.
3. Connect a VGA monitor to the server’s VGA port and a keyboard to a USB port.
4. Run the configuration script for the switch: ./switch_configuration.sh (see Figure 11).

Figure 11: Switch Configuration Command

5. When you receive the switch configuration prompt (see Figure 12) select < Yes > and press [ENTER].

Figure 12: Switch Configuration Confirmation

6. If you are prompted that the server is not connected (see Figure 13), check the connections of both the server and its
port (it should be the correct port on the server’s network card). Then select < OK > and press [ENTER].

Figure 13: Server Not Connected

7. If you are prompted that the switch configuration file was not found on the CONFIG partition (see Figure 14), select <
OK > and press [ENTER]. Then follow the steps in Chapter 2: USB Boot Disk Creation and make sure you moved
the switch configuration file to the CONFIG partition (look for a file name beginning with JPE, FOC, SSZ, or HSH).
Restart the server with the USB flash drive connected.

Figure 14: Switch Configuration Not Found on CONFIG Partition

 FNA: USB Based Software Installation Guide | 11

8. Select < OK > and press [ENTER] at the configuration process prompt (see Figure 15) to start the configuration
procedure.

Figure 15: Configuration Process Prompt

9. Contact the FNA Ops team ([email protected]), providing them with the cluster name.
10. Wait up to 30 minutes. During the process, the FNA team will monitor the state of the switch. Echo requests will be
sent to predefined IP addresses and a response will be made when it is configured (see Figure 16).

Figure 16: Switch Configuration Monitoring

11. Verify whether switch configuration is complete by following either step a, b or c:

a. If you receive a prompt that switch configuration was successful (see Figure 17), select < OK > and press
[ENTER].

Figure 17: Successful Switch Configuration

b. If you receive a prompt that the switch serial number was not found on the CONFIG partition (see Figure 18), note
the number listed in the prompt and contact FNA Operations at [email protected].

Figure 18: Switch Serial Number Not Found on CONFIG Partition

 FNA: USB Based Software Installation Guide | 12

c. If no prompts at all appear (such as Figure 17 or Figure 18) after 30 minutes, press [Ctrl + C] to end the current
flow and follow the procedures as described in Chapter 5: Re-Image the Servers to BSD.

4.2 Next Step

1. Re-image all cluster servers as described in Chapter 5: Re-Image the Servers to BSD.
 FNA: USB Based Software Installation Guide | 13

Chapter 5: Re-Image the Servers to BSD

This section describes how to use the USB boot disk to re-image the FNA cluster servers.
1. Connect the flash drive containing the USB boot disk (see Chapter 2: USB Boot Disk Creation)
to the FNA server (see Figure 19).

Figure 19: USB Port Locations on FNA Servers

2. Ensure that the server is powered on and connected to the FNA cluster switch.
3. Connect a VGA monitor to the server’s VGA port and a keyboard to a USB port.
4. Run the re-image script for the switch: ./local_image.sh command.
5. Verify that re-imaging was successful (see Figure 20).

Figure 20: Server Re-Image Command

6. If re-imaging fails you may receive a prompt that the system disk was not found (see Figure 21).

Figure 21: System Disk Not Found

7. If you need to shut down the server after the re-image process (for example, to install it somewhere else), run the
poweroff command. Otherwise, powering off is not necessary.
8. Repeat the preceding steps for all cluster servers that need to be re-imaged.
 FNA: USB Based Software Installation Guide | 14

Chapter 6: Troubleshooting
If any issues were encountered during the creation of the USB boot disk or IP configuration, use the
following methods for general troubleshooting.
FNA Operations can be reached at [email protected].

Operating System Boot Issues

1. Take a picture of the screen where the error has occurred.
2. Send this image with brief description of the issue to FNA Operations.

Network Connectivity Test

If a network connectivity test is requested after you contact FNA Operations, follow these steps:
1. Ping the default gateway IP address.
2. If the ping was successful (minimal packet loss), ping any known working address on the Internet (for example,
Facebook.com).
3. If the ping was unsuccessful, troubleshoot for network connectivity (per next method).

Network Connectivity Checks

1. Ensure all equipment in the uplink / downlink path is powered on.
2. Ensure that the network cables between each server in the FNA cluster are properly connected to the FNA cluster
switch.
3. Ensure that the SFP+ optical transceivers are properly seated in the switch uplink ports.
4. Ensure that the optical cables in the path between the FNA cluster and the facility router have a good connection.
5. Ensure that the signal strength (light level) for the FNA cluster is between -2 and -7 dB.

Switch Configuration Issues

1. If the switch has not been configured, press [Ctrl + C] to end the current flow.
2. Take a picture of the output in this log:
cat /var/log/nginx/access.log
3. Send the pictures with a brief description of the issue and its duration to FNA Operations.