FNA Edge Network Appliance

OPERATIONS GUIDE

Apr 2022
Copyright & Trademarks
© 2022 Meta. All rights reserved.

FNA Operations Guide | 2

Table of Contents
COPYRIGHT & TRADEMARKS ........................................................................................................................................... 2
Equipment ........................................................................................................................................................................................................ 5

1. FNA NEW CLUSTER KIT (WITH SWITCH) ................................................................................................................. 5

2. FNA UPGRADE KIT .............................................................................................................................................. 6
3. HARDWARE PROVIDED BY ISP ............................................................................................................................... 7
4. OPTICAL LIGHT LEVELS ........................................................................................................................................ 8
5. EQUIPMENT LED VERIFICATION............................................................................................................................. 9
6. AC/DC POWER AND TEMPERATURE REQUIREMENTS............................................................................................... 11
7. HARDWARE SPECIFICATIONS ............................................................................................................................... 13
Adding to FNA Cluster Capacity ........................................................................................................................................................ 16

8. GROWTH PATH ................................................................................................................................................. 16

9. UPGRADES WITH ARISTA SWITCH CLUSTERS.......................................................................................................... 17
10. ADDING PORTS TO INCREASE UPLINK CAPACITY .................................................................................................. 17
11. REFRESHING CLUSTERS .................................................................................................................................. 17
Planned Maintenance Procedures .................................................................................................................................................... 18

12. NOTIFY META OF PLANNED MAINTENANCE ...................................................................................................... 18

13. DRAINING NETWORK TRAFFIC ......................................................................................................................... 19
14. SHUTTING DOWN THE FNA ........................................................................................................................... 19
15. MONITORING AND AUTOMATED DRAINS ........................................................................................................... 19
Updating Network Configuration ..................................................................................................................................................... 20

16. UPDATING THE IP CONFIGURATION OF THE FNA ............................................................................................... 20

17. CHANGING THE AS NUMBER IN THE BGP SESSION ............................................................................................. 20
18. UPDATING PREFIX LIMITS ................................................................................................................................ 20
19. USING BGP COMMUNITIES ............................................................................................................................. 20

FNA Operations Guide | 3

Introduction

Thank you for choosing to install the FNA Edge Network Appliance (FNA)!
FNA is Meta’s content caching program. FNA provides Internet Service Providers (ISPs) with an efficient
means of delivering static Meta content from within their network. Upon deployment, an ISP will offload a
significant amount of Meta content from its backbone network and vastly improve the Meta user
experience.

An FNA cluster consists of a Top-of-Rack (ToR) switch and from two to twenty servers. The hardware is
suitable for deployment in data centers, colocation facilities, and outside plant environments (the industry
standard 19-inch form factor allows integration into most network environments).

This document provides hardware specifications, instructions on adding cluster capacity, and maintenance
instructions.

Frequently Asked Questions

You can also review a list of Frequently Asked Questions, available in the NPP portal https://fb.me/npp, at
the bottom of the Support section.

Some questions you can find in the FAQ section:

FNA Operations Guide | 4

Equipment
1. FNA New Cluster kit (with switch)

For all shipments, ensure that you have received all of the hardware required. If any equipment is
missing, contact your installation vendor for assistance.

FNA Provided Hardware – New Cluster kit

FNA Operations Guide | 5

 QTY Item Dimensions

(1) Arista DCS-7060CX-32S-R-DC Switch 1 RU

(between 2 to 20) Hewlett Packard ProLiant Server DL380p G10 2 RU

(1 per server) 4-Post rail kit N/A

(2 per server) AC Power Cord (only if AC power required) 2 Meters

(1 per server) Network Cable 3 Meters

(1) HP ProLiant DL380 G10 server user manual1 N/A

(1) Switch user manual N/A

(1 per server) For 10G uplink, SFP+ optical transceiver, LR or SR2 N/A

(1 per SFP+) For 10G uplink, QSFP to SFP+ adapter2 N/A

(2) For 100G uplink, QSFP optical transceiver, LR4 or SR42 N/A
1
Model numbers vary per cluster.

2
Uplink capacity is selected for every order, either with 10G or 100G ports. Optical transceivers
will be sent according to the port type selected. Meta provides transceivers only for the FNA, while
optics for the ISP devices must be provided by the ISP.

Table 1: FNA Cluster Provided Hardware

2. FNA Upgrade Kit

The following table shows equipment that is included in an FNA order that is used for adding additional
servers to a cluster. This upgrade kit does not include a switch. However, the order includes optical
transceivers for additional uplink capacity. Use the existing switch at your FNA deployment.

The quantity of servers you receive depends on your current and forecasted Meta traffic volume and
are shipped in groups of two. The equipment will arrive at your facility in one shipment. FNA’s
installation vendor will contact you with shipping information.

FNA Operations Guide | 6

 FNA Provided Hardware -Upgrade kit

QTY Item Dimensions

(between 2 to 16) Hewlett Packard ProLiant DL380p G10 2 RU1

(1 per server) 4-Post rail kit N/A

(2 per server) AC Power Cord (only if AC power required) 2 Meters

(1 per server) Network cable 3 Meters

(1) HP ProLiant DL380p G10 server user manual1 N/A

(1) Text configuration file N/A

(1 per server) For 10G uplink, SFP+ optical transceiver, LR or SR2 N/A

(1 per SFP+) For 10G uplink, QSFP to SFP+ adapter2 N/A

(2) For 100G uplink, QSFP optical transceiver, LR4 or SR42 N/A
1
Model numbers vary per cluster.

2
Uplink capacity is selected for every order, either with 10G or 100G ports. Optical transceivers will be
sent according to the port type selected. Meta provides transceivers only for the FNA, while the optics
for the ISP devices must be provided by the ISP.

Table 2: FNA Cluster Provided Hardware

3. Hardware Provided by ISP

The following table shows additional equipment required for the FNA installation that is not provided
by Meta.

FNA Equipment Required But Not Included

QTY Item

(1 per uplink port) Fiber patch cables from Arista DCS-7060CX-32S-R-DC Switch to ISP router

As needed QSFP / SFP from router side and rack

FNA Operations Guide | 7

 1
The FNA accepts 10G SR and LR, and 100G SR4 and LR4 optical transceivers. Meta provides
the most optimal optical transceiver type, corresponding to the type you currently use to
connect your routers. FNA does not support 40GE interfaces for the uplink.

2
Fiber connectivity on the FNA side is the Lucent Connector (LC) type. Fiber connectivity on the
router side can be LC-LC, LC-SC, (Lucent Connector – Subscriber Connector), etc.

Table 3: FNA Equipment Required But Not Included

4. Optical Light Levels

Check the optical light levels to ensure that the cluster has adequate signal strength. Light levels need
to be between -2 dB and -7 dB.

In cases where light levels are outside the normal range (i.e., higher than -2dB or lower than -7dB), see
Table 4: Signal Level Troubleshooting to resolve the issue.

Signal Level Troubleshooting

Light Level Possible Cause(s) Possible Solution(s)

(dB)

Above -2 The signal is too strong. This could be Fix with your normal operating procedures
Caused by many reasons. For strong signals.

-2 to -7 None (Normal Operating Range) None (Normal Operating Range)

• Disconnect, clean, and reconnect

-8 to -20 • A degraded fiber optic cable is the fiber optic cable(s) and/or SFP+
installed. optical transceiver(s).
• A degraded SFP+ optical • If this does not resolve the issue,
transceiver is installed. either replace
• A fiber optic cable has a dirty • TX/RX hardware or solve with your
connection. normal fiber replacement
procedures.

• A fiber optic cable is bent. • Inspect the fiber optic cable(s)

• Network connectivity has not along its length. Gently roll the fiber
been established. if you observe bends or kinks.

-21 to -40 • All cables are connected to their • Check that the port connection is
proper ports, but the ports are enabled.

FNA Operations Guide | 8

 disabled.
• Ensure the cables are connected to
• All ports are enabled, but a the proper port interface. See
cable is connected to the Chapter 3: Adding to FNA Cluster
incorrect port. Capacity for connectivity
recommendations.

Table 4: Signal Level Troubleshooting

5. Equipment LED Verification

In cases where light levels are outside the normal range (i.e., higher than -2dB or lower than -7dB), see
Table 4: Signal Level Troubleshooting to resolve the issue.

Figure 1: LED Verification

Each server and switch have LEDs that can be used to diagnose the FNA for issues. The following table
describes the behavior and equipment status indicated by these LEDs.

Arista Switch (7000 Series)

Power Supply On, Steady Green The power supply is on.

On, Steady Green The fan is powered on.

Single fan is removed or malfunctioning. It is also orange
On, Steady Orange when a PSU is completely removed or has a stuck fan
rotor.
Fan
Two or more fans (any combination of fans or PSU fans)
On, Steady Red are disconnected or malfunctioning. The switch will
automatically execute a “graceful shutdown” shortly.
On, Steady Green The system is operating normally.

FNA Operations Guide | 9

 On, Steady Blue The locator function is active
System Status Two or more fans (any combination of fan modules or
On, Steady Orange PSU fans) are disconnected or malfunctioning. The switch
will automatically execute a “graceful shutdown” shortly.

ProLiant DL380 - Power Supply Unit

LED Name Behavior Status

Off The server is powered off or the power supply has failed.
Power LED
On, Steady Green The power supply is on.

ProLiant DL380 - Network Interface Controller

Activity (ACT) On, Blinking Green There is network activity.

Link (LNK) On, Steady Green A connection exists between the server and the network.

ProLiant DL380 - NVMe SSD LED definitions

LED Name Behavior Status

Solid blue The drive is being identified by a host application.

Locate (1) The drive carrier firmware is being updated or requires na
Flashing blue
update.

Rotating green Drive activity

Activity ring (2)
Off No drive activity

FNA Operations Guide | 10

 Solid green The drive is a member of one or more logical drives.

The drive is doing one of the following:

Rebuilding
Performing a RAID migration
Flashing green Performing a stripe size migration
Performing a capacity expansion
Performing a logical drive extension
Drive status (3)
Erasing

Flashing amber The drive is a member of one or more logical drives and
/ green predicts the drive will fail.

Flashing amber The drive is not configured and predicts the drive will fail.

Solid amber The drive has failed.

Off The drive is not configured by a RAID controller.

Do not remove the drive. The drive must be ejected from
Solid white
the PCIe bus prior to removal.
Do not remove
(4) Flashing white The drive ejection request is pending.
Off The drive has been ejected.
Do not remove the drive. The drive must be ejected from
Solid green
the PCIe bus prior to removal.
Power (5) Flashing green The drive ejection request is pending.
Off The drive has been ejected.

Table 5: Equipment LED Behavior

6. AC/DC Power and Temperature Requirements

Parameter Specification

Power Consumption Total

Power drain for FNA cluster Will depend on number of servers. Typical: ~300W per server and ~200W
for the Arista switch. Then for example for a 4-server FNA cluster, typical
drain would be ~1.4kW

ProLiant DL380 500W Flexslot Platinum AC Power Supply

FNA Operations Guide | 11

 5.6 A @ 100 V
Input Current (by PSU) 2.7 A @ 200 V
Steady State Power 500 W @ 100 V to 240 V

Frequency 50-60 Hz

Dual Path Power Required

Dual PDUs Required

Operating Temperature
Range 10°C to 35°C

C13-C14 AC Power Cord

Cordset Rating 10 A @ 250 V

Table 6: AC Power and Temperature Requirements

Parameter Specification

HP 800W Flexslot DC Power Supply

Input Voltage Range (VDC) 40-72 VDC

Nominal Input Voltage (VDC) 40/48/72 VDC

Maximum Rated Output Wattage Rating (W)= 800 W @ 40 /48/72 VDC

22.1A @ 40 VDC
Nominal Input Current (A) 18.2A @ 48 VDC
12 A @ 72 VDC
874W @ 40 VDC
Maximum Rated Input Wattage Rating (W) 865W @ 48 VDC
854W @ 72 VDC
2983 BTU-Hr @ 40 VDC
Maximum British Thermal Unit Rating (BTU-Hr) 2951 BTU-Hr @ 48 VDC
2912 BTU-Hr @ 72 VDC

Operating Temperature Range 10°C to 35°C

Arista DCS-7060CX-32S-R Switch

Table 7: DC Power and Temperature Requirements

FNA Operations Guide | 12

 Power Supply Specifications

Power Supply PWR-500AC PWR-500DC

Input Voltage 100-240AC 40-72V DC

13.1 – 7.3A
Typical Input Current 6.3 – 2.3A
11A at -48V

Input Frequency 50/60Hz DC

Input Connector IEC 320-C13 AWG #16-#12

Efficiency (Typical) 93% Platinum 90%

Compatibility 7060SX2-48YC6, 7060CX-32S, 7060CX2-32S

Note: Not supported on 7260QX-64 QSFP+ ports

Environmental Characteristics

Opening Temperature 0-40oC (32 to 104oF)

Storage Temperature -40 to 70oC (-40 to 158oF)

Relative Humidity 5 to 95%

Operating Altitude 0 to 10,000 ft, (0-3,000m)

Table 8: AC/DC Power and Temperature Requirements

7. Hardware Specifications

Refer to the specifications for the equipment included in your FNA kit (this can vary according to when the
order was placed).

Parameter Specification
Hewlett Packard ProLiant DL380p G10
Dimensions (Height x Width x Depth) 8.73 x 44.55 x 73.02 cm (3.44 x 17.54 x 28.75 in)
Weight (Approximate) 14.9 kg (32 lb)
Arista DCS-7060CX-32S-R Switch
Dimensions (Height x Width x Depth) 8.73 x 44.55 x 73.02 cm (1.8 x 19 x 16 in)
Weight (Approximate) 9.5 kg (21 lb)

FNA Operations Guide | 13

 C13 – C14 AC Power Cord
Length 4.4 x 40.6 x 44.5 cm (1.75 x 16 x 19 in)
Plug 8.6 kg (19 lb)
Connector
10G SFP+ Cable, Passive 2 Meters
American Wire Gauge (AWG) SS10A
Length HS10S

SFP+ Optical Transceiver

Dimensions (Height x Width x Depth) 30
Weight (Approximate) 3 Meters

Feature Description

Hewlett Packard ProLiant DL380 – Electrical Specifications

100 to 120 V ac
Rated line voltage
200 to 240 V ac
Maximum
For 800W DC power supply
2983 BTU/hr (At 40V DC)
2951 BTU/hr (At 48V DC)
BTU rating 2912 BTU/hr (At 72V DC)
For 500W AC power supply
1904 BTU/hr (At 100 V AC)
1837 BTU/hr (At 200 V AC)
1832 BTU/hr (At 240 V AC)

Hewlett Packard ProLiant DL380 – Environmental Specifications

• 10° to 35°C (50° to 95°F) at sea level with an altitude derating

of 1.0°Cper every 305 m (1.8°F per every 1000 ft) above sea
level to a maximum of 3050 m (10,000 ft), no direct sustained
sunlight
Standard operating temperature •
• Maximum rate of change is 20°C/hr (36°F/hr)
•
• The upper limit and rate of change may be limited by the type
and number of options installed
• Operating
Extended ambient • 5° to 10°C (41° to 50°F) and 35° to 40°C (95° to 104°F) at sea
level with an altitude derating of 1.0°C per every 175 m (1.8°F

FNA Operations Guide | 14

 per every 574 ft) above 900 m (2953 ft) to a maximum of
3050 m (10,000 ft).

• 40° to 45°C (104° to 113°F) at sea level with an altitude

derating of 1.0°C per every 125 m (1.8°F per every 410 ft)
above 900 m (2953 ft) to a maximum of 3050 m (10,000 ft).

• Non-operating
• -30° to 60°C (-22° to 140°F)
Maximum rate of change is 20°C/hr (36°F/hr)
• Operating
• 8% to 90% - Rh
• 28°C (82°F) maximum wet bulb temperature
• Non-condensing
Relative humidity (Rh) •
• Non-operating
• 5% to 95% - Rh
• 38.7°C (101.7°F) maximum wet bulb temperature
Non-condensing
• Operating
• 3050 m (10,000 ft)
• This value may be limited by the type and number of options
installed
• Maximum allowable altitude change rate is 457 m/min (1500
Altitude ft/min)
•
• Non-operating
• 9144 m (30,000 ft)
Maximum allowable altitude change rate is 457 m/min (1500
ft/min)
• Idle
• LWAd
• 4.7 B entry
• 4.9 B base
• 4.8 B perf
• LpAm
• 31 dBA entry
• 34 dBA base
• 33 dBA perf
Acoustic noise1 •
• Operating
• LWAd
• 4.7 B entry
• 4.9 B base
• 4.8 B perf
• LpAm
• 31 dBA entry
• 34 dBA base
33 dBA perf

FNA Operations Guide | 15

 Note: Acoustics levels presented here are generated by the test configuration only. Acoustics levels will
vary depending on system configuration. Values are subject to change without notification and are for
reference only

Product conformance to cited product specifications is based on sample (Type) testing, evaluation, or
assessment. This product or family of products is eligible to bear the appropriate compliance logos and
statements

The listed sound levels apply to standard shipping configurations. Additional options may result in
increased sound levels

Table 9: Hardware Dimensions and Weight

Adding to FNA Cluster Capacity

This section describes how to add additional capacity to your original FNA cluster and provides a suggested
growth path reference.

Warning: This procedure requires updating the Link Aggregation (LAG) configuration. Updating the LAG
may cause service interruption. If this is the case, it is recommended to drain network traffic first. See
section 13. Draining Network Traffic.

8. Growth Path

This section provides a reference for best practices regarding augmentations. The growth path optimizes
for deployment failover scenarios by reducing the quantity of Single Points Of Failure (SPOF).

Figure 2: Recommended FNA Growth Path

Note: When building an FNA cluster beyond eight servers, it is best practice to install the additional servers
in a new rack and maintain equal size clusters (as depicted in growth path configuration 3 and 4).

FNA Operations Guide | 16

 9. Upgrades with Arista Switch Clusters

Upgrading a (4 or more) Server Cluster up to a (20) Server Cluster

Figure 3: Arista Switch Port Reference

i. Make sure all of the hardware is available. Refer to sections 1 and 2.

ii. Rack the additional servers, connect the peripheral cables (power and network), and power on
the servers. Refer to the document Quick Start Guide from the Documents section of NPP

iii. For clusters over (4) servers, follow the port assignments as illustrated. Other port maps are
available in the Quick Start Guide

10. Adding ports to increase uplink capacity

No new servers are added, only connecting 10G or 100G ports).

i. Install QSFP/Optic if necessary

ii. Connect the new port in the FNA switch to the ISP router as instructed in figure 3

iii. Update the ISP router configuration to add the new port to the LACP interface.

FNA switch is pre-configured, therefore, this process usually does not require intervention from Meta.
however, to upgrade from 10G to 100G please create a Support ticket in the NPP asking for assistance.

Note: 10G to 100G upgrade is only applicable for FNA caches with an Arista switch. Older FNA caches with
Cisco switches only support 10G ports.

11. Refreshing clusters

After the equipment has been in use for a few years, it might be necessary to replace the switch and/or

FNA Operations Guide | 17

servers for new devices. This process is known as Refresh, and just like with the initial deployment, Meta
and its partners will take care of shipping the new devices to you, as well as removing the old ones. When
an FNA cluster is due for a refresh, we will reach out to the ISP to coordinate it.

For refresh please consider the following points.

i. We request you Not To power down or replace any of the hardware as we must access the
servers to prepare the cluster for refresh.

ii. The decommissioning of the old servers may take up to 6hrs so request you to arrange site tech
accordingly

If you are going through this process, please review the FNA Refresh Guide, available in the NPP portal
Documents section.

Planned Maintenance Procedures

Use the Network Partner Portal (NPP) as the central hub for ordering FNA kits, reviewing deployment
information and metrics, scheduling maintenance, and troubleshooting any issues you may encounter with
your deployment.

12. Notify Meta of Planned Maintenance

When performing either singular or recurring maintenance jobs, create a task in the NPP https://fb.me/npp
within the FNA Maintenance queue.

If you don’t have access to the NPP, you may instead provide the following information in an email:

Item Detail

To [email protected], [email protected]

Subject [FNA] [FNA_CLUSTER] [ISP_NAME] Maintenance Notification1

[ISP_NAME]1 Maintenance Notification

FNA clusters affected:
Maintenance Ticket #:
Start Date:
End Date:
Body
Reason for maintenance:
Contact information:

FNA Operations Guide | 18

 Meta Maintenance Notification
FNA clusters affected: fsnc1c01
fsnc2c02
Maintenance Ticket #: 4103
Start Date: APRIL 27, 2018 (04:00 GMT)
Body Example End Date: APRIL 27, 2018 (12:00 GMT)
Reason for maintenance: cabling upgrades
Contact information: [email protected]

Note: Update items between brackets, these are unique to your deployment.

13. Draining Network Traffic

Before powering down the FNA cluster for maintenance, all network traffic must be properly drained. This
procedure describes how to properly drain traffic from the cluster.

Important: This procedure requires withdrawing prefixes. It is important that the Border Gateway Protocol
(BGP) peering session is maintained while withdrawing prefixes.

i. Withdraw prefixes. While maintaining the BGP peering session, begin withdrawing BGP prefixes.

ii. Traffic will begin to drain. Traffic will fully drain in less than one hour.

14. Shutting Down the FNA

You should shut down the FNA only if it is absolutely required and traffic has been properly drained from
the system (see 13. Draining Network Traffic). This should be a rare occurrence. To shut down the FNA,
follow the procedure outlined below or create a task in the NPP:

i. Connect a monitor and keyboard to the server you want to power off.
ii. Use the arrow keys and navigate to the Shut Down text.
iii. Press the [Enter] key. The FNA will power off.

15. Monitoring and Automated Drains

Meta regularly monitors internet reachability from various simulated endpoints. When reachability issues
are detected on these Virtual IPs (VIPs), Meta may drain traffic from an FNA cluster. This measure ensures
that Meta maintains a high quality of service for the Meta platform.

FNA Operations Guide | 19

Updating Network Configuration

16. Updating the IP configuration of the FNA

If you need to update the configuration of the IP addresses used by the FNA cluster, such as the P2P uplink
addresses, the BGP neighbor address, or even the /26 IPv4 or /64 IPv6 prefixes reserved for caching, please
create a Support ticket in the NPP asking for assistance with the new IP details.

Important note: Changing the FNA caching prefixes (the /26 IPv4 or /64 IPv6) takes between 24 to 48hs to
be in effect in our systems. Please note that if you are requesting this change, the cache will be drained with
no traffic between 24 to 48hs until traffic returns to normal levels. On the other hand, requesting changes
to P2P or BGP IP addresses as well as updates to your ASN number usually only take 2hrs or less to be in
effect.

17. Changing the AS number in the BGP session

The FNA clusters have a BGP sessions between Meta’s AS63293 and the ISP ASN. If you are going through
the process of updating the AS number you use to set up BGP sessions, you can request a change of
configuration.

To do so,
i. First make sure that the ASN has been added to your organization profile in NPP. You can check
this in the Settings section (cog wheel icon). You can find more details about this and about how to
request the addition of a new ASN in the Portal User Guide, available in the NPP portal Documents
section
ii. Once the new ASN is already available, create a new Support ticket in NPP requesting the AS
number change.

18. Updating prefix limits

We hold a limit on the number of IPv4 / IPv6 advertisements allowed to your FNA:
i. For IPv4 we allow a maximum of 50k
ii. For IPv6 we allow a maximum of 8k

As this may be indicative of prefix leakage, we will inform you via a Direct Support ticket in the Network
Partner Portal when you have reached 90% of these thresholds. Should you cross either of these
thresholds, traffic will be drained from your cluster. To prevent this, please ensure your BGP
advertisements are below the defined thresholds. Should you have a reason to cross this, please contact
[email protected] or open a ticket in the NPP, Support section.

19. Using BGP communities

Meta maintains interconnection points of presence for networks around the globe and strives to deliver
traffic as close to end users as possible for optimal performance. This mapping generally includes latency as
a decision factor. However, in certain circumstances peer networks may wish to influence path selection
away from what Meta is seeing as the best path, or proactively determine redundant path selection. To this
end, Meta supports the exchange of BGP community tags on prefix advertisements to influence specific

FNA Operations Guide | 20

egress paths. Under all circumstances, such community tags are considered advisory and may be
overridden by internal Meta logic, policy, or configuration considerations.

You can find out more in the BGP Community Signaling Guide, available in the NPP portal Documents
section

FNA Operations Guide | 21

FNA Operations Guide | 22