CA-NFS

Paper: A. Batsakis, R. Burns, A. Kanevsky, J. Letini and T. Talpey

Slides: Joe Buck for CMPS 292 - Spring 2010

04/12/2010

Tuesday, April 13, 2010

My slides are based heavily on Batsakis’ FAST ’09 slides. Link in the last slide
 The Goal

✤ Create a framework for influencing clients in a distributed file system

in response to server load

Tuesday, April 13, 2010

 How this is new

✤ Current clients operate selfishly

✤ Greedy scheduling

✤ Throttled via TCP back-offs

✤ Each request has an associated cost

✤ memory (buffering), disk, network

Tuesday, April 13, 2010

Memory is used on both the client, which can flush cache, or on the server
Keep going until bad things happen
 This is the status quo

✤ Current distributed systems put entire responsibility on the servers

✤ Clients only look out for themselves

✤ Don’t consider server load

✤ Don’t consider other clients

Tuesday, April 13, 2010

 Solution

✤ Solution: Inform the client

✤ Be aware of the wider impact of their actions

✤ Have a metric to compare the cost

Tuesday, April 13, 2010

The clients have no knowledge of the server loads or client population.

Use cost on client vs server to make decisions.
 Solution Design

✤ CA-NFS: Congregation Aware - Network Filesystem

✤ Monitor resource usage and quantify congestion

✤ Inform clients so they can act accordingly

Tuesday, April 13, 2010

Make clients aware of other clients resource usage (cost for for further use)
Makes client aware of server’s resource usage ( and cost for further use)
 How to quantify congestion

✤ What metrics matter

✤ throughput, latency, cpu usage, memory usage

✤ How to measure

✤ Grey box, Black box, ... ?

✤ Heterogeneity causes hiccups

Tuesday, April 13, 2010

-Heterogeneous workloads, devices, machines

-How does one compare 80% cpu usage vs 100 outstanding disk IOs
 Congestion

✤ Create a single metric

✤ Congestion price = function over resource utilization

✤ Pi (ui) = Pmax x

✤ log-k competitive off-line algorithm

✤ system and workload independent

Tuesday, April 13, 2010

Metric based on B. Awerbuch, Y. Azar and S. Plotkin “Throughput-competitive online routing”

FOCS ’93
Ui: utilization of resource i
Pi, Pmax: price for resource i, max price
Ki: degredation factor
for disks, smaller K means congestion is used more quickly but for memory a large K means
congestion is felt much later
 Congestion - continued

✤ Device-specific

✤ All things being equal, client set higher than servers

Tuesday, April 13, 2010

 Resource Measurements

✤ It doesn’t matter what resources are measured

✤ The only requirement is that the usage can be expressed as a value

between 0 - 1

✤ Monitored devices can be real

✤ Network, CPU, memory, disk

✤ Or they can be virtual

✤ Read-ahead effectiveness, cache effectiveness

Tuesday, April 13, 2010

Any device or heuristic can be added at any time.

Weighting can be adjusted
 Current NFS assumptions

✤ Current NFS servers are based on false pretenses

✤ Assume that increasing server throughput translates into client

benefits

✤ All clients, and all their operations, are equally worthy

Tuesday, April 13, 2010

Client priorities can be handled out of band by QoS software

Backup, indexing are low priority and can be pre-empted
Asynchronous operations are what this project focuses on
 Client Realities

✤ All packets are not created equal

✤ Process priorities vary

✤ Synchronous vs Asynchronous operations

Tuesday, April 13, 2010

Client priorities can be handled out of band by QoS software

Backup, indexing are low priority and can be pre-empted
Asynchronous operations are what this project focuses on
 Implicit Priorities

✤ Synchronous versus Asynch operations

✤ Synchronous ops block

✤ Asynch do not block

✤ Can be time-shifted

Tuesday, April 13, 2010

Synch ops: reads, metadata

Asynch ops: write, read-ahead
 Using Metrics to Inform Clients

✤ Goal is to shift asynch, lower-priority in favor of synchronous, high-

priority requests

Tuesday, April 13, 2010

 Reverse Auction

✤ Both clients and servers encode their resource availability in costs

✤ Servers and clients exchange prices

✤ Clients use pricing to:

✤ issue more or less read-aheads

✤ defer or accelerate a write

Tuesday, April 13, 2010

Clients make their decisions based on server prices and local prices
 Accelerating Writes

✤ Clients inform servers to synch data immediately

✤ Client does not buffer the data

✤ Preserves clients cache

✤ Saves client memory

✤ Consumes server resources immediately

✤ Write-behind is bypassed

✤ Non-issue

Tuesday, April 13, 2010

Servers sync immediately

Based on the hit rate ratio of the client metric
prevents double-caching
Since this only happens during lower server loads, write-behind bypassing okay
 Deferring Writes

✤ Writes are held locally

✤ Saves server resources

✤ Consumes client memory

✤ Maybe be deferring the inevitable

Tuesday, April 13, 2010

If the server is still loaded later then the writes will have a bigger latency delay (the damn
bursting)
Typically caused by high server load
Heuristics throttle small write deferral
 Example 1
CA-NFS in Practice

memory: 80% memory: 10%

Client 1 RA eff: 10% Client 2 RA eff: 85%
hit rate: 40% hit rate: 90%

price Writes Reads price Writes Reads

85 4 20 17

memory: 65%
Server network: 20%
disk: 90%
hit rate: 40%
price Writes Reads
40 12

Tuesday, April 13, 2010

Client 2 defers writes since the cost on the server is higher than the local cost.
Also,©client 2 is seeing good cache rates so this is good.
2008 NetApp. All rights reserved. 14

14
 Example
CA-NFS 2
in Practice

memory: 50% memory: 50%

Client 1 RA eff: 10% Client 2 RA eff: 85%
hit rate: 40% hit rate: 90%

price Writes Reads price Writes Reads

64 4 60 17

memory: 65%
Server network: 20%
disk: 60%
hit rate: 40%
price Writes Reads
55 12

CA-NFS “exchanges” resource congestion

Tuesday, April 13, 2010

Client 1 has freed up some memory and client 2 is using more memory so it now starts
writing again. among clients and the server
Writes cheaper
© 2008 atrights
NetApp. All thereserved.
server than either client 15

15
 Benchmarks

✤ Fileserver test

✤ 1000s of real NFS traces

✤ many asynch operations

✤ creates, deletes, reads, writes, etc.

✤ OLTP (database benchmark)

✤ many random reads and writes

✤ mostly synchronous

Tuesday, April 13, 2010

oltp based on oracle work

 Fileserver Results
Fileserver – Results I

Tuesday, April 13, 2010

Performance improves for a single client because writes and reads are batched a bit better
© 2008 NetApp. All rights reserved. 17

17
 Fileserver
Fileserver –Results
Results II 2

Tuesday, April 13, 2010

© 2008 NetApp. All rights reserved. 18
NFS schedules write-backs early
CA-NFS smooths the graph for better utilization 18
 Fileserver
Fileserver –Results
Results II 2

Tuesday, April 13, 2010

© 2008 NetApp. All rights reserved. 18
NFS schedules write-backs early
CA-NFS smooths the graph for better utilization 18
 OLTP
OLTP -Results
Results

Tuesday, April 13, 2010

© 2008 NetApp. All rights reserved. 19

19
 Future Work

✤ Asynch scheduling proof of concept

✤ Policies for synchronous operations

✤ Graph fairness over time

✤ Implement reservations by varying prices

✤ Salaries for QoS

Tuesday, April 13, 2010

synchronous operations could stop applications after .8

Track client usage to prevent abuse
 Reflections

✤ Importance of proper economic model

✤ Clients do not always benefit from increased server throughput

✤ Server cost always increases with load

✤ Contributed a framework for performance management based on

congestion metrics

Tuesday, April 13, 2010

Client requests come at different priority, caching effectiveness

 Other Resources

✤ Original Slides: http://www.usenix.org/events/fast/tech/slides/

batsakis.pdf

✤ Video of FAST ’09 paper presentation: http://www.usenix.com/

multimedia/fast09batsakis

✤ Cost metrics are based on this paper: B. Awerbuch, Y. Azar and S. Plotkin
“Throughput-Competitive On-Line Routing” FOCS ’93

Tuesday, April 13, 2010

 The End

✤ Questions?

✤ Comments?

✤ Contact: [email protected]

Tuesday, April 13, 2010