Queueing in the Linux Network Stack

Sep 23, 2013 By Dan Siemon (/users/dan-siemon) in

46

Starvation and Latency Despite its necessity and benefits, the queue between the IP stack and the hardware introduces two problems: starvation and latency. If the NIC driver wakes to pull packets off of the queue for transmission and the queue is empty, the hardware will miss a transmission opportunity, thereby reducing the throughput of the system. This is referred to as starvation. Note that an empty queue when the system does not have anything to transmit is not starvationthis is normal. The complication associated with avoiding starvation is that the IP stack that is filling the queue and the hardware driver draining the queue run asynchronously. Worse, the duration between fill or drain events varies with the load on the system and external conditions, such as the network interface's physical medium. For example, on a busy system, the IP stack will get fewer opportunities to add packets to the queue, which increases the chances that the hardware will drain the queue before more packets are queued. For this reason, it is advantageous to have a very large queue to reduce the probability of starvation and ensure high throughput. Although a large queue is necessary for a busy system to maintain high throughput, it has the downside of allowing for the introduction of a large amount of latency. Figure 4 shows a driver queue that is almost full with TCP segments for a single highbandwidth, bulk traffic flow (blue). Queued last is a packet from a VoIP or gaming flow (yellow). Interactive applications like VoIP or gaming typically emit small packets at fixed intervals that are latency-sensitive, while a high-bandwidth data transfer generates a higher packet rate and larger packets. This higher packet rate can fill the queue between interactive packets, causing the transmission of the interactive packet to be delayed.

Figure 4. Interactive Packet (Yellow) behind Bulk Flow Packets (Blue)

To illustrate this behaviour further, consider a scenario based on the following assumptions: A network interface that is capable of transmitting at 5 Mbit/sec or 5,000,000 bits/sec. Each packet from the bulk flow is 1,500 bytes or 12,000 bits. Each packet from the interactive flow is 500 bytes. The depth of the queue is 128 descriptors. There are 127 bulk data packets and one interactive packet queued last. Given the above assumptions, the time required to drain the 127 bulk packets and create a transmission opportunity for the interactive packet is (127 * 12,000) / 5,000,000 = 0.304 seconds (304 milliseconds for those who think of latency in terms of ping results). This amount of latency is well beyond what is acceptable for interactive applications, and this does not even represent the complete round-trip timeit is only the time required to transmit the packets queued before the interactive one. As described earlier, the size of the packets in the driver queue can be larger than 1,500 bytes, if TSO, UFO or GSO are enabled. This makes the latency problem correspondingly worse. Large latencies introduced by over-sized, unmanaged queues is known as Bufferbloat (http://en.wikipedia.org/wiki/Bufferbloat (http://en.wikipedia.org/wiki/Bufferbloat) ). For a more detailed explanation of this phenomenon, see the Resources for this article. As the above discussion illustrates, choosing the correct size for the driver queue is a Goldilocks problemit can't be too small, or throughput suffers; it can't be too big, or latency suffers. Byte Queue Limits (BQL) Byte Queue Limits (BQL) is a new feature in recent Linux kernels (> 3.3.0) that attempts to solve the problem of driver queue sizing automatically. This is accomplished by adding a layer that enables and disables queueing to the driver queue based on calculating the minimum queue size required to avoid starvation under the current system conditions. Recall from earlier that the smaller the amount of queued data, the lower the maximum latency experienced by queued packets. It is key to understand that the actual size of the driver queue is not changed by BQL. Rather, BQL calculates a limit of how much data (in bytes) can be queued at the current time. Any bytes over this limit must be held or dropped by the layers above the driver queue. A real-world example may help provide a sense of how much BQL affects the amount of data that can be queued. On one of the author's servers, the driver queue size defaults to 256 descriptors. Since the Ethernet MTU is 1,500 bytes, this means up to 256 * 1,500 = 384,000 bytes can be queued to the driver queue (TSO, GSO and so forth are disabled, or this would be much higher). However, the limit value calculated by BQL is 3,012 bytes. As you can see, BQL greatly constrains the amount of data that can be queued. BQL reduces network latency by limiting the amount of data in the driver queue to the minimum required to avoid starvation. It also has the important side effect of moving the point where most packets are queued from the driver queue, which is a simple FIFO, to the queueing discipline (QDisc) layer, which is capable of implementing much more complicated queueing strategies. ______________________ Dan Siemon is a longtime Linux user and former network admin who now spends most of his time doing business stuff.

Comments
Comment viewing options
Threaded list - expanded Date - newest first 50 comments per page Save settings

Select your preferred way to display the comments and click "Save settings" to activate your changes.

I just added this feed to my (/content/queueing-linux-network-stack#comment-377654) Submitted by Personal VPN Service (https://www.blacklogic.com/) (not verified) on Tue, 11/19/2013 - 01:07. I just added this feed to my bookmarks. I have to say, I very much enjoy reading your blogs. Keep it up!

safe flight academy with linux system (/content/queueing-linux-network-stack#comment-377599) Submitted by lordtn (/users/lordtn) on Sat, 11/09/2013 - 02:37. with safe flight academy (http://www.sfa.tn) : Get quality flying lessons in Tunisia (http://www.sfa.tn) using Flying Schools in Tunisia for advanced pilots and beginner pilot lessons base in linux systems

(/users/lordtn)

I found this is an (/content/queueing-linux-network-stack#comment-377562) Submitted by laurawillson (not verified) on Thu, 10/31/2013 - 10:54. I found this is an informative and interesting post so i think so it is very useful and knowledgeable. I would like to thank you for the efforts you have made in writing this article http://www.actual-braindumps.com (http://www.actual-braindumps.com/) braindumps

IT Network Support (/content/queueing-linux-network-stack#comment-377472) Submitted by IT Network Support (http://www.marnetllc.com/) (not verified) on Sat, 10/19/2013 07:36. I am really pleased to read this webpage posts which contains lots of helpful data, thanks for providing such statistics. IT Network Support (//www.marnetllc.com )

The cotton bag at (/content/queueing-linux-network-stack#comment-377448) Submitted by Anonymous (not verified) on Wed, 10/16/2013 - 21:59. The cotton bag at http://www.irisweb.co.uk (http://www.irisweb.co.uk) has a lovely and valuable property. It is feasible to recycle and reuse the cotton bag. It is speedy. You can make use of the cotton bag for numerous significant purposes.

good stuff (/content/queueing-linux-network-stack#comment-377347)

Submitted by lordtn (/users/lordtn) on Tue, 10/01/2013 - 16:01. great work! Thanks for the post and this fantastic website. tunisie annonce (http://www.tunisie-annonce.biz)
(/users/lordtn)

Comment Spam (/content/queueing-linux-network-stack#comment-377318) Submitted by the critic (not verified) on Fri, 09/27/2013 - 04:34. why is it that a supposedly professional website such als LJ can'r provide proper anti-spam measures in their comment system?

Typo in tag (/content/queueing-linux-network-stack#comment-377311) Submitted by Nigel (not verified) on Thu, 09/26/2013 - 09:56. I think this should be tagged 'Networking' not 'Netowrking' ?

It is very nice to read this (/content/queueing-linux-network-stack#comment-377310) Submitted by Biker (http://carbon650.com/) (not verified) on Thu, 09/26/2013 - 08:56. It is very nice to read this post. I appreciate your views about this. It helps us a lot. Thank you very much.

Thank you for the effort you (/content/queueing-linux-network-stack#comment-377306) Submitted by question voyance gratuite (http://www.dana-voyance.com) (not verified) on Thu, 09/26/2013 - 06:34. Thank you for the effort you have made in creating this blog, better shared information that's also one of the values of democracy ... if I can do anything to help this site I 'd be happy .. Good luck !

Thanks for the post. Good and (/content/queueing-linux-network-stack#comment-377304) Submitted by Anonymous (not verified) on Thu, 09/26/2013 - 05:26. Thanks for the post. Good and succint overview of packet queueing in Linux.

Reply to comment | Linux Journal (/content/queueing-linux-network-stack#comment-377283) Submitted by logo design (http://www.logonow.com.au) (not verified) on Tue, 09/24/2013 - 03:14. You have published a fantastic website.

Post new comment

Homepage:

Subject:

Comment: *

Allowed HTML tags: <a> <em> <strong> <cite> <code> <pre><tt> <ul> <ol> <li> <dl> <dt> <dd> <i> <b> <blockquote> Lines and paragraphs break automatically. Web page addresses and e-mail addresses turn into links automatically.

Notify me when new comments are posted All comments Preview Replies to my comment

By submitting this form, you accept the Mollom privacy policy (http://mollom.com/web-service-privacy-policy) .

White Paper

More Resources (/whitepapers)

White Paper

More Resources (/whitepapers)

Linux Management with Red Hat Satellite: Measuring Business Impact and ROI
(http://w w w .linuxjournal.com/content/dow nload-linuxmanagement-red-hat-satellite-measuring-businessimpact-and-roi)

Private PaaS for the Agile Enterprise

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality availability, security, and performance becomes an essential requirement for business success. Learn More
(http://www.linuxjournal.com/content/download-linuxmanagement-red-hat-satellite-measuring-businessimpact-and-roi) Sponsored by Red Hat

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesnt scale? In todays hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control. Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise. Learn More
(http://www.linuxjournal.com/content/download-privatepaas-agile-enterprise) Sponsored by ActiveState