Scaling the

Grammys®
with Drupal
Sunday, June 27, 2010
 • Nathan Haug
• "quicksketch" on d.o

Sunday, June 27, 2010

 • Nathan Haug
• "quicksketch" on d.o

Sunday, June 27, 2010

 • Nathan Haug
• "quicksketch" on d.o

Sunday, June 27, 2010

 • Nathan Haug
• "quicksketch" on d.o

Sunday, June 27, 2010

Sunday, June 27, 2010
 • The Recording Academy wanted more
"social" features and interaction.

Sunday, June 27, 2010

 • The Recording Academy wanted more
"social" features and interaction.

• Easier maintenance of site and

functionality.

Sunday, June 27, 2010

 • The Recording Academy wanted more
"social" features and interaction.

• Easier maintenance of site and

functionality.

• Single place for Nominee and Winner

history.

Sunday, June 27, 2010

 Some Numbers

Sunday, June 27, 2010

 A c c u r a te
Some^Numbers

Sunday, June 27, 2010

 A c c u r a te
Some^Numbers
• 214 million page views on Grammy
Award Show day, most over the 6 hour
window of the broadcast.

Sunday, June 27, 2010

 A c c u r a te
Some^Numbers
• 214 million page views on Grammy
Award Show day, most over the 6 hour
window of the broadcast.

• 50,000 new registered users over the

same period.

Sunday, June 27, 2010

 A c c u r a te
Some^Numbers
• 214 million page views on Grammy
Award Show day, most over the 6 hour
window of the broadcast.

• 50,000 new registered users over the

same period.

• 5% Database load during show.

Sunday, June 27, 2010

 A c c u r a te
Some^Numbers
• 214 million page views on Grammy
Award Show day, most over the 6 hour
window of the broadcast.

• 50,000 new registered users over the

same period.

• 5% Database load during show.

• 1% Web server load during show.
Sunday, June 27, 2010
 Hardware

Sunday, June 27, 2010

 Hardware

Web Servers

Sunday, June 27, 2010

 Hardware
Load Balancer

Web Servers

Sunday, June 27, 2010

 Hardware
Load Balancer

Web Servers

Sunday, June 27, 2010

 Hardware
Load Balancer

Web Servers

MySQL Server

Sunday, June 27, 2010

 Hardware
Load Balancer

Web Servers

MySQL Server

Solr Server

Sunday, June 27, 2010

 Hardware
Load Balancer

Web Servers

MySQL Server

Solr Server

NAS (files directory)

Sunday, June 27, 2010
 Hardware
Load Balancer LB Backup

Web Servers

MySQL Server MySQL Backup

Solr Server Solr Backup

NAS (files directory)

Sunday, June 27, 2010
 Hardware
Load Balancer LB Backup

Web Servers

MySQL Server MySQL Backup

Solr Server Solr Backup

NAS (files directory)

Sunday, June 27, 2010
 Hardware
Load Balancer LB Backup

Web Servers

MySQL Server MySQL Backup

Solr Server Solr Backup

NAS (files directory)

Sunday, June 27, 2010
 Hardware

Sunday, June 27, 2010

 Software

Sunday, June 27, 2010

 Software

Sunday, June 27, 2010

 Software
• Use caching mechanisms and software

Sunday, June 27, 2010

 Software
• Use caching mechanisms and software
• Configure Drupal to utilize this caching

Sunday, June 27, 2010

 Software
• Use caching mechanisms and software
• Configure Drupal to utilize this caching
• Give servers multiple roles

Sunday, June 27, 2010

 Software
• Use caching mechanisms and software
• Configure Drupal to utilize this caching
• Give servers multiple roles
• Dedicate servers for "expensive"
operations (i.e. Search)

Sunday, June 27, 2010

 Web Server

Sunday, June 27, 2010

 Web Server

• PressFlow

Sunday, June 27, 2010

 Web Server

• PressFlow
• APC Op-Code Cache for PHP

Sunday, June 27, 2010

 Web Server

• PressFlow
• APC Op-Code Cache for PHP
• Database Slave

Sunday, June 27, 2010

 Web Server

• PressFlow
• APC Op-Code Cache for PHP
• Database Slave
• Varnish in front of Apache

Sunday, June 27, 2010

 Web Server

Sunday, June 27, 2010

 Web Server

• Distribution of Drupal (100% API

Compatible with existing modules)

Sunday, June 27, 2010

 Web Server

• Distribution of Drupal (100% API

Compatible with existing modules)

• Enables use of CDNs, Reverse

Proxies, and Slave database servers

Sunday, June 27, 2010

 Web Server

• Distribution of Drupal (100% API

Compatible with existing modules)

• Enables use of CDNs, Reverse

Proxies, and Slave database servers

• Does not create sessions for

Anonymous users
Sunday, June 27, 2010
 Web Server

Sunday, June 27, 2010

 Web Server

• APC (Another PHP Cache) reduces

disk access when parsing PHP files.

Sunday, June 27, 2010

 Web Server

• APC (Another PHP Cache) reduces

disk access when parsing PHP files.

• Keeps fully parsed PHP files in

memory.

Sunday, June 27, 2010

 Web Server

Sunday, June 27, 2010

 Web Server

• SQL Slave

Sunday, June 27, 2010

 Web Server

• SQL Slave

• All "Pagered" and Limit queries hit

the slave server.

Sunday, June 27, 2010

Sunday, June 27, 2010
Sunday, June 27, 2010
Sunday, June 27, 2010
Sunday, June 27, 2010
Sunday, June 27, 2010
Sunday, June 27, 2010
Sunday, June 27, 2010
Sunday, June 27, 2010
Sunday, June 27, 2010
Sunday, June 27, 2010
Sunday, June 27, 2010
Sunday, June 27, 2010
Sunday, June 27, 2010
Sunday, June 27, 2010
 Web Server

Sunday, June 27, 2010

 Web Server

• The MOST important cache on our

infrastructure.

Sunday, June 27, 2010

 Web Server

• The MOST important cache on our

infrastructure.

• Serves up images, static files, and

anonymous pages from memory.

Sunday, June 27, 2010

 Web Server

• The MOST important cache on our

infrastructure.

• Serves up images, static files, and

anonymous pages from memory.

• Cookie-based skipping of Varnish

lets users fall through to Apache.
Sunday, June 27, 2010
 Master Database

Sunday, June 27, 2010

 Master Database

• MySQL

Sunday, June 27, 2010

 Master Database

• MySQL
• Doubles as Memcache server

Sunday, June 27, 2010

 Master Database

• MySQL
• Doubles as Memcache server
• Mirrored as a backup, NOT a Master-
Master configuration.

Sunday, June 27, 2010

 Master Database

Sunday, June 27, 2010

 Master Database

• MySQL 5.2

Sunday, June 27, 2010

 Master Database

• MySQL 5.2

• Turn on query caching in my.cnf:

Sunday, June 27, 2010

 Master Database

• MySQL 5.2

• Turn on query caching in my.cnf:

query_cache_limit = 1M
query_cache_size = 16M

Sunday, June 27, 2010

 Master Database

MEMCACHED

Sunday, June 27, 2010

 Master Database

MEMCACHED
• All Cache tables and Sessions
stored in memory.

Sunday, June 27, 2010

 Master Database

MEMCACHED
• All Cache tables and Sessions
stored in memory.

• 16 Cache tables on Grammy.com

Sunday, June 27, 2010

 Master Database

MEMCACHED
• All Cache tables and Sessions
stored in memory.

• 16 Cache tables on Grammy.com

• Anonymous pages served entirely

from memory with page cache
enabled.
Sunday, June 27, 2010
 Solr Server

Sunday, June 27, 2010

 Solr Server

• Outsources search to dedicated box.

Sunday, June 27, 2010

 Solr Server

• Outsources search to dedicated box.

• Java server. Not a PHP technology.

Sunday, June 27, 2010

 Solr Server

• Outsources search to dedicated box.

• Java server. Not a PHP technology.

• TomCat as host.

Sunday, June 27, 2010

 All Together Now
Anonymous
Pure
Anonymous w/ NO_CACHE Authenticated

MEMCA CHED

Slave

Master

Sunday, June 27, 2010

 All Together Now
Anonymous
Pure
Anonymous w/ NO_CACHE Authenticated

Peak Traffic Times

MEMCA CHED

Slave

Master

Sunday, June 27, 2010

 All Together Now
Anonymous
Pure
Anonymous w/ NO_CACHE Authenticated

Peak Traffic Times

Off-Peak

MEMCA CHED

Slave

Master

Sunday, June 27, 2010

 All Together Now
Anonymous
Pure
Anonymous w/ NO_CACHE Authenticated

Peak Traffic Times

Off-Peak

MEMCA CHED

Slave

Master

Sunday, June 27, 2010

 All Together Now
Anonymous
Pure
Anonymous w/ NO_CACHE Authenticated

Peak Traffic Times

Off-Peak

MEMCA CHED

Slave

Master

Sunday, June 27, 2010

 All Together Now
Anonymous
Pure
Anonymous w/ NO_CACHE Authenticated

Peak Traffic Times

Off-Peak

MEMCA CHED

Slave

Master

Sunday, June 27, 2010

 Drupal Integration

Sunday, June 27, 2010

 Drupal Integration
• Most all configuration in settings.php

Sunday, June 27, 2010

 Drupal Integration
• Most all configuration in settings.php
• Slave database setup

Sunday, June 27, 2010

 Drupal Integration
• Most all configuration in settings.php
• Slave database setup
• Memcache server setup

Sunday, June 27, 2010

 Drupal Integration
• Most all configuration in settings.php
• Slave database setup
• Memcache server setup
• Varnish Reverse Proxy setup

Sunday, June 27, 2010

 Drupal Integration
• Most all configuration in settings.php
• Slave database setup
• Memcache server setup
• Varnish Reverse Proxy setup
• NO_CACHE cookie checking

Sunday, June 27, 2010

 Drupal Integration
• Most all configuration in settings.php
• Slave database setup
• Memcache server setup
• Varnish Reverse Proxy setup
• NO_CACHE cookie checking
• One Custom module to handle all cache
edge-cases and set the NO_CACHE
cookie.
Sunday, June 27, 2010