SlideShare a Scribd company logo

Advanced nginx in mercari - How to handle over 1,200,000 HTTPS Reqs/Min

•
•
Masahiro Nagano
Masahiro Nagano

Advanced nginx in mercari - How to handle over 1,200,000 HTTPS Reqs/Min 「nginx実践入門」出版記念！執筆者らが語る nginx Tech Talks

1 of 33
Downloaded 46 times
Advanced nginx in Mercari 「nginx実践入門」出版記念！執筆者らが語る nginx Tech Talks Masahiro Nagano How to handle over 1,200,000 HTTPS Reqs/Min
Me • Masahiro Nagano @kazeburo • Mercari, inc. • Principal Engineer, Site Reliability Engineering Team • BASE, inc. Technical Advisor
Mercari Your Friendly Mobile MarketPlace JP US
Mercari KPI DOWNLOAD GMV ITEMS 3100万DL(JP+US) 月間数十億円 1日数十万品以上
API Traffic 1,200,000 reqs/min (HTTPS only)
nginx in mercari ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Users Client Multimedia Corporate data center Trad se Mobile Client AWS Management Console IAM Add-on Example: IAM Add-on Tasks (HIT) TaskTurk ice Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporatedata center Mobile Client net AWS Management Console IAM Add-on Example:IAM Add-on Requester azon W eb Services LLC or its affiliates. All rights reserved. ser Users Client Multim edia Corporate data center Tradit server Mobile Client AW S Managem ent Console IAM Add-on Exam ple: IAM Add-on ence HIT) Assignm ent/ Task W orkers L7 load balancer Reverse Proxy TLS Terminator Contents Distribution©2011 Am azon W eb Services LLC or its affiliates. All rights reserve Users Client M ultim edia Corporate data center Tradition server M obile Client Internet AW S M anagem ent Console IAM Add-on Exam ple: IAM Add-on m ent/ Task Requester W orkers
nginx Extension in mercari • ngx_dynamic_upstream • Operate upsteam servers Up/Down dynamically • OpenResty (ngx_lua) • More powerful L7 load balancer • Application log collector
Agenda • Continuous updating nginx • TLS Optimization • Automated TLS session ticket key updating • SPDY to HTTP/2
Continuous updating nginx
Keep nginx the latest • Use nginx mainline version • stableは安定しているという意味ではなく新機能が追加さ れず、バグ修正も緊急な修正を除いて行われないというこ と。すべての新機能やバグ修正はmainlineにて行われるの で結果的にmainlineが安定している
 https://www.nginx.com/blog/nginx-1-6-1-7-released/ • nginxは無停止でのアップデートが可能 • Periodic OpenSSL vulnerability report 😓
nginx-build • nginx builder command written in Go
 https://github.com/cubicdaiya/nginx-build • Automate download, configure and make • Download OpenSSL, zlib and PCRE. build them statically ( independent from OS’s library version) • Support Tengine and OpenResty
nginx-build $ EXPORT $VERSION=1.9.10 $ nginx-build -clear -d work -v $VERSION -c nginx-build-conf/configure -m nginx-build-conf/modules3rd.ini -zlib -zlibversion=1.2.8 -verbose -pcre -pcreversion=8.38 -openssl -opensslversion=1.0.2f $ cd work/$VERSION/nginx-$VERSION $ sudo make install
TLS Optimization
TLS Optimization • PFS - Perfect Forward Secrecy • TTFB - Time To First Byte
Perfect Forward Secrecy • Mozilla Wiki
 https://wiki.mozilla.org/Security/Server_Side_TLS • Mozilla SSL Configuration Generator
 https://mozilla.github.io/server-side-tls/ssl-config- generator/ • *最新の情報を参照し、プロダクション環境に投 入する前にテストを行ってください
TTFB • TLS Session Cache • TLS Session Tickets • OCSP Stapling • SSL buffer size
TLS Session Cache • Server side session cache • Client software does not matter • nginx could’t share session cache between servers. ssl_session_cache shared:SSL:120m; ssl_session_timeout 5m; CONS PROS
TLS Session Cache • OpenResty will support it?
 https://twitter.com/agentzh/status/ 686655229828403201
TLS Session Tickets • Client Side Session Cache
 (like a CookieStore) • No need to share cache between servers • It’s required share ssl_session_ticket_key file • Client support required ssl_session_tickets on; ssl_session_ticket_key /path/to/ssl_session_ticket; PROS CONS
TLS Session Tickets User Agent Session Tickets Android 2.3.7 No Android 4.0.4 YES Android 5 YES Chrome 47 YES Firefox 42 YES IE11/Win7 No IE11/Win10 YES Edge/Win10 YES Safari 9/iOS9 No Safari 9/MacOSX10.11 No !!!! !!!! https://www.ssllabs.com/ssltest/clients.html
OCSP Stapling https://wiki.mozilla.org/Security/Server_Side_TLS Client required to checking the revocation status. If OCSP Responder is down, page load is failed. Server retrieves OCSP record, caches it and servers it to client directly. Fast!
OSCP Stapling • All Android version don’t support yet • iOS < 9 doesn’t support it CONS CONS ssl_stapling on; ssl_stapling_verify on; ssl_trusted_certificate /path/to/server.crt; resolver 8.8.8.8 valid=30s; resolver_timeout 5s;
TLS buffer size • TLS record size. Default 16KB, it’s too large. That’s effect to latency
 https://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-to-first-byte/ ssl_buffer_size 1k;
Automated TLS session tickets key updating
TLS Session Tickets • Should share ssl_session_ticket_key file between servers • Mozilla recommends replace it every day for forward secrecy • Previously we changed the key file almost every month in manual operation😓
Update Tickets with Consul • Inspired by Stretcher (fujiwara👍) • Key_file is very small (48byte), It can store into a payload of consul event • We’re updating key_file twice a day by using Consul
Update Tickets with Consul JOB Consul Servers nginx Consul Agent $ consul event -name=“ssl-session-ticket-refresh" $(openssl rand 48 | base64) nginx Consul Agent nginx Consul Agent cat | jq -r '.Payload' | base64 -d | base64 -d > /path/to/ticket.key service nginx reload ❶ ❷ ❷ ❷ ❸
SPDY to HTTP/2
SPDY is alive Request to a webview 728 1,009,268 270,527 HTTP/1.1 SPDY HTTP2
SPDY to HTTP/2 • In Mobile App, transition to HTTP/2 from SPDY is not so quickly • Android 4.x, iOS 8... • nginx dropped SPDY in 1.9.4, but we need to transition gracefully in a few and more years • We need a server that speak both HTTP/2 and SPDY
Tengine Now we’re trying Tengine (at a previous slide)
cloudflare patch? https://blog.cloudflare.com/introducing-http2/
End

More Related Content

Advanced nginx in mercari - How to handle over 1,200,000 HTTPS Reqs/Min

  • 1. Advanced nginx in Mercari 「nginx実践入門」出版記念！執筆者らが語る nginx Tech Talks Masahiro Nagano How to handle over 1,200,000 HTTPS Reqs/Min
  • 2. Me • Masahiro Nagano @kazeburo • Mercari, inc. • Principal Engineer, Site Reliability Engineering Team • BASE, inc. Technical Advisor
  • 3. Mercari Your Friendly Mobile MarketPlace JP US
  • 6. nginx in mercari ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Users Client Multimedia Corporate data center Trad se Mobile Client AWS Management Console IAM Add-on Example: IAM Add-on Tasks (HIT) TaskTurk ice Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporatedata center Mobile Client net AWS Management Console IAM Add-on Example:IAM Add-on Requester azon W eb Services LLC or its affiliates. All rights reserved. ser Users Client Multim edia Corporate data center Tradit server Mobile Client AW S Managem ent Console IAM Add-on Exam ple: IAM Add-on ence HIT) Assignm ent/ Task W orkers L7 load balancer Reverse Proxy TLS Terminator Contents Distribution©2011 Am azon W eb Services LLC or its affiliates. All rights reserve Users Client M ultim edia Corporate data center Tradition server M obile Client Internet AW S M anagem ent Console IAM Add-on Exam ple: IAM Add-on m ent/ Task Requester W orkers
  • 7. nginx Extension in mercari • ngx_dynamic_upstream • Operate upsteam servers Up/Down dynamically • OpenResty (ngx_lua) • More powerful L7 load balancer • Application log collector
  • 8. Agenda • Continuous updating nginx • TLS Optimization • Automated TLS session ticket key updating • SPDY to HTTP/2
  • 10. Keep nginx the latest • Use nginx mainline version • stableは安定しているという意味ではなく新機能が追加さ れず、バグ修正も緊急な修正を除いて行われないというこ と。すべての新機能やバグ修正はmainlineにて行われるの で結果的にmainlineが安定している
 https://www.nginx.com/blog/nginx-1-6-1-7-released/ • nginxは無停止でのアップデートが可能 • Periodic OpenSSL vulnerability report 😓
  • 11. nginx-build • nginx builder command written in Go
 https://github.com/cubicdaiya/nginx-build • Automate download, configure and make • Download OpenSSL, zlib and PCRE. build them statically ( independent from OS’s library version) • Support Tengine and OpenResty
  • 12. nginx-build $ EXPORT $VERSION=1.9.10 $ nginx-build -clear -d work -v $VERSION -c nginx-build-conf/configure -m nginx-build-conf/modules3rd.ini -zlib -zlibversion=1.2.8 -verbose -pcre -pcreversion=8.38 -openssl -opensslversion=1.0.2f $ cd work/$VERSION/nginx-$VERSION $ sudo make install
  • 14. TLS Optimization • PFS - Perfect Forward Secrecy • TTFB - Time To First Byte
  • 15. Perfect Forward Secrecy • Mozilla Wiki
 https://wiki.mozilla.org/Security/Server_Side_TLS • Mozilla SSL Configuration Generator
 https://mozilla.github.io/server-side-tls/ssl-config- generator/ • *最新の情報を参照し、プロダクション環境に投 入する前にテストを行ってください
  • 16. TTFB • TLS Session Cache • TLS Session Tickets • OCSP Stapling • SSL buffer size
  • 17. TLS Session Cache • Server side session cache • Client software does not matter • nginx could’t share session cache between servers. ssl_session_cache shared:SSL:120m; ssl_session_timeout 5m; CONS PROS
  • 18. TLS Session Cache • OpenResty will support it?
 https://twitter.com/agentzh/status/ 686655229828403201
  • 19. TLS Session Tickets • Client Side Session Cache
 (like a CookieStore) • No need to share cache between servers • It’s required share ssl_session_ticket_key file • Client support required ssl_session_tickets on; ssl_session_ticket_key /path/to/ssl_session_ticket; PROS CONS
  • 20. TLS Session Tickets User Agent Session Tickets Android 2.3.7 No Android 4.0.4 YES Android 5 YES Chrome 47 YES Firefox 42 YES IE11/Win7 No IE11/Win10 YES Edge/Win10 YES Safari 9/iOS9 No Safari 9/MacOSX10.11 No !!!! !!!! https://www.ssllabs.com/ssltest/clients.html
  • 21. OCSP Stapling https://wiki.mozilla.org/Security/Server_Side_TLS Client required to checking the revocation status. If OCSP Responder is down, page load is failed. Server retrieves OCSP record, caches it and servers it to client directly. Fast!
  • 22. OSCP Stapling • All Android version don’t support yet • iOS < 9 doesn’t support it CONS CONS ssl_stapling on; ssl_stapling_verify on; ssl_trusted_certificate /path/to/server.crt; resolver 8.8.8.8 valid=30s; resolver_timeout 5s;
  • 23. TLS buffer size • TLS record size. Default 16KB, it’s too large. That’s effect to latency
 https://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-to-first-byte/ ssl_buffer_size 1k;
  • 25. TLS Session Tickets • Should share ssl_session_ticket_key file between servers • Mozilla recommends replace it every day for forward secrecy • Previously we changed the key file almost every month in manual operation😓
  • 26. Update Tickets with Consul • Inspired by Stretcher (fujiwara👍) • Key_file is very small (48byte), It can store into a payload of consul event • We’re updating key_file twice a day by using Consul
  • 27. Update Tickets with Consul JOB Consul Servers nginx Consul Agent $ consul event -name=“ssl-session-ticket-refresh" $(openssl rand 48 | base64) nginx Consul Agent nginx Consul Agent cat | jq -r '.Payload' | base64 -d | base64 -d > /path/to/ticket.key service nginx reload ❶ ❷ ❷ ❷ ❸
  • 29. SPDY is alive Request to a webview 728 1,009,268 270,527 HTTP/1.1 SPDY HTTP2
  • 30. SPDY to HTTP/2 • In Mobile App, transition to HTTP/2 from SPDY is not so quickly • Android 4.x, iOS 8... • nginx dropped SPDY in 1.9.4, but we need to transition gracefully in a few and more years • We need a server that speak both HTTP/2 and SPDY
  • 31. Tengine Now we’re trying Tengine (at a previous slide)
  • 33. End