Cloud Foundry Summit 2017

Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
2017年6月21日
The Road to "JYU-BAI"
- Adopting Cloud Foundry at Yahoo! JAPAN -
2017年6月20日

About me
2
Software Engineer
Manager
Yahoo! JAPAN
Yasuhiko Kubono

3
• Introducing Cloud Foundry into our
services
-Yasuhiko Kubono
• How do we Actually Operate
-Yusuke Kondo

Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Introducing Cloud Foundry
into our services

Agenda
5
• About Yahoo! JAPAN
• Why we use Cloud Foundry?
• Introducing Cloud Foundry into our services
• Case study

About
Yahoo! JAPAN

Outline
Yahoo Japan Corporation
(SoftBank Group)
Businesses: Internet Advertising - e-Commerce
- Member services - Others
Headquarters: Tokyo Japan
Founded: January 31, 1996
# of Employees: 5,826 (As of March 31, 2017)
7

# of Engineers & Designers
2,500

More than 100
Web Services

39,89MActive User IDs
1Month
※2017年1-3月の平均
67,4B Page Views
Total requests

Why we use
Cloud Foundry?

Why we use Cloud Foundry？
Speed up
development time
JYU-BAI
increase productivity by 10 times
12

Adoption Plan
13
2016
Initial introduction to
a few services
2017
Full-scale
implementation
2018
Expand
implementation
Here

Introducing
Cloud Foundry
into our Services

Programming Languages
16
C, Perl, C++,
PHP, Node.js, Java...

Architecture Differs by Web Service
17
e.g. Travel tips e.g. Yahoo! Auction
Small-scale web services
list
cart
logic
logic
logic
Large-scale web services
Search
API
Gate
way

Obstacles
18
Same architecture does not fit
in each web services

Solutions
Enroll CF Coach in each web services
around 20 staff / 15 services
Coaches role:
Promote cloud design methods that suit for
each web services
19

Role map
20
Core Team
CF Coach
For
Shopping
Shopping
engineer
Shopping
engineer
CF Coach
For
Auction
Auction
engineer
CF Coach
For
Media
Media engineer
…

Case study

Where we started from

List Necessary Functions
23
Service A Service B Service C Service D Service E Service F Service G
MySQL ● ● ● ●
Oracle ● ● ●
KVS
ObjectStrage ●
C/C++ ●
PHP ● ● ● ●
Node.js ● ● ● ● ●
Java ● ● ●
advertisement ● ● ● ● ●
beacon ● ●

Challenges we encountered
Functions that can’t be used in the cloud because of
complicated dependency
Internal security polices are not suited for cloud
environment
Most of our web services were stateful design
24

How we started
We selected one web service, and started
by preparing the necessary functions for
that service
Resolve issues each time they occur
25

So, which web service
did we start with?

Criteria for the web service
1. Simplicity
• Service with limited functions and external
PF that can be used
2. Actively developed
• Web services that actively developed
so that the effectiveness of introducing CF
can be measured
27

First target : CS tool
• Constructed with few servers in OpenStack environment
• WebServer: apache
• Apache Traffic Server (ATS) : Reverse Proxy
28
 Characteristics
• Language: PHP
• Framework： cakephp
• Uses REST API
• MySQL
 Server Configuration
ATS
CS tool
(apache)
API
ATS
MySQL
HTTPS
HTTP MySQL
CS tool
(apache)
(our auction service)

Partial release using ATS (entry points)
Partially diverted entry point to CF apps using ATS:
• So that CF or OpenStack can be switched in entry points
29
ATS
CS tool
(apache)
CS tool
(apache)
API
HTTPS
HTTP
Some
entry
points
ATS
CS tool
(CF)

Lessons learned from the first target
• How to Implement in Production
• Development method based on OSS
• How to make service stateless on CF
30

Adopting & Expanding to other services
31
Decide
target
Investigate
issues
Adopting
knowledge
Solve issues
Knowhow
accumulated

Next Presentation,
How do we Actually Operate
Photo by: Aflo

Hello CF Summit 2017!
Yusuke Kondo or @konfoo
Responsible to...
• operating Cloud Foundry & Concourse on IaaS
• increasing engineers’ productivity by providing
tools and best practices around CI/CD

Overview of Yahoo! JAPAN proprietary Infrastructure
More than four DCs in Japan
More than 90,000 VM running on OpenStack
34

Cluster Spec
35
dev production
Load Balancer Software Hardware x2
IaaS Openstack Openstack
Hypervisor # 40 40

Current Status (As of Jun. 9, 2017)
36
dev production
Cluster # 1 1
Cell # per Cluster 40 30
Org # 136 38
App Instance # approx. 2,000 approx. 400
Rps at peak time N/A approx. 2,000

Future Plan (As of Jun. 9, 2017)
37
dev production
Cluster # 1 1 => 6
Cell # per Cluster 40 30 => 100
Org # 136 38
App Instance # approx. 2,000 approx. 400
Rps at peak time N/A approx. 2,000

Integration
with
Backend Services

App Role Based
ACL
MQ
Service
RDB
Object
Storage
Key Value
Store
Cache
Service
FaaS
Existent Platforms
39

Integration with Existent Platforms
• Cookie off-loading Route Service
• On-demand MySQL (OpenStack Trove API)
• Distributed pubsub service (Pulsar)
40

Marketplace Dashboard
41
Goal: Providing all PFs in CF Marketplace

Issues we faced
Platform ACL is based on IP address or hostname
=> Requesting for exceptional permission for
accessing via IP Range with a limited term.
=> Migrating from Host-based ACL to Role-
Based ACL in the long term
42

Integration
with
Logs and Metrics

What we already have
In-house Monitoring & Alerting PF based on
Apache Kafka, Hbase,
Splunk, an enterprise log analytics platform
44

User-side Logs and Metrics
45
PCF Cluster-1
VM
APP
VM
App
VM
VM
APP
App
Loggregator
Splunk
Monitoring PF
App App
App
APP
APP
APP
APP
No action is needed for App developers

What we prepared
Firehose Nozzle and Relay Server
• Nozzle filters and formats the App
logs streamed by Firehose
• Relay Server forwards the log stream
to specific index
46

Issue we faced
High log traffic. 900 lines per sec !
(as of Jun. 8, 2017)
=> Provided users with CF friendly logger
47

Operator-side logs and Metrics
Splunk
• Platform logs such as CF component syslog
Prometheus
• Bosh metrics, VM metrics, Firehose metrics
• Emitting alerts to our smartphone
48

49

Integration
with
other System

Integration with package monitoring tool
51
Application Source Code
Dependent packages
Runtime Buildpack version
Vulnerable
Package
Monitoring Tool
Track the buildpack version which the App are staged with and report outdated apps.

Integration with package monitoring tool
52
Application Source Code
Dependent packages
Runtime
Scan package version
Scan whole source code

Integration with Concourse
We use Concourse for
• deploying new Cloud Foundry release
• updating buildpacks
• syncing employee accounts with UAA
• backup database to object storage
• ...
53

Lessons learned

We are still on the way to change mind
Changing your organization mind is the most
essential part.
• Educate not only users, but also platform
division where you belong.
• Work closely with your security paranoid
team. Involve them to update the policy
55

Cloud Foundry Summit 2017

More Related Content

Cloud Foundry Summit 2017