Building a Python
web service with Ray
Philipp Moritz
September 30, 2020
What this talk is about
What this talk is about
Show design patterns for building Python web services with
Ray using Ray tasks and actors
What this talk is about
Show design patterns for building Python web services with
Ray using Ray tasks and actors
Show how we are building Anyscale as a Ray application
What this talk is about
Show design patterns for building Python web services with
Ray using Ray tasks and actors
Show how we are building Anyscale as a Ray application
Show how to address practical challenges like type checking,
testing, tracing, monitoring and deployment
Requirements for a web service
Needs to be available 24/7
Requirements for a web service
Needs to be available 24/7
Needs to be scalable according to user demand
Requirements for a web service
Needs to be available 24/7
Needs to be scalable according to user demand
Needs to integrate external Python libraries and frameworks,
e.g. web serving frameworks or machine learning libraries
Traditional Python web service architecture
Flask server
aiohttp server
fastAPI server
Web logic
Traditional Python web service architecture
Redis
Flask server
Celery
Redis Queue
aiohttp server Multiprocessing
Service 1 Service 2 Service 3
fastAPI server
Web logic Business logic
Traditional Python web service architecture
Database
Redis
Flask server
Celery
Redis Queue
aiohttp server Multiprocessing Blob store
Service 1 Service 2 Service 3
fastAPI server
Web logic Business logic Data
Traditional Python web service architecture
Database
Redis
Flask server
Celery
Redis Queue
aiohttp server Multiprocessing Blob store
Service 1 Service 2 Service 3
fastAPI server
Challenges: Programming, scaling,
Web logic Business logic Data
monitoring, tracing, fault tolerance
Ray web service architecture
Database
Redis
Flask server
Celery
Redis Queue
aiohttp server Multiprocessing Blob store
Service 1 Service 2 Service 3
fastAPI server
Web logic Business logic Data
Ray web service architecture
Database
Service 1 Service 2 Service 3
Flask server
Task Actor Task
Task Actor Task
aiohttp server Blob store
Actor Actor Task
Actor Actor
fastAPI server Task
Web logic Business logic Data
Ray web service architecture
Database
Service 1 Service 2 Service 3
Advantages of Ray:
Flask server
Task Actor Task
● Unified programming model
Task Actor Task
aiohttp server Blob store
Actor Actor Task
Actor Actor
fastAPI server Task
Web logic Business logic Data
Ray web service architecture
Database
Service 1 Service 2 Service 3
Advantages of Ray:
Flask server
Task Actor Task
● Unified programming model
Task Actor Task
● Automatic scheduling,
aiohttp server resource management Blob store
Actor Actor Task
Actor Actor
fastAPI server Task
Web logic Business logic Data
Ray web service architecture
Database
Service 1 Service 2 Service 3
Advantages of Ray:
Flask server
Task Actor Task
● Unified programming model
Task Actor Task
● Automatic scheduling,
aiohttp server resource management Blob store
● Autoscaling Actor Actor Task
Actor Actor
fastAPI server Task
Web logic Business logic Data
Ray web service architecture
Database
Service 1 Service 2 Service 3
Advantages of Ray:
Flask server
Task Actor Task
● Unified programming model
Task Actor Task
● Automatic scheduling,
aiohttp server resource management Blob store
● Autoscaling Actor Actor Task
● Built-in facilities for monitoring
Actor Actor
fastAPI server Task
Web logic Business logic Data
Ray web service architecture
Database
Service 1 Service 2 Service 3
Advantages of Ray:
Flask server
Task Actor Task
● Unified programming model
Task Actor Task
● Automatic scheduling,
aiohttp server resource management Blob store
● Autoscaling Actor Actor Task
● Built-in facilities for monitoring
● Great
fastAPIsupport
server for ML
Actor Actor Task
Web logic Business logic Data
Reminder: The Anyscale Platform
1. Laptop experience with the power of a cluster
2. Serverless experience without serverless limitations
3. Real-time collaboration
Architecture of Anyscale
fastAPI server Service 1 Service 2 Database
Task Task
Task Task
Actor Actor
Web logic Business logic Data
Scaling up with Ray tasks
fastAPI server Sessions Websockets Database
Task Actor Session 1
Session 2
Web logic Business logic Data
/api/v2/session/start
Session 1
Session 2
Scaling up with Ray tasks
fastAPI server Sessions Websockets Database
Task Actor Session 1
Task Session 2
Task
Web logic Business logic Data
/api/v2/session/1/execute
/api/v2/session/1/execute
/api/v2/session/1/execute Session 1
Session 2
Scaling up with Ray tasks
fastAPI server Sessions Websockets Database
Task Actor Session 1
Task Session 2
Task
Task
Web logic Business logic Data
Web logic Business logic
/api/v2/session/1/execute
/api/v2/session/1/execute
/api/v2/session/1/execute Session 1
/api/v2/session/2/execute
Session 2
Managing state with Ray actors
Update
fastAPI server Sessions Notifications
Actor Actor
Task
Update
Web logic Business logic
/api/v2/session/start
Session
Writing an API server with fastAPI
● Makes it easy to define a REST API
● Schema validation
● Typing
@router.get(“/{command_id}/execution_logs”)
async def get_execution_logs(
command_id: int, ...) ->
Response[LogOutput]:
Ray asyncio support
Object Reference
Ray object references are awaitable!
async def get_execution_logs(session_record, session_command_id, log_params):
log = await session_tasks_service.get_execution_log.remote(
session_record["id"],
session_command_id,
logs_params
)
Ray asyncio support
Object Reference
Ray object references are awaitable!
async def get_execution_logs(session_record, session_command_id, log_params):
log = await session_tasks_service.get_execution_log.remote(
@ray.remote
session_record["id"],
class WebSocketActor:
session_command_id, def __init__(self) -> None:
self.sio = socketio.AsyncServer()
logs_params
async def emit(self, message_name: str,
) data: Dict[str, Any]) -> None:
await self.sio.emit(message_name, data)
Ray actors can also be
Typing
fastAPI server Service 1 Service 2
Task Task
Task Task
Actor Actor
Web logic Business logic
Frontend
executeCommand({ async def execute_command( @ray.remote
sessionId, session_id: int, options: Options): def execute_command(
options: { command_record = db.create_command( command: CommandRecord):
command: input session_id, options) runner = AnyscaleSessionRunner()
} execute_command.remote( runner.execute_command(command)
}) command_record)
TypeScript Python Python
Testing
Unit testing: Use the Ray local mode for unit testing:
ray.init(local_mode=True)
Everything runs in a single process -> can mock out interfaces
Integration testing: Use a Ray instance running on the laptop/CI server,
testing web logic, business logic and database
End-to-end testing: Test full functionality in staging environment
Stress testing: Test scalability limits of the system
Metrics and Monitoring
Use Ray’s built in metrics API:
from ray.experimental import metrics
self.create_cluster_stats = metrics.Histogram(
"Anyscale_create_cluster", "Num of seconds took to create cluster",
"second",
[float(i) for i in range(10, 300, 10)],
["step"],
Metrics and Monitoring
Use Ray’s built in metrics API:
from ray.experimental import metrics
self.create_cluster_stats = metrics.Histogram(
"Anyscale_create_cluster", "Num of seconds took to create cluster",
"second",
[float(i) for i in range(10, 300, 10)],
["step"],
Tracing
We use OpenTelemetry for tracing
Can generate detailed traces for a number of Python libraries, including database
clients, web frameworks. requirements.txt:
opentelemetry-api
opentelemetry-sdk
opentelemetry-ext-asgi
Automatic tracing for Ray tasks and actors
opentelemetry-ext-asyncpg
opentelemetry-ext-botocore
opentelemetry-instrumentatio
Can also add custom traces: n-starlette
Tracing
We use OpenTelemetry for tracing
It can generate detailed traces automatically for a number of Python libraries,
including database clients, web frameworks.
requirements.txt
opentelemetry-api
opentelemetry-sdk
Full automatic tracing for Ray tasks and actors opentelemetry-ext-asgi
opentelemetry-ext-asyncpg
opentelemetry-ext-botocore
opentelemetry-instrumentation-starlette
Can also add custom traces:
Deployment
The cloud environment for our web service is set up with
Terraform, to make the setup easily reproducible for
● Development
● Staging,
● Production
The web service is deployed on Docker and Kubernetes, which
integrate well with Ray
Summary
We showed how the Python web serving ecosystem
integrates with Ray
We showed how Ray makes it easy to scale up your web
services and manage their state
We showed how to type, test, monitor and deploy your web
service with Ray
Thanks to the Team @ Anyscale
