Trove Project Onboarding Final2

13.11.
2018
Trove Onboarding Session

Introductory course for contributors and reviewers
Bartosz Żurkowski IRC: EMAIL:

Cloud and Big Data Engineer, bzurkowski [email protected]
Samsung R&D Institute Poland
Przemysław Godek IRC: EMAIL:

Senior Cloud and Big Data Engineer, pgodek [email protected]
Samsung R&D Institute Poland
Outline
● Introduction ● How do I get started?
○ Trove mission ○ Development environment setup
○ Features overview ○ Building guest images
○ Datastore support ○ Guest customization via cloudinit
● Deep dive ● Where contribution is needed?
○ Overall architecture
○ Handling datastore diversity using
strategy pattern
What is Trove?
Application optimization
● Database as a Service for OpenStack DB performance tuning

● Provides full database lifecycle management Replication and clustering
○ Provisioning, configuration, backups, scaling
Scaling
● Multi-datastore support
Periodic backups
○ 11 database engines
○ Relational, non-relational DB software upgrades
○ Single-instance and clustered deployments Hardening
● Unified management interface DB software setup and config
● Built entirely on OpenStack Virtual resource provisioning

○ Synergy of Nova, Cinder, Swift, Glance and Neutron
DBA responsibilities
DBaaS (Trove) responsibilities
Features overview
● Instance provisioning ● Cluster provisioning
● Instance resizing (volume, flavor) ● Cluster sizing (grow, shrink)
● Database and user management ● Replication setup
● Configuration groups ● Replication failover
● Backups (full, incremental, scheduled) (promote read replica, eject source)
● Datastore upgrades
● Logs (guest, database)
● Security groups management
● Flavors management
Did you find your
favourite database
Datastore capability matrix on the slide?
Schema Backup
Datastore Type Provisioning Resizing Replication Clustering
management and restore
MySQL ✓ ✓ ✓ ✓ ✓ ✓
MariaDB ✓ ✓ ✓ ✓ ✓ ✓
PostgreSQL Relational ✓ ✓ ✓ ✓ ✓ ✗
Percona ✓ ✓ ✓ ✓ ✓ ✓
DB2 ✓ ✓ ✓ ✓ ✗ ✗
Redis Key-value ✓ － ✓ ✓ ✓ ✓
Cassandra ✓ ✓ ✓ ✓ － ✓
Column
Vertica ✓ ✓ ✗ ✓ － ✓
MongoDB ✓ ✓ ✓ ✓ － ✓
Document
CouchDB ✓ ✓ ✓ ✓ － ✗
Couchbase Multi-model ✓ ✗ ✓ ✓ － ✗
Let’s dive into the internals
Deliverables
Trove Python Trove Client

openstack/trove openstack/python-troveclient
Trove Dashboard Trove Specs Trove Tempest Plugin

openstack/trove-dashboard openstack/trove-specs openstack/trove-tempest-plugin
Overall architecture
Trove
DynamoDB RedShift
RDBMS NoSQL
RDS Elasticache DW
EC2 S3 Nova Swift
EBS VPC Cinder Neutron

Overall architecture
● Trove API
○ Exposes REST API for end user
○ Handles simple tasks
● Trove Task Manager
○ Handles heavy database operations
enqueued by API (provisioning,
backups, scaling)
● Trove Guest Agent
○ Exposes abstract interface for
interacting with the database service
● Trove Conductor
○ Serves as a proxy for Guest Agent to
communicate back with the control
plane
Overall architecture: Performing database backup
1. User hits Trove API via CLI or Horizon
1 4 2. API creates initial backup record in the
6 7 database
3. ... queues backup job onto the message
bus
2 3
4. … and responds to the user
5. Taskmanager grabs job from message bus
5 6. … and asks guestagent to perform backup
9 7. Guestagent performs backup and streams
output to Swift
8. … then reports backup completion back to
8 Conductor
9. Conductor updates backup status in the
database
How to handle datastore diversity?
● Configuration ● Clustering
○ Procedures ○ Procedures
○ Formats (INI, YAML, JSON) ○ Primitives
○ Directories (Cassandra tokens, Redis slots)
○ Sets of parameters ○ Management tools (nodetool, redis-cli)
● Backup and restore ● Replication
○ Procedures ● Storage
○ Backuping tools ● ...
… with Strategy pattern
● Behavioral software design pattern
● Enables selecting an algorithm at runtime
● Code receives run-time instructions of which algorithms to use
● Depending on datastore type, strategy implements a specific backup, replication,
storage or clustering routine
● Locations in code:
○ guestagent/strategies/* – backup, restore, replication
○ guestagent/datastore/experimental/* – database service management
○ common/strategies/cluster/experimental/* - cluster management
Strategy pattern examples: Database service
DatabaseService
● prepare()
● list_databases()
● create_database()
● list_users()
● create_user()
● get_status()
● fetch_logs()
MariaDBService RedisService CassandraService
● prepare() ● prepare() ● prepare()

● list_databases() ● list_databases() ● list_databases()
● create_database() ● create_database() ● create_database()
● ... ● ... ● ...
Strategy pattern examples: Clustering
ClusteringStrategy
● create_cluster()
● grow_cluster()
● shrink_cluster()
● upgrade_cluster()
GaleraClusteringStrategy RedisClusteringStrategy CassandraClusteringStrategy
● create_cluster() ● create_cluster() ● create_cluster()

● grow_cluster() ● grow_cluster() ● grow_cluster()
● shrink_cluster() ● shrink_cluster() ● shrink_cluster()
● upgrade_cluster() ● upgrade_cluster() ● upgrade_cluster()
Determining strategy for datastore
Loading strategy
Strategy configuration
How do I get started?
Development environment setup: Devstack
● Download Devstack
git clone git://git.openstack.org/openstack-dev/devstack.git && cd devstack
● Edit local.conf
[[local|localrc]]
...
enable_plugin trove git://git.openstack.org/openstack/trove
LIBS_FROM_GIT=python-troveclient
● Run Devstack utility
./stack.sh
● Trove sources reside in Useful documentation:
/opt/stack/trove ❏ https://github.com/openstack/trove/tree/master/devstack
❏ https://docs.openstack.org/devstack/latest/configuration.html
Development environment setup: Kolla Ansible
● Install Kolla Ansible: https://docs.openstack.org/kolla-ansible/latest/user/quickstart.html
● Edit /etc/kolla/globals.yaml
enable_trove: "yes"
enable_cinder: "yes"
enable_cinder_backend_lvm: "yes" # required setting LVM volumes group
trove_dev_mode: "yes"
● Run deployment
kolla-ansible -i all-in-one prechecks
kolla-ansible -i all-in-one deploy
● Trove sources reside in /opt/stack/trove
Building guest images
● Install Disk Image Builder:
https://docs.openstack.org/diskimage-builder/latest/user_guide/installation.html
● Clone Trove repository
git clone https://github.com/openstack/trove.git
● Run build-image command
./trove/integration/scripts/trovestack build-image mysql
● Guest image will be built into $HOME/images directory
Useful documentation:
❏ https://docs.openstack.org/trove/latest/admin/building_guest_images.html
Load images into Trove
● Upload image to Glance
openstack image create --file $HOME/images/<image-name>.qcow2 \
--disk-format qcow2 --public ubuntu_mysql
● Setup datastore in Trove
trove-manage datastore_update mysql ""
trove-manage datastore_version_update mysql 5.7 mysql <glance-image-id> "" 1
trove-manage datastore_update mysql 5.7
trove-manage db_load_datastore_config_parameters \
mysql 5.7 \
./trove/trove/templates/$DATASTORE_TYPE/validation-rules.json
Cloudinit scripts
● Allow to customize VM on the first boot
● Each datastore defines its own <datastore-name>.cloudinit script in
/etc/trove/cloudinit directory
● Task Manager collects cloudinit content and injects it into VM via user data
● Example use cases
○ Setting authorized keys for connection from the host
○ Synchronizing guest code with host or external repository
○ Configuring monitoring agents
○ Running Ansible playbooks
Where contribution
is needed?
Appealing work awaiting
● Python 3 support
● Aligning OpenStack CLI with Trove CLI
● Cluster management
(backups, rolling upgrades, flavor resizing)
● Datastore support enhancement
(Oracle, SQL Server, HBase, Druid, etc.)
● Cluster status monitoring
● Backup encryption
Check out full list of our ideas

https://blueprints.launchpad.net/trove
Meet core contributors & community members
● With whom could I chat about Trove at the conference?
○ Bartosz Żurkowski, [email protected]
○ Przemysław Godek, [email protected]
● Check out the ongoing work: https://etherpad.openstack.org/p/trove-general
● You are invited to test Trove, subsequently share your bug fixes, and enhance the
documentation
● Weekly meetings: Wed 14:00 UTC on IRC #openstack-meeting-alt

channel, up-to-date schedule:
https://wiki.openstack.org/wiki/Meetings/TroveMeeting
● Obtain help at Trove IRC channel: #openstack-trove:
http://webchat.freenode.net/?channels=openstack-trove
Questions & Answers
openstack @OpenStack openstack OpenStackFoundation

Thank You!
openstack @OpenStack openstack OpenStackFoundation

Trove Project Onboarding Final2

Uploaded by

Copyright:

Available Formats

Trove Project Onboarding Final2

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Trove Project Onboarding Final2

Uploaded by

Copyright:

Available Formats

13.11.

Trove Onboarding Session

Bartosz Żurkowski IRC: EMAIL:

Przemysław Godek IRC: EMAIL:

● Database as a Service for OpenStack DB performance tuning

○ Single-instance and clustered deployments Hardening

● Unified management interface DB software setup and config

● Built entirely on OpenStack Virtual resource provisioning

Trove Python Trove Client

Trove Dashboard Trove Specs Trove Tempest Plugin

EC2 S3 Nova Swift

EBS VPC Cinder Neutron

MariaDBService RedisService CassandraService

● prepare() ● prepare() ● prepare()

GaleraClusteringStrategy RedisClusteringStrategy CassandraClusteringStrategy

● create_cluster() ● create_cluster() ● create_cluster()

Check out full list of our ideas

● Weekly meetings: Wed 14:00 UTC on IRC #openstack-meeting-alt

openstack @OpenStack openstack OpenStackFoundation

openstack @OpenStack openstack OpenStackFoundation

You might also like