Replicate Elasticsearch Data with Cross-Cluster Replication (CCR)
5 likes•3,176 views
The document provides an overview of Cross-Cluster Replication (CCR) in Elasticsearch, highlighting its importance for high availability, disaster recovery, and content distribution. It introduces how CCR allows replication of indices between clusters, detailing its architecture, security considerations, and use cases. Additionally, it includes resources for getting started with CCR and mentions a demonstration of its implementation.
Replicate Elasticsearch Data with Cross-Cluster Replication (CCR)
- 1. Replicate Elasticsearch Data with Cross-Cluster Replication (CCR) Jason Zucchetto | Senior Product Manager January 31, 2019
- 2. Jason Zucchetto Senior Product Manager Elastic
- 3. Housekeeping & Logistics • Recording will be available following the webinar • Chat via IRC #elastic-webinar ◦ #elastic-webinar @ Freenode ◦ Click "Join the Chat" link, create an IRC account • Please select high resolution in the YouTube video player
- 4. 4 Agenda Elastic Stack Introduction1 CCR Use Cases Demo Additional Resources Cross Cluster Replication (CCR) Overview2 3 4 5
- 5. 5 Store, Search, & Analyze Visualize & Manage Ingest Elastic Stack SOLUTIONS Elastic Stack Kibana Elasticsearch Beats Logstash SaaS SELF-MANAGED Elastic cloud Elastic cloud Enterprise Standalone
- 6. Solutions Store, Search, & Analyze Visualize & Manage Ingest Kibana Elasticsearch Beats Logstash Elastic Stack SaaS SELF-MANAGED App Search Site Search Enterprise Search FUTURE Metrics APM Business Analytics Logging Security Analytics
- 7. 7 Store, Search, & Analyze Visualize & Manage Ingest Deployment options SOLUTIONS Elastic Stack Kibana Elasticsearch Beats Logstash SaaS SELF-MANAGED Elastic Cloud Elastic Cloud Enterprise Standalone
- 8. 8 Cross Cluster Replication (CCR) in Elasticsearch
- 9. 99 Data Center 1 Data Center 2 Previous Cross Data Center Replication Strategies Previous methods for replicating across Elasticsearch clusters / datacenters ● Double writes ● Middleware ● Snapshot / restore Kafka Snapshot / Restore
- 10. 10 Cross Cluster Replication (CCR) in Elasticsearch ● High availability and disaster recovery ○ Across networks, data centers, or regions ● Content distribution / data locality ○ Elasticsearch clusters close to the application servers ● Read-only cluster from regional source clusters ○ Hub/spoke model, ad-hoc searches don’t require ad-hoc cross-network communication Replicate indices to one or more clusters Cluster 1 Index A Index A Cluster 2
- 11. 11 Cross Cluster Replication (CCR) in Elasticsearch ● Introduced as beta feature in Elasticsearch 6.5 ○ Short beta period (GA soon) ○ Get any feedback to ES team ■ github.com/elastic/elasticsearch ● Platinum level feature ○ Easy to test with 30 day trial (demo) Replicate indices to one or more clusters Cluster 1 Index A Index A Cluster 2
- 12. 12 ● Index based ○ Replicated index is read-only ● Pull based model ○ Follower asks the source for a range of documents to replicate ○ Source cluster is unaware of its followers. ● Many tunable options, e.g. ○ Number of operations replicated per fetch (from the follower) Index B Index A Index B ... ... ... Index A Cluster 1 Cluster 2 Cluster 3 Cross Cluster Replication (CCR) in Elasticsearch Replicate indices to one or more clusters
- 13. 13 Cross Cluster Replication (CCR) Security Security considerations for replicating between two Elasticsearch clusters ● Permissions requirements ○ Leader / source cluster ■ “read_ccr” cluster privilege ■ “monitor” and “read” index privileges ○ Follower / destination cluster ■ “manage_ccr” cluster privilege ■ “monitor”, “read”, “write”, and “manage_follow_index” index privileges ● Same security infrastructure as CCS ● Demo will not use security, to simplify demo setup ○ Demo can also use the built-in “elastic” user
- 14. 14 Cross Cluster Replication (CCR) Use Cases
- 15. 1515 Production Data Center Disaster Recovery Data Center Leader Follower High Availability and Disaster Recovery
- 16. 1616 Production Data Center Disaster Recovery Data Center Leader Follower High Availability and Disaster Recovery
- 17. 1717 Central DCSingapore DC Ireland DC Canada DC Leader Follower Data Locality
- 18. 1818 Central ReportingSingapore DC Ireland DC Canada DC Leader Follower Centralized Reporting
- 19. 1919 Central ReportingSingapore DC Ireland DC Canada DC Leader Follower Centralized Reporting
- 20. 20 Cross Cluster Replication (CCR) Demo
- 21. 21 Cross Cluster Replication (CCR) Demo ● Use two local clusters of Elasticsearch ● Replicate a Metricbeat index from one cluster to another ● Become familiar with the CCR API ● Walk through the CCR Kibana Management UI ● API commands available in Github
- 23. 23 ● Getting started with Cross Cluster Replication ○ https://www.elastic.co/guide/en/elastic-stack-overview/current/ ccr-getting-started.html ● Cross Cluster Replication APIs ○ https://www.elastic.co/guide/en/elasticsearch/reference/curren t/ccr-apis.html Additional Resources
- 24. 24 Housekeeping & Logistics • Recording will be available following the webinar • Chat via IRC #elastic-webinar ◦ #elastic-webinar @ Freenode ◦ Click "Join the Chat" link, create an IRC account • Please select high resolution in the YouTube video player
- 25. Thank You ● Web : www.elastic.co ● Demos: demo.elastic.co ● Products : https://www.elastic.co/products ● Forums : https://discuss.elastic.co/ ● Community : https://www.elastic.co/community/meetups ● Twitter : @elastic
- 26. Questions?