Elastic Cloud Enterprise in Azure with Devon
0 likes5,333 views
Devon Energy, in collaboration with Accenture, has developed a cloud-based analytics platform that evolved through various stages from 2013 to 2019, enhancing security and log management practices. The platform addresses traditional SIEM challenges such as retention and parsing while improving data ingestion rates and operational efficiency. Future plans include advanced data collection techniques and dedicated nodes for security and non-security operations.
Elastic Cloud Enterprise in Azure with Devon
- 1. NYSE: DVN devonenergy.com Elastic Cloud Enterprise in Azure A Devon Energy Story
- 2. Devon - Internal 2 Paul PC • Alphabet soup: MBA, GSE, GREM, GCIA, GCIH, GSEC, GPEN, GPYC, CISSP • Security Architect / Team Lead for Devon Energy, an Independent E&P headquartered in OKC • Fluent in Romanian, English, Python • Loves the cloud almost as much as his Ducati Prakhar S • Educational Kaizen : Bachelor of Engineering (CSE), RHCE, SAS® Certified • Works for Accenture Technology, collaborating with Devon Energy currently • Close to nine years of experience, aspires to contribute in cutting edge tech Who Are We?
- 3. Devon - Internal 3 Agenda / History 2013: ES 1.7 and Moloch 2015: Production-grade Servers 2017: Elastic Search in Security Analytics Platform 2019: SIEM replacement
- 4. Devon - Internal 4 2017: Analytics Platform v1.0 – Working Experiments Axioms: • Collaboration with Data Analytics team • Only Cloud components • Use as much azure 1st party as possible • Be able to ingest 500GB/ day
- 5. Devon - Internal 5 Approximately • 100 billion documents • 100 TB of Indexed data • 1.5 – 2.0 TB DAILY • 1300 indices • 10000 shards • Ingestion rate approximately 20-25k EPS Scale
- 6. Devon - Internal 6 2019: Analytics Platform v3.0 – Production SIEM++ Enrichment moved to Logstash DYI Retention to BLOB First shard optimizations efforts Automation is our best L1 HoD More data, more user stories • Network • AKS • Accounting • SAP security
- 7. Devon - Internal 7 • Traditional SIEM problems: • Retention • Parsing - legacy logs are terrible • WEF is great, until it isn’t • Just enough normalization • Enrichment > Correlation • Atomic indicators • Identity information • Cluster > purpose collectors • Logstash nodes are cattle, not pets • Puppet was good, K8s is great • Throughput vs. node count Logstash - The Core of Our FrankenSIEM
- 8. Devon - Internal 8 Intent : Security should not be managing it What’s cool about ECE • It saves Time : Streamline scaling, securing, upgrading the stack • Great way to enhance utility and value was to grant other teams access to their logs • Centralized management of logstash pipelines and cluster user settings is cool • Inbuilt monitoring gives a great insight It was very different than traditional ES • Troubleshooting is a bit different • Elasticsearch.yml is not to be trifled with • Initially, had to rely a lot more on support Why ECE 8
- 9. Devon - Internal 9 Indexing throughput • Use dedicated Master Nodes • Index refresh interval refresh_interval : 30s (even -1 for initial load) • Tune Indexing buffer size OOMs • Give JVM 50% of available memory • Prevent JVM resizing : minHeap=maxHeap Cautiously budget your cache • Limit and monitor Field Data cache : it never goes away • Using circuit breaker can save the day Disk Sizing • Monitor number of replica for your use case • Experiment with Sharding : Larger shards means better indexing rate but needs can vary; more shards come with cost Appraise your precedence, go Quid-pro-quo • Memory Intensive queries vs near-real-time data vs long-term retention Major Gotchas
- 10. Devon - Internal 10 • It hurt being the first Azure customers for ECE • Docker ignorance didn’t help • ECE 1.1.x supported specific version of docker with specific version of kernel • Updates tanked the cluster at-least twice – multi zone helps • Problems at our scale: • Migrating shards between containers takes days • In-place container upgrades: advanced configuration will save your day/week • Moving to dedicated master nodes – election problems • Unexpected socket hang ups, difficult to diagnose • Monitoring: • Independent Monitoring cluster • Created our own health check scripts What We Learned – Operational Glitches
- 11. Devon - Internal 11 What’s Next For Us Chugging the Docker and Kubernetes Kool-Aid: • Curator • Enrichment data collection • Logstash + autoscaling • Health checks Dedicated nodes: • ML Nodes • Search Only Nodes • Ingest nodes Separate security and non-security to their own spaces SAML / OpenIDc with MFA
- 12. Thank you. Come call BS at Happy Hour Contact info: @p4ulpc – [email protected] [email protected] Code: https://github.com/paulpc/elasticon