Security Events Logging at Bell with the Elastic Stack
13 likes13,315 views
The document outlines a comprehensive approach to security events logging, emphasizing the need for scalable log management solutions to handle increasing volumes of security data. It discusses the architecture for logging, data storage, and visualization, employing tools like Kafka, Logstash, and Elasticsearch to create a robust, automated pipeline. Additionally, it highlights the importance of secure data handling and smart detection through machine learning and correlation engines.
Security Events Logging at Bell with the Elastic Stack
- 1. 1 Sylvain Proulx Mathew Vandystadt October, Date, 2018 Security Events Logging at
- 2. 2 Sylvain Proulx 18 years in security Senior Security Manager Who are we Mathew V. 5 years in security Security Specialist Software Engineer
- 3. 3 Our Mission • Ingest all security logs • Enrich, normalize, analyze, and contextualize • Automation • Build threat model detection • Visualize security data
- 4. 4 What’s going on Our current challenges
- 5. 5 Volume of logs keeps increasing Challenges Normalization of many new types of logs No one stop solution for our needs STOP
- 6. 6 More data means more alerts Challenges Limited amount of analysts Limited detection mechanism
- 7. 7 Challenges Share logs between different branches Own our security data Secure the data
- 8. 8 An in-depth look at our solution Building our Pipelines
- 9. 9 Building One Piece at a Time Logging 1 2 3 4 Data engineering Log storage and long term retention Visualization and alerting End to end solution
- 10. 10 Where Our Data Comes From • Bare metal servers • Virtual machines • Containers 1 2 3 4
- 11. 11 Requirements For Our Log Shippers • Simple way to ship logs • Something that can buffer logs in case of outage • Something that’s lightweight, but gives us the possibility to perform light filtering at the source • Something uniform throughout our fleet • Automated deployment capability 1 2 3 4
- 12. 12 Filebeats and Winlogbeats • Generic beats configuration per service logged • Simple installation and configuration • Minimal impact on systems • No loss of data in case of network outage 1 2 3 4
- 13. 13 Adding Beats to Our Architecture Diagram 1 2 3 4
- 14. 14 Being an ISP • Large quantity and variety of network devices • Unique ISP applications • Logs also come from security devices • Network devices can be very chatty 1 2 3 4 Different data sources to consider that other businesses don’t
- 15. 15 What If Beats Can’t Handle Special Cases? • Most of the devices send logs only via syslog • Losing data is not an option • Need to receive data from geographically diverse locations 1 2 3 4
- 16. 16 Rsyslog • Adding Rsyslog servers close to data sources • Acts as buffer • Basic parsing and serialization in JSON of logs with Rsyslog • Send logs to our security data center in TCP and minimize the risk of data loss 1 2 3 4
- 17. 17 Adding Rsyslog to Our Architecture Diagram 1 2 3 4
- 18. 18 Building One Piece at a Time Logging 1 2 3 4 Data engineering Log storage and long term retention Visualization and alerting End to end solution
- 19. 19 Incoming Logs • All logs are serialized in JSON • The ability to sustain large spikes of traffic without over provisioning • Buffer data allowing for higher availability • Data accessible to multiple consumers 1 2 3 4 Our past experiences and requirements
- 20. 20 Kafka as Our Message Queue • Kafka allows us to handle spikes of logs • Provide data buffering for potential downstream issue • Provide controls to share data securely across other teams using open formats • Kafka supports JSON out of the box • Rsyslog and Beats can write to Kafka 1 2 3 4 Our past experiences and requirements
- 21. 21 Adding Kafka to Our Architecture Diagram 1 2 3 4
- 22. 22 Parsing and Normalizing • Use resources efficiently by taking advantage of auto-scaling • Every unique technology requires it’s own set of configuration for parsing and normalization • Needs integration of CI/CD for ease of test and deployment 1 2 3 4 Our past experiences and requirements
- 23. 23 Logstash on Openshift • We decided to run all our logstash instances on openshift • Containers consumes less resources than multiple virtual machines • We get auto scaling through openshift • We can scale quickly by adding more nodes if needed to our openshift cluster. 1 2 3 4 Logstash containers
- 24. 24 Adding Openshift and Logstash to Our Architecture Diagram 1 2 3 4
- 25. 25 Logstash on Openshift • Centralize configurations in Gitlab • Gitlab allows us to create CI pipelines quickly • Run Logstash configurations through rspec for testing • Review and deploy to production on merge requests • Openshift provides the ability to build CD pipelines 1 2 3 4 Logstash CI/CD
- 26. 26 Adding CI/CD to Our Architecture Diagram 1 2 3 4
- 27. 27 Building One Piece at a Time Logging 1 2 3 4 Data engineering Log storage and long term retention Visualization and alerting End to end solution
- 28. 28 Log Storage • Most the searching is going to be done the same day • Documents need to be easily searchable for the previous 90 days • Horizontal scalability • Highly available and redundant data 1 2 3 4 Our past experiences and requirements
- 29. 29 Log Storage • No real surprise, we store our logs in elasticsearch • Implementing the Hot-Warm architecture provides the best solution to meet our requirements • Our process allows for automated deployment of new nodes • Elasticsearch provides the required HA and redundancy 1 2 3 4 Elasticsearch
- 30. 30 Adding Elasticsearch to Our Architecture Diagram 1 2 3 4
- 31. 31 Long-Term Data Retention • For forensic and legal issues, data needs to be stored for a minimum of 12 months • Needs to be stored outside of the elasticsearch cluster • Fast retrieval of data in the existing elastic cluster • Minimize cost for long-term storage solution 1 2 3 4 Our past experiences and requirements
- 32. 32 Long-Term Data Retention • Openstack Swift allows us to store our index snapshots in object storage • Reusability of S3 snapshot plugin from elasticsearch • Acceptable retrieval times • Use of curator to automate snapshots 1 2 3 4 S3 object storage
- 33. 33 Adding S3 Storage to Our Architecture Diagram 1 2 3 4
- 34. 34 Securing Data • Control over who has access to the data • Ease of RBAC management • Add layer of encryption over data transportation • Use of existing and tested solutions 1 2 3 4 Our past experiences and requirements
- 35. 35 Adding X-Pack to Our Architecture Diagram 1 2 3 4
- 36. 36 Building One Piece at a Time End to end solution Logging 1 3 42 Data engineering Log storage and long term retention Visualization and alerting
- 37. 37 Handling and Visualization Our Data • Easy front-end to query logs • Reusable query • Ability to meaningfully visualize data • Front-end that’s used by a wide range of security specialists ‒ Analysts ‒ Threat hunters ‒ Data scientists Our past experiences and requirements 1 2 3 4
- 38. 38 Adding Kibana to Our Architecture Diagram 1 2 3 4
- 39. 39 Alerting on Security Events • Need to filter on meaningful security events • Ease of building and deploying detection rules • Automate deployment • Easily track life cycle of rules Our past experiences and requirements 1 2 3 4
- 40. 40 Alerting on Security Events • Simple way of writing queries • Use of YAML text files solves maintainability issues with version control tools • Auto deployment through CI/CD tools tied to version control Elastalert 1 2 3 4
- 41. 41 Adding ElastAlert to Our Architecture Diagram 1 2 3 4
- 42. 42 Smart Detection • Data must be easily accessible • Develop custom machine learning models • Automated deployment of machine learning models • Flexibility in using different algorithms Our past experiences and requirements 1 2 3 4
- 43. 43 1 2 3 4 Smart Detection In-house machine learning • Models developed with open source, ML centric libraries • Deployment pipeline from data scientists to production
- 44. 44 Adding Machine Learning to Our Architecture Diagram 1 2 3 4
- 45. 45 Security Event Correlation • Ability to correlate security events • Ability to write complex rules • Simple front end to help our analysts • Central point for alerting Our past experiences and requirements 1 2 3 4
- 46. 46 Security Event Correlation • Provides one of the best correlation engines for security events • Allows for aggregation, correlation, trending, and more • ESM provides a GUI and it’s a well known product throughout Bell security teams • Can receive and send data to multiple sources Arcsight 1 2 3 4
- 47. 47 Adding Arcsight to Our Architecture Diagram 1 2 3 4
- 48. 48 Today’s Situation With Elastic • Elastic allows for horizontal scaling to support constant increase of log volume • Elastic allows for simple integration with open security protocols • Elastic’s X-Pack solution provides a built-in secure data environment • New architecture using elastic allows us to build more detection mechanism using different techniques Where we at STOP 1 2 3 4