Karandeep Anand
Vancouver, British Columbia, Canada
803 followers
500+ connections
About
Lead Data/Cloud Engineer
Extensive experience in Banking and Financial domain in Analytics/Business Intelligence domain.
Innovative strategist with track record of simplifying and transforming legacy data platform to build AWS Cloud based data platform to achieve cost savings / operational efficiency.
Worked on creating different cloud infrastructures, architectures and delivery to build next generation, scalable Cloud ecosystem using AWS/Azure Cloud/GCP and Big Data tools/technologies
Strong working knowledge of Data Warehousing techniques and Concepts experience including ETL processes, Star Schema/Snowflake Schema and Database performance tuning.
Extensive experience in client handling including Product owners, System Integrators, Business, IT, Architects.
Experienced in cloud architecture and data engineering with a demonstrated history of working in various industries including banking, healthcare, retail, oil & gas, insurance, and forestry.
Significant global experience in enterprise solution delivery in complex organizations and virtual environments. Local work experience in different countries including United States of America, India and Canada.
Activity
-
Former Infosys exec joins Siemens as CIO https://lnkd.in/gZcapGhk
Former Infosys exec joins Siemens as CIO https://lnkd.in/gZcapGhk
Liked by Karandeep Anand
-
Here it is! We have 120Mhz and 60mhz variants available at www.toolworld.in
Here it is! We have 120Mhz and 60mhz variants available at www.toolworld.in
Liked by Karandeep Anand
-
It’s been an incredible experience working at Capital One, Canada. I’m really going to miss this place. Thank you for the memories, the challenges…
It’s been an incredible experience working at Capital One, Canada. I’m really going to miss this place. Thank you for the memories, the challenges…
Liked by Karandeep Anand
Experience
-
-
-
-
-
-
-
-
-
-
-
-
-
Education
Licenses & Certifications
-
-
-
Abinitio basic
-
Languages
-
Punjabi
-
-
Hindi
-
-
English
Full professional proficiency
More activity by Karandeep
-
Looking forward to an excellent opportunity to be part of a panel and engage in meaningful discussions with like-minded professionals at the upcoming…
Looking forward to an excellent opportunity to be part of a panel and engage in meaningful discussions with like-minded professionals at the upcoming…
Liked by Karandeep Anand
-
Was fantastic to be there today for this very exciting launch! Kudos to our very own Innovation team and the team from A&K Robotics #yvr…
Was fantastic to be there today for this very exciting launch! Kudos to our very own Innovation team and the team from A&K Robotics #yvr…
Liked by Karandeep Anand
Other similar profiles
Explore more posts
-
Rohan Rekhi
Here's my latest article on Medium for various aspects of Snowflake Technology Infrastructure which needs to be managed and governed . I will be diving into each of these aspects in my upcoming articles . Please feel free to subscribe , share and comment. Reach out for any professional help or consulting services in these areas . #snowflakecm #snowflakeintrastructure #snowflakedevops
-
Shailesh Mishra
🚀 Optimise INSERT, UPDATE and DELETE in PostgreSQL using best indexing strategy! 📊 Struggling with sluggish DML operations due to indexes? Refer my new medium blog https://lnkd.in/gTxmyWaY which elaborates best indexing strategies to boost your PostgreSQL performance, especially for tables in the TB range. 📈 1. Evaluate and Refine Index Types: - 🧹 Review Index Usage: Identify and drop unused indexes using `pg_stat_all_indexes`. - 🔍 Choose the Right Index Type: Use B-tree for high-cardinality columns and BRIN for range queries on naturally sorted columns. - ✂️ Partial Indexes: Index only frequently queried subsets to reduce size and improve speed. - 📚 Covering Indexes: Ensure indexes cover common queries to minimize extra lookups. 2. Remove Redundant Indexes: - 🗑️ Unused indexes can slow down operations. Regularly review and drop them to save space and enhance performance. 3. Advanced Index Types: - ⚡ Expression Indexes: Index results of expressions or functions for efficient querying based on calculations. - 🧩 Partial Indexes: Optimize for specific data subsets to reduce index size. 4. Use Efficient Index Types: - 🏷️ BRIN Indexes: Ideal for large, naturally sorted tables. Efficient for range queries. - 🔢 HASH Indexes: Perfect for exact-match queries on large datasets. 5. Reorganize Indexes: - 🔄 Reindex During Off-Peak Hours: Schedule maintenance to minimize impact. - ⚙️ REINDEX CONCURRENTLY: Reduce downtime by using PostgreSQL's concurrent reindexing. 6. Consider Index Maintenance Tools: - 🛠️ pg_repack: Reorganize indexes without locking tables using this powerful extension. - 🌐 Logical Replication: Perform maintenance on a replica, then promote it to primary. 7. Tune Configuration Parameters: - 🔧 Increase maintenance_work_mem: Speed up index operations by allocating more memory. By following these strategies, you can ensure your PostgreSQL database performs optimally, even with massive datasets. Stay tuned for more tips on table structure optimization, partitioning, and sharding in our future posts! 🚀📈 #SQL #DatabaseOptimization #PostgreSQL #DataManagement #TechTips
-
Ishan Mehta
🌟 Certificate Accomplishment 🌟 Course - Get Started with Data Analysis on Databricks This is an introductory course for data analysts onboarding onto the Databricks Lakehouse Platform. In this 2-hour video series, you'll learn about the functionality offered by Databricks SQL, as well as how to use Databricks SQL to complete basic daily workflows. #certification #databricks #SQL #DataLakehouse
5 Comments -
Mark H.
AWS Glue Data Quality: Elevate Your ETL Pipelines with Dynamic Rules AWS Glue has significantly enhanced ETL pipeline data quality with dynamic rules that automatically adjust to changing data properties. This innovation ensures your data remains accurate and reliable, crucial for complex data environments. Technical Highlights: Dynamic Rule Adjustment: Leveraging the last(k) operator, these rules adapt based on historical data trends, maintaining high data integrity. Diverse Validation Options: Includes rules like CustomSQL for custom checks, Mean for statistical validations, Sum for aggregations, RowCount for completeness, and DistinctValuesCount for uniqueness. Integration Ease: Set up effortlessly using AWS CloudFormation and manage through Glue Studio for streamlined operations. Dynamic rules support a range of validation scenarios: CustomSQL: Craft specific SQL queries for tailored data checks. Mean and Sum Rules: Validate statistical properties and aggregations within your data sets. RowCount and Completeness: Ensure comprehensive data validation by checking the number of rows and data completeness. DistinctValuesCount: Maintain data uniqueness, crucial for deduplication processes. Embrace these dynamic rules to maintain superior data quality in your ETL pipelines, adapt to evolving data patterns, and ensure robust data governance. This advancement is a game-changer for businesses relying on consistent and high-quality data. #AWSGlue #DataQuality #ETL #BigData
-
Darrius Wright
Triggers are one of the most critical components to controlling cost in Spark Structured Streaming. Consider the example of streaming data from Kafka into S3. Neglecting the trigger option means writing to S3 up to the default of 500ms for each interval. As compared to assigning the processingTime option of the trigger to 5 mins, which could mean the difference of 600x files or more depending on your partition strategy. Be sure to consider your SLA when determining streaming triggers, and save yourself time and money! Check out the docs: https://lnkd.in/gbuDuJRx #spark #structuredstreaming
-
Wasif Kabeer
*Enabling batch+streaming integration in on-premises data platforms* Modern cloud-based analytics platforms support batch & streaming data ingestion, referred to as cold+hot paths or delta/lambda architecture. However, enabling this capability on on-premises data platforms requires careful considerations. Here are key aspects to consider while planning cold+hot data pipelines in a unified on-prem data platform. 1) Assessment This is a fundamental step to assess gaps in tools & skills before the kickoff. 2) Latency A data refresh frequency should be defined based on business needs. e.g., milliseconds-to-seconds for streaming, minutes-to-an-hour for micro-batches & hours-to-a-day for batch integrations. 3) Impact on source systems Properly assessing the impact on source systems & aligning with software engineers is crucial, to avoid any unpleasant surprises during pipelines execution. 4) Design patterns for data curation Various design patterns can be adopted to logically organize the data, such as Databrick's Medallion architecture (bronze, silver, gold), Teradata's IDW architecture (staging, core, semantic), etc. The design pattern should support quick time to market, a unified enterprise-wide view, timely refresh, enhanced data quality, ease of implementation, data transformation & enrichment. 5) Workload management Enterprise data platforms are used by a variety of users. It is vital to ensure proper workload management, especially for frequent jobs with low-latency data requirements & complex jobs with significant data requirements. Allocating an off-peak nightly window outside of peak business hours for heavy batch processing pipelines is usually recommended. For micro-batch & streaming pipelines with distinguishing nature, proper quota & resource pooling should be applied to ensure optimal utilization of system resources. 6) Orchestration & scheduling Batch pipelines have been around for a while, so data engineers are familiar with scheduling best practices. Micro-batch pipelines inherit many features from batch pipelines; however, streaming pipelines need to be continuously "listening" & ingest data (with re-try features) as it is created from source to destination. 7) Data security & privacy The data security & privacy should be implemented by design to ensure regulatory compliance. 8) Scalability The infrastructure & data curation design should ensure scalability across system resources, users & all types of data pipelines. 9) Observability There should be a built-in capability to capture pipeline execution metadata, logs, alerts & other pipeline features to ensure proactive monitoring. 10) Data retention Batch pipelines handle large data volumes, while micro-batch & streaming pipelines handle lower data volumes. To ensure efficient pipeline execution, proper data retention policies must be applied. 11) Documentation Regardless of the refresh frequency, adequate documentation should be created. If I missed anything, pls comment.
7 Comments
Explore collaborative articles
We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
Explore MoreOthers named Karandeep Anand
5 others named Karandeep Anand are on LinkedIn
See others named Karandeep Anand