Mohamed Khalil JAMAI

𝐃𝐞𝐚𝐫 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐬 🔥 Here I come with another PySpark Transformation Challenge 📣 𝐈𝐧𝐟𝐨: You have a DataFrame with two columns: 𝐢𝐝 and 𝐩𝐥𝐚𝐧_𝐝𝐚𝐭𝐞. 𝐘𝐨𝐮𝐫 𝐭𝐚𝐬𝐤 𝐢𝐬 𝐭𝐨 𝐜𝐫𝐞𝐚𝐭𝐞 𝐭𝐡𝐫𝐞𝐞 𝐧𝐞𝐰 𝐜𝐨𝐥𝐮𝐦𝐧𝐬: 1. 𝐈𝐒_𝐋𝐀𝐒𝐓_𝐒𝐈𝐗_𝐌𝐎𝐍𝐓𝐇𝐒: For each id, check whether the date is within the last 6 months compared to the previous row for that id. The first row for each id should always be "YES", and subsequent rows should check if the previous date is less than 6 months before. If it is within 6 months, mark it as "YES", otherwise, mark it as "NO". 2. 𝐈𝐒_𝐂𝐔𝐑𝐑𝐄𝐍𝐓_𝐅𝐈𝐍𝐀𝐍𝐂𝐈𝐀𝐋_𝐘𝐄𝐀𝐑: For each id, check whether the date belongs to the current financial year (April 1, 2024 - March 31, 2025). 3. 𝐈𝐒_𝐂𝐔𝐑𝐑𝐄𝐍𝐓_𝐘𝐄𝐀𝐑: For each id, check whether the date belongs to the current calendar year (January 1, 2024 - December 31, 2024). 𝐂𝐫𝐞𝐚𝐭𝐞 𝐒𝐚𝐦𝐩𝐥𝐞 𝐃𝐚𝐭𝐚𝐟𝐫𝐚𝐦𝐞 (Input Data): 𝘥𝘢𝘵𝘢 = [   (1, "2024-08-15"),   (1, "2024-03-10"),   (1, "2023-05-05"),   (2, "2025-02-24"),   (2, "2024-07-15"),   (2, "2024-03-15") ] 𝘤𝘰𝘭𝘶𝘮𝘯𝘴 = ["𝘪𝘥", "𝘱𝘭𝘢𝘯_𝘥𝘢𝘵𝘦"] 𝘥𝘧 = 𝘴𝘱𝘢𝘳𝘬.𝘤𝘳𝘦𝘢𝘵𝘦𝘋𝘢𝘵𝘢𝘍𝘳𝘢𝘮𝘦(𝘥𝘢𝘵𝘢, 𝘤𝘰𝘭𝘶𝘮𝘯𝘴) 𝐏𝐲𝐒𝐩𝐚𝐫𝐤 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧 𝐂𝐨𝐝𝐞? (Expected Data) Let me know the answer in the comments 😀 #pyspark #dataengineering

81

6 commentaires

Here are 3 reasons why I am switching my content in English : (Petit sondage à la fin!) Little survey at the end 👇 3 reasons : 1️⃣ Industry Standard: In the tech and data fields, most of the literature, tools, and conversations happen in English. 2️⃣ Access to the Latest Information: Most cutting-edge research, case studies, and innovations in data science, machine learning, and AI are published in English first. 3️⃣ English-language content offers a wider variety of educational resources, from technical documentation and tutorials to expert blogs and podcasts. For my information, what if your level of english ? Celebrate (Bravo) : fluent, I can make jokes in English Love (Coeur) : professional level, I can speak and understand easily Insightful (Idée) : basic level Like (Pouce bleu) : difficult but I try to manage Support (Soutien) : je ne connais pas l'anglais Funny (Drôle) : je fais mes requêtes SQL en français

13

3 commentaires

🎓 After graduation, I set out to achieve my dream: starting a beauty salon software company in Morocco. Here's what happened (Part 4): March 2024 – I had been working as a software developer for six months. Finally, we launched the big project we had been working on for months at the company. At the same time, I finished a stable version of my software and was ready to show it to beauty salons in Morocco for feedback. But then, our work setup changed from fully remote to hybrid, with one week in the office and one week online. I faced a tough choice: -Delay my startup to save more money. -Start right away, with enough funds to keep going for one year. I asked myself: "Would I ever ever regret leaving my job if my startup failed and I had to find a new job in a year?" My answer was 'No.' I calculated the worst-case (no revenue, no part-time jobs) and best-case scenarios (finding part-time work, some revenue). I spent a lot of time thinking and getting advice from friends. It was a hard decision. In the end, I knew I wouldn’t regret trying. So, I quit my job, and shortly after I have came across a lot of part time opportunities that help me currently sustain the project. In Part 5, I will be sharing the first struggles after leaving my secure job. #Entrepreneurship #StartupJourney #CareerChange #FollowYourDreams #TechInBeauty #Morocco #LifeDecisions #Inspiration #SoftwareDevelopment #BusinessLaunch #StayTuned

8

2 commentaires

I'm thrilled to have earned my Hands-On Essentials: Data Engineering Workshop credential from Snowflake! 🚀 This workshop was an incredible learning experience, providing hands-on skills and insights crucial for data engineering. this workshop was packed with hands-on labs and interactive sessions. Excited to apply this knowledge to real-world challenges! #DataEngineering #Snowflake #ContinuousLearning #TimezonesandTimestampFormats #CTAS #SnowflakeStreamsCDC #SnowflakeTasks #Parsing #JSON with Paths and Casts #QueryProfiles #SQL_Merge_Statements #Task History and Dependency #Snowpipe #SQL_Window_Function #Snowflake #Dashboards #Storing #Metadata #Advanced_SQL

50

5 commentaires

These are very important points to understand how to work in data engineering while handling BIG data. Thanks Zach Wilson for summarizing it.

2

New tool bar with nice order, ability to fold/unfold and what I like even more : possibility to search a tool by name. Nice update on Amphi last release. Good job Thibaut Gourdel and thanks for listening to users (especially when the user is myself). For everyone, if you don't know Amphi : it's a (young and WIP) visual python-based ETL. https://amphi.ai/

9

I recently deployed a partial Data ML stack in the Hetzner cloud using bare-metal Kubernetes. Currently, 24 developers and ML engineers from the Basel infrastructure team are using it. To ensure we stay within a strict budget of $200, with a maximum threshold of $500, I implemented several optimizations: - For each JupyterHub user, I allocated 1 CPU and 2 GB of memory. To optimize resource usage, I disabled the terminal and provided pre-installed Jupyter templates. These templates are heavily optimized for specific jobs and do not support all pip libraries, keeping the image size small and controlling compute resources. - I'm running Apache Superset on just 2 CPUs and 4 GB of RAM. Surprisingly, it's performing well, although traffic isn't heavy. I configured middleware to reject new logins if CPU or memory usage reaches critical levels. - Auto-scaling is enabled with extremely fast node boot-up times. The cheapest nodes proved less useful, so I switched to medium-sized nodes. We average 10 nodes, with peaks up to 25 nodes. - I used Hetzner volumes for persistence and leveraged LocalStack for AWS services. This setup works amazingly well and is extremely fast. Each user has their own 5 GB of storage space. - DuckDB has a dedicated 100 GB space for testing, costing around $1.50. It's configured to use an entire node with affinity settings, allowing us to monitor traffic and CPU spikes. I may consider using a dedicated node for DuckDB in the future. - Streaming data is now reduced to 1 MB/sec, and data is retained for two days before being dropped. This helps control egress costs, and we appear to be within budget.

60

4 commentaires

Yesterday, I spent all day at various events. First in line was Snowflake, organised by Infotrust. There, I finally met Mathias Granberg face-to-face, talked to Ajith Menon Kidaparambil, and saw Olli Ek in action (in the photo, he looks like an angry Viking, but he's super chill and not that intimidating like he looks in the photo 😂). I see that, finally, we have some Snowflake love in Lithuania! Yeah, I also talked about my experience across different companies with Snowflake (I consider myself technology agnostic, so if you have some questions I can answer from my own experience - feel free to DM me. I'm not affiliated with Snowflake). After a quick lunch with Edvinas Radvilavičius, we rushed to a Google event about marketing data analytics. Interestingly, I have almost no experience in marketing analytics, so it was an excellent place to learn from the industry experts. I had no idea about counterfactual testing, its application, and the difference from A/B testing. Apart from learning, I got out with a Google sweatshirt (who doesn't like swag?). The fun thing is that Flo Health Inc. and Databricks are also organising an event in Vilnius today! Various data communities seem to be coming back from the dead and making a big bang! If you're coming to today's event, say hi. I'm super excited to nerd out on various data topics! #learning #knowledge #sharing #community #data

47

12 commentaires

𝐃𝐞𝐚𝐫𝐬 𝐌𝐢𝐜𝐫𝐨𝐬𝐨𝐟𝐭 𝐅𝐀𝐁𝐑𝐈𝐂 𝐚𝐧𝐝 𝐏𝐨𝐰𝐞𝐫 𝐁𝐈 𝐅𝐚𝐧𝐬 🧭 May I share with you 1) on architecture side, an advise around => 𝐢𝐧𝐭𝐞𝐫𝐧𝐚𝐥 𝐞𝐱𝐢𝐬𝐭𝐢𝐧𝐠 𝐒𝐐𝐋 "𝐁𝐈" 𝐝𝐚𝐭𝐚 (somewhere on my company DWH or Lakehouse) consumption on Power BI semantic models ✔️Avoid a 𝐝𝐚𝐭𝐚𝐟𝐥𝐨𝐰 𝐛𝐞𝐭𝐰𝐞𝐞𝐧 your SQL DWH/LH domain data product and your Semantic models (also data product ? ...another discussion 😁) 2) and a "big" requirement for Microsoft ==> 𝐓𝐡𝐞 "𝐦𝐚𝐠𝐢𝐜" & "𝐠𝐥𝐨𝐛𝐚𝐥" 𝐄𝐱𝐭𝐞𝐫𝐧𝐚𝐥 𝐝𝐚𝐭𝐚 𝐬𝐨𝐮𝐫𝐜𝐞 𝐇𝐔𝐁 (the famous universal data broker) around 𝐄𝐱𝐭𝐞𝐫𝐧𝐚𝐥 to 𝐅𝐀𝐁𝐑𝐈𝐂 𝐃𝐚𝐭𝐚 (ERP , SAP , Folders , SQL , API access token , Bigquery DWH , Snowflake DWH , Databricks etc ..) usages ==> a 𝐇𝐔𝐁 access also on pbi desktop ✔️ Useful : On all "ingestion type" artifacts , a "one click"<== ready to use usage ✔️Don't forget pbi desktop please MS <== currently we give Bigquery service account secrets to PBI Desktop builders persons - not safe for bigquery ✔️Isolation : External data sources are already setuped (and shared with me) somewhere by domain owners ✔️ FABRIC consumers are data factory dfgen2 & Data pipeline , PBI semantic model & data flow , notebook , mirroring setup , event stream , T SQL stored proc , python script , shortcut etc.. ✔️ Datamesh governance compliant : By domain management ✔️ Security : Avoid to transmit credentials to several persons ✔️ Cloud or on prem data sources (offer a 360° vision) ✔️ Credentials fill in directly or Azure key vault system access or (managed the SAAS access to external BI system ? ) This day INTEGRATION word 'll reach a strong step on Fabric #data #dataanalytics #datagovernance #powerbi #microsoftfabric #bi #azure #microsoft #DataTransformation #DataCulture #BusinessExperts #DataAcculturation #DataAwareness #DataMesh #DataCatalog

5

2 commentaires

𝐃𝐞𝐚𝐫𝐬 𝐌𝐢𝐜𝐫𝐨𝐬𝐨𝐟𝐭 𝐅𝐀𝐁𝐑𝐈𝐂 𝐚𝐧𝐝 𝐏𝐨𝐰𝐞𝐫 𝐁𝐈 𝐅𝐚𝐧𝐬 🧭 May I share with you 1) on architecture side, an advise around => 𝐢𝐧𝐭𝐞𝐫𝐧𝐚𝐥 𝐞𝐱𝐢𝐬𝐭𝐢𝐧𝐠 𝐒𝐐𝐋 "𝐁𝐈" 𝐝𝐚𝐭𝐚 (somewhere on my company DWH or Lakehouse) consumption on Power BI semantic models ✔️Avoid a 𝐝𝐚𝐭𝐚𝐟𝐥𝐨𝐰 𝐛𝐞𝐭𝐰𝐞𝐞𝐧 your SQL DWH/LH domain data product and your Semantic models (also data product ? ...another discussion 😁) 2) and a "big" requirement for Microsoft ==> 𝐓𝐡𝐞 "𝐦𝐚𝐠𝐢𝐜" & "𝐠𝐥𝐨𝐛𝐚𝐥" 𝐄𝐱𝐭𝐞𝐫𝐧𝐚𝐥 𝐝𝐚𝐭𝐚 𝐬𝐨𝐮𝐫𝐜𝐞 𝐇𝐔𝐁 (the famous universal data broker) around 𝐄𝐱𝐭𝐞𝐫𝐧𝐚𝐥 to 𝐅𝐀𝐁𝐑𝐈𝐂 𝐃𝐚𝐭𝐚 (ERP , SAP , Folders , SQL , API access token , Bigquery DWH , Snowflake DWH , Databricks etc ..) usages ==> a 𝐇𝐔𝐁 access also on pbi desktop ✔️ Useful : On all "ingestion type" artifacts , a "one click"<== ready to use usage ✔️Don't forget pbi desktop please MS <== currently we give Bigquery service account secrets to PBI Desktop builders persons - not safe for bigquery ✔️Isolation : External data sources are already setuped (and shared with me) somewhere by domain owners ✔️ FABRIC consumers are data factory dfgen2 & Data pipeline , PBI semantic model & data flow , notebook , mirroring setup , event stream , T SQL stored proc , python script , shortcut etc.. ✔️ Datamesh governance compliant : By domain management ✔️ Security : Avoid to transmit credentials to several persons ✔️ Cloud or on prem data sources (offer a 360° vision) ✔️ Credentials fill in directly or Azure key vault system access or (managed the SAAS access to external BI system ? ) This day INTEGRATION word 'll reach a strong step on Fabric #data #dataanalytics #datagovernance #powerbi #microsoftfabric #bi #azure #microsoft #DataTransformation #DataCulture #BusinessExperts #DataAcculturation #DataAwareness #DataMesh #DataCatalog

13

1 commentaire

A good view on Tools for Realtime Data Projects .

3

Excited to share a RAG LLM Quiz Generator I worked on for next week's Databricks' #DAIS2024 ! Discover how LangChain and a vector database can come together to create dynamic, context-aware quizzes for any topic you input, complete with incorrect distractor answers, explanations for all answers and links to sources for each. Because we're going to be showing this off at DAIS 2024 next week, this flavor of the quiz is based on... Databricks! Try The Quiz For Yourself: https://lnkd.in/e9WiuUdj Read our blog post fully breaking down the techniques and technologies & services used to implement this on Databricks : https://lnkd.in/efyPyzJp Or if that's not enough, take the notebook that powers this pipeline and try it for yourself in your Databrick's environment: https://lnkd.in/eS7Fnxex We'll be there all week at Booth E9, if you want to know more about how we built this on Databricks, some of the other projects were working on or are curious how Rearc may be able to support your Generative AI endeavors - stop by and say hello! #Rearc #DataAISummit #LLMWhisperer

28

11 commentaires

Suhas Yogish new innovations in building front ends for spark ? what's your opinion of this given your depth in Streamlit Suhas? it looks like even hacker like me can create a fast beautiful UI for different Spark insurance and wealth management product models from coherent.global. I could see this stock as a way of rapidly building apps that are highly custom, user-centric, and very simple to use. so instead of trying to jam everyone in a general user experience.. build out more all driven from a "product brain" when we were building major frameworks back in 2010... we dreamed of building domain specific model driven applications... only implemented them in insurance and wealth management companies they cost millions of dollars to create and millions of dollars to maintain. And the software was at most average. It seems to me that enterprises have to rethink how they build their components and use front end tools for speed and utility that many subject matter experts can use, and then they should be domain specific and use models about their products that they sell. and I think it's really cool that we can instantiate those models using Excel which again is broadly understood by subject matter experts like it actuaries and underwriters and that model enables ease of use applications for agents and financial advisors. Use the right tools for the right purposes right?!🤘 https://lnkd.in/gM-V9PVg #tools #customsoftware

4

2 commentaires

France plays a significant role in the field of artificial intelligence A lot of French firms are driving global advancements in AI and NLP. Here are some key players: ↳ Hugging Face: Transforming NLP with their open-source library and work closely with OpenAI. ↳ Mistral AI: Specializes in advanced language models and AI technologies. ↳ Kyutai (Moshi): Innovates in natural language processing and other AI applications. ↳ Dataiku: Their platform is widely used for data science and machine learning. We also see many French experts at major companies like OpenAI (Romain Huet Head of Developer Experience) and Meta (Yann LeCun Director of AI Research) How do you see French innovation shaping the future of AI ?

59

46 commentaires

How bad is it doc? 🤒 In case you missed it, last month my approach to the SLB #SemanticSegmentation challenge for defect detection on Challenge Data platform by Ecole normale supérieure and Collège de France proved to be the winning one at the mid-season breakdown. If you think making CNNs is too complicated for you and you shy away from embarking on the #DeepLearning journey, like I did 4 months ago, let me try to persuade you otherwise. On my website 🔽, I have shared what I learned along the way - approaches towards data preprocessing, network architecture and training that allowed me to score 0.67 IoU for the challenge of labelling defects on ultrasound images. It was a learning process and it still is, so if you have any questions or remarks, hit me up and let’s grow together. 🌳 (Credits: U-Net architecture diagram; Long, Shelhamer, and Darrell, 2014)

42

7 commentaires

☀ Blog1 : Embarking on My PySpark Journey as a Data Engineer 🔥 Hi everyone, I am excited to embark on this journey of learning and exploring PySpark with all of you, starting with a deep dive into the fundamentals of PySpark and will be covering everything from initial setup to advanced level and I'm happy to hear any ideas, help, or fixes you have along with this journey. ✨ First Blog Topic: Before we dive into the world of PySpark, lets explore the basic introduction of Apache Spark.     1. What is Spark? 👉 Spark is designed to support wide range of task over the same computing engine. It is a unified analytics engine for large-scale distributed data processing. 👉 It is an open-source cluster computing framework, which handles both batch data and streaming data  👉 It does not have any separate storage or file system, it is limited to a computing engine       👉 Spark is much faster than Hadoop, because it does in-memory processing (spark hold the data in memory for processing while the Hadoop writes the data back to disk and read it again from disk to in-memory) 👉 Spark can work with different resource manager/Cluster manager. For eg. 🔥 Yarn 🔥 Mesos 🔥 Kubernetes 🔥 Standalone (Spark own cluster) 👉 Spark provides high level API or libraries for different languages like Python, Java, Scala and R    2. Why Spark?         👉 Speed: Spark can process large volumes of data at fast speed.        👉 Easy to use: Spark offers different languages and libraries as per the need of user.        👉 Versatility: It is not limited to one specific task, it can handle all sorts of tasks like analyzing data, running machine learning algorithms and processing streaming data in real-time.         👉 Reliability and scalability: Spark is highly reliable with built in fault tolerance mechanisms and scales seamlessly to handle increasing computational demands. In this blog post, I have covered the basic intro of Apache Spark. In the next blog I will be delve deeper into the Eco-system of Spark and its components. Stay tuned! I would love your thoughts on it, check it out when you get a chance Sanjeeb Mohapatra Rajas Walavalkar #Spark #DataEngineering #Learning       

32

6 commentaires

Ressources to stay updated about the latest news in the AI field. #AI #ML #data #python

4

Dear Linkedin Network 👋 , Join me on a journey through the dynamic landscape of global demographics with my seconde interactive project in #PowerBI . 🔴 To see the project with Interactive Visualizations : https://lnkd.in/eeCWJ66C #project #PopulationProjections #powerBI #interactiveproject #InteractiveVisualizations #sql #Data #Tableau #Data_power_bi #challengeproject #formation #dataAnalytics #excel #datascience #datavisualization #statistics #innovation #developer #programmer #business #ml #database #bigdata #businessintelligence .

37