Dr. Tim Dörnemann

My new article on medium: how to simply enable temporal / history table in PostgreSQL. No need for: - Special extensions. - Specialized databases. - Special techniques like event storming or soft deletes. None of that is needed. Don't overcomplicate things. https://lnkd.in/dQDNrCq8

20

🧠 Qualcomm 𝐣𝐮𝐬𝐭 𝐡𝐚𝐝 𝐚𝐧 #𝐀𝐈 𝐚𝐧𝐚𝐥𝐲𝐬𝐭 𝐝𝐚𝐲 𝐰𝐡𝐞𝐫𝐞 𝐭𝐡𝐞𝐲 𝐝𝐞𝐦𝐨𝐞𝐝 𝐭𝐡𝐞𝐢𝐫 𝐨𝐧-𝐝𝐞𝐯𝐢𝐜𝐞 𝐀𝐈 𝐜𝐚𝐩𝐚𝐛𝐢𝐥𝐢𝐭𝐢𝐞𝐬. 𝐈𝐬 𝐢𝐭 𝐰𝐨𝐫𝐭𝐡 𝐜𝐨𝐧𝐬𝐢𝐝𝐞𝐫𝐢𝐧𝐠? 🎯 My thoughts on this topic in a blog post and video analysis in the first two comments below. ✅ 𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 𝗳𝗼𝗿 𝘁𝗵𝗲 𝗹𝗮𝘁𝗲𝘀𝘁 𝐢𝐧 𝐀𝐈, 𝐌𝐋, 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞𝐀𝐈 𝗿𝗲𝗹𝗮𝘁𝗲𝗱 𝗻𝗲𝘄𝘀 𝗮𝗻𝗱 𝗮𝗻𝗮𝗹𝘆𝘀𝗶𝘀. - https://lnkd.in/eESnAbFm 💡 Here are my top five takeaways from the event: ✨ Smaller models and more efficient architecture will rule. ✨ Edge is faster & scalable. ✨ On -Device is private & faster. ✨ Qualcomm is more than chips. With their AI hub. ✨ Qualcomm AI research. 💥My Final thoughts on the video analysis are in the first comment below. ❓Do you agree? Please leave your thoughts in the comments section. Thanks for having me at the event. Qualcomm Clare Conley Lauren Miller Durga Malladi Jilei Hou Tony O'Connor Constellation Research, Inc. R "Ray" Wang #Qualcomm #GenerativeAI #ArtificialIntelligence #OnDeviceAI #LLM #SLM

39

8 Kommentare

When developing new algorithms for performance there is always a long period of time when you have no idea whether it will actually, you know, work. Developing pgvectorscale was no different. I spent months developing the algorithm without much luck. Then, one day, we ran benchmarks and my colleague sent me a slack message “DUDE, you hit gold with this branch”. That’s when I knew I was onto something. The end-results were impressive. PostgreSQL + pgvector + pgvectorscale vs Pinecone produces 28x lower p95 latency and 16x higher query throughput for approximate nearest neighbor queries at 99% recall. It does this on EC2 hardware that costs 75% less. I’ll admit to being giddy. You should be too. https://tsdb.co/mat-pg-ai

232

2 Kommentare

New article in #APIDesignMatters: API Naming Patterns https://lnkd.in/eppRvi-J

36

3 Kommentare

Improve your CI/CD automated on-target testing: #SEGGER has added --args in J-Run for J-Link. Now you can pass arguments to the application on the embedded target: https://lnkd.in/eFfEEPaF

43

I'm curious to hear why people (I mean developers) fight so hard against having logic in database. Here is interesting Reddit thread on subject: https://lnkd.in/dE9BZuv6 Top comment gave three examples from the real world, where moving logic out of application and into the database gave the following performance improvements: Example 1: from seconds to about 80ms. Example 2: from 30 seconds down out 0.3 seconds. Example 3: from 4 hours to 10 minutes. Crazy, isn't it... That's more or less mine experience too. But he(or she) starts the post saying it's difficult because you can't easily debug or get useful logging for tracing problems. As a matter of a fact, you can configure pgAdmin for debugging easily. But I get it, you need to reconfigure your server and you can only use that debugger on local instances. But you can always use RAISE INFO statement and wire them up in your application to be normally logged, even with exact source location. So I don't see a problem there. Most commentators seem to agree that there are always some reasons to move logic into database, either performance or security, so most system trying top keep logic out of the database will end up as a hybrid with logic scattered between application and database, which certainly takes some cognitive and maintenance toll. So, why even bother with logic in application, you know what I mean? What are some other argument against? Scalability? Testability? Lack of trained operators/developers?

42

70 Kommentare

Yesterday's holiday (June 11, 2024) activity includes playing and learning RAG implementation with Elasticsearch (ELSER - Elastic Learned Sparse EncodeR, a natural language processing (NLP) model trained by Elastic) and E5 embeddings. The thought process includes crawling a public website using Elastic Web crawler, ingest and format the data in Elasticsearch index for these data to be use when asking question to openAI (LLM) - the result will then be displayed a webapp built using Streamlit ( a free and open-source framework to rapidly build and share machine learning and data science web apps). Sharing the resulting code for other to use and some screenshots of the questions and answers using openAI. RAG Implementation for your own experimentation: Giving credit where it's due Grant Patterson - thanks for sharing the initial code for me to play with.

17

Just read a great article by Max Kremer about their experience using the AWS Postgres managed service: https://lnkd.in/eaVdxyKp I would say that managed services are great for the 80% of all use cases. This is probably true for all AWS (and any other supplier) managed services. They save a lot of work but are much less flexible. For edge use cases, when you need specific performance in certain areas or require to change specific settings to optimize your environment, that's when its time to start moving off of the managed services and set up your own services. In general, I would recommend starting with the managed services. This gives you an easy and quick starting point. As you scale and get moving in production, you may notice some aspects aren't ideal. That is the point where you make the decision, is the pain of managing the services on your own worth the performance gains? #iworkforgm #GMrs_lead

7

2 Kommentare

Because so many have asked: a great project when it comes to dynamic load balancing in #Proxmox. Hopefully soon also directly integrated, but until then an excellent solution.

13

Just released sqlaxe, an open source for splitting up SQL files by table name. It can also pretty print and transpile sql dialects - e.g. MySQL to PostgreSQL. https://lnkd.in/eWSBbZbY For example: > python sqlaxe.py mysql_test.sql >> tokenizing >> done tokenizing 100%|███████████████████████████████████████████████████| 204/204 [00:00<00:00, 1191696.40it/s]  0%|                                  | 0/7 [00:00<?, ?it/s]>> writing to sqlaxe_mysql_test/0001_orm_test.sql 2 statements >> writing to sqlaxe_mysql_test/0002_users.sql 1 statements >> writing to sqlaxe_mysql_test/0003_posts.sql 1 statements >> writing to sqlaxe_mysql_test/0004_comments.sql 1 statements >> writing to sqlaxe_mysql_test/0005_tags.sql 1 statements >> writing to sqlaxe_mysql_test/0006_post_tags.sql sqlaxe, as you can see, writes each table to a separate file. They are indexed numerically, which helps if you wish to re-assemble the files after editing. Note that this isn't a text manipulation approach, as you might DIY with sed/awk and friends - this uses sqlglot to actually parse the SQL, so at least in theory you shouldn't be tripped up by, say, a stray semicolon or ten inside a text column. The primary use case here is for working with SQL backups - you can, for example, use this to performing surgery on SQL backups to fix, for example, accidentally dropped tables or somesuch.

16

Here is something interesting about databases: The first top ten database engines are mostly relational and multi-model. The only two that are not multi-model are just relational. Multi-model means that it supports multiple data models, such as Document Model, Graph Model, Spatial Model, Vector Model, Search Engine Model, Time Series Model, etc. Which makes those multi-model databases a general purpose database. And top five from this popularity ranking are all relational and multi-model databases. However, the relational data model is a general-purpose model itself: - AI, tell me, is a relational data model a general-purpose data model? - AI: Yes, the relational data model is considered a general-purpose data model. So, the top five most popular databases in the world - are not only general-purpose databases - they are very, very, VERY much general-purpose databases. But, somehow, modern software architecture strongly encourages developers to postpone decisions on the database type and NOT make decisions about the database until much later in the development cycle. As Uncle Bob says - when you have more information. About requirements. Because, you know, the requirements might change, and we might switch to another database type. Maybe one general purpose is more general purposy than the other; I don't know. It doesn't make sense. How about pick any, and just get the most of it, how about that. https://lnkd.in/duz7gjN5

32

11 Kommentare

Rust vs Python: Performance Showdown in Kubernetes I recently compared Python and Rust performance for computationally intensive tasks in a Kubernetes environment. Our challenge: calculate prime numbers up to ten million. Test Environment - No GPUs - Google Kubernetes Engine (GKE) - Cluster: 3 nodes, e2-medium (2 vCPUs, 4 GB memory) - Kubernetes Version: 1.29.6-gke.1254000 Results - Python: Prime search up to 10,000,000 completed in 222 seconds - Rust: Prime search up to 10,000,000 completed in 7.6 seconds Rust completed the task over 29 times faster than Python! Key Takeaways 1. **Performance Gap**: Rust's speed showcases its potential for performance-critical applications, completing the task significantly faster than Python. 2. **Resource Efficiency**: Rust's efficiency could lead to substantial cost savings in cloud environments, especially for compute-intensive tasks. 3. **Development Trade-offs**: Python's ease of use and vast ecosystem remain advantageous for many scenarios, despite the performance difference. 4. **Kubernetes Flexibility**: We demonstrated seamless switching between implementations, allowing teams to use the right tool for each task. Conclusion This experiment highlights the importance of choosing the right language for specific needs in cloud-native development. While Python excels in simplicity and library support, Rust shines in performance-critical components. The combination of Rust's speed and Kubernetes' flexibility offers a powerful toolkit for building scalable, efficient systems. As cloud computing evolves, there's room for both Python's simplicity and Rust's performance, enabling developers to optimize their applications for both development speed and runtime efficiency. The stark difference in execution times for this prime number search underscores the potential impact of language choice on application performance.

2

pgroll - zero-downtime schema changes (keeps 2 version of the schema accessible at the same time), reversible, schema migrations (using JSON format) for Postgres (CLI) https://buff.ly/3VWoEol

16

𝑯𝒆𝒍𝒍𝒐 𝑬𝒗𝒆𝒓𝒚𝒐𝒏𝒆 🌠 🌳 𝐃𝐚𝐲 𝟜 of my progress on #GeekStreak2024 by GeeksforGeeks x Deutsche Bank 💻 🔹𝐏𝐫𝐨𝐛𝐥𝐞𝐦 - Longest Valid Parentheses 🔹𝐃𝐢𝐟𝐟𝐢𝐜𝐮𝐥𝐭𝐲 - Hard 🔹𝐃𝐞𝐬𝐜𝐫𝐢𝐩𝐭𝐢𝐨𝐧 - Given a string str consisting of opening and closing parenthesis '(' and ')'. Find length of the longest valid parenthesis substring. A parenthesis string is valid if: For every opening parenthesis, there is a closing parenthesis. Opening parenthesis must be closed in the correct order 👨🏻‍💻 🔹𝐊𝐞𝐲 𝐓𝐚𝐤𝐞𝐚𝐰𝐚𝐲𝐬 - Stack, Strings💡✨ #DSA #CodingChallenge #CodingJourney #ProblemSolving #GeeksforGeeks #21daystreak

13

It looks like that the new, upcoming PostgreSQL 17 is mostly performance oriented. Many improvements to the optimizer. Here are the release docs, see for yourself, maybe I'm wrong. https://lnkd.in/dyUw4d9w

7

{✒NEW BLOG} How to validate rows with CHECK constraints in #MariaDB? Will CHECK constraints affect performance? Get the answers in the new blog post by Federico Razzoli: https://lnkd.in/ec_sc69V #database #databases #databaseadministration #MySQL #mysqldba #dbms #rdbms

7

NEW § The second pattern in Juntao Qiu's article on data fetching is on how to avoid the dreaded Request Waterfall. Parallel Data Fetching queries multiple sources in parallel, so that the latency of the overall fetch is largest of the queries rather than the sum of them. https://lnkd.in/eFJu25Eg

176

1 Kommentar