Mehmet Bektas

"Which one is faster?" - Great question Introducing: 🤖 LLM Arena 🚥 📊 This app allows you to measure the speed and word count across OpenAI, NVIDIA, and Groq language models API's and compare their performance. Why? This is a solution to a question I had while implementing an agentic solution, I was thinking all the possible ways to make it run faster. I came up with this app just to illustrate the concept of how useful could it be to choose the right models and providers to accelerate your flow. 🛠️ Stack: Langchain, FastAPI, Streamlit and Docker. Take a look at my repo here: https://lnkd.in/e4Q8Q2V5 🎙️This app will be one of many things I'll be covering in my upcoming talk! If you are in Madrid on 📆25th September, see you there! Free 🍕 🍻 powered by InnoIT Consulting and My Tech Plan 👉Register here: https://lnkd.in/ev_-n9vx #genAI #OpenAI #Groq #Nvidia #AI #Langchain

12

3 Comments

Generating JSON with LLMs is broadly applicable to many tasks. At Substrate, we believe it's the *glue* we need to do *less prompt engineering*, and *more software engineering*. JSON generation with LLMs makes data so much more malleable. Now we can easily extract data from unstructured sources, or even "shapeshift" structured data into any shape we want (Substrate code below). A lot of the new terms in AI engineering simply describe multi-step flows. And a lot of these flows can be reframed in terms of JSON generation. RAG? That's generating JSON for your vector DB query, searching, and then calling an LLM. (JSON generation can be useful on both the "Retrieval" and "Augmented Generation" sides). Function calling? Tool use? That's generating JSON for a function call, calling the function, and then calling an LLM. (Any multi-step LLM flow is a form of "Augmented Generation"). And there's a real benefit to reframing this way. JSON generation can improve reliability, and it's well established that LLM programs improve when multiple calls are chained together. You can also save on inference costs: (1) By using smaller models for each step. (2) By using smaller prompts (you can just define a schema, instead of coaxing with multi-shot examples) What's the catch? JSON generation and multi-step flows are known to be slow, and unreliable. That's where Substrate comes in. We've relentlessly optimized our JSON generation to make sure it's fast, and follows your schema with 100% accuracy. And Substrate's unique inference approach enables multi-step flows to run with maximum parallelism, and zero unnecessary data roundtrips. LLMs offer programmatic access to heuristic computation. Heuristic and symbolic computation are good for different things – and structured generation is the glue we need to combine them into a new kind of AI-integrated program.

23

1 Comment

Code as Extensible as a Lotus Flower "Very good. Enough for today, Grasshopper. I would like for you to go and meditate further on this topic... Remember, code should be closed (to change) like the lotus flower in the evening, yet open (to extension) like the lotus flower in the morning. ...Design Principle: Classes should be open for extension, but closed for modification." ― Eric Freeman, Ph.D. & Elisabeth Robson, Head First Design Patterns: Building Extensible and Maintainable Object-Oriented Software https://lnkd.in/dPYV-8jF #refactoring #extremeprogramming #modernagile #agility #tdd #testdrivendevelopment #legacycode #continuousimprovement #maintainability #softwarequality #codequality #softwaredesign #lean #oop #solid #designpatterns

3

Really interesting blog post came up from Character.ai team! They handle 20,000 requests per second, so the task is challenging and needs to be optimized. How they did it: ➕ Reducing KV Cache: - 85% of layers use only Local Attention: Instead of standard attention, they use Local Attention (https://lnkd.in/gmE3iEE3), significantly reducing cache size, especially for long contexts. This approach is also used in Gemini and Gemma 2 (https://t.me/ai_newz/2925) and is reminiscent of Jamba (https://t.me/ai_newz/2519), where heavy attention was used only for certain layers. ➕ Multi-Query Attention: They use Multi-Query Attention (https://lnkd.in/grppzadw) instead of the currently dominant Group Query Attention, reducing cache size by eight times compared to the industry standard, albeit with significant quality drops. ➕ Shared KV Cache between layers: This method (https://lnkd.in/g3yRgi3d) reduces KV Cache size by over 20 times, enabling: - One machine can store the KV Cache of thousands of users. - Segmenting KV Cache for each message (second image) allows conversation continuation from any point without regenerating the cache. - Sticky Sessions are used to maintain the efficiency of over 95%, by directing users to servers where their KV Cache is already stored. ➕ Cost Reduction Benefit: Due to these optimizations, the cost of inference for the startup has decreased by 33 times over a year and a half, making it 13 times cheaper than the nearest competitor. Full Article here: https://lnkd.in/gXQrGNZx #AI #MachineLearning #ArtificialIntelligence #LLM #LocalAttention #MultiQueryAttention #KVCache #TechInnovation #CharacterAI #Startup #InferenceOptimization #Int8Training #AIWaifu #TechStartups #AIResearch #Transformers #Innovation #NoamShazeer #TechNews #AIModels

13

4 Comments

Thank you community for your feedback and very constructive responses on the GenAI Ref Arch. All the way from adding a TOC for better navigation through these patterns that help you design your Gen AI Reference Architecture to more coverage on Multi-agent Architectures. Let me know, in the comments if there are further areas you want to probe deeper. #community #generativeai #referencearchitecture Google #patterns #designpatterns #architecturepatterns #bestpractices #llm

189

14 Comments

Anthropic’s newest and now most intelligent model, Claude 3.5 Sonnet, is generally available on #VertexAI! Learn what makes Claude 3.5 Sonnet and Vertex AI the perfect match → goo.gle/claude

28

7 Comments

This might freak some people out, but don’t let AI anxiety distort the facts. It’s perfectly fine to let LLMs check code, but as OpenAI and many others have noted, keep humans in the loop to ensure quality outputs. Semi-autonomous systems are the best we can hope for right now. #AI #LLMs #TechSafety #CodeReview #AIAnxiety #HumanInTheLoop #TechInnovation #AIApplications #QualityAssurance

4

1 Comment

if you know, you know NVIDIA annual shareholder meeting today: june 26 2024 09:00 pst in about an hour we'll have a chance to listen to new developments. it's breath-taking. the speed of not only NVIDIA's ability to navigate (well i guess they must be using their own (ai) to decide what where how to do and then some. huh?) more importantly just as Apple Microsoft defined the pervasiveness of computers. their applications. we believe NVIDIA holds that position currently. they have been handsomely rewarded in the markets. a valuation as the most valuable in the world. 3.3 trillys. and going. so every and all announcements are truly where the centre of the universe of chip design, manufacturing, exists. it's more than simply offering state of the art chip design for (ai) modelling. where Alphabet Inc. organizes the world's information. NVIDIA makes sense of it. and everyone else applies it, to their subject matter expertise. this is the change that is afoot. at this moment. where it was a race to offer advertisers of services/products a better understanding of their clientele, now the lens is sharper. there is no need to suggest age/sex/ethnicity/generational differences. there are behaviours. theses behaviours are independent of what data was used to point adverts to. to get these people to buy more sh^t. that's the base of the u.s. economy. it's apparently consumer driven. and so in this post pandemic world where it literally stopped. and we were forced to sit still. wait it out as the storm clouds passed. in hopes no one. not us. or loved ones would perish. that time was a generational shift. we all felt. it. no matter what class you were. NVIDIA is the fulcrum Google is the waterfall or Google is the head NVIDIA the neck wherever the neck turns that is where the head has their attention. the neck keeps the attention of the head there. so it appears as if in many ways Google has in essence offered the search part of it's business up for grabs. this is why Perplexity exists, and others. a new dynamic pricing auction model is about to emerge. it will as it is by nature adversarial. let's not fool ourselves. we are designed to be adversarial. it's our nature. and in coveting $$$$$ as the prime objective. money for the sake of money this will destroy a lot of thoughtful, mindful, intentionally beneficial innovation. why? well. it's not playing according to the whims of a recommendation machine that rewards those that are part of it's invisible army. today is a crowning. it's a crown. upon the head of jensen. this is akin to king charles and his crowning. we wonder who will be at the crowning. to recognize this eventful day. it's good to be king. ;) < . > source: https://lnkd.in/eumX4num

1

Since this talk from 2011 - not doing a significant percentage of your refactoring via automation - is an outdated way of working when it comes to improving code quality in a product. Refactoring by module or file or function, in every team, is still relevant as part of eg TDD and other flows, but certain forms of code modernization for higher quality and better security can gain a lot of momentum and efficiency by being automation, in any larger scale or safety/security critical software project. This includes - of course - not only the support provided by third party tools like clang, but also tons of own, product specific needs.

1

Meta Introduces Chameleon: A State-of-the-Art Multimodal Model 🦎 Meta has unveiled Chameleon, a groundbreaking multimodal model that pushes the boundaries of AI capabilities. Chameleon seamlessly integrates vision and language, enabling it to understand and generate images and text with remarkable accuracy. Key Takeaways: 🎨 Chameleon excels in both image captioning and text-to-image generation, showcasing its versatility across multiple tasks. 🧠 The model leverages a novel architecture called Mixture-of-Attention (MoA), which efficiently processes and combines visual and textual information. 🌐 Chameleon outperforms existing state-of-the-art models in zero-shot image captioning and text-to-image generation on various benchmarks. 🔍 The model's ability to handle complex, compositional prompts sets it apart from other multimodal models. 🔒 Meta has open-sourced Chameleon's code and trained weights, fostering transparency and enabling further research in the field. Chameleon represents a significant milestone in the development of multimodal AI systems. Its ability to understand and generate both images and text opens up exciting possibilities for applications in fields such as creative content generation, assistive technologies, and more. Read the full article to learn more about Chameleon and its implications for the future of AI: https://lnkd.in/gk_ZTZdR #chameleon #multimodalai #artificialintelligence #ai #machinelearning #ml

4

The #OSAID effort by #OSI appears to be a good start at generating a definition of what "open-source" means to AI models. It remains to be seen if they can harmonize their efforts with others that have made progress, such as the Linux Foundation. I'll also be waiting to see, once the hot takes give way to measured and thoughtful responses, if any significant holes can be poked in it and then addressed. But ultimately, it appears to be a good-faith effort at describing full transparency and replicability. #opensource https://lnkd.in/gEtRJ29S

8

1 Comment

Impressive gains from fine-tuning Llama 3.1, especially regarding inference speed & VRAM reduction ! Open-source version available for 1 GPU instances!

4

Chameleon, an early fusion model developed by Meta, uniquely integrates and processes multiple information modalities—such as text and images —right from the input stage. Unlike traditional models that handle each modality separately and merge them later, Chameleon processes them together from the beginning. This approach enables a more comprehensive understanding and nuanced responses. https://lnkd.in/ejF65K7y #Meta #MachineLearning #AI #Technology #Innovation #FutureTech

7

1 Comment

Anthropic's latest Claude AI model is taking human-AI interaction to a whole new level! 🖥️🤖 The new Claude 3.5 Sonnet can now use computers just like a human would - by looking at the screen, moving the cursor, clicking buttons, and typing text. This is a major leap forward in AI capabilities. Key features: Operates based on screenshots, emulating human computer use Can perform complex multi-step tasks across different applications Self-corrects and retries when encountering obstacles2 Accessible via API for developers to build with Why it's game-changing: This opens up incredible possibilities for AI assistance in data entry, form filling, software testing, and automating repetitive computer tasks13. Imagine having an AI assistant that can seamlessly work alongside you, handling tedious computer work while you focus on higher-level tasks. While still in beta, this technology demonstrates how AI is rapidly advancing toward more human-like interaction with digital interfaces. As it develops further, it could revolutionize how we work with computers and boost productivity across industries. The future of human-AI collaboration just got a lot more exciting! What do you think about this new capability? #AIInnovation #FutureOfWork Here is a video about it: https://lnkd.in/gh8aNYEu

6

The world is waiting Groq. But, don’t hurt us to much on ghe pricing though, we know you can’t get something for nothing. We are using #groq, but only for experiments due to the rate amd token limits atbthe moment. There are ao many usecases, especially on backend for us to do some amazing things in almost realtime. 3GPUs seem legacy compared THE NEW #LLM optimized #LPUs. Hopefully someone will figure out a way to strain moremoutnof gpusmormoptimize the #llms to be more #gpu friendly.

Toolhouse AI and Groq are presenting a technical webinar on the challenges of building agentic AI. It'll be a high level overview of agentic architecture, highlight the current challenges of building agents and explain how you can overcome them with Toolhouse AI and Groq. https://lnkd.in/dm9pPf6H

2