Arun C Murthy

Really excited about KIP-932 (Queues for #Kafka) and its impact on the AI world: massive worker architectures to index unstructured documents at scale, feeding LLMs. Combined to inference cost reduction, lower latency, and smaller specialized models: local #SLMs will be a game-changer for scaling AI workloads cost-effectively in the streaming paradigm. (SLM: Small Language Model)

58

1 Comment

Task-specific models and routing architectures are the future. Robert Nishihara, co-founder and CEO of Anyscale, is seeing hybrid strategies emerge, where businesses are evaluating different types of models for different use cases. With the rapid advancement of AI tech, it won't be long before open-source models are "good enough" for the majority of business tasks. The natural progression from here is to develop smaller, task-focused models using proprietary company data. As Robert puts it: "You're going to need small, fast models. And task-specific models have a huge advantage when it comes to cost and speed because you can achieve the same quality with a smaller model if it's specialized." Another interesting development: the growing use of routing architectures. Imagine a classifier as a kind of traffic controller for your AI queries. It assesses the complexity of each query and directs it to the most suitable model: simpler requests are handled by smaller models, while complex ones are passed on to larger, more sophisticated models. “If you do it right,” Robert says, “you can achieve something like the cost of the smaller model with the quality of the more advanced model.” Dive deeper into AI architecture trends in my full discussion with Robert on the B2BaCEO podcast here: https://lnkd.in/ghigvERn

46

3 Comments

What do 4 Ascend.io data 🤓s talk about on a Hackathon hike? 🤖 multi agent orchestration with #LangGraph 🦆 #DuckDB benchmarks and CSV parsing 👷streaming with #Confluent It's an exciting time in data & AI!

60

1 Comment

Batch processing for LLM inference activity: "One sign of that is the rise of batch processing. Today, popular consumer AI products like ChatGPT and Perplexity provide users with near instantaneous responses, otherwise known as real-time inference. However, businesses don't always need immediate responses and are often willing to wait hours, days, or even weeks for responses, as long as they don't have to shell out as much money. Founders of cloud and inference providers have told me there's growing pressure from business customers for this kind of flexibility, otherwise known as batch processing." -- Nathaniel Whittemore on "The AI Daily Brief" (with much deserved shout-out to his inspo from Stephanie Palazzolo at The Information) I fall into this bucket as we have a lot of backfilling to do as we're building out the underlying data in our service, and some of that could just as well be done in the wee hours of the night. The new stuff flowing in I want processed ASAFP, but these backfills—I'd mos def take more cost-efficient delayed batch processing. There's also use cases like needing (near) real-time inference for the first inference pass for timeliness / contemporaneousness value, but then a second pass slow-time inference for wider classification exercises over an ever-growing data corpus. I could see us mixing Sonnet & Haiku models in both cases. Just a heads up to Anthropic Mike Krieger as you contemplate this, along with Amazon Web Services (AWS) Jeff Barr Mark Roy Swami Sivasubramanian on the Bedrock side of things...and I don't mean the "batch" mode that doesn't seem to have any real value other than helping AWS manage its own rate limit challenges. (DeepCast is using Bedrock.)

11

2 Comments

Lowest income taxes on the west coast too!

2

Can we trust machines to judge machines when it comes to LLMs? There's an ongoing battle between manual and automated evaluation of LLMs. Manual evaluation is predominantly used by many companies, especially large ones with sufficient budgets. This approach involves internal QA teams or external consultants who manually create test cases and evaluate the outputs in spreadsheets. This process is time-consuming, but can provide insights into: - model's performance - specific failure modes such as hallucinations - tone discrepancies - biases - copyright violations - privacy issues Smaller companies without such budgets often resort to using expensive engineering time to manually review outputs, which can be inefficient. The primary advantage of manual evaluation is the accuracy and reliability of human judgment. But it's not scalable and introduces human biases, which can affect the consistency of the evaluations. Also manual processes cannot keep up with the rapid development cycles and the volume of outputs generated by modern LLMs. Automated evaluation aims to reduce the reliance on human labor by employing metrics and models to assess LLM performance. Techniques used in automated evaluation include: - Metrics-Based Evaluation: It utilizes established metrics such as BLEU score, ROUGE score, and perplexity. These metrics help assess aspects like fluency, accuracy, and relevance. But these metrics often do not correlate well with human judgments and can be inadequate for comprehensive evaluation. - Heuristic-Based Functions: These are specific functions to test particular capabilities or behaviors of the LLM e.g. toxicity detection or PII (Personally Identifiable Information) detection. These approaches can be tailored to specific use cases but may lack generalizability. - Model-Based Evaluation: It uses another trusted language model to evaluate the outputs of the target LLM. This can automate the evaluation process but requires careful calibration to ensure reliability. Automating this process can be a huge unlock. But developers attempting to create their own automated evaluation systems report very low confidence in the results. Lot of opportunity on this front.

3

1 Comment

At ScreensAI, we're focused on driving as much accuracy as we can with LLMs. But for contract review and playbook execution, it's often the case that vague instructions and subjectivity are as much to blame for undesirable results as the LLM's reasoning ability. Say you want to use an AI system to help determine if the following standard is met across thousands of contracts in your repository: 📜 The contract should limit the vendor's liability to 12 months’ fees. From our recently published Screens Accuracy Evaluation Report: "While any given individual contract reviewer may have a reasonable understanding of what they’d expect to pass or fail this standard, we see many edge cases in practice. If preferences for those edge cases are not clearly articulated, the LLM is left to assume and make a judgment call. Consider these scenarios: - What if there are major exceptions to this liability cap that weaken it so substantially that it might as well not exist? - What if this liability cap doesn’t apply to beta features? Or some specific set of features? What if the service is free? - What if the liability is capped at either 12 months’ fees or some fixed amount, whichever is greater? Or whichever is lesser? What if the fixed amount is very large? Or trivially small? - What if there is a secondary cap or a backup cap that applies under certain conditions at a much different amount than 12 months’ fees? - What if the cap is only for fees paid for the services that gave rise to the liability rather than fees paid in general under the agreement? In any of these cases, the LLM is left to assume. The leading LLMs are capable of reasoning through this type of nuance, but if they are told they must make a decision as to whether or not the standard passes, they might not make the same decision that you would if you don’t expose your preferences." 📊 If you want quantitative aggregate results to assess the risk of a portfolio of contracts, thousands of verbose "it depends" responses won't cut it. Contract professionals have strong opinions, institutional knowledge, and well-defined playbooks. ScreensAI helps them articulate this to the platform up front, systematically validating the types of LLM-powered results they'll get for the given set of expressions of their preferences. Then they can deploy this expression of preferences at scale in perpetuity. 🧙‍♂️ We think this is where a lot of the magic happens in LLM-powered contract review.

21

5 Comments

Day 3/3 GenAI Summit San Francisco Causal AI, causal grounded RAG - cRAG, scaling from POC to production in AWS, a deep dive into the evolution of Llama models and what's coming next, synthetic data in the age of genAI, PII, and domain randomization. I can't count how many times I've heard the words AI, genAI, data, cloud, RAG, LLM, and multimodal in the last 3 intense days. And when I finally went to have lunch at a restaurant, at the adjacent table, a fintech AI startup founder was pitching to a VC, while the people at the table behind me were talking about data engineering. It feels so good to be in SF, where at the airport’s security check-in a big Anthropic advertisement greets you! 😊 The last 3 days were amazing, filled with inspiration and thought-provoking new tech. I can't wait to get back to building DiffuseDrive and working with our customers!

22

I am incredibly excited to announce Arch gateway https://lnkd.in/ePAf-VK8, an open source project for developers to build genai apps faster. It is engineered with purpose-built LLMs, Arch handles the critical but undifferentiated tasks related to the handling and processing of prompts, including detecting and rejecting jailbreak attempts, intelligently calling “backend” APIs to fulfill the user’s request represented in a prompt, routing to and offering disaster recovery between upstream LLMs, and managing the observability of prompts and LLM interactions in a centralized way - all outside business logic. Here are some additional key details of the project, * Built on top of Envoy and is written in rust. It runs alongside application servers, and uses Envoy's proven HTTP management and scalability features to handle traffic related to prompts and LLMs. * Function calling for fast agentic and RAG apps. Engineered with purpose-built fast LLMs to handle fast, cost-effective, and accurate prompt-based tasks like function/API calling, and parameter extraction from prompts. * Prompt guardrails to prevent jailbreak attempts and ensure safe user interactions without writing a single line of code. * Manages LLM calls, offering smart retries, automatic cutover, and resilient upstream connections for continuous availability. * Uses the W3C Trace Context standard to enable complete request tracing across applications, ensuring compatibility with observability tools, and provides metrics to monitor latency, token usage, and error rates, helping optimize AI application performance. This is our first release, and would love to build alongside the community. We are just getting started on reinventing what we could do at the networking layer for prompts. Do check it out on GitHub at https://lnkd.in/ePAf-VK8 Please leave a comment or feedback here or in GH and I will be happy to answer. #gateway #envoy #rust #genai #developers #ai cc Salman Paracha | Muhammad Ikramul Haq | Syed Anas Hashmi | Fahim ul Haq | Farheen Omar | Umar Farooq | Arslan Ahmad | Abdul Rafay Abbasi |

56

9 Comments

The new bytewax-redis module allows you to seamlessly integrate Redis streams and key-value stores into your Bytewax dataflows. Whether you're consuming data from Redis streams, writing processed data back to them, or interacting with Redis' key-value storage, bytewax-redis has you covered. cc: Zander Matheson

42

1 Comment

Another good post by Sohrab Rahimi at McKinsey & Company. He raises four concepts and techniques as foundational elements of a "highly powerful system". However, in my view they are just a set of discrete capabilities that need to be under management of platform. There's a bias in tech coverage towards all things probabilistic. I recognize that it sounds future forward, but the problem is that almost none of these papers ground AI to the real-world - there is no meaningful #recursion and #reflection without it. On their own, they'd be incomplete and outright irresponsible for anything #EnterpriseGrade and mission critical. Simply compounding probabilities in ever increasingly complex and brute force services, never replaces a set of good facts, a well-modeled domain and real-time connections to virtual and physical devices. It's worth noting that there are things that classic AI, analytics and good old fashioned rules just do better, faster, cheaper (more sustainably too!). The industry needs balanced position. Let's get back to architecture, non-functional concerns and cost/performance. Personally, the only meaningful way forward is through #NeuroSymbolic #AI and #ontologies. As is often the case, we'll "go back to the future" - the return of #ExpertSystems.

18

5 Comments

Google DeepMind has just released two exciting updates to the Gemini family: Gemini-1.5-Pro-002 and Gemini-1.5-Flash-002. These models bring faster, smarter, and more efficient AI capabilities, making them a game-changer for developers and businesses alike. Here’s what’s new: • >50% price reduction on Gemini 1.5 Pro • 2x faster output with 3x lower latency • Rate limit boosts: 2x higher for Flash and ~3x for Pro • Concise, focused responses, Enhanced safety and customizable filters #AI #MachineLearning #Gemini #DeepMind #AIInnovation #TechUpdate https://lnkd.in/gGGbwEFt

18

With the release of o1, a model with advanced meta-thinking, using a "mixture of agents" becomes even more appropriate. For instance, o1 can now be used to plan prompt sequences to be performed by less expensive or task-specific models and tools. Meta-prompt engineering is now officially a thing. https://lnkd.in/eTqwp_mt

17

1 Comment

It’s still early to predict which companies will rise as the new giants of the AI application layer. Significant breakthroughs are likely still ahead at the foundational model level, and companies operating in the application layer will need to stay highly agile. Any major advancement in foundational models will disrupt their business models.

7

A new tool for evaluating LLM. This could be useful for enterprises.

7

Please media types writing about the cost of AI: recognize there is a difference between huge GPU/TPU clusters needed to train foundation models vs. inference costs to run most apps on top of foundation models. The two are not synonymous, so you can dial back some of the doom and gloom about AI startups and your commingling of the two different AI business types.

11

We used AI to uncover nearly invisible malicious code that was costing us $35k/year! Here's how: BACKGROUND: Our team was running a Lambda optimization scan with our new CloudFix AI finder and it flagged a Lambda function that was costing us $35k/yr... This function was creating a "fork bomb" effect, constantly spawning new instances of itself. We traced the issue to an outdated index update process. The AI helped us identify the problematic code and proposed a fix. This function had been running for a few years... Maybe it was malicious. Maybe not. But to diagnose that it was a useless function spawning instances of itself would have been very difficult without using AI to ingest and analyze the code. TAKEAWAY: This issue was nearly invisible in our typical cost reports due to its relatively small impact on our overall AWS spend. Without AI, we might never have found it. $35k might be small, but how many of these hard to find functions are running wild across your AWS instances ? Just a few of them running is the budget to hire another engineer. Or to buy a margarita machine. Whatever you're into.

24

11 Comments

We’ve already seen that software eats the world. Ali Ghodsi predicts that, in the next phase, AI eats software. During a conversation on the B2BaCEO podcast, Ali, the Co-founder & CEO of Databricks, said AI will not just be a feature; it will be the foundation of most platforms, as we move toward a world where AI is seamlessly integrated into nearly every tool. Imagine every piece of software evolving from static code into an autonomous decision-maker, as intelligence and automation are built into all data architecture underlying enterprise software. Our full conversation on the B2BaCEO podcast here: https://lnkd.in/gXvSSf_C

56

4 Comments