Scott Kramer

Batch processing for LLM inference activity: "One sign of that is the rise of batch processing. Today, popular consumer AI products like ChatGPT and Perplexity provide users with near instantaneous responses, otherwise known as real-time inference. However, businesses don't always need immediate responses and are often willing to wait hours, days, or even weeks for responses, as long as they don't have to shell out as much money. Founders of cloud and inference providers have told me there's growing pressure from business customers for this kind of flexibility, otherwise known as batch processing." -- Nathaniel Whittemore on "The AI Daily Brief" (with much deserved shout-out to his inspo from Stephanie Palazzolo at The Information) I fall into this bucket as we have a lot of backfilling to do as we're building out the underlying data in our service, and some of that could just as well be done in the wee hours of the night. The new stuff flowing in I want processed ASAFP, but these backfills—I'd mos def take more cost-efficient delayed batch processing. There's also use cases like needing (near) real-time inference for the first inference pass for timeliness / contemporaneousness value, but then a second pass slow-time inference for wider classification exercises over an ever-growing data corpus. I could see us mixing Sonnet & Haiku models in both cases. Just a heads up to Anthropic Mike Krieger as you contemplate this, along with Amazon Web Services (AWS) Jeff Barr Mark Roy Swami Sivasubramanian on the Bedrock side of things...and I don't mean the "batch" mode that doesn't seem to have any real value other than helping AWS manage its own rate limit challenges. (DeepCast is using Bedrock.)

11

2 Comments

Tembo is leading in the 'Postgres as a Platform' movement. Their most recent blog post(s) below do a pretty good job of explaining why 'Postgres as a Platform' is rapidly becoming an anecdote for database sprawl.

8

1 Comment

RLHF is a traditional and well-known technique that helps practitioners to correct ML/DL over time. With the rise of LLMs, it's essential to have a proper RLHF as LLMs are non-trivial by nature. Right now, only foundation model companies and big tech can afford it in its current form at scale as it is complex and costly. Additionally, the collection of human feedback is usually done via a very simple mechanism: they are prompted to choose from two options of generations for each given prompt. This aims to reduce the labeling costs but provides a weak learning signal for the reward model that will be trained on this data. Finally, evaluators do not have sufficient context of the task and that’s the reason they introduce bias. We introduce an alternative to RLHF that optimizes LLMs through 'personalized' feedback. Read why RLHF matters in the context of LLMs and more here: https://rb.gy/1xcx05 #llms #rlhf #feedbackintelligence

45

4 Comments

Prompting for scale will soon be obsolete... i think...i hope. Prompting is an error-prone, discovery-based process, unsuitable for scalable AI. Many folks rely on black-box LLMs behind API walls (e.g., Claude, GPT-4), lacking direct control over LLM model weights. Programmatically custom-pruned LLMs will be the future of controllable, production AI, not prompting. hint--> #steeringvectors

17

6 Comments

Not Diamond now supports routing to custom models. You can route between proprietary models like GPT-4o, private fine-tunes, agents... this is something a ton of people have asked for. Watch me quickly train a model router from scratch with both proprietary and custom LLMs:

36

Two weeks ago Sam Levan (CEO of MadKudu) put out a 5,000 word “Unbiased Review of RB2B” which RUTHLESSLY compared us to Clearbit Reveal across 5 metrics. I couldn’t believe what he discovered: _______ THE GOOD #1: The accuracy of our de-anonymization ABSOLUTELY CRUSHED Clearbit. RB2B - 84% accurate Clearbit - 59% accurate You might be asking yourself: “How do they determine the accuracy of the data?” The answer: They used the % of records that matched their form fill data. So, we’re way more accurate than Clearbit, and probably 6Sense, Demandbase, and ZoomInfo, since they all do a similar reverse IP technology to resolve companies. Period. _______ THE GOOD #2: The overlap of audiences between the two solutions was TINY. ONLY 4% OVERLAP in the audiences between Clearbit and RB2B. We found the same to be true with 6Sense’s tool when we bought it. What does that mean? If you are using Clearbit, 6Sense, or Demandbase, you need to USE US TOO. It’s not only MORE information you’re getting about your site visitors … It’s DIFFERENT SITE VISITORS altogether. _______ THE BAD Our integrations game is terrible. We’re missing Salesforce, Marketo, Zapier, Salesloft, and Outreach. But, we’ve been growing like crazy… So that tells you who we’ve been selling to (SMB). We’ll fix this over the next two quarters. _______ THE UGLY   This is hard to write. We only resolved 11% of Mad Kudu’s total traffic. Clearbit resolved 31%. However, I asked Sam in the comments how much of their traffic was non-US. Clearbit can resolve EU traffic … We cannot. The answer? 60% of MadKudu’s traffic is US-based. What percentage of their US traffic did we resolve? RB2B resolved 18.3% of US traffic (probably in line w/ Clearbit’s US rate) That’s below the 20-25% we’re quoting people, but not by much. _______ TAKEAWAY Cold outreach doesn’t work anymore. Creating intent signals that trigger sales outreach is the only way to grow your SaaS. In Sam’s words, your first-party data is the best kind of intent. So is RB2B the best source of first-party intent? 1. It’s 40% more accurate than Clearbit 2. Our coverage of US-traffic is similar  3. You don’t have to GUESS who the visitor was Not only that… It’s free. If you are using any of the company-based identity tools like Clearbit, if it’s working, keep using them. But based on Sam’s analysis, there’s one REALLY obvious course of action... Use us too! You can sign up here 100% free: https://www.rb2b.com/

315

84 Comments