Ian W.

Yesterday, my team at Vareto announced the first-ever multiplayer modeling platform for Finance. 😍 Now that I can finally speak publicly about what we've been working on, the first thing I want to share is a screenshot from a live multiplayer session we did internally at the company. One of the interesting things about building enterprise software as a startup is that we're still too small to *need* our own product. So instead of doing what our customers do--expense budgeting, headcount planning, etc.--we use Vareto's multiplayer modeling to forecast fun things like total pet spend. Yep, that's right. We had 40 people go into a Vareto model and enter in bottom-up inputs for how much they expect to spend on their cats and dogs this year. 😂 While this isn't very useful--especially since many people entered in clearly exaggerated numbers--sessions like this helped prove that our new 2.0 platform can handle even "extreme" usage situations. (In reality, even if some enterprise customers have thousands of users in Vareto, only a handful are ever likely to be in the same model page at the same exact time given granular data permissions.) You can see more useful, realistic ways that finance and business teams can use Vareto multiplayer here: https://lnkd.in/gV9gqKwz

123

22 Comments

LLMs rarely produce a finished copy on the first try. Your best prompts typically give you a rough first draft at best. That’s quickly changing – look at our progress in just 2 years. LLMs have faster response times, more data, and accurate outputs. Imagine 5 years from now?

15

5 Comments

I am more and more excited about Local ML on Apple Silicon. In particular, Apple Silicon + Core ML is starting to be a very nice stack and they've improved the Developer Experience significantly at the latest WWDC. Pedro Cuenca and Christopher Fleetwood wrote a blogpost on HF about the main DX and performance improvements announced at WWDC 2024: - Swift Tensor, a much simpler Tensor API (numpy, but for Swift) - Stateful Buffers (pretty much required for modern ML) - new built-in quantization techniques The post then describes how to convert Mistral 7B to Core ML, and the end result is: You have a 7B parameter model running at 30+ tokens/second using less than 4GB of memory on your Mac. 🔥🔥🔥🔥

1,532

81 Comments

Generating JSON with LLMs is broadly applicable to many tasks. At Substrate, we believe it's the *glue* we need to do *less prompt engineering*, and *more software engineering*. JSON generation with LLMs makes data so much more malleable. Now we can easily extract data from unstructured sources, or even "shapeshift" structured data into any shape we want (Substrate code below). A lot of the new terms in AI engineering simply describe multi-step flows. And a lot of these flows can be reframed in terms of JSON generation. RAG? That's generating JSON for your vector DB query, searching, and then calling an LLM. (JSON generation can be useful on both the "Retrieval" and "Augmented Generation" sides). Function calling? Tool use? That's generating JSON for a function call, calling the function, and then calling an LLM. (Any multi-step LLM flow is a form of "Augmented Generation"). And there's a real benefit to reframing this way. JSON generation can improve reliability, and it's well established that LLM programs improve when multiple calls are chained together. You can also save on inference costs: (1) By using smaller models for each step. (2) By using smaller prompts (you can just define a schema, instead of coaxing with multi-shot examples) What's the catch? JSON generation and multi-step flows are known to be slow, and unreliable. That's where Substrate comes in. We've relentlessly optimized our JSON generation to make sure it's fast, and follows your schema with 100% accuracy. And Substrate's unique inference approach enables multi-step flows to run with maximum parallelism, and zero unnecessary data roundtrips. LLMs offer programmatic access to heuristic computation. Heuristic and symbolic computation are good for different things – and structured generation is the glue we need to combine them into a new kind of AI-integrated program.

23

1 Comment

At ScreensAI, we're focused on driving as much accuracy as we can with LLMs. But for contract review and playbook execution, it's often the case that vague instructions and subjectivity are as much to blame for undesirable results as the LLM's reasoning ability. Say you want to use an AI system to help determine if the following standard is met across thousands of contracts in your repository: 📜 The contract should limit the vendor's liability to 12 months’ fees. From our recently published Screens Accuracy Evaluation Report: "While any given individual contract reviewer may have a reasonable understanding of what they’d expect to pass or fail this standard, we see many edge cases in practice. If preferences for those edge cases are not clearly articulated, the LLM is left to assume and make a judgment call. Consider these scenarios: - What if there are major exceptions to this liability cap that weaken it so substantially that it might as well not exist? - What if this liability cap doesn’t apply to beta features? Or some specific set of features? What if the service is free? - What if the liability is capped at either 12 months’ fees or some fixed amount, whichever is greater? Or whichever is lesser? What if the fixed amount is very large? Or trivially small? - What if there is a secondary cap or a backup cap that applies under certain conditions at a much different amount than 12 months’ fees? - What if the cap is only for fees paid for the services that gave rise to the liability rather than fees paid in general under the agreement? In any of these cases, the LLM is left to assume. The leading LLMs are capable of reasoning through this type of nuance, but if they are told they must make a decision as to whether or not the standard passes, they might not make the same decision that you would if you don’t expose your preferences." 📊 If you want quantitative aggregate results to assess the risk of a portfolio of contracts, thousands of verbose "it depends" responses won't cut it. Contract professionals have strong opinions, institutional knowledge, and well-defined playbooks. ScreensAI helps them articulate this to the platform up front, systematically validating the types of LLM-powered results they'll get for the given set of expressions of their preferences. Then they can deploy this expression of preferences at scale in perpetuity. 🧙‍♂️ We think this is where a lot of the magic happens in LLM-powered contract review.

21

5 Comments

We've been sharing some benchmarks recently about Encore's pretty impressive performance (9x faster than Express.js). Now if you're curious, here's a brand new step by step guide for how to migrate your Express.js API to Encore.ts, unlocking that sweet sweet performance boost 🔥 https://lnkd.in/djzudGYn

21

Anthropic's Claude 3.5 Sonnet is really useful. I've been playing around with Claude's coding functionality and artifacts. Claude Artifacts basically allow Claude to generate content based on your prompts. This includes code, designs and even interactive stuff like simple games in a separate window. You literally watch Claude build what you want in super fast speed! I was explaining to an entrepreneur the difference between venture debt and venture capital. It's a bit difficult to illustrate the difference in equity dilution when a founder raise venture debt. I decided to generate a very simple venture debt dilution calculator using Claude. Checkout my artifact below and please give me ideas on features I should add onto my super simplistic creation. 🙂 https://lnkd.in/g3zBi2cd #AI #Claude #Copilot

2

My latest for TechCrunch: Another Black Exit! Open source compliance and security platform FOSSA has acquired developer community platform StackShare, the company confirmed to TechCrunch. Founded by Yonas Beshawred, StackShare is one of the more popular platforms for developers to discuss, track, and share the tools they use to build applications. This encompasses everything from which front-end JavaScript framework to use to which cloud provider to use for specific tasks. Read more to learn about the company and its acquisition!

61

2 Comments

How does the speed of SwiftData compare to Realm? What about memory usage? Size? ⚖ Do the results change if you're mutating 100 objects vs. 1,000,000? How about a simple data model vs. a complex one? 🤔 We have answers 🤓 https://lnkd.in/gpymY7wU

58

9 Comments

Replit Agent. If you don't know what this is, whether you code daily, or have never written a line of code. Take a few minutes to take a look - this is a complete game-changer IMO. Still the early days, but absolutely going to change how people build software, and who can build it. 🔗 to docs in comment section below.

2

1 Comment

Llama3 performs worse on curated "Hard" LMSys Arena benchmarks. These are more problem-specific complex prompts. Models trained for reasoning in fact go up ⬆️ in Elo scores on these hard examples. Looks like we need to curate the incoming user prompts and create benchmarks out of them. Fortunately, there is an endless supply of prompts from users! Curate not only the training data but also the eval data continuously! --- Subscribe and Discover more GenAI Tech insights with our newsletter: offnote.substack.com

9

HITRUST chose Vanta as the first pre-built HITRUST e1 solution, which means Vanta comes "out of the box" with the necessary controls, documents, and policies for an e1 assessment and eliminates the manual “do-it-yourself” approach that other platforms require. Up to 80% of requirements are automated by: *️⃣ Guidance around requirements: 44 new controls and 72 automated tests to ensure comprehensive and continuous compliance. *️⃣ Documents and policy templates: 80 new documents and 10 policy addendums to outline practices around managing sensitive data. *️⃣ Automated evidence collection: With over 300 integrations, Vanta automatically and continuously collects evidence from an organization’s technology stack. Our platform also applies overlapping, implemented controls from other supported frameworks, including up to 50% of SOC 2 and ISO 27001, to eliminate duplicative work across compliance programs. Coming soon, our partnership with HITRUST will expand into an integration with their audit portal, MyCSF, so customer evidence is transfered automatically from Vanta to MyCSF, further streamlining the HITRUST validation process Really excited for this collaboration!

251

14 Comments

Have you ever wondered what's possible to build off of the Parcl Labs API? To start, we open source all of our work. Why? Helping others build and understand the housing market differently is very important to us. Building a community to critique methods is very important to us. Together, we can evolve how housing is understood across many dimensions. A short summary of what we have open sourced to date: New Construction - Rates of growth broken out by product type - Percentage of for sale inventory that is new construction - Price performance across listings, sales and rentals - Hot spot analysis of on market, unit level inventory experiencing price reductions Unit Level Analysis - Unit level rental performance by buy box - Advanced on market unit level analysis - when are units changing prices, where are they increasing/decreasing, and at what velocity by every unit Real Time Housing Market Pricing - Technical analysis of daily updating housing market indices Experimental - Algorithm to detect markets in distress, spanning supply/demand trending skews coupled with strong seller signals/activity. Come build with us.

12

Really interesting blog post came up from Character.ai team! They handle 20,000 requests per second, so the task is challenging and needs to be optimized. How they did it: ➕ Reducing KV Cache: - 85% of layers use only Local Attention: Instead of standard attention, they use Local Attention (https://lnkd.in/gmE3iEE3), significantly reducing cache size, especially for long contexts. This approach is also used in Gemini and Gemma 2 (https://t.me/ai_newz/2925) and is reminiscent of Jamba (https://t.me/ai_newz/2519), where heavy attention was used only for certain layers. ➕ Multi-Query Attention: They use Multi-Query Attention (https://lnkd.in/grppzadw) instead of the currently dominant Group Query Attention, reducing cache size by eight times compared to the industry standard, albeit with significant quality drops. ➕ Shared KV Cache between layers: This method (https://lnkd.in/g3yRgi3d) reduces KV Cache size by over 20 times, enabling: - One machine can store the KV Cache of thousands of users. - Segmenting KV Cache for each message (second image) allows conversation continuation from any point without regenerating the cache. - Sticky Sessions are used to maintain the efficiency of over 95%, by directing users to servers where their KV Cache is already stored. ➕ Cost Reduction Benefit: Due to these optimizations, the cost of inference for the startup has decreased by 33 times over a year and a half, making it 13 times cheaper than the nearest competitor. Full Article here: https://lnkd.in/gXQrGNZx #AI #MachineLearning #ArtificialIntelligence #LLM #LocalAttention #MultiQueryAttention #KVCache #TechInnovation #CharacterAI #Startup #InferenceOptimization #Int8Training #AIWaifu #TechStartups #AIResearch #Transformers #Innovation #NoamShazeer #TechNews #AIModels

13

4 Comments

Thanks Joanne Chen and Jaya Gupta from Foundation Capital for the nod! Great to be in the cohort of top notch AI companies. Observability data is untapped goldmine of information, but when the time comes (in the middle of incident), it still requires "expert" skills to find the needle in the haystack. We at Deductive AI are democratizing the expertise to everyone in the organization.

35

4 Comments

Every startup CEO should be using this. Meet your new AI SDR.

22

4 Comments

pro tip: i made the "blazing fast" AI dev stack for y'all! ⌘ Langbase is supa fast!! Deploy your AI agent pipes today. Excited to see what y'all will ship with composable AI. Here's an entirely free and open-source chatbot. ⌘ Open Pipe: https://lnkd.in/g-wU_ZyF 📖 Open-source Chatbot: https://lnkd.in/gT5_wFkc Let's go!

69

11 Comments

Anthrpic's revenue is going to explode in 2025. Here's why: Claude Enterprise is priced at $60 per seat with a minimum of 70 seats. That’s a $50K annual commitment for: - leading frontier model - 500k context window - Projects - Artifacts - Native plugins (GitHub for now) Anthropic has added added Workspaces to the Anthropic Console too. Anthropic is most likely to facilitate text-to-action capabilities that meaningfully impact how frontier models can increase revenue and profitability of Enterprises, SMBs and SMEs. OpenAI will cede some of their marketshare in 2025 to Claude 3.5 Opus and Anthropic's innovative and customer-centric products.

1

Trying out Replit agents for the first time. Some thoughts: It’s quicker to go from idea to production app because of the full stack nature of it. Super easy to get prompted to set up secrets and environment variables like OpenAI API keys. I like the checkpoint nature of building that it breaks down the task into smaller requirements. It makes sure that you're fully satisfied with one requirement before proceeding with another One problem that I have found that it is hard to discard requirements that I don’t need anymore. That should be a quick update by Amjad Masad and team I find that there were no errors while executing a simple app, whereas I would've got a lot of errors while trying to build something with Cursor So Replit is probably better suited for beginners I was able to go from idea to production app in less than 10 minutes I really feel you're only limited by your ideas, taste, and distribution anymore What a time to be alive! How have you used replit agents or cursor so far? What do you think? Got any questions, reply or DM me!

7

2 Comments