Timescale

PostgreSQL and pgvector: Now Faster than Pinecone, 75% cheaper, 100% open-source. Introducing pgvectorscale, an open-source PostgreSQL extension that builds on pgvector, enabling greater performance and scalability. Here’s how pgvectorscale helps pgvector outperform specialized vector database like Pinecone: 1️⃣ StreamingDiskANN:  A new vector search index that overcomes limitations of in-memory indexes like HNSW the index on disk, making it more cost-efficient to run and scale as vector workloads grow. Inspired by the DiskANN paper from Microsoft. 2️⃣ Statistical Binary Quantization (SBQ): Developed by researchers at Timescale, this technique improves on standard binary quantization techniques by improving accuracy when using quantization to reduce the space needed for vector storage 3️⃣ Written in Rust, giving the PostgreSQL community to contribute to vector support. 📈The result? On our benchmark of 50 million Cohere embeddings (768 dimensions each), PostgreSQL with pgvector and pgvectorscale achieves 28x lower p95 latency and 16x higher query throughput compared to Pinecone for approximate nearest neighbor queries at 99 % recall, all at 75 % less cost when self-hosted on AWS EC2. We also tested it against Pinecone’s p2 high performance index, see the blog post at the end of this post for full results (spoiler: It’s just as impressive). Pgvectorscale is open-source under the PostgreSQL license and free for you to use on any PostgreSQL database for your AI projects. To get started, see the pgvectorscale github repo: https://lnkd.in/ghXj2e-U Or try it on Timescale Cloud on any new database service. Eager to learn more about pgvectorscale and how it works? Head over to our blog post with all the details: https://lnkd.in/gcMcxrVb

If you're building an AI app with PostgreSQL/ pgvector, you'll probably need to add filters to your semantic search to get better results. Here's an overview of 5 most useful filter options for building search and RAG apps: 𝟏/ 𝐌𝐞𝐭𝐚𝐝𝐚𝐭𝐚 𝐟𝐢𝐥𝐭𝐞𝐫 Use case: A technical documentation search system for a software company with multiple products. This query searches for documents related to the CRM Software product, specifically API reference documents. 𝟐/ 𝐂𝐨𝐦𝐩𝐨𝐬𝐢𝐭𝐞 𝐟𝐢𝐥𝐭𝐞𝐫 Use case: An e-commerce product recommendation system that considers both user preferences and product attributes. This query combines multiple filters to find relevant products within specific categories, price range, stock status, and rating. 𝟑/ 𝐓𝐢𝐦𝐞-𝐛𝐚𝐬𝐞𝐝 𝐟𝐢𝐥𝐭𝐞𝐫 Use case: A news article recommendation system that prioritizes recent content. This query retrieves semantically similar news articles published within the last 7 days. 𝟒/ 𝐏𝐞𝐫𝐦𝐢𝐬𝐬𝐢𝐨𝐧𝐬-𝐛𝐚𝐬𝐞𝐝 𝐟𝐢𝐥𝐭𝐞𝐫 Use case: A company-wide knowledge base where access to certain documents is restricted based on user roles. (This is super common for internal RAG apps!) This query ensures that only documents the user has permission to access are included in the search results. 𝟓/ 𝐆𝐞𝐨-𝐬𝐩𝐚𝐭𝐢𝐚𝐥 𝐟𝐢𝐥𝐭𝐞𝐫 Use case: A location-based service recommendation system for tourists. This query combines semantic search with geospatial filtering to find tourist attractions within a 5km radius of the user's location, ordered by semantic relevance and distance. 𝐏𝐫𝐨-𝐭𝐢𝐩: Use pgvectorscale and the StreamingDiskANN index to get higher search accuracy for filtered search. You can learn more about how streaming filtering works in the blog post linked in the comments: pgvectorscale is open-source and free to use under the PostgreSQL license. Head over to GitHub to get started. I hope this thread was helpful, let me know what other topics you'd like me to cover regarding building RAG and search systems with Postgres and pgvector. #semanticsearch #vectordatabase #postgresql #rag #pgvector #devtool #pinecone #search

Just ICYMI: The 2024 State of PostgreSQL survey is now open!💥🤯 It aims to gather insights from the PostgreSQL community and help track trends such as the increasing popularity of PostgreSQL, growth in managed services adoption, and evolving learning preferences.✨ Last year's report showed a rise in PostgreSQL usage among small businesses and developers, with its open-source nature, reliability, and strong documentation being top reasons for its use. The survey will remain open until September 30, 2024. Take it now! https://lnkd.in/gER82F4y

Data/AI folks in New York - Data Driven NYC #114 just announced (September 18 at 6pm) 🚨 🚨 🚨 * Captions AI with co-founder Dwight Churchill * Timescale with co-founder 🐯 Michael Freedman * TitanML with co-founder Meryem Arik Everyone welcome! But you must RSVP, link below 👇 #AI #ML #data #startups #LLM

New webinar alert! 🚨 https://lu.ma/0v7nwfxd

Me and Sven (Timescale colleague) gave a talk about improving #PostgreSQL plans by “constifying” expressions at #pgibiza2024. It’s an optimization we’ve implemented in TimescaleDB. References: https://lnkd.in/dvJb_pmi https://lnkd.in/d8SJjpQ4

I came across this article about "SQL as fast as NoSQL" today (as always great content Franck Pachot and Yugabyte). I know it's not the point of the article, but what took my eye was the table showing independent ingest benchmarking, with TimescaleDB topping the list 👀. I got old mate ChatGPT to work out the relative ingest rates as percentages 🔥: --- Here are the relative insert speeds as percentages compared to TimescaleDB #TimescaleDB: 100.00% (600,000 inserts/sec) #InfluxDB: 76.67% (460,000 inserts/sec) #PostgreSQL: 71.33% (428,000 inserts/sec) #Cassandra async inserts: 68.33% (410,000 inserts/sec) #Cassandra sync inserts: 64.83% (389,000 inserts/sec) #YugabyteDB YSQL: 49.17% (295,000 inserts/sec) #YugabyteDB YCQL: 48.00% (288,000 inserts/sec) #Elasticsearch: 28.33% (170,000 inserts/sec) #ArrangoDB: 22.83% (137,000 inserts/sec) #CockroachDB: 15.17% (91,000 inserts/sec)

Time After Time: Why the ‘When’ Matters 🤷 This DM Radio episode features insights from Eric Kavanagh and 🔥 Avthar Sewrathan as they lead the discussion exploring how the timing of data processing is critical in modern database architectures. The episode delves into the shift toward multi-modal databases and the growing importance of real-time analytics. Tune in to understand how these innovations are shaping the future of data management and why timing truly matters in this field.👇🤓 https://lnkd.in/gUmXhVAY

𝗟𝗲𝗮𝗿𝗻 𝗳𝗿𝗼𝗺 𝗙𝗼𝘂𝗻𝗱𝗲𝗿𝘀, 𝗖𝗘𝗢𝘀, 𝗮𝗻𝗱 𝘁𝗵𝗶𝗻𝗸𝗲𝗿𝘀 𝗮𝘁 𝘁𝗵𝗲 𝗳𝗼𝗿𝗲𝗳𝗿𝗼𝗻𝘁 𝗼𝗳 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴, 𝗔𝗜, 𝗮𝗻𝗱 𝗕𝗶𝗴 𝗗𝗮𝘁𝗮 𝗮𝘁 𝗗𝗮𝘁𝗮 𝗗𝗿𝗶𝘃𝗲𝗻 𝗡𝗬𝗖 — 𝗮𝗻 𝗲𝘃𝗲𝗻𝘁 𝗳𝗲𝗮𝘁𝘂𝗿𝗶𝗻𝗴 𝗖𝗮𝗽𝘁𝗶𝗼𝗻𝘀 𝗔𝗜, 𝗧𝗶𝗺𝗲𝘀𝗰𝗮𝗹𝗲, 𝗮𝗻𝗱 𝗧𝗶𝘁𝗮𝗻𝗠𝗟 ➡ 𝗗𝗮𝘁𝗲 𝗮𝗻𝗱 𝘁𝗶𝗺𝗲: Wednesday, September 18 · 5:45 - 8pm EDT ➡ 𝗟𝗼𝗰𝗮𝘁𝗶𝗼𝗻: 19 West 22nd Street # fl 2 New York, NY 10010 ➡ 𝗣𝗿𝗲𝘀𝗲𝗻𝘁𝗲𝗿𝘀: • Dwight Churchill, Co-Founder & COO of Captions AI • Mike Freedman, Co-Founder & CEO of Timescale • Meryem Arik, Co-Founder & CEO of TitanML 𝗥𝗲𝘀𝗲𝗿𝘃𝗲 𝗮 𝘀𝗽𝗼𝘁: https://lnkd.in/gten-H3j #timescale #machinelearning #ai #bigdata #nyc #events

Love Postgres? Now's your chance to have your say in the 2024 State of Postgres Survey hosted by Timescale. We're proudly supporting the survey this year and kindly ask that if you use Postgres, please share your experience. All feedback provides invaluable insights for improving Postgres and the supporting community. Take the survey 👉 https://lnkd.in/gER82F4y