Craig Kerstiens - Scalable Uniques in Postgres @ Postgres Open
3 likesâą2,431 views
This document discusses using HyperLogLog in PostgreSQL to estimate cardinality or unique counts within a small memory footprint. It introduces HyperLogLog concepts like KMV, bit patterns, and stochastic averaging. It then demonstrates creating a PostgreSQL extension, inserting data into an HLL column, and using HLL functions to estimate unique counts across rows and tables. It also covers tuning HLL parameters and best practices like batching updates. The document presents HLL as a way to estimate large unique counts with only 1280 bytes and a few percent error rate.
Craig Kerstiens - Scalable Uniques in Postgres @ Postgres Open
- 1. Scalable Uniques in Postgres - Craig Kerstiens Heroku Postgres
- 3. Truviso âą Extended Postgres to do streaming âą Various markets âą Ad space âą Wanted unique impressions âą Sort of wanted unique impressions
- 5. Approx Top K
- 7. HyperLogLog
- 8. HyperLogLog âą KMV - K minimum value
- 9. HyperLogLog âą KMV - K minimum value âą Bit observable patterns
- 10. HyperLogLog âą KMV - K minimum value âą Bit observable patterns âą Stochastic averaging
- 11. HyperLogLog âą KMV - K minimum value âą Bit observable patterns âą Stochastic averaging âą Harmonic averaging
- 12. HyperLogLog âą KMV - K minimum value âą Bit observable patterns âą Stochastic averaging âą Harmonic averaging
- 13. HyperLogLog âą KMV - K minimum value âą Bit observable patterns âą Stochastic averaging âą Harmonic averaging âą Implemented by Aggregate Knowledge
- 15. HyperLogLog Probabilistic uniques with small footprint
- 16. HyperLogLog Probabilistic uniques with small footprint Close enough distinct with small footprint
- 17. Use cases
- 18. Use cases âą Semi distinct count âą Think pg_stat_statements âą Ad networks âą Web traïŹc
- 19. Use cases âą Semi distinct count âą Think pg_stat_statements âą Ad networks âą Web traïŹc âą With rollups/groupings
- 20. Digging in CREATE  EXTENSION  hll;   CREATE  TABLE  helloworld  (       id     integer,       set    hll   );
- 21. Digging in CREATE  EXTENSION  hll;   CREATE  TABLE  helloworld  (       id     integer,       set    hll   );
- 22. Inserting data UPDATE  helloworld  SET  set  =  hll_add(set,  hll_hash_integer(12345))  WHERE  id  =  1; UPDATE  helloworld  SET  set  =  hll_add(set,  hll_hash_text('hello  world'))  WHERE  id  =  1;
- 23. Real world CREATE  TABLE  daily_uniques  (     date             date  UNIQUE,     users            hll );
- 24. Real world INSERT  INTO  daily_uniques(date,  users)   SELECT  date,  hll_add_agg(hll_hash_integer(user_id))   FROM  users   GROUP  BY  1;
- 25. Real world SELECT         EXTRACT(MONTH  FROM  date)  AS  month,         hll_cardinality(hll_union_agg(users)) FROM  daily_uniques WHERE  date  >=  '2012-Ââ01-Ââ01'  AND       date  <   '2013-Ââ01-Ââ01' GROUP  BY  1;
- 26. Real world SELECT         EXTRACT(MONTH  FROM  date)  AS  month,         hll_cardinality(hll_union_agg(users)) FROM  daily_uniques WHERE  date  >=  '2012-Ââ01-Ââ01'  AND       date  <   '2013-Ââ01-Ââ01' GROUP  BY  1;
- 27. Good practices
- 28. Good practices
- 29. Good practices âą It uses update
- 30. Good practices âą It uses update âą Do as a batch in most cases
- 31. Good practices âą It uses update âą Do as a batch in most cases âą Tweak the conïŹg
- 33. Tuning Parameters âą log2m - log base 2 of registers âą Between 4 and 17 âą Each 1 increase doubles storage
- 34. Tuning Parameters âą log2m - log base 2 of registers âą Between 4 and 17 âą Each 1 increase doubles storage âą regwidth - bits per register
- 35. Tuning Parameters âą log2m - log base 2 of registers âą Between 4 and 17 âą Each 1 increase doubles storage âą regwidth - bits per register âą expthresh - threshold for explicit vs sparse
- 36. Tuning Parameters âą log2m - log base 2 of registers âą Between 4 and 17 âą Each 1 increase doubles storage âą regwidth - bits per register âą expthresh - threshold for explicit vs sparse âą spareson - on/oïŹ for sparse
- 37. Is it better?
- 38. 1280 bytes Estimate count of 10s of billions Few percent error
- 39. Resources âą https://github.com/aggregateknowledge/ postgresql-hll âą http://blog.aggregateknowledge.com/ 2013/02/04/open-source-release- postgresql-hll/ âą http://tapoueh.org/blog/2013/02/25- postgresql-hyperloglog
- 40. Questions