data is a json blob from the perspective of SQLite, but it is parsed into a strictly typed structure in the Rust application code. After a year of collecting data, the SQLite database is already 7.6 GByte in size. Of course this is an extreme example of redundancy, but storing similar redundant data in databases is pretty common. SQLite doesn’t have any compression features, so of course I had to
Bolt is an algorithm for compressing vectors of real-valued data and running mathematical operations directly on the compressed representations. If you have a large collection of mostly-dense vectors and can tolerate lossy compression, Bolt can probably save you 10-200x space and compute time. Bolt also has theoretical guarantees bounding the errors in its approximations. EDIT: this repo now also
// By Rishabh Jain and Daniel Reiter Horn • Aug 04, 2020 Dropbox syncs petabytes of data every day across millions of desktop clients. It is vital that we constantly improve the sync experience for our users, to increase our users’ productivity in their everyday lives. We also constantly strive to better leverage our infrastructure, increasing Dropbox’s operational efficiency. Because the files th
DwarFS is a read-only file system with a focus on achieving very high compression ratios in particular for very redundant data. This probably doesn't sound very exciting, because if it's redundant, it should compress well. However, I found that other read-only, compressed file systems don't do a very good job at making use of this redundancy. See here for a comparison with other compressed file sy
SMASH: an efficient compression algorithm for microcontrollers One of the things that is driven from the top in SEGGER is that we can always do better. Not satisfied with standard schemes, we wanted to optimize emCompress, SEGGER’s compression library, for: Very fast decompression High compression ratio (uncompressed size divided by compressed size) Small decompressor Limited state in RAM when dec
====================================== Fastest (Branchless) Range Coder / Arithmetic Coder 100% C (C++ headers). OS/Arch: Linux amd/intel, arm, PowerPC, s390x, MacOs+Apple M1. Windows: Mingw, visual c++ No other Range Coder / Arithmetic Coder encode or decode faster with better compression Up to 3 times faster than the next fastest range coder with similar compression ratio Can work as bitwise or/
処理を実行中です
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く