-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Alternative interchange formats #58
Comments
Here is an extremely good write up on the shortcomings of Apache arrow seems to very much be a solution to many of the prior memory constraint and inter-process ailments of big data with Anyone wanting to take a look at the ipc section in |
You might be interested in this as well: https://github.com/real-logic/aeron and the binary encoding it uses: https://github.com/real-logic/simple-binary-encoding Designed for extremely low latency trading systems. There is a C++ implementation, and there is no python interface atm though. Not sure exactly what sauce they are using that is better than say, CapNProto. All of them are probably useful in different situations. Which complicates things.. Blosc AFAIK is just a compression algorithm. Still useful, and can be used transparently (would require intelligence about when data is moving over I/O), but perhaps should be a user level thing. My suspicion is that Arrow has compression specifically accounted for, although I don't know. |
For the sake of interestingness, although its likely of no use to use is: https://kaitai.io/ |
Also #8 mentions While not a new interchange it is a system worth comparing against when considering alternatives. |
Interesting historical format SBE - simple binary encoding that's (was?) used in financial systems.
Sounds like it would need to be compared with |
The list I've been meaning to look through/support:
json
(if peeps want it overmsgpack
)pycapnp
Maybe more?
We'll need to abstract the channel API to take in different types of stream types.
This work will require coordination for the alt transport work in #19.
The text was updated successfully, but these errors were encountered: