Comment by recursivecaveat
Comment by recursivecaveat 2 days ago
My dream binary format is schema driven, as compact and efficient as Capt Proto or such, but just optionally embeds the entire schema into the message. Then we can write a vim plugin that just opens the file in human readable form without having to fish for the schema. Whenever I am using binary formats, it's because I have a list of millions of objects of the same types. Seems to me that you may as well tack 1KB of schema onto a 2GB message and make it self-describing so everyone's life is easier.
It’s possible to build this around protobuf. Google has a rich internal protobuf ecosystem that does this and supports querying large amounts of protobuf data without specifying schemas. They are only selectively open sourced. Have a look at riegeli if you are interested.
https://github.com/google/riegeli