Comment by dmos62

I'm working on a plaintext, decetralized, distributed (multi-device), trustless (multi-user), immutable, schemaless database that's syncable via SyncThing. The trustless part, fundamentally, comes from records being signed by the authors, each device/user keeping track of signatures (ids) seen, and each device periodically publishing all ids it's aware of: that seems to defeat all attacks or accidental corruptions I care about. My most demanding use case is multi-user expense tracking. Legal/financial document (a la google drive), family photo tracking (a la google photos) is also up there.

Also, I've recently picked up modeling my financial choices and circumstances in Python. Modeling uncertainty is especially interesting I've found. I might share the Jupyter notebook some time to get some feedback.

It's early days.

- It will work as a library that a tool (e.g. family finance tracker) would be based on;

- Every record will be immutable and undeleteable; the whole thing is space inefficient, though I've some mechanisms in mind for pruning away unnecessary records, and it's just plain text, so I'm not worried: should compress well; I wouldn't envision something like this working well on a very large scale though;

- Editing of preexisting records will be implemented as adding a new record that simply references the previous one as its previous version; also, you can implement a ledger by creating a parent-child chain (though the tracking of signatures I mentioned previously might be a simpler approach);

- I like the append-only model because it gives you history of edits for everything, and protects you in case of mistakes (or malice);

- You'll be able to collaborate on records (and the whole db) with other devices/people; every record will be signed by its author/device; conflicting edits (records marking the same record as their parent) will not be deconflicted automatically by the db: the high level app can choose how it wants to do that: throw up an alarm for the user, ignore the older fork, whatever;

- SyncThing-compatibility will be achieved by simply having each device only edit files specific to it; there won't be a single file that is edited by more than a single device/user, thus, SyncThing conflicts can't happen;

- The db will have fairly good guarantees: if it runs its checks and tells you all's good, you can be sure that records were not changed, there is not a new record masquerading as an old record, a record was not secretely deleted, records weren't reordered, another device didn't edit some other device's files, every record's author is its author;

- It was important for me to make the database easily editable and readable via regular text editors, i.e. not only plaintext, but text-editing-centric, but I've not found a way to do that at the lowest level; right now I'm contemplating using some variant of JSON for the records; however, I do envision a text-interface where the database, or a database query/subset, is rendered as a text file, and edits are automatically picked up by a daemon and translated into new records in the actual db, but that would be just another tool using this db, not a feature of the db itself;

- Like for anything synced via SyncThing, the user (or an app using the db) will want to implement some kind of backup (SyncThing is not meant for backups), unless the data is not sensitive to loss.

ZoeNews 4 days ago

As someone that uses Syncthing for pretty much all the usecases you mentioned that sounds very interesting! Do you mind sharing some details about the process so far?

Reply View 1 reply

dmos62 4 days ago

It's early days.
- It will work as a library that a tool (e.g. family finance tracker) would be based on;
- Every record will be immutable and undeleteable; the whole thing is space inefficient, though I've some mechanisms in mind for pruning away unnecessary records, and it's just plain text, so I'm not worried: should compress well; I wouldn't envision something like this working well on a very large scale though;
- Editing of preexisting records will be implemented as adding a new record that simply references the previous one as its previous version; also, you can implement a ledger by creating a parent-child chain (though the tracking of signatures I mentioned previously might be a simpler approach);
- I like the append-only model because it gives you history of edits for everything, and protects you in case of mistakes (or malice);
- You'll be able to collaborate on records (and the whole db) with other devices/people; every record will be signed by its author/device; conflicting edits (records marking the same record as their parent) will not be deconflicted automatically by the db: the high level app can choose how it wants to do that: throw up an alarm for the user, ignore the older fork, whatever;
- SyncThing-compatibility will be achieved by simply having each device only edit files specific to it; there won't be a single file that is edited by more than a single device/user, thus, SyncThing conflicts can't happen;
- The db will have fairly good guarantees: if it runs its checks and tells you all's good, you can be sure that records were not changed, there is not a new record masquerading as an old record, a record was not secretely deleted, records weren't reordered, another device didn't edit some other device's files, every record's author is its author;
- It was important for me to make the database easily editable and readable via regular text editors, i.e. not only plaintext, but text-editing-centric, but I've not found a way to do that at the lowest level; right now I'm contemplating using some variant of JSON for the records; however, I do envision a text-interface where the database, or a database query/subset, is rendered as a text file, and edits are automatically picked up by a daemon and translated into new records in the actual db, but that would be just another tool using this db, not a feature of the db itself;
- Like for anything synced via SyncThing, the user (or an app using the db) will want to implement some kind of backup (SyncThing is not meant for backups), unless the data is not sensitive to loss.

Reply View | 0 replies