Comment by rienbdj
A new McSharry post! Excellent
Last I checked, VMWare had moved away from differential datalog?
A new McSharry post! Excellent
Last I checked, VMWare had moved away from differential datalog?
They did, and their product is great.
It is the only database/query engine that allows you to use the same SQL for both batch and streaming (with UDFs).
I have made an accessible version of a subset of Differential Dataflow (DBSP) in Python right here: https://github.com/brurucy/pydbsp
DBSP is so expressive that I have implemented a fully incremental dynamic datalog engine as a DBSP program.
Think of SQL/Datalog where the query can change in runtime, and the changes themselves (program diffs) are incrementally computed: https://github.com/brurucy/pydbsp/blob/master/notebooks/data...
> It is the only database/query engine that allows you to use the same SQL for both batch and streaming (with UDFs).
Flink SQL also checks that box.
Flink SQL is quite limited compared to Feldera/DBSP or Frank’s Materialize.com, and has some correctness limitations: it’s “eventually consistent” but until you stop the data it’s unlikely to ever be actually correct when working with streaming joins. https://www.scattered-thoughts.net/writing/internal-consiste...
Not true.
There has to be some change in the code, and they will not share the same semantics (and perhaps won't work when retractions/deletions also appear whilst streaming). And let's not even get to the leaky abstractions for good performance (watermarks et al).
The Differential Datalog team founded Feldera: https://www.feldera.com/
They switched from differential Datalog to differential SQL, I think because they realized Datalog is a really tough sell.