Comment by stephen

Comment by stephen a day ago

I really want to use pipelining for our "em.flush" of sending all INSERTs & UPDATEs to the db as part of a transaction, b/c my initial prototyping showed a 3-6x increase:

https://joist-orm.io/blog/initial-pipelining-benchmark/

If you're not in a transaction, afaiu pipelining is not as applicable/useful b/c any SQL statement failing in the pipeline fails all other queries after it, and imo it would suck for separate/unrelated web requests that "share a pipeline" to have one request fail the others -- but for a single txn/single request, these semantics are what you expect anyway.

Unfortunately in the TypeScript ecosystem, the node-pg package/driver doesn't support pipelining yet, instead this "didn't quite hit mainstream adoption and now the author is AWOL" driver does: https://github.com/porsager/postgres

I've got a branch to convert our TypeScript ORM to postgres.js solely for this "send all our INSERTs/UPDATEs/DELETEs in parallel" perf benefit, and have some great stats so far:

https://github.com/joist-orm/joist-orm/pull/1373#issuecommen...

But it's not "must have" for us atm, so haven't gotten time to rebase/ship/etc...hoping to rebase & land the PR by eoy...

porsager a day ago

I'm right here - what are you missing?

Reply View 5 replies

stephen a day ago

Oh hello! Very happy to hear from you, and even happier to be wrong about your "AWOL-ness" (since I want to ship postgres.js to prod). :-)
My assumption was just from, afaict, the general lack of triage on GitHub issues, i.e. for a few needs we have like tracing/APM, and then also admittedly esoteric topics like this stack trace fixing:
https://github.com/porsager/postgres/issues/963#issuecomment...
Fwiw I definitely sympathize with issue triage being time-consuming/sometimes a pita, i.e. where a nontrivial/majority of issues are from well-meaning but maybe naive users asking for free support/filing incorrect/distracting issues.
I don't have an answer, but just saying that's where my impression came from.
Thanks for replying!

Reply View | 4 replies
- porsager a day ago
  
  Thanks a lot. You're spot on about issue triage etc. I haven't had the time to keep up, but I read all issues when they're created and deal with anything critical. I'm using Postgres.js myself in big deployments and know others are too. The metrics branch should be usable, and I could probably find time to get that part released. It's been ready for a while. I do have some important changes in the pipeline for v4, but won't be able to focus on it until December.
  
  Reply View | 1 reply
  
  stephen a day ago
  
  Great to hear you're using postgres.js in prod/large deployments! That sort of real-world-driven usage/improvements/roadmap imo leads to the best results for open source projects.
  Also interesting about a potential v4! I'll keep lurking on the github project and hope to see what it brings!
  
  Reply View | 0 replies
- hombre_fatal a day ago
  
  That was a pretty nasty assumption you made about them though: That they're MIA because they're upset that their pet project isn't as popular as they'd like.
  Jeez.
  That said, I hope node-postgres can support this soon. As it stands, every single query you add to a transaction adds a serial network roundtrip which is devastating not just in execution time but how long you're holding any locks inside the transaction.
  
  Reply View | 1 reply
  
  porsager a day ago
  
  Hehe. I didn't read it like that at all, so no worries
  
  Reply View | 0 replies