Comment by edublancas

Comment by edublancas 10 months ago

View on Hacker News

Papermill is great but has quite some limitations because it spins up a new process to run the notebook:

- You cannot extract live variables (needed for testing)

- Cannot use pdb for debugging

- Cannot profile memory usage

You can do all of that with ploomber-engine (https://github.com/ploomber/ploomber-engine).

Disclaimer: I'm the author of this package

ziddoap 10 months ago

Not disclosed in this comment is that edublancas is

>Ploomber (YC W22) co-founder.

Reply View 4 replies

Kalanos 10 months ago

who is a great technologist with a lot of hands on experience. if it made sense to leverage papermill, he would have done so and focused on something else.

Reply View | 3 replies
- ziddoap 10 months ago
  
  What does any of this have to do with disclosure?
  
  Reply View | 2 replies
  
  Kalanos 10 months ago
  
  calling attention to disclosure suggests bias. i'm obviously saying that i trust him not to be biased.
  
  Reply View | 1 reply
  
  fastasucan 10 months ago
  
  Not obvious at all. And no. It doesn't have to suggest bias, just a lack of disclosure.
  
  Reply View | 0 replies

throwpoaster 10 months ago

iirc, a few years back I was able to do all of these things with the Papermill IPython runtime.

Papermill is great, but yes: lots of room to hack on it and make it better.

Reply View 2 replies

edublancas 10 months ago

has papermill deprecated the ipython runtime? I used papermill extensively in the past and I never saw that in their docs.

Reply View | 1 reply
- throwpoaster 10 months ago
  
  It’s been a while but you do it with a custom kernel and maybe some entry point tweaks. IIRC.
  
  Reply View | 0 replies