Comment by edublancas

Comment by edublancas 2 days ago

View on Hacker News

Papermill is great but has quite some limitations because it spins up a new process to run the notebook:

- You cannot extract live variables (needed for testing)

- Cannot use pdb for debugging

- Cannot profile memory usage

You can do all of that with ploomber-engine (https://github.com/ploomber/ploomber-engine).

Disclaimer: I'm the author of this package

ziddoap 2 days ago

Not disclosed in this comment is that edublancas is

>Ploomber (YC W22) co-founder.

Reply View 3 replies

Kalanos 2 days ago

who is a great technologist with a lot of hands on experience. if it made sense to leverage papermill, he would have done so and focused on something else.

Reply View | 2 replies
- ziddoap 2 days ago
  
  What does any of this have to do with disclosure?
  
  Reply View | 1 reply
  
  Kalanos 2 days ago
  
  calling attention to disclosure suggests bias. i'm obviously saying that i trust him not to be biased.
  
  Reply View | 0 replies

throwpoaster 2 days ago

iirc, a few years back I was able to do all of these things with the Papermill IPython runtime.

Papermill is great, but yes: lots of room to hack on it and make it better.

Reply View 2 replies

edublancas 2 days ago

has papermill deprecated the ipython runtime? I used papermill extensively in the past and I never saw that in their docs.

Reply View | 1 reply
- throwpoaster a day ago
  
  It’s been a while but you do it with a custom kernel and maybe some entry point tweaks. IIRC.
  
  Reply View | 0 replies