Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ability to execute notebooks via the CLI #643

Open
bt-maps opened this issue Oct 25, 2021 · 9 comments
Open

Add the ability to execute notebooks via the CLI #643

bt-maps opened this issue Oct 25, 2021 · 9 comments
Labels
area:CLI Related to the CLI discussion Needs to be discussed before moving forward

Comments

@bt-maps
Copy link

bt-maps commented Oct 25, 2021

Add options to cli to inject input parameters at runtime and allow parameterized livebooks to programmatically execute a workflow/broadway multi-stage data pipeline.

Basically, allow livebooks with inputs (or exported scripts) to be used as-is in multi-stage broadway pipelines and pass values downstream.

Brent

@josevalim
Copy link
Contributor

You can use inputs to parameterized Livebook and fill those in the UI. Because our inputs use the IO API, they also work when the notebooks run in the CLI (i.e. we will request the user to fill info in as it happens). So this functionality is already in place but in a way.

The biggest issue is that we don't have an option to execute notebooks via the CLI.

@josevalim josevalim changed the title Add the ability to execute and parameterize livebooks at run-time Add the ability to execute notebooks via the CLI Oct 25, 2021
@jonatanklosko jonatanklosko added area:CLI Related to the CLI discussion Needs to be discussed before moving forward labels Dec 29, 2021
@jonatanklosko
Copy link
Member

Depends on livebook-dev/kino#54.

@houllette
Copy link

I was hoping to bump this issue as I'm currently working on a pretty novel idea using Livebooks in an educational manner and would like a way to execute notebooks via the CLI in order to automate grading code output. Any help would be appreciated! 😄

@jonatanklosko
Copy link
Member

Hey @houllette, this is not a priority for us right now, but I put together a script for basic evaluation here. It needs to be run within the Livebook mix project:

git clone https://github.com/livebook-dev/livebook.git
cd livebook
mix dev.setup

# Download the script and use as:
mix run livebook_run.exs ./path/to/notebook.livemd

Hopefully this helps or gives you a starting point :)

@houllette
Copy link

Hey @jonatanklosko - Sorry to bother you again after so much time as elapsed since you wrote the aforementioned script. I was just starting to work with it more and am running into issues - I tried debugging it myself as best as possible but it appears I'm running into a wall.

I'm getting an error when the code enters the Livebook.Session.Data.apply_operation/2 function during your script execution on Line 38. The error I'm specifically seeing is 1st argument: not an atom.

At first I thought it was the .livemd file I was running the script on, but I've come to realize that it won't work even on just a super basic .livemd file that only has one code block returning an atom.

@jonatanklosko
Copy link
Member

Hey @houllette! There was an internal change that broke it, I've just updated the script :)

@sonic182
Copy link

sonic182 commented Jan 9, 2024

In jupyter there is nbconvert for notebooks to other formats (python scripts included of course) https://nbconvert.readthedocs.io/en/latest/index.html

It may be nice to have something like nbconvert in elixir for livebooks

@dderooy
Copy link

dderooy commented May 30, 2024

Bumping this idea b/c I think its great. These features would open up lots of potential new app ideas.

> livebook run ./playbook.livemd -x someval -y anotherval

Also a companion hex package would be great too.

output_livemd = livebook.run(file, params) 

An app user story:

  1. user creates a livebook to automate events from a stream (to do ETL, data enrichments etc).
  2. user saves livebook and goes to do other things in life.
  3. an orchestrator process consumes the stream and runs the livebook for each event. A runtime log is created and saved.
  4. User later logs into app and can view a runtime for some past event. When the livebook is loaded for the past event, any front end rendered things (visualizations etc) are created from saved output of code cells.

Applications: ETL, Security SOAR, if-this-then-that low code apps, any event driven system handling.

Livebook is great b/c it can bridge the gap between non-coders and developers. In the security industry as example, a developer could make an advanced livebook to run and enrich a security alert stream. Later a SOC analyst could investigate an enriched alert with an already-ran livebook and see pretty readable output with visualizations etc.

@joshuataylor
Copy link

Sorry for the bump, for those coming from Jupyter Notebooks (plus to help SEO), this would be equivalent to the following command:

jupyter nbconvert --inplace --to notebook --execute example.ipynb

docs for reference

Anything more complicated than that requires papermill etc.

My use case is I could edit this outside the browser and rerun the workbook as I need.

However, IMHO, if you need to do this as a script, please write a mix task or consider something else :).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:CLI Related to the CLI discussion Needs to be discussed before moving forward
Projects
None yet
Development

No branches or pull requests

7 participants