Update on lineapy.to_pipeline
#742
mingjerli
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We are one step closer to removing the duplicated code execution in pipelines generated from
lineapy.to_pipeline
.Last week, we have merged the following two pull requests PR-714 and PR-734.
In PR-714, we've analyzed the session graph to identify the dependencies between any two artifacts within the same session and extract common code to avoid executing multiple times.
In PR-734, we've extended the work of PR-714 from a single session to multiple sessions, so we can easily stitch artifacts in multiple notebooks into a single pipeline.
Note that, we no longer need to specify task dependencies within the same session (thanks to PR-714), but we still need to specify task dependencies between different sessions(but only need one task from each session).
We are currently rewriting the current
lineapy.to_pipeline
using the work from these two PRs, and users should not expect any API changes once we update the implementation.Beta Was this translation helpful? Give feedback.
All reactions