Skip to content

Commit

Permalink
Simple quote marks
Browse files Browse the repository at this point in the history
  • Loading branch information
ewels committed Mar 27, 2024
1 parent 40fb7b4 commit e2e9b79
Showing 1 changed file with 6 additions and 6 deletions.
12 changes: 6 additions & 6 deletions src/content/blog/2024/nextflow-colored-logs.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ author: Phil Ewels
icon: phil.jpg
---

Nextflow is a command-line interface (CLI) tool that runs in the terminal. Everyone who has launched Nextflow from the command line knows what it’s like to follow the console output as a pipeline runs: the excitement of watching jobs zipping off as they’re submitted, the satisfaction of the phrase _Pipeline completed successfully!_ and occasionally, the sinking feeling of seeing an error message.
Nextflow is a command-line interface (CLI) tool that runs in the terminal. Everyone who has launched Nextflow from the command line knows what it’s like to follow the console output as a pipeline runs: the excitement of watching jobs zipping off as they’re submitted, the satisfaction of the phrase _"Pipeline completed successfully!"_ and occasionally, the sinking feeling of seeing an error message.

Because the CLI is the primary way that people interact with Nextflow, a little bit of polish can have a big effect. In this article, I’m excited to describe an upgrade for the console output that should make monitoring workflow progress just a little easier.

Expand All @@ -22,7 +22,7 @@ NXF_EDGE=1 nextflow self-update

## Background

The Nextflow console output hasn’t changed much over the 10 years that it’s been around. The biggest update happened in 2018 when ANSI logging was released in version `18.10.0`. This replaced the stream of log messages announcing each task submission with a view that updates dynamically, giving an overview of each process. This gives an overview of the pipeline’s progress rather than being swamped with thousands of individual task submissions.
The Nextflow console output hasn’t changed much over the 10 years that it’s been around. The biggest update happened in 2018 when "ANSI logging" was released in version `18.10.0`. This replaced the stream of log messages announcing each task submission with a view that updates dynamically, giving an overview of each process. This gives an overview of the pipeline’s progress rather than being swamped with thousands of individual task submissions.

<figure>
<img src="/img/blog-nextflow-colored-logs/nextflow_log_with_without_ansi.png" alt="Nextflow console output with and without ANSI logging">
Expand All @@ -31,7 +31,7 @@ The Nextflow console output hasn’t changed much over the 10 years that it’s
</figcaption>
</figure>

I can be a little obsessive tool user interfaces. The nf-core template, as well as MultiQC and nf-core/tools all have coloured terminal output, mostly using the excellent [textualize/rich](https://github.com/Textualize/rich). I’ve also written a couple of general-use tools around this such as [ewels/rich-click](https://github.com/ewels/rich-click/) for Python CLI help texts, and [ewels/rich-codex](https://github.com/ewels/rich-codex) to auto-generate screenshots from code / commands in markdown. The problem with being surrounded by so much colored CLI output is that any tools _without_ colors start to stand out. Dropping hints to the Nextflow team didn’t work, so eventually I whipped up [a proposal](https://github.com/nextflow-io/nextflow/issues/3976) of what the console output could look like using the tools I knew: Python and Rich. Paolo knows me well and [offered up a bait](https://github.com/nextflow-io/nextflow/issues/3976#issuecomment-1568071479) that I couldn’t resist: _“Phil. I think this a great opportunity to improve your Groovy skills 😆”._
I can be a little obsessive tool user interfaces. The nf-core template, as well as MultiQC and nf-core/tools all have coloured terminal output, mostly using the excellent [textualize/rich](https://github.com/Textualize/rich). I’ve also written a couple of general-use tools around this such as [ewels/rich-click](https://github.com/ewels/rich-click/) for Python CLI help texts, and [ewels/rich-codex](https://github.com/ewels/rich-codex) to auto-generate screenshots from code / commands in markdown. The problem with being surrounded by so much colored CLI output is that any tools _without_ colors start to stand out. Dropping hints to the Nextflow team didn’t work, so eventually I whipped up [a proposal](https://github.com/nextflow-io/nextflow/issues/3976) of what the console output could look like using the tools I knew: Python and Rich. Paolo knows me well and [offered up a bait](https://github.com/nextflow-io/nextflow/issues/3976#issuecomment-1568071479) that I couldn’t resist: _"Phil. I think this a great opportunity to improve your Groovy skills 😆"._

## Showing what’s important

Expand All @@ -44,9 +44,9 @@ The console output shown by Nextflow describes a range of information. Much of i
</figcaption>
</figure>

With some judicious use of the `dim` style, we can make less important information fade into the background. For example, the stem of the fully qualified process identifiers now step back to allow the process name to stand out. Secondary information such as the number of tasks that were cached, or the executor that is being submitted to, are still there to see but take a back seat. Doing the reverse with some `bold` text helps to highlight the run name – key information for identifying and resuming pipeline runs. Using color allows different fields to be easily distinguished, such as process labels and task hashes. Greens, blues and reds in the task statuses allow a reader to get an impression of the run progress without needing to read every number.
With some judicious use of the `dim` style, we can make less important information fade into the background. For example, the "stem" of the fully qualified process identifiers now step back to allow the process name to stand out. Secondary information such as the number of tasks that were cached, or the executor that is being submitted to, are still there to see but take a back seat. Doing the reverse with some `bold` text helps to highlight the run name – key information for identifying and resuming pipeline runs. Using color allows different fields to be easily distinguished, such as process labels and task hashes. Greens, blues and reds in the task statuses allow a reader to get an impression of the run progress without needing to read every number.

Probably the most difficult aspect technically was the `NEXTFLOW` header line. I knew I wanted to use the “Nextflow green” here, or as close to it as possible. But colors in the terminal are tricky. What the ANSI standard defines as `green`, `black` and `blue` can vary significantly across different systems and terminal themes. Some people use a light-color scheme and others run in dark mode. This hadn’t mattered much for most of the colors up until this point, I could use the [Jansi](https://github.com/fusesource/jansi) library to use named colors and they should look ok. But for the specific RGB of the _Nextflow Green_ I had to [hardcode specific ANSI control characters](https://github.com/nextflow-io/nextflow/blob/c9c7032c2e34132cf721ffabfea09d893adf3761/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy#L379-L389). But it got worse - it turns out that the default Terminal app that ships with OS X only supports 256 colors, so I had to find the closest match (_“light sea green”_ if you’re curious). Even once the green was ok, using `black` as the text color meant that it would actually render as white with some terminal color themes and be unreadable. In the end, the header text is a very dark gray.
Probably the most difficult aspect technically was the `NEXTFLOW` header line. I knew I wanted to use the "Nextflow green" here, or as close to it as possible. But colors in the terminal are tricky. What the ANSI standard defines as `green`, `black` and `blue` can vary significantly across different systems and terminal themes. Some people use a light-color scheme and others run in dark mode. This hadn’t mattered much for most of the colors up until this point, I could use the [Jansi](https://github.com/fusesource/jansi) library to use named colors and they should look ok. But for the specific RGB of the _Nextflow Green_ I had to [hardcode specific ANSI control characters](https://github.com/nextflow-io/nextflow/blob/c9c7032c2e34132cf721ffabfea09d893adf3761/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy#L379-L389). But it got worse - it turns out that the default Terminal app that ships with OS X only supports 256 colors, so I had to find the closest match (_"light sea green"_ if you’re curious). Even once the green was ok, using `black` as the text color meant that it would actually render as white with some terminal color themes and be unreadable. In the end, the header text is a very dark gray.

<figure>
<img src="/img/blog-nextflow-colored-logs/testing_terminal_themes.png" alt="Testing many horrible terminal themes">
Expand Down Expand Up @@ -78,7 +78,7 @@ The end result is console output that makes the most of the available space in y

## Contributing to Nextflow

Despite building tools that use Nextflow for many years, I’ve spent relatively little time venturing into the main codebase myself. Just as with any contributor, part of the challenge was figuring out how to build Nextflow, how to navigate its code structure and how to write tests. I found it quite a fun experience, so I described and demoed the process in a recent nf-core Bytesize talk titled [Contributing to Nextflow](https://nf-co.re/events/2024/bytesize_nextflow_dev). You can watch the talk on [YouTube](https://www.youtube.com/watch?v=R0fqk5OS-nw), where I explain the mechanics of forking Nextflow, enhancing, compiling, and testing changes locally, and contributing enhancements back to the main code base.
Despite building tools that use Nextflow for many years, I’ve spent relatively little time venturing into the main codebase myself. Just as with any contributor, part of the challenge was figuring out how to build Nextflow, how to navigate its code structure and how to write tests. I found it quite a fun experience, so I described and demoed the process in a recent nf-core Bytesize talk titled "[Contributing to Nextflow](https://nf-co.re/events/2024/bytesize_nextflow_dev)". You can watch the talk on [YouTube](https://www.youtube.com/watch?v=R0fqk5OS-nw), where I explain the mechanics of forking Nextflow, enhancing, compiling, and testing changes locally, and contributing enhancements back to the main code base.

## But wait, there’s more!

Expand Down

0 comments on commit e2e9b79

Please sign in to comment.