Nextflow is growing up. The past week marked five years since the first commit of the project on GitHub. Like a parent reflecting on their child attending school for the first time, we know reaching this point hasn’t been an entirely solo journey, despite Paolo’s best efforts!
A lot has happened recently and we thought it was time to highlight some of the recent evolutions. We also take the opportunity to extend the warmest of thanks to all those who have contributed to the development of Nextflow as well as the fantastic community of users who consistently provide ideas, feedback and the occasional late night banter on the Gitter channel.
Here are a few neat developments churning out of the birthday cake mix.
nf-core is a community effort to provide a home for high quality, production-ready, curated analysis pipelines built using Nextflow. The project has been initiated and is being led by Phil Ewels of MultiQC fame. The principle is that nf-core pipelines can be used out-of-the-box or as inspiration for something different.
As well as being a place for best-practise pipelines, other features of nf-core include the cookie cutter template tool which provides a fast way to create a dependable workflow using many of Nextflow’s sweet capabilities such as:
There is also a Python package with helper tools for Nextflow.
You can find more information about the community via the project website, GitHub repository, Twitter account or join the dedicated Gitter chat.
As of version 0.28.0 Nextflow now has support for Kubernetes. If you don’t know much about Kubernetes, at its heart it is an open-source platform for the management and deployment of containers at scale. Google led the initial design and it is now maintained by the Cloud Native Computing Foundation. I found the The Illustrated Children’s Guide to Kubernetes particularly useful in explaining the basic vocabulary and concepts.
Kubernetes looks be one of the key technologies for the application of containers in the cloud as well as for building Infrastructure as a Service (IaaS) and Platform and a Service (PaaS) applications. We have been approached by many users who wish to use Nextflow with Kubernetes to be able to deploy workflows across both academic and commercial settings. With enterprise versions of Kubernetes such as Red Hat’s OpenShift, it was becoming apparent there was a need for native execution with Nextflow.
The new command nextflow kuberun launches the Nextflow driver as a pod which is then able to run workflow tasks as other pods within a Kubernetes cluster. You can read more in the documentation on Kubernetes support for Nextflow here.
Following the hackathon in September we wrote about the addition of HTML trace reports that allow for the generation HTML detailing resource usage (CPU time, memory, disk i/o etc).
Thanks to valuable feedback there has continued to be many improvements to the reports as tracked through the Nextflow GitHub issues page. Reports are now able to display thousands of tasks and include extra information such as the container engine used. Tasks can be filtered and an overall progress bar has been added.
You can explore a real-world HTML report and more information on HTML reports can be found in the documentation.
There has also been additions to workflow notifications. Currently these can be configured to automatically send a notification email when a workflow execution terminates. You can read more about how to setup notifications in the documentation.
Writing workflows no longer has to be done in monochrome. There is now syntax highlighting for Nextflow in the popular Atom editor as well as in Visual Studio Code.
You can find the Atom plugin by searching for Nextflow in Atoms package installer or clicking here. The Visual Studio plugin can be downloaded here.
On a related note, Nextflow is now an official language on GitHub!
Nextflow developments are progressing faster than ever and with the help of the community, there are a ton of great new features on the way. If you have any suggestions of your killer NF idea then please drop us a line, open an issue or even better, join in the fun.
Over the coming months Nextflow will be reaching out with several training and presentation sessions across the US and Europe. We hope to see as many of you as possible on the road.