RC 3.16.0 #1395

maxulysse · 2024-09-30T17:12:35Z

Release Candidate 3.16.0 aka Fire Ferret

Merge only once #1394 is merged

PR checklist

Co-authored-by: Matthias Zepper <6963520+MatthiasZepper@users.noreply.github.com>

patch bedtools/genomecov to use quay.io to solve issue with nf-core download

github-actions · 2024-09-30T17:14:50Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 00a0857

+| ✅ 173 tests passed       |+
#| ❔   9 tests were ignored |#
!| ❗   8 tests had warnings |!

❗ Test warnings:

files_exist - File not found: assets/multiqc_config.yml
files_exist - File not found: .github/workflows/awstest.yml
files_exist - File not found: .github/workflows/awsfulltest.yml
nextflow_config - Config manifest.version should end in dev: 3.16.0
pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

files_exist - File is ignored: conf/modules.config
nextflow_config - Config default ignored: params.ribo_database_manifest
files_unchanged - File ignored due to lint config: assets/email_template.html
files_unchanged - File ignored due to lint config: assets/email_template.txt
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
actions_ci - actions_ci
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/rnaseq/rnaseq/.github/workflows/awstest.yml
multiqc_config - multiqc_config
modules_config - modules_config

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-rnaseq_logo_light.png
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-rnaseq_logo_light.png
files_exist - File found: docs/images/nf-core-rnaseq_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-rnaseq_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowRnaseq.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.hisat2_build_memory= 200.GB
nextflow_config - Config default value correct: params.gtf_extra_attributes= gene_name
nextflow_config - Config default value correct: params.gtf_group_features= gene_id
nextflow_config - Config default value correct: params.featurecounts_group_type= gene_biotype
nextflow_config - Config default value correct: params.featurecounts_feature_type= exon
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.trimmer= trimgalore
nextflow_config - Config default value correct: params.min_trimmed_reads= 10000
nextflow_config - Config default value correct: params.umitools_extract_method= string
nextflow_config - Config default value correct: params.umitools_grouping_method= directional
nextflow_config - Config default value correct: params.aligner= star_salmon
nextflow_config - Config default value correct: params.pseudo_aligner_kmer_size= 31
nextflow_config - Config default value correct: params.min_mapped_reads= 5.0
nextflow_config - Config default value correct: params.kallisto_quant_fraglen= 200
nextflow_config - Config default value correct: params.kallisto_quant_fraglen_sd= 200
nextflow_config - Config default value correct: params.stranded_threshold= 0.8
nextflow_config - Config default value correct: params.unstranded_threshold= 0.1
nextflow_config - Config default value correct: params.deseq2_vst= true
nextflow_config - Config default value correct: params.rseqc_modules= bam_stat,inner_distance,infer_experiment,junction_annotation,junction_saturation,read_distribution,read_duplication
nextflow_config - Config default value correct: params.bracken_precision= S
nextflow_config - Config default value correct: params.skip_bbsplit= true
nextflow_config - Config default value correct: params.skip_preseq= true
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/7f1614baeb0ddf66e60be78c3d9fa55440465ac8/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-rnaseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-rnaseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-rnaseq_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (590 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: cloud_tests_full.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: cloud_tests_small.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 2.14.1

Run details

nf-core/tools version 2.14.1
Run at 2024-10-02 12:25:51

Improve pipeline level snapshots

prepare RC 3.16.0

MatthiasZepper · 2024-10-02T12:04:42Z

What do we do with #1369 ?

@siddharthab made such an awesome effort, and it is essentially done - the tool is just horribly slow on transcriptomic alignments, so we are since weeks waiting that the tool's author will merge a tiny PR that fixes that. I am seriously considering forking the repo and merge it there...because I so badly want that tool in the pipeline.

jfy133

Only thing I would say I'm unhappy with is the extremely minimal description of the input file for --kraken2_db.

A kraken2 database is not the same as a bracken database (bracken requires multiple extra files alongside the normal kraken2 database files), and the Ben Langmead things I guess is being assumed is the ones most people use are not actually really the gold standard (there aren't any).

I would suggest improving the description of what gets supplied to that paramneter (a tar.gz) and also in usage explain further what is accepted and when depending on if you're supplying kraken vs bracken parameters.

But otherwise everything else is self-explanatory (thus the approval)

jfy133 · 2024-10-02T11:45:23Z

CHANGELOG.md

+| `Bracken` | ----------- | 2.9 |
+
+> **NB:** Dependency has been **updated** if both old and new version information is present.
+>


Suggested change

>

I like the line breaks here

jfy133 · 2024-10-02T11:45:28Z

CHANGELOG.md

+> **NB:** Dependency has been **updated** if both old and new version information is present.
+>
+> **NB:** Dependency has been **added** if just the new version information is present.
+>


Suggested change

>

jfy133 · 2024-10-02T11:48:02Z

nextflow_schema.json

+ },
+ "kraken_db": {
+ "type": "string",
+ "description": "Database when using Kraken2/Bracken for contaminant screening.",


In what form? Should this be a directory or a tar.gz file?

OR both? The Ben Langmead versions are tar.gz for example

@egreenberg7 any idea about that?

I'll make sure to include the answer to that when we make tests for this new feature in the next release

Should be a tar.gz file since the nf-core Kraken2 and Bracken modules use the --gzip-compressed option. It probably should be clarified that kraken2 and bracken databases are distinct, but they are often zipped together (as in the pre-built indices). A separate --bracken_db option potentially could be added.

While the pre-built files are not gold standard, they provide a quick/convenient option for people who want to run contamination detection without getting into the weeds of the tools. I'll note that I did not offer the ability to make a standard database in an indexing run of the pipeline because it is so much slower than just downloading pre-built indices (and rather computationally expensive).

In any case, @jfy133 has much more experience than me with metagenomics tools, so I would defer to any changes he recommends to either the Usage tab or the input

subworkflows/local/utils_nfcore_rnaseq_pipeline/main.nf

jfy133 · 2024-10-02T11:51:53Z

subworkflows/local/utils_nfcore_rnaseq_pipeline/main.nf

+ //Check that Kraken/Bracken parameters are not provided when Kraken2 is not being used
+ } else {
+ if (!params.bracken_precision.equals('S')) {
+ brackenPrecisionWithoutKrakenDBWarn()


Is a function here really necessary? You only use it once...

Good spot, that'll be looked after in some slight refactoring, but out of scope for this release

tests/default.nf.test

workflows/rnaseq/main.nf

maxulysse · 2024-10-02T12:12:40Z

What do we do with #1369 ?

@siddharthab made such an awesome effort, and it is essentially done - the tool is just horribly slow on transcriptomic alignments, so we are since weeks waiting that the tool's author will merge a tiny PR that fixes that. I am seriously considering forking the repo and merge it there...because I so badly want that tool in the pipeline.

This was intended to be just a patch release, Kraken2/Bracken was already in so we release as 3.16.0, but it was really meant to be 3.15.2

#1369 will be included in the next release

Co-authored-by: James A. Fellows Yates <jfy133@gmail.com>

MatthiasZepper · 2024-10-02T12:32:01Z

What do we do with #1369 ?
@siddharthab made such an awesome effort, and it is essentially done - the tool is just horribly slow on transcriptomic alignments, so we are since weeks waiting that the tool's author will merge a tiny PR that fixes that. I am seriously considering forking the repo and merge it there...because I so badly want that tool in the pipeline.

This was intended to be just a patch release, Kraken2/Bracken was already in so we release as 3.16.0, but it was really meant to be 3.15.2

#1369 will be included in the next release

Fine with me.

BTW: You are not very patriotic, are you? Why would you chose "Fire" as an element, if you could have chosen Francium and honor your home country 😉

maxulysse · 2024-10-02T12:41:27Z

What do we do with #1369 ?
@siddharthab made such an awesome effort, and it is essentially done - the tool is just horribly slow on transcriptomic alignments, so we are since weeks waiting that the tool's author will merge a tiny PR that fixes that. I am seriously considering forking the repo and merge it there...because I so badly want that tool in the pipeline.

This was intended to be just a patch release, Kraken2/Bracken was already in so we release as 3.16.0, but it was really meant to be 3.15.2
#1369 will be included in the next release

Fine with me.

BTW: You are not very patriotic, are you? Why would you chose "Fire" as an element, if you could have chosen Francium and honor your home country 😉

https://avatar.fandom.com/wiki/Fire_ferret

adamrtalbot · 2024-10-02T14:43:42Z

docs/images/nf-core-rnaseq_metro_map_grey_animated.svg

adamrtalbot · 2024-10-02T14:45:42Z

tests/.nftignore

Wow, so much ignored!

egreenberg7 and others added 30 commits July 30, 2024 16:03

Install kraken2 and bracken

3e0793f

Change nextflow configurations

a7281e5

Initial addition of kraken and bracken to pipeline

734919e

Fix include statements

8695f50

Fix include statements

44ba282

Include module config files

3657e3e

Use presence of kraken_db to determine bracken run, add save params

7ab90f6

Fix multiqc for Bracken/Kraken

39cfdef

Add Bracken/Kraken citations

2e9c243

Fix bracken config

03314c9

Update bracken

7913cbc

Adjust multiqc configs for updated bracken

063b4ac

Add input validation and update scheme

075bca8

Change default Bracken precision to species

a0a225c

Debug input validation

bd4ea1c

Documentation/output image

7789827

Update changelog

336a7b0

Update Kraken2 module

a1e1d99

Linting

33c9483

Change to --contaminant_screening param

102152e

Fixing save unaligned default

6dd7521

Debugging

fdd85ad

Update usage

c0099d4

Provide motivation for Kraken2 parameters

c25aa50

Fix typo

0063c2d

Update metro map

53f46b2

Merge branch 'dev' into dev

2dc5746

Update schema

2a322c4

Update docs/usage.md

75a10f7

Co-authored-by: Matthias Zepper <6963520+MatthiasZepper@users.noreply.github.com>

Change output directory for kraken2/bracken

cfc8945

maxulysse added 4 commits September 30, 2024 14:50

use url for singularity

5c61ed3

use proper url for singularity

894a770

Merge pull request #1393 from maxulysse/patch_registry

fcbfa1b

patch bedtools/genomecov to use quay.io to solve issue with nf-core download

prepare RC 3.15.2

30483c7

maxulysse added 9 commits September 30, 2024 19:22

update version in snapshots too

1424156

use nft-utils

5d79bdc

update nftignore

07ba1b9

Merge branch 'dev' into better_snapshots

00a87fc

update pipeline versions in snapshot

a83ce56

update CHANGELOG

b80c6c5

update nftignore

7d14c74

update nftignore

2b856b7

3.16.0 instead of 3.15.2

1d50d31

maxulysse changed the title ~~RC 3.15.2~~ RC 3.16.0 Oct 1, 2024

maxulysse and others added 7 commits October 1, 2024 15:56

Update CHANGELOG.md

d19cea4

use nft-utils 0.0.2 to get relative path in snapshots

1d35e2a

Merge pull request #1379 from maxulysse/better_snapshots

07197c5

Improve pipeline level snapshots

Merge branch 'dev' into prepare_3.15.2

a5d636a

forgot a merge

73c8101

Merge pull request #1394 from maxulysse/prepare_3.15.2

2867938

prepare RC 3.16.0

Merge branch 'master' into dev

f901e4f

jfy133 approved these changes Oct 2, 2024

View reviewed changes

Apply suggestions from code review

00a0857

Co-authored-by: James A. Fellows Yates <jfy133@gmail.com>

adamrtalbot approved these changes Oct 2, 2024

View reviewed changes

maxulysse merged commit 33df0c0 into master Oct 2, 2024
48 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RC 3.16.0 #1395

RC 3.16.0 #1395

maxulysse commented Sep 30, 2024 •

edited

Loading

github-actions bot commented Sep 30, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

MatthiasZepper commented Oct 2, 2024

jfy133 left a comment •

edited

Loading

jfy133 Oct 2, 2024

maxulysse Oct 2, 2024

jfy133 Oct 2, 2024

jfy133 Oct 2, 2024

jfy133 Oct 2, 2024

maxulysse Oct 2, 2024

maxulysse Oct 2, 2024

egreenberg7 Oct 2, 2024

jfy133 Oct 2, 2024

maxulysse Oct 2, 2024

maxulysse commented Oct 2, 2024

MatthiasZepper commented Oct 2, 2024

maxulysse commented Oct 2, 2024

adamrtalbot Oct 2, 2024

adamrtalbot Oct 2, 2024

RC 3.16.0 #1395

RC 3.16.0 #1395

Conversation

maxulysse commented Sep 30, 2024 • edited Loading

PR checklist

github-actions bot commented Sep 30, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

MatthiasZepper commented Oct 2, 2024

jfy133 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxulysse commented Oct 2, 2024

MatthiasZepper commented Oct 2, 2024

maxulysse commented Oct 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxulysse commented Sep 30, 2024 •

edited

Loading

github-actions bot commented Sep 30, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️

jfy133 left a comment •

edited

Loading