-
Notifications
You must be signed in to change notification settings - Fork 1
/
wdlconfig.qmd
70 lines (57 loc) · 9.42 KB
/
wdlconfig.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
# WDL Configuration Guide {{< iconify file-icons wdl >}} {#sec-wdlconfig}
This WILDS WDL configuration guide was inspired by the [BioWDL](https://biowdl.github.io/styleGuidelines.html) and [WARP](https://broadinstitute.github.io/warp/docs/Best_practices/suggested_formats) guidelines and is intended to cater to the pedagogical "proof-of-concept" nature of the WILDS.
## WILDS WDL Philosophy
- The mindset behind WILDS WDLs is for each repository to be a self-contained demonstration of a particular bioinformatic functionality. An ideal use-case would proceed as follows:
1. A researcher reviews the repository to deem whether it is relevant for their needs, starting with the README for the over-arching purpose of the workflow, but extending to the the input json and WDL script itself for specific questions about toolsets, settings, and input/output data types.
2. If the workflow seems useful, the researcher clones the repository locally, makes minimal updates to the input json, and executes the code with minimal effort using their favorite WDL executor.
3. If the researcher would like to add their own flavor to the workflow, they can fork the repository, customize it as necessary to fit their exact research needs, and even resubmit the changes back to the original repository for consideration and review.
- To that end, WILDS WDL repositories are relatively minimal and will usually consist of:
- a detailed [README](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/README.md) describing the intended functionality and input/output file types
- a single [WDL script](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl) containing the workflow as well as the tasks that make up the workflow
- a [input json](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram-inputs.json) template providing examples of expected inputs
- a [test case](https://github.com/getwilds/fastq-to-cram/tree/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/tests/data) to ensure the workflow is running as expected
- We believe the minimal nature of this setup will aid from a readability/ease-of-use standpoint.
## Structural Guidelines
- [Structs](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L5) should be at the top of the WDL script, followed by the [workflow](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L21) itself, followed by all of its corresponding [tasks](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L80).
- While any order is technically allowed, we recommend this arrangement to promote consistency and improve readability.
- Tasks should be broken down into as small of operations as possible.
- If a task uses more than two command line tools, it should probably be broken up into individual tasks.
- [Docker containers](https://hutchdatascience.org/WDL_Workflows_Guide/the-first-task.html#docker-images-and-containers) should be assigned to every task to ensure uniform execution, regardless of local context.
- Outside of very basic images from very trusted sources, Docker images should be pulled directly from [WILDS' Docker Library](https://github.com/getwilds/wilds-docker-library) whenever possible.
- If you think a particular tool should be added to that library, [submit an issue](https://github.com/getwilds/wilds-docker-library/issues) or email us at wilds@fredhutch.org.
- In general, [runtime attributes](https://hutchdatascience.org/WDL_Workflows_Guide/the-first-task.html#runtime-attributes) should be defined whenever possible in order to enable execution on as many [backends](https://hutchdatascience.org/WDL_Workflows_Guide/appendix-backends-and-executors.html) as possible.
- Some runtime attributes will be ignored/required based on the backend/WDL engine being used to run your workflow. Refer to the [WDL 101 guide](https://hutchdatascience.org/WDL_Workflows_Guide/appendix-backends-and-executors.html#commonly-used-runtime-attributes) for more details.
## Stylistic Guidelines
- **Indentation**: [braces contents](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L7C23-L11C2), [inputs](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L36C9-L44C31), and [line continuations](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L143C5-L147C67) should all be indented by two spaces (not four).
- **White Space**: different [input groups](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L36C1-L44C31) and [code blocks](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L95C11-L107C6) should be separated by a single blank line.
- **Line Breaks**: line breaks should only occur in the following places:
- After a [comma](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L37C49-L37C50)
- Before the `else` of an `if` statement
- [Between inputs](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L36C1-L44C31)
- [Opening](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L23C37-L23C38) and [closing](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L45C1-L46C29) braces
- **Line Character Limit**: lines should be a maximum of 100 characters.
- **Expression Spacing**: spaces should surround [operators](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L39) to increase clarity and readability.
- **Naming Conventions**:
- [Tasks](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L83C6-L83C24), [workflows](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L23C10-L23C36), and [structs](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L7C8-L7C22) should follow upper camel case (`SuperAwesomeTask`)
- Call [aliases](https://github.com/getwilds/wdl-101/blob/b1de97d360b524e1932368c04b6e2dec2c85f134/mutation_calling.wdl#L55C20-L55C31) should follow lower camel case (`superAwesomeCall`)
- [Variables](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L28C10-L28C21) should follow lowercase underscore (`super_awesome_variable`)
- **Descriptive Commenting**:
- [Comments](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L82C1-L82C41) should be placed above each task in the workflow describing its function.
- Input descriptors should be provided in the [`parameter_meta`](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/fastq-to-cram.wdl#L71C3-L77C4) component.
- **Command Syntax**:
- Command sections within a WDL task should use [Heredoc syntax](https://hutchdatascience.org/WDL_Workflows_Guide/the-first-task.html#referencing-inputs-in-the-command-section) for added clarity in terms of input variables.
- Quotation marks around string/file variables are recommended within the command section to avoid confusion involving spaces.
- While it is usually not an issue within the context of Cromwell, [file localization](https://hutchdatascience.org/WDL_Workflows_Guide/the-first-task.html#file-localization) is also recommended in order to maximize the utility of the workflow across different WDL executors.
## Repository Guidelines
- As with all repositories, each workflow should include a detailed [README](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/README.md) containing:
- Purpose and functionality of the workflow
- Basic diagram illustrating the flow of data
- Contact information in case issues pop up
- [WILDS Badge](https://github.com/getwilds/badges) at the top describing the development status of the workflow
- Make sure to include an example input json in the repository for users to modify and easily execute the workflow.
- For a skeleton template, try the `inputs` action of [WOMtool](https://cromwell.readthedocs.io/en/stable/WOMtool/#inputs).
- A GitHub Action executing [WOMtool](https://cromwell.readthedocs.io/en/stable/WOMtool/#validate) `validate` is highly recommended as a [check](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/.github/workflows/womtools-validate.yml) before merging new features into the main branch.
- If you're feeling adventurous, try automating an [entire test run](https://github.com/getwilds/fastq-to-cram/blob/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/.github/workflows/cromwell-test-run.yml) using a very small [validation dataset](https://github.com/getwilds/fastq-to-cram/tree/489ccdf0697ab902ca6f775b8e51d4f0603c0c01/tests/data).
## Additional Resources
- [Fred Hutch DaSL WDL 101 Online Course](https://hutchdatascience.org/WDL_Workflows_Guide/introduction-to-wdl.html)
- [OpenWDL Documentation](https://docs.openwdl.org/en/latest/)