Create a new component

Building a reproducible Viash component.

A Viash component can be translated into one or more engines: Native, Docker; and one or more runners: Executable, Nextflow. Each of these engines and runners result in a different artifact:

Below we will create our first Viash component using any of the languages natively supported by Viash.

Create a script

When creating a new Viash component, you can write a new script or use a pre-existing script. Below is a script that simply copies an input file to an output destination.

Create a new file named script.sh and copy the following content inside of it. This script will copy an input file to an output destination.

#!/bin/bash

## VIASH START
par_input=path/to/file.txt
par_output=output.txt
## VIASH END

# copy file
echo "Copying '$par_input' to '$par_output'."
cp -r "$par_input" "$par_output"

Create a new file named script.csx and copy the following content inside of it. This script will copy an input file to an output destination.

using System.IO;

// VIASH START
var par = new {
  input = "path/to/file.txt",
  output = "output.txt"
};
// VIASH END

// copy file
Console.WriteLine($"Copying '{par.input}' to '{par.output}'.");
File.Copy(par.input, par.output, true);

Create a new file named script.js and copy the following content inside of it. This script will copy an input file to an output destination.

const fs = require('fs');

// VIASH START
let par = {
  'input': 'path/to/file.txt',
  'output': 'output.txt'
};
// VIASH END

// copy file
console.log(`Copying '${par['input']}' to '${par['output']}'`)
fs.copyFile(par['input'], par['output'], (err) => {
  if (err) throw err;
});

Create a new file named script.py and copy the following content inside of it. This script will copy an input file to an output destination.

import shutil

## VIASH START
par = {
  'input': 'file.txt',
  'output': 'output.txt'
}
## VIASH END

# copy file
print(f"Copying '{par['input']}' to '{par['output']}'.")
shutil.copyfile(par['input'], par['output'])

Create a new file named script.R and copy the following content inside of it. This script will copy an input file to an output destination.

## VIASH START
par <- list(
  "input" = 'file.txt',
  "output" = 'output.txt'
)
## VIASH END

# copy file
cat("Copying '", par$input, "' to '", par$output, "'.\n", sep = "")
file.copy(par$input, par$output)

Create a new file named script.scala and copy the following content inside of it. This script will copy an input file to an output destination.

import java.nio.file.StandardCopyOption.REPLACE_EXISTING
import java.nio.file.Files
import java.nio.file.Paths

// VIASH START
case class ViashPar(input: String, output: String)
val par = ViashPar(
  "path/to/file.txt",
  "output.txt"
)
// VIASH END

// copy file
println(s"Copying '${par.input}' to '${par.output}'.")
val fileIn = Paths.get(par.input)
val fileOut = Paths.get(par.output)
Files.copy(fileIn, fileOut, REPLACE_EXISTING)
Note

The par variable(s) appear to be hard coded, but they’re not! When running this script with Viash, Viash will strip away the section between VIASH START and VIASH END, and replace it with parameter values at runtime. The values included in this script are thus entirely for development and debugging purposes. More information on how this works will be given in Variables and meta-variables.

Create a config

A Viash config file is a YAML file that describes the functionality of a component as well as the engine(s) and runner(s) it targets.

Create a file named config.vsh.yaml and add the contents below based on your chosen scripting language.

name: example_bash
description: A minimal example component.
arguments:
  - type: file
    name: --input
    example: file.txt
    required: true
  - type: file
    name: --output
    direction: output
    example: output.txt
    required: true
resources:
  - type: bash_script
    path: script.sh
engines:
  - type: docker
    image: "bash:4.0"
  - type: native
runners:
  - type: executable
  - type: nextflow

Create a file named config.vsh.yaml and add the contents below based on your chosen scripting language.

name: example_csharp
description: A minimal example component.
arguments:
  - type: file
    name: --input
    example: file.txt
    required: true
  - type: file
    name: --output
    direction: output
    example: output.txt
    required: true
resources:
  - type: csharp_script
    path: script.csx
engines:
  - type: docker
    image: "ghcr.io/data-intuitive/dotnet-script:1.3.1"
  - type: native
runners:
  - type: executable
  - type: nextflow

Create a file named config.vsh.yaml and add the contents below based on your chosen scripting language.

name: example_js
description: A minimal example component.
arguments:
  - type: file
    name: --input
    example: file.txt
    required: true
  - type: file
    name: --output
    direction: output
    example: output.txt
    required: true
resources:
  - type: javascript_script
    path: script.js
engines:
  - type: docker
    image: "node:19-bullseye-slim"
  - type: native
runners:
  - type: executable
  - type: nextflow

Create a file named config.vsh.yaml and add the contents below based on your chosen scripting language.

name: example_python
description: A minimal example component.
arguments:
  - type: file
    name: --input
    example: file.txt
    required: true
  - type: file
    name: --output
    direction: output
    example: output.txt
    required: true
resources:
  - type: python_script
    path: script.py
engines:
  - type: docker
    image: "python:3.10-slim"
  - type: native
runners:
  - type: executable
  - type: nextflow

Create a file named config.vsh.yaml and add the contents below based on your chosen scripting language.

name: example_r
description: A minimal example component.
arguments:
  - type: file
    name: --input
    example: file.txt
    required: true
  - type: file
    name: --output
    direction: output
    example: output.txt
    required: true
resources:
  - type: r_script
    path: script.R
engines:
  - type: docker
    image: "eddelbuettel/r2u:22.04"
  - type: native
runners:
  - type: executable
  - type: nextflow

Create a file named config.vsh.yaml and add the contents below based on your chosen scripting language.

name: example_scala
description: A minimal example component.
arguments:
  - type: file
    name: --input
    example: file.txt
    required: true
  - type: file
    name: --output
    direction: output
    example: output.txt
    required: true
resources:
  - type: scala_script
    path: script.scala
engines:
  - type: docker
    image: "sbtscala/scala-sbt:eclipse-temurin-19_36_1.7.2_2.13.10"
  - type: native
runners:
  - type: executable
  - type: nextflow

Here’s a breakdown of the different sections:

  • name: The name of the component.
  • description: A description of what the component does
  • arguments: The input and output parameters of the script.
  • resources: References to all necessary files and folders to make the component work.
  • engines: Lists which engines a component can target (i.e. Native or Docker).
  • runners: Lists which runners a component can utilize (i.e. Executable or Nextflow).

Run the component

That’s it! With these two steps, you created your first component.

Next, you can use the viash run command to test whether it actually works as intended.

You can call use the component’s --help functionality to get an overview its parameters and descriptions.

viash run config.vsh.yaml -- --help
example_bash

A minimal example component.

Arguments:
    --input
        type: file, required parameter, file must exist
        example: file.txt

    --output
        type: file, required parameter, output, file must exist
        example: output.txt

As expected, this component has an --input and --output parameter. You can execute the component by providing values for these parameters.

viash run config.vsh.yaml -- --input config.vsh.yaml --output foo.txt
[notice] Checking if Docker image is available at 'example_bash:latest'
[warning] Could not pull from 'example_bash:latest'. Docker image doesn't exist or is not accessible.
[notice] Building container 'example_bash:latest' with Dockerfile
Copying '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/bash/config.vsh.yaml' to '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/bash/foo.txt'.

You can call use the component’s --help functionality to get an overview its parameters and descriptions.

viash run config.vsh.yaml -- --help
example_csharp

A minimal example component.

Arguments:
    --input
        type: file, required parameter, file must exist
        example: file.txt

    --output
        type: file, required parameter, output, file must exist
        example: output.txt

As expected, this component has an --input and --output parameter. You can execute the component by providing values for these parameters.

viash run config.vsh.yaml -- --input config.vsh.yaml --output foo.txt
[notice] Checking if Docker image is available at 'example_csharp:latest'
[warning] Could not pull from 'example_csharp:latest'. Docker image doesn't exist or is not accessible.
[notice] Building container 'example_csharp:latest' with Dockerfile
Copying '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/csharp/config.vsh.yaml' to '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/csharp/foo.txt'.

You can call use the component’s --help functionality to get an overview its parameters and descriptions.

viash run config.vsh.yaml -- --help
example_js

A minimal example component.

Arguments:
    --input
        type: file, required parameter, file must exist
        example: file.txt

    --output
        type: file, required parameter, output, file must exist
        example: output.txt

As expected, this component has an --input and --output parameter. You can execute the component by providing values for these parameters.

viash run config.vsh.yaml -- --input config.vsh.yaml --output foo.txt
[notice] Checking if Docker image is available at 'example_js:latest'
[warning] Could not pull from 'example_js:latest'. Docker image doesn't exist or is not accessible.
[notice] Building container 'example_js:latest' with Dockerfile
Copying '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/js/config.vsh.yaml' to '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/js/foo.txt'

You can call use the component’s --help functionality to get an overview its parameters and descriptions.

viash run config.vsh.yaml -- --help
example_python

A minimal example component.

Arguments:
    --input
        type: file, required parameter, file must exist
        example: file.txt

    --output
        type: file, required parameter, output, file must exist
        example: output.txt

As expected, this component has an --input and --output parameter. You can execute the component by providing values for these parameters.

viash run config.vsh.yaml -- --input config.vsh.yaml --output foo.txt
[notice] Checking if Docker image is available at 'example_python:latest'
[warning] Could not pull from 'example_python:latest'. Docker image doesn't exist or is not accessible.
[notice] Building container 'example_python:latest' with Dockerfile
Copying '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/python/config.vsh.yaml' to '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/python/foo.txt'.

You can call use the component’s --help functionality to get an overview its parameters and descriptions.

viash run config.vsh.yaml -- --help
example_r

A minimal example component.

Arguments:
    --input
        type: file, required parameter, file must exist
        example: file.txt

    --output
        type: file, required parameter, output, file must exist
        example: output.txt

As expected, this component has an --input and --output parameter. You can execute the component by providing values for these parameters.

viash run config.vsh.yaml -- --input config.vsh.yaml --output foo.txt
[notice] Checking if Docker image is available at 'example_r:latest'
[warning] Could not pull from 'example_r:latest'. Docker image doesn't exist or is not accessible.
[notice] Building container 'example_r:latest' with Dockerfile
Copying '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/r/config.vsh.yaml' to '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/r/foo.txt'.
[1] TRUE

You can call use the component’s --help functionality to get an overview its parameters and descriptions.

viash run config.vsh.yaml -- --help
example_scala

A minimal example component.

Arguments:
    --input
        type: file, required parameter, file must exist
        example: file.txt

    --output
        type: file, required parameter, output, file must exist
        example: output.txt

As expected, this component has an --input and --output parameter. You can execute the component by providing values for these parameters.

viash run config.vsh.yaml -- --input config.vsh.yaml --output foo.txt
[notice] Checking if Docker image is available at 'example_scala:latest'
[warning] Could not pull from 'example_scala:latest'. Docker image doesn't exist or is not accessible.
[notice] Building container 'example_scala:latest' with Dockerfile
warning: 1 deprecation; re-run with -deprecation for details
Copying '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/scala/config.vsh.yaml' to '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/scala/foo.txt'.
Note

The double dash (--) between the viash command and the arguments is used to signify the end of the arguments passed to Viash and the start of those passed to the script. If you forgot to add these, you’ll get an error similar to this:

viash run config.vsh.yaml \
  --input foo.txt \
  --output bar.txt
[scallop] Error: Unknown option 'input'

Build an executable

We will now turn the Viash component into an executable.

Use the viash build command to generate an executable.

viash build config.vsh.yaml --output target

This will generate an executable in the target/ directory:

tree
.
├── config.vsh.yaml
├── foo.txt
├── script.sh
└── target
    └── example_bash

1 directory, 4 files
viash build config.vsh.yaml --output target

This will generate an executable in the target/ directory:

tree
.
├── config.vsh.yaml
├── foo.txt
├── script.csx
└── target
    └── example_csharp

1 directory, 4 files
viash build config.vsh.yaml --output target

This will generate an executable in the target/ directory:

tree
.
├── config.vsh.yaml
├── foo.txt
├── script.js
└── target
    └── example_js

1 directory, 4 files
viash build config.vsh.yaml --output target

This will generate an executable in the target/ directory:

tree
.
├── config.vsh.yaml
├── foo.txt
├── script.py
└── target
    └── example_python

1 directory, 4 files
viash build config.vsh.yaml --output target

This will generate an executable in the target/ directory:

tree
.
├── config.vsh.yaml
├── foo.txt
├── script.R
└── target
    └── example_r

1 directory, 4 files
viash build config.vsh.yaml --output target

This will generate an executable in the target/ directory:

tree
.
├── config.vsh.yaml
├── foo.txt
├── script.scala
└── target
    └── example_scala

1 directory, 4 files

Displaying the help text

It’s often useful to know what arguments an executable expects before trying to run it.

To display its documentation, run the executable with just the --help argument:

target/example_bash --help
example_bash

A minimal example component.

Arguments:
    --input
        type: file, required parameter, file must exist
        example: file.txt

    --output
        type: file, required parameter, output, file must exist
        example: output.txt

To display its documentation, run the executable with just the --help argument:

target/example_csharp --help
example_csharp

A minimal example component.

Arguments:
    --input
        type: file, required parameter, file must exist
        example: file.txt

    --output
        type: file, required parameter, output, file must exist
        example: output.txt

To display its documentation, run the executable with just the --help argument:

target/example_js --help
example_js

A minimal example component.

Arguments:
    --input
        type: file, required parameter, file must exist
        example: file.txt

    --output
        type: file, required parameter, output, file must exist
        example: output.txt

To display its documentation, run the executable with just the --help argument:

target/example_python --help
example_python

A minimal example component.

Arguments:
    --input
        type: file, required parameter, file must exist
        example: file.txt

    --output
        type: file, required parameter, output, file must exist
        example: output.txt

To display its documentation, run the executable with just the --help argument:

target/example_r --help
example_r

A minimal example component.

Arguments:
    --input
        type: file, required parameter, file must exist
        example: file.txt

    --output
        type: file, required parameter, output, file must exist
        example: output.txt

To display its documentation, run the executable with just the --help argument:

target/example_scala --help
example_scala

A minimal example component.

Arguments:
    --input
        type: file, required parameter, file must exist
        example: file.txt

    --output
        type: file, required parameter, output, file must exist
        example: output.txt

This executable takes a file as input and will create an output file.

Running the executable

Running an executable is the same as any other executable on your system.

You can run the executable by providing a value for --input and --output:

target/example_bash --input config.vsh.yaml --output output.txt
Copying '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/bash/config.vsh.yaml' to '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/bash/output.txt'.

This results in the following output:

ls -l
total 20
-rw-r--r-- 1 runner docker  411 Sep  6 22:17 config.vsh.yaml
-rw-r--r-- 1 runner docker  411 Sep  6 22:17 foo.txt
-rw-r--r-- 1 runner docker  411 Sep  6 22:19 output.txt
-rwxr-xr-x 1 runner docker  181 Sep  6 22:17 script.sh
drwxr-xr-x 2 runner docker 4096 Sep  6 22:19 target

You can run the executable by providing a value for --input and --output:

target/example_csharp --input config.vsh.yaml --output output.txt
Copying '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/csharp/config.vsh.yaml' to '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/csharp/output.txt'.

This results in the following output:

ls -l
total 20
-rw-r--r-- 1 runner docker  450 Sep  6 22:17 config.vsh.yaml
-rw-r--r-- 1 runner docker  450 Sep  6 22:17 foo.txt
-rw-r--r-- 1 runner docker  450 Sep  6 22:17 output.txt
-rw-r--r-- 1 runner docker  237 Sep  6 22:17 script.csx
drwxr-xr-x 2 runner docker 4096 Sep  6 22:19 target

You can run the executable by providing a value for --input and --output:

target/example_js --input config.vsh.yaml --output output.txt
Copying '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/js/config.vsh.yaml' to '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/js/output.txt'

This results in the following output:

ls -l
total 20
-rw-r--r-- 1 runner docker  428 Sep  6 22:17 config.vsh.yaml
-rw-r--r-- 1 runner docker  428 Sep  6 22:18 foo.txt
-rw-r--r-- 1 runner docker  428 Sep  6 22:20 output.txt
-rwxr-xr-x 1 runner docker  282 Sep  6 22:17 script.js
drwxr-xr-x 2 runner docker 4096 Sep  6 22:19 target

You can run the executable by providing a value for --input and --output:

target/example_python --input config.vsh.yaml --output output.txt
Copying '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/python/config.vsh.yaml' to '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/python/output.txt'.

This results in the following output:

ls -l
total 20
-rw-r--r-- 1 runner docker  423 Sep  6 22:17 config.vsh.yaml
-rw-r--r-- 1 runner docker  423 Sep  6 22:18 foo.txt
-rw-r--r-- 1 runner docker  423 Sep  6 22:20 output.txt
-rwxr-xr-x 1 runner docker  216 Sep  6 22:17 script.py
drwxr-xr-x 2 runner docker 4096 Sep  6 22:19 target

You can run the executable by providing a value for --input and --output:

target/example_r --input config.vsh.yaml --output output.txt
Copying '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/r/config.vsh.yaml' to '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/r/output.txt'.
[1] TRUE

This results in the following output:

ls -l
total 20
-rw-r--r-- 1 runner docker  418 Sep  6 22:17 config.vsh.yaml
-rw-r--r-- 1 runner docker  418 Sep  6 22:18 foo.txt
-rw-r--r-- 1 runner docker  418 Sep  6 22:20 output.txt
-rwxr-xr-x 1 runner docker  207 Sep  6 22:17 script.R
drwxr-xr-x 2 runner docker 4096 Sep  6 22:19 target

You can run the executable by providing a value for --input and --output:

target/example_scala --input config.vsh.yaml --output output.txt
warning: 1 deprecation; re-run with -deprecation for details
Copying '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/scala/config.vsh.yaml' to '/viash_automount/tmp/RtmpfaEtNu/create_new_component68e34a987626/scala/output.txt'.

This results in the following output:

ls -l
total 20
-rw-r--r-- 1 runner docker  462 Sep  6 22:17 config.vsh.yaml
-rw-r--r-- 1 runner docker  462 Sep  6 22:19 foo.txt
-rw-r--r-- 1 runner docker  462 Sep  6 22:20 output.txt
-rw-r--r-- 1 runner docker  435 Sep  6 22:17 script.scala
drwxr-xr-x 2 runner docker 4096 Sep  6 22:19 target