From d847ebb29a5da19e928b8701336b9966a92aa255 Mon Sep 17 00:00:00 2001 From: "Documenter.jl" Date: Tue, 12 Mar 2024 19:45:12 -0400 Subject: [PATCH] build based on 077bcdd --- dev/.documenter-siteinfo.json | 2 +- dev/getting-started/index.html | 4 +-- dev/guides/authentication/index.html | 2 +- dev/guides/datasets/index.html | 2 +- dev/guides/jobs/index.html | 4 +-- dev/index.html | 2 +- dev/internal/index.html | 4 +-- dev/objects.inv | Bin 1576 -> 1576 bytes dev/reference/authentication/index.html | 8 ++--- dev/reference/datasets/index.html | 14 ++++---- dev/reference/exceptions/index.html | 4 +-- dev/reference/job-submission/index.html | 42 ++++++++++++------------ dev/reference/jobs/index.html | 16 ++++----- dev/search_index.js | 2 +- 14 files changed, 53 insertions(+), 53 deletions(-) diff --git a/dev/.documenter-siteinfo.json b/dev/.documenter-siteinfo.json index fbfbd171d..bf432add8 100644 --- a/dev/.documenter-siteinfo.json +++ b/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.10.2","generation_timestamp":"2024-03-11T00:13:06","documenter_version":"1.3.0"}} \ No newline at end of file +{"documenter":{"julia_version":"1.10.2","generation_timestamp":"2024-03-12T23:45:07","documenter_version":"1.3.0"}} \ No newline at end of file diff --git a/dev/getting-started/index.html b/dev/getting-started/index.html index ada607915..d3f40740e 100644 --- a/dev/getting-started/index.html +++ b/dev/getting-started/index.html @@ -48,7 +48,7 @@ ENV["RESULTS"] = string(mat_sum) """ sha256(project_toml) = 93a83d60d4a9c6a3d1438259fd506929eaad296b7e112e886b305781b85cb85b -sha256(manifest_toml) = cb7bf4658b8d6daea993e9d71acd44b9e172a044b99ae83fa04c7ac07ee2f3aa
Job environment

In most cases, you also submit a Julia package environment (i.e. Project.toml and Manifest.toml files together with a job). That environment then gets instantiated before the user-provided code is run.

The script"" string macro, by default, attaches the currently active environment to the job. This means that any packages that you are currently using should also be available on the job (although only registered packages added as non-development dependencies will work). You can use Base.active_project() or pkg> status to see what environment is currently active.

To submit a job, you can simply call submit_job on it.

julia> j = JuliaHub.submit_job(s)JuliaHub.Job: jr-xf4tslavut (Submitted)
+sha256(manifest_toml) = 7f6a3052cad3e8097b148270b3350caaedc38034c123866b503450e77744966b
Job environment

In most cases, you also submit a Julia package environment (i.e. Project.toml and Manifest.toml files together with a job). That environment then gets instantiated before the user-provided code is run.

The script"" string macro, by default, attaches the currently active environment to the job. This means that any packages that you are currently using should also be available on the job (although only registered packages added as non-development dependencies will work). You can use Base.active_project() or pkg> status to see what environment is currently active.

To submit a job, you can simply call submit_job on it.

julia> j = JuliaHub.submit_job(s)JuliaHub.Job: jr-xf4tslavut (Submitted)
  submitted: 2023-03-15T07:56:50.974+00:00
  started:   2023-03-15T07:56:51.251+00:00
  finished:  2023-03-15T07:56:59.000+00:00

The submit_job function also allows you to specify configure how the job gets run, such as how many CPUs or how much memory it has available. By default, though, it runs your code on a single node, picking the smallest instance that is available.

At this point, if you go to the "Jobs" page web UI, you should see the job there. It may take a few moments to actually start running. You can also call job on the returned Job object to refresh the status of the job.

julia> j = JuliaHub.job(j)JuliaHub.Job: jr-xf4tslavut (Running)
@@ -63,4 +63,4 @@
     startOnLoad: true,
     theme: "neutral"
 });
-
+
diff --git a/dev/guides/authentication/index.html b/dev/guides/authentication/index.html
index ef648c479..b5723d103 100644
--- a/dev/guides/authentication/index.html
+++ b/dev/guides/authentication/index.html
@@ -8,4 +8,4 @@
     startOnLoad: true,
     theme: "neutral"
 });
-
+
diff --git a/dev/guides/datasets/index.html b/dev/guides/datasets/index.html
index c5abd89dc..b7c0b558f 100644
--- a/dev/guides/datasets/index.html
+++ b/dev/guides/datasets/index.html
@@ -107,4 +107,4 @@
     startOnLoad: true,
     theme: "neutral"
 });
-
+
diff --git a/dev/guides/jobs/index.html b/dev/guides/jobs/index.html
index 974905803..1ba6b2006 100644
--- a/dev/guides/jobs/index.html
+++ b/dev/guides/jobs/index.html
@@ -54,7 +54,7 @@
   println("hello world")
   """
   sha256(project_toml) = 93a83d60d4a9c6a3d1438259fd506929eaad296b7e112e886b305781b85cb85b
-  sha256(manifest_toml) = cb7bf4658b8d6daea993e9d71acd44b9e172a044b99ae83fa04c7ac07ee2f3aa
+  sha256(manifest_toml) = 7f6a3052cad3e8097b148270b3350caaedc38034c123866b503450e77744966b
 compute:
   JuliaHub.ComputeConfig
    Node: 3.5 GHz Intel Xeon Platinum 8375C
@@ -117,4 +117,4 @@
     startOnLoad: true,
     theme: "neutral"
 });
-
  • 1You can also e.g. upload datasets etc. But in that case the resulting data is not, strictly speaking, related to a specific job.

+
  • 1You can also e.g. upload datasets etc. But in that case the resulting data is not, strictly speaking, related to a specific job.

diff --git a/dev/index.html b/dev/index.html
index b63891743..038674c03 100644
--- a/dev/index.html
+++ b/dev/index.html
@@ -4,4 +4,4 @@
     startOnLoad: true,
     theme: "neutral"
 });
-
+
diff --git a/dev/internal/index.html b/dev/internal/index.html
index 316bc36f0..97c278d24 100644
--- a/dev/internal/index.html
+++ b/dev/internal/index.html
@@ -1,7 +1,7 @@
 
-Internal · JuliaHub.jl
GitHub
Internal
_PackageBundler
JuliaHub._PackageBundler.bundle — Function
bundle(dir; output = "",  force=false, allownoenv=false, verbose = true) -> String
Creates a .tar file with the contents of dir as well as any packages that are either tracked by path (developed) outside dir or packages that are not tracked by the PkgServer. Artifacts are also bundled. The bundled packages and artifacts go into a .bundled/depot directory and is set up like a depot and can thus be made available by adding it to DEPOT_PATH.
.git and globs listed in .juliabundleignore are excluded form the bundle.
Returns the hex-encoded SHA256 of the Manifest.toml file that is packed into the appbundle. The return value is used when requesting a sysimage build, in which case we have to pass the manifest's hash with the submit request.
source
JuliaHub._PackageBundler.path_filterer — Function
path_filterer(top)
Returns a function that takes a file or directory path and checks whether that is excluded by the nearest .juliabundleignore file. The function will also ignore any .git files and directories.
The top argument specifies the highest directory up the tree that will be searched for the .juliabundleignore file.
The function will return false for any excluded files and true otherwise, and can be used as a predicate for filtering files that should be bundled.
source
Index
GitHub
Internal
_PackageBundler
JuliaHub._PackageBundler.bundle — Function
bundle(dir; output = "",  force=false, allownoenv=false, verbose = true) -> String
Creates a .tar file with the contents of dir as well as any packages that are either tracked by path (developed) outside dir or packages that are not tracked by the PkgServer. Artifacts are also bundled. The bundled packages and artifacts go into a .bundled/depot directory and is set up like a depot and can thus be made available by adding it to DEPOT_PATH.
.git and globs listed in .juliabundleignore are excluded form the bundle.
Returns the hex-encoded SHA256 of the Manifest.toml file that is packed into the appbundle. The return value is used when requesting a sysimage build, in which case we have to pass the manifest's hash with the submit request.
source
JuliaHub._PackageBundler.path_filterer — Function
path_filterer(top)
Returns a function that takes a file or directory path and checks whether that is excluded by the nearest .juliabundleignore file. The function will also ignore any .git files and directories.
The top argument specifies the highest directory up the tree that will be searched for the .juliabundleignore file.
The function will return false for any excluded files and true otherwise, and can be used as a predicate for filtering files that should be bundled.
source
Index

+

diff --git a/dev/objects.inv b/dev/objects.inv
index 10b3bcf5bbaaa86d9cce720c4ec6ca8f69190452..3168223930da0f05ffc275575de324120b6ca5aa 100644
GIT binary patch
delta 12
TcmZ3%vw~-WGo$53mnW
-Authentication · JuliaHub.jl
GitHub
Authentication
In order to talk to a JuliaHub instance, you need to have a valid authentication token. JuliaHub reuses the Julia's built-in package server authentication tokens for this purpose. By default, the authentication uses the JULIA_PKG_SERVER environment variable to determine which JuliaHub instance to connect to, but this can be overridden by passing an argument to authenticate.
The authenticate function can be used to construct a token. If a valid token is available in ~/.julia/servers, it gets reused. Otherwise, a browser window is opened, starting an interactive authentication procedure.
All the functions that require authentication accept an auth keyword argument. However, JuliaHub.jl also stores the authentication token from the last authenticate call in a global variable and automatically uses that if auth is not provided, and also tries to authenticate automatically. The current global authentication object can be accessed via the current_authentication() function.
See also: authentication guide, authentication section on help.juliahub.com, PkgAuthentication.
Token expiration and refresh tokens
By default, JuliaHub access tokens expire in 24 hours. However, the tokens usually also have a refresh token, which is valid for 30 days. If the access token has expired, but there is a valid refresh token available, authenticate will automatically try to use that, to re-acquire an access token without starting an interactive authentication.
In JuliaHub job and cloud IDE environments, the authentication token on disk will be continuously kept up to date. The reauthenticate! function can be used to reload the token from disk.
Reference
JuliaHub.authenticate — Function
JuliaHub.authenticate(server = Pkg.pkg_server(); force::Bool = false, maxcount::Integer = 3, [hook::Base.Callable])
Authenticates with a JuliaHub server. If a valid authentication token does not exist in the Julia depot, a new token is acquired via an interactive browser based prompt. Returns an Authentication object if the authentication was successful, or throws an AuthenticationError if authentication fails.
The interactive prompts tries to authenticate for a maximum of maxcount times. If force is set to true, an existing authentication token is first deleted. This can be useful when the existing authentication token is causing the authentication to fail.
Extended help
By default, it attemps to connect to the currently configured Julia package server URL (configured e.g. via the JULIA_PKG_SERVER environment variable). However, this can be overridden by passing the server argument.
hook can be set to a function taking a single string-type argument, and will be passed the authorization URL the user should interact with in the browser. This can be used to override the default behavior coming from PkgAuthentication.
The returned Authentication object is also cached globally (overwriting any previously cached authentications), making it unnecessary to pass the returned object manually to other function calls. This is useful for interactive use, but should not be used in library code, as different authentication calls may clash.
source
JuliaHub.Authentication — Type
mutable struct Authentication
Authentication object constructed by the authenticate function that can be passed to the various JuliaHub.jl function via the auth keyword argument.
Objects have the following properties:
  • server :: URIs.URI: URL of the JuliaHub instance this authentication token applies to.
  • username :: String: user's JuliaHub username (used for e.g. to namespace datasets)
  • token :: JuliaHub.Secret: a Secret object storing the JuliaHub authentication token
Note that the object is mutable, and hence will be shared as it is passed around. And at the same time, functions such as reauthenticate! may modify the object.
See also: authenticate, reauthenticate!, current_authentication.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.current_authentication — Function
JuliaHub.current_authentication() -> Union{Authentication, Nothing}
Returns the current globally active Authentication object, or nothing if authenticate has not yet been called.
julia> JuliaHub.current_authentication()
-JuliaHub.Authentication("https://juliahub.com", "username", *****)
Note
Calling this function will not initialize authentication.
source
JuliaHub.check_authentication — Function
JuliaHub.check_authentication(; [auth::Authentication]) -> Bool
Checks if the authentication to a JuliaHub instance is still valid or not.
This can be used to periodically check an authentication token, to see if it is necessary to re-authenticate.
See also: reauthenticate!.
source
JuliaHub.reauthenticate! — Function
JuliaHub.reauthenticate!([auth::Authentication]; force::Bool = false, maxcount::Integer = 3, [hook::Base.Callable])
Attempts to update the authentication token in auth:
  • If the original auth.toml file has been updated, it simply reloads the token from the file.
  • If loading from auth.toml fails or force=true, it will attempt to re-authenticate with the server, possibly interactively.
If auth is omitted, it will reauthenticate the global Authentication object. The force, maxcount and hook are relevant for interactive authentication, and behave the same way as in the authenticate function.
This is mostly meant to be used to re-acquire authentication tokens in long-running sessions, where the initial authentication token may have expired.
As Authentication objects are mutable, the token will be updated in all contexts where the reference to the Authentication has been passed to.
See also: authenticate, current_authentication, Authentication, check_authentication.
source
JuliaHub.Secret — Type
mutable struct Secret
A helper type for storing secrets. Internally it is a covenience wrapper around Base.SecretBuffer. Predominantly used in Authentication objects to store the JuliaHub authentication token.
The String(::Secret) function can be used to obtain an unsecure string copy of the secret stored in the object.
julia> s = JuliaHub.Secret("secret-string")
+Authentication · JuliaHub.jl
GitHub
Authentication
In order to talk to a JuliaHub instance, you need to have a valid authentication token. JuliaHub reuses the Julia's built-in package server authentication tokens for this purpose. By default, the authentication uses the JULIA_PKG_SERVER environment variable to determine which JuliaHub instance to connect to, but this can be overridden by passing an argument to authenticate.
The authenticate function can be used to construct a token. If a valid token is available in ~/.julia/servers, it gets reused. Otherwise, a browser window is opened, starting an interactive authentication procedure.
All the functions that require authentication accept an auth keyword argument. However, JuliaHub.jl also stores the authentication token from the last authenticate call in a global variable and automatically uses that if auth is not provided, and also tries to authenticate automatically. The current global authentication object can be accessed via the current_authentication() function.
See also: authentication guide, authentication section on help.juliahub.com, PkgAuthentication.
Token expiration and refresh tokens
By default, JuliaHub access tokens expire in 24 hours. However, the tokens usually also have a refresh token, which is valid for 30 days. If the access token has expired, but there is a valid refresh token available, authenticate will automatically try to use that, to re-acquire an access token without starting an interactive authentication.
In JuliaHub job and cloud IDE environments, the authentication token on disk will be continuously kept up to date. The reauthenticate! function can be used to reload the token from disk.
Reference
JuliaHub.authenticate — Function
JuliaHub.authenticate(server = Pkg.pkg_server(); force::Bool = false, maxcount::Integer = 3, [hook::Base.Callable])
Authenticates with a JuliaHub server. If a valid authentication token does not exist in the Julia depot, a new token is acquired via an interactive browser based prompt. Returns an Authentication object if the authentication was successful, or throws an AuthenticationError if authentication fails.
The interactive prompts tries to authenticate for a maximum of maxcount times. If force is set to true, an existing authentication token is first deleted. This can be useful when the existing authentication token is causing the authentication to fail.
Extended help
By default, it attemps to connect to the currently configured Julia package server URL (configured e.g. via the JULIA_PKG_SERVER environment variable). However, this can be overridden by passing the server argument.
hook can be set to a function taking a single string-type argument, and will be passed the authorization URL the user should interact with in the browser. This can be used to override the default behavior coming from PkgAuthentication.
The returned Authentication object is also cached globally (overwriting any previously cached authentications), making it unnecessary to pass the returned object manually to other function calls. This is useful for interactive use, but should not be used in library code, as different authentication calls may clash.
source
JuliaHub.Authentication — Type
mutable struct Authentication
Authentication object constructed by the authenticate function that can be passed to the various JuliaHub.jl function via the auth keyword argument.
Objects have the following properties:
  • server :: URIs.URI: URL of the JuliaHub instance this authentication token applies to.
  • username :: String: user's JuliaHub username (used for e.g. to namespace datasets)
  • token :: JuliaHub.Secret: a Secret object storing the JuliaHub authentication token
Note that the object is mutable, and hence will be shared as it is passed around. And at the same time, functions such as reauthenticate! may modify the object.
See also: authenticate, reauthenticate!, current_authentication.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.current_authentication — Function
JuliaHub.current_authentication() -> Union{Authentication, Nothing}
Returns the current globally active Authentication object, or nothing if authenticate has not yet been called.
julia> JuliaHub.current_authentication()
+JuliaHub.Authentication("https://juliahub.com", "username", *****)
Note
Calling this function will not initialize authentication.
source
JuliaHub.check_authentication — Function
JuliaHub.check_authentication(; [auth::Authentication]) -> Bool
Checks if the authentication to a JuliaHub instance is still valid or not.
This can be used to periodically check an authentication token, to see if it is necessary to re-authenticate.
See also: reauthenticate!.
source
JuliaHub.reauthenticate! — Function
JuliaHub.reauthenticate!([auth::Authentication]; force::Bool = false, maxcount::Integer = 3, [hook::Base.Callable])
Attempts to update the authentication token in auth:
  • If the original auth.toml file has been updated, it simply reloads the token from the file.
  • If loading from auth.toml fails or force=true, it will attempt to re-authenticate with the server, possibly interactively.
If auth is omitted, it will reauthenticate the global Authentication object. The force, maxcount and hook are relevant for interactive authentication, and behave the same way as in the authenticate function.
This is mostly meant to be used to re-acquire authentication tokens in long-running sessions, where the initial authentication token may have expired.
As Authentication objects are mutable, the token will be updated in all contexts where the reference to the Authentication has been passed to.
See also: authenticate, current_authentication, Authentication, check_authentication.
source
JuliaHub.Secret — Type
mutable struct Secret
A helper type for storing secrets. Internally it is a covenience wrapper around Base.SecretBuffer. Predominantly used in Authentication objects to store the JuliaHub authentication token.
The String(::Secret) function can be used to obtain an unsecure string copy of the secret stored in the object.
julia> s = JuliaHub.Secret("secret-string")
 JuliaHub.Secret("*******")
 
 julia> String(s)
 "secret-string"
Constructors
Secret(::AbstractString)
-Secret(::Vector{UInt8})
Create a Secret object from the input strings.
source
Index

+

diff --git a/dev/reference/datasets/index.html b/dev/reference/datasets/index.html
index a36c63f8a..47dbed96b 100644
--- a/dev/reference/datasets/index.html
+++ b/dev/reference/datasets/index.html
@@ -1,5 +1,5 @@
 
-Datasets · JuliaHub.jl
GitHub
Datasets
These APIs allow you to create, read, update, and delete datasets owned by the currently authenticated user.
See also: help.julialang.org on datasets, DataSets.jl.
Dataset types
JuliaHub currently has two distinct types of datasets:
  1. Blob: a single file; or, more abstractly, a collection of bytes
  2. BlobTree: a directory or a file; more abstractly a tree-like collection of Blobs, indexed by file system paths
These types mirror the concepts in DataSets.jl
JuliaHub.jl APIs do not rely that much on the dataset type for anything, except when downloading or uploading. In that case, a local file always corresponds to a Blob, and a local directory corresponds to a BlobTree. For example, when trying to upload a file as a new version of a BlobTree-type dataset will fail, because the dataset type can not change.
The upload_dataset function uses information filesystem to determine whether the created dataset is a Blob or a BlobTree, and similarly download_dataset will always download a Blob into a file, and a BlobTree as a directory.
Dataset versions
A JuliaHub dataset can have zero or more versions. A newly created dataset usually has at least one version, but it may have zero versions if, for example, the upload did not finish. The versions are indexed with a linear list of integers starting from 1.
Reference
JuliaHub.Dataset — Type
struct Dataset
Information about a dataset stored on JuliaHub, and the following fields are considered to be public API:
  • uuid :: UUID: dataset UUID
  • owner :: String: username of the dataset owner
  • name :: String: dataset name
  • dtype :: String: generally either Blob or BlobTree, but additional values may be added in the future
  • versions :: Vector{DatasetVersion}: an ordered list of DatasetVersion objects, one for each dataset version, sorted from oldest to latest (i.e. you can use last to get the newest version).
  • size :: Int: total size of the whole dataset (including all the dataset versions) in bytes
  • Fields to access user-provided dataset metadata:
    • description :: String: dataset description
    • tags :: Vector{String}: a list of tags
Canonical fully qualified dataset name
In some contexts, like when accessing JuliaHub datasets with DataSets.jl, the .owner-.name tuple constitutes the fully qualifed dataset name, uniquely identifying a dataset on a JuliaHub instance. I.e. for a dataset object dataset, it can be constructed as "$(dataset.owner)/$(dataset.name)".
Non-dynamic dataset objects
Dataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.DatasetVersion — Type
struct DatasetVersion
Represents one version of a dataset.
Objects have the following properties:
  • .id: unique dataset version identifier (used e.g. in download_dataset to identify the dataset version).
  • .size :: Int: size of the dataset version in bytes
  • .timestamp :: ZonedDateTime: dataset version timestamp
julia> JuliaHub.datasets()
See also: Dataset, datasets, dataset.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.datasets — Function
JuliaHub.datasets([username::AbstractString]; shared::Bool=false, [auth::Authentication]) -> Vector{Dataset}
List all datasets owned by username, returning a list of Dataset objects.
If username is omitted, it returns the datasets owned by the currently authenticated user. If username is different from the currently authenticated user, it only returns the datasets that are readable to (i.e. somehow shared with) the currently authenticated user.
If shared = true, it also returns datasets that belong to other users that have that have been shared with the currently authenticated user. In this case, username is effectively ignored.
julia> JuliaHub.datasets()
+Datasets · JuliaHub.jl
GitHub
Datasets
These APIs allow you to create, read, update, and delete datasets owned by the currently authenticated user.
See also: help.julialang.org on datasets, DataSets.jl.
Dataset types
JuliaHub currently has two distinct types of datasets:
  1. Blob: a single file; or, more abstractly, a collection of bytes
  2. BlobTree: a directory or a file; more abstractly a tree-like collection of Blobs, indexed by file system paths
These types mirror the concepts in DataSets.jl
JuliaHub.jl APIs do not rely that much on the dataset type for anything, except when downloading or uploading. In that case, a local file always corresponds to a Blob, and a local directory corresponds to a BlobTree. For example, when trying to upload a file as a new version of a BlobTree-type dataset will fail, because the dataset type can not change.
The upload_dataset function uses information filesystem to determine whether the created dataset is a Blob or a BlobTree, and similarly download_dataset will always download a Blob into a file, and a BlobTree as a directory.
Dataset versions
A JuliaHub dataset can have zero or more versions. A newly created dataset usually has at least one version, but it may have zero versions if, for example, the upload did not finish. The versions are indexed with a linear list of integers starting from 1.
Reference
JuliaHub.Dataset — Type
struct Dataset
Information about a dataset stored on JuliaHub, and the following fields are considered to be public API:
  • uuid :: UUID: dataset UUID
  • owner :: String: username of the dataset owner
  • name :: String: dataset name
  • dtype :: String: generally either Blob or BlobTree, but additional values may be added in the future
  • versions :: Vector{DatasetVersion}: an ordered list of DatasetVersion objects, one for each dataset version, sorted from oldest to latest (i.e. you can use last to get the newest version).
  • size :: Int: total size of the whole dataset (including all the dataset versions) in bytes
  • Fields to access user-provided dataset metadata:
    • description :: String: dataset description
    • tags :: Vector{String}: a list of tags
Canonical fully qualified dataset name
In some contexts, like when accessing JuliaHub datasets with DataSets.jl, the .owner-.name tuple constitutes the fully qualifed dataset name, uniquely identifying a dataset on a JuliaHub instance. I.e. for a dataset object dataset, it can be constructed as "$(dataset.owner)/$(dataset.name)".
Non-dynamic dataset objects
Dataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.DatasetVersion — Type
struct DatasetVersion
Represents one version of a dataset.
Objects have the following properties:
  • .id: unique dataset version identifier (used e.g. in download_dataset to identify the dataset version).
  • .size :: Int: size of the dataset version in bytes
  • .timestamp :: ZonedDateTime: dataset version timestamp
julia> JuliaHub.datasets()
See also: Dataset, datasets, dataset.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.datasets — Function
JuliaHub.datasets([username::AbstractString]; shared::Bool=false, [auth::Authentication]) -> Vector{Dataset}
List all datasets owned by username, returning a list of Dataset objects.
If username is omitted, it returns the datasets owned by the currently authenticated user. If username is different from the currently authenticated user, it only returns the datasets that are readable to (i.e. somehow shared with) the currently authenticated user.
If shared = true, it also returns datasets that belong to other users that have that have been shared with the currently authenticated user. In this case, username is effectively ignored.
julia> JuliaHub.datasets()
 2-element Vector{JuliaHub.Dataset}:
  JuliaHub.dataset(("username", "example-dataset"))
  JuliaHub.dataset(("username", "blobtree/example"))
@@ -12,7 +12,7 @@
 
 julia> JuliaHub.datasets("anotheruser")
 1-element Vector{JuliaHub.Dataset}:
- JuliaHub.dataset(("anotheruser", "publicdataset"))
Non-dynamic dataset objects
Dataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.
source
JuliaHub.DatasetReference — Type
const DatasetReference :: Type
Type constraint on the first argument of most of the datasets-related functions, that is used to uniquely specify the dataset that the operation will affect.
There are three different objects that can be passed as a dataset reference (dsref::DatasetReference):
  • (owner::AbstractString, dataset_name::AbstractString)::Tuple{AbstractString,AbstractString}
    A tuple of the owner's username and the dataset's name.
  • dataset_name::AbstractString
    Just a string with the dataset name; in this case the dataset's owner will be assumed to be the currently authenticated user (with the username determined from the Authentication objects passed via the auth keyword).
  • dataset::Dataset
    Uses the owner and dataset name information from a Dataset object.
No UUID mismatch checks
When using the third option (i.e. passing a Dataset), the dataset UUID will not be checked. So if the dataset with the same owner and username has been deleted and re-created as a new dataset (potentially of a different dtype etc), the functions will then act on the new dataset.
source
JuliaHub.dataset — Function
JuliaHub.dataset(dataset::DatasetReference; throw::Bool=true, [auth::Authentication]) -> Dataset
Looks up a dataset based on the dataset reference dataset. Returns the Dataset object corresponding to dataset_name, or throws a InvalidRequestError if the dataset can not be found (if throw=false is passed, returns nothing instead).
By passing a Dataset object as dataset, this can be used to update the Dataset object.
julia> dataset = JuliaHub.dataset("example-dataset")
+ JuliaHub.dataset(("anotheruser", "publicdataset"))
Non-dynamic dataset objects
Dataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.
source
JuliaHub.DatasetReference — Type
const DatasetReference :: Type
Type constraint on the first argument of most of the datasets-related functions, that is used to uniquely specify the dataset that the operation will affect.
There are three different objects that can be passed as a dataset reference (dsref::DatasetReference):
  • (owner::AbstractString, dataset_name::AbstractString)::Tuple{AbstractString,AbstractString}
    A tuple of the owner's username and the dataset's name.
  • dataset_name::AbstractString
    Just a string with the dataset name; in this case the dataset's owner will be assumed to be the currently authenticated user (with the username determined from the Authentication objects passed via the auth keyword).
  • dataset::Dataset
    Uses the owner and dataset name information from a Dataset object.
No UUID mismatch checks
When using the third option (i.e. passing a Dataset), the dataset UUID will not be checked. So if the dataset with the same owner and username has been deleted and re-created as a new dataset (potentially of a different dtype etc), the functions will then act on the new dataset.
source
JuliaHub.dataset — Function
JuliaHub.dataset(dataset::DatasetReference; throw::Bool=true, [auth::Authentication]) -> Dataset
Looks up a dataset based on the dataset reference dataset. Returns the Dataset object corresponding to dataset_name, or throws a InvalidRequestError if the dataset can not be found (if throw=false is passed, returns nothing instead).
By passing a Dataset object as dataset, this can be used to update the Dataset object.
julia> dataset = JuliaHub.dataset("example-dataset")
 Dataset: example-dataset (Blob)
  owner: username
  description: An example dataset
@@ -26,12 +26,12 @@
  description: An example dataset
  versions: 2
  size: 388 bytes
- tags: tag1, tag2
If the specifed username is not the currently authenticated user, the dataset must be shared with the currently authenticated user (i.e. contained in datasets(; shared=true)).
Note
This will call datasets every time, which might become a problem if you are processing a large number of datasets. In that case, you should call datasets and process the returned list yourself.
Non-dynamic dataset objects
Dataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.
source
JuliaHub.download_dataset — Function
download_dataset(
+ tags: tag1, tag2
If the specifed username is not the currently authenticated user, the dataset must be shared with the currently authenticated user (i.e. contained in datasets(; shared=true)).
Note
This will call datasets every time, which might become a problem if you are processing a large number of datasets. In that case, you should call datasets and process the returned list yourself.
Non-dynamic dataset objects
Dataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.
source
JuliaHub.download_dataset — Function
download_dataset(
     dataset::DatasetReference, local_path::AbstractString;
     replace::Bool = false, [version::Integer],
     [quiet::Bool = false], [auth::Authentication]
-) -> String
Downloads the dataset specified by the dataset reference dataset to local_path (which must not exist, unless replace = true), returning the absolute path to the downloaded file or directory. If the dataset is a Blob, then the created local_path will be a file, and if the dataset is a BlobTree the local_path will be a directory.
By default, it downloads the latest version, but an older version can be downloaded by specifying the version keyword argument. Caution: you should never assume that the index of the .versions property of Dataset matches the version number – always explicitly use the .id propert of the DatasetVersion object.
The function also prints download progress to standard output. This can be disabled by setting quiet=true. Any error output from the download is still printed.
Warning
Setting replace = true will recursively erase any existing data at local_path before replacing it with the dataset contents.
source
JuliaHub.upload_dataset — Function
JuliaHub.upload_dataset(dataset::DatasetReference, local_path; [auth,] kwargs...) -> Dataset
Uploads a new dataset or a new version of an existing dataset, with the dataset specified by the dataset reference dataset. The dataset type is determined from the local path (Blob if a file, BlobTree if a directory). If a Dataset object is passed, it attempts to update that dataset. Returns an updated Dataset object.
The following keyword arguments can be used to control the exact behavior of the function:
  • create :: Bool (default: true): Create the dataset, if it already does not exist.
  • update :: Bool (default: false): Upload the data as a new dataset version, if the dataset exists.
  • replace :: Bool (default: false): If a dataset exists, delete all existing data and create a new dataset with the same name instead. Excludes update = true, and only creates a completely new dataset if create=true as well.
In addition, the following keyword arguments can be passed to set or updated the dataset metadata when uploading:
  • description: description of the dataset (a string)
  • tags: an iterable of strings of all the tags of the dataset
  • visibility: a string with possible values public or private
  • license: a valid SPDX license identifier, or a tuple (:fulltext, license_text), where license_text is the full text string of a custom license
  • groups: an iterable of valid group names
If a dataset already exists, then these fields are updated as if update_dataset was called.
The function will throw an ArgumentError for invalid argument combinations.
Use the progress keyword argument to suppress upload progress from being printed.
Note
Presently, it is only possible to upload datasets for the currently authenticated user.
source
JuliaHub.update_dataset — Function
JuliaHub.update_dataset(dataset::DatasetReference; kwargs..., [auth]) -> Dataset
Updates the metadata of the dataset specified by the dataset reference dataset, as according to the keyword arguments keyword arguments. If the keywords are omitted, the metadata corresponding to it remains unchanged. Returns the Dataset object corresponding to the updated dataset.
The supported keywords are:
  • description: description of the dataset (a string)
  • tags: an iterable of strings of all the tags of the dataset
  • visibility: a string with possible values public or private
  • license: a valid SPDX license identifier, or a tuple (:fulltext, license_text), where license_text is the full text string of a custom license
  • groups: an iterable of valid group names
For example, to add a new tag to a dataset:
dataset = JuliaHub.dataset("my_dataset")
-JuliaHub.update(dataset; tags = [dataset.tags..., "newtag"])
Note
Presently, it is only possible to update datasets for the currently authenticated user.
source
JuliaHub.delete_dataset — Function
JuliaHub.delete_dataset(dataset::DatasetReference; force::Bool=false, [auth::Authentication]) -> Nothing
Delete the dataset specified by the dataset reference dataset. Will return nothing if the delete was successful, or throws an error if it was not.
Normally, when the dataset to be deleted does not exist, the function throws an error. This can be overridden by setting force = true.
julia> JuliaHub.datasets()
+) -> String
Downloads the dataset specified by the dataset reference dataset to local_path (which must not exist, unless replace = true), returning the absolute path to the downloaded file or directory. If the dataset is a Blob, then the created local_path will be a file, and if the dataset is a BlobTree the local_path will be a directory.
By default, it downloads the latest version, but an older version can be downloaded by specifying the version keyword argument. Caution: you should never assume that the index of the .versions property of Dataset matches the version number – always explicitly use the .id propert of the DatasetVersion object.
The function also prints download progress to standard output. This can be disabled by setting quiet=true. Any error output from the download is still printed.
Warning
Setting replace = true will recursively erase any existing data at local_path before replacing it with the dataset contents.
source
JuliaHub.upload_dataset — Function
JuliaHub.upload_dataset(dataset::DatasetReference, local_path; [auth,] kwargs...) -> Dataset
Uploads a new dataset or a new version of an existing dataset, with the dataset specified by the dataset reference dataset. The dataset type is determined from the local path (Blob if a file, BlobTree if a directory). If a Dataset object is passed, it attempts to update that dataset. Returns an updated Dataset object.
The following keyword arguments can be used to control the exact behavior of the function:
  • create :: Bool (default: true): Create the dataset, if it already does not exist.
  • update :: Bool (default: false): Upload the data as a new dataset version, if the dataset exists.
  • replace :: Bool (default: false): If a dataset exists, delete all existing data and create a new dataset with the same name instead. Excludes update = true, and only creates a completely new dataset if create=true as well.
In addition, the following keyword arguments can be passed to set or updated the dataset metadata when uploading:
  • description: description of the dataset (a string)
  • tags: an iterable of strings of all the tags of the dataset
  • visibility: a string with possible values public or private
  • license: a valid SPDX license identifier, or a tuple (:fulltext, license_text), where license_text is the full text string of a custom license
  • groups: an iterable of valid group names
If a dataset already exists, then these fields are updated as if update_dataset was called.
The function will throw an ArgumentError for invalid argument combinations.
Use the progress keyword argument to suppress upload progress from being printed.
Note
Presently, it is only possible to upload datasets for the currently authenticated user.
source
JuliaHub.update_dataset — Function
JuliaHub.update_dataset(dataset::DatasetReference; kwargs..., [auth]) -> Dataset
Updates the metadata of the dataset specified by the dataset reference dataset, as according to the keyword arguments keyword arguments. If the keywords are omitted, the metadata corresponding to it remains unchanged. Returns the Dataset object corresponding to the updated dataset.
The supported keywords are:
  • description: description of the dataset (a string)
  • tags: an iterable of strings of all the tags of the dataset
  • visibility: a string with possible values public or private
  • license: a valid SPDX license identifier, or a tuple (:fulltext, license_text), where license_text is the full text string of a custom license
  • groups: an iterable of valid group names
For example, to add a new tag to a dataset:
dataset = JuliaHub.dataset("my_dataset")
+JuliaHub.update(dataset; tags = [dataset.tags..., "newtag"])
Note
Presently, it is only possible to update datasets for the currently authenticated user.
source
JuliaHub.delete_dataset — Function
JuliaHub.delete_dataset(dataset::DatasetReference; force::Bool=false, [auth::Authentication]) -> Nothing
Delete the dataset specified by the dataset reference dataset. Will return nothing if the delete was successful, or throws an error if it was not.
Normally, when the dataset to be deleted does not exist, the function throws an error. This can be overridden by setting force = true.
julia> JuliaHub.datasets()
 2-element Vector{JuliaHub.Dataset}:
  JuliaHub.dataset(("username", "example-dataset"))
  JuliaHub.dataset(("username", "blobtree"))
@@ -40,9 +40,9 @@
 
 julia> JuliaHub.datasets()
 1-element Vector{JuliaHub.Dataset}:
- JuliaHub.dataset(("username", "blobtree"))
Note
Presently, it is only possible to delete datasets for the currently authenticated user.
source
Index

+

diff --git a/dev/reference/exceptions/index.html b/dev/reference/exceptions/index.html
index 8fb0a0017..98d675b43 100644
--- a/dev/reference/exceptions/index.html
+++ b/dev/reference/exceptions/index.html
@@ -1,7 +1,7 @@
 
-Exceptions · JuliaHub.jl
GitHub
Exceptions
JuliaHub.jl is designed in a way that the only errors it should throw under normal circumstances are subtypes of JuliaHubException (in addition to standard ArgumentErrors and MethodErrors etc. from invalid function calls). Any unhandled errors from JuliaHub.jl or its dependencies should be considered a bug.
Debugging JuliaHub.jl issues
You can also enable debug logging for JuliaHub, which will make JuliaHub.jl print out additional debug messages, by setting the JULIA_DEBUG environment variable:
ENV["JULIA_DEBUG"]="JuliaHub"
JuliaHub.AuthenticationError — Type
struct AuthenticationError <: JuliaHubException
Exception thrown if the authentication fails. The .msg fields contains a human-readable error message.
source
JuliaHub.InvalidRequestError — Type
struct InvalidRequestError <: JuliaHubException
An exception thrown if the request was rejected by the backend due to request parameters that are inconsistent with the backend state. The .msg field contains the error message.
source
JuliaHub.JuliaHubConnectionError — Type
struct JuliaHubConnectionError <: JuliaHubException
An exception thrown if there is a communication error with JuliaHub.
The .msg field contains the error message. If there is an underlying exception, it is stored in the .exception field.
source
JuliaHub.JuliaHubError — Type
struct JuliaHubError <: JuliaHubException
An exception thrown if there is an unexpected response from or backend failure in JuliaHub.
The .msg field contains the error message. If there is an underlying exception, it is stored in the .exception field.
source
JuliaHub.PermissionError — Type
struct PermissionError <: JuliaHubException
Thrown if the currently authenticated user does not have the necessary permissions to perform the operation. The .msg field contains the error message, and .response may contain the raw server response.
source
Index
GitHub
Exceptions
JuliaHub.jl is designed in a way that the only errors it should throw under normal circumstances are subtypes of JuliaHubException (in addition to standard ArgumentErrors and MethodErrors etc. from invalid function calls). Any unhandled errors from JuliaHub.jl or its dependencies should be considered a bug.
Debugging JuliaHub.jl issues
You can also enable debug logging for JuliaHub, which will make JuliaHub.jl print out additional debug messages, by setting the JULIA_DEBUG environment variable:
ENV["JULIA_DEBUG"]="JuliaHub"
JuliaHub.AuthenticationError — Type
struct AuthenticationError <: JuliaHubException
Exception thrown if the authentication fails. The .msg fields contains a human-readable error message.
source
JuliaHub.InvalidRequestError — Type
struct InvalidRequestError <: JuliaHubException
An exception thrown if the request was rejected by the backend due to request parameters that are inconsistent with the backend state. The .msg field contains the error message.
source
JuliaHub.JuliaHubConnectionError — Type
struct JuliaHubConnectionError <: JuliaHubException
An exception thrown if there is a communication error with JuliaHub.
The .msg field contains the error message. If there is an underlying exception, it is stored in the .exception field.
source
JuliaHub.JuliaHubError — Type
struct JuliaHubError <: JuliaHubException
An exception thrown if there is an unexpected response from or backend failure in JuliaHub.
The .msg field contains the error message. If there is an underlying exception, it is stored in the .exception field.
source
JuliaHub.PermissionError — Type
struct PermissionError <: JuliaHubException
Thrown if the currently authenticated user does not have the necessary permissions to perform the operation. The .msg field contains the error message, and .response may contain the raw server response.
source
Index

+

diff --git a/dev/reference/job-submission/index.html b/dev/reference/job-submission/index.html
index 4d2ab1c6c..a50b8fe76 100644
--- a/dev/reference/job-submission/index.html
+++ b/dev/reference/job-submission/index.html
@@ -1,13 +1,13 @@
 
 Job submission · JuliaHub.jl
GitHub
Job submission
On JuliaHub you can submit jobs, which are user-defined workloads that get allocated a dedicated compute capacity. For example, this includes running scripts in batch computations, cloud IDEs, interactive notebooks and so on. The functions and types here deal with starting up such jobs and apps. The functions to inspect running or finished jobs are documented separately.
A complete JuliaHub job workload is defined by the following configuration pieces:
  • Job configuration. This specifies the computation that gets run, and includes information such as the type of computation (batch script vs starting a GUI application), any arguments or Julia code that gets passed to the job etc.
    As of now, there are three general types of jobs that can be run on JuliaHub:
    Each of these categories are configured slightly differently, and are described in more detail below.
  • Compute configuration. This specifies the hardware and cluster topology that will be used to execute the job (such as the number of CPUs per node, whether there is a GPU present, the number of nodes), how the Julia processes are configured (e.g. single process per node vs process per cpu), and other low-level technical configuration.
  • Runtime parameters. These are various additional parameters that control how a job behaves. Currently, this is limited to passing environment variables to the jobs, overriding the job's name in the UI, and to various configuration options related to running interactive jobs.
In the JuliaHub.jl code, the first two categories are encapsulated by the AbstractJobConfig and ComputeConfig types. These two, together with the additional runtime parameters, make up a WorkloadConfig object that can be submitted to JuliaHub for executing with the submit_job function.
The following sections on this page explain the different aspects of job submission in more detail. See the guide on submitting batch jobs to see more practical examples of how to submit jobs.
Compute configuration
JuliaHub supports a predefined set of node configurations, each of which have a specific number of CPUs, memory, GPUs etc. A JuliaHub job must pick one of these node types to run on, although a distributed job can run across multiple instances of the same node type. A list of these node specifications can be obtained with the nodespecs function (returning a list of NodeSpec objects).
julia> JuliaHub.nodespecs()
 9-element Vector{JuliaHub.NodeSpec}:
- JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 0.33/hr =#; ncpu=4, memory=16, ngpu=false, exactmatch=true)
- JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 0.65/hr =#; ncpu=8, memory=32, ngpu=false, exactmatch=true)
- JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 2.4/hr =#; ncpu=32, memory=128, ngpu=false, exactmatch=true)
+ JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 0.17/hr =#; ncpu=2, memory=8, ngpu=false, exactmatch=true)
  JuliaHub.nodespec(#= r6: 3.5 GHz Intel Xeon Platinum 8375C, 0.22/hr =#; ncpu=2, memory=16, ngpu=false, exactmatch=true)
+ JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 0.33/hr =#; ncpu=4, memory=16, ngpu=false, exactmatch=true)
  JuliaHub.nodespec(#= r6: 3.5 GHz Intel Xeon Platinum 8375C, 0.42/hr =#; ncpu=4, memory=32, ngpu=false, exactmatch=true)
- JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 0.17/hr =#; ncpu=2, memory=8, ngpu=false, exactmatch=true)
+ JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 0.65/hr =#; ncpu=8, memory=32, ngpu=false, exactmatch=true)
  JuliaHub.nodespec(#= r6: 3.5 GHz Intel Xeon Platinum 8375C, 1.3/hr =#; ncpu=8, memory=64, ngpu=false, exactmatch=true)
+ JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 2.4/hr =#; ncpu=32, memory=128, ngpu=false, exactmatch=true)
  JuliaHub.nodespec(#= p2: Intel Xeon E5-2686 v4 (Broadwell), 1.4/hr =#; ncpu=4, memory=61, ngpu=true, exactmatch=true)
  JuliaHub.nodespec(#= p3: Intel Xeon E5-2686 v4 (Broadwell), 4.5/hr =#; ncpu=8, memory=61, ngpu=true, exactmatch=true)
While you can manually index into the list returned by nodespecs, that is generally inconvenient. Instead, the nodespec function should be used to find a suitable node for a particular job.
julia> JuliaHub.nodespec(ncpu=2, memory=8)
 Node: 3.5 GHz Intel Xeon Platinum 8375C
@@ -37,7 +37,7 @@
   @info "Extracting 'MY_PARAMETER'" get(ENV, "MY_PARAMETER", nothing)
   """
   sha256(project_toml) = 93a83d60d4a9c6a3d1438259fd506929eaad296b7e112e886b305781b85cb85b
-  sha256(manifest_toml) = cb7bf4658b8d6daea993e9d71acd44b9e172a044b99ae83fa04c7ac07ee2f3aa
+  sha256(manifest_toml) = 7f6a3052cad3e8097b148270b3350caaedc38034c123866b503450e77744966b
 compute:
   JuliaHub.ComputeConfig
    Node: 3.5 GHz Intel Xeon Platinum 8375C
@@ -86,27 +86,27 @@
  - GPU: no
  - vCores: 2
  - Memory: 8 Gb
- - Price: 0.17 $/hr
They can be used to contruct explicit compute configuration objects when submitting JuliaHub jobs.
See also: submit_job, ComputeConfig.
source
JuliaHub.nodespecs — Function
JuliaHub.nodespecs(; auth::Authentication) -> Vector{NodeSpec}
Query node specifications available on the current server, returning a list of NodeSpec objects.
source
JuliaHub.nodespecs — Function
JuliaHub.nodespecs(; auth::Authentication) -> Vector{NodeSpec}
Query node specifications available on the current server, returning a list of NodeSpec objects.
source
JuliaHub.nodespec — Function
JuliaHub.nodespec(
     [nodes::Vector{NodeSpec}];
     ncpu::Integer=1, ngpu::Integer=false, memory::Integer=1,
     exactmatch::Bool=false, throw::Bool=true,
     [auth::Authentication]
-) -> Union{NodeSpec, Nothing}
Finds the node matching the specified node parameters. Throws an InvalidRequestError if it is unable to find a node with the specific parameters. However, if throw is set to false, it will return nothing instead in that situation.
By default, it searches for the smallest node that has the specified parameters or more higher. If exactmatch is set to true, it only returns a node specification if it can find one that matches the parameters exactly.
A list of nodes (e.g. from nodespecs) can also be passed, so that the function does not have to query the server for the list. When this method is used, it is not necessary to pass auth.
source
JuliaHub.BatchImage — Type
struct BatchImage
Represents an available JuliaHub batch job image. These can be passed to BatchJob to specify which underlying job image will be used for the job.
A list of available batch images can be accessed with batchimages and specific images can be constructed with batchimage.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
See also: batchimages, batchimage, BatchJob, script, appbundle.
source
JuliaHub.batchimages — Function
JuliaHub.batchimages([product::AbstractString]; [auth::Authentication]) -> Vector{BatchImage}
Return the list of all batch job images available to the currently authenticated user, as a list of BatchImage objects. These can be passed to BatchJob.
Optionally, by passing a product identifier, the list can be narrowed down to images available for that specific product.
Batch images on older instances
When using the package with an older JuliaHub instance (<= 6.1), the non-default batch images show up with @legacy as the product name. This indicates that the package is using an older API, and not that the images themselves are outdated.
See also: BatchImage, batchimage, BatchJob, script, appbundle.
source
JuliaHub.batchimage — Function
JuliaHub.batchimage(
+) -> Union{NodeSpec, Nothing}
Finds the node matching the specified node parameters. Throws an InvalidRequestError if it is unable to find a node with the specific parameters. However, if throw is set to false, it will return nothing instead in that situation.
By default, it searches for the smallest node that has the at least the specified parameters (prioritizing GPU count, CPU count, and memory in this order when deciding). If exactmatch is set to true, it only returns a node specification if it can find one that matches the parameters exactly.
A list of nodes (e.g. from nodespecs) can also be passed, so that the function does not have to query the server for the list. When this method is used, it is not necessary to pass auth.
source
JuliaHub.BatchImage — Type
struct BatchImage
Represents an available JuliaHub batch job image. These can be passed to BatchJob to specify which underlying job image will be used for the job.
A list of available batch images can be accessed with batchimages and specific images can be constructed with batchimage.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
See also: batchimages, batchimage, BatchJob, script, appbundle.
source
JuliaHub.batchimages — Function
JuliaHub.batchimages([product::AbstractString]; [auth::Authentication]) -> Vector{BatchImage}
Return the list of all batch job images available to the currently authenticated user, as a list of BatchImage objects. These can be passed to BatchJob.
Optionally, by passing a product identifier, the list can be narrowed down to images available for that specific product.
Batch images on older instances
When using the package with an older JuliaHub instance (<= 6.1), the non-default batch images show up with @legacy as the product name. This indicates that the package is using an older API, and not that the images themselves are outdated.
See also: BatchImage, batchimage, BatchJob, script, appbundle.
source
JuliaHub.batchimage — Function
JuliaHub.batchimage(
     [product::AbstractString, [image::AbstractString]];
     throw::Bool=true, [auth::Authentication]
-) -> BatchImage
Pick a product job batch image from the list of all batch image, returning a BatchImage object. If image is omitted, it will return the default image corresponding to product. If product is omitted as well, it will return the default image of the instance (generally the standard Julia batch image).
Will throw an InvalidRequestError if the specified image can not be found. If throw=false, it will return nothing instead in this situation.
Batch images on older instances
When using the package with an older JuliaHub instance (<= 6.1), the non-default batch images show up with @legacy as the product name. This indicates that the package is using an older API, and not that the images themselves are outdated.
See also: BatchImage, batchimages, BatchJob, script, appbundle.
source
JuliaHub.BatchJob — Type
struct BatchJob <: AbstractJobConfig
Represents the application configuration of a JuliaHub batch job. A batch job is defined by the following information:
  • The Julia code that is to be executed in the job.
  • Julia package environment (i.e. Project.toml, Manifest.toml) and other files, such as the appbundle.
  • The underlying batch job container image (see also batchimages), which defaults to the standard Julia image by default.
Instances of this types should normally not be constructed directly, and the following functions should be used instead:
  • script or @script_str: for submitting simple Julia scripts or code snippets
  • appbundle: for submitting more complex "appbundles" that include additional file, private or modified package dependencies etc.
Optional arguments
  • image  :: Union{BatchImage, Nothing}: can be used to specify which product's batch job image will be used when running the job, by passing the appropriate BatchImage object (see also: batchimage and batchimages). If set to nothing (the default), the job runs with the default Julia image.
  • sysimage :: Bool: if set to true, requests that a system image is built from the job's Manifest.toml file before starting the job. Defaults to false.
JuliaHub compatibility
The sysimage = true option requires JuliaHub 6.3 to have an effect. When running against older JuliaHub versions, it does not have an effect.
Constructors
BatchJob(::BatchJob; [image::BatchImage], [sysimage::Bool]) -> BatchJob
Construct a BatchJob, but override some of the optional arguments documented above. When the argument is omitted, the value from the underlying BatchJob object is used. This is the only constructor that is part of the public API.
This method is particularly useful when used in in combination with the @script_str string macro, to be able to specify the job image or trigger a sysimage build. For example, the following snippet will set a different batch image for the script-type job:
JuliaHub.BatchJob(
+) -> BatchImage
Pick a product job batch image from the list of all batch image, returning a BatchImage object. If image is omitted, it will return the default image corresponding to product. If product is omitted as well, it will return the default image of the instance (generally the standard Julia batch image).
Will throw an InvalidRequestError if the specified image can not be found. If throw=false, it will return nothing instead in this situation.
Batch images on older instances
When using the package with an older JuliaHub instance (<= 6.1), the non-default batch images show up with @legacy as the product name. This indicates that the package is using an older API, and not that the images themselves are outdated.
See also: BatchImage, batchimages, BatchJob, script, appbundle.
source
JuliaHub.BatchJob — Type
struct BatchJob <: AbstractJobConfig
Represents the application configuration of a JuliaHub batch job. A batch job is defined by the following information:
  • The Julia code that is to be executed in the job.
  • Julia package environment (i.e. Project.toml, Manifest.toml) and other files, such as the appbundle.
  • The underlying batch job container image (see also batchimages), which defaults to the standard Julia image by default.
Instances of this types should normally not be constructed directly, and the following functions should be used instead:
  • script or @script_str: for submitting simple Julia scripts or code snippets
  • appbundle: for submitting more complex "appbundles" that include additional file, private or modified package dependencies etc.
Optional arguments
  • image  :: Union{BatchImage, Nothing}: can be used to specify which product's batch job image will be used when running the job, by passing the appropriate BatchImage object (see also: batchimage and batchimages). If set to nothing (the default), the job runs with the default Julia image.
  • sysimage :: Bool: if set to true, requests that a system image is built from the job's Manifest.toml file before starting the job. Defaults to false.
JuliaHub compatibility
The sysimage = true option requires JuliaHub 6.3 to have an effect. When running against older JuliaHub versions, it does not have an effect.
Constructors
BatchJob(::BatchJob; [image::BatchImage], [sysimage::Bool]) -> BatchJob
Construct a BatchJob, but override some of the optional arguments documented above. When the argument is omitted, the value from the underlying BatchJob object is used. This is the only constructor that is part of the public API.
This method is particularly useful when used in in combination with the @script_str string macro, to be able to specify the job image or trigger a sysimage build. For example, the following snippet will set a different batch image for the script-type job:
JuliaHub.BatchJob(
     JuliaHub.script"""
     @info "Hello World"
     """,
     image = JuliaHub.batchimage("...")
-)
source
JuliaHub.script — Function
JuliaHub.script(...) -> BatchJob
Constructs the configuration for a script-type batch job, returning the respective BatchJob object that can then be passed to submit_job. A script-type batch job is defined by the following:
  • A user-provided Julia script that gets executed on the server. Note that no validation of the input code is done.
  • An optional Julia package environment (e.g. Project.toml, Manifest.toml and Artifacts.toml). If any of the TOML files are provided, they must parse as valid TOML files, but no further validation is done client-side.
    If the manifest is not provided, the project environment must be instantiated from scratch, generally pulling in the latest versions of all the dependencies (although [compat] sections are honored).
    It is also fine to omit the project file, and just provide the manifest, and the environment defined by the manifest still gets instantiated. If both are omitted, the job runs in an empty environment.
  • A JuliaHub job image, which determines the container environment that will be used to execute the code in (see batchimage, batchimages, BatchImage). If omitted, the default Julia image is used.
See also the @script_str string macro to more easily submit simple scripts that are defined in code.
Methods
script(
+)
source
JuliaHub.script — Function
JuliaHub.script(...) -> BatchJob
Constructs the configuration for a script-type batch job, returning the respective BatchJob object that can then be passed to submit_job. A script-type batch job is defined by the following:
  • A user-provided Julia script that gets executed on the server. Note that no validation of the input code is done.
  • An optional Julia package environment (e.g. Project.toml, Manifest.toml and Artifacts.toml). If any of the TOML files are provided, they must parse as valid TOML files, but no further validation is done client-side.
    If the manifest is not provided, the project environment must be instantiated from scratch, generally pulling in the latest versions of all the dependencies (although [compat] sections are honored).
    It is also fine to omit the project file, and just provide the manifest, and the environment defined by the manifest still gets instantiated. If both are omitted, the job runs in an empty environment.
  • A JuliaHub job image, which determines the container environment that will be used to execute the code in (see batchimage, batchimages, BatchImage). If omitted, the default Julia image is used.
See also the @script_str string macro to more easily submit simple scripts that are defined in code.
Methods
script(
     scriptfile::AbstractString;
     [project_directory::AbstractString], [image::BatchImage], [sysimage::Bool]
 ) -> BatchJob
Constructs a script-type batch job configuration the will execute the code in scriptfile. Optionally, a path to a project environment directory can be passed via project_directory, which will be searched for the environment TOML files, and a job image can be specified via image.
script(;
     code::AbstractString,
     [project::AbstractString], [manifest::AbstractString], [artifacts::AbstractString],
     [image::BatchImage], [sysimage::Bool]
-) -> BatchJob
A lower-level method that can be used to construct the script-type BatchJob configuration directly in memory (i.e. without having to write out intermediate files).
The code keyword argument is mandatory and will specify contents of the Julia script that gets executed on the server. The Julia project environment can be specified by passing the contents of the TOML files via the corresponding arguments (project, manifest, artifacts). The job image can be specified via image.
Optional arguments
See BatchJob for a more thorough description of the optional arguments.
source
JuliaHub.@script_str — Macro
JuliaHub.@script_str -> JuliaHub.BatchJob
A string macro to conveniently construct a script-type batch job configuration (BatchJob) that can be submitted as a JuliaHub job.
script = JuliaHub.script"""
+) -> BatchJob
A lower-level method that can be used to construct the script-type BatchJob configuration directly in memory (i.e. without having to write out intermediate files).
The code keyword argument is mandatory and will specify contents of the Julia script that gets executed on the server. The Julia project environment can be specified by passing the contents of the TOML files via the corresponding arguments (project, manifest, artifacts). The job image can be specified via image.
Optional arguments
See BatchJob for a more thorough description of the optional arguments.
source
JuliaHub.@script_str — Macro
JuliaHub.@script_str -> JuliaHub.BatchJob
A string macro to conveniently construct a script-type batch job configuration (BatchJob) that can be submitted as a JuliaHub job.
script = JuliaHub.script"""
 @info "Hello World!"
 """
This allows for an easy submission of simple single-script jobs to JuliaHub:
JuliaHub.submit_job(
     JuliaHub.script"""
@@ -121,7 +121,7 @@
         """,
         image = JuliaHub.batchimage(...)
     )
-)
You can also use this pattern to set the sysimage option.
source
JuliaHub.appbundle — Function
JuliaHub.appbundle(
     directory::AbstractString, codefile::AbstractString;
     [image::BatchImage], [sysimage::Bool]
 ) -> BatchJob
@@ -133,12 +133,12 @@
     code = """
     @show ENV
     """
-)
See BatchJob for a description of the optional arguments.
Extended help
The following should be kept in mind about how appbundles are handled:
  • The bundler looks for a Julia environment (i.e. Project.toml and/or Manifest.toml files) at the root of the directory. If the environment does not exist (i.e. the files are missing), the missing files are created. If the manifest is missing, then the environment is re-instantiated from scratch based on the contents of Project.toml. The generated files will also be left in the user-provided directory directory.
  • Development dependencies of the environment (i.e. packages added with pkg> develop or Pkg.develop()) are also bundled up into the archive that gets submitted to JuliaHub (including any current, uncommitted changes). Registered packages are installed via the package manager via the standard environment instantiation, and their source code is not included in the bundle directly.
  • When the JuliaHub job starts, the bundle is unpacked and the job's starting working directory is set to the appbundle/ directory, and you can e.g. load the data from those files with just read("my-data.txt", String).
    Note that @__DIR__ points elsewhere and, relatedly, include in the main script should be used with an absolute path (e.g. include(joinpath(pwd(), "my-julia-file.jl"))).
    JuliaHub 6.2 and older
    On some older JuliaHub versions (6.2 and older), the working directory was set to the parent directory of appbundle/, and so it was necessary to do joinpath("appbundle", "mydata.dat") to load the code.
source
JuliaHub.ComputeConfig — Type
struct ComputeConfig
This type encapsulates the configuration of a jobs's compute cluster, including the hardware configuration and the cluster topology.
See also: submit_job.
Constructors
JuliaHub.ComputeConfig(
+)
See BatchJob for a description of the optional arguments.
Extended help
The following should be kept in mind about how appbundles are handled:
  • The bundler looks for a Julia environment (i.e. Project.toml and/or Manifest.toml files) at the root of the directory. If the environment does not exist (i.e. the files are missing), the missing files are created. If the manifest is missing, then the environment is re-instantiated from scratch based on the contents of Project.toml. The generated files will also be left in the user-provided directory directory.
  • Development dependencies of the environment (i.e. packages added with pkg> develop or Pkg.develop()) are also bundled up into the archive that gets submitted to JuliaHub (including any current, uncommitted changes). Registered packages are installed via the package manager via the standard environment instantiation, and their source code is not included in the bundle directly.
  • When the JuliaHub job starts, the bundle is unpacked and the job's starting working directory is set to the appbundle/ directory, and you can e.g. load the data from those files with just read("my-data.txt", String).
    Note that @__DIR__ points elsewhere and, relatedly, include in the main script should be used with an absolute path (e.g. include(joinpath(pwd(), "my-julia-file.jl"))).
    JuliaHub 6.2 and older
    On some older JuliaHub versions (6.2 and older), the working directory was set to the parent directory of appbundle/, and so it was necessary to do joinpath("appbundle", "mydata.dat") to load the code.
source
JuliaHub.ComputeConfig — Type
struct ComputeConfig
This type encapsulates the configuration of a jobs's compute cluster, including the hardware configuration and the cluster topology.
See also: submit_job.
Constructors
JuliaHub.ComputeConfig(
     node::NodeSpec;
     nnodes::Integer = 1,
     process_per_node::Bool = true,
     elastic::Bool = false,
-)
  • node: a NodeSpec object that specifies the hardware of a single node.
  • nnodes::Union{Integer, Tuple{Integer, Integer}} = 1: specifies the number of nodes of type node that will be allocated. Alternatively, a two-integer tuple can also be passed, where the first value specifies the minimum number of nodes required to start a job. By default, a single-node job is started.
  • process_per_node::Bool = true: if true, there will only be a single Julia process per node, and the total number of Julia processes will be nnodes. If set to false, however, each core on each node will be allocated a separate Julia process (running in an isolated container on the same node), and so the total number of Julia processes will be nnodes × ncpu, and it will essentially always be a multi-process job.
  • elastic::Bool = false: if set, the job will be started in an elastic cluster mode. In this case, a minimum number of nnodes must not be passed.
source
JuliaHub.submit_job — Function
JuliaHub.submit_job(
+)
  • node: a NodeSpec object that specifies the hardware of a single node.
  • nnodes::Union{Integer, Tuple{Integer, Integer}} = 1: specifies the number of nodes of type node that will be allocated. Alternatively, a two-integer tuple can also be passed, where the first value specifies the minimum number of nodes required to start a job. By default, a single-node job is started.
  • process_per_node::Bool = true: if true, there will only be a single Julia process per node, and the total number of Julia processes will be nnodes. If set to false, however, each core on each node will be allocated a separate Julia process (running in an isolated container on the same node), and so the total number of Julia processes will be nnodes × ncpu, and it will essentially always be a multi-process job.
  • elastic::Bool = false: if set, the job will be started in an elastic cluster mode. In this case, a minimum number of nnodes must not be passed.
source
JuliaHub.submit_job — Function
JuliaHub.submit_job(
     app::Union{AbstractJuliaHubApp, AbstractJobConfig},
     [compute::ComputeConfig];
     # Compute keyword arguments
@@ -152,7 +152,7 @@
     # General keyword arguments
     dryrun::Bool = false,
     [auth :: Authentication]
-) -> Job
Submits the specified application config app as a job to JuliaHub. Returns a Job object corresponding to the submitted job.
Compute arguments. If compute is passed, the compute keyword arguments can not be passed. If compute is not passed, the following arguments can be used to specify the compute configration via keyword arguments:
  • ncpu, ngpu and memory are used to pick a node type that will be used to run the job. The node type will be a minimum one that satisfies the constraints, but may have more compute resources than specified by the arguments (it corresponds to the exactmatch = false case of nodespec).
  • nnodes, minimum_nnodes, process_per_node, and elastic specify the corresponding arguments in ComputeConfig.
Runtime configuration. These are used to set the Runtime configuration of the job.
  • alias :: Union{AbstractString, Nothing}: can be used to override the name of the job that gets displayed in the UI. Passing nothing is equivalent to omitting the argument.
  • timelimit :: Limit: sets the job's time limit (see Limit for valid values)
  • env: an iterable of key-value pairs that can be used to set environment variable that get set before, the job code gets executed.
  • project :: Union{UUID, AbstractString, Nothing}: the UUID of the project that the job will be associated with. If a string is passed, it must parse as a valid UUID. Passing nothing is equivalent to omitting the argument.
General arguments.
  • auth :: Authentication: optional authentication object (see the authentication section for more information)
  • dryrun :: Bool: if set to true, submit_job does not actually submit the job, but instead returns a WorkloadConfig object, which can be used to inspect the configuration that would be submitted.
    The WorkloadConfig object can then be submitted to JuliaHub with the additional submit_job method:
    JuliaHub.submit_job(::WorkloadConfig; [auth::Authentication])
JuliaHub compatibility
The timelimit = JuliaHub.Unlimited() argument requires JuliaHub 6.3+.
source
JuliaHub.Limit — Type
JuliaHub.Limit
Type-constraint on JuliaHub job timelimit arguments in submit_job.
The job time limit can either be a time period (an instance of Dates.Period), an Integer, (interpreted as the number of hours), or JuliaHub.Unlimited().
Only an integer number of hours are accepted by JuliaHub, and fractional hours from get rounded up to the next full integer number of hours (e.g. Dates.Minute(90) will be interpreted as 2 hours).
source
JuliaHub.Unlimited — Type
struct Unlimited
An instance of this type can be passed as the [timelimit] option to submit_job to start jobs that run indefinitely, until killed manually.
JuliaHub.submit_job(..., timelimit = JuliaHub.Unlimited())
source
JuliaHub.WorkloadConfig — Type
struct WorkloadConfig
Represents a full job configuration, including the application, compute and runtime configuration.
Instances of this type can be constructed by passing dryrun = true to submit_job, and can also be directly submitted to JuliaHub with the same function.
source
Experimental APIs
Experimental features
Starting application jobs with JuliaHub.jl is considered to be experimental. The APIs are likely to change in future JuliaHub.jl version.
JuliaHub.AbstractJuliaHubApp — Type
abstract type AbstractJuliaHubApp
Abstract supertype for JuliaHub applications object types.
Experimental API
Applications-related APIs are experimental, and may be changed or removed without notice.
source
JuliaHub.applications — Function
JuliaHub.applications([category::Symbol]; [auth::Authentication]) -> Vector{AbstractJuliaHubApp}
Returns the list of applications enabled for the authenticated user, optionally in the specified category only. Returns a vector of AbstractJuliaHubApp instances.
julia> JuliaHub.applications()
+) -> Job
Submits the specified application config app as a job to JuliaHub. Returns a Job object corresponding to the submitted job.
Compute arguments. If compute is passed, the compute keyword arguments can not be passed. If compute is not passed, the following arguments can be used to specify the compute configration via keyword arguments:
  • ncpu, ngpu and memory are used to pick a node type that will be used to run the job. The node type will be a minimum one that satisfies the constraints, but may have more compute resources than specified by the arguments (it corresponds to the exactmatch = false case of nodespec).
  • nnodes, minimum_nnodes, process_per_node, and elastic specify the corresponding arguments in ComputeConfig.
Runtime configuration. These are used to set the Runtime configuration of the job.
  • alias :: Union{AbstractString, Nothing}: can be used to override the name of the job that gets displayed in the UI. Passing nothing is equivalent to omitting the argument.
  • timelimit :: Limit: sets the job's time limit (see Limit for valid values)
  • env: an iterable of key-value pairs that can be used to set environment variable that get set before, the job code gets executed.
  • project :: Union{UUID, AbstractString, Nothing}: the UUID of the project that the job will be associated with. If a string is passed, it must parse as a valid UUID. Passing nothing is equivalent to omitting the argument.
General arguments.
  • auth :: Authentication: optional authentication object (see the authentication section for more information)
  • dryrun :: Bool: if set to true, submit_job does not actually submit the job, but instead returns a WorkloadConfig object, which can be used to inspect the configuration that would be submitted.
    The WorkloadConfig object can then be submitted to JuliaHub with the additional submit_job method:
    JuliaHub.submit_job(::WorkloadConfig; [auth::Authentication])
JuliaHub compatibility
The timelimit = JuliaHub.Unlimited() argument requires JuliaHub 6.3+.
source
JuliaHub.Limit — Type
JuliaHub.Limit
Type-constraint on JuliaHub job timelimit arguments in submit_job.
The job time limit can either be a time period (an instance of Dates.Period), an Integer, (interpreted as the number of hours), or JuliaHub.Unlimited().
Only an integer number of hours are accepted by JuliaHub, and fractional hours from get rounded up to the next full integer number of hours (e.g. Dates.Minute(90) will be interpreted as 2 hours).
source
JuliaHub.Unlimited — Type
struct Unlimited
An instance of this type can be passed as the [timelimit] option to submit_job to start jobs that run indefinitely, until killed manually.
JuliaHub.submit_job(..., timelimit = JuliaHub.Unlimited())
source
JuliaHub.WorkloadConfig — Type
struct WorkloadConfig
Represents a full job configuration, including the application, compute and runtime configuration.
Instances of this type can be constructed by passing dryrun = true to submit_job, and can also be directly submitted to JuliaHub with the same function.
source
Experimental APIs
Experimental features
Starting application jobs with JuliaHub.jl is considered to be experimental. The APIs are likely to change in future JuliaHub.jl version.
JuliaHub.AbstractJuliaHubApp — Type
abstract type AbstractJuliaHubApp
Abstract supertype for JuliaHub applications object types.
Experimental API
Applications-related APIs are experimental, and may be changed or removed without notice.
source
JuliaHub.applications — Function
JuliaHub.applications([category::Symbol]; [auth::Authentication]) -> Vector{AbstractJuliaHubApp}
Returns the list of applications enabled for the authenticated user, optionally in the specified category only. Returns a vector of AbstractJuliaHubApp instances.
julia> JuliaHub.applications()
 7-element Vector{JuliaHub.AbstractJuliaHubApp}:
  JuliaHub.application(:default, "Linux Desktop")
  JuliaHub.application(:default, "Julia IDE")
@@ -177,7 +177,7 @@
 julia> JuliaHub.applications(:user)
 1-element Vector{JuliaHub.UserApp}:
  JuliaHub.application(:user, "ExampleApp.jl")
-
Experimental API
Applications-related APIs are experimental, and may be changed or removed without notice.
source
JuliaHub.application — Function
JuliaHub.application(
+
Experimental API
Applications-related APIs are experimental, and may be changed or removed without notice.
source
JuliaHub.application — Function
JuliaHub.application(
     category::Symbol, name::AbstractString;
     throw::Bool=true, [auth::Authentication]
 ) -> AbstractJuliaHubApp
Returns the application corresponding to name from the specified category of applications. Will throw an InvalidRequestError if the application can't be found, or returns nothing in this situation if throw=false is passed.
category specifies the application category and must be one of: :default, :package, or :user. This is necessary to disambiguate apps with the same name in the different categories.
See also: applications.
Examples
julia> JuliaHub.applications()
@@ -188,7 +188,7 @@
  JuliaHub.application(:default, "Windows Workstation")
  JuliaHub.application(:package, "RegisteredPackageApp")
  JuliaHub.application(:package, "CustomDashboardApp")
- JuliaHub.application(:user, "ExampleApp.jl")
Experimental API
Applications-related APIs are experimental, and may be changed or removed without notice.
source
JuliaHub.DefaultApp — Type
struct DefaultApp <: AbstractJuliaHubApp
Represents a default JuliaHub instance application, and they can be started as jobs with submit_job.
The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.
julia> apps = JuliaHub.applications(:default)
+ JuliaHub.application(:user, "ExampleApp.jl")
Experimental API
Applications-related APIs are experimental, and may be changed or removed without notice.
source
JuliaHub.DefaultApp — Type
struct DefaultApp <: AbstractJuliaHubApp
Represents a default JuliaHub instance application, and they can be started as jobs with submit_job.
The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.
julia> apps = JuliaHub.applications(:default)
 4-element Vector{JuliaHub.DefaultApp}:
  JuliaHub.application(:default, "Linux Desktop")
  JuliaHub.application(:default, "Julia IDE")
@@ -198,7 +198,7 @@
 julia> JuliaHub.application(:default, "Pluto")
 DefaultApp
  name: Pluto
- key: pluto
Experimental API
Applications-related APIs are experimental, and may be changed or removed without notice.
source
JuliaHub.PackageApp — Type
struct PackageApp <: AbstractJuliaHubApp
Represents a JuliaHub package application that is available in one of the instance's package registries. These packages can be started as JuliaHub jobs with submit_job.
The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.
julia> apps = JuliaHub.applications(:package)
+ key: pluto
Experimental API
Applications-related APIs are experimental, and may be changed or removed without notice.
source
JuliaHub.PackageApp — Type
struct PackageApp <: AbstractJuliaHubApp
Represents a JuliaHub package application that is available in one of the instance's package registries. These packages can be started as JuliaHub jobs with submit_job.
The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.
julia> apps = JuliaHub.applications(:package)
 2-element Vector{JuliaHub.PackageApp}:
  JuliaHub.application(:package, "RegisteredPackageApp")
  JuliaHub.application(:package, "CustomDashboardApp")
@@ -207,14 +207,14 @@
 PackageApp
  name: RegisteredPackageApp
  uuid: db8b4d46-bfad-4aa5-a5f8-40df1e9542e5
- registry: General (23338594-aafe-5451-b93e-139f81909106)
See also: help.juliahub.com on applications
Experimental API
Applications-related APIs are experimental, and may be changed or removed without notice.
source
JuliaHub.UserApp — Type
struct UserApp <: AbstractJuliaHubApp
Represents a private application that has been added to the user account via a Git repository. These applications can be started as JuliaHub jobs with submit_job.
The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.
julia> apps = JuliaHub.applications(:user)
+ registry: General (23338594-aafe-5451-b93e-139f81909106)
See also: help.juliahub.com on applications
Experimental API
Applications-related APIs are experimental, and may be changed or removed without notice.
source
JuliaHub.UserApp — Type
struct UserApp <: AbstractJuliaHubApp
Represents a private application that has been added to the user account via a Git repository. These applications can be started as JuliaHub jobs with submit_job.
The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.
julia> apps = JuliaHub.applications(:user)
 1-element Vector{JuliaHub.UserApp}:
  JuliaHub.application(:user, "ExampleApp.jl")
 
 julia> JuliaHub.application(:user, "ExampleApp.jl")
 UserApp
  name: ExampleApp.jl
- repository: https://github.com/JuliaHubExampleOrg/ExampleApp.jl
See also: help.juliahub.com on applications
Experimental API
Applications-related APIs are experimental, and may be changed or removed without notice.
source
JuliaHub.PackageJob — Type
struct PackageJob <: AbstractJobConfig
AbstractJobConfig that wraps a PackageApp or UserApp. This is primarily used internally and should rarely be constructed explicitly.
Constructors
JuliaHub.PackageJob(app::Union{JuliaHub.PackageApp,JuliaHub.UserApp}; [sysimage::Bool = false])
Can be used to construct a PackageApp or UserApp based job, but allows for some job parameters to be overridden. Currently, only support the enabling of a system image based job by setting sysimage = true.
julia> app = JuliaHub.application(:package, "RegisteredPackageApp")
+ repository: https://github.com/JuliaHubExampleOrg/ExampleApp.jl
See also: help.juliahub.com on applications
Experimental API
Applications-related APIs are experimental, and may be changed or removed without notice.
source
JuliaHub.PackageJob — Type
struct PackageJob <: AbstractJobConfig
AbstractJobConfig that wraps a PackageApp or UserApp. This is primarily used internally and should rarely be constructed explicitly.
Constructors
JuliaHub.PackageJob(app::Union{JuliaHub.PackageApp,JuliaHub.UserApp}; [sysimage::Bool = false])
Can be used to construct a PackageApp or UserApp based job, but allows for some job parameters to be overridden. Currently, only support the enabling of a system image based job by setting sysimage = true.
julia> app = JuliaHub.application(:package, "RegisteredPackageApp")
 PackageApp
  name: RegisteredPackageApp
  uuid: db8b4d46-bfad-4aa5-a5f8-40df1e9542e5
@@ -224,9 +224,9 @@
 JuliaHub.Job: jr-xf4tslavut (Submitted)
  submitted: 2023-03-15T07:56:50.974+00:00
  started:   2023-03-15T07:56:51.251+00:00
- finished:  2023-03-15T07:56:59.000+00:00
source
Index

+
diff --git a/dev/reference/jobs/index.html b/dev/reference/jobs/index.html
index 5ee7a637c..a0b5f893d 100644
--- a/dev/reference/jobs/index.html
+++ b/dev/reference/jobs/index.html
@@ -13,18 +13,18 @@
 }
 FINISHED:::user
 
-classDef user fill:lightgray
Refreshing a Job object
An instance of a Job object reflects the state of the job when the job function was called. If you want to inspect the current state of a job, you must first "refresh" the job object, which can simply be done with the help of the job function.
job = JuliaHub.job(job)
Reference
JuliaHub.JobReference — Type
const JobReference :: Type
A type constraint on the arguments of many jobs-related functions that is used to specify the job. A job reference must be either a Job object, or an AbstractString containing the unique job ID.
source
JuliaHub.jobs — Function
JuliaHub.jobs(; [limit::Integer], [auth::Authentication]) -> Vector{Job}
Retrieve the list of jobs, latest first, visible to the currently authenticated user.
By default, JuliaHub only returns up to 20 jobs. However, this default limit can be overridden by passing the limit keyword (which must be a positive integer).
Non-dynamic job objects
Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.
source
JuliaHub.job — Function
JuliaHub.job(job::JobReference; throw::Bool=true, [auth::Authentication]) -> Job
Fetch the details of a job based on the job reference ref. Will throw an InvalidRequestError if the job does not exist, or returns nothing if throw=false is passed.
Non-dynamic job objects
Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.
source
JuliaHub.isdone — Function
JuliaHub.isdone(::Job)
A helper function to check if a Job is "done", i.e. its status is one of Completed, Stopped, or Failed.
Non-dynamic job objects
Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. As such, the result from this function may not represent the current live state of the job. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.
source
JuliaHub.wait_job — Function
wait_job(
+classDef user fill:lightgray
Refreshing a Job object
An instance of a Job object reflects the state of the job when the job function was called. If you want to inspect the current state of a job, you must first "refresh" the job object, which can simply be done with the help of the job function.
job = JuliaHub.job(job)
Reference
JuliaHub.JobReference — Type
const JobReference :: Type
A type constraint on the arguments of many jobs-related functions that is used to specify the job. A job reference must be either a Job object, or an AbstractString containing the unique job ID.
source
JuliaHub.jobs — Function
JuliaHub.jobs(; [limit::Integer], [auth::Authentication]) -> Vector{Job}
Retrieve the list of jobs, latest first, visible to the currently authenticated user.
By default, JuliaHub only returns up to 20 jobs. However, this default limit can be overridden by passing the limit keyword (which must be a positive integer).
Non-dynamic job objects
Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.
source
JuliaHub.job — Function
JuliaHub.job(job::JobReference; throw::Bool=true, [auth::Authentication]) -> Job
Fetch the details of a job based on the job reference ref. Will throw an InvalidRequestError if the job does not exist, or returns nothing if throw=false is passed.
Non-dynamic job objects
Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.
source
JuliaHub.isdone — Function
JuliaHub.isdone(::Job)
A helper function to check if a Job is "done", i.e. its status is one of Completed, Stopped, or Failed.
Non-dynamic job objects
Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. As such, the result from this function may not represent the current live state of the job. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.
source
JuliaHub.wait_job — Function
wait_job(
     job::AbstractString;
     interval::Integer = 30, [auth::Authentication]
-) -> Job
Blocks until remote job referred to by the job reference job has completed, by polling it with every interval seconds. Returns an updated Job object.
source
JuliaHub.extend_job — Function
JuliaHub.extend_job(job::JobReference, extension::Limit; [auth::Authentication]) -> Job
Extends the time limit of the job referred to by the job reference ref by extension (Dates.Period, or Integer number of hours). Returns an updated Job object.
See Limit for more information on how the extension argument is interpreted. Note that Unlimited is not allowed as extension.
See also: Job.
source
JuliaHub.JobLogMessage — Type
struct JobLogMessage
Contains a single JuliaHub job log message, and has the following fields:
  • timestamp :: Union{ZonedDateTime, Nothing}: log message timestamp (in UTC)
  • message :: Union{String, Nothing}: log message string. This generally corresponds to one line of printed output
Fields that can also be nothing may be missing for some log messages.
See also: job_logs, job_logs_buffered.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.job_logs — Function
JuliaHub.job_logs(job; offset::Integer = 0, [limit::Integer], [auth::Authentication]) -> Vector{JobLogMessage}
Fetches the log messages for the specified JuliaHub job. The job is specifed by passing the job name as a string, or by passing a Job object (i.e. job::Union{AbstractString,Job}). Returns the log messages as an array of JobLogMessage objects.
Optionally, the function takes the following keyword arguments:
  • offset::Integer: the offset of the first log message fetched (0 corresponds to the first message); for the first method, this defaults to 0; however, in the second (callback) case, if offset is not specified, any existing logs will be ignored.
  • limit::Integer: the maximum number of messages fetched (all by default)
No default limit
The limit keyword does not have a default limit, and so by default job_logs fetches all the log messages. This may take a while and require many requests to JuliaHub if the job has a huge number of log messages.
source
JuliaHub.extend_job — Function
JuliaHub.extend_job(job::JobReference, extension::Limit; [auth::Authentication]) -> Job
Extends the time limit of the job referred to by the job reference ref by extension (Dates.Period, or Integer number of hours). Returns an updated Job object.
See Limit for more information on how the extension argument is interpreted. Note that Unlimited is not allowed as extension.
See also: Job.
source
JuliaHub.JobLogMessage — Type
struct JobLogMessage
Contains a single JuliaHub job log message, and has the following fields:
  • timestamp :: Union{ZonedDateTime, Nothing}: log message timestamp (in UTC)
  • message :: Union{String, Nothing}: log message string. This generally corresponds to one line of printed output
Fields that can also be nothing may be missing for some log messages.
See also: job_logs, job_logs_buffered.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.job_logs — Function
JuliaHub.job_logs(job; offset::Integer = 0, [limit::Integer], [auth::Authentication]) -> Vector{JobLogMessage}
Fetches the log messages for the specified JuliaHub job. The job is specifed by passing the job name as a string, or by passing a Job object (i.e. job::Union{AbstractString,Job}). Returns the log messages as an array of JobLogMessage objects.
Optionally, the function takes the following keyword arguments:
  • offset::Integer: the offset of the first log message fetched (0 corresponds to the first message); for the first method, this defaults to 0; however, in the second (callback) case, if offset is not specified, any existing logs will be ignored.
  • limit::Integer: the maximum number of messages fetched (all by default)
No default limit
The limit keyword does not have a default limit, and so by default job_logs fetches all the log messages. This may take a while and require many requests to JuliaHub if the job has a huge number of log messages.
source
JuliaHub.job_logs_buffered — Function
JuliaHub.job_logs_buffered(
     [f::Base.Callable], job::Union{Job,AbstractString};
     streaming::Bool=true, [offset::Integer],
     [auth::Authentication]
-) -> AbstractJobLogsBuffer
A lower-level function to work with log streams, and is particularly useful when working with jobs that have not finished yet and are actively producing new log messages.
The function accepts the following arguments:
  • f :: Base.Callable: an optional callback function that gets called every time the buffer is updated. The callback must take two arguments: f(::AbstractJobLogsBuffer, ::AbstractVector). The first argument is the buffer object itself, and the second argument will be passed a read-only view of all the logs that have been loaded into the buffer, including the new ones.
  • job :: Union{Job,AbstractString}: either the job name or a Job object.
  • streaming :: Bool: if set to true, the buffer object The streaming can be stopped with interrupt!.
  • offset :: Integer: optional non-negative value to specify the starting point of the buffer
Interface of the returned object
Returns an instance of the abstract AbstractJobLogsBuffer type. These objects contain log messages (of type JobLogMessage), but not all the log messages are immediately available. Instead, at any given time the buffer represents a continous section of logs that can be extended in either direction.
The following functions can be used to interact with log buffers: job_logs_newer!, job_logs_older!, JuliaHub.hasfirst, JuliaHub.haslast. Additionally, the objects will have a .logs :: Vector{JobLogMessage} property that can be used to access the log messages that have been loaded into the buffer.
See also: job_logs, Job.
source
JuliaHub.job_logs_older! — Function
JuliaHub.job_logs_older!(
+) -> AbstractJobLogsBuffer
A lower-level function to work with log streams, and is particularly useful when working with jobs that have not finished yet and are actively producing new log messages.
The function accepts the following arguments:
  • f :: Base.Callable: an optional callback function that gets called every time the buffer is updated. The callback must take two arguments: f(::AbstractJobLogsBuffer, ::AbstractVector). The first argument is the buffer object itself, and the second argument will be passed a read-only view of all the logs that have been loaded into the buffer, including the new ones.
  • job :: Union{Job,AbstractString}: either the job name or a Job object.
  • streaming :: Bool: if set to true, the buffer object The streaming can be stopped with interrupt!.
  • offset :: Integer: optional non-negative value to specify the starting point of the buffer
Interface of the returned object
Returns an instance of the abstract AbstractJobLogsBuffer type. These objects contain log messages (of type JobLogMessage), but not all the log messages are immediately available. Instead, at any given time the buffer represents a continous section of logs that can be extended in either direction.
The following functions can be used to interact with log buffers: job_logs_newer!, job_logs_older!, JuliaHub.hasfirst, JuliaHub.haslast. Additionally, the objects will have a .logs :: Vector{JobLogMessage} property that can be used to access the log messages that have been loaded into the buffer.
See also: job_logs, Job.
source
JuliaHub.job_logs_older! — Function
JuliaHub.job_logs_older!(
     buffer::AbstractJobLogsBuffer; [count::Integer], [auth::Authentication]
-) -> AbstractJobLogsBuffer
Updates the AbstractJobLogsBuffer object by adding up to count log messages to the beginning of the buffer. If count is omitted, it will seek all the way to the beginning of the logs.
If all the logs have already been loaded into the buffer (i.e. JuliaHub.hasfirst(buffer) is true), the function is a no-op.
See also: job_logs_buffered, job_logs_newer!.
source
JuliaHub.job_logs_newer! — Function
JuliaHub.job_logs_newer!(
+) -> AbstractJobLogsBuffer
Updates the AbstractJobLogsBuffer object by adding up to count log messages to the beginning of the buffer. If count is omitted, it will seek all the way to the beginning of the logs.
If all the logs have already been loaded into the buffer (i.e. JuliaHub.hasfirst(buffer) is true), the function is a no-op.
See also: job_logs_buffered, job_logs_newer!.
source
JuliaHub.job_logs_newer! — Function
JuliaHub.job_logs_newer!(
     buffer::AbstractJobLogsBuffer; [count::Integer], [auth::Authentication]
-) -> AbstractJobLogsBuffer
Updates the AbstractJobLogsBuffer object by adding up to count log messages to the end of the buffer. If count is omitted, it will seek all the way to the end of the current logs.
For a finished job, if all the logs have already been loaded into the buffer (i.e. JuliaHub.haslast(buffer) is true), the function is a no-op. If the buffer is actively streaming new logs for a running job,  then the function is also a no-op.
See also: job_logs_buffered, job_logs_older!.
source
JuliaHub.haslast — Function
JuliaHub.haslast(::AbstractJobLogsBuffer) -> Bool
Determines whether the job log buffer has the last message of the job logs. Note that if the job has not finished, this will always be false, since the job may produce additional logs.
See also: hasfirst, job_logs_buffered.
source
JuliaHub.interrupt! — Function
JuliaHub.interrupt!(::AbstractJobLogsBuffer; wait::Bool=true)
Can be use to interrupt the asynchronous log streaming task. If the log buffer is not streaming, this function is a no-op.
Note that the effect of JuliaHub.interrupt! may not be immediate and the function will block until the task has stopped. wait = false can be passed to make interrupt! return immediately, but in that case the buffer may stream for a little while longer.
source
JuliaHub.job_files — Function
JuliaHub.job_files(job::Job, [filetype::Symbol]) -> Vector{JobFile}
Return the list of inputs and/or output files associated job.
The optional filetype argument should be one of :input, :source, :result or :project, and can be used to filter the file list down to either just job input files (such as the appbundle or Julia environment files), or output files (such as the one uploaded via RESULTS_FILE).
Note: job_file(job) is equivalent to job.files, and the latter is preferred. This function is primarily meant to be used when filtering by file type.
See also: Job.
source
JuliaHub.job_file — Function
JuliaHub.job_file(job::Job, type::Symbol, filename::AbstractString) -> JobFile | Nothing
Searches for a job output file of a specified type and with the specific filename for job job, or nothing if the file was not found.
type should be one of the standard job file types. See JobFile and job_files for more information.
source
JuliaHub.download_job_file — Function
JuliaHub.download_job_file(file::JobFile, path::AbstractString; [auth]) -> String
+) -> AbstractJobLogsBuffer
Updates the AbstractJobLogsBuffer object by adding up to count log messages to the end of the buffer. If count is omitted, it will seek all the way to the end of the current logs.
For a finished job, if all the logs have already been loaded into the buffer (i.e. JuliaHub.haslast(buffer) is true), the function is a no-op. If the buffer is actively streaming new logs for a running job,  then the function is also a no-op.
See also: job_logs_buffered, job_logs_older!.
source
JuliaHub.haslast — Function
JuliaHub.haslast(::AbstractJobLogsBuffer) -> Bool
Determines whether the job log buffer has the last message of the job logs. Note that if the job has not finished, this will always be false, since the job may produce additional logs.
See also: hasfirst, job_logs_buffered.
source
JuliaHub.interrupt! — Function
JuliaHub.interrupt!(::AbstractJobLogsBuffer; wait::Bool=true)
Can be use to interrupt the asynchronous log streaming task. If the log buffer is not streaming, this function is a no-op.
Note that the effect of JuliaHub.interrupt! may not be immediate and the function will block until the task has stopped. wait = false can be passed to make interrupt! return immediately, but in that case the buffer may stream for a little while longer.
source
JuliaHub.job_files — Function
JuliaHub.job_files(job::Job, [filetype::Symbol]) -> Vector{JobFile}
Return the list of inputs and/or output files associated job.
The optional filetype argument should be one of :input, :source, :result or :project, and can be used to filter the file list down to either just job input files (such as the appbundle or Julia environment files), or output files (such as the one uploaded via RESULTS_FILE).
Note: job_file(job) is equivalent to job.files, and the latter is preferred. This function is primarily meant to be used when filtering by file type.
See also: Job.
source
JuliaHub.job_file — Function
JuliaHub.job_file(job::Job, type::Symbol, filename::AbstractString) -> JobFile | Nothing
Searches for a job output file of a specified type and with the specific filename for job job, or nothing if the file was not found.
type should be one of the standard job file types. See JobFile and job_files for more information.
source
JuliaHub.download_job_file — Function
JuliaHub.download_job_file(file::JobFile, path::AbstractString; [auth]) -> String
 JuliaHub.download_job_file(file::JobFile, io::IO; [auth])
Downloads a JobFile to a local path. Alternative, writeable stream object can be passed as the second argument to write the contents directly into the stream.
When a local path is passed, it returns the path (which can be useful when calling the function as e.g. JuliaHub.download_job_file(file, tempname()))). When an IO object is passed, it returns nothing.
For example, to download a file into a temporary file:
julia> file = JuliaHub.job_file(JuliaHub.job("jr-eezd3arpcj"), :result, "outdir.tar.gz")
 JuliaHub.JobFile outdir.tar.gz (jr-eezd3arpcj, :result, 632143 bytes)
 sha2_256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
@@ -42,7 +42,7 @@
 julia> JuliaHub.download_job_file(file, buffer)
 
 julia> bytes2hex(sha2_256(take!(buffer)))
-"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
See also: Job, JobFile.
source
JuliaHub.Job — Type
struct Job
Represents a single job submitted to JuliaHub. Objects have the following properties:
  • id :: String: the unique, automatically generated ID of the job
  • alias :: String: a non-unique, but descriptive alias for the job (often set by e.g. applications)
  • status :: JobStatus: a string-like JobStatus object storing the state of the job
  • env :: Dict: a dictionary of environment variables that were set when the job was submitted
  • results :: String: the output value set via ENV["RESULTS"] (an empty string if it was not explicitly set)
  • files :: Vector{JobFiles}: a list of JobFile objects, representing the input and output files of the job (see: job_files, job_file, download_job_file).
See also: job, jobs.
Non-dynamic job objects
Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.JobStatus — Type
struct JobStatus
Type of the .status field of a Job object, representing the current state of the job. Should be one of: Submitted, Running, Failed, Stopped, Completed.
In practice, the .status field should be treated as string and only used in string comparisons.
See also: isdone.
julia> job = JuliaHub.job("jr-novcmdtiz6")
+"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
See also: Job, JobFile.
source
JuliaHub.Job — Type
struct Job
Represents a single job submitted to JuliaHub. Objects have the following properties:
  • id :: String: the unique, automatically generated ID of the job
  • alias :: String: a non-unique, but descriptive alias for the job (often set by e.g. applications)
  • status :: JobStatus: a string-like JobStatus object storing the state of the job
  • env :: Dict: a dictionary of environment variables that were set when the job was submitted
  • results :: String: the output value set via ENV["RESULTS"] (an empty string if it was not explicitly set)
  • files :: Vector{JobFiles}: a list of JobFile objects, representing the input and output files of the job (see: job_files, job_file, download_job_file).
See also: job, jobs.
Non-dynamic job objects
Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.JobStatus — Type
struct JobStatus
Type of the .status field of a Job object, representing the current state of the job. Should be one of: Submitted, Running, Failed, Stopped, Completed.
In practice, the .status field should be treated as string and only used in string comparisons.
See also: isdone.
julia> job = JuliaHub.job("jr-novcmdtiz6")
 JuliaHub.Job: jr-novcmdtiz6 (Completed)
  submitted: 2023-03-15T07:56:50.974+00:00
  started:   2023-03-15T07:56:51.251+00:00
@@ -58,9 +58,9 @@
 false
 
 julia> job.status == "Completed"
-true
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.JobFile — Type
struct JobFile
A reference to a job input or output file, with the following properties:
  • .name :: String: the name of the Job this file is attached to
  • .type :: Symbol: indicated the file type (see below)
  • .filename :: String: file name
  • .size :: Int: size of the file in bytes (reported to be zero in cases where the file contents is missing)
  • .hash :: Union{FileHash, Nothing}: a FileHash object containing the file hash, but may also be missing (nothing) in some cases, like when the file upload has not completed yet.
The file is uniquely identified by the (job, type, filename) triplet.
The type of the file should be one of:
  • :input: various job input files, which includes code, environment, and appbundle files.
  • :source: source
  • :project: Julia project environment files (e.g. Project.toml, Manifest.toml, Artifacts.toml)
  • :result: output results file, defined via the RESULTS_FILE environment variable in the job
Note that some files are duplicated under multiple categories.
See also: Job, job_files, job_file.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.FileHash — Type
struct FileHash
Stores a hash and the algorithm used to calcute it. The object has the following properties:
  • .algorithm :: Symbol: hash algorithm
  • .hash :: Vector{UInt8}: the hash as a sequence of bytes
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
Index

+
diff --git a/dev/search_index.js b/dev/search_index.js
index ede03e706..35f85faa4 100644
--- a/dev/search_index.js
+++ b/dev/search_index.js
@@ -1,3 +1,3 @@
 var documenterSearchIndex = {"docs":
-[{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"CurrentModule=JuliaHub\nDocTestSetup = :(using JuliaHub)","category":"page"},{"location":"reference/authentication/#authentication","page":"Authentication","title":"Authentication","text":"","category":"section"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"In order to talk to a JuliaHub instance, you need to have a valid authentication token. JuliaHub reuses the Julia's built-in package server authentication tokens for this purpose. By default, the authentication uses the JULIA_PKG_SERVER environment variable to determine which JuliaHub instance to connect to, but this can be overridden by passing an argument to authenticate.","category":"page"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"The authenticate function can be used to construct a token. If a valid token is available in ~/.julia/servers, it gets reused. Otherwise, a browser window is opened, starting an interactive authentication procedure.","category":"page"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"All the functions that require authentication accept an auth keyword argument. However, JuliaHub.jl also stores the authentication token from the last authenticate call in a global variable and automatically uses that if auth is not provided, and also tries to authenticate automatically. The current global authentication object can be accessed via the current_authentication() function.","category":"page"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"See also: authentication guide, authentication section on help.juliahub.com, PkgAuthentication.","category":"page"},{"location":"reference/authentication/#Token-expiration-and-refresh-tokens","page":"Authentication","title":"Token expiration and refresh tokens","text":"","category":"section"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"By default, JuliaHub access tokens expire in 24 hours. However, the tokens usually also have a refresh token, which is valid for 30 days. If the access token has expired, but there is a valid refresh token available, authenticate will automatically try to use that, to re-acquire an access token without starting an interactive authentication.","category":"page"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"In JuliaHub job and cloud IDE environments, the authentication token on disk will be continuously kept up to date. The reauthenticate! function can be used to reload the token from disk.","category":"page"},{"location":"reference/authentication/#Reference","page":"Authentication","title":"Reference","text":"","category":"section"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"authenticate\nAuthentication\ncurrent_authentication\ncheck_authentication\nreauthenticate!\nSecret","category":"page"},{"location":"reference/authentication/#JuliaHub.authenticate","page":"Authentication","title":"JuliaHub.authenticate","text":"JuliaHub.authenticate(server = Pkg.pkg_server(); force::Bool = false, maxcount::Integer = 3, [hook::Base.Callable])\n\nAuthenticates with a JuliaHub server. If a valid authentication token does not exist in the Julia depot, a new token is acquired via an interactive browser based prompt. Returns an Authentication object if the authentication was successful, or throws an AuthenticationError if authentication fails.\n\nThe interactive prompts tries to authenticate for a maximum of maxcount times. If force is set to true, an existing authentication token is first deleted. This can be useful when the existing authentication token is causing the authentication to fail.\n\nExtended help\n\nBy default, it attemps to connect to the currently configured Julia package server URL (configured e.g. via the JULIA_PKG_SERVER environment variable). However, this can be overridden by passing the server argument.\n\nhook can be set to a function taking a single string-type argument, and will be passed the authorization URL the user should interact with in the browser. This can be used to override the default behavior coming from PkgAuthentication.\n\nThe returned Authentication object is also cached globally (overwriting any previously cached authentications), making it unnecessary to pass the returned object manually to other function calls. This is useful for interactive use, but should not be used in library code, as different authentication calls may clash.\n\n\n\n\n\n","category":"function"},{"location":"reference/authentication/#JuliaHub.Authentication","page":"Authentication","title":"JuliaHub.Authentication","text":"mutable struct Authentication\n\nAuthentication object constructed by the authenticate function that can be passed to the various JuliaHub.jl function via the auth keyword argument.\n\nObjects have the following properties:\n\nserver :: URIs.URI: URL of the JuliaHub instance this authentication token applies to.\nusername :: String: user's JuliaHub username (used for e.g. to namespace datasets)\ntoken :: JuliaHub.Secret: a Secret object storing the JuliaHub authentication token\n\nNote that the object is mutable, and hence will be shared as it is passed around. And at the same time, functions such as reauthenticate! may modify the object.\n\nSee also: authenticate, reauthenticate!, current_authentication.\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/authentication/#JuliaHub.current_authentication","page":"Authentication","title":"JuliaHub.current_authentication","text":"JuliaHub.current_authentication() -> Union{Authentication, Nothing}\n\nReturns the current globally active Authentication object, or nothing if authenticate has not yet been called.\n\njulia> JuliaHub.current_authentication()\nJuliaHub.Authentication(\"https://juliahub.com\", \"username\", *****)\n\nnote: Note\nCalling this function will not initialize authentication.\n\n\n\n\n\n","category":"function"},{"location":"reference/authentication/#JuliaHub.check_authentication","page":"Authentication","title":"JuliaHub.check_authentication","text":"JuliaHub.check_authentication(; [auth::Authentication]) -> Bool\n\nChecks if the authentication to a JuliaHub instance is still valid or not.\n\nThis can be used to periodically check an authentication token, to see if it is necessary to re-authenticate.\n\nSee also: reauthenticate!.\n\n\n\n\n\n","category":"function"},{"location":"reference/authentication/#JuliaHub.reauthenticate!","page":"Authentication","title":"JuliaHub.reauthenticate!","text":"JuliaHub.reauthenticate!([auth::Authentication]; force::Bool = false, maxcount::Integer = 3, [hook::Base.Callable])\n\nAttempts to update the authentication token in auth:\n\nIf the original auth.toml file has been updated, it simply reloads the token from the file.\nIf loading from auth.toml fails or force=true, it will attempt to re-authenticate with the server, possibly interactively.\n\nIf auth is omitted, it will reauthenticate the global Authentication object. The force, maxcount and hook are relevant for interactive authentication, and behave the same way as in the authenticate function.\n\nThis is mostly meant to be used to re-acquire authentication tokens in long-running sessions, where the initial authentication token may have expired.\n\nAs Authentication objects are mutable, the token will be updated in all contexts where the reference to the Authentication has been passed to.\n\nSee also: authenticate, current_authentication, Authentication, check_authentication.\n\n\n\n\n\n","category":"function"},{"location":"reference/authentication/#JuliaHub.Secret","page":"Authentication","title":"JuliaHub.Secret","text":"mutable struct Secret\n\nA helper type for storing secrets. Internally it is a covenience wrapper around Base.SecretBuffer. Predominantly used in Authentication objects to store the JuliaHub authentication token.\n\nThe String(::Secret) function can be used to obtain an unsecure string copy of the secret stored in the object.\n\njulia> s = JuliaHub.Secret(\"secret-string\")\nJuliaHub.Secret(\"*******\")\n\njulia> String(s)\n\"secret-string\"\n\nConstructors\n\nSecret(::AbstractString)\nSecret(::Vector{UInt8})\n\nCreate a Secret object from the input strings.\n\n\n\n\n\n","category":"type"},{"location":"reference/authentication/#Index","page":"Authentication","title":"Index","text":"","category":"section"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"Pages = [\"authentication.md\"]","category":"page"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"CurrentModule=JuliaHub\nDocTestSetup = :(using JuliaHub)","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"using JuliaHub","category":"page"},{"location":"guides/datasets/#guide-datasets","page":"Datasets","title":"Datasets","text":"","category":"section"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.jl offers a programmatic way to work with your JuliaHub datasets, and this section demonstrates a few common workflows you can use with these APIs.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"Pages = [\"datasets.md\"]\nDepth = 2:10","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"See the datasets reference page for a detailed reference of the datasets-related functionality.","category":"page"},{"location":"guides/datasets/#Accessing-datasets","page":"Datasets","title":"Accessing datasets","text":"","category":"section"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"The datasets function can be use to list all the datasets owned by the currently authenticated user, returning an array of Dataset objects.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"julia> JuliaHub.datasets()\n2-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"username\", \"example-dataset\"))\n JuliaHub.dataset((\"username\", \"blobtree/example\"))","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"If you know the name of the dataset, you can also directly access it with the dataset function, and you can access the dataset metadata via the properties of the Dataset object.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"julia> ds = JuliaHub.dataset(\"example-dataset\")\nDataset: example-dataset (Blob)\n owner: username\n description: An example dataset\n versions: 2\n size: 388 bytes\n tags: tag1, tag2\n\njulia> ds.owner\n\"username\"\n\njulia> ds.description\n\"An example dataset\"\n\njulia> ds.size\n388","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"If you want to work with dataset that you do not own but is shared with you in JuliaHub, you can pass shared=true to datasets, or specify the username.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"julia> JuliaHub.datasets(shared=true)\n3-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"username\", \"example-dataset\"))\n JuliaHub.dataset((\"anotheruser\", \"publicdataset\"))\n JuliaHub.dataset((\"username\", \"blobtree/example\"))\n\njulia> JuliaHub.datasets(\"anotheruser\")\n1-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"anotheruser\", \"publicdataset\"))\n\njulia> JuliaHub.dataset((\"anotheruser\", \"publicdataset\"))\nDataset: publicdataset (Blob)\n owner: anotheruser\n description: An example dataset\n versions: 1\n size: 57 bytes\n tags: tag1, tag2","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"Finally, JuliaHub.jl can also be used to download to your local machine with the download_dataset function.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"julia> JuliaHub.download_dataset(\"example-dataset\", \"mydata\")\nTransferred:       86.767 KiB / 86.767 KiB, 100%, 0 B/s, ETA -\nTransferred:            1 / 1, 100%\nElapsed time:         2.1s\n\"/home/username/my-project/mydata\"","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"As datasets can have multiple versions, the .versions property of Dataset can be used to see information about the individual versions (represented with DatasetVersion objects). When downloading, you can also specify the version you wish to download (with the default being the newest version).","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"julia> ds.versions\n2-element Vector{JuliaHub.DatasetVersion}:\n JuliaHub.DatasetVersion(dataset = (\"username\", \"example-dataset\"), version = 1)\n JuliaHub.DatasetVersion(dataset = (\"username\", \"example-dataset\"), version = 2)\n\njulia> ds.versions[1]\nDatasetVersion: example-dataset @ v1\n owner: username\n timestamp: 2022-10-13T01:39:42.963-04:00\n size: 57 bytes\n\njulia> JuliaHub.download_dataset(\"example-dataset\", \"mydata\", version=ds.versions[1].id)\nTransferred:       86.767 KiB / 86.767 KiB, 100%, 0 B/s, ETA -\nTransferred:            1 / 1, 100%\nElapsed time:         2.1s\n\"/home/username/my-project/mydata\"\n","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"The dataset version are sorted with oldest first. To explicitly access the newest dataset, you can use the last function on the .versions property.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"julia> last(ds.versions)\nDatasetVersion: example-dataset @ v2\n owner: username\n timestamp: 2022-10-14T01:39:43.237-04:00\n size: 331 bytes\n","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"tip: Tip: DataSets.jl\nIn JuliaHub jobs and Cloud IDEs you can also use the DataSets.jl package to access and work with datasets. See the help.julialang.org section on datasets for more information.","category":"page"},{"location":"guides/datasets/#Create,-update,-or-replace","page":"Datasets","title":"Create, update, or replace","text":"","category":"section"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"The upload_dataset function can be used to programmatically create new datasets on JuliaHub.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"touch(\"local-file\")\nmkdir(\"local-directory\")\nMain.MOCK_JULIAHUB_STATE[:existing_datasets] = []","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.upload_dataset(\"example-dataset\", \"local-file\")","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"The type of the dataset (Blob or BlobTree) depends on whether the uploaded object is a file or a directory. A directory will be stored as a BlobTree-type dataset on JuliaHub.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.upload_dataset(\"example-blobtree\", \"local-directory\")","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"The create, update, and replace options control how upload_dataset behaves with respect to existing datasets. By default, the function only creates brand new datasets, and trying to upload a dataset that already exists will fail with an error.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.upload_dataset(\"example-dataset\", \"local-file\")","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"This behavior can be overridden by setting update=true, which will then upload a new version of a dataset if it already exists. This is useful for jobs and workflows that are meant to be re-run, updating the dataset each time they run.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.upload_dataset(\"example-dataset\", \"local-file\"; update=true)","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"The replace=true option can be used to erase earlier versions of a dataset. This will delete all information about the existing dataset and is a destructive, non-recoverable action. This may also lead to the dataset type being changed.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.upload_dataset(\"example-dataset\", \"local-file\"; replace=true)","category":"page"},{"location":"guides/datasets/#Bulk-updates","page":"Datasets","title":"Bulk updates","text":"","category":"section"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"You can also use the package to perform bulk updates or deletions of datasets. The following example, adds a new tag to all the datasets where the name matches a particular pattern.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"# Find all the datasets that have names that start with 'my-analysis-'\nmyanalysis_datasets = filter(\n    dataset -> startswith(dataset.name, r\"my-analysis-.*\"),\n    JuliaHub.datasets()\n)\n# .. and now add a 'new-tag' tag to each of them\nfor dataset in myanalysis_datasets\n    @info \"Updating\" dataset\n    # Note: tags = ... overrides the whole list, so you need to manually retain\n    # old tags.\n    new_tags = [dataset.tags..., \"new-tag\"]\n    JuliaHub.update_dataset(dataset, tags = new_tags)\nend","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"While this example shows the update_dataset, for example, the delete_dataset function could be used in the same way.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"CurrentModule=JuliaHub\nDocTestSetup = :(using JuliaHub)","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"using JuliaHub","category":"page"},{"location":"reference/datasets/#datasets","page":"Datasets","title":"Datasets","text":"","category":"section"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"These APIs allow you to create, read, update, and delete datasets owned by the currently authenticated user.","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"You can use datasets, dataset, and download_dataset to access datasets or their metadata.\nupload_dataset, update_dataset, and delete_dataset can be used to create, update, or delete datasets.","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"See also: help.julialang.org on datasets, DataSets.jl.","category":"page"},{"location":"reference/datasets/#Dataset-types","page":"Datasets","title":"Dataset types","text":"","category":"section"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub currently has two distinct types of datasets:","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"Blob: a single file; or, more abstractly, a collection of bytes\nBlobTree: a directory or a file; more abstractly a tree-like collection of Blobs, indexed by file system paths","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"These types mirror the concepts in DataSets.jl","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.jl APIs do not rely that much on the dataset type for anything, except when downloading or uploading. In that case, a local file always corresponds to a Blob, and a local directory corresponds to a BlobTree. For example, when trying to upload a file as a new version of a BlobTree-type dataset will fail, because the dataset type can not change.","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"The upload_dataset function uses information filesystem to determine whether the created dataset is a Blob or a BlobTree, and similarly download_dataset will always download a Blob into a file, and a BlobTree as a directory.","category":"page"},{"location":"reference/datasets/#Dataset-versions","page":"Datasets","title":"Dataset versions","text":"","category":"section"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"A JuliaHub dataset can have zero or more versions. A newly created dataset usually has at least one version, but it may have zero versions if, for example, the upload did not finish. The versions are indexed with a linear list of integers starting from 1.","category":"page"},{"location":"reference/datasets/#Reference","page":"Datasets","title":"Reference","text":"","category":"section"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.Dataset\nJuliaHub.DatasetVersion\nJuliaHub.datasets\nJuliaHub.DatasetReference\nJuliaHub.dataset\nJuliaHub.download_dataset\nJuliaHub.upload_dataset\nJuliaHub.update_dataset\nJuliaHub.delete_dataset","category":"page"},{"location":"reference/datasets/#JuliaHub.Dataset","page":"Datasets","title":"JuliaHub.Dataset","text":"struct Dataset\n\nInformation about a dataset stored on JuliaHub, and the following fields are considered to be public API:\n\nuuid :: UUID: dataset UUID\nowner :: String: username of the dataset owner\nname :: String: dataset name\ndtype :: String: generally either Blob or BlobTree, but additional values may be added in the future\nversions :: Vector{DatasetVersion}: an ordered list of DatasetVersion objects, one for each dataset version, sorted from oldest to latest (i.e. you can use last to get the newest version).\nsize :: Int: total size of the whole dataset (including all the dataset versions) in bytes\nFields to access user-provided dataset metadata:\ndescription :: String: dataset description\ntags :: Vector{String}: a list of tags\n\nnote: Canonical fully qualified dataset name\nIn some contexts, like when accessing JuliaHub datasets with DataSets.jl, the .owner-.name tuple constitutes the fully qualifed dataset name, uniquely identifying a dataset on a JuliaHub instance. I.e. for a dataset object dataset, it can be constructed as \"$(dataset.owner)/$(dataset.name)\".\n\nwarning: Non-dynamic dataset objects\nDataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/datasets/#JuliaHub.DatasetVersion","page":"Datasets","title":"JuliaHub.DatasetVersion","text":"struct DatasetVersion\n\nRepresents one version of a dataset.\n\nObjects have the following properties:\n\n.id: unique dataset version identifier (used e.g. in download_dataset to identify the dataset version).\n.size :: Int: size of the dataset version in bytes\n.timestamp :: ZonedDateTime: dataset version timestamp\n\njulia> JuliaHub.datasets()\n\nSee also: Dataset, datasets, dataset.\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/datasets/#JuliaHub.datasets","page":"Datasets","title":"JuliaHub.datasets","text":"JuliaHub.datasets([username::AbstractString]; shared::Bool=false, [auth::Authentication]) -> Vector{Dataset}\n\nList all datasets owned by username, returning a list of Dataset objects.\n\nIf username is omitted, it returns the datasets owned by the currently authenticated user. If username is different from the currently authenticated user, it only returns the datasets that are readable to (i.e. somehow shared with) the currently authenticated user.\n\nIf shared = true, it also returns datasets that belong to other users that have that have been shared with the currently authenticated user. In this case, username is effectively ignored.\n\njulia> JuliaHub.datasets()\n2-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"username\", \"example-dataset\"))\n JuliaHub.dataset((\"username\", \"blobtree/example\"))\n\njulia> JuliaHub.datasets(shared=true)\n3-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"username\", \"example-dataset\"))\n JuliaHub.dataset((\"anotheruser\", \"publicdataset\"))\n JuliaHub.dataset((\"username\", \"blobtree/example\"))\n\njulia> JuliaHub.datasets(\"anotheruser\")\n1-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"anotheruser\", \"publicdataset\"))\n\nwarning: Non-dynamic dataset objects\nDataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.\n\n\n\n\n\n","category":"function"},{"location":"reference/datasets/#JuliaHub.DatasetReference","page":"Datasets","title":"JuliaHub.DatasetReference","text":"const DatasetReference :: Type\n\nType constraint on the first argument of most of the datasets-related functions, that is used to uniquely specify the dataset that the operation will affect.\n\nThere are three different objects that can be passed as a dataset reference (dsref::DatasetReference):\n\n(owner::AbstractString, dataset_name::AbstractString)::Tuple{AbstractString,AbstractString}\nA tuple of the owner's username and the dataset's name.\ndataset_name::AbstractString\nJust a string with the dataset name; in this case the dataset's owner will be assumed to be the currently authenticated user (with the username determined from the Authentication objects passed via the auth keyword).\ndataset::Dataset\nUses the owner and dataset name information from a Dataset object.\n\nwarning: No UUID mismatch checks\nWhen using the third option (i.e. passing a Dataset), the dataset UUID will not be checked. So if the dataset with the same owner and username has been deleted and re-created as a new dataset (potentially of a different dtype etc), the functions will then act on the new dataset.\n\n\n\n\n\n","category":"type"},{"location":"reference/datasets/#JuliaHub.dataset","page":"Datasets","title":"JuliaHub.dataset","text":"JuliaHub.dataset(dataset::DatasetReference; throw::Bool=true, [auth::Authentication]) -> Dataset\n\nLooks up a dataset based on the dataset reference dataset. Returns the Dataset object corresponding to dataset_name, or throws a InvalidRequestError if the dataset can not be found (if throw=false is passed, returns nothing instead).\n\nBy passing a Dataset object as dataset, this can be used to update the Dataset object.\n\njulia> dataset = JuliaHub.dataset(\"example-dataset\")\nDataset: example-dataset (Blob)\n owner: username\n description: An example dataset\n versions: 2\n size: 388 bytes\n tags: tag1, tag2\n\njulia> JuliaHub.dataset(dataset)\nDataset: example-dataset (Blob)\n owner: username\n description: An example dataset\n versions: 2\n size: 388 bytes\n tags: tag1, tag2\n\nIf the specifed username is not the currently authenticated user, the dataset must be shared with the currently authenticated user (i.e. contained in datasets(; shared=true)).\n\nnote: Note\nThis will call datasets every time, which might become a problem if you are processing a large number of datasets. In that case, you should call datasets and process the returned list yourself.\n\nwarning: Non-dynamic dataset objects\nDataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.\n\n\n\n\n\n","category":"function"},{"location":"reference/datasets/#JuliaHub.download_dataset","page":"Datasets","title":"JuliaHub.download_dataset","text":"download_dataset(\n    dataset::DatasetReference, local_path::AbstractString;\n    replace::Bool = false, [version::Integer],\n    [quiet::Bool = false], [auth::Authentication]\n) -> String\n\nDownloads the dataset specified by the dataset reference dataset to local_path (which must not exist, unless replace = true), returning the absolute path to the downloaded file or directory. If the dataset is a Blob, then the created local_path will be a file, and if the dataset is a BlobTree the local_path will be a directory.\n\nBy default, it downloads the latest version, but an older version can be downloaded by specifying the version keyword argument. Caution: you should never assume that the index of the .versions property of Dataset matches the version number – always explicitly use the .id propert of the DatasetVersion object.\n\nThe function also prints download progress to standard output. This can be disabled by setting quiet=true. Any error output from the download is still printed.\n\nwarning: Warning\nSetting replace = true will recursively erase any existing data at local_path before replacing it with the dataset contents.\n\n\n\n\n\n","category":"function"},{"location":"reference/datasets/#JuliaHub.upload_dataset","page":"Datasets","title":"JuliaHub.upload_dataset","text":"JuliaHub.upload_dataset(dataset::DatasetReference, local_path; [auth,] kwargs...) -> Dataset\n\nUploads a new dataset or a new version of an existing dataset, with the dataset specified by the dataset reference dataset. The dataset type is determined from the local path (Blob if a file, BlobTree if a directory). If a Dataset object is passed, it attempts to update that dataset. Returns an updated Dataset object.\n\nThe following keyword arguments can be used to control the exact behavior of the function:\n\ncreate :: Bool (default: true): Create the dataset, if it already does not exist.\nupdate :: Bool (default: false): Upload the data as a new dataset version, if the dataset exists.\nreplace :: Bool (default: false): If a dataset exists, delete all existing data and create a new dataset with the same name instead. Excludes update = true, and only creates a completely new dataset if create=true as well.\n\nIn addition, the following keyword arguments can be passed to set or updated the dataset metadata when uploading:\n\ndescription: description of the dataset (a string)\ntags: an iterable of strings of all the tags of the dataset\nvisibility: a string with possible values public or private\nlicense: a valid SPDX license identifier, or a tuple (:fulltext, license_text), where license_text is the full text string of a custom license\ngroups: an iterable of valid group names\n\nIf a dataset already exists, then these fields are updated as if update_dataset was called.\n\nThe function will throw an ArgumentError for invalid argument combinations.\n\nUse the progress keyword argument to suppress upload progress from being printed.\n\nnote: Note\nPresently, it is only possible to upload datasets for the currently authenticated user.\n\n\n\n\n\n","category":"function"},{"location":"reference/datasets/#JuliaHub.update_dataset","page":"Datasets","title":"JuliaHub.update_dataset","text":"JuliaHub.update_dataset(dataset::DatasetReference; kwargs..., [auth]) -> Dataset\n\nUpdates the metadata of the dataset specified by the dataset reference dataset, as according to the keyword arguments keyword arguments. If the keywords are omitted, the metadata corresponding to it remains unchanged. Returns the Dataset object corresponding to the updated dataset.\n\nThe supported keywords are:\n\ndescription: description of the dataset (a string)\ntags: an iterable of strings of all the tags of the dataset\nvisibility: a string with possible values public or private\nlicense: a valid SPDX license identifier, or a tuple (:fulltext, license_text), where license_text is the full text string of a custom license\ngroups: an iterable of valid group names\n\nFor example, to add a new tag to a dataset:\n\ndataset = JuliaHub.dataset(\"my_dataset\")\nJuliaHub.update(dataset; tags = [dataset.tags..., \"newtag\"])\n\nnote: Note\nPresently, it is only possible to update datasets for the currently authenticated user.\n\n\n\n\n\n","category":"function"},{"location":"reference/datasets/#JuliaHub.delete_dataset","page":"Datasets","title":"JuliaHub.delete_dataset","text":"JuliaHub.delete_dataset(dataset::DatasetReference; force::Bool=false, [auth::Authentication]) -> Nothing\n\nDelete the dataset specified by the dataset reference dataset. Will return nothing if the delete was successful, or throws an error if it was not.\n\nNormally, when the dataset to be deleted does not exist, the function throws an error. This can be overridden by setting force = true.\n\njulia> JuliaHub.datasets()\n2-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"username\", \"example-dataset\"))\n JuliaHub.dataset((\"username\", \"blobtree\"))\n\njulia> JuliaHub.delete_dataset(\"example-dataset\")\n\njulia> JuliaHub.datasets()\n1-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"username\", \"blobtree\"))\n\nnote: Note\nPresently, it is only possible to delete datasets for the currently authenticated user.\n\n\n\n\n\n","category":"function"},{"location":"reference/datasets/#Index","page":"Datasets","title":"Index","text":"","category":"section"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"Pages = [\"datasets.md\"]","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"CurrentModule=JuliaHub","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"using JuliaHub\nENV[\"JULIA_PKG_SERVER\"] = \"juliahub.com\"","category":"page"},{"location":"getting-started/#getting-started","page":"Getting Started","title":"Getting Started with JuliaHub.jl","text":"","category":"section"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"This tutorial walks you through the basic operations you can do with JuliaHub.jl, from installation to submitting simple jobs and working with datasets. If you are unfamiliar with JuliaHub.jl, this is a good place to get started.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"If you already know what you wish to achieve with JuliaHub.jl, you can also skip this and jump directly into one of the more detailed how-to guides.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"In particular, the tutorial will show","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"How to install JuliaHub.jl and connect it to a JuliaHub instance.\nHow to create, access and update a simple dataset.\nHow to submit a simple job.","category":"page"},{"location":"getting-started/#Installation","page":"Getting Started","title":"Installation","text":"","category":"section"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.jl is a registered Julia package and can be installed using Julia's package manager. You can access the Julia package manager REPL mode by pressing ], and you can install JuliaHub.jl with","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"pkg> add JuliaHub","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Alternatively, you can use the Pkg standard library functions to install it.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"import Pkg\nPkg.add(\"JuliaHub\")","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Once it is installed, simply use import or using to load JuliaHub.jl into your current Julia session.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"julia> using JuliaHub","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"note: No exported names\nJuliaHub.jl does not have any exported names, so doing using JuliaHub does not introduce any functions or types in Main. Instead, JuliaHub.jl functions are designed to be used by prefixing them with JuliaHub. (e.g. JuliaHub.authenticate(...) or JuliaHub.submit_job(...))That said, there is nothing stopping you from explicitly bringing some names into your current scope, by doing e.g. using JuliaHub: submit_job, if you so wish!","category":"page"},{"location":"getting-started/#Authentication","page":"Getting Started","title":"Authentication","text":"","category":"section"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"In order to communicate with a JuliaHub instance, you need a valid authentication token. If you are working in a JuliaHub Cloud IDE, you actually do not need to do anything to be authenticated, as the authentication tokens are automatically set up in the cloud environment. To verify this, you can still call authenticate, which should load the pre-configured token.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.authenticate()","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"If you are working on a local computer, the easiest way to get started is to pass the URL of the JuliaHub instance to authenticate. Unless you have authenticated before, this will initiate an interactive browser-based authentication.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"julia> JuliaHub.authenticate(\"juliahub.com\")\nAuthentication required: please authenticate in browser.\nThe authentication page should open in your browser automatically, but you may need to switch to the opened window or tab. If the authentication page is not automatically opened, you can authenticate by manually opening the following URL: ...","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Once you have completed the steps in the browser, the function should return a valid authentication token.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"The authenticate function returns an Authentication object, which hold the authentication token. In principle, you can pass these objects directly to JuliaHub.jl function via the auth keyword argument. However, in practice, this is usually not needed, because JuliaHub.jl also remembers the last authentication in the Julia session in a global variable. You can see the current globally stored authentication token with current_authentication.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.current_authentication()","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"note: Authentication guide\nThere is more to authentication than this, including its relationship to the Julia package server and JULIA_PKG_SERVER environment variable. See the Authentication how-to if you want to learn more.","category":"page"},{"location":"getting-started/#Creating-and-accessing-datasets","page":"Getting Started","title":"Creating & accessing datasets","text":"","category":"section"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.jl allows you to create, access, and update the datasets that are hosted on JuliaHub. This section shows some of the basic operations you can perform with datasets.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Main.MOCK_JULIAHUB_STATE[:existing_datasets] = String[]\nMain.MOCK_JULIAHUB_STATE[:dataset_params] = Dict(\n    \"description\" => \"\",\n    \"tags\" => String[],\n)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"The datasets function allows you to list the datasets you have. Optionally, you can also make it show any other datasets you have access to.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.datasets()","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Unless you have created datasets in the web UI or in the IDE, this list will likely be empty currently. To fix that, let us upload a simple dataset using JuliaHub.jl.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Just as an example, we'll generate a simple 5-by-5 matrix, and save it in a file using the using the DelimitedFiles standard library.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"using DelimitedFiles\nmat = [i^2 + j^2 for i=1:5, j=1:5]\nwritedlm(\"matrix.dat\", mat)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Now that the matrix has been serialized into a text file on the disk, we can upload that file to JuliaHub with upload_dataset.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.upload_dataset(\"tutorial-matrix\", \"matrix.dat\")","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"warning: Existing dataset\nIf you already happen to have a dataset with the same name, the upload_dataset call will fail. It is designed to be safe by default. However, you can pass update=true or replace=true to either upload your file as a new version of the dataset, or to delete all existing versions and upload a brand new version.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"If we now call datasets, it should show up in the list of datasets.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.datasets()","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"To see more details about the dataset, you can index into the array returned by datasets. Alternatively, you can also use the dataset function to pick out a single dataset by its name.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.dataset(\"tutorial-matrix\")","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub datasets also support basic metadata, such as tags and a description field. You could set it directly in the upload_dataset function, but we did not. But that is fine, since we can use update_dataset to update the metadata at any time.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.update_dataset(\"tutorial-matrix\", description=\"An i^2 + j^2 matrix\")","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"The function also immediately queries JuliaHub for the updated dataset metadata by internally calling JuliaHub.dataset(\"tutorial-matrix\").","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Finally, JuliaHub.jl also allows you to download the datasets you have with the download_dataset function. We can also imagine doing this on a different computer or in a JuliaHub job.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.download_dataset(\"tutorial-matrix\", \"matrix-downloaded.dat\")","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"cp(\"matrix.dat\", \"matrix-downloaded.dat\")","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"This downloads the dataset into a local file, after which you can e.g. read it back into Julia and do operations on it.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"mat = readdlm(\"matrix-downloaded.dat\", '\\t', Int)\nsum(mat)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"tip: Directories as datasets\nWhile this demo uploaded a single file as a dataset, JuliaHub also supports uploading whole directories as a single dataset. For that, you can simply point upload_dataset to a directory, rather than a file. See the datasets how-to for more information on how to work with datasets.","category":"page"},{"location":"getting-started/#Submitting-a-job","page":"Getting Started","title":"Submitting a job","text":"","category":"section"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.jl allows for an easy programmatic submission of JuliaHub jobs. In this example, we submit a simple script that downloads the dataset from the previous step, does a simple calculations and then upload the result. We then access the result locally with JuliaHub.jl.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"First, we need to specify the code that we want to run in the job. There are a few options for this, but in this example we use the @script_str string macro to construct a script-type computation, that simply runs the code snippet we specify.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"The following script will access the dataset, calculates the sum of all the elements, and stores the value in the job results. You will be able to access the contents of RESULTS in both the web UI, but also via JuliaHub.jl.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"s = JuliaHub.script\"\"\"\nusing JuliaHub, DelimitedFiles\n@info JuliaHub.authenticate()\nJuliaHub.download_dataset(\"tutorial-matrix\", \"matrix-downloaded.dat\")\nmat = readdlm(\"matrix-downloaded.dat\", '\\t', Int)\nmat_sum = @show sum(mat)\nENV[\"RESULTS\"] = string(mat_sum)\n\"\"\"","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"note: Job environment\nIn most cases, you also submit a Julia package environment (i.e. Project.toml and Manifest.toml files together with a job). That environment then gets instantiated before the user-provided code is run.The script\"\" string macro, by default, attaches the currently active environment to the job. This means that any packages that you are currently using should also be available on the job (although only registered packages added as non-development dependencies will work). You can use Base.active_project() or pkg> status to see what environment is currently active.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"To submit a job, you can simply call submit_job on it.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Main.MOCK_JULIAHUB_STATE[:jobs] = Dict(\n    \"jr-xf4tslavut\" => Dict(\n        \"status\" => \"Submitted\",\n        \"files\" => [],\n        \"outputs\" => \"\",\n    )\n)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"j = JuliaHub.submit_job(s)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"The submit_job function also allows you to specify configure how the job gets run, such as how many CPUs or how much memory it has available. By default, though, it runs your code on a single node, picking the smallest instance that is available.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"At this point, if you go to the \"Jobs\" page web UI, you should see the job there. It may take a few moments to actually start running. You can also call job on the returned Job object to refresh the status of the job.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Main.MOCK_JULIAHUB_STATE[:jobs] = Dict(\n    \"jr-xf4tslavut\" => Dict(\n        \"status\" => \"Running\",\n        \"files\" => [],\n        \"outputs\" => \"\",\n    )\n)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"j = JuliaHub.job(j)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Finally, after the job has completed, if you refresh the Job it should reflect the final status of the job, and also give you access to the","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Main.MOCK_JULIAHUB_STATE[:jobs] = Dict(\n    \"jr-xf4tslavut\" => Dict(\n        \"status\" => \"Completed\",\n        \"files\" => [],\n        \"outputs\" => \"550\",\n    )\n)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"j = JuliaHub.job(j)\nj.results","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"See the jobs how-to guide for more details on the different options when it comes to job submission.","category":"page"},{"location":"getting-started/#Next-steps","page":"Getting Started","title":"Next steps","text":"","category":"section"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"This tutorial has hopefully given an overview of basic JuliaHub.jl usage. For more advanced usage, you may want to read through the more detailed how-to guides.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Pages = Main.PAGES_GUIDES\nDepth = 1:1","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"# Clear the mocking state, so that other pages and docstrings would not\n# be affected.\nempty!(Main.MOCK_JULIAHUB_STATE)\n# We don't actually care about the generated files\nrm(\"matrix.dat\", force=true)\nrm(\"matrix-downloaded.dat\", force=true)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"CurrentModule=JuliaHub\nDocTestSetup = :(using JuliaHub)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"using JuliaHub","category":"page"},{"location":"guides/jobs/#Jobs","page":"Jobs","title":"Jobs","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"JuliaHub.jl can be used to both submit new jobs, and to inspect running or finished jobs.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Pages = [\"jobs.md\"]\nDepth = 2:10","category":"page"},{"location":"guides/jobs/#jobs-guide-batch","page":"Jobs","title":"Submitting batch jobs","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"A common use case for this package is to programmatically submit Julia scripts as batch jobs to JuliaHub, to start non-interactive workloads. In a nutshell, these are Julia scripts, together with an optional Julia environment, that get executed on the allocated hardware.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The easiest way to start a batch job is to submit a single Julia script, which can optionally also include a Julia environment with the job. However, for more complex jobs, with multiple inputs files etc., appbundles are likely more suitable.","category":"page"},{"location":"guides/jobs/#jobs-batch-script","page":"Jobs","title":"Script jobs","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"import JuliaHub\n_temp_path = mktempdir()\ncd(_temp_path)\nwrite(\"myscript.jl\", \"@warn \\\"Hello World!\\\"\\n\")","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The simplest job one can submit is a humble Julia script, together with an optional Julia environment (i.e. Project.toml, Manifest.toml, and/or Artifacts.toml). These jobs can be created with the JuliaHub.@script_str string macro, for inline instantiation:","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"JuliaHub.submit_job(\n    JuliaHub.script\"\"\"\n    @warn \"Hello World!\"\n    \"\"\",\n)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Alternatively, they can be created with the script function, which can load the Julia code from a script file:","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"cd(_temp_path) do # hide\nJuliaHub.submit_job(\n    JuliaHub.script(\"myscript.jl\"),\n)\nend # hide","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The string macro also picks up the currently running environment (i.e. Project.toml, Manifest.toml, and Artifacts.toml files), which then gets instantiated on JuliaHub when the script is started. If necessary, this can be disabled by appending the noenv suffix to the string macro.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"JuliaHub.script\"\"\"\n@warn \"Hello World!\"\n\"\"\"noenv","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"With the script function, you can also specify a path to directory containing the Julia package environment, if necessary.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"If an environment is passed with the job, it gets instantiated on the JuliaHub node, and the script is run in that environment. As such, any packages that are not available in the package registries or added via public Git URLs will not work. If that is the case, appbundles can be used instead to submit jobs that include private or local dependencies.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"rm(_temp_path, recursive=true)","category":"page"},{"location":"guides/jobs/#jobs-batch-appbundles","page":"Jobs","title":"Appbundles","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"A more advanced way of submitting a batch job is as an appbundle, which \"bundles up\" a whole directory and submits it together with the script. The Julia environment in the directory is also immediately added into the bundle.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"An appbundle can be constructed with the appbundle function, which takes as arguments the path to the directory to be bundled up, and a script within that directory. This is meant to be used for project directories where you have your Julia environment in the top level of the directory or repository. For example, you can submit an bundle from a submit script on the top level of your project directory as follows:","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"import JuliaHub # hide\n# We need to override the @__DIR__ here, because we actually construct a real appbundle # hide\nmacro __DIR__(); joinpath(dirname(dirname(pathof(JuliaHub))), \"test\", \"jobenvs\", \"job1\"); end # hide\nJuliaHub.submit_job(\n    JuliaHub.appbundle(@__DIR__, \"script.jl\"),\n    ncpu = 4, memory = 16,\n)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The bundler looks for a Julia environment (i.e. Project.toml, Manifest.toml, and/or Artifacts.toml files) at the root of the directory. If the environment does not exist (i.e. the files are missing), one is created. When the job starts on JuliaHub, this environment is instantiated.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"A key feature of the appbundle is that development dependencies of the environment (i.e. packages added with pkg> develop or Pkg.develop()) are also bundled up into the archive that gets submitted to JuliaHub (including any current, uncommitted changes). Registered packages are installed via the package manager via the standard environment instantiation, and their source code is not included in the bundle directly.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"When the JuliaHub job starts, the bundle is unpacked into the appbundle/ directory (relative to the starting working directory). E.g. if you have a mydata.dat file in the bundled directory, you can access it in the script at joinpath(\"appbundle\", \"mydata.dat\").","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Finally, a .juliabundleignore file can be used to exclude certain directories, by adding the relevant globs, similar to how .gitignore files work. In addition, .git directories are also automatically excluded from the bundle.","category":"page"},{"location":"guides/jobs/#Examining-job-configuration","page":"Jobs","title":"Examining job configuration","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The dryrun option to submit_job can be used to inspect the full job workload configuration that would be submitted to JuliaHub.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"import JuliaHub # hide\nJuliaHub.submit_job(\n    JuliaHub.script\"\"\"\n    println(\"hello world\")\n    \"\"\",\n    ncpu = 4, memory = 8,\n    env = Dict(\"ARG\" => \"value\"),\n    dryrun = true\n)","category":"page"},{"location":"guides/jobs/#Query,-extend,-kill","page":"Jobs","title":"Query, extend, kill","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The package has function that can be used to interact with running and past jobs. The jobs function can be used to list jobs, returning an array of Job objects.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"js = JuliaHub.jobs(limit=3)\njs[1]","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"If you know the name of the job, you can also query the job directly with job.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Main.setup_job_results_file!()","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"job = JuliaHub.job(\"jr-eezd3arpcj\")\njob.status\nJuliaHub.isdone(job)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Similarly, the kill_job function can be used to stop a running job, and the extend_job function can be used to extend the job's time limit.","category":"page"},{"location":"guides/jobs/#Waiting-on-jobs","page":"Jobs","title":"Waiting on jobs","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"A common pattern in a script is to submit one or more jobs, and then wait until the jobs complete, to then process their outputs. isdone can be used to see if a job has completed.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Main.MOCK_JULIAHUB_STATE[:jobs] = Dict(\"jr-novcmdtiz6\" => Dict(\"status\" => \"Running\"))","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"job = JuliaHub.job(\"jr-novcmdtiz6\")\nJuliaHub.isdone(job)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"empty!(Main.MOCK_JULIAHUB_STATE)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The wait_job function also provides a convenient way for a script to wait for a job to finish.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"job = JuliaHub.wait_job(\"jr-novcmdtiz6\")\nJuliaHub.isdone(job)","category":"page"},{"location":"guides/jobs/#Accessing-job-outputs","page":"Jobs","title":"Accessing job outputs","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"There are two ways a JuliaHub job can store outputs that are directly related to a specific job[1]:","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Small, simple outputs can be stored by setting the ENV[\"RESULTS\"] environment variable. Conventionally, this is often set to a JSON object, and will act as a dictionary of key value pairs.\nFiles or directories can be uploaded by setting the ENV[\"RESULTS_FILE\"] to a local file path on the job. Note that directories are combined into a single tarball when uploaded.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"[1]: You can also e.g. upload datasets etc. But in that case the resulting data is not, strictly speaking, related to a specific job.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The values set via the RESULTS environment variable can be accessed with the .results field of a Job object:","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Main.MOCK_JULIAHUB_STATE[:jobs] = Dict(\n    \"jr-novcmdtiz6\" => Dict(\n        \"outputs\" => \"\"\"\n        {\"user_param\": 2, \"output_value\": 4}\n        \"\"\"\n    )\n)\nimport JuliaHub\njob = JuliaHub.job(\"jr-novcmdtiz6\")","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"job.results","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"As the .results string is often a JSON object, you can use the the JSON.jl or JSON3.jl packages to easily parse it. For example","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"import JSON\nJSON.parse(job.results)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"When it comes to job result files, they can all be accessed via the .files field.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"job.files","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The job_files function can be used to filter down to specific file types.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"JuliaHub.job_files(job, :result)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"And if you know the name of the file, you can also use the job_files to get the specific JobFile object for a particular file directly.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"jobfile = JuliaHub.job_file(job, :result, \"outdir.tar.gz\")","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"To actually fetch the contents of a file, you can use the download_job_file function on the JobFile objects.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"empty!(Main.MOCK_JULIAHUB_STATE)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"","category":"page"},{"location":"reference/exceptions/","page":"Exceptions","title":"Exceptions","text":"CurrentModule=JuliaHub","category":"page"},{"location":"reference/exceptions/#Exceptions","page":"Exceptions","title":"Exceptions","text":"","category":"section"},{"location":"reference/exceptions/","page":"Exceptions","title":"Exceptions","text":"JuliaHub.jl is designed in a way that the only errors it should throw under normal circumstances are subtypes of JuliaHubException (in addition to standard ArgumentErrors and MethodErrors etc. from invalid function calls). Any unhandled errors from JuliaHub.jl or its dependencies should be considered a bug.","category":"page"},{"location":"reference/exceptions/","page":"Exceptions","title":"Exceptions","text":"tip: Debugging JuliaHub.jl issues\nYou can also enable debug logging for JuliaHub, which will make JuliaHub.jl print out additional debug messages, by setting the JULIA_DEBUG environment variable:ENV[\"JULIA_DEBUG\"]=\"JuliaHub\"","category":"page"},{"location":"reference/exceptions/","page":"Exceptions","title":"Exceptions","text":"JuliaHubException\nAuthenticationError\nInvalidAuthentication\nInvalidRequestError\nJuliaHubConnectionError\nJuliaHubError\nPermissionError","category":"page"},{"location":"reference/exceptions/#JuliaHub.JuliaHubException","page":"Exceptions","title":"JuliaHub.JuliaHubException","text":"abstract type JuliaHubException <: Exception\n\nAbstract supertype of all JuliaHub.jl exception types.\n\n\n\n\n\n","category":"type"},{"location":"reference/exceptions/#JuliaHub.AuthenticationError","page":"Exceptions","title":"JuliaHub.AuthenticationError","text":"struct AuthenticationError <: JuliaHubException\n\nException thrown if the authentication fails. The .msg fields contains a human-readable error message.\n\n\n\n\n\n","category":"type"},{"location":"reference/exceptions/#JuliaHub.InvalidAuthentication","page":"Exceptions","title":"JuliaHub.InvalidAuthentication","text":"struct InvalidAuthentication <: JuliaHubException\n\nThis exception is thrown if the authentication token is invalid or has expired. Re-authenticating with JuliaHub.authenticate should generally be sufficient to resolve the issue.\n\n\n\n\n\n","category":"type"},{"location":"reference/exceptions/#JuliaHub.InvalidRequestError","page":"Exceptions","title":"JuliaHub.InvalidRequestError","text":"struct InvalidRequestError <: JuliaHubException\n\nAn exception thrown if the request was rejected by the backend due to request parameters that are inconsistent with the backend state. The .msg field contains the error message.\n\n\n\n\n\n","category":"type"},{"location":"reference/exceptions/#JuliaHub.JuliaHubConnectionError","page":"Exceptions","title":"JuliaHub.JuliaHubConnectionError","text":"struct JuliaHubConnectionError <: JuliaHubException\n\nAn exception thrown if there is a communication error with JuliaHub.\n\nThe .msg field contains the error message. If there is an underlying exception, it is stored in the .exception field.\n\n\n\n\n\n","category":"type"},{"location":"reference/exceptions/#JuliaHub.JuliaHubError","page":"Exceptions","title":"JuliaHub.JuliaHubError","text":"struct JuliaHubError <: JuliaHubException\n\nAn exception thrown if there is an unexpected response from or backend failure in JuliaHub.\n\nThe .msg field contains the error message. If there is an underlying exception, it is stored in the .exception field.\n\n\n\n\n\n","category":"type"},{"location":"reference/exceptions/#JuliaHub.PermissionError","page":"Exceptions","title":"JuliaHub.PermissionError","text":"struct PermissionError <: JuliaHubException\n\nThrown if the currently authenticated user does not have the necessary permissions to perform the operation. The .msg field contains the error message, and .response may contain the raw server response.\n\n\n\n\n\n","category":"type"},{"location":"reference/exceptions/#Index","page":"Exceptions","title":"Index","text":"","category":"section"},{"location":"reference/exceptions/","page":"Exceptions","title":"Exceptions","text":"Pages = [\"exceptions.md\"]","category":"page"},{"location":"reference/exceptions/","page":"Exceptions","title":"Exceptions","text":"","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"CurrentModule=JuliaHub","category":"page"},{"location":"guides/authentication/#guide-authentication","page":"Authentication","title":"Authentication","text":"","category":"section"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"In order to talk to a JuliaHub instance, you must authenticate with your user account, and have an authentication token. JuliaHub.jl aims to make it as easy and painless as possible to manage your authentication tokens.","category":"page"},{"location":"guides/authentication/#Overview","page":"Authentication","title":"Overview","text":"","category":"section"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"To use this package to authenticate in your current Julia session, you can call the authenticate function. It will return an Authentication object, containing the token.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"In principle, nearly every function here takes an (optional) auth keyword argument, that should be passed an Authentication object. However, this is usually not needed in practice, since JuliaHub.jl remembers your last authentication, and uses that as the default value for auth. Additionally, if there is no active authentication, calling any function that requires authentication will try to automatically authenticate (and may also trigger an interactive authentication). You can access the currently active authentication object with current_authentication.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"In order for authenticate to work, you must specify which JuliaHub instance to authenticate against (e.g. juliahub.com). There are two options for this:","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"You can configure the Julia package server to point to the JuliaHub instance by explicitly setting the JULIA_PKG_SERVER environment variable. See Local environments below to for more information on how to do it in local environments. In JuliaHub jobs, however, it is automatically set.\nYou can directly pass the JuliaHub instance URL to authenticate, e.g. JuliaHub.authenticate(\"juliahub.com\").","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"If a valid token for the JuliaHub instance exists on the disk, authenticate will use that directly. Otherwise it will attempt an interactive, browser-based authentication against the Julia instance. For the latter, under the hood, it uses the PkgAuthentication package to perform the interactive authentication","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"tip: Authentication token storage\nJuliaHub (or, more generally, package server) tokens are stored in the ~/.julia/servers/ directory. Inspecting the auth.toml files, or deleting them and re-authenticating may help if you are having unexpected issues with authentication.Note that this applies mainly when running JuliaHub.jl on your local computer. In JuliaHub cloud IDEs and jobs, the authentication token is actively managed by JuliaHub, and tampering with it may lead to other issues.","category":"page"},{"location":"guides/authentication/#JuliaHub-cloud-environments","page":"Authentication","title":"JuliaHub cloud environments","text":"","category":"section"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"When working with JuliaHub.jl in JuliaHub cloud environment, such as in JuliaHub Cloud IDEs or in JuliaHub jobs, you generally do not have to worry about authentication. The JULIA_PKG_SERVER should always be correctly set up, and the auth.toml file will also be present and up-to-date. This means that authenticate should automatically be able to authenticate your session, and you should never be prompted for interactive authentication.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"tip: No need to authenticate()\nAs in a JuliaHub environment everything is already set up and pointing to the correct server, you do not need to call authenticate yourself, nor set up JULIA_PKG_SERVER. You should be able to start using JuliaHub.jl functions right away.However, in JuliaHub jobs, you may still want to log the Authentication object in the job logs. You can use something like this at the top of your script:@info \"JuliaHub authentication token\" JuliaHub.authenticate()","category":"page"},{"location":"guides/authentication/#Local-environments","page":"Authentication","title":"Local environments","text":"","category":"section"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"When using JuliaHub.jl outside of JuliaHub (e.g. on your local laptop), you must explicitly configure the Julia package server, or pass the JuliaHub URL to authenticate.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"The Julia package server can be configured via the JULIA_PKG_SERVER environment variable, and there are a few options for this. In a live Julia session, you can simply set it via ENV, e.g.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"ENV[\"JULIA_PKG_SERVER\"] = \"juliahub.com\"","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"You can also do it this way in the Julia startup.jl script.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"Alternatively, on an UNIX-y system, you can set it in the shell or a script with e.g.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"export JULIA_PKG_SERVER=juliahub.com","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"If you're exclusively using the VS Code editor with the Julia extension, you can configure the package server in the UI directly, and it will apply to any REPLs you open in VS Code.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"(Image: Julia: Package Server option in Julia VS Code extension)","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"When running JuliaHub.jl on your local computer for the first time, or when you have not used JuliaHub.jl in a while, you will likely see something like:","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"julia> using JuliaHub\n\njulia> JuliaHub.authenticate()\nAuthentication required: please authenticate in browser.\nThe authentication page should open in your browser automatically, but you may need to switch to the opened window or tab. If the authentication page is not automatically opened, you can authenticate by manually opening the following URL: ...","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"Once you perform the browser-based authentication, the local auth.toml file gets updated, and JuliaHub.jl will be able to talk to the JuliaHub instance.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"note: JULIA_PKG_SERVER\nSetting JULIA_PKG_SERVER will also make the Julia package manager operations use the JuliaHub instance for Pkg operations. This is necessary for accessing packages on the private registries only accessible via JuliaHub. But it also means that you must have a valid JuliaHub authentication token in order to pull packages from the General registry.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"note: Enterprise JuliaHub instances\nEnterprise users generally have their own JuliaHub instance hosted under a different domain. In those cases, the recommended workflow is to globally set the JULIA_PKG_SERVER variable to point to the enterprise instance. This has the following benefits:JuliaHub.jl will, by default, authenticate with the server defined by JULIA_PKG_SERVER, opening a browser window if needed to acquire the token.\nAll package manager operations (i.e. pkg> or Pkg.*) will be routed through your enterprise JuliaHub instances as well. This is most useful if you need access to packages that are available in your company's private registries.Note: the same authentication token is used for package operations and JuliaHub operations.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"","category":"page"},{"location":"internal/#Internal","page":"Internal","title":"Internal","text":"","category":"section"},{"location":"internal/#_PackageBundler","page":"Internal","title":"_PackageBundler","text":"","category":"section"},{"location":"internal/","page":"Internal","title":"Internal","text":"JuliaHub._PackageBundler.bundle\nJuliaHub._PackageBundler.path_filterer","category":"page"},{"location":"internal/#JuliaHub._PackageBundler.bundle","page":"Internal","title":"JuliaHub._PackageBundler.bundle","text":"bundle(dir; output = \"\",  force=false, allownoenv=false, verbose = true) -> String\n\nCreates a .tar file with the contents of dir as well as any packages that are either tracked by path (developed) outside dir or packages that are not tracked by the PkgServer. Artifacts are also bundled. The bundled packages and artifacts go into a .bundled/depot directory and is set up like a depot and can thus be made available by adding it to DEPOT_PATH.\n\n.git and globs listed in .juliabundleignore are excluded form the bundle.\n\nReturns the hex-encoded SHA256 of the Manifest.toml file that is packed into the appbundle. The return value is used when requesting a sysimage build, in which case we have to pass the manifest's hash with the submit request.\n\n\n\n\n\n","category":"function"},{"location":"internal/#JuliaHub._PackageBundler.path_filterer","page":"Internal","title":"JuliaHub._PackageBundler.path_filterer","text":"path_filterer(top)\n\nReturns a function that takes a file or directory path and checks whether that is excluded by the nearest .juliabundleignore file. The function will also ignore any .git files and directories.\n\nThe top argument specifies the highest directory up the tree that will be searched for the .juliabundleignore file.\n\nThe function will return false for any excluded files and true otherwise, and can be used as a predicate for filtering files that should be bundled.\n\n\n\n\n\n","category":"function"},{"location":"internal/#Index","page":"Internal","title":"Index","text":"","category":"section"},{"location":"internal/","page":"Internal","title":"Internal","text":"Pages = [\"internal.md\"]","category":"page"},{"location":"internal/","page":"Internal","title":"Internal","text":"","category":"page"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"CurrentModule = JuliaHub\nDocTestSetup = :(using JuliaHub)","category":"page"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"using JuliaHub","category":"page"},{"location":"reference/jobs/#job-apis","page":"Jobs","title":"Jobs","text":"","category":"section"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"JuliaHub.jl provides various methods to inspect and interact jobs that have been submitted to JuliaHub. It is also possible to get access to job outputs programmatically.","category":"page"},{"location":"reference/jobs/#Job-status-states","page":"Jobs","title":"Job status states","text":"","category":"section"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"A job will be in various \"states\" during its execution, and its current state can be inspected via the [.status] field.","category":"page"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"The following diagram illustrates the possible states a job can be in, and how it moves between them during a job lifecycle.","category":"page"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"stateDiagram-v2\ndirection LR\n[*] --> SUBMIT:::user\nSUBMIT --> Submitted\nSubmitted --> Running\nRunning --> FINISHED\nstate FINISHED {\n  direction LR\n  [*] --> Failed\n  [*] --> Stopped\n  [*] --> Completed\n}\nFINISHED:::user\n\nclassDef user fill:lightgray","category":"page"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"note: Refreshing a Job object\nAn instance of a Job object reflects the state of the job when the job function was called. If you want to inspect the current state of a job, you must first \"refresh\" the job object, which can simply be done with the help of the job function.job = JuliaHub.job(job)","category":"page"},{"location":"reference/jobs/#Reference","page":"Jobs","title":"Reference","text":"","category":"section"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"JuliaHub.JobReference\nJuliaHub.jobs\nJuliaHub.job\nJuliaHub.isdone\nJuliaHub.wait_job\nJuliaHub.kill_job\nJuliaHub.extend_job\nJuliaHub.JobLogMessage\nJuliaHub.job_logs\nJuliaHub.job_logs_buffered\nJuliaHub.job_logs_older!\nJuliaHub.job_logs_newer!\nJuliaHub.AbstractJobLogsBuffer\nJuliaHub.hasfirst\nJuliaHub.haslast\nJuliaHub.interrupt!\nJuliaHub.job_files\nJuliaHub.job_file\nJuliaHub.download_job_file\nJuliaHub.Job\nJuliaHub.JobStatus\nJuliaHub.JobFile\nJuliaHub.FileHash","category":"page"},{"location":"reference/jobs/#JuliaHub.JobReference","page":"Jobs","title":"JuliaHub.JobReference","text":"const JobReference :: Type\n\nA type constraint on the arguments of many jobs-related functions that is used to specify the job. A job reference must be either a Job object, or an AbstractString containing the unique job ID.\n\n\n\n\n\n","category":"type"},{"location":"reference/jobs/#JuliaHub.jobs","page":"Jobs","title":"JuliaHub.jobs","text":"JuliaHub.jobs(; [limit::Integer], [auth::Authentication]) -> Vector{Job}\n\nRetrieve the list of jobs, latest first, visible to the currently authenticated user.\n\nBy default, JuliaHub only returns up to 20 jobs. However, this default limit can be overridden by passing the limit keyword (which must be a positive integer).\n\nwarning: Non-dynamic job objects\nJob objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.job","page":"Jobs","title":"JuliaHub.job","text":"JuliaHub.job(job::JobReference; throw::Bool=true, [auth::Authentication]) -> Job\n\nFetch the details of a job based on the job reference ref. Will throw an InvalidRequestError if the job does not exist, or returns nothing if throw=false is passed.\n\nwarning: Non-dynamic job objects\nJob objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.isdone","page":"Jobs","title":"JuliaHub.isdone","text":"JuliaHub.isdone(::Job)\n\nA helper function to check if a Job is \"done\", i.e. its status is one of Completed, Stopped, or Failed.\n\nwarning: Non-dynamic job objects\nJob objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. As such, the result from this function may not represent the current live state of the job. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.wait_job","page":"Jobs","title":"JuliaHub.wait_job","text":"wait_job(\n    job::AbstractString;\n    interval::Integer = 30, [auth::Authentication]\n) -> Job\n\nBlocks until remote job referred to by the job reference job has completed, by polling it with every interval seconds. Returns an updated Job object.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.kill_job","page":"Jobs","title":"JuliaHub.kill_job","text":"JuliaHub.kill_job(job::JobRefererence; [auth::Authentication]) -> Job\n\nStop the job referred to by the job reference ref. Returns the updated Job object.\n\nSee also: Job.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.extend_job","page":"Jobs","title":"JuliaHub.extend_job","text":"JuliaHub.extend_job(job::JobReference, extension::Limit; [auth::Authentication]) -> Job\n\nExtends the time limit of the job referred to by the job reference ref by extension (Dates.Period, or Integer number of hours). Returns an updated Job object.\n\nSee Limit for more information on how the extension argument is interpreted. Note that Unlimited is not allowed as extension.\n\nSee also: Job.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.JobLogMessage","page":"Jobs","title":"JuliaHub.JobLogMessage","text":"struct JobLogMessage\n\nContains a single JuliaHub job log message, and has the following fields:\n\ntimestamp :: Union{ZonedDateTime, Nothing}: log message timestamp (in UTC)\nmessage :: Union{String, Nothing}: log message string. This generally corresponds to one line of printed output\n\nFields that can also be nothing may be missing for some log messages.\n\nSee also: job_logs, job_logs_buffered.\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/jobs/#JuliaHub.job_logs","page":"Jobs","title":"JuliaHub.job_logs","text":"JuliaHub.job_logs(job; offset::Integer = 0, [limit::Integer], [auth::Authentication]) -> Vector{JobLogMessage}\n\nFetches the log messages for the specified JuliaHub job. The job is specifed by passing the job name as a string, or by passing a Job object (i.e. job::Union{AbstractString,Job}). Returns the log messages as an array of JobLogMessage objects.\n\nOptionally, the function takes the following keyword arguments:\n\noffset::Integer: the offset of the first log message fetched (0 corresponds to the first message); for the first method, this defaults to 0; however, in the second (callback) case, if offset is not specified, any existing logs will be ignored.\nlimit::Integer: the maximum number of messages fetched (all by default)\n\nnote: No default limit\nThe limit keyword does not have a default limit, and so by default job_logs fetches all the log messages. This may take a while and require many requests to JuliaHub if the job has a huge number of log messages.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.job_logs_buffered","page":"Jobs","title":"JuliaHub.job_logs_buffered","text":"JuliaHub.job_logs_buffered(\n    [f::Base.Callable], job::Union{Job,AbstractString};\n    streaming::Bool=true, [offset::Integer],\n    [auth::Authentication]\n) -> AbstractJobLogsBuffer\n\nA lower-level function to work with log streams, and is particularly useful when working with jobs that have not finished yet and are actively producing new log messages.\n\nThe function accepts the following arguments:\n\nf :: Base.Callable: an optional callback function that gets called every time the buffer is updated. The callback must take two arguments: f(::AbstractJobLogsBuffer, ::AbstractVector). The first argument is the buffer object itself, and the second argument will be passed a read-only view of all the logs that have been loaded into the buffer, including the new ones.\njob :: Union{Job,AbstractString}: either the job name or a Job object.\nstreaming :: Bool: if set to true, the buffer object The streaming can be stopped with interrupt!.\noffset :: Integer: optional non-negative value to specify the starting point of the buffer\n\nInterface of the returned object\n\nReturns an instance of the abstract AbstractJobLogsBuffer type. These objects contain log messages (of type JobLogMessage), but not all the log messages are immediately available. Instead, at any given time the buffer represents a continous section of logs that can be extended in either direction.\n\nThe following functions can be used to interact with log buffers: job_logs_newer!, job_logs_older!, JuliaHub.hasfirst, JuliaHub.haslast. Additionally, the objects will have a .logs :: Vector{JobLogMessage} property that can be used to access the log messages that have been loaded into the buffer.\n\nSee also: job_logs, Job.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.job_logs_older!","page":"Jobs","title":"JuliaHub.job_logs_older!","text":"JuliaHub.job_logs_older!(\n    buffer::AbstractJobLogsBuffer; [count::Integer], [auth::Authentication]\n) -> AbstractJobLogsBuffer\n\nUpdates the AbstractJobLogsBuffer object by adding up to count log messages to the beginning of the buffer. If count is omitted, it will seek all the way to the beginning of the logs.\n\nIf all the logs have already been loaded into the buffer (i.e. JuliaHub.hasfirst(buffer) is true), the function is a no-op.\n\nSee also: job_logs_buffered, job_logs_newer!.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.job_logs_newer!","page":"Jobs","title":"JuliaHub.job_logs_newer!","text":"JuliaHub.job_logs_newer!(\n    buffer::AbstractJobLogsBuffer; [count::Integer], [auth::Authentication]\n) -> AbstractJobLogsBuffer\n\nUpdates the AbstractJobLogsBuffer object by adding up to count log messages to the end of the buffer. If count is omitted, it will seek all the way to the end of the current logs.\n\nFor a finished job, if all the logs have already been loaded into the buffer (i.e. JuliaHub.haslast(buffer) is true), the function is a no-op. If the buffer is actively streaming new logs for a running job,  then the function is also a no-op.\n\nSee also: job_logs_buffered, job_logs_older!.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.AbstractJobLogsBuffer","page":"Jobs","title":"JuliaHub.AbstractJobLogsBuffer","text":"abstract type AbstractJobLogsBuffer\n\nSupertype of possible objects returned by job_logs_buffered. See the job_logs_buffered function for a description of the interface.\n\n\n\n\n\n","category":"type"},{"location":"reference/jobs/#JuliaHub.hasfirst","page":"Jobs","title":"JuliaHub.hasfirst","text":"JuliaHub.hasfirst(::AbstractJobLogsBuffer) -> Bool\n\nDetermines whether the job log buffer has the first message of the job logs.\n\nSee also: haslast, job_logs_buffered.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.haslast","page":"Jobs","title":"JuliaHub.haslast","text":"JuliaHub.haslast(::AbstractJobLogsBuffer) -> Bool\n\nDetermines whether the job log buffer has the last message of the job logs. Note that if the job has not finished, this will always be false, since the job may produce additional logs.\n\nSee also: hasfirst, job_logs_buffered.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.interrupt!","page":"Jobs","title":"JuliaHub.interrupt!","text":"JuliaHub.interrupt!(::AbstractJobLogsBuffer; wait::Bool=true)\n\nCan be use to interrupt the asynchronous log streaming task. If the log buffer is not streaming, this function is a no-op.\n\nNote that the effect of JuliaHub.interrupt! may not be immediate and the function will block until the task has stopped. wait = false can be passed to make interrupt! return immediately, but in that case the buffer may stream for a little while longer.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.job_files","page":"Jobs","title":"JuliaHub.job_files","text":"JuliaHub.job_files(job::Job, [filetype::Symbol]) -> Vector{JobFile}\n\nReturn the list of inputs and/or output files associated job.\n\nThe optional filetype argument should be one of :input, :source, :result or :project, and can be used to filter the file list down to either just job input files (such as the appbundle or Julia environment files), or output files (such as the one uploaded via RESULTS_FILE).\n\nNote: job_file(job) is equivalent to job.files, and the latter is preferred. This function is primarily meant to be used when filtering by file type.\n\nSee also: Job.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.job_file","page":"Jobs","title":"JuliaHub.job_file","text":"JuliaHub.job_file(job::Job, type::Symbol, filename::AbstractString) -> JobFile | Nothing\n\nSearches for a job output file of a specified type and with the specific filename for job job, or nothing if the file was not found.\n\ntype should be one of the standard job file types. See JobFile and job_files for more information.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.download_job_file","page":"Jobs","title":"JuliaHub.download_job_file","text":"JuliaHub.download_job_file(file::JobFile, path::AbstractString; [auth]) -> String\nJuliaHub.download_job_file(file::JobFile, io::IO; [auth])\n\nDownloads a JobFile to a local path. Alternative, writeable stream object can be passed as the second argument to write the contents directly into the stream.\n\nWhen a local path is passed, it returns the path (which can be useful when calling the function as e.g. JuliaHub.download_job_file(file, tempname()))). When an IO object is passed, it returns nothing.\n\nFor example, to download a file into a temporary file:\n\njulia> file = JuliaHub.job_file(JuliaHub.job(\"jr-eezd3arpcj\"), :result, \"outdir.tar.gz\")\nJuliaHub.JobFile outdir.tar.gz (jr-eezd3arpcj, :result, 632143 bytes)\nsha2_256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nUploaded: 2023-03-15T07:59:29.473898+00:00\n\njulia> tmp = tempname()\n\"/tmp/jl_nE3uvkZwvC\"\n\njulia> JuliaHub.download_job_file(file, tmp)\n\"/tmp/jl_BmHgj8rQXe\"\n\njulia> bytes2hex(open(sha2_256, tmp))\n\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"\n\nAlternatively, you can also download the file into a writable IO stream, such as IOBuffer:\n\njulia> buffer = IOBuffer();\n\njulia> JuliaHub.download_job_file(file, buffer)\n\njulia> bytes2hex(sha2_256(take!(buffer)))\n\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"\n\nSee also: Job, JobFile.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.Job","page":"Jobs","title":"JuliaHub.Job","text":"struct Job\n\nRepresents a single job submitted to JuliaHub. Objects have the following properties:\n\nid :: String: the unique, automatically generated ID of the job\nalias :: String: a non-unique, but descriptive alias for the job (often set by e.g. applications)\nstatus :: JobStatus: a string-like JobStatus object storing the state of the job\nenv :: Dict: a dictionary of environment variables that were set when the job was submitted\nresults :: String: the output value set via ENV[\"RESULTS\"] (an empty string if it was not explicitly set)\nfiles :: Vector{JobFiles}: a list of JobFile objects, representing the input and output files of the job (see: job_files, job_file, download_job_file).\n\nSee also: job, jobs.\n\nwarning: Non-dynamic job objects\nJob objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/jobs/#JuliaHub.JobStatus","page":"Jobs","title":"JuliaHub.JobStatus","text":"struct JobStatus\n\nType of the .status field of a Job object, representing the current state of the job. Should be one of: Submitted, Running, Failed, Stopped, Completed.\n\nIn practice, the .status field should be treated as string and only used in string comparisons.\n\nSee also: isdone.\n\njulia> job = JuliaHub.job(\"jr-novcmdtiz6\")\nJuliaHub.Job: jr-novcmdtiz6 (Completed)\n submitted: 2023-03-15T07:56:50.974+00:00\n started:   2023-03-15T07:56:51.251+00:00\n finished:  2023-03-15T07:56:59.000+00:00\n files:\n  - code.jl (input; 3 bytes)\n  - code.jl (source; 3 bytes)\n  - Project.toml (project; 244 bytes)\n  - Manifest.toml (project; 9056 bytes)\n outputs: \"{}\"\n\njulia> job.status == \"Submitted\"\nfalse\n\njulia> job.status == \"Completed\"\ntrue\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/jobs/#JuliaHub.JobFile","page":"Jobs","title":"JuliaHub.JobFile","text":"struct JobFile\n\nA reference to a job input or output file, with the following properties:\n\n.name :: String: the name of the Job this file is attached to\n.type :: Symbol: indicated the file type (see below)\n.filename :: String: file name\n.size :: Int: size of the file in bytes (reported to be zero in cases where the file contents is missing)\n.hash :: Union{FileHash, Nothing}: a FileHash object containing the file hash, but may also be missing (nothing) in some cases, like when the file upload has not completed yet.\n\nThe file is uniquely identified by the (job, type, filename) triplet.\n\nThe type of the file should be one of:\n\n:input: various job input files, which includes code, environment, and appbundle files.\n:source: source\n:project: Julia project environment files (e.g. Project.toml, Manifest.toml, Artifacts.toml)\n:result: output results file, defined via the RESULTS_FILE environment variable in the job\n\nNote that some files are duplicated under multiple categories.\n\nSee also: Job, job_files, job_file.\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/jobs/#JuliaHub.FileHash","page":"Jobs","title":"JuliaHub.FileHash","text":"struct FileHash\n\nStores a hash and the algorithm used to calcute it. The object has the following properties:\n\n.algorithm :: Symbol: hash algorithm\n.hash :: Vector{UInt8}: the hash as a sequence of bytes\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/jobs/#Index","page":"Jobs","title":"Index","text":"","category":"section"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"Pages = [\"jobs.md\"]","category":"page"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"CurrentModule=JuliaHub\nDocTestSetup = :(using JuliaHub)","category":"page"},{"location":"reference/job-submission/#job-submission","page":"Job submission","title":"Job submission","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"On JuliaHub you can submit jobs, which are user-defined workloads that get allocated a dedicated compute capacity. For example, this includes running scripts in batch computations, cloud IDEs, interactive notebooks and so on. The functions and types here deal with starting up such jobs and apps. The functions to inspect running or finished jobs are documented separately.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"Pages=[\"job-submission.md\"]\nDepth=2:3","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"A complete JuliaHub job workload is defined by the following configuration pieces:","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"Job configuration. This specifies the computation that gets run, and includes information such as the type of computation (batch script vs starting a GUI application), any arguments or Julia code that gets passed to the job etc.\nAs of now, there are three general types of jobs that can be run on JuliaHub:\nBatch jobs: non-interactive Julia scripts\nDefault Applications: special, built-in, interactive applications (such as IDEs or product-specific dashboards)\nExternal package applications: Julia packages that can be run as either interactive or non-interactive jobs\nEach of these categories are configured slightly differently, and are described in more detail below.\nCompute configuration. This specifies the hardware and cluster topology that will be used to execute the job (such as the number of CPUs per node, whether there is a GPU present, the number of nodes), how the Julia processes are configured (e.g. single process per node vs process per cpu), and other low-level technical configuration.\nRuntime parameters. These are various additional parameters that control how a job behaves. Currently, this is limited to passing environment variables to the jobs, overriding the job's name in the UI, and to various configuration options related to running interactive jobs.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"In the JuliaHub.jl code, the first two categories are encapsulated by the AbstractJobConfig and ComputeConfig types. These two, together with the additional runtime parameters, make up a WorkloadConfig object that can be submitted to JuliaHub for executing with the submit_job function.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"The following sections on this page explain the different aspects of job submission in more detail. See the guide on submitting batch jobs to see more practical examples of how to submit jobs.","category":"page"},{"location":"reference/job-submission/#Compute-configuration","page":"Job submission","title":"Compute configuration","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"JuliaHub supports a predefined set of node configurations, each of which have a specific number of CPUs, memory, GPUs etc. A JuliaHub job must pick one of these node types to run on, although a distributed job can run across multiple instances of the same node type. A list of these node specifications can be obtained with the nodespecs function (returning a list of NodeSpec objects).","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"julia> JuliaHub.nodespecs()\n9-element Vector{JuliaHub.NodeSpec}:\n JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 0.33/hr =#; ncpu=4, memory=16, ngpu=false, exactmatch=true)\n JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 0.65/hr =#; ncpu=8, memory=32, ngpu=false, exactmatch=true)\n JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 2.4/hr =#; ncpu=32, memory=128, ngpu=false, exactmatch=true)\n JuliaHub.nodespec(#= r6: 3.5 GHz Intel Xeon Platinum 8375C, 0.22/hr =#; ncpu=2, memory=16, ngpu=false, exactmatch=true)\n JuliaHub.nodespec(#= r6: 3.5 GHz Intel Xeon Platinum 8375C, 0.42/hr =#; ncpu=4, memory=32, ngpu=false, exactmatch=true)\n JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 0.17/hr =#; ncpu=2, memory=8, ngpu=false, exactmatch=true)\n JuliaHub.nodespec(#= r6: 3.5 GHz Intel Xeon Platinum 8375C, 1.3/hr =#; ncpu=8, memory=64, ngpu=false, exactmatch=true)\n JuliaHub.nodespec(#= p2: Intel Xeon E5-2686 v4 (Broadwell), 1.4/hr =#; ncpu=4, memory=61, ngpu=true, exactmatch=true)\n JuliaHub.nodespec(#= p3: Intel Xeon E5-2686 v4 (Broadwell), 4.5/hr =#; ncpu=8, memory=61, ngpu=true, exactmatch=true)","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"While you can manually index into the list returned by nodespecs, that is generally inconvenient. Instead, the nodespec function should be used to find a suitable node for a particular job.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"julia> JuliaHub.nodespec(ncpu=2, memory=8)\nNode: 3.5 GHz Intel Xeon Platinum 8375C\n - GPU: no\n - vCores: 2\n - Memory: 8 Gb\n - Price: 0.17 $/hr","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"By default, nodespec finds the smallest node that satisfies the specified requirements. However, it also supports the exactmatch argument, which can be use to find the exactly matching node configuration.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"julia> JuliaHub.nodespec(ncpu=3, memory=5; exactmatch=true)\nERROR: InvalidRequestError: Unable to find a nodespec: ncpu=3 memory=5 gpu=false\nStacktrace:\n ...\n\njulia> JuliaHub.nodespec(ncpu=3, memory=5)\nNode: 3.5 GHz Intel Xeon Platinum 8375C\n - GPU: no\n - vCores: 4\n - Memory: 16 Gb\n - Price: 0.33 $/hr","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"By default, JuliaHub jobs run on a single node. However, for a distributed job, additional nodes can be allocated to a job by specifying the nnodes parameter. In that case, a Julia process is started on each node, and the additional nodes are linked to the main process via the Distributed module.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"While by default only a single Julia process is started on each node, by setting the process_per_cpu parameter, multiple Julia processes are started on the same node. The processes are isolated from each other by running in separate containers, but they share the CPUs, GPUs, and most crucially, the memory.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"See ComputeConfig and submit_job for more details on how exactly to set up this configuration.","category":"page"},{"location":"reference/job-submission/#jobs-runtime-config","page":"Job submission","title":"Runtime configuration","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"The submit_job function accepts various additional parameters that control aspects of the job. See the function's docstring for more details.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"Parameter Description\nname can be used to override the name of the job shown in the UI\nproject specifies the JuliaHub project UUID that the job is associate with\ntimelimit sets the time limit after which the job gets killed\nenv environment variables set at runtime","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"As an example, to have an environment variable set while the job is running, you could call submit_job as follows:","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"import JuliaHub # hide\nJuliaHub.submit_job(\n    JuliaHub.script\"\"\"\n    @info \"Extracting 'MY_PARAMETER'\" get(ENV, \"MY_PARAMETER\", nothing)\n    \"\"\",\n    env = Dict(\"MY_PARAMETER\" => \"example value\"),\n    dryrun = true\n)","category":"page"},{"location":"reference/job-submission/#jobs-batch","page":"Job submission","title":"Batch jobs","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"Batch jobs are Julia scripts with (optional) associated Julia package environments (Project.toml, Manifest.toml and/or Artifacts.toml) that run on the cluster non-interactively.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"See also: @script_str, script, appbundle for more details, and the guide on submitting batch jobs for a tutorial.","category":"page"},{"location":"reference/job-submission/#Specifying-the-job-image","page":"Job submission","title":"Specifying the job image","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"JuliaHub batch jobs can run in various container images. Different JuliaHub products often have their own image tailored for the application (e.g. that come with custom sysimages to reduce load times).","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"The list of all images available to the user can be obtained with batchimages, and a specific one can be picked out with batchimage. The latter function is particularly useful when submitting jobs.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"JuliaHub.submit_job(\n    JuliaHub.BatchJob(\n        JuliaHub.script\"\"\"\n        using SpecialProductModule\n        SpecialProductModule.run()\n        \"\"\",\n        image = JuliaHub.batchimage(\"specialproduct\"),\n    )\n)","category":"page"},{"location":"reference/job-submission/#jobs-default-apps","page":"Job submission","title":"Default Applications","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"compat: Experimental feature\nStarting application jobs with JuliaHub.jl is considered to be experimental. The APIs are likely to change in future JuliaHub.jl version.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"Default applications are the JuliaHub-built in applications (such as dashboards and IDEs), generally associated with specific JuliaHub products. Specific examples of default applications available to everyone include the Pluto notebooks and the Julia IDE.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"julia> apps = JuliaHub.applications()\n7-element Vector{JuliaHub.AbstractJuliaHubApp}:\n JuliaHub.application(:default, \"Linux Desktop\")\n JuliaHub.application(:default, \"Julia IDE\")\n JuliaHub.application(:default, \"Pluto\")\n JuliaHub.application(:default, \"Windows Workstation\")\n JuliaHub.application(:package, \"RegisteredPackageApp\")\n JuliaHub.application(:package, \"CustomDashboardApp\")\n JuliaHub.application(:user, \"ExampleApp.jl\")\n\njulia> JuliaHub.application(:default, \"Pluto\")\nDefaultApp\n name: Pluto\n key: pluto","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"A JuliaHub job that launches an application can be started by passing the object returned by the application function to submit_job.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"import JuliaHub # hide\nJuliaHub.submit_job(\n    JuliaHub.application(:default, \"Pluto\"),\n    ncpu = 8, memory = 16,\n)","category":"page"},{"location":"reference/job-submission/#jobs-packages","page":"Job submission","title":"External package applications","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"compat: Experimental feature\nStarting package application jobs with JuliaHub.jl is considered to be experimental. The APIs are likely to change in future JuliaHub.jl version.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"Specially crafted Julia packages can also be launched as JuliaHub jobs. They are either automatically picked up from the packages registries served by the JuliaHub instance (PackageApp), or added by users themselves (UserApp) via a Git URL.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"A package application must have a top-level package environment (i.e. it has a Project.toml the declares a name and a UUID), and it must have a entry script at bin/main.jl. When a package job starts, the package is added to an environment and bin/main.jl is called. Note that the bin/main.jl script does not have any access to any of the package dependencies.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"The applications function can list all available packages (with filtering for user or registered). JuliaHub.application can be used to pick a package by name. It works for both registered packages","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"JuliaHub.submit_job(\n    JuliaHub.application(:package, \"RegisteredPackageApp\"),\n    ncpu = 4,\n)","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"and for private, user applications","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"JuliaHub.submit_job(\n    JuliaHub.application(:user, \"ExampleApp.jl\"),\n    ncpu = 4,\n    env = Dict(\"example_parameter\" => \"2\")\n)","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"tip: Environment variables\nEnvironment variables (i.e. env) are a common way to communicate options and settings to package applications.","category":"page"},{"location":"reference/job-submission/#Reference","page":"Job submission","title":"Reference","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"NodeSpec\nnodespecs\nnodespec\nBatchImage\nbatchimages\nbatchimage\nAbstractJobConfig\nBatchJob\nscript\n@script_str\nappbundle\nComputeConfig\nsubmit_job\nLimit\nUnlimited\nWorkloadConfig","category":"page"},{"location":"reference/job-submission/#JuliaHub.NodeSpec","page":"Job submission","title":"JuliaHub.NodeSpec","text":"struct NodeSpec\n\nStores information about a compute node that can be allocated for JuliaHub jobs. The list of all available node specifications can be accessed with nodespecs, or specific ones searched with nodespec.\n\njulia> JuliaHub.nodespec()\nNode: 3.5 GHz Intel Xeon Platinum 8375C\n - GPU: no\n - vCores: 2\n - Memory: 8 Gb\n - Price: 0.17 $/hr\n\nThey can be used to contruct explicit compute configuration objects when submitting JuliaHub jobs.\n\nSee also: submit_job, ComputeConfig.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.nodespecs","page":"Job submission","title":"JuliaHub.nodespecs","text":"JuliaHub.nodespecs(; auth::Authentication) -> Vector{NodeSpec}\n\nQuery node specifications available on the current server, returning a list of NodeSpec objects.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.nodespec","page":"Job submission","title":"JuliaHub.nodespec","text":"JuliaHub.nodespec(\n    [nodes::Vector{NodeSpec}];\n    ncpu::Integer=1, ngpu::Integer=false, memory::Integer=1,\n    exactmatch::Bool=false, throw::Bool=true,\n    [auth::Authentication]\n) -> Union{NodeSpec, Nothing}\n\nFinds the node matching the specified node parameters. Throws an InvalidRequestError if it is unable to find a node with the specific parameters. However, if throw is set to false, it will return nothing instead in that situation.\n\nBy default, it searches for the smallest node that has the specified parameters or more higher. If exactmatch is set to true, it only returns a node specification if it can find one that matches the parameters exactly.\n\nA list of nodes (e.g. from nodespecs) can also be passed, so that the function does not have to query the server for the list. When this method is used, it is not necessary to pass auth.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.BatchImage","page":"Job submission","title":"JuliaHub.BatchImage","text":"struct BatchImage\n\nRepresents an available JuliaHub batch job image. These can be passed to BatchJob to specify which underlying job image will be used for the job.\n\nA list of available batch images can be accessed with batchimages and specific images can be constructed with batchimage.\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\nSee also: batchimages, batchimage, BatchJob, script, appbundle.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.batchimages","page":"Job submission","title":"JuliaHub.batchimages","text":"JuliaHub.batchimages([product::AbstractString]; [auth::Authentication]) -> Vector{BatchImage}\n\nReturn the list of all batch job images available to the currently authenticated user, as a list of BatchImage objects. These can be passed to BatchJob.\n\nOptionally, by passing a product identifier, the list can be narrowed down to images available for that specific product.\n\nnote: Batch images on older instances\nWhen using the package with an older JuliaHub instance (<= 6.1), the non-default batch images show up with @legacy as the product name. This indicates that the package is using an older API, and not that the images themselves are outdated.\n\nSee also: BatchImage, batchimage, BatchJob, script, appbundle.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.batchimage","page":"Job submission","title":"JuliaHub.batchimage","text":"JuliaHub.batchimage(\n    [product::AbstractString, [image::AbstractString]];\n    throw::Bool=true, [auth::Authentication]\n) -> BatchImage\n\nPick a product job batch image from the list of all batch image, returning a BatchImage object. If image is omitted, it will return the default image corresponding to product. If product is omitted as well, it will return the default image of the instance (generally the standard Julia batch image).\n\nWill throw an InvalidRequestError if the specified image can not be found. If throw=false, it will return nothing instead in this situation.\n\nnote: Batch images on older instances\nWhen using the package with an older JuliaHub instance (<= 6.1), the non-default batch images show up with @legacy as the product name. This indicates that the package is using an older API, and not that the images themselves are outdated.\n\nSee also: BatchImage, batchimages, BatchJob, script, appbundle.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.AbstractJobConfig","page":"Job submission","title":"JuliaHub.AbstractJobConfig","text":"abstract type AbstractJobConfig\n\nAbstract supertype of all application configuration types that can be passed to submit_job for submission as a JuliaHub job. The package has built-in support for the following application configurations:\n\nJuliaHub.BatchJob\nJuliaHub.ApplicationJob\nJuliaHub.PackageApp\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.BatchJob","page":"Job submission","title":"JuliaHub.BatchJob","text":"struct BatchJob <: AbstractJobConfig\n\nRepresents the application configuration of a JuliaHub batch job. A batch job is defined by the following information:\n\nThe Julia code that is to be executed in the job.\nJulia package environment (i.e. Project.toml, Manifest.toml) and other files, such as the appbundle.\nThe underlying batch job container image (see also batchimages), which defaults to the standard Julia image by default.\n\nInstances of this types should normally not be constructed directly, and the following functions should be used instead:\n\nscript or @script_str: for submitting simple Julia scripts or code snippets\nappbundle: for submitting more complex \"appbundles\" that include additional file, private or modified package dependencies etc.\n\nOptional arguments\n\nimage  :: Union{BatchImage, Nothing}: can be used to specify which product's batch job image will be used when running the job, by passing the appropriate BatchImage object (see also: batchimage and batchimages). If set to nothing (the default), the job runs with the default Julia image.\nsysimage :: Bool: if set to true, requests that a system image is built from the job's Manifest.toml file before starting the job. Defaults to false.\n\ncompat: JuliaHub compatibility\nThe sysimage = true option requires JuliaHub 6.3 to have an effect. When running against older JuliaHub versions, it does not have an effect.\n\nConstructors\n\nBatchJob(::BatchJob; [image::BatchImage], [sysimage::Bool]) -> BatchJob\n\nConstruct a BatchJob, but override some of the optional arguments documented above. When the argument is omitted, the value from the underlying BatchJob object is used. This is the only constructor that is part of the public API.\n\nThis method is particularly useful when used in in combination with the @script_str string macro, to be able to specify the job image or trigger a sysimage build. For example, the following snippet will set a different batch image for the script-type job:\n\nJuliaHub.BatchJob(\n    JuliaHub.script\"\"\"\n    @info \"Hello World\"\n    \"\"\",\n    image = JuliaHub.batchimage(\"...\")\n)\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.script","page":"Job submission","title":"JuliaHub.script","text":"JuliaHub.script(...) -> BatchJob\n\nConstructs the configuration for a script-type batch job, returning the respective BatchJob object that can then be passed to submit_job. A script-type batch job is defined by the following:\n\nA user-provided Julia script that gets executed on the server. Note that no validation of the input code is done.\nAn optional Julia package environment (e.g. Project.toml, Manifest.toml and Artifacts.toml). If any of the TOML files are provided, they must parse as valid TOML files, but no further validation is done client-side.\nIf the manifest is not provided, the project environment must be instantiated from scratch, generally pulling in the latest versions of all the dependencies (although [compat] sections are honored).\nIt is also fine to omit the project file, and just provide the manifest, and the environment defined by the manifest still gets instantiated. If both are omitted, the job runs in an empty environment.\nA JuliaHub job image, which determines the container environment that will be used to execute the code in (see batchimage, batchimages, BatchImage). If omitted, the default Julia image is used.\n\nSee also the @script_str string macro to more easily submit simple scripts that are defined in code.\n\nMethods\n\nscript(\n    scriptfile::AbstractString;\n    [project_directory::AbstractString], [image::BatchImage], [sysimage::Bool]\n) -> BatchJob\n\nConstructs a script-type batch job configuration the will execute the code in scriptfile. Optionally, a path to a project environment directory can be passed via project_directory, which will be searched for the environment TOML files, and a job image can be specified via image.\n\nscript(;\n    code::AbstractString,\n    [project::AbstractString], [manifest::AbstractString], [artifacts::AbstractString],\n    [image::BatchImage], [sysimage::Bool]\n) -> BatchJob\n\nA lower-level method that can be used to construct the script-type BatchJob configuration directly in memory (i.e. without having to write out intermediate files).\n\nThe code keyword argument is mandatory and will specify contents of the Julia script that gets executed on the server. The Julia project environment can be specified by passing the contents of the TOML files via the corresponding arguments (project, manifest, artifacts). The job image can be specified via image.\n\nnote: Optional arguments\nSee BatchJob for a more thorough description of the optional arguments.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.@script_str","page":"Job submission","title":"JuliaHub.@script_str","text":"JuliaHub.@script_str -> JuliaHub.BatchJob\n\nA string macro to conveniently construct a script-type batch job configuration (BatchJob) that can be submitted as a JuliaHub job.\n\nscript = JuliaHub.script\"\"\"\n@info \"Hello World!\"\n\"\"\"\n\nThis allows for an easy submission of simple single-script jobs to JuliaHub:\n\nJuliaHub.submit_job(\n    JuliaHub.script\"\"\"\n    @info \"Hello World!\"\n    \"\"\"\n)\n\nBy default, the macro picks up the currently active Julia project environment (via Base.active_project()), and attaches the environment .toml files to the script. To disable this, you can call the macro with the noenv suffix, e.g.\n\nscript = JuliaHub.script\"\"\"\n@info \"Hello World!\"\n\"\"\"noenv\n\nHowever, if your local environment has development dependencies, you likely need to use an appbundle instead (see appbundle).\n\nnote: Using a different job image\nThere is no way to specify the job image with the string macro, and it will use the default Julia image. To use a different job image, you should either use the script function, either by fully constructing the batch job configuration with the keyword arguments, or by using the BatchJob(::BatchJob; image=...) method.JuliaHub.submit_job(\n    JuliaHub.BatchJob(\n        JuliaHub.script\"\"\"\n        @info \"Hello World!\"\n        \"\"\",\n        image = JuliaHub.batchimage(...)\n    )\n)You can also use this pattern to set the sysimage option.\n\n\n\n\n\n","category":"macro"},{"location":"reference/job-submission/#JuliaHub.appbundle","page":"Job submission","title":"JuliaHub.appbundle","text":"JuliaHub.appbundle(\n    directory::AbstractString, codefile::AbstractString;\n    [image::BatchImage], [sysimage::Bool]\n) -> BatchJob\nJuliaHub.appbundle(\n    directory::AbstractString;\n    code::AbstractString, [image::BatchImage], [sysimage::Bool]\n) -> BatchJob\n\nConstruct an appbundle-type JuliaHub batch job configuration. An appbundle is a directory containing a Julia environment that is bundled up, uploaded to JuliaHub, and then unpacked and instantiated as the job starts.\n\nThe code that gets executed is read from codefile, which should be a path to Julia source file relative to directory.\n\nJuliaHub.appbundle(@__DIR__, \"my-script.jl\")\n\nAlternatively, if codefile is omitted, the code can be provided as a string via the code keyword argument.\n\nJuliaHub.AppBundle(\n    @__DIR__,\n    code = \"\"\"\n    @show ENV\n    \"\"\"\n)\n\nSee BatchJob for a description of the optional arguments.\n\nExtended help\n\nThe following should be kept in mind about how appbundles are handled:\n\nThe bundler looks for a Julia environment (i.e. Project.toml and/or Manifest.toml files) at the root of the directory. If the environment does not exist (i.e. the files are missing), the missing files are created. If the manifest is missing, then the environment is re-instantiated from scratch based on the contents of Project.toml. The generated files will also be left in the user-provided directory directory.\nDevelopment dependencies of the environment (i.e. packages added with pkg> develop or Pkg.develop()) are also bundled up into the archive that gets submitted to JuliaHub (including any current, uncommitted changes). Registered packages are installed via the package manager via the standard environment instantiation, and their source code is not included in the bundle directly.\nWhen the JuliaHub job starts, the bundle is unpacked and the job's starting working directory is set to the appbundle/ directory, and you can e.g. load the data from those files with just read(\"my-data.txt\", String).\nNote that @__DIR__ points elsewhere and, relatedly, include in the main script should be used with an absolute path (e.g. include(joinpath(pwd(), \"my-julia-file.jl\"))).\ncompat: JuliaHub 6.2 and older\nOn some older JuliaHub versions (6.2 and older), the working directory was set to the parent directory of appbundle/, and so it was necessary to do joinpath(\"appbundle\", \"mydata.dat\") to load the code.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.ComputeConfig","page":"Job submission","title":"JuliaHub.ComputeConfig","text":"struct ComputeConfig\n\nThis type encapsulates the configuration of a jobs's compute cluster, including the hardware configuration and the cluster topology.\n\nSee also: submit_job.\n\nConstructors\n\nJuliaHub.ComputeConfig(\n    node::NodeSpec;\n    nnodes::Integer = 1,\n    process_per_node::Bool = true,\n    elastic::Bool = false,\n)\n\nnode: a NodeSpec object that specifies the hardware of a single node.\nnnodes::Union{Integer, Tuple{Integer, Integer}} = 1: specifies the number of nodes of type node that will be allocated. Alternatively, a two-integer tuple can also be passed, where the first value specifies the minimum number of nodes required to start a job. By default, a single-node job is started.\nprocess_per_node::Bool = true: if true, there will only be a single Julia process per node, and the total number of Julia processes will be nnodes. If set to false, however, each core on each node will be allocated a separate Julia process (running in an isolated container on the same node), and so the total number of Julia processes will be nnodes × ncpu, and it will essentially always be a multi-process job.\nelastic::Bool = false: if set, the job will be started in an elastic cluster mode. In this case, a minimum number of nnodes must not be passed.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.submit_job","page":"Job submission","title":"JuliaHub.submit_job","text":"JuliaHub.submit_job(\n    app::Union{AbstractJuliaHubApp, AbstractJobConfig},\n    [compute::ComputeConfig];\n    # Compute keyword arguments\n    ncpu::Integer = 1, ngpu::Integer = 0, memory::Integer = 1,\n    nnodes::Integer = 1, minimum_nnodes::Union{Integer,Nothing} = nothing,\n    elastic::Bool = false,\n    process_per_node::Bool = true,\n    # Runtime configuration keyword arguments\n    [alias::AbstractString], [env], [project::Union{UUID, AbstractString}],\n    timelimit::Limit = Hour(1),\n    # General keyword arguments\n    dryrun::Bool = false,\n    [auth :: Authentication]\n) -> Job\n\nSubmits the specified application config app as a job to JuliaHub. Returns a Job object corresponding to the submitted job.\n\nCompute arguments. If compute is passed, the compute keyword arguments can not be passed. If compute is not passed, the following arguments can be used to specify the compute configration via keyword arguments:\n\nncpu, ngpu and memory are used to pick a node type that will be used to run the job. The node type will be a minimum one that satisfies the constraints, but may have more compute resources than specified by the arguments (it corresponds to the exactmatch = false case of nodespec).\nnnodes, minimum_nnodes, process_per_node, and elastic specify the corresponding arguments in ComputeConfig.\n\nRuntime configuration. These are used to set the Runtime configuration of the job.\n\nalias :: Union{AbstractString, Nothing}: can be used to override the name of the job that gets displayed in the UI. Passing nothing is equivalent to omitting the argument.\ntimelimit :: Limit: sets the job's time limit (see Limit for valid values)\nenv: an iterable of key-value pairs that can be used to set environment variable that get set before, the job code gets executed.\nproject :: Union{UUID, AbstractString, Nothing}: the UUID of the project that the job will be associated with. If a string is passed, it must parse as a valid UUID. Passing nothing is equivalent to omitting the argument.\n\nGeneral arguments.\n\nauth :: Authentication: optional authentication object (see the authentication section for more information)\ndryrun :: Bool: if set to true, submit_job does not actually submit the job, but instead returns a WorkloadConfig object, which can be used to inspect the configuration that would be submitted.\nThe WorkloadConfig object can then be submitted to JuliaHub with the additional submit_job method:\nJuliaHub.submit_job(::WorkloadConfig; [auth::Authentication])\n\ncompat: JuliaHub compatibility\nThe timelimit = JuliaHub.Unlimited() argument requires JuliaHub 6.3+.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.Limit","page":"Job submission","title":"JuliaHub.Limit","text":"JuliaHub.Limit\n\nType-constraint on JuliaHub job timelimit arguments in submit_job.\n\nThe job time limit can either be a time period (an instance of Dates.Period), an Integer, (interpreted as the number of hours), or JuliaHub.Unlimited().\n\nOnly an integer number of hours are accepted by JuliaHub, and fractional hours from get rounded up to the next full integer number of hours (e.g. Dates.Minute(90) will be interpreted as 2 hours).\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.Unlimited","page":"Job submission","title":"JuliaHub.Unlimited","text":"struct Unlimited\n\nAn instance of this type can be passed as the [timelimit] option to submit_job to start jobs that run indefinitely, until killed manually.\n\nJuliaHub.submit_job(..., timelimit = JuliaHub.Unlimited())\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.WorkloadConfig","page":"Job submission","title":"JuliaHub.WorkloadConfig","text":"struct WorkloadConfig\n\nRepresents a full job configuration, including the application, compute and runtime configuration.\n\nInstances of this type can be constructed by passing dryrun = true to submit_job, and can also be directly submitted to JuliaHub with the same function.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#Experimental-APIs","page":"Job submission","title":"Experimental APIs","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"compat: Experimental features\nStarting application jobs with JuliaHub.jl is considered to be experimental. The APIs are likely to change in future JuliaHub.jl version.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"AbstractJuliaHubApp\napplications\napplication\nDefaultApp\nPackageApp\nUserApp\nApplicationJob\nPackageJob","category":"page"},{"location":"reference/job-submission/#JuliaHub.AbstractJuliaHubApp","page":"Job submission","title":"JuliaHub.AbstractJuliaHubApp","text":"abstract type AbstractJuliaHubApp\n\nAbstract supertype for JuliaHub applications object types.\n\ncompat: Experimental API\nApplications-related APIs are experimental, and may be changed or removed without notice.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.applications","page":"Job submission","title":"JuliaHub.applications","text":"JuliaHub.applications([category::Symbol]; [auth::Authentication]) -> Vector{AbstractJuliaHubApp}\n\nReturns the list of applications enabled for the authenticated user, optionally in the specified category only. Returns a vector of AbstractJuliaHubApp instances.\n\njulia> JuliaHub.applications()\n7-element Vector{JuliaHub.AbstractJuliaHubApp}:\n JuliaHub.application(:default, \"Linux Desktop\")\n JuliaHub.application(:default, \"Julia IDE\")\n JuliaHub.application(:default, \"Pluto\")\n JuliaHub.application(:default, \"Windows Workstation\")\n JuliaHub.application(:package, \"RegisteredPackageApp\")\n JuliaHub.application(:package, \"CustomDashboardApp\")\n JuliaHub.application(:user, \"ExampleApp.jl\")\n\njulia> JuliaHub.applications(:default)\n4-element Vector{JuliaHub.DefaultApp}:\n JuliaHub.application(:default, \"Linux Desktop\")\n JuliaHub.application(:default, \"Julia IDE\")\n JuliaHub.application(:default, \"Pluto\")\n JuliaHub.application(:default, \"Windows Workstation\")\n\njulia> JuliaHub.applications(:package)\n2-element Vector{JuliaHub.PackageApp}:\n JuliaHub.application(:package, \"RegisteredPackageApp\")\n JuliaHub.application(:package, \"CustomDashboardApp\")\n\njulia> JuliaHub.applications(:user)\n1-element Vector{JuliaHub.UserApp}:\n JuliaHub.application(:user, \"ExampleApp.jl\")\n\n\ncompat: Experimental API\nApplications-related APIs are experimental, and may be changed or removed without notice.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.application","page":"Job submission","title":"JuliaHub.application","text":"JuliaHub.application(\n    category::Symbol, name::AbstractString;\n    throw::Bool=true, [auth::Authentication]\n) -> AbstractJuliaHubApp\n\nReturns the application corresponding to name from the specified category of applications. Will throw an InvalidRequestError if the application can't be found, or returns nothing in this situation if throw=false is passed.\n\ncategory specifies the application category and must be one of: :default, :package, or :user. This is necessary to disambiguate apps with the same name in the different categories.\n\nSee also: applications.\n\nExamples\n\njulia> JuliaHub.applications()\n7-element Vector{JuliaHub.AbstractJuliaHubApp}:\n JuliaHub.application(:default, \"Linux Desktop\")\n JuliaHub.application(:default, \"Julia IDE\")\n JuliaHub.application(:default, \"Pluto\")\n JuliaHub.application(:default, \"Windows Workstation\")\n JuliaHub.application(:package, \"RegisteredPackageApp\")\n JuliaHub.application(:package, \"CustomDashboardApp\")\n JuliaHub.application(:user, \"ExampleApp.jl\")\n\ncompat: Experimental API\nApplications-related APIs are experimental, and may be changed or removed without notice.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.DefaultApp","page":"Job submission","title":"JuliaHub.DefaultApp","text":"struct DefaultApp <: AbstractJuliaHubApp\n\nRepresents a default JuliaHub instance application, and they can be started as jobs with submit_job.\n\nThe list of available applications can be accessed via the applications function, and specific applications can be picked out with application.\n\njulia> apps = JuliaHub.applications(:default)\n4-element Vector{JuliaHub.DefaultApp}:\n JuliaHub.application(:default, \"Linux Desktop\")\n JuliaHub.application(:default, \"Julia IDE\")\n JuliaHub.application(:default, \"Pluto\")\n JuliaHub.application(:default, \"Windows Workstation\")\n\njulia> JuliaHub.application(:default, \"Pluto\")\nDefaultApp\n name: Pluto\n key: pluto\n\ncompat: Experimental API\nApplications-related APIs are experimental, and may be changed or removed without notice.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.PackageApp","page":"Job submission","title":"JuliaHub.PackageApp","text":"struct PackageApp <: AbstractJuliaHubApp\n\nRepresents a JuliaHub package application that is available in one of the instance's package registries. These packages can be started as JuliaHub jobs with submit_job.\n\nThe list of available applications can be accessed via the applications function, and specific applications can be picked out with application.\n\njulia> apps = JuliaHub.applications(:package)\n2-element Vector{JuliaHub.PackageApp}:\n JuliaHub.application(:package, \"RegisteredPackageApp\")\n JuliaHub.application(:package, \"CustomDashboardApp\")\n\njulia> JuliaHub.application(:package, \"RegisteredPackageApp\")\nPackageApp\n name: RegisteredPackageApp\n uuid: db8b4d46-bfad-4aa5-a5f8-40df1e9542e5\n registry: General (23338594-aafe-5451-b93e-139f81909106)\n\nSee also: help.juliahub.com on applications\n\ncompat: Experimental API\nApplications-related APIs are experimental, and may be changed or removed without notice.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.UserApp","page":"Job submission","title":"JuliaHub.UserApp","text":"struct UserApp <: AbstractJuliaHubApp\n\nRepresents a private application that has been added to the user account via a Git repository. These applications can be started as JuliaHub jobs with submit_job.\n\nThe list of available applications can be accessed via the applications function, and specific applications can be picked out with application.\n\njulia> apps = JuliaHub.applications(:user)\n1-element Vector{JuliaHub.UserApp}:\n JuliaHub.application(:user, \"ExampleApp.jl\")\n\njulia> JuliaHub.application(:user, \"ExampleApp.jl\")\nUserApp\n name: ExampleApp.jl\n repository: https://github.com/JuliaHubExampleOrg/ExampleApp.jl\n\nSee also: help.juliahub.com on applications\n\ncompat: Experimental API\nApplications-related APIs are experimental, and may be changed or removed without notice.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.ApplicationJob","page":"Job submission","title":"JuliaHub.ApplicationJob","text":"struct ApplicationJob <: AbstractJobConfig\n\nAbstractJobConfig that wraps a DefaultApp. This is primarily used internally and should rarely be constructed explicitly.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.PackageJob","page":"Job submission","title":"JuliaHub.PackageJob","text":"struct PackageJob <: AbstractJobConfig\n\nAbstractJobConfig that wraps a PackageApp or UserApp. This is primarily used internally and should rarely be constructed explicitly.\n\nConstructors\n\nJuliaHub.PackageJob(app::Union{JuliaHub.PackageApp,JuliaHub.UserApp}; [sysimage::Bool = false])\n\nCan be used to construct a PackageApp or UserApp based job, but allows for some job parameters to be overridden. Currently, only support the enabling of a system image based job by setting sysimage = true.\n\njulia> app = JuliaHub.application(:package, \"RegisteredPackageApp\")\nPackageApp\n name: RegisteredPackageApp\n uuid: db8b4d46-bfad-4aa5-a5f8-40df1e9542e5\n registry: General (23338594-aafe-5451-b93e-139f81909106)\n\njulia> JuliaHub.submit_job(JuliaHub.PackageJob(app; sysimage = true))\nJuliaHub.Job: jr-xf4tslavut (Submitted)\n submitted: 2023-03-15T07:56:50.974+00:00\n started:   2023-03-15T07:56:51.251+00:00\n finished:  2023-03-15T07:56:59.000+00:00\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#Index","page":"Job submission","title":"Index","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"Pages = [\"job-submission.md\"]","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"","category":"page"},{"location":"","page":"Home","title":"Home","text":"CurrentModule=JuliaHub","category":"page"},{"location":"#JuliaHub.jl","page":"Home","title":"JuliaHub.jl","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"The JuliaHub.jl package offers a programmatic Julia interface to the JuliaHub platform.","category":"page"},{"location":"","page":"Home","title":"Home","text":"With JuliaHub.jl you can do things such as start JuliaHub jobs, access job outputs, and manage your datasets, programmatically, directly in the REPL or in your Julia script. It can also be used in JuliaHub jobs to interact with the platform (to upload datasets, for example).","category":"page"},{"location":"","page":"Home","title":"Home","text":"If you are unfamiliar with JuliaHub.jl, you may want to start out by reading through the package's Getting Started tutorial. If you want to know in detail how to programmatically work with a particular JuliaHub feature, you may want to skim through the applicable how-to guide:","category":"page"},{"location":"","page":"Home","title":"Home","text":"Pages = Main.PAGES_GUIDES\nDepth = 1:1","category":"page"},{"location":"","page":"Home","title":"Home","text":"Finally, detailed explanations and API references of JuliaHub.jl features and functions are available in the reference section of the manual:","category":"page"},{"location":"","page":"Home","title":"Home","text":"Pages = Main.PAGES_REFERENCE\nDepth = 1:1","category":"page"},{"location":"","page":"Home","title":"Home","text":"tip: JuliaHub platform documentation\nThe documentation here focuses on working with the JuliaHub.jl Julia library. See the main JuliaHub documentation to learn more about the JuliaHub platform, and see product (e.g. JuliaSim) documentation for product-specific questions.","category":"page"},{"location":"","page":"Home","title":"Home","text":"note: Enterprise use\nJuliaHub.jl works with both juliahub.com and private enterprise instances. For enterprise users: the instance URL can be passed to the authenticate function either directly via an argument, or via the JULIA_PKG_SERVER environment variable.","category":"page"},{"location":"","page":"Home","title":"Home","text":"","category":"page"}]
+[{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"CurrentModule=JuliaHub\nDocTestSetup = :(using JuliaHub)","category":"page"},{"location":"reference/authentication/#authentication","page":"Authentication","title":"Authentication","text":"","category":"section"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"In order to talk to a JuliaHub instance, you need to have a valid authentication token. JuliaHub reuses the Julia's built-in package server authentication tokens for this purpose. By default, the authentication uses the JULIA_PKG_SERVER environment variable to determine which JuliaHub instance to connect to, but this can be overridden by passing an argument to authenticate.","category":"page"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"The authenticate function can be used to construct a token. If a valid token is available in ~/.julia/servers, it gets reused. Otherwise, a browser window is opened, starting an interactive authentication procedure.","category":"page"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"All the functions that require authentication accept an auth keyword argument. However, JuliaHub.jl also stores the authentication token from the last authenticate call in a global variable and automatically uses that if auth is not provided, and also tries to authenticate automatically. The current global authentication object can be accessed via the current_authentication() function.","category":"page"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"See also: authentication guide, authentication section on help.juliahub.com, PkgAuthentication.","category":"page"},{"location":"reference/authentication/#Token-expiration-and-refresh-tokens","page":"Authentication","title":"Token expiration and refresh tokens","text":"","category":"section"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"By default, JuliaHub access tokens expire in 24 hours. However, the tokens usually also have a refresh token, which is valid for 30 days. If the access token has expired, but there is a valid refresh token available, authenticate will automatically try to use that, to re-acquire an access token without starting an interactive authentication.","category":"page"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"In JuliaHub job and cloud IDE environments, the authentication token on disk will be continuously kept up to date. The reauthenticate! function can be used to reload the token from disk.","category":"page"},{"location":"reference/authentication/#Reference","page":"Authentication","title":"Reference","text":"","category":"section"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"authenticate\nAuthentication\ncurrent_authentication\ncheck_authentication\nreauthenticate!\nSecret","category":"page"},{"location":"reference/authentication/#JuliaHub.authenticate","page":"Authentication","title":"JuliaHub.authenticate","text":"JuliaHub.authenticate(server = Pkg.pkg_server(); force::Bool = false, maxcount::Integer = 3, [hook::Base.Callable])\n\nAuthenticates with a JuliaHub server. If a valid authentication token does not exist in the Julia depot, a new token is acquired via an interactive browser based prompt. Returns an Authentication object if the authentication was successful, or throws an AuthenticationError if authentication fails.\n\nThe interactive prompts tries to authenticate for a maximum of maxcount times. If force is set to true, an existing authentication token is first deleted. This can be useful when the existing authentication token is causing the authentication to fail.\n\nExtended help\n\nBy default, it attemps to connect to the currently configured Julia package server URL (configured e.g. via the JULIA_PKG_SERVER environment variable). However, this can be overridden by passing the server argument.\n\nhook can be set to a function taking a single string-type argument, and will be passed the authorization URL the user should interact with in the browser. This can be used to override the default behavior coming from PkgAuthentication.\n\nThe returned Authentication object is also cached globally (overwriting any previously cached authentications), making it unnecessary to pass the returned object manually to other function calls. This is useful for interactive use, but should not be used in library code, as different authentication calls may clash.\n\n\n\n\n\n","category":"function"},{"location":"reference/authentication/#JuliaHub.Authentication","page":"Authentication","title":"JuliaHub.Authentication","text":"mutable struct Authentication\n\nAuthentication object constructed by the authenticate function that can be passed to the various JuliaHub.jl function via the auth keyword argument.\n\nObjects have the following properties:\n\nserver :: URIs.URI: URL of the JuliaHub instance this authentication token applies to.\nusername :: String: user's JuliaHub username (used for e.g. to namespace datasets)\ntoken :: JuliaHub.Secret: a Secret object storing the JuliaHub authentication token\n\nNote that the object is mutable, and hence will be shared as it is passed around. And at the same time, functions such as reauthenticate! may modify the object.\n\nSee also: authenticate, reauthenticate!, current_authentication.\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/authentication/#JuliaHub.current_authentication","page":"Authentication","title":"JuliaHub.current_authentication","text":"JuliaHub.current_authentication() -> Union{Authentication, Nothing}\n\nReturns the current globally active Authentication object, or nothing if authenticate has not yet been called.\n\njulia> JuliaHub.current_authentication()\nJuliaHub.Authentication(\"https://juliahub.com\", \"username\", *****)\n\nnote: Note\nCalling this function will not initialize authentication.\n\n\n\n\n\n","category":"function"},{"location":"reference/authentication/#JuliaHub.check_authentication","page":"Authentication","title":"JuliaHub.check_authentication","text":"JuliaHub.check_authentication(; [auth::Authentication]) -> Bool\n\nChecks if the authentication to a JuliaHub instance is still valid or not.\n\nThis can be used to periodically check an authentication token, to see if it is necessary to re-authenticate.\n\nSee also: reauthenticate!.\n\n\n\n\n\n","category":"function"},{"location":"reference/authentication/#JuliaHub.reauthenticate!","page":"Authentication","title":"JuliaHub.reauthenticate!","text":"JuliaHub.reauthenticate!([auth::Authentication]; force::Bool = false, maxcount::Integer = 3, [hook::Base.Callable])\n\nAttempts to update the authentication token in auth:\n\nIf the original auth.toml file has been updated, it simply reloads the token from the file.\nIf loading from auth.toml fails or force=true, it will attempt to re-authenticate with the server, possibly interactively.\n\nIf auth is omitted, it will reauthenticate the global Authentication object. The force, maxcount and hook are relevant for interactive authentication, and behave the same way as in the authenticate function.\n\nThis is mostly meant to be used to re-acquire authentication tokens in long-running sessions, where the initial authentication token may have expired.\n\nAs Authentication objects are mutable, the token will be updated in all contexts where the reference to the Authentication has been passed to.\n\nSee also: authenticate, current_authentication, Authentication, check_authentication.\n\n\n\n\n\n","category":"function"},{"location":"reference/authentication/#JuliaHub.Secret","page":"Authentication","title":"JuliaHub.Secret","text":"mutable struct Secret\n\nA helper type for storing secrets. Internally it is a covenience wrapper around Base.SecretBuffer. Predominantly used in Authentication objects to store the JuliaHub authentication token.\n\nThe String(::Secret) function can be used to obtain an unsecure string copy of the secret stored in the object.\n\njulia> s = JuliaHub.Secret(\"secret-string\")\nJuliaHub.Secret(\"*******\")\n\njulia> String(s)\n\"secret-string\"\n\nConstructors\n\nSecret(::AbstractString)\nSecret(::Vector{UInt8})\n\nCreate a Secret object from the input strings.\n\n\n\n\n\n","category":"type"},{"location":"reference/authentication/#Index","page":"Authentication","title":"Index","text":"","category":"section"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"Pages = [\"authentication.md\"]","category":"page"},{"location":"reference/authentication/","page":"Authentication","title":"Authentication","text":"","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"CurrentModule=JuliaHub\nDocTestSetup = :(using JuliaHub)","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"using JuliaHub","category":"page"},{"location":"guides/datasets/#guide-datasets","page":"Datasets","title":"Datasets","text":"","category":"section"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.jl offers a programmatic way to work with your JuliaHub datasets, and this section demonstrates a few common workflows you can use with these APIs.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"Pages = [\"datasets.md\"]\nDepth = 2:10","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"See the datasets reference page for a detailed reference of the datasets-related functionality.","category":"page"},{"location":"guides/datasets/#Accessing-datasets","page":"Datasets","title":"Accessing datasets","text":"","category":"section"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"The datasets function can be use to list all the datasets owned by the currently authenticated user, returning an array of Dataset objects.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"julia> JuliaHub.datasets()\n2-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"username\", \"example-dataset\"))\n JuliaHub.dataset((\"username\", \"blobtree/example\"))","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"If you know the name of the dataset, you can also directly access it with the dataset function, and you can access the dataset metadata via the properties of the Dataset object.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"julia> ds = JuliaHub.dataset(\"example-dataset\")\nDataset: example-dataset (Blob)\n owner: username\n description: An example dataset\n versions: 2\n size: 388 bytes\n tags: tag1, tag2\n\njulia> ds.owner\n\"username\"\n\njulia> ds.description\n\"An example dataset\"\n\njulia> ds.size\n388","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"If you want to work with dataset that you do not own but is shared with you in JuliaHub, you can pass shared=true to datasets, or specify the username.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"julia> JuliaHub.datasets(shared=true)\n3-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"username\", \"example-dataset\"))\n JuliaHub.dataset((\"anotheruser\", \"publicdataset\"))\n JuliaHub.dataset((\"username\", \"blobtree/example\"))\n\njulia> JuliaHub.datasets(\"anotheruser\")\n1-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"anotheruser\", \"publicdataset\"))\n\njulia> JuliaHub.dataset((\"anotheruser\", \"publicdataset\"))\nDataset: publicdataset (Blob)\n owner: anotheruser\n description: An example dataset\n versions: 1\n size: 57 bytes\n tags: tag1, tag2","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"Finally, JuliaHub.jl can also be used to download to your local machine with the download_dataset function.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"julia> JuliaHub.download_dataset(\"example-dataset\", \"mydata\")\nTransferred:       86.767 KiB / 86.767 KiB, 100%, 0 B/s, ETA -\nTransferred:            1 / 1, 100%\nElapsed time:         2.1s\n\"/home/username/my-project/mydata\"","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"As datasets can have multiple versions, the .versions property of Dataset can be used to see information about the individual versions (represented with DatasetVersion objects). When downloading, you can also specify the version you wish to download (with the default being the newest version).","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"julia> ds.versions\n2-element Vector{JuliaHub.DatasetVersion}:\n JuliaHub.DatasetVersion(dataset = (\"username\", \"example-dataset\"), version = 1)\n JuliaHub.DatasetVersion(dataset = (\"username\", \"example-dataset\"), version = 2)\n\njulia> ds.versions[1]\nDatasetVersion: example-dataset @ v1\n owner: username\n timestamp: 2022-10-13T01:39:42.963-04:00\n size: 57 bytes\n\njulia> JuliaHub.download_dataset(\"example-dataset\", \"mydata\", version=ds.versions[1].id)\nTransferred:       86.767 KiB / 86.767 KiB, 100%, 0 B/s, ETA -\nTransferred:            1 / 1, 100%\nElapsed time:         2.1s\n\"/home/username/my-project/mydata\"\n","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"The dataset version are sorted with oldest first. To explicitly access the newest dataset, you can use the last function on the .versions property.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"julia> last(ds.versions)\nDatasetVersion: example-dataset @ v2\n owner: username\n timestamp: 2022-10-14T01:39:43.237-04:00\n size: 331 bytes\n","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"tip: Tip: DataSets.jl\nIn JuliaHub jobs and Cloud IDEs you can also use the DataSets.jl package to access and work with datasets. See the help.julialang.org section on datasets for more information.","category":"page"},{"location":"guides/datasets/#Create,-update,-or-replace","page":"Datasets","title":"Create, update, or replace","text":"","category":"section"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"The upload_dataset function can be used to programmatically create new datasets on JuliaHub.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"touch(\"local-file\")\nmkdir(\"local-directory\")\nMain.MOCK_JULIAHUB_STATE[:existing_datasets] = []","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.upload_dataset(\"example-dataset\", \"local-file\")","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"The type of the dataset (Blob or BlobTree) depends on whether the uploaded object is a file or a directory. A directory will be stored as a BlobTree-type dataset on JuliaHub.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.upload_dataset(\"example-blobtree\", \"local-directory\")","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"The create, update, and replace options control how upload_dataset behaves with respect to existing datasets. By default, the function only creates brand new datasets, and trying to upload a dataset that already exists will fail with an error.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.upload_dataset(\"example-dataset\", \"local-file\")","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"This behavior can be overridden by setting update=true, which will then upload a new version of a dataset if it already exists. This is useful for jobs and workflows that are meant to be re-run, updating the dataset each time they run.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.upload_dataset(\"example-dataset\", \"local-file\"; update=true)","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"The replace=true option can be used to erase earlier versions of a dataset. This will delete all information about the existing dataset and is a destructive, non-recoverable action. This may also lead to the dataset type being changed.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.upload_dataset(\"example-dataset\", \"local-file\"; replace=true)","category":"page"},{"location":"guides/datasets/#Bulk-updates","page":"Datasets","title":"Bulk updates","text":"","category":"section"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"You can also use the package to perform bulk updates or deletions of datasets. The following example, adds a new tag to all the datasets where the name matches a particular pattern.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"# Find all the datasets that have names that start with 'my-analysis-'\nmyanalysis_datasets = filter(\n    dataset -> startswith(dataset.name, r\"my-analysis-.*\"),\n    JuliaHub.datasets()\n)\n# .. and now add a 'new-tag' tag to each of them\nfor dataset in myanalysis_datasets\n    @info \"Updating\" dataset\n    # Note: tags = ... overrides the whole list, so you need to manually retain\n    # old tags.\n    new_tags = [dataset.tags..., \"new-tag\"]\n    JuliaHub.update_dataset(dataset, tags = new_tags)\nend","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"While this example shows the update_dataset, for example, the delete_dataset function could be used in the same way.","category":"page"},{"location":"guides/datasets/","page":"Datasets","title":"Datasets","text":"","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"CurrentModule=JuliaHub\nDocTestSetup = :(using JuliaHub)","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"using JuliaHub","category":"page"},{"location":"reference/datasets/#datasets","page":"Datasets","title":"Datasets","text":"","category":"section"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"These APIs allow you to create, read, update, and delete datasets owned by the currently authenticated user.","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"You can use datasets, dataset, and download_dataset to access datasets or their metadata.\nupload_dataset, update_dataset, and delete_dataset can be used to create, update, or delete datasets.","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"See also: help.julialang.org on datasets, DataSets.jl.","category":"page"},{"location":"reference/datasets/#Dataset-types","page":"Datasets","title":"Dataset types","text":"","category":"section"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub currently has two distinct types of datasets:","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"Blob: a single file; or, more abstractly, a collection of bytes\nBlobTree: a directory or a file; more abstractly a tree-like collection of Blobs, indexed by file system paths","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"These types mirror the concepts in DataSets.jl","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.jl APIs do not rely that much on the dataset type for anything, except when downloading or uploading. In that case, a local file always corresponds to a Blob, and a local directory corresponds to a BlobTree. For example, when trying to upload a file as a new version of a BlobTree-type dataset will fail, because the dataset type can not change.","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"The upload_dataset function uses information filesystem to determine whether the created dataset is a Blob or a BlobTree, and similarly download_dataset will always download a Blob into a file, and a BlobTree as a directory.","category":"page"},{"location":"reference/datasets/#Dataset-versions","page":"Datasets","title":"Dataset versions","text":"","category":"section"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"A JuliaHub dataset can have zero or more versions. A newly created dataset usually has at least one version, but it may have zero versions if, for example, the upload did not finish. The versions are indexed with a linear list of integers starting from 1.","category":"page"},{"location":"reference/datasets/#Reference","page":"Datasets","title":"Reference","text":"","category":"section"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"JuliaHub.Dataset\nJuliaHub.DatasetVersion\nJuliaHub.datasets\nJuliaHub.DatasetReference\nJuliaHub.dataset\nJuliaHub.download_dataset\nJuliaHub.upload_dataset\nJuliaHub.update_dataset\nJuliaHub.delete_dataset","category":"page"},{"location":"reference/datasets/#JuliaHub.Dataset","page":"Datasets","title":"JuliaHub.Dataset","text":"struct Dataset\n\nInformation about a dataset stored on JuliaHub, and the following fields are considered to be public API:\n\nuuid :: UUID: dataset UUID\nowner :: String: username of the dataset owner\nname :: String: dataset name\ndtype :: String: generally either Blob or BlobTree, but additional values may be added in the future\nversions :: Vector{DatasetVersion}: an ordered list of DatasetVersion objects, one for each dataset version, sorted from oldest to latest (i.e. you can use last to get the newest version).\nsize :: Int: total size of the whole dataset (including all the dataset versions) in bytes\nFields to access user-provided dataset metadata:\ndescription :: String: dataset description\ntags :: Vector{String}: a list of tags\n\nnote: Canonical fully qualified dataset name\nIn some contexts, like when accessing JuliaHub datasets with DataSets.jl, the .owner-.name tuple constitutes the fully qualifed dataset name, uniquely identifying a dataset on a JuliaHub instance. I.e. for a dataset object dataset, it can be constructed as \"$(dataset.owner)/$(dataset.name)\".\n\nwarning: Non-dynamic dataset objects\nDataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/datasets/#JuliaHub.DatasetVersion","page":"Datasets","title":"JuliaHub.DatasetVersion","text":"struct DatasetVersion\n\nRepresents one version of a dataset.\n\nObjects have the following properties:\n\n.id: unique dataset version identifier (used e.g. in download_dataset to identify the dataset version).\n.size :: Int: size of the dataset version in bytes\n.timestamp :: ZonedDateTime: dataset version timestamp\n\njulia> JuliaHub.datasets()\n\nSee also: Dataset, datasets, dataset.\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/datasets/#JuliaHub.datasets","page":"Datasets","title":"JuliaHub.datasets","text":"JuliaHub.datasets([username::AbstractString]; shared::Bool=false, [auth::Authentication]) -> Vector{Dataset}\n\nList all datasets owned by username, returning a list of Dataset objects.\n\nIf username is omitted, it returns the datasets owned by the currently authenticated user. If username is different from the currently authenticated user, it only returns the datasets that are readable to (i.e. somehow shared with) the currently authenticated user.\n\nIf shared = true, it also returns datasets that belong to other users that have that have been shared with the currently authenticated user. In this case, username is effectively ignored.\n\njulia> JuliaHub.datasets()\n2-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"username\", \"example-dataset\"))\n JuliaHub.dataset((\"username\", \"blobtree/example\"))\n\njulia> JuliaHub.datasets(shared=true)\n3-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"username\", \"example-dataset\"))\n JuliaHub.dataset((\"anotheruser\", \"publicdataset\"))\n JuliaHub.dataset((\"username\", \"blobtree/example\"))\n\njulia> JuliaHub.datasets(\"anotheruser\")\n1-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"anotheruser\", \"publicdataset\"))\n\nwarning: Non-dynamic dataset objects\nDataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.\n\n\n\n\n\n","category":"function"},{"location":"reference/datasets/#JuliaHub.DatasetReference","page":"Datasets","title":"JuliaHub.DatasetReference","text":"const DatasetReference :: Type\n\nType constraint on the first argument of most of the datasets-related functions, that is used to uniquely specify the dataset that the operation will affect.\n\nThere are three different objects that can be passed as a dataset reference (dsref::DatasetReference):\n\n(owner::AbstractString, dataset_name::AbstractString)::Tuple{AbstractString,AbstractString}\nA tuple of the owner's username and the dataset's name.\ndataset_name::AbstractString\nJust a string with the dataset name; in this case the dataset's owner will be assumed to be the currently authenticated user (with the username determined from the Authentication objects passed via the auth keyword).\ndataset::Dataset\nUses the owner and dataset name information from a Dataset object.\n\nwarning: No UUID mismatch checks\nWhen using the third option (i.e. passing a Dataset), the dataset UUID will not be checked. So if the dataset with the same owner and username has been deleted and re-created as a new dataset (potentially of a different dtype etc), the functions will then act on the new dataset.\n\n\n\n\n\n","category":"type"},{"location":"reference/datasets/#JuliaHub.dataset","page":"Datasets","title":"JuliaHub.dataset","text":"JuliaHub.dataset(dataset::DatasetReference; throw::Bool=true, [auth::Authentication]) -> Dataset\n\nLooks up a dataset based on the dataset reference dataset. Returns the Dataset object corresponding to dataset_name, or throws a InvalidRequestError if the dataset can not be found (if throw=false is passed, returns nothing instead).\n\nBy passing a Dataset object as dataset, this can be used to update the Dataset object.\n\njulia> dataset = JuliaHub.dataset(\"example-dataset\")\nDataset: example-dataset (Blob)\n owner: username\n description: An example dataset\n versions: 2\n size: 388 bytes\n tags: tag1, tag2\n\njulia> JuliaHub.dataset(dataset)\nDataset: example-dataset (Blob)\n owner: username\n description: An example dataset\n versions: 2\n size: 388 bytes\n tags: tag1, tag2\n\nIf the specifed username is not the currently authenticated user, the dataset must be shared with the currently authenticated user (i.e. contained in datasets(; shared=true)).\n\nnote: Note\nThis will call datasets every time, which might become a problem if you are processing a large number of datasets. In that case, you should call datasets and process the returned list yourself.\n\nwarning: Non-dynamic dataset objects\nDataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.\n\n\n\n\n\n","category":"function"},{"location":"reference/datasets/#JuliaHub.download_dataset","page":"Datasets","title":"JuliaHub.download_dataset","text":"download_dataset(\n    dataset::DatasetReference, local_path::AbstractString;\n    replace::Bool = false, [version::Integer],\n    [quiet::Bool = false], [auth::Authentication]\n) -> String\n\nDownloads the dataset specified by the dataset reference dataset to local_path (which must not exist, unless replace = true), returning the absolute path to the downloaded file or directory. If the dataset is a Blob, then the created local_path will be a file, and if the dataset is a BlobTree the local_path will be a directory.\n\nBy default, it downloads the latest version, but an older version can be downloaded by specifying the version keyword argument. Caution: you should never assume that the index of the .versions property of Dataset matches the version number – always explicitly use the .id propert of the DatasetVersion object.\n\nThe function also prints download progress to standard output. This can be disabled by setting quiet=true. Any error output from the download is still printed.\n\nwarning: Warning\nSetting replace = true will recursively erase any existing data at local_path before replacing it with the dataset contents.\n\n\n\n\n\n","category":"function"},{"location":"reference/datasets/#JuliaHub.upload_dataset","page":"Datasets","title":"JuliaHub.upload_dataset","text":"JuliaHub.upload_dataset(dataset::DatasetReference, local_path; [auth,] kwargs...) -> Dataset\n\nUploads a new dataset or a new version of an existing dataset, with the dataset specified by the dataset reference dataset. The dataset type is determined from the local path (Blob if a file, BlobTree if a directory). If a Dataset object is passed, it attempts to update that dataset. Returns an updated Dataset object.\n\nThe following keyword arguments can be used to control the exact behavior of the function:\n\ncreate :: Bool (default: true): Create the dataset, if it already does not exist.\nupdate :: Bool (default: false): Upload the data as a new dataset version, if the dataset exists.\nreplace :: Bool (default: false): If a dataset exists, delete all existing data and create a new dataset with the same name instead. Excludes update = true, and only creates a completely new dataset if create=true as well.\n\nIn addition, the following keyword arguments can be passed to set or updated the dataset metadata when uploading:\n\ndescription: description of the dataset (a string)\ntags: an iterable of strings of all the tags of the dataset\nvisibility: a string with possible values public or private\nlicense: a valid SPDX license identifier, or a tuple (:fulltext, license_text), where license_text is the full text string of a custom license\ngroups: an iterable of valid group names\n\nIf a dataset already exists, then these fields are updated as if update_dataset was called.\n\nThe function will throw an ArgumentError for invalid argument combinations.\n\nUse the progress keyword argument to suppress upload progress from being printed.\n\nnote: Note\nPresently, it is only possible to upload datasets for the currently authenticated user.\n\n\n\n\n\n","category":"function"},{"location":"reference/datasets/#JuliaHub.update_dataset","page":"Datasets","title":"JuliaHub.update_dataset","text":"JuliaHub.update_dataset(dataset::DatasetReference; kwargs..., [auth]) -> Dataset\n\nUpdates the metadata of the dataset specified by the dataset reference dataset, as according to the keyword arguments keyword arguments. If the keywords are omitted, the metadata corresponding to it remains unchanged. Returns the Dataset object corresponding to the updated dataset.\n\nThe supported keywords are:\n\ndescription: description of the dataset (a string)\ntags: an iterable of strings of all the tags of the dataset\nvisibility: a string with possible values public or private\nlicense: a valid SPDX license identifier, or a tuple (:fulltext, license_text), where license_text is the full text string of a custom license\ngroups: an iterable of valid group names\n\nFor example, to add a new tag to a dataset:\n\ndataset = JuliaHub.dataset(\"my_dataset\")\nJuliaHub.update(dataset; tags = [dataset.tags..., \"newtag\"])\n\nnote: Note\nPresently, it is only possible to update datasets for the currently authenticated user.\n\n\n\n\n\n","category":"function"},{"location":"reference/datasets/#JuliaHub.delete_dataset","page":"Datasets","title":"JuliaHub.delete_dataset","text":"JuliaHub.delete_dataset(dataset::DatasetReference; force::Bool=false, [auth::Authentication]) -> Nothing\n\nDelete the dataset specified by the dataset reference dataset. Will return nothing if the delete was successful, or throws an error if it was not.\n\nNormally, when the dataset to be deleted does not exist, the function throws an error. This can be overridden by setting force = true.\n\njulia> JuliaHub.datasets()\n2-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"username\", \"example-dataset\"))\n JuliaHub.dataset((\"username\", \"blobtree\"))\n\njulia> JuliaHub.delete_dataset(\"example-dataset\")\n\njulia> JuliaHub.datasets()\n1-element Vector{JuliaHub.Dataset}:\n JuliaHub.dataset((\"username\", \"blobtree\"))\n\nnote: Note\nPresently, it is only possible to delete datasets for the currently authenticated user.\n\n\n\n\n\n","category":"function"},{"location":"reference/datasets/#Index","page":"Datasets","title":"Index","text":"","category":"section"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"Pages = [\"datasets.md\"]","category":"page"},{"location":"reference/datasets/","page":"Datasets","title":"Datasets","text":"","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"CurrentModule=JuliaHub","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"using JuliaHub\nENV[\"JULIA_PKG_SERVER\"] = \"juliahub.com\"","category":"page"},{"location":"getting-started/#getting-started","page":"Getting Started","title":"Getting Started with JuliaHub.jl","text":"","category":"section"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"This tutorial walks you through the basic operations you can do with JuliaHub.jl, from installation to submitting simple jobs and working with datasets. If you are unfamiliar with JuliaHub.jl, this is a good place to get started.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"If you already know what you wish to achieve with JuliaHub.jl, you can also skip this and jump directly into one of the more detailed how-to guides.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"In particular, the tutorial will show","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"How to install JuliaHub.jl and connect it to a JuliaHub instance.\nHow to create, access and update a simple dataset.\nHow to submit a simple job.","category":"page"},{"location":"getting-started/#Installation","page":"Getting Started","title":"Installation","text":"","category":"section"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.jl is a registered Julia package and can be installed using Julia's package manager. You can access the Julia package manager REPL mode by pressing ], and you can install JuliaHub.jl with","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"pkg> add JuliaHub","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Alternatively, you can use the Pkg standard library functions to install it.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"import Pkg\nPkg.add(\"JuliaHub\")","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Once it is installed, simply use import or using to load JuliaHub.jl into your current Julia session.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"julia> using JuliaHub","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"note: No exported names\nJuliaHub.jl does not have any exported names, so doing using JuliaHub does not introduce any functions or types in Main. Instead, JuliaHub.jl functions are designed to be used by prefixing them with JuliaHub. (e.g. JuliaHub.authenticate(...) or JuliaHub.submit_job(...))That said, there is nothing stopping you from explicitly bringing some names into your current scope, by doing e.g. using JuliaHub: submit_job, if you so wish!","category":"page"},{"location":"getting-started/#Authentication","page":"Getting Started","title":"Authentication","text":"","category":"section"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"In order to communicate with a JuliaHub instance, you need a valid authentication token. If you are working in a JuliaHub Cloud IDE, you actually do not need to do anything to be authenticated, as the authentication tokens are automatically set up in the cloud environment. To verify this, you can still call authenticate, which should load the pre-configured token.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.authenticate()","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"If you are working on a local computer, the easiest way to get started is to pass the URL of the JuliaHub instance to authenticate. Unless you have authenticated before, this will initiate an interactive browser-based authentication.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"julia> JuliaHub.authenticate(\"juliahub.com\")\nAuthentication required: please authenticate in browser.\nThe authentication page should open in your browser automatically, but you may need to switch to the opened window or tab. If the authentication page is not automatically opened, you can authenticate by manually opening the following URL: ...","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Once you have completed the steps in the browser, the function should return a valid authentication token.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"The authenticate function returns an Authentication object, which hold the authentication token. In principle, you can pass these objects directly to JuliaHub.jl function via the auth keyword argument. However, in practice, this is usually not needed, because JuliaHub.jl also remembers the last authentication in the Julia session in a global variable. You can see the current globally stored authentication token with current_authentication.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.current_authentication()","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"note: Authentication guide\nThere is more to authentication than this, including its relationship to the Julia package server and JULIA_PKG_SERVER environment variable. See the Authentication how-to if you want to learn more.","category":"page"},{"location":"getting-started/#Creating-and-accessing-datasets","page":"Getting Started","title":"Creating & accessing datasets","text":"","category":"section"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.jl allows you to create, access, and update the datasets that are hosted on JuliaHub. This section shows some of the basic operations you can perform with datasets.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Main.MOCK_JULIAHUB_STATE[:existing_datasets] = String[]\nMain.MOCK_JULIAHUB_STATE[:dataset_params] = Dict(\n    \"description\" => \"\",\n    \"tags\" => String[],\n)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"The datasets function allows you to list the datasets you have. Optionally, you can also make it show any other datasets you have access to.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.datasets()","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Unless you have created datasets in the web UI or in the IDE, this list will likely be empty currently. To fix that, let us upload a simple dataset using JuliaHub.jl.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Just as an example, we'll generate a simple 5-by-5 matrix, and save it in a file using the using the DelimitedFiles standard library.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"using DelimitedFiles\nmat = [i^2 + j^2 for i=1:5, j=1:5]\nwritedlm(\"matrix.dat\", mat)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Now that the matrix has been serialized into a text file on the disk, we can upload that file to JuliaHub with upload_dataset.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.upload_dataset(\"tutorial-matrix\", \"matrix.dat\")","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"warning: Existing dataset\nIf you already happen to have a dataset with the same name, the upload_dataset call will fail. It is designed to be safe by default. However, you can pass update=true or replace=true to either upload your file as a new version of the dataset, or to delete all existing versions and upload a brand new version.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"If we now call datasets, it should show up in the list of datasets.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.datasets()","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"To see more details about the dataset, you can index into the array returned by datasets. Alternatively, you can also use the dataset function to pick out a single dataset by its name.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.dataset(\"tutorial-matrix\")","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub datasets also support basic metadata, such as tags and a description field. You could set it directly in the upload_dataset function, but we did not. But that is fine, since we can use update_dataset to update the metadata at any time.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.update_dataset(\"tutorial-matrix\", description=\"An i^2 + j^2 matrix\")","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"The function also immediately queries JuliaHub for the updated dataset metadata by internally calling JuliaHub.dataset(\"tutorial-matrix\").","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Finally, JuliaHub.jl also allows you to download the datasets you have with the download_dataset function. We can also imagine doing this on a different computer or in a JuliaHub job.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.download_dataset(\"tutorial-matrix\", \"matrix-downloaded.dat\")","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"cp(\"matrix.dat\", \"matrix-downloaded.dat\")","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"This downloads the dataset into a local file, after which you can e.g. read it back into Julia and do operations on it.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"mat = readdlm(\"matrix-downloaded.dat\", '\\t', Int)\nsum(mat)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"tip: Directories as datasets\nWhile this demo uploaded a single file as a dataset, JuliaHub also supports uploading whole directories as a single dataset. For that, you can simply point upload_dataset to a directory, rather than a file. See the datasets how-to for more information on how to work with datasets.","category":"page"},{"location":"getting-started/#Submitting-a-job","page":"Getting Started","title":"Submitting a job","text":"","category":"section"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"JuliaHub.jl allows for an easy programmatic submission of JuliaHub jobs. In this example, we submit a simple script that downloads the dataset from the previous step, does a simple calculations and then upload the result. We then access the result locally with JuliaHub.jl.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"First, we need to specify the code that we want to run in the job. There are a few options for this, but in this example we use the @script_str string macro to construct a script-type computation, that simply runs the code snippet we specify.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"The following script will access the dataset, calculates the sum of all the elements, and stores the value in the job results. You will be able to access the contents of RESULTS in both the web UI, but also via JuliaHub.jl.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"s = JuliaHub.script\"\"\"\nusing JuliaHub, DelimitedFiles\n@info JuliaHub.authenticate()\nJuliaHub.download_dataset(\"tutorial-matrix\", \"matrix-downloaded.dat\")\nmat = readdlm(\"matrix-downloaded.dat\", '\\t', Int)\nmat_sum = @show sum(mat)\nENV[\"RESULTS\"] = string(mat_sum)\n\"\"\"","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"note: Job environment\nIn most cases, you also submit a Julia package environment (i.e. Project.toml and Manifest.toml files together with a job). That environment then gets instantiated before the user-provided code is run.The script\"\" string macro, by default, attaches the currently active environment to the job. This means that any packages that you are currently using should also be available on the job (although only registered packages added as non-development dependencies will work). You can use Base.active_project() or pkg> status to see what environment is currently active.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"To submit a job, you can simply call submit_job on it.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Main.MOCK_JULIAHUB_STATE[:jobs] = Dict(\n    \"jr-xf4tslavut\" => Dict(\n        \"status\" => \"Submitted\",\n        \"files\" => [],\n        \"outputs\" => \"\",\n    )\n)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"j = JuliaHub.submit_job(s)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"The submit_job function also allows you to specify configure how the job gets run, such as how many CPUs or how much memory it has available. By default, though, it runs your code on a single node, picking the smallest instance that is available.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"At this point, if you go to the \"Jobs\" page web UI, you should see the job there. It may take a few moments to actually start running. You can also call job on the returned Job object to refresh the status of the job.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Main.MOCK_JULIAHUB_STATE[:jobs] = Dict(\n    \"jr-xf4tslavut\" => Dict(\n        \"status\" => \"Running\",\n        \"files\" => [],\n        \"outputs\" => \"\",\n    )\n)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"j = JuliaHub.job(j)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Finally, after the job has completed, if you refresh the Job it should reflect the final status of the job, and also give you access to the","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Main.MOCK_JULIAHUB_STATE[:jobs] = Dict(\n    \"jr-xf4tslavut\" => Dict(\n        \"status\" => \"Completed\",\n        \"files\" => [],\n        \"outputs\" => \"550\",\n    )\n)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"j = JuliaHub.job(j)\nj.results","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"See the jobs how-to guide for more details on the different options when it comes to job submission.","category":"page"},{"location":"getting-started/#Next-steps","page":"Getting Started","title":"Next steps","text":"","category":"section"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"This tutorial has hopefully given an overview of basic JuliaHub.jl usage. For more advanced usage, you may want to read through the more detailed how-to guides.","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"Pages = Main.PAGES_GUIDES\nDepth = 1:1","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"# Clear the mocking state, so that other pages and docstrings would not\n# be affected.\nempty!(Main.MOCK_JULIAHUB_STATE)\n# We don't actually care about the generated files\nrm(\"matrix.dat\", force=true)\nrm(\"matrix-downloaded.dat\", force=true)","category":"page"},{"location":"getting-started/","page":"Getting Started","title":"Getting Started","text":"","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"CurrentModule=JuliaHub\nDocTestSetup = :(using JuliaHub)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"using JuliaHub","category":"page"},{"location":"guides/jobs/#Jobs","page":"Jobs","title":"Jobs","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"JuliaHub.jl can be used to both submit new jobs, and to inspect running or finished jobs.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Pages = [\"jobs.md\"]\nDepth = 2:10","category":"page"},{"location":"guides/jobs/#jobs-guide-batch","page":"Jobs","title":"Submitting batch jobs","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"A common use case for this package is to programmatically submit Julia scripts as batch jobs to JuliaHub, to start non-interactive workloads. In a nutshell, these are Julia scripts, together with an optional Julia environment, that get executed on the allocated hardware.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The easiest way to start a batch job is to submit a single Julia script, which can optionally also include a Julia environment with the job. However, for more complex jobs, with multiple inputs files etc., appbundles are likely more suitable.","category":"page"},{"location":"guides/jobs/#jobs-batch-script","page":"Jobs","title":"Script jobs","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"import JuliaHub\n_temp_path = mktempdir()\ncd(_temp_path)\nwrite(\"myscript.jl\", \"@warn \\\"Hello World!\\\"\\n\")","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The simplest job one can submit is a humble Julia script, together with an optional Julia environment (i.e. Project.toml, Manifest.toml, and/or Artifacts.toml). These jobs can be created with the JuliaHub.@script_str string macro, for inline instantiation:","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"JuliaHub.submit_job(\n    JuliaHub.script\"\"\"\n    @warn \"Hello World!\"\n    \"\"\",\n)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Alternatively, they can be created with the script function, which can load the Julia code from a script file:","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"cd(_temp_path) do # hide\nJuliaHub.submit_job(\n    JuliaHub.script(\"myscript.jl\"),\n)\nend # hide","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The string macro also picks up the currently running environment (i.e. Project.toml, Manifest.toml, and Artifacts.toml files), which then gets instantiated on JuliaHub when the script is started. If necessary, this can be disabled by appending the noenv suffix to the string macro.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"JuliaHub.script\"\"\"\n@warn \"Hello World!\"\n\"\"\"noenv","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"With the script function, you can also specify a path to directory containing the Julia package environment, if necessary.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"If an environment is passed with the job, it gets instantiated on the JuliaHub node, and the script is run in that environment. As such, any packages that are not available in the package registries or added via public Git URLs will not work. If that is the case, appbundles can be used instead to submit jobs that include private or local dependencies.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"rm(_temp_path, recursive=true)","category":"page"},{"location":"guides/jobs/#jobs-batch-appbundles","page":"Jobs","title":"Appbundles","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"A more advanced way of submitting a batch job is as an appbundle, which \"bundles up\" a whole directory and submits it together with the script. The Julia environment in the directory is also immediately added into the bundle.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"An appbundle can be constructed with the appbundle function, which takes as arguments the path to the directory to be bundled up, and a script within that directory. This is meant to be used for project directories where you have your Julia environment in the top level of the directory or repository. For example, you can submit an bundle from a submit script on the top level of your project directory as follows:","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"import JuliaHub # hide\n# We need to override the @__DIR__ here, because we actually construct a real appbundle # hide\nmacro __DIR__(); joinpath(dirname(dirname(pathof(JuliaHub))), \"test\", \"jobenvs\", \"job1\"); end # hide\nJuliaHub.submit_job(\n    JuliaHub.appbundle(@__DIR__, \"script.jl\"),\n    ncpu = 4, memory = 16,\n)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The bundler looks for a Julia environment (i.e. Project.toml, Manifest.toml, and/or Artifacts.toml files) at the root of the directory. If the environment does not exist (i.e. the files are missing), one is created. When the job starts on JuliaHub, this environment is instantiated.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"A key feature of the appbundle is that development dependencies of the environment (i.e. packages added with pkg> develop or Pkg.develop()) are also bundled up into the archive that gets submitted to JuliaHub (including any current, uncommitted changes). Registered packages are installed via the package manager via the standard environment instantiation, and their source code is not included in the bundle directly.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"When the JuliaHub job starts, the bundle is unpacked into the appbundle/ directory (relative to the starting working directory). E.g. if you have a mydata.dat file in the bundled directory, you can access it in the script at joinpath(\"appbundle\", \"mydata.dat\").","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Finally, a .juliabundleignore file can be used to exclude certain directories, by adding the relevant globs, similar to how .gitignore files work. In addition, .git directories are also automatically excluded from the bundle.","category":"page"},{"location":"guides/jobs/#Examining-job-configuration","page":"Jobs","title":"Examining job configuration","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The dryrun option to submit_job can be used to inspect the full job workload configuration that would be submitted to JuliaHub.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"import JuliaHub # hide\nJuliaHub.submit_job(\n    JuliaHub.script\"\"\"\n    println(\"hello world\")\n    \"\"\",\n    ncpu = 4, memory = 8,\n    env = Dict(\"ARG\" => \"value\"),\n    dryrun = true\n)","category":"page"},{"location":"guides/jobs/#Query,-extend,-kill","page":"Jobs","title":"Query, extend, kill","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The package has function that can be used to interact with running and past jobs. The jobs function can be used to list jobs, returning an array of Job objects.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"js = JuliaHub.jobs(limit=3)\njs[1]","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"If you know the name of the job, you can also query the job directly with job.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Main.setup_job_results_file!()","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"job = JuliaHub.job(\"jr-eezd3arpcj\")\njob.status\nJuliaHub.isdone(job)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Similarly, the kill_job function can be used to stop a running job, and the extend_job function can be used to extend the job's time limit.","category":"page"},{"location":"guides/jobs/#Waiting-on-jobs","page":"Jobs","title":"Waiting on jobs","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"A common pattern in a script is to submit one or more jobs, and then wait until the jobs complete, to then process their outputs. isdone can be used to see if a job has completed.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Main.MOCK_JULIAHUB_STATE[:jobs] = Dict(\"jr-novcmdtiz6\" => Dict(\"status\" => \"Running\"))","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"job = JuliaHub.job(\"jr-novcmdtiz6\")\nJuliaHub.isdone(job)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"empty!(Main.MOCK_JULIAHUB_STATE)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The wait_job function also provides a convenient way for a script to wait for a job to finish.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"job = JuliaHub.wait_job(\"jr-novcmdtiz6\")\nJuliaHub.isdone(job)","category":"page"},{"location":"guides/jobs/#Accessing-job-outputs","page":"Jobs","title":"Accessing job outputs","text":"","category":"section"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"There are two ways a JuliaHub job can store outputs that are directly related to a specific job[1]:","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Small, simple outputs can be stored by setting the ENV[\"RESULTS\"] environment variable. Conventionally, this is often set to a JSON object, and will act as a dictionary of key value pairs.\nFiles or directories can be uploaded by setting the ENV[\"RESULTS_FILE\"] to a local file path on the job. Note that directories are combined into a single tarball when uploaded.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"[1]: You can also e.g. upload datasets etc. But in that case the resulting data is not, strictly speaking, related to a specific job.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The values set via the RESULTS environment variable can be accessed with the .results field of a Job object:","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"Main.MOCK_JULIAHUB_STATE[:jobs] = Dict(\n    \"jr-novcmdtiz6\" => Dict(\n        \"outputs\" => \"\"\"\n        {\"user_param\": 2, \"output_value\": 4}\n        \"\"\"\n    )\n)\nimport JuliaHub\njob = JuliaHub.job(\"jr-novcmdtiz6\")","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"job.results","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"As the .results string is often a JSON object, you can use the the JSON.jl or JSON3.jl packages to easily parse it. For example","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"import JSON\nJSON.parse(job.results)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"When it comes to job result files, they can all be accessed via the .files field.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"job.files","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"The job_files function can be used to filter down to specific file types.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"JuliaHub.job_files(job, :result)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"And if you know the name of the file, you can also use the job_files to get the specific JobFile object for a particular file directly.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"jobfile = JuliaHub.job_file(job, :result, \"outdir.tar.gz\")","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"To actually fetch the contents of a file, you can use the download_job_file function on the JobFile objects.","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"empty!(Main.MOCK_JULIAHUB_STATE)","category":"page"},{"location":"guides/jobs/","page":"Jobs","title":"Jobs","text":"","category":"page"},{"location":"reference/exceptions/","page":"Exceptions","title":"Exceptions","text":"CurrentModule=JuliaHub","category":"page"},{"location":"reference/exceptions/#Exceptions","page":"Exceptions","title":"Exceptions","text":"","category":"section"},{"location":"reference/exceptions/","page":"Exceptions","title":"Exceptions","text":"JuliaHub.jl is designed in a way that the only errors it should throw under normal circumstances are subtypes of JuliaHubException (in addition to standard ArgumentErrors and MethodErrors etc. from invalid function calls). Any unhandled errors from JuliaHub.jl or its dependencies should be considered a bug.","category":"page"},{"location":"reference/exceptions/","page":"Exceptions","title":"Exceptions","text":"tip: Debugging JuliaHub.jl issues\nYou can also enable debug logging for JuliaHub, which will make JuliaHub.jl print out additional debug messages, by setting the JULIA_DEBUG environment variable:ENV[\"JULIA_DEBUG\"]=\"JuliaHub\"","category":"page"},{"location":"reference/exceptions/","page":"Exceptions","title":"Exceptions","text":"JuliaHubException\nAuthenticationError\nInvalidAuthentication\nInvalidRequestError\nJuliaHubConnectionError\nJuliaHubError\nPermissionError","category":"page"},{"location":"reference/exceptions/#JuliaHub.JuliaHubException","page":"Exceptions","title":"JuliaHub.JuliaHubException","text":"abstract type JuliaHubException <: Exception\n\nAbstract supertype of all JuliaHub.jl exception types.\n\n\n\n\n\n","category":"type"},{"location":"reference/exceptions/#JuliaHub.AuthenticationError","page":"Exceptions","title":"JuliaHub.AuthenticationError","text":"struct AuthenticationError <: JuliaHubException\n\nException thrown if the authentication fails. The .msg fields contains a human-readable error message.\n\n\n\n\n\n","category":"type"},{"location":"reference/exceptions/#JuliaHub.InvalidAuthentication","page":"Exceptions","title":"JuliaHub.InvalidAuthentication","text":"struct InvalidAuthentication <: JuliaHubException\n\nThis exception is thrown if the authentication token is invalid or has expired. Re-authenticating with JuliaHub.authenticate should generally be sufficient to resolve the issue.\n\n\n\n\n\n","category":"type"},{"location":"reference/exceptions/#JuliaHub.InvalidRequestError","page":"Exceptions","title":"JuliaHub.InvalidRequestError","text":"struct InvalidRequestError <: JuliaHubException\n\nAn exception thrown if the request was rejected by the backend due to request parameters that are inconsistent with the backend state. The .msg field contains the error message.\n\n\n\n\n\n","category":"type"},{"location":"reference/exceptions/#JuliaHub.JuliaHubConnectionError","page":"Exceptions","title":"JuliaHub.JuliaHubConnectionError","text":"struct JuliaHubConnectionError <: JuliaHubException\n\nAn exception thrown if there is a communication error with JuliaHub.\n\nThe .msg field contains the error message. If there is an underlying exception, it is stored in the .exception field.\n\n\n\n\n\n","category":"type"},{"location":"reference/exceptions/#JuliaHub.JuliaHubError","page":"Exceptions","title":"JuliaHub.JuliaHubError","text":"struct JuliaHubError <: JuliaHubException\n\nAn exception thrown if there is an unexpected response from or backend failure in JuliaHub.\n\nThe .msg field contains the error message. If there is an underlying exception, it is stored in the .exception field.\n\n\n\n\n\n","category":"type"},{"location":"reference/exceptions/#JuliaHub.PermissionError","page":"Exceptions","title":"JuliaHub.PermissionError","text":"struct PermissionError <: JuliaHubException\n\nThrown if the currently authenticated user does not have the necessary permissions to perform the operation. The .msg field contains the error message, and .response may contain the raw server response.\n\n\n\n\n\n","category":"type"},{"location":"reference/exceptions/#Index","page":"Exceptions","title":"Index","text":"","category":"section"},{"location":"reference/exceptions/","page":"Exceptions","title":"Exceptions","text":"Pages = [\"exceptions.md\"]","category":"page"},{"location":"reference/exceptions/","page":"Exceptions","title":"Exceptions","text":"","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"CurrentModule=JuliaHub","category":"page"},{"location":"guides/authentication/#guide-authentication","page":"Authentication","title":"Authentication","text":"","category":"section"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"In order to talk to a JuliaHub instance, you must authenticate with your user account, and have an authentication token. JuliaHub.jl aims to make it as easy and painless as possible to manage your authentication tokens.","category":"page"},{"location":"guides/authentication/#Overview","page":"Authentication","title":"Overview","text":"","category":"section"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"To use this package to authenticate in your current Julia session, you can call the authenticate function. It will return an Authentication object, containing the token.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"In principle, nearly every function here takes an (optional) auth keyword argument, that should be passed an Authentication object. However, this is usually not needed in practice, since JuliaHub.jl remembers your last authentication, and uses that as the default value for auth. Additionally, if there is no active authentication, calling any function that requires authentication will try to automatically authenticate (and may also trigger an interactive authentication). You can access the currently active authentication object with current_authentication.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"In order for authenticate to work, you must specify which JuliaHub instance to authenticate against (e.g. juliahub.com). There are two options for this:","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"You can configure the Julia package server to point to the JuliaHub instance by explicitly setting the JULIA_PKG_SERVER environment variable. See Local environments below to for more information on how to do it in local environments. In JuliaHub jobs, however, it is automatically set.\nYou can directly pass the JuliaHub instance URL to authenticate, e.g. JuliaHub.authenticate(\"juliahub.com\").","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"If a valid token for the JuliaHub instance exists on the disk, authenticate will use that directly. Otherwise it will attempt an interactive, browser-based authentication against the Julia instance. For the latter, under the hood, it uses the PkgAuthentication package to perform the interactive authentication","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"tip: Authentication token storage\nJuliaHub (or, more generally, package server) tokens are stored in the ~/.julia/servers/ directory. Inspecting the auth.toml files, or deleting them and re-authenticating may help if you are having unexpected issues with authentication.Note that this applies mainly when running JuliaHub.jl on your local computer. In JuliaHub cloud IDEs and jobs, the authentication token is actively managed by JuliaHub, and tampering with it may lead to other issues.","category":"page"},{"location":"guides/authentication/#JuliaHub-cloud-environments","page":"Authentication","title":"JuliaHub cloud environments","text":"","category":"section"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"When working with JuliaHub.jl in JuliaHub cloud environment, such as in JuliaHub Cloud IDEs or in JuliaHub jobs, you generally do not have to worry about authentication. The JULIA_PKG_SERVER should always be correctly set up, and the auth.toml file will also be present and up-to-date. This means that authenticate should automatically be able to authenticate your session, and you should never be prompted for interactive authentication.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"tip: No need to authenticate()\nAs in a JuliaHub environment everything is already set up and pointing to the correct server, you do not need to call authenticate yourself, nor set up JULIA_PKG_SERVER. You should be able to start using JuliaHub.jl functions right away.However, in JuliaHub jobs, you may still want to log the Authentication object in the job logs. You can use something like this at the top of your script:@info \"JuliaHub authentication token\" JuliaHub.authenticate()","category":"page"},{"location":"guides/authentication/#Local-environments","page":"Authentication","title":"Local environments","text":"","category":"section"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"When using JuliaHub.jl outside of JuliaHub (e.g. on your local laptop), you must explicitly configure the Julia package server, or pass the JuliaHub URL to authenticate.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"The Julia package server can be configured via the JULIA_PKG_SERVER environment variable, and there are a few options for this. In a live Julia session, you can simply set it via ENV, e.g.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"ENV[\"JULIA_PKG_SERVER\"] = \"juliahub.com\"","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"You can also do it this way in the Julia startup.jl script.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"Alternatively, on an UNIX-y system, you can set it in the shell or a script with e.g.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"export JULIA_PKG_SERVER=juliahub.com","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"If you're exclusively using the VS Code editor with the Julia extension, you can configure the package server in the UI directly, and it will apply to any REPLs you open in VS Code.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"(Image: Julia: Package Server option in Julia VS Code extension)","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"When running JuliaHub.jl on your local computer for the first time, or when you have not used JuliaHub.jl in a while, you will likely see something like:","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"julia> using JuliaHub\n\njulia> JuliaHub.authenticate()\nAuthentication required: please authenticate in browser.\nThe authentication page should open in your browser automatically, but you may need to switch to the opened window or tab. If the authentication page is not automatically opened, you can authenticate by manually opening the following URL: ...","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"Once you perform the browser-based authentication, the local auth.toml file gets updated, and JuliaHub.jl will be able to talk to the JuliaHub instance.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"note: JULIA_PKG_SERVER\nSetting JULIA_PKG_SERVER will also make the Julia package manager operations use the JuliaHub instance for Pkg operations. This is necessary for accessing packages on the private registries only accessible via JuliaHub. But it also means that you must have a valid JuliaHub authentication token in order to pull packages from the General registry.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"note: Enterprise JuliaHub instances\nEnterprise users generally have their own JuliaHub instance hosted under a different domain. In those cases, the recommended workflow is to globally set the JULIA_PKG_SERVER variable to point to the enterprise instance. This has the following benefits:JuliaHub.jl will, by default, authenticate with the server defined by JULIA_PKG_SERVER, opening a browser window if needed to acquire the token.\nAll package manager operations (i.e. pkg> or Pkg.*) will be routed through your enterprise JuliaHub instances as well. This is most useful if you need access to packages that are available in your company's private registries.Note: the same authentication token is used for package operations and JuliaHub operations.","category":"page"},{"location":"guides/authentication/","page":"Authentication","title":"Authentication","text":"","category":"page"},{"location":"internal/#Internal","page":"Internal","title":"Internal","text":"","category":"section"},{"location":"internal/#_PackageBundler","page":"Internal","title":"_PackageBundler","text":"","category":"section"},{"location":"internal/","page":"Internal","title":"Internal","text":"JuliaHub._PackageBundler.bundle\nJuliaHub._PackageBundler.path_filterer","category":"page"},{"location":"internal/#JuliaHub._PackageBundler.bundle","page":"Internal","title":"JuliaHub._PackageBundler.bundle","text":"bundle(dir; output = \"\",  force=false, allownoenv=false, verbose = true) -> String\n\nCreates a .tar file with the contents of dir as well as any packages that are either tracked by path (developed) outside dir or packages that are not tracked by the PkgServer. Artifacts are also bundled. The bundled packages and artifacts go into a .bundled/depot directory and is set up like a depot and can thus be made available by adding it to DEPOT_PATH.\n\n.git and globs listed in .juliabundleignore are excluded form the bundle.\n\nReturns the hex-encoded SHA256 of the Manifest.toml file that is packed into the appbundle. The return value is used when requesting a sysimage build, in which case we have to pass the manifest's hash with the submit request.\n\n\n\n\n\n","category":"function"},{"location":"internal/#JuliaHub._PackageBundler.path_filterer","page":"Internal","title":"JuliaHub._PackageBundler.path_filterer","text":"path_filterer(top)\n\nReturns a function that takes a file or directory path and checks whether that is excluded by the nearest .juliabundleignore file. The function will also ignore any .git files and directories.\n\nThe top argument specifies the highest directory up the tree that will be searched for the .juliabundleignore file.\n\nThe function will return false for any excluded files and true otherwise, and can be used as a predicate for filtering files that should be bundled.\n\n\n\n\n\n","category":"function"},{"location":"internal/#Index","page":"Internal","title":"Index","text":"","category":"section"},{"location":"internal/","page":"Internal","title":"Internal","text":"Pages = [\"internal.md\"]","category":"page"},{"location":"internal/","page":"Internal","title":"Internal","text":"","category":"page"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"CurrentModule = JuliaHub\nDocTestSetup = :(using JuliaHub)","category":"page"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"using JuliaHub","category":"page"},{"location":"reference/jobs/#job-apis","page":"Jobs","title":"Jobs","text":"","category":"section"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"JuliaHub.jl provides various methods to inspect and interact jobs that have been submitted to JuliaHub. It is also possible to get access to job outputs programmatically.","category":"page"},{"location":"reference/jobs/#Job-status-states","page":"Jobs","title":"Job status states","text":"","category":"section"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"A job will be in various \"states\" during its execution, and its current state can be inspected via the [.status] field.","category":"page"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"The following diagram illustrates the possible states a job can be in, and how it moves between them during a job lifecycle.","category":"page"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"stateDiagram-v2\ndirection LR\n[*] --> SUBMIT:::user\nSUBMIT --> Submitted\nSubmitted --> Running\nRunning --> FINISHED\nstate FINISHED {\n  direction LR\n  [*] --> Failed\n  [*] --> Stopped\n  [*] --> Completed\n}\nFINISHED:::user\n\nclassDef user fill:lightgray","category":"page"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"note: Refreshing a Job object\nAn instance of a Job object reflects the state of the job when the job function was called. If you want to inspect the current state of a job, you must first \"refresh\" the job object, which can simply be done with the help of the job function.job = JuliaHub.job(job)","category":"page"},{"location":"reference/jobs/#Reference","page":"Jobs","title":"Reference","text":"","category":"section"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"JuliaHub.JobReference\nJuliaHub.jobs\nJuliaHub.job\nJuliaHub.isdone\nJuliaHub.wait_job\nJuliaHub.kill_job\nJuliaHub.extend_job\nJuliaHub.JobLogMessage\nJuliaHub.job_logs\nJuliaHub.job_logs_buffered\nJuliaHub.job_logs_older!\nJuliaHub.job_logs_newer!\nJuliaHub.AbstractJobLogsBuffer\nJuliaHub.hasfirst\nJuliaHub.haslast\nJuliaHub.interrupt!\nJuliaHub.job_files\nJuliaHub.job_file\nJuliaHub.download_job_file\nJuliaHub.Job\nJuliaHub.JobStatus\nJuliaHub.JobFile\nJuliaHub.FileHash","category":"page"},{"location":"reference/jobs/#JuliaHub.JobReference","page":"Jobs","title":"JuliaHub.JobReference","text":"const JobReference :: Type\n\nA type constraint on the arguments of many jobs-related functions that is used to specify the job. A job reference must be either a Job object, or an AbstractString containing the unique job ID.\n\n\n\n\n\n","category":"type"},{"location":"reference/jobs/#JuliaHub.jobs","page":"Jobs","title":"JuliaHub.jobs","text":"JuliaHub.jobs(; [limit::Integer], [auth::Authentication]) -> Vector{Job}\n\nRetrieve the list of jobs, latest first, visible to the currently authenticated user.\n\nBy default, JuliaHub only returns up to 20 jobs. However, this default limit can be overridden by passing the limit keyword (which must be a positive integer).\n\nwarning: Non-dynamic job objects\nJob objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.job","page":"Jobs","title":"JuliaHub.job","text":"JuliaHub.job(job::JobReference; throw::Bool=true, [auth::Authentication]) -> Job\n\nFetch the details of a job based on the job reference ref. Will throw an InvalidRequestError if the job does not exist, or returns nothing if throw=false is passed.\n\nwarning: Non-dynamic job objects\nJob objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.isdone","page":"Jobs","title":"JuliaHub.isdone","text":"JuliaHub.isdone(::Job)\n\nA helper function to check if a Job is \"done\", i.e. its status is one of Completed, Stopped, or Failed.\n\nwarning: Non-dynamic job objects\nJob objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. As such, the result from this function may not represent the current live state of the job. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.wait_job","page":"Jobs","title":"JuliaHub.wait_job","text":"wait_job(\n    job::AbstractString;\n    interval::Integer = 30, [auth::Authentication]\n) -> Job\n\nBlocks until remote job referred to by the job reference job has completed, by polling it with every interval seconds. Returns an updated Job object.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.kill_job","page":"Jobs","title":"JuliaHub.kill_job","text":"JuliaHub.kill_job(job::JobRefererence; [auth::Authentication]) -> Job\n\nStop the job referred to by the job reference ref. Returns the updated Job object.\n\nSee also: Job.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.extend_job","page":"Jobs","title":"JuliaHub.extend_job","text":"JuliaHub.extend_job(job::JobReference, extension::Limit; [auth::Authentication]) -> Job\n\nExtends the time limit of the job referred to by the job reference ref by extension (Dates.Period, or Integer number of hours). Returns an updated Job object.\n\nSee Limit for more information on how the extension argument is interpreted. Note that Unlimited is not allowed as extension.\n\nSee also: Job.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.JobLogMessage","page":"Jobs","title":"JuliaHub.JobLogMessage","text":"struct JobLogMessage\n\nContains a single JuliaHub job log message, and has the following fields:\n\ntimestamp :: Union{ZonedDateTime, Nothing}: log message timestamp (in UTC)\nmessage :: Union{String, Nothing}: log message string. This generally corresponds to one line of printed output\n\nFields that can also be nothing may be missing for some log messages.\n\nSee also: job_logs, job_logs_buffered.\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/jobs/#JuliaHub.job_logs","page":"Jobs","title":"JuliaHub.job_logs","text":"JuliaHub.job_logs(job; offset::Integer = 0, [limit::Integer], [auth::Authentication]) -> Vector{JobLogMessage}\n\nFetches the log messages for the specified JuliaHub job. The job is specifed by passing the job name as a string, or by passing a Job object (i.e. job::Union{AbstractString,Job}). Returns the log messages as an array of JobLogMessage objects.\n\nOptionally, the function takes the following keyword arguments:\n\noffset::Integer: the offset of the first log message fetched (0 corresponds to the first message); for the first method, this defaults to 0; however, in the second (callback) case, if offset is not specified, any existing logs will be ignored.\nlimit::Integer: the maximum number of messages fetched (all by default)\n\nnote: No default limit\nThe limit keyword does not have a default limit, and so by default job_logs fetches all the log messages. This may take a while and require many requests to JuliaHub if the job has a huge number of log messages.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.job_logs_buffered","page":"Jobs","title":"JuliaHub.job_logs_buffered","text":"JuliaHub.job_logs_buffered(\n    [f::Base.Callable], job::Union{Job,AbstractString};\n    streaming::Bool=true, [offset::Integer],\n    [auth::Authentication]\n) -> AbstractJobLogsBuffer\n\nA lower-level function to work with log streams, and is particularly useful when working with jobs that have not finished yet and are actively producing new log messages.\n\nThe function accepts the following arguments:\n\nf :: Base.Callable: an optional callback function that gets called every time the buffer is updated. The callback must take two arguments: f(::AbstractJobLogsBuffer, ::AbstractVector). The first argument is the buffer object itself, and the second argument will be passed a read-only view of all the logs that have been loaded into the buffer, including the new ones.\njob :: Union{Job,AbstractString}: either the job name or a Job object.\nstreaming :: Bool: if set to true, the buffer object The streaming can be stopped with interrupt!.\noffset :: Integer: optional non-negative value to specify the starting point of the buffer\n\nInterface of the returned object\n\nReturns an instance of the abstract AbstractJobLogsBuffer type. These objects contain log messages (of type JobLogMessage), but not all the log messages are immediately available. Instead, at any given time the buffer represents a continous section of logs that can be extended in either direction.\n\nThe following functions can be used to interact with log buffers: job_logs_newer!, job_logs_older!, JuliaHub.hasfirst, JuliaHub.haslast. Additionally, the objects will have a .logs :: Vector{JobLogMessage} property that can be used to access the log messages that have been loaded into the buffer.\n\nSee also: job_logs, Job.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.job_logs_older!","page":"Jobs","title":"JuliaHub.job_logs_older!","text":"JuliaHub.job_logs_older!(\n    buffer::AbstractJobLogsBuffer; [count::Integer], [auth::Authentication]\n) -> AbstractJobLogsBuffer\n\nUpdates the AbstractJobLogsBuffer object by adding up to count log messages to the beginning of the buffer. If count is omitted, it will seek all the way to the beginning of the logs.\n\nIf all the logs have already been loaded into the buffer (i.e. JuliaHub.hasfirst(buffer) is true), the function is a no-op.\n\nSee also: job_logs_buffered, job_logs_newer!.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.job_logs_newer!","page":"Jobs","title":"JuliaHub.job_logs_newer!","text":"JuliaHub.job_logs_newer!(\n    buffer::AbstractJobLogsBuffer; [count::Integer], [auth::Authentication]\n) -> AbstractJobLogsBuffer\n\nUpdates the AbstractJobLogsBuffer object by adding up to count log messages to the end of the buffer. If count is omitted, it will seek all the way to the end of the current logs.\n\nFor a finished job, if all the logs have already been loaded into the buffer (i.e. JuliaHub.haslast(buffer) is true), the function is a no-op. If the buffer is actively streaming new logs for a running job,  then the function is also a no-op.\n\nSee also: job_logs_buffered, job_logs_older!.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.AbstractJobLogsBuffer","page":"Jobs","title":"JuliaHub.AbstractJobLogsBuffer","text":"abstract type AbstractJobLogsBuffer\n\nSupertype of possible objects returned by job_logs_buffered. See the job_logs_buffered function for a description of the interface.\n\n\n\n\n\n","category":"type"},{"location":"reference/jobs/#JuliaHub.hasfirst","page":"Jobs","title":"JuliaHub.hasfirst","text":"JuliaHub.hasfirst(::AbstractJobLogsBuffer) -> Bool\n\nDetermines whether the job log buffer has the first message of the job logs.\n\nSee also: haslast, job_logs_buffered.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.haslast","page":"Jobs","title":"JuliaHub.haslast","text":"JuliaHub.haslast(::AbstractJobLogsBuffer) -> Bool\n\nDetermines whether the job log buffer has the last message of the job logs. Note that if the job has not finished, this will always be false, since the job may produce additional logs.\n\nSee also: hasfirst, job_logs_buffered.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.interrupt!","page":"Jobs","title":"JuliaHub.interrupt!","text":"JuliaHub.interrupt!(::AbstractJobLogsBuffer; wait::Bool=true)\n\nCan be use to interrupt the asynchronous log streaming task. If the log buffer is not streaming, this function is a no-op.\n\nNote that the effect of JuliaHub.interrupt! may not be immediate and the function will block until the task has stopped. wait = false can be passed to make interrupt! return immediately, but in that case the buffer may stream for a little while longer.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.job_files","page":"Jobs","title":"JuliaHub.job_files","text":"JuliaHub.job_files(job::Job, [filetype::Symbol]) -> Vector{JobFile}\n\nReturn the list of inputs and/or output files associated job.\n\nThe optional filetype argument should be one of :input, :source, :result or :project, and can be used to filter the file list down to either just job input files (such as the appbundle or Julia environment files), or output files (such as the one uploaded via RESULTS_FILE).\n\nNote: job_file(job) is equivalent to job.files, and the latter is preferred. This function is primarily meant to be used when filtering by file type.\n\nSee also: Job.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.job_file","page":"Jobs","title":"JuliaHub.job_file","text":"JuliaHub.job_file(job::Job, type::Symbol, filename::AbstractString) -> JobFile | Nothing\n\nSearches for a job output file of a specified type and with the specific filename for job job, or nothing if the file was not found.\n\ntype should be one of the standard job file types. See JobFile and job_files for more information.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.download_job_file","page":"Jobs","title":"JuliaHub.download_job_file","text":"JuliaHub.download_job_file(file::JobFile, path::AbstractString; [auth]) -> String\nJuliaHub.download_job_file(file::JobFile, io::IO; [auth])\n\nDownloads a JobFile to a local path. Alternative, writeable stream object can be passed as the second argument to write the contents directly into the stream.\n\nWhen a local path is passed, it returns the path (which can be useful when calling the function as e.g. JuliaHub.download_job_file(file, tempname()))). When an IO object is passed, it returns nothing.\n\nFor example, to download a file into a temporary file:\n\njulia> file = JuliaHub.job_file(JuliaHub.job(\"jr-eezd3arpcj\"), :result, \"outdir.tar.gz\")\nJuliaHub.JobFile outdir.tar.gz (jr-eezd3arpcj, :result, 632143 bytes)\nsha2_256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nUploaded: 2023-03-15T07:59:29.473898+00:00\n\njulia> tmp = tempname()\n\"/tmp/jl_nE3uvkZwvC\"\n\njulia> JuliaHub.download_job_file(file, tmp)\n\"/tmp/jl_BmHgj8rQXe\"\n\njulia> bytes2hex(open(sha2_256, tmp))\n\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"\n\nAlternatively, you can also download the file into a writable IO stream, such as IOBuffer:\n\njulia> buffer = IOBuffer();\n\njulia> JuliaHub.download_job_file(file, buffer)\n\njulia> bytes2hex(sha2_256(take!(buffer)))\n\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"\n\nSee also: Job, JobFile.\n\n\n\n\n\n","category":"function"},{"location":"reference/jobs/#JuliaHub.Job","page":"Jobs","title":"JuliaHub.Job","text":"struct Job\n\nRepresents a single job submitted to JuliaHub. Objects have the following properties:\n\nid :: String: the unique, automatically generated ID of the job\nalias :: String: a non-unique, but descriptive alias for the job (often set by e.g. applications)\nstatus :: JobStatus: a string-like JobStatus object storing the state of the job\nenv :: Dict: a dictionary of environment variables that were set when the job was submitted\nresults :: String: the output value set via ENV[\"RESULTS\"] (an empty string if it was not explicitly set)\nfiles :: Vector{JobFiles}: a list of JobFile objects, representing the input and output files of the job (see: job_files, job_file, download_job_file).\n\nSee also: job, jobs.\n\nwarning: Non-dynamic job objects\nJob objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/jobs/#JuliaHub.JobStatus","page":"Jobs","title":"JuliaHub.JobStatus","text":"struct JobStatus\n\nType of the .status field of a Job object, representing the current state of the job. Should be one of: Submitted, Running, Failed, Stopped, Completed.\n\nIn practice, the .status field should be treated as string and only used in string comparisons.\n\nSee also: isdone.\n\njulia> job = JuliaHub.job(\"jr-novcmdtiz6\")\nJuliaHub.Job: jr-novcmdtiz6 (Completed)\n submitted: 2023-03-15T07:56:50.974+00:00\n started:   2023-03-15T07:56:51.251+00:00\n finished:  2023-03-15T07:56:59.000+00:00\n files:\n  - code.jl (input; 3 bytes)\n  - code.jl (source; 3 bytes)\n  - Project.toml (project; 244 bytes)\n  - Manifest.toml (project; 9056 bytes)\n outputs: \"{}\"\n\njulia> job.status == \"Submitted\"\nfalse\n\njulia> job.status == \"Completed\"\ntrue\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/jobs/#JuliaHub.JobFile","page":"Jobs","title":"JuliaHub.JobFile","text":"struct JobFile\n\nA reference to a job input or output file, with the following properties:\n\n.name :: String: the name of the Job this file is attached to\n.type :: Symbol: indicated the file type (see below)\n.filename :: String: file name\n.size :: Int: size of the file in bytes (reported to be zero in cases where the file contents is missing)\n.hash :: Union{FileHash, Nothing}: a FileHash object containing the file hash, but may also be missing (nothing) in some cases, like when the file upload has not completed yet.\n\nThe file is uniquely identified by the (job, type, filename) triplet.\n\nThe type of the file should be one of:\n\n:input: various job input files, which includes code, environment, and appbundle files.\n:source: source\n:project: Julia project environment files (e.g. Project.toml, Manifest.toml, Artifacts.toml)\n:result: output results file, defined via the RESULTS_FILE environment variable in the job\n\nNote that some files are duplicated under multiple categories.\n\nSee also: Job, job_files, job_file.\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/jobs/#JuliaHub.FileHash","page":"Jobs","title":"JuliaHub.FileHash","text":"struct FileHash\n\nStores a hash and the algorithm used to calcute it. The object has the following properties:\n\n.algorithm :: Symbol: hash algorithm\n.hash :: Vector{UInt8}: the hash as a sequence of bytes\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\n\n\n\n\n","category":"type"},{"location":"reference/jobs/#Index","page":"Jobs","title":"Index","text":"","category":"section"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"Pages = [\"jobs.md\"]","category":"page"},{"location":"reference/jobs/","page":"Jobs","title":"Jobs","text":"","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"CurrentModule=JuliaHub\nDocTestSetup = :(using JuliaHub)","category":"page"},{"location":"reference/job-submission/#job-submission","page":"Job submission","title":"Job submission","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"On JuliaHub you can submit jobs, which are user-defined workloads that get allocated a dedicated compute capacity. For example, this includes running scripts in batch computations, cloud IDEs, interactive notebooks and so on. The functions and types here deal with starting up such jobs and apps. The functions to inspect running or finished jobs are documented separately.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"Pages=[\"job-submission.md\"]\nDepth=2:3","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"A complete JuliaHub job workload is defined by the following configuration pieces:","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"Job configuration. This specifies the computation that gets run, and includes information such as the type of computation (batch script vs starting a GUI application), any arguments or Julia code that gets passed to the job etc.\nAs of now, there are three general types of jobs that can be run on JuliaHub:\nBatch jobs: non-interactive Julia scripts\nDefault Applications: special, built-in, interactive applications (such as IDEs or product-specific dashboards)\nExternal package applications: Julia packages that can be run as either interactive or non-interactive jobs\nEach of these categories are configured slightly differently, and are described in more detail below.\nCompute configuration. This specifies the hardware and cluster topology that will be used to execute the job (such as the number of CPUs per node, whether there is a GPU present, the number of nodes), how the Julia processes are configured (e.g. single process per node vs process per cpu), and other low-level technical configuration.\nRuntime parameters. These are various additional parameters that control how a job behaves. Currently, this is limited to passing environment variables to the jobs, overriding the job's name in the UI, and to various configuration options related to running interactive jobs.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"In the JuliaHub.jl code, the first two categories are encapsulated by the AbstractJobConfig and ComputeConfig types. These two, together with the additional runtime parameters, make up a WorkloadConfig object that can be submitted to JuliaHub for executing with the submit_job function.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"The following sections on this page explain the different aspects of job submission in more detail. See the guide on submitting batch jobs to see more practical examples of how to submit jobs.","category":"page"},{"location":"reference/job-submission/#Compute-configuration","page":"Job submission","title":"Compute configuration","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"JuliaHub supports a predefined set of node configurations, each of which have a specific number of CPUs, memory, GPUs etc. A JuliaHub job must pick one of these node types to run on, although a distributed job can run across multiple instances of the same node type. A list of these node specifications can be obtained with the nodespecs function (returning a list of NodeSpec objects).","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"julia> JuliaHub.nodespecs()\n9-element Vector{JuliaHub.NodeSpec}:\n JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 0.17/hr =#; ncpu=2, memory=8, ngpu=false, exactmatch=true)\n JuliaHub.nodespec(#= r6: 3.5 GHz Intel Xeon Platinum 8375C, 0.22/hr =#; ncpu=2, memory=16, ngpu=false, exactmatch=true)\n JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 0.33/hr =#; ncpu=4, memory=16, ngpu=false, exactmatch=true)\n JuliaHub.nodespec(#= r6: 3.5 GHz Intel Xeon Platinum 8375C, 0.42/hr =#; ncpu=4, memory=32, ngpu=false, exactmatch=true)\n JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 0.65/hr =#; ncpu=8, memory=32, ngpu=false, exactmatch=true)\n JuliaHub.nodespec(#= r6: 3.5 GHz Intel Xeon Platinum 8375C, 1.3/hr =#; ncpu=8, memory=64, ngpu=false, exactmatch=true)\n JuliaHub.nodespec(#= m6: 3.5 GHz Intel Xeon Platinum 8375C, 2.4/hr =#; ncpu=32, memory=128, ngpu=false, exactmatch=true)\n JuliaHub.nodespec(#= p2: Intel Xeon E5-2686 v4 (Broadwell), 1.4/hr =#; ncpu=4, memory=61, ngpu=true, exactmatch=true)\n JuliaHub.nodespec(#= p3: Intel Xeon E5-2686 v4 (Broadwell), 4.5/hr =#; ncpu=8, memory=61, ngpu=true, exactmatch=true)","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"While you can manually index into the list returned by nodespecs, that is generally inconvenient. Instead, the nodespec function should be used to find a suitable node for a particular job.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"julia> JuliaHub.nodespec(ncpu=2, memory=8)\nNode: 3.5 GHz Intel Xeon Platinum 8375C\n - GPU: no\n - vCores: 2\n - Memory: 8 Gb\n - Price: 0.17 $/hr","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"By default, nodespec finds the smallest node that satisfies the specified requirements. However, it also supports the exactmatch argument, which can be use to find the exactly matching node configuration.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"julia> JuliaHub.nodespec(ncpu=3, memory=5; exactmatch=true)\nERROR: InvalidRequestError: Unable to find a nodespec: ncpu=3 memory=5 gpu=false\nStacktrace:\n ...\n\njulia> JuliaHub.nodespec(ncpu=3, memory=5)\nNode: 3.5 GHz Intel Xeon Platinum 8375C\n - GPU: no\n - vCores: 4\n - Memory: 16 Gb\n - Price: 0.33 $/hr","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"By default, JuliaHub jobs run on a single node. However, for a distributed job, additional nodes can be allocated to a job by specifying the nnodes parameter. In that case, a Julia process is started on each node, and the additional nodes are linked to the main process via the Distributed module.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"While by default only a single Julia process is started on each node, by setting the process_per_cpu parameter, multiple Julia processes are started on the same node. The processes are isolated from each other by running in separate containers, but they share the CPUs, GPUs, and most crucially, the memory.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"See ComputeConfig and submit_job for more details on how exactly to set up this configuration.","category":"page"},{"location":"reference/job-submission/#jobs-runtime-config","page":"Job submission","title":"Runtime configuration","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"The submit_job function accepts various additional parameters that control aspects of the job. See the function's docstring for more details.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"Parameter Description\nname can be used to override the name of the job shown in the UI\nproject specifies the JuliaHub project UUID that the job is associate with\ntimelimit sets the time limit after which the job gets killed\nenv environment variables set at runtime","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"As an example, to have an environment variable set while the job is running, you could call submit_job as follows:","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"import JuliaHub # hide\nJuliaHub.submit_job(\n    JuliaHub.script\"\"\"\n    @info \"Extracting 'MY_PARAMETER'\" get(ENV, \"MY_PARAMETER\", nothing)\n    \"\"\",\n    env = Dict(\"MY_PARAMETER\" => \"example value\"),\n    dryrun = true\n)","category":"page"},{"location":"reference/job-submission/#jobs-batch","page":"Job submission","title":"Batch jobs","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"Batch jobs are Julia scripts with (optional) associated Julia package environments (Project.toml, Manifest.toml and/or Artifacts.toml) that run on the cluster non-interactively.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"See also: @script_str, script, appbundle for more details, and the guide on submitting batch jobs for a tutorial.","category":"page"},{"location":"reference/job-submission/#Specifying-the-job-image","page":"Job submission","title":"Specifying the job image","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"JuliaHub batch jobs can run in various container images. Different JuliaHub products often have their own image tailored for the application (e.g. that come with custom sysimages to reduce load times).","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"The list of all images available to the user can be obtained with batchimages, and a specific one can be picked out with batchimage. The latter function is particularly useful when submitting jobs.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"JuliaHub.submit_job(\n    JuliaHub.BatchJob(\n        JuliaHub.script\"\"\"\n        using SpecialProductModule\n        SpecialProductModule.run()\n        \"\"\",\n        image = JuliaHub.batchimage(\"specialproduct\"),\n    )\n)","category":"page"},{"location":"reference/job-submission/#jobs-default-apps","page":"Job submission","title":"Default Applications","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"compat: Experimental feature\nStarting application jobs with JuliaHub.jl is considered to be experimental. The APIs are likely to change in future JuliaHub.jl version.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"Default applications are the JuliaHub-built in applications (such as dashboards and IDEs), generally associated with specific JuliaHub products. Specific examples of default applications available to everyone include the Pluto notebooks and the Julia IDE.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"julia> apps = JuliaHub.applications()\n7-element Vector{JuliaHub.AbstractJuliaHubApp}:\n JuliaHub.application(:default, \"Linux Desktop\")\n JuliaHub.application(:default, \"Julia IDE\")\n JuliaHub.application(:default, \"Pluto\")\n JuliaHub.application(:default, \"Windows Workstation\")\n JuliaHub.application(:package, \"RegisteredPackageApp\")\n JuliaHub.application(:package, \"CustomDashboardApp\")\n JuliaHub.application(:user, \"ExampleApp.jl\")\n\njulia> JuliaHub.application(:default, \"Pluto\")\nDefaultApp\n name: Pluto\n key: pluto","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"A JuliaHub job that launches an application can be started by passing the object returned by the application function to submit_job.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"import JuliaHub # hide\nJuliaHub.submit_job(\n    JuliaHub.application(:default, \"Pluto\"),\n    ncpu = 8, memory = 16,\n)","category":"page"},{"location":"reference/job-submission/#jobs-packages","page":"Job submission","title":"External package applications","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"compat: Experimental feature\nStarting package application jobs with JuliaHub.jl is considered to be experimental. The APIs are likely to change in future JuliaHub.jl version.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"Specially crafted Julia packages can also be launched as JuliaHub jobs. They are either automatically picked up from the packages registries served by the JuliaHub instance (PackageApp), or added by users themselves (UserApp) via a Git URL.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"A package application must have a top-level package environment (i.e. it has a Project.toml the declares a name and a UUID), and it must have a entry script at bin/main.jl. When a package job starts, the package is added to an environment and bin/main.jl is called. Note that the bin/main.jl script does not have any access to any of the package dependencies.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"The applications function can list all available packages (with filtering for user or registered). JuliaHub.application can be used to pick a package by name. It works for both registered packages","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"JuliaHub.submit_job(\n    JuliaHub.application(:package, \"RegisteredPackageApp\"),\n    ncpu = 4,\n)","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"and for private, user applications","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"JuliaHub.submit_job(\n    JuliaHub.application(:user, \"ExampleApp.jl\"),\n    ncpu = 4,\n    env = Dict(\"example_parameter\" => \"2\")\n)","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"tip: Environment variables\nEnvironment variables (i.e. env) are a common way to communicate options and settings to package applications.","category":"page"},{"location":"reference/job-submission/#Reference","page":"Job submission","title":"Reference","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"NodeSpec\nnodespecs\nnodespec\nBatchImage\nbatchimages\nbatchimage\nAbstractJobConfig\nBatchJob\nscript\n@script_str\nappbundle\nComputeConfig\nsubmit_job\nLimit\nUnlimited\nWorkloadConfig","category":"page"},{"location":"reference/job-submission/#JuliaHub.NodeSpec","page":"Job submission","title":"JuliaHub.NodeSpec","text":"struct NodeSpec\n\nStores information about a compute node that can be allocated for JuliaHub jobs. The list of all available node specifications can be accessed with nodespecs, or specific ones searched with nodespec.\n\njulia> JuliaHub.nodespec()\nNode: 3.5 GHz Intel Xeon Platinum 8375C\n - GPU: no\n - vCores: 2\n - Memory: 8 Gb\n - Price: 0.17 $/hr\n\nThey can be used to contruct explicit compute configuration objects when submitting JuliaHub jobs.\n\nSee also: submit_job, ComputeConfig.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.nodespecs","page":"Job submission","title":"JuliaHub.nodespecs","text":"JuliaHub.nodespecs(; auth::Authentication) -> Vector{NodeSpec}\n\nQuery node specifications available on the current server, returning a list of NodeSpec objects.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.nodespec","page":"Job submission","title":"JuliaHub.nodespec","text":"JuliaHub.nodespec(\n    [nodes::Vector{NodeSpec}];\n    ncpu::Integer=1, ngpu::Integer=false, memory::Integer=1,\n    exactmatch::Bool=false, throw::Bool=true,\n    [auth::Authentication]\n) -> Union{NodeSpec, Nothing}\n\nFinds the node matching the specified node parameters. Throws an InvalidRequestError if it is unable to find a node with the specific parameters. However, if throw is set to false, it will return nothing instead in that situation.\n\nBy default, it searches for the smallest node that has the at least the specified parameters (prioritizing GPU count, CPU count, and memory in this order when deciding). If exactmatch is set to true, it only returns a node specification if it can find one that matches the parameters exactly.\n\nA list of nodes (e.g. from nodespecs) can also be passed, so that the function does not have to query the server for the list. When this method is used, it is not necessary to pass auth.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.BatchImage","page":"Job submission","title":"JuliaHub.BatchImage","text":"struct BatchImage\n\nRepresents an available JuliaHub batch job image. These can be passed to BatchJob to specify which underlying job image will be used for the job.\n\nA list of available batch images can be accessed with batchimages and specific images can be constructed with batchimage.\n\ncompat: No public constructors\nObjects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.\n\nSee also: batchimages, batchimage, BatchJob, script, appbundle.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.batchimages","page":"Job submission","title":"JuliaHub.batchimages","text":"JuliaHub.batchimages([product::AbstractString]; [auth::Authentication]) -> Vector{BatchImage}\n\nReturn the list of all batch job images available to the currently authenticated user, as a list of BatchImage objects. These can be passed to BatchJob.\n\nOptionally, by passing a product identifier, the list can be narrowed down to images available for that specific product.\n\nnote: Batch images on older instances\nWhen using the package with an older JuliaHub instance (<= 6.1), the non-default batch images show up with @legacy as the product name. This indicates that the package is using an older API, and not that the images themselves are outdated.\n\nSee also: BatchImage, batchimage, BatchJob, script, appbundle.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.batchimage","page":"Job submission","title":"JuliaHub.batchimage","text":"JuliaHub.batchimage(\n    [product::AbstractString, [image::AbstractString]];\n    throw::Bool=true, [auth::Authentication]\n) -> BatchImage\n\nPick a product job batch image from the list of all batch image, returning a BatchImage object. If image is omitted, it will return the default image corresponding to product. If product is omitted as well, it will return the default image of the instance (generally the standard Julia batch image).\n\nWill throw an InvalidRequestError if the specified image can not be found. If throw=false, it will return nothing instead in this situation.\n\nnote: Batch images on older instances\nWhen using the package with an older JuliaHub instance (<= 6.1), the non-default batch images show up with @legacy as the product name. This indicates that the package is using an older API, and not that the images themselves are outdated.\n\nSee also: BatchImage, batchimages, BatchJob, script, appbundle.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.AbstractJobConfig","page":"Job submission","title":"JuliaHub.AbstractJobConfig","text":"abstract type AbstractJobConfig\n\nAbstract supertype of all application configuration types that can be passed to submit_job for submission as a JuliaHub job. The package has built-in support for the following application configurations:\n\nJuliaHub.BatchJob\nJuliaHub.ApplicationJob\nJuliaHub.PackageApp\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.BatchJob","page":"Job submission","title":"JuliaHub.BatchJob","text":"struct BatchJob <: AbstractJobConfig\n\nRepresents the application configuration of a JuliaHub batch job. A batch job is defined by the following information:\n\nThe Julia code that is to be executed in the job.\nJulia package environment (i.e. Project.toml, Manifest.toml) and other files, such as the appbundle.\nThe underlying batch job container image (see also batchimages), which defaults to the standard Julia image by default.\n\nInstances of this types should normally not be constructed directly, and the following functions should be used instead:\n\nscript or @script_str: for submitting simple Julia scripts or code snippets\nappbundle: for submitting more complex \"appbundles\" that include additional file, private or modified package dependencies etc.\n\nOptional arguments\n\nimage  :: Union{BatchImage, Nothing}: can be used to specify which product's batch job image will be used when running the job, by passing the appropriate BatchImage object (see also: batchimage and batchimages). If set to nothing (the default), the job runs with the default Julia image.\nsysimage :: Bool: if set to true, requests that a system image is built from the job's Manifest.toml file before starting the job. Defaults to false.\n\ncompat: JuliaHub compatibility\nThe sysimage = true option requires JuliaHub 6.3 to have an effect. When running against older JuliaHub versions, it does not have an effect.\n\nConstructors\n\nBatchJob(::BatchJob; [image::BatchImage], [sysimage::Bool]) -> BatchJob\n\nConstruct a BatchJob, but override some of the optional arguments documented above. When the argument is omitted, the value from the underlying BatchJob object is used. This is the only constructor that is part of the public API.\n\nThis method is particularly useful when used in in combination with the @script_str string macro, to be able to specify the job image or trigger a sysimage build. For example, the following snippet will set a different batch image for the script-type job:\n\nJuliaHub.BatchJob(\n    JuliaHub.script\"\"\"\n    @info \"Hello World\"\n    \"\"\",\n    image = JuliaHub.batchimage(\"...\")\n)\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.script","page":"Job submission","title":"JuliaHub.script","text":"JuliaHub.script(...) -> BatchJob\n\nConstructs the configuration for a script-type batch job, returning the respective BatchJob object that can then be passed to submit_job. A script-type batch job is defined by the following:\n\nA user-provided Julia script that gets executed on the server. Note that no validation of the input code is done.\nAn optional Julia package environment (e.g. Project.toml, Manifest.toml and Artifacts.toml). If any of the TOML files are provided, they must parse as valid TOML files, but no further validation is done client-side.\nIf the manifest is not provided, the project environment must be instantiated from scratch, generally pulling in the latest versions of all the dependencies (although [compat] sections are honored).\nIt is also fine to omit the project file, and just provide the manifest, and the environment defined by the manifest still gets instantiated. If both are omitted, the job runs in an empty environment.\nA JuliaHub job image, which determines the container environment that will be used to execute the code in (see batchimage, batchimages, BatchImage). If omitted, the default Julia image is used.\n\nSee also the @script_str string macro to more easily submit simple scripts that are defined in code.\n\nMethods\n\nscript(\n    scriptfile::AbstractString;\n    [project_directory::AbstractString], [image::BatchImage], [sysimage::Bool]\n) -> BatchJob\n\nConstructs a script-type batch job configuration the will execute the code in scriptfile. Optionally, a path to a project environment directory can be passed via project_directory, which will be searched for the environment TOML files, and a job image can be specified via image.\n\nscript(;\n    code::AbstractString,\n    [project::AbstractString], [manifest::AbstractString], [artifacts::AbstractString],\n    [image::BatchImage], [sysimage::Bool]\n) -> BatchJob\n\nA lower-level method that can be used to construct the script-type BatchJob configuration directly in memory (i.e. without having to write out intermediate files).\n\nThe code keyword argument is mandatory and will specify contents of the Julia script that gets executed on the server. The Julia project environment can be specified by passing the contents of the TOML files via the corresponding arguments (project, manifest, artifacts). The job image can be specified via image.\n\nnote: Optional arguments\nSee BatchJob for a more thorough description of the optional arguments.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.@script_str","page":"Job submission","title":"JuliaHub.@script_str","text":"JuliaHub.@script_str -> JuliaHub.BatchJob\n\nA string macro to conveniently construct a script-type batch job configuration (BatchJob) that can be submitted as a JuliaHub job.\n\nscript = JuliaHub.script\"\"\"\n@info \"Hello World!\"\n\"\"\"\n\nThis allows for an easy submission of simple single-script jobs to JuliaHub:\n\nJuliaHub.submit_job(\n    JuliaHub.script\"\"\"\n    @info \"Hello World!\"\n    \"\"\"\n)\n\nBy default, the macro picks up the currently active Julia project environment (via Base.active_project()), and attaches the environment .toml files to the script. To disable this, you can call the macro with the noenv suffix, e.g.\n\nscript = JuliaHub.script\"\"\"\n@info \"Hello World!\"\n\"\"\"noenv\n\nHowever, if your local environment has development dependencies, you likely need to use an appbundle instead (see appbundle).\n\nnote: Using a different job image\nThere is no way to specify the job image with the string macro, and it will use the default Julia image. To use a different job image, you should either use the script function, either by fully constructing the batch job configuration with the keyword arguments, or by using the BatchJob(::BatchJob; image=...) method.JuliaHub.submit_job(\n    JuliaHub.BatchJob(\n        JuliaHub.script\"\"\"\n        @info \"Hello World!\"\n        \"\"\",\n        image = JuliaHub.batchimage(...)\n    )\n)You can also use this pattern to set the sysimage option.\n\n\n\n\n\n","category":"macro"},{"location":"reference/job-submission/#JuliaHub.appbundle","page":"Job submission","title":"JuliaHub.appbundle","text":"JuliaHub.appbundle(\n    directory::AbstractString, codefile::AbstractString;\n    [image::BatchImage], [sysimage::Bool]\n) -> BatchJob\nJuliaHub.appbundle(\n    directory::AbstractString;\n    code::AbstractString, [image::BatchImage], [sysimage::Bool]\n) -> BatchJob\n\nConstruct an appbundle-type JuliaHub batch job configuration. An appbundle is a directory containing a Julia environment that is bundled up, uploaded to JuliaHub, and then unpacked and instantiated as the job starts.\n\nThe code that gets executed is read from codefile, which should be a path to Julia source file relative to directory.\n\nJuliaHub.appbundle(@__DIR__, \"my-script.jl\")\n\nAlternatively, if codefile is omitted, the code can be provided as a string via the code keyword argument.\n\nJuliaHub.AppBundle(\n    @__DIR__,\n    code = \"\"\"\n    @show ENV\n    \"\"\"\n)\n\nSee BatchJob for a description of the optional arguments.\n\nExtended help\n\nThe following should be kept in mind about how appbundles are handled:\n\nThe bundler looks for a Julia environment (i.e. Project.toml and/or Manifest.toml files) at the root of the directory. If the environment does not exist (i.e. the files are missing), the missing files are created. If the manifest is missing, then the environment is re-instantiated from scratch based on the contents of Project.toml. The generated files will also be left in the user-provided directory directory.\nDevelopment dependencies of the environment (i.e. packages added with pkg> develop or Pkg.develop()) are also bundled up into the archive that gets submitted to JuliaHub (including any current, uncommitted changes). Registered packages are installed via the package manager via the standard environment instantiation, and their source code is not included in the bundle directly.\nWhen the JuliaHub job starts, the bundle is unpacked and the job's starting working directory is set to the appbundle/ directory, and you can e.g. load the data from those files with just read(\"my-data.txt\", String).\nNote that @__DIR__ points elsewhere and, relatedly, include in the main script should be used with an absolute path (e.g. include(joinpath(pwd(), \"my-julia-file.jl\"))).\ncompat: JuliaHub 6.2 and older\nOn some older JuliaHub versions (6.2 and older), the working directory was set to the parent directory of appbundle/, and so it was necessary to do joinpath(\"appbundle\", \"mydata.dat\") to load the code.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.ComputeConfig","page":"Job submission","title":"JuliaHub.ComputeConfig","text":"struct ComputeConfig\n\nThis type encapsulates the configuration of a jobs's compute cluster, including the hardware configuration and the cluster topology.\n\nSee also: submit_job.\n\nConstructors\n\nJuliaHub.ComputeConfig(\n    node::NodeSpec;\n    nnodes::Integer = 1,\n    process_per_node::Bool = true,\n    elastic::Bool = false,\n)\n\nnode: a NodeSpec object that specifies the hardware of a single node.\nnnodes::Union{Integer, Tuple{Integer, Integer}} = 1: specifies the number of nodes of type node that will be allocated. Alternatively, a two-integer tuple can also be passed, where the first value specifies the minimum number of nodes required to start a job. By default, a single-node job is started.\nprocess_per_node::Bool = true: if true, there will only be a single Julia process per node, and the total number of Julia processes will be nnodes. If set to false, however, each core on each node will be allocated a separate Julia process (running in an isolated container on the same node), and so the total number of Julia processes will be nnodes × ncpu, and it will essentially always be a multi-process job.\nelastic::Bool = false: if set, the job will be started in an elastic cluster mode. In this case, a minimum number of nnodes must not be passed.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.submit_job","page":"Job submission","title":"JuliaHub.submit_job","text":"JuliaHub.submit_job(\n    app::Union{AbstractJuliaHubApp, AbstractJobConfig},\n    [compute::ComputeConfig];\n    # Compute keyword arguments\n    ncpu::Integer = 1, ngpu::Integer = 0, memory::Integer = 1,\n    nnodes::Integer = 1, minimum_nnodes::Union{Integer,Nothing} = nothing,\n    elastic::Bool = false,\n    process_per_node::Bool = true,\n    # Runtime configuration keyword arguments\n    [alias::AbstractString], [env], [project::Union{UUID, AbstractString}],\n    timelimit::Limit = Hour(1),\n    # General keyword arguments\n    dryrun::Bool = false,\n    [auth :: Authentication]\n) -> Job\n\nSubmits the specified application config app as a job to JuliaHub. Returns a Job object corresponding to the submitted job.\n\nCompute arguments. If compute is passed, the compute keyword arguments can not be passed. If compute is not passed, the following arguments can be used to specify the compute configration via keyword arguments:\n\nncpu, ngpu and memory are used to pick a node type that will be used to run the job. The node type will be a minimum one that satisfies the constraints, but may have more compute resources than specified by the arguments (it corresponds to the exactmatch = false case of nodespec).\nnnodes, minimum_nnodes, process_per_node, and elastic specify the corresponding arguments in ComputeConfig.\n\nRuntime configuration. These are used to set the Runtime configuration of the job.\n\nalias :: Union{AbstractString, Nothing}: can be used to override the name of the job that gets displayed in the UI. Passing nothing is equivalent to omitting the argument.\ntimelimit :: Limit: sets the job's time limit (see Limit for valid values)\nenv: an iterable of key-value pairs that can be used to set environment variable that get set before, the job code gets executed.\nproject :: Union{UUID, AbstractString, Nothing}: the UUID of the project that the job will be associated with. If a string is passed, it must parse as a valid UUID. Passing nothing is equivalent to omitting the argument.\n\nGeneral arguments.\n\nauth :: Authentication: optional authentication object (see the authentication section for more information)\ndryrun :: Bool: if set to true, submit_job does not actually submit the job, but instead returns a WorkloadConfig object, which can be used to inspect the configuration that would be submitted.\nThe WorkloadConfig object can then be submitted to JuliaHub with the additional submit_job method:\nJuliaHub.submit_job(::WorkloadConfig; [auth::Authentication])\n\ncompat: JuliaHub compatibility\nThe timelimit = JuliaHub.Unlimited() argument requires JuliaHub 6.3+.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.Limit","page":"Job submission","title":"JuliaHub.Limit","text":"JuliaHub.Limit\n\nType-constraint on JuliaHub job timelimit arguments in submit_job.\n\nThe job time limit can either be a time period (an instance of Dates.Period), an Integer, (interpreted as the number of hours), or JuliaHub.Unlimited().\n\nOnly an integer number of hours are accepted by JuliaHub, and fractional hours from get rounded up to the next full integer number of hours (e.g. Dates.Minute(90) will be interpreted as 2 hours).\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.Unlimited","page":"Job submission","title":"JuliaHub.Unlimited","text":"struct Unlimited\n\nAn instance of this type can be passed as the [timelimit] option to submit_job to start jobs that run indefinitely, until killed manually.\n\nJuliaHub.submit_job(..., timelimit = JuliaHub.Unlimited())\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.WorkloadConfig","page":"Job submission","title":"JuliaHub.WorkloadConfig","text":"struct WorkloadConfig\n\nRepresents a full job configuration, including the application, compute and runtime configuration.\n\nInstances of this type can be constructed by passing dryrun = true to submit_job, and can also be directly submitted to JuliaHub with the same function.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#Experimental-APIs","page":"Job submission","title":"Experimental APIs","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"compat: Experimental features\nStarting application jobs with JuliaHub.jl is considered to be experimental. The APIs are likely to change in future JuliaHub.jl version.","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"AbstractJuliaHubApp\napplications\napplication\nDefaultApp\nPackageApp\nUserApp\nApplicationJob\nPackageJob","category":"page"},{"location":"reference/job-submission/#JuliaHub.AbstractJuliaHubApp","page":"Job submission","title":"JuliaHub.AbstractJuliaHubApp","text":"abstract type AbstractJuliaHubApp\n\nAbstract supertype for JuliaHub applications object types.\n\ncompat: Experimental API\nApplications-related APIs are experimental, and may be changed or removed without notice.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.applications","page":"Job submission","title":"JuliaHub.applications","text":"JuliaHub.applications([category::Symbol]; [auth::Authentication]) -> Vector{AbstractJuliaHubApp}\n\nReturns the list of applications enabled for the authenticated user, optionally in the specified category only. Returns a vector of AbstractJuliaHubApp instances.\n\njulia> JuliaHub.applications()\n7-element Vector{JuliaHub.AbstractJuliaHubApp}:\n JuliaHub.application(:default, \"Linux Desktop\")\n JuliaHub.application(:default, \"Julia IDE\")\n JuliaHub.application(:default, \"Pluto\")\n JuliaHub.application(:default, \"Windows Workstation\")\n JuliaHub.application(:package, \"RegisteredPackageApp\")\n JuliaHub.application(:package, \"CustomDashboardApp\")\n JuliaHub.application(:user, \"ExampleApp.jl\")\n\njulia> JuliaHub.applications(:default)\n4-element Vector{JuliaHub.DefaultApp}:\n JuliaHub.application(:default, \"Linux Desktop\")\n JuliaHub.application(:default, \"Julia IDE\")\n JuliaHub.application(:default, \"Pluto\")\n JuliaHub.application(:default, \"Windows Workstation\")\n\njulia> JuliaHub.applications(:package)\n2-element Vector{JuliaHub.PackageApp}:\n JuliaHub.application(:package, \"RegisteredPackageApp\")\n JuliaHub.application(:package, \"CustomDashboardApp\")\n\njulia> JuliaHub.applications(:user)\n1-element Vector{JuliaHub.UserApp}:\n JuliaHub.application(:user, \"ExampleApp.jl\")\n\n\ncompat: Experimental API\nApplications-related APIs are experimental, and may be changed or removed without notice.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.application","page":"Job submission","title":"JuliaHub.application","text":"JuliaHub.application(\n    category::Symbol, name::AbstractString;\n    throw::Bool=true, [auth::Authentication]\n) -> AbstractJuliaHubApp\n\nReturns the application corresponding to name from the specified category of applications. Will throw an InvalidRequestError if the application can't be found, or returns nothing in this situation if throw=false is passed.\n\ncategory specifies the application category and must be one of: :default, :package, or :user. This is necessary to disambiguate apps with the same name in the different categories.\n\nSee also: applications.\n\nExamples\n\njulia> JuliaHub.applications()\n7-element Vector{JuliaHub.AbstractJuliaHubApp}:\n JuliaHub.application(:default, \"Linux Desktop\")\n JuliaHub.application(:default, \"Julia IDE\")\n JuliaHub.application(:default, \"Pluto\")\n JuliaHub.application(:default, \"Windows Workstation\")\n JuliaHub.application(:package, \"RegisteredPackageApp\")\n JuliaHub.application(:package, \"CustomDashboardApp\")\n JuliaHub.application(:user, \"ExampleApp.jl\")\n\ncompat: Experimental API\nApplications-related APIs are experimental, and may be changed or removed without notice.\n\n\n\n\n\n","category":"function"},{"location":"reference/job-submission/#JuliaHub.DefaultApp","page":"Job submission","title":"JuliaHub.DefaultApp","text":"struct DefaultApp <: AbstractJuliaHubApp\n\nRepresents a default JuliaHub instance application, and they can be started as jobs with submit_job.\n\nThe list of available applications can be accessed via the applications function, and specific applications can be picked out with application.\n\njulia> apps = JuliaHub.applications(:default)\n4-element Vector{JuliaHub.DefaultApp}:\n JuliaHub.application(:default, \"Linux Desktop\")\n JuliaHub.application(:default, \"Julia IDE\")\n JuliaHub.application(:default, \"Pluto\")\n JuliaHub.application(:default, \"Windows Workstation\")\n\njulia> JuliaHub.application(:default, \"Pluto\")\nDefaultApp\n name: Pluto\n key: pluto\n\ncompat: Experimental API\nApplications-related APIs are experimental, and may be changed or removed without notice.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.PackageApp","page":"Job submission","title":"JuliaHub.PackageApp","text":"struct PackageApp <: AbstractJuliaHubApp\n\nRepresents a JuliaHub package application that is available in one of the instance's package registries. These packages can be started as JuliaHub jobs with submit_job.\n\nThe list of available applications can be accessed via the applications function, and specific applications can be picked out with application.\n\njulia> apps = JuliaHub.applications(:package)\n2-element Vector{JuliaHub.PackageApp}:\n JuliaHub.application(:package, \"RegisteredPackageApp\")\n JuliaHub.application(:package, \"CustomDashboardApp\")\n\njulia> JuliaHub.application(:package, \"RegisteredPackageApp\")\nPackageApp\n name: RegisteredPackageApp\n uuid: db8b4d46-bfad-4aa5-a5f8-40df1e9542e5\n registry: General (23338594-aafe-5451-b93e-139f81909106)\n\nSee also: help.juliahub.com on applications\n\ncompat: Experimental API\nApplications-related APIs are experimental, and may be changed or removed without notice.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.UserApp","page":"Job submission","title":"JuliaHub.UserApp","text":"struct UserApp <: AbstractJuliaHubApp\n\nRepresents a private application that has been added to the user account via a Git repository. These applications can be started as JuliaHub jobs with submit_job.\n\nThe list of available applications can be accessed via the applications function, and specific applications can be picked out with application.\n\njulia> apps = JuliaHub.applications(:user)\n1-element Vector{JuliaHub.UserApp}:\n JuliaHub.application(:user, \"ExampleApp.jl\")\n\njulia> JuliaHub.application(:user, \"ExampleApp.jl\")\nUserApp\n name: ExampleApp.jl\n repository: https://github.com/JuliaHubExampleOrg/ExampleApp.jl\n\nSee also: help.juliahub.com on applications\n\ncompat: Experimental API\nApplications-related APIs are experimental, and may be changed or removed without notice.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.ApplicationJob","page":"Job submission","title":"JuliaHub.ApplicationJob","text":"struct ApplicationJob <: AbstractJobConfig\n\nAbstractJobConfig that wraps a DefaultApp. This is primarily used internally and should rarely be constructed explicitly.\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#JuliaHub.PackageJob","page":"Job submission","title":"JuliaHub.PackageJob","text":"struct PackageJob <: AbstractJobConfig\n\nAbstractJobConfig that wraps a PackageApp or UserApp. This is primarily used internally and should rarely be constructed explicitly.\n\nConstructors\n\nJuliaHub.PackageJob(app::Union{JuliaHub.PackageApp,JuliaHub.UserApp}; [sysimage::Bool = false])\n\nCan be used to construct a PackageApp or UserApp based job, but allows for some job parameters to be overridden. Currently, only support the enabling of a system image based job by setting sysimage = true.\n\njulia> app = JuliaHub.application(:package, \"RegisteredPackageApp\")\nPackageApp\n name: RegisteredPackageApp\n uuid: db8b4d46-bfad-4aa5-a5f8-40df1e9542e5\n registry: General (23338594-aafe-5451-b93e-139f81909106)\n\njulia> JuliaHub.submit_job(JuliaHub.PackageJob(app; sysimage = true))\nJuliaHub.Job: jr-xf4tslavut (Submitted)\n submitted: 2023-03-15T07:56:50.974+00:00\n started:   2023-03-15T07:56:51.251+00:00\n finished:  2023-03-15T07:56:59.000+00:00\n\n\n\n\n\n","category":"type"},{"location":"reference/job-submission/#Index","page":"Job submission","title":"Index","text":"","category":"section"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"Pages = [\"job-submission.md\"]","category":"page"},{"location":"reference/job-submission/","page":"Job submission","title":"Job submission","text":"","category":"page"},{"location":"","page":"Home","title":"Home","text":"CurrentModule=JuliaHub","category":"page"},{"location":"#JuliaHub.jl","page":"Home","title":"JuliaHub.jl","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"The JuliaHub.jl package offers a programmatic Julia interface to the JuliaHub platform.","category":"page"},{"location":"","page":"Home","title":"Home","text":"With JuliaHub.jl you can do things such as start JuliaHub jobs, access job outputs, and manage your datasets, programmatically, directly in the REPL or in your Julia script. It can also be used in JuliaHub jobs to interact with the platform (to upload datasets, for example).","category":"page"},{"location":"","page":"Home","title":"Home","text":"If you are unfamiliar with JuliaHub.jl, you may want to start out by reading through the package's Getting Started tutorial. If you want to know in detail how to programmatically work with a particular JuliaHub feature, you may want to skim through the applicable how-to guide:","category":"page"},{"location":"","page":"Home","title":"Home","text":"Pages = Main.PAGES_GUIDES\nDepth = 1:1","category":"page"},{"location":"","page":"Home","title":"Home","text":"Finally, detailed explanations and API references of JuliaHub.jl features and functions are available in the reference section of the manual:","category":"page"},{"location":"","page":"Home","title":"Home","text":"Pages = Main.PAGES_REFERENCE\nDepth = 1:1","category":"page"},{"location":"","page":"Home","title":"Home","text":"tip: JuliaHub platform documentation\nThe documentation here focuses on working with the JuliaHub.jl Julia library. See the main JuliaHub documentation to learn more about the JuliaHub platform, and see product (e.g. JuliaSim) documentation for product-specific questions.","category":"page"},{"location":"","page":"Home","title":"Home","text":"note: Enterprise use\nJuliaHub.jl works with both juliahub.com and private enterprise instances. For enterprise users: the instance URL can be passed to the authenticate function either directly via an argument, or via the JULIA_PKG_SERVER environment variable.","category":"page"},{"location":"","page":"Home","title":"Home","text":"","category":"page"}]
 }