Is it possible for an expression to get the filename?

[eadlam]

I would like to capture the filename in a let expression scoped to that file.

[mvdan]

What would you need the filename for? For example, if it's for reporting an error, #943 would allow you to use error messages for "bottom", meaning that a user could see a friendly error with the filename and position where it's written.

[eadlam]

@mvdan

I have a system in which I configure a set of agents. Each agent has a type and an id (among other things) and they live at the root level of the package. Each agent config is in it's own file. For example:

red_0.cue

package agents

red_0: {
  type: "red"
  id: 0
  ...
}

blue_1.cue

package agents

blue_1: {
  type: "blue"
  id: 1
  ...
}

blue_2.cue

package agents

blue_2: {
  type: "blue"
  id: 2
  ...
}

Now let's say I want to change all blue agents to green. For each blue agent I have to change three things:

1. the filename (by convention)
2. the struct key (also by convention, but I can "type check" it via template regex)
3. the actual type values

This was an inefficient and redundant workflow.

I recently improved it by deriving the type and id from the agent struct key. For example:

#common.cue

package agents

import (
    "regexp"
    "strconv"
)

[AgentName= =~"^[a-zA-Z]_[0-9]+$"]: {
    let match = regexp.FindSubmatch("^([a-zA-Z]+)_([0-9]+)$", AgentName)
    type: match[1]
    id: strconv.Atoi(match[2])
}

Now my agent files don't need to manually specify type and id:

blue_2.cue

package agents

blue_2: {
  ...
}

However, if I change all blue agents to green, I still need to modify both the filename and the agent struct key. I would love it if I could just change the filename. For example, imagine the filename was available as a reserved tag. I could do this:

#common.cue

package agents

import (
    "regexp"
    "strconv"
)

#AgentName: {
  #filename: string
  regexp.FindSubmatch("^([a-zA-Z]+_[0-9]+).cue$", #filename)[1]
}

[AgentName= =~"^[a-zA-Z]_[0-9]+$"]: {
    let match = regexp.FindSubmatch("^([a-zA-Z]+)_([0-9]+)$", AgentName)
    type: match[1]
    id: strconv.Atoi(match[2])
}

blue_2.cue

package agents

let Agent = {#AgentName, #filename: string @tag(filename)}
"\(Agent)": {
  ...
}

With this, the filename is the source of truth of the agent name, type, and id. This may seem like a niche problem, but I think it's a primary missing piece for enabling a workflow in which you compose complex configurations by dragging/dropping/copying/renaming files.

[mvdan]

I see what you mean, thanks for elaborating. There is no such method that I am aware right now. I guess what we want is to inject the filename at the syntax level, where the expression is written, regardless of where that value ends up getting unified into or used as.

The only alternative that comes to mind right now would be to enforce this via the tooling layer, which can list files in a directory and read/write to them. But then you'd have to run something like cue cmd applynames to enforce the consistency, rather than getting it by default when evaluating.

I'll CC @mpvl @myitcv to get their thoughts; this seems like a reasonable request to me. A few follow-up questions come to mind:

• Would the filename just be the base name (blue_2.cue), a full absolute path (/home/john/project/somepkg/blue_2.cue), or a path relative to the module root (somepkg/blue_2.cue)?
• If we don't use base names, do we always use forward slashes as separators?
• What would the variable contain in cases where there wasn't a file on disk, such as feeding CUE via stdin?

Perhaps a relatively composable solution would be to only do the base name now, and then later we can expose other variables like the package path or the module path.

[myitcv]

@mvdan I think it would be worth relaying some of the offline conversations to augment this discussion?

[davidovich]

I need this kind of metadata also, and have opted to use the --with-context and -l flags. It is a bit of a mystery how they work, but by trial and error I got to extract the filename. I would like a native way to have the filename instead of the following.

In a library utility:

package entry

import (
	"path"
	"strings"
)

// parameters is the top level field for the user serviced config
// Ensure we have this field.
#hasParametersField: {
	parameters: _
}

// extract name from filename
// #in is the input injected at call site by --with-context, it can be placed in the main cue schema. 
// we also lift the enclosing parameters to top level (it is a domain requirement to not have the parameter field)
#nameFromFilename: {
	#in: _

	for _name, _content in #in {
		{
			_isYaml: _name & =~#"\.yaml$"#
			Name:    strings.TrimSuffix(path.Base(_name), ".yaml")
		} & _content.parameters // lift the contained parameters
	}
}

Given a file tree like so:

configs
├── my-config-filename.yaml
└── config-shema.cue

Where my-config-filename.yaml is

parameters:
    otherField: "user-defined"

And config-schema.cue:

// input has the form input: "filename": content when called --with-context -l '_input' -l 'filename'
_input: [string]: entry.#hasParametersField

// build a hidden _parameters which has the deduced Name 
_parameters: {#in:_input} & entry.#nameFromFilename

#config: {
    // enforced schema
    Name: string // needed to satisfy the unification below
    otherField: string // user defined constraint
}

// yield a unified config
_parameters & #config

I validate using this call:

cue vet -c --with-context -l '_input:' -l 'filename' configs/my-config-filename.yaml configs/config-schema.cue

I export using this call:

cue export --with-context -l '_input:' -l 'filename' configs/my-config-filename.yaml configs/config-schema.cue

{
    Name: "my-config-filename"
    otherField: "user-defined"
}

[myitcv]

@davidovich - just out of interest, had you seen https://cuelang.org/docs/howto/combine-multiple-yaml-files-by-using-file-metadata/?

Secondly, would file embedding of a glob (https://cuelang.org/docs/howto/embed-files-in-cue-evaluation/), combined with some use of https://pkg.go.dev/cuelang.org/go/pkg/path and possibly the various https://pkg.go.dev/cuelang.org/go/pkg/encoding packages perhaps be easier?

[davidovich]

@myitcv yes, I had seen that entry of combining multiple files, this is exactly where I got the idea on how to extract the filename. I have an implementation where I associate n yaml files to one cue schema for validation. I use a make driven repeated call to cue to validate all my config yaml files.

Having a context of metadata available would remove the need to change invocation (--with-context -l) and the extract name from filename struct.

[myitcv]

The tricky point here, at least as far as I can see, is establishing "during evaluation" where a value has come from. Because by that time, the answer is not, generally speaking, a single place. The reason being because of unification: CUE can, and does, happily combine values from multiple sources, at which point the concept of a single source has disappeared. Hence why the pre-evaluation placing of data is a much cleaner approach, whether by using @embed file embedding, or via -l (with --with-context).

@davidovich

I would like a native way to have the filename instead of the following.

Please can you help me understand what you mean by "native way" here? Perhaps with a pseudo-code example?

[davidovich]

Maybe a metadata construct to capture filenames passed on the commandline, that we could access natively?

#metadata: {
   cuefiles: [string] // or perhaps a {file: content} mapping, but this may be the @embed feature
   datafiles: [string]
}

I'll have a look at embed, though my initial try failed with this error:

cannot embed files when not in a module

But I am, I have a module at root.

[myitcv]

@davidovich ah right, if all you want is a list of CUE files, non-CUE files that have gone into an evaluation, that's a much "simpler" problem indeed.

I'll have a look at embed, though my initial try failed with this error:

You definitely shouldn't be getting that error. If you have a repro, please do feel free to raise an issue/bug.