Skip to content

Commit

Permalink
Enhance artifact pull and push commands (#112)
Browse files Browse the repository at this point in the history
* Rename GetCreds() to GenerateCraneOptions(), check for oci urls for 'oci://' prefix

- Users can now pass Registry Tokens as credential while using pull/push commands.
- OCI URLs need to provided prefixed with 'oci://' to differenciate between HTTP and OCI operations.
- Implementation of exponential retry logic while pulling and pushing from registries.
- Update OCI urls for pulling deault policies for Rego validation.
Signed-off-by: Santosh <ksantosh@intelops.dev>

* Add completion command

Signed-off-by: Santosh <ksantosh@intelops.dev>

* Add Rego validation for Dockerfiles with policies stored in OCI registries

Signed-off-by: Santosh <ksantosh@intelops.dev>

* Fix: lint errors

Signed-off-by: santoshkal <ksantosh@intelops.dev>

* Fix: lint errors

Signed-off-by: santoshkal <ksantosh@intelops.dev>

* Update: accept Rego policies from oci registries

Signed-off-by: santoshkal <ksantosh@intelops.dev>

* Add examples in commands and fix typos

Signed-off-by: santoshkal <ksantosh@intelops.dev>

* Update: validate with default policies and policies stored in OCI registries for dockerfile command

Signed-off-by: santoshkal <ksantosh@intelops.dev>

* Refactor conditional logic to accept default Rego policies and Rego policies from OCI registries for all Rego commands

Signed-off-by: santoshkal <ksantosh@intelops.dev>

* Refactor Auth and OCI Push Options in ociClient pkg

- Move out auth related logic to its own GetCreds().
- Now, users can pass creds through --credentials flag
- accepts auth in <$USER:$PAT> or <$TOKEN> format
- If none provided falls back to /home/santosh/.docker/config.json
- Updated examples for all commands for using default policies and policies from OCI registries

Signed-off-by: santoshkal <ksantosh@intelops.dev>

* Update: Reading of OCI URLs for cuemods from a const instead of a .env file

This behaviour is for testing the commands and would be updated to read the URLs for all commands from a .env file
stored in a repo

Signed-off-by: santoshkal <ksantosh@intelops.dev>

* Update retry logic in GenerateCraneOptions() in line with go-containerregistry

Signed-off-by: santoshkal <ksantosh@intelops.dev>

* Update README about new workflows of validating iwth default policies and policies from OCI registries

Signed-off-by: santoshkal <ksantosh@intelops.dev>

* Update rootCmd for supressing the usage info printed with errors

Signed-off-by: santoshkal <ksantosh@intelops.dev>

* Fix changelog printing for releases in goreleaser

Signed-off-by: santoshkal <ksantosh@intelops.dev>

---------

Signed-off-by: Santosh <ksantosh@intelops.dev>
Signed-off-by: santoshkal <ksantosh@intelops.dev>
  • Loading branch information
santoshkal authored Jun 21, 2024
1 parent ce9bc91 commit dd7f5e3
Show file tree
Hide file tree
Showing 20 changed files with 504 additions and 204 deletions.
2 changes: 0 additions & 2 deletions .env

This file was deleted.

2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ input_dockerfile.json
Dockerfile*
genval
cosign
.todo
todo.md
!.devcontainer/Dockerfile
results.json
.env
Expand Down
9 changes: 9 additions & 0 deletions .goreleaser.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,16 @@ source:

# Adds Changelog to the release
changelog:
use: github
format: "{{.SHA}}: {{.Message}} (@{{.AuthorUsername}})"
sort: asc
groups: # Regex use RE2 syntax as defined here: https://github.com/google/re2/wiki/Syntax.
- title: 'Updates'
regexp: ^(?i).*?update\w*(\([[:word:]]+\))??!?:.+$
order: 100
- title: 'Bug fixes'
regexp: ^(?i).*?fix\w*(\([[:word:]]+\))??!?:.+$
order: 200
filters:
exclude:
- "^docs:"
Expand Down
44 changes: 42 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ If verification is successful, you'll see "**Verified OK.**"
## Quick Start


For a quick start, pre-built templates for Dockerfile generation for popular languages can be found in the `./templates/inputs/dockerfile_input` folder.
For a quick start, pre-built templates for Dockerfile generation for popular languages can be found in the `./templates/inputs/dockerfile_input` folder. We also maintain all the default policies and input templates in a dedicated [repository](https://github.com/intelops/policyhub).


## Building from Source
Expand Down Expand Up @@ -188,12 +188,52 @@ $ genval dockerfile --reqinput=./templates/inputs/dockerfile_input/golang_input.
All the rego policies are housed in a hirearchy containing a `rego` policy and a `JSON` file containing all the metadata related to policy. A user needs to pass the directory containing boththe `.rego` and `.json` files. Genval also accpets a top leval directory containing multiple sub-directories containing multiple rego and accompanied JSON files for validating with more than one policy.

Users can use policies to validate input JSON as well as generated Dockerfile with policies stored in their OCI registries
or with Genval's default Rego policies. Behind the scenes, this action iteracts with OCI registries for pulling the policies.

To facilitate authentication with OCI compliant container registries,
Users can provide credentials through --credentials flag. The creds can be provided via <$USER:$PAT> or <REGISTRY_PAT> format.
If no credentials are provided, Genval searches for the "./docker/config.json" file in the user's $HOME directory.
If this file is found, Genval utilizes it for authentication.

**Validating with Default policies**

```shell
./genval dockerfile --reqinput https://github.com/intelops/genval-security-policies/blob/patch-1/input-templates/dockerfile_input/clang_input.json \
--output ./output/Dockefile-cobra
// No credntials provided, will default to $HOME/.docker/config.json for credentials
```

**Validating with policies stored in OCI compliant container registries**

```shell
./genval dockerfile --reqinput https://github.com/intelops/genval-security-policies/blob/patch-1/input-templates/dockerfile_input/clang_input.json \
--output ./output/Dockefile-cobra \
inputpolicy oci://ghcr.io/intelops/policyhub/genval/input_policies:v0.0.1 \
--outputpolicy oci://ghcr.io/intelops/policyhub/genval/dockerfile_policies:v0.0.1 \
--credentials <$GITHUB_PAT> or <$USER:$PAT> format
```
**Review Feedback**: Genval provides feedback based on best practice validation. Use this feedback to refine your Dockerfile.

### Validation of the Dockerfile, Kubernetes manifests and Terraform files using Rego policies

Genval manages validation with Rego polcies with `regoval` command and for validation of each of the technology a separate subcommand is provided:
Users can leverage Genval's feature of Validating of resources using policies stored in OCI compliant registries or provide policies stored in their own OCI compliant registries.

To facilitate authentication with OCI compliant container registries, Users can provide credentials through `--credentials` flag while invoking a regoval subcommand. The credentials can
be provided via <$USER:$PAT> or <$REGISTRY_PAT> format. If no credentials are provided, Genval searches for the `./docker/config.json` file in the user's `$HOME` directory. If this file is found, Genval utilizes it for authentication.

**Example**:

./genval regoval dockerfileval --reqinput=Dockerfile \
--policy oci://ghcr.io/intelops/policyhub/genval/dockerfile_policies:v0.0.1
--credentials <GITHUB_PAT> or <USER:PAT>


Users can you use default policies maintained by the community stored in the https://github.com/intelops/policyhub repo

./genval regoval dockerfileval --reqinput <Path to Dockerfile>
// No credntials provided, will default to $HOME/.docker/config.json for credentials


#### Validation of Dockerfiles with Rego policies

Expand Down
12 changes: 7 additions & 5 deletions cmd/artifact_pull.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ The artifact pull command pull the artifact stored in the remote container regis
If --verify flag is set to true (false by default), Genval will verify the artifact's signature
which is signed by Cosign in Keyless mode and pull the artifact, and unpack the tar.gz archive in desired path.
To facilitate authentication with container registries, Genval initially searches for the "./docker/config.json"
file in the user's $HOME directory. If this file is found, Genval utilizes it for authentication.
However, if the file is not present, users must set the ARTIFACT_REGISTRY_USERNAME and ARTIFACT_REGISTRY_PASSWORD
environment variables to authenticate with the container registry.
To facilitate authentication with OCI compliant container registries,
Users can provide credentials through --creds flag. The creds can be provided via <USER:PAT> or <REGISTRY_PAT> format.
If no credentials are provided, Genval searches for the "./docker/config.json" file in the user's $HOME directory.
If this file is found, Genval utilizes it for authentication.
`,
Expand All @@ -45,12 +45,14 @@ environment variables to authenticate with the container registry.
--path ./output \
--verify true \
--pub-key ./cosign/cosign.pub
--credentials <GITHUB_PAT> or <USER:PAT>
# Uses can also pull the artifact with verifying the signatures of the artifact
in the container registry and unpack the archive in desired path
./genval artifact pull --dest ghcr.io/santoshkal/artifacts/genval:test \
--path ./output
// No credentials provided, will default to $HOME/.docker/config.json for credentials
`,
RunE: runPullArtifactCmd,
}
Expand Down Expand Up @@ -103,7 +105,7 @@ func runPullArtifactCmd(cmd *cobra.Command, args []string) error {
spin := utils.StartSpinner("Pulling Artifact...")
defer spin.Stop()

if err := oci.PullArtifact(context.Background(), pullArgs.dest, pullArgs.path); err != nil {
if err := oci.PullArtifact(context.Background(), pullArgs.creds, pullArgs.dest, pullArgs.path); err != nil {
color.Red("Error pulling artifact from remote : %v", err)
return err
}
Expand Down
42 changes: 26 additions & 16 deletions cmd/artifact_push.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,13 @@ import (
"path/filepath"
"time"

"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/compression"
"github.com/google/go-containerregistry/pkg/crane"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/empty"
"github.com/google/go-containerregistry/pkg/v1/mutate"
"github.com/google/go-containerregistry/pkg/v1/remote/transport"
"github.com/google/go-containerregistry/pkg/v1/tarball"
"github.com/google/go-containerregistry/pkg/v1/types"
"github.com/intelops/genval/pkg/oci"
Expand All @@ -27,10 +28,10 @@ var pushCmd = &cobra.Command{
The artifact push command takes in a tar.gz bundle of configuration files generated/validated by Genval
and pushes them into a OCI complient container registry
To facilitate authentication with container registries, Genval initially searches for the "./docker/config.json"
file in the user's $HOME directory. If this file is found, Genval utilizes it for authentication.
However, if the file is not present, users must set the ARTIFACT_REGISTRY_USERNAME and ARTIFACT_REGISTRY_PASSWORD
environment variables to authenticate with the container registry.
To facilitate authentication with OCI compliant container registries,
Users can provide credentials through --creds flag. The creds can be provided via <USER:PAT> or <REGISTRY_PAT> format.
If no credentials are provided, Genval searches for the "./docker/config.json" file in the user's $HOME directory.
If this file is found, Genval utilizes it for authentication.
`,
Example: `
# Build and push the provided file/files as OCI artifact to registry
Expand All @@ -42,13 +43,15 @@ environment variables to authenticate with the container registry.
./genval artifact push --reqinput ./templates/defaultpolicies/rego \
--dest ghcr.io/santoshkal/artifacts/genval:test \
--sign true
// No credentials provided, will default to $HOME/.docker/config.json for credentials
# Alternatively, users may provide the Cosign generated private key for signing the artifact
./genval artifact push --reqinput ./templates/defaultpolicies/rego \
--dest ghcr.io/santoshkal/artifacts/genval:test \
--sign true
--cosign-key <Path to Cosign private Key>
--credentials <GITHUB_PAT> or <USER:PAT>
# User can pass additional annotations in <key=value> pair while pushing the artifact
Expand Down Expand Up @@ -114,21 +117,20 @@ func runPushCmd(cmd *cobra.Command, args []string) error {
if err := oci.CreateTarball(inputPath, outputPath); err != nil {
return fmt.Errorf("creating tarball: %w", err)
}
log.Println("✔ Artifact created successfully")
log.Info("✔ Artifact created successfully")

ref, err := name.ParseReference(source)
ref, err := oci.ParseOCIReference(source)
if err != nil {
log.Printf("Error parsing source: %v", err)
log.Errorf("Error parsing source: %v", err)
}

remoteURL, err := oci.GetRemoteURL()
fmt.Printf("Remote Name: %v", remoteURL)
if err != nil {
return err
return fmt.Errorf("error fetching credentials: %v", err)
}
annotations, err := oci.ParseAnnotations(pushArgs.annotations)
if err != nil {
return err
return fmt.Errorf("error parsing annotations %s: %v", pushArgs.annotations, err)
}

img := mutate.MediaType(empty.Image, types.OCIManifestSchema1)
Expand All @@ -155,9 +157,17 @@ func runPushCmd(cmd *cobra.Command, args []string) error {
}
spin := utils.StartSpinner("pushing artifact")
defer spin.Stop()
opts, err := oci.GetCreds()
auth, err := oci.GetCreds(pushArgs.creds)
if err != nil {
log.Errorf("Error reading credentials: %v", err)
return fmt.Errorf("error getting credentials: %v", err)
}
opts, err := oci.GenerateCraneOptions(cmd.Context(), ref, auth, []string{ref.Context().Scope(transport.PushScope)})
if err != nil {
log.Errorf("Error creating options required for push: %v", err)
}
opts = append(opts, crane.WithAuth(auth))
if pushArgs.creds == "" {
opts = append(opts, crane.WithAuthFromKeychain(authn.DefaultKeychain))
}

if err := crane.Push(img, ref.String(), opts...); err != nil {
Expand All @@ -174,11 +184,11 @@ func runPushCmd(cmd *cobra.Command, args []string) error {
if pushArgs.sign {
err := oci.SignCosign(digestURL, pushArgs.cosignKey)
if err != nil {
return err
return fmt.Errorf("error signing artifact %s: %v", digestURL, err)
}
}

log.Printf("✔ Artifact pushed successfully to: %v\n, with Digest: %v\n", source, digest)
log.Printf("Digest URL: %v\n", digestURL)
log.Infof("✔ Artifact pushed successfully to: %v\n,Artifact Digest: %v\n", source, digest)
log.Infof("Digest URL: %v\n", digestURL)
return nil
}
86 changes: 86 additions & 0 deletions cmd/completion.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
/*
Copyright © 2024 NAME HERE <EMAIL ADDRESS>
*/
package cmd

import (
"fmt"
"os"

log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
)

var completionCmd = &cobra.Command{
Use: "completion [bash|zsh|fish|powershell]",
Short: "Generate completion script",
Long: fmt.Sprintf(`To load completions:
Bash:
$ source <(%[1]s completion bash)
# To load completions for each session, execute once:
# Linux:
$ %[1]s completion bash > /etc/bash_completion.d/%[1]s
# macOS:
$ %[1]s completion bash > $(brew --prefix)/etc/bash_completion.d/%[1]s
Zsh:
# If shell completion is not already enabled in your environment,
# you will need to enable it. You can execute the following once:
$ echo "autoload -U compinit; compinit" >> ~/.zshrc
# To load completions for each session, execute once:
$ %[1]s completion zsh > "${fpath[1]}/_%[1]s"
# You will need to start a new shell for this setup to take effect.
fish:
$ %[1]s completion fish | source
# To load completions for each session, execute once:
$ %[1]s completion fish > ~/.config/fish/completions/%[1]s.fish
PowerShell:
PS> %[1]s completion powershell | Out-String | Invoke-Expression
# To load completions for every new session, run:
PS> %[1]s completion powershell > %[1]s.ps1
# and source this file from your PowerShell profile.
`, rootCmd.Name()),
DisableFlagsInUseLine: true,
ValidArgs: []string{"bash", "zsh", "fish", "powershell"},
Args: cobra.MatchAll(cobra.ExactArgs(1), cobra.OnlyValidArgs),
Run: func(cmd *cobra.Command, args []string) {
if len(args) == 0 {
log.Error("No shell specified")
}
var err error
switch args[0] {
case "bash":
err = rootCmd.GenBashCompletion(os.Stdout)
case "zsh":
err = rootCmd.GenZshCompletion(os.Stdout)
case "fish":
err = rootCmd.GenFishCompletion(os.Stdout, true)
case "powershell":
err = rootCmd.GenPowerShellCompletionWithDesc(os.Stdout)
default:
log.Printf("Unknown shell: %s", args[0])
return
}
if err != nil {
log.Printf("Error generating completion for %s: %v", args[0], err)
return
}
},
}

func init() {
rootCmd.AddCommand(completionCmd)
}
Loading

0 comments on commit dd7f5e3

Please sign in to comment.