Merge pull request #26 from daos-stack/develop

DAOSGCP-77 Merge v0.1.0 changes from develop to main

.pre-commit-config.yaml

@@ -0,0 +1,27 @@
+---
+repos:
+  - repo: https://github.com/antonbabenko/pre-commit-terraform
+    rev: v1.64.0
+    hooks:
+      - id: terraform_fmt
+      - id: terraform_tflint
+        args:
+          - --args=--config=__GIT_WORKING_DIR__/.tflint.hcl
+      - id: terraform_validate
+  - repo: local
+    hooks:
+      - id: terraform-readme
+        name: terraform-readme
+        entry: tools/autodoc/terraform_docs.sh
+        language: script
+        types: ['terraform']
+        exclude: \.terraform\/.*$
+        pass_filenames: true
+        require_serial: true
+      - id: packer-readme
+        name: packer-readme
+        entry: tools/autodoc/terraform_docs.sh
+        language: script
+        files: ^.*\.pkr\.hcl$
+        pass_filenames: true
+        require_serial: true

.tfdocs-json.yaml

@@ -0,0 +1,25 @@
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+formatter: json
+
+# do not use lockfile to determine version ranges placed into docs
+settings:
+  lockfile: false
+
+output:
+  file: module.json
+  mode: replace
+  template: |
+    {{ .Content }}

.tfdocs-markdown.yaml

@@ -0,0 +1,27 @@
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+formatter: markdown
+
+# do not use lockfile to determine version ranges placed into docs
+settings:
+  lockfile: false
+
+output:
+  file: README.md
+  mode: inject
+  template: |-
+    <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+    {{ .Content }}
+    <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

.tflint.hcl

@@ -0,0 +1,38 @@
+plugin "google" {
+  enabled = true
+  version = "0.12.1"
+  source  = "github.com/terraform-linters/tflint-ruleset-google"
+}
+rule "terraform_deprecated_index" {
+  enabled = true
+}
+rule "terraform_unused_declarations" {
+  enabled = true
+}
+rule "terraform_documented_variables" {
+  enabled = true
+}
+rule "terraform_comment_syntax" {
+  enabled = true
+}
+rule "terraform_documented_outputs" {
+  enabled = true
+}
+rule "terraform_documented_variables" {
+  enabled = true
+}
+rule "terraform_typed_variables" {
+  enabled = true
+}
+rule "terraform_naming_convention" {
+  enabled = true
+}
+rule "terraform_required_version" {
+  enabled = true
+}
+rule "terraform_required_providers" {
+  enabled = true
+}
+rule "terraform_unused_required_providers" {
+  enabled = true
+}

README.md

@@ -1,7 +1,98 @@
-# Distributed Asynchronous Object Storage (DAOS) on Google Cloud Platform (GCP)
+# DAOS on GCP
 
-This repository contains scripts to deploy DAOS on GCP.
+Distributed Asynchronous Object Storage ([DAOS](https://docs.daos.io/)) on Google Cloud Platform ([GCP](https://cloud.google.com/))
 
-It consists of the directories:
-- [images](images) - which contains scripts to prepare DAOS images for GCP used by Terraform code
-- [terraform](terraform) - which contains Terraform code used to deploy DAOS on GCP
+This repository contains:
+
+- [Packer](https://www.packer.io/) scripts used to build DAOS images with [Google Cloud Build](https://cloud.google.com/build)
+- [Terraform](https://www.terraform.io/) modules that can be used to deploy DAOS Server and Client instances
+- [Terraform](https://www.terraform.io/) examples that demonstrate how to use the Terraform modules
+
+**Directory structure**
+
+```
+.
+├── docs            Miscellaneousc documentation and Cloud Shell tutorials
+├── images          Cloud Build config files and Packer templates
+│   └── scripts     Scripts that Packer runs to configure images
+├── terraform       Terraform content
+│   ├── examples    Examples that demonstrate how to use the DAOS Terraform modules
+│   └── modules     Terraform modules for deploying DAOS server and client instances
+└── tools           Tools used by pre-commit
+```
+
+### Prerequisites
+
+In order to deploy DAOS on GCP you will need
+
+- **Access to the Google Cloud Platform (GCP)**
+
+   See [Get Started with Google Cloud](https://cloud.google.com/docs/get-started)
+
+- **A GCP Project**
+
+  See [Creating and managing projects](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
+
+- **Required Software**
+
+  The documentation in this repository assumes that you will use [Cloud Shell](https://cloud.google.com/shell).
+
+  If you use [Cloud Shell](https://cloud.google.com/shell), you do not need to install any software on your system.
+
+  If you do not want to use Cloud Shell, you will need to install
+    - [Git](https://git-scm.com/)
+    - [Google Cloud CLI](https://cloud.google.com/sdk/docs/install)
+    - [Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli)
+
+### Deploy DAOS on GCP
+
+Steps to deploy DAOS on GCP
+
+1. **Set defaults for Google Cloud CLI (```gcloud```)**
+
+   Only needs to be done once in your shell (Cloud Shell or local shell).
+
+2. **Create a Packer image in your GCP project**
+
+   In order to build DAOS images with Cloud Build your GCP project must contain a Packer image.
+
+   Building the Packer images only needs to be done once for a GCP project.
+
+3. **Build DAOS Server and Client images**
+
+   DAOS Server and Client instances are deployed using images that have DAOS pre-installed.
+
+   Therefore, the images need to be built prior to running Terraform.
+
+  >  Click the button to open an interactive walk-through in Cloud Shell which will guide you through the steps listed above.
+  >
+  >  [![DAOS on GCP Setup](http://gstatic.com/cloudssh/images/open-btn.png)](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/daos-stack/google-cloud-daos&cloudshell_git_branch=main&shellonly=true&tutorial=docs/tutorials/daosgcp_setup.md)
+
+4. **Use DAOS Terraform modules in your Terraform code**
+
+   You will need to write your own Terraform code for your particular use case.
+
+   Your Terraform code can use the modules in ```terraform/modules``` to deploy DAOS server and client instances.
+
+   The example Terraform configurations provided in ```terraform/examples``` can be used as a reference.
+
+   See the [DAOS Cluster](terraform/examples/daos_cluster/README.md) example to learn more about how to use the ```terraform/modules```.
+
+## Links
+
+- [Distributed Asynchronous Object Storage (DAOS)](https://docs.daos.io/)
+- [Google Cloud Platform (GCP)](https://cloud.google.com/)
+- [Google Cloud CLI (gcloud)](https://cloud.google.com/cli)
+- [Google Cloud Build](https://cloud.google.com/build)
+- [Cloud Shell](https://cloud.google.com/shell)
+- [Packer](https://www.packer.io/)
+- [Terraform](https://www.terraform.io/)
+
+
+## Development
+
+If you are contributing to the code in this repo, see [Development](docs/development.md)
+
+## License
+
+[Apache License Version 2.0](LICENSE)

docs/development.md

@@ -0,0 +1,139 @@
+# Development
+
+Please use the [pre-commit](https://pre-commit.com/) hooks configured in this repository to ensure that all Terraform modules are validated and properly documented before pushing code changes.
+
+
+## Install pre-commit and dependencies
+
+In order to use [pre-commit](https://pre-commit.com/) you will need to install it on your system.
+
+You will also need to install the dependencies that are required for the pre-commit plugins used in this repository.
+
+1. Install [pre-commit](https://pre-commit.com/).
+
+   [pre-commit](https://pre-commit.com/) can be installed using standard package managers.
+
+   Instructions can be found at the [pre-commit website](https://pre-commit.com/#install).
+
+
+2. Install [TFLint](https://github.com/terraform-linters/tflint)
+
+   See the [installation instructions](https://github.com/terraform-linters/tflint#installation)
+
+   After installing tflint change into the root of the locally cloned git repo and run the `init` command.
+
+   ```shell
+   cd <root of google-cloud-daos repo>
+   tflint --init
+   ```
+
+3. Install terraform-docs
+
+   See [https://github.com/terraform-docs/terraform-docs](https://github.com/terraform-docs/terraform-docs)
+
+
+4. MacOS only
+
+   MacOS users will need to install `findutils` and `coreutils`.
+
+   Before installing coreutils read the
+   [gotchas about coreutils](https://www.pixelbeat.org/docs/coreutils-gotchas.html)
+   to ensure that the installation will not negatively impact your
+   system.
+
+   **Homebrew**
+
+   ```shell
+   brew install findutils
+   brew install coreutils
+   ```
+
+   **Conda**
+
+   ```shell
+   brew install findutils
+   conda install coreutils
+   ```
+
+   Update your PATH  in your `~/.bashrc` or `~/.bash_profile`
+   ```shell
+   PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
+   ```
+
+## Install the pre-commit hook
+
+After you have installed [pre-commit](https://pre-commit.com/) and its dependencies on your system you can need to install the pre-commit hook in
+your local clone of the google-cloud-daos git repository.
+
+```shell
+cd <root of google-cloud-daos repo>
+pre-commit install
+```
+
+## Running pre-commit
+
+[pre-commit](https://pre-commit.com/) will now run on any files that are staged when you run `git commit -s`.
+
+To run [pre-commit](https://pre-commit.com/) on all files prior to staging them
+
+```shell
+pre-commit run --all-files
+```
+
+## Updating Cloud Shell urls in documentation
+
+Several of the README.md files in this repository contain links that open tutorials in Cloud Shell.
+
+In order for these links to work properly during development the URLs must be changed to point to the correct branch.
+
+Currently Cloud Shell tutorials do not have an automatic way to detect a branch.  Therefore, the branch parameter in the URL must be updated manually.
+
+The `tools/autodoc/cloudshell_urls.sh` script should be used to update the branch parameter in all Cloud Shell URLs that are present in *.md files in this repo.
+
+### Update Cloud Shell URLs when submitting a PR
+
+If your PR changes README.md files that contain Cloud Shell URLs, then prior to requesting a review you should run the following command and push any changes to your dev branch.
+
+```bash
+tools/autodoc/cloudshell_urls.sh --repo-url <your_forked_repo_url> --branch <your_dev_branch_name>
+```
+
+This will allow the reviewers to run Cloud Shell tutorials from your PR branch.
+
+### Update Cloud Shell URLs before merging to a branch
+
+If you are merging changes to `*.md` files with Cloud Shell URLs in them you need to ensure that the URLs are updated with the name of the target branch before you merge.
+
+This is not ideal but it's the only way we can think of doing things for now.
+
+Let's say that you have a PR that has been approved and you want to merge it to the `develop` branch.
+
+Prior to merging you need to run
+
+```bash
+tools/autodoc/cloudshell_urls.sh --repo-url <your_forked_repo_url> --branch develop
+```
+
+And then commit the changes in your dev branch.
+
+Once that is done you can then merge to the `develop` branch.
+
+Now let's say that you want to merge the `develop` branch into the `main` branch.
+
+You will need to check out the https://github.com/daos-stack/google-cloud-daos `develop` branch and run
+
+```bash
+tools/autodoc/cloudshell_urls.sh --repo-url https://github.com/daos-stack/google-cloud-daos --branch main
+```
+
+Commit the changes and push them to the develop branch. After doing this you can merge the `develop` branch to `main`.
+
+Now you will need to set the URLs back to the develop branch.
+
+```bash
+tools/autodoc/cloudshell_urls.sh --repo-url https://github.com/daos-stack/google-cloud-daos --branch develop
+```
+
+Commit the changes and push them to the develop branch.
+
+This is very tedious. We will continue to seek out a better solution for maintaining the Cloud Shell URLs.