DAOSGCP-75 Example and documentation updates (#25)

1. Renamed example directories for clarity and consistency 2. Updated variable names in examples to be consistent 3. Added terraform.tfvars.*.example files. Updated variable values in terraform.tfvars.*.example files This takes care of the changes required for DAOSGCP-76 4. Updated main README.md to contain more instructions for getting started 5. Created Cloud Shell tutorial that walks through the one-time setup steps 6. Created Cloud Shell tutorial that walks through the daos_cluster example 7. Set default DAOS version to 2.0 Signed-off-by: Mark A. Olson <mark.a.olson@intel.com>
daos-stack · Mar 30, 2022 · 4a9c8a9 · 4a9c8a9
1 parent 17b08ac
commit 4a9c8a9
Show file tree

Hide file tree

Showing 36 changed files with 1,892 additions and 672 deletions.
diff --git a/README.md b/README.md
@@ -1,21 +1,98 @@
-# [Distributed Asynchronous Object Storage (DAOS)](https://docs.daos.io/) on [Google Cloud Platform (GCP)](https://cloud.google.com/)
+# DAOS on GCP
 
-This repository contains scripts to deploy [DAOS](https://docs.daos.io/) on [GCP](https://cloud.google.com/).
+Distributed Asynchronous Object Storage ([DAOS](https://docs.daos.io/)) on Google Cloud Platform ([GCP](https://cloud.google.com/))
 
-It consists of the directories:
-- [images](images) - which contains scripts to prepare DAOS images for GCP used by Terraform code
-- [terraform](terraform) - which contains Terraform code used to deploy DAOS on GCP
+This repository contains:
 
-## Dependencies
+- [Packer](https://www.packer.io/) scripts used to build DAOS images with [Google Cloud Build](https://cloud.google.com/build)
+- [Terraform](https://www.terraform.io/) modules that can be used to deploy DAOS Server and Client instances
+- [Terraform](https://www.terraform.io/) examples that demonstrate how to use the Terraform modules
 
-In order to build images and deploy DAOS instances in Google Cloud Compute Engine you must have the following software installed.
+**Directory structure**
+
+```
+.
+├── docs            Miscellaneousc documentation and Cloud Shell tutorials
+├── images          Cloud Build config files and Packer templates
+│   └── scripts     Scripts that Packer runs to configure images
+├── terraform       Terraform content
+│   ├── examples    Examples that demonstrate how to use the DAOS Terraform modules
+│   └── modules     Terraform modules for deploying DAOS server and client instances
+└── tools           Tools used by pre-commit
+```
+
+### Prerequisites
+
+In order to deploy DAOS on GCP you will need
+
+- **Access to the Google Cloud Platform (GCP)**
+
+   See [Get Started with Google Cloud](https://cloud.google.com/docs/get-started)
+
+- **A GCP Project**
+
+  See [Creating and managing projects](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
+
+- **Required Software**
+
+  The documentation in this repository assumes that you will use [Cloud Shell](https://cloud.google.com/shell).
+
+  If you use [Cloud Shell](https://cloud.google.com/shell), you do not need to install any software on your system.
+
+  If you do not want to use Cloud Shell, you will need to install
+    - [Git](https://git-scm.com/)
+    - [Google Cloud CLI](https://cloud.google.com/sdk/docs/install)
+    - [Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli)
+
+### Deploy DAOS on GCP
+
+Steps to deploy DAOS on GCP
+
+1. **Set defaults for Google Cloud CLI (```gcloud```)**
+
+   Only needs to be done once in your shell (Cloud Shell or local shell).
+
+2. **Create a Packer image in your GCP project**
+
+   In order to build DAOS images with Cloud Build your GCP project must contain a Packer image.
+
+   Building the Packer images only needs to be done once for a GCP project.
+
+3. **Build DAOS Server and Client images**
+
+   DAOS Server and Client instances are deployed using images that have DAOS pre-installed.
+
+   Therefore, the images need to be built prior to running Terraform.
+
+  >  Click the button to open an interactive walk-through in Cloud Shell which will guide you through the steps listed above.
+  >
+  >  [![DAOS on GCP Setup](http://gstatic.com/cloudssh/images/open-btn.png)](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/daos-stack/google-cloud-daos&cloudshell_git_branch=main&shellonly=true&tutorial=docs/tutorials/daosgcp_setup.md)
+
+4. **Use DAOS Terraform modules in your Terraform code**
+
+   You will need to write your own Terraform code for your particular use case.
+
+   Your Terraform code can use the modules in ```terraform/modules``` to deploy DAOS server and client instances.
+
+   The example Terraform configurations provided in ```terraform/examples``` can be used as a reference.
+
+   See the [DAOS Cluster](terraform/examples/daos_cluster/README.md) example to learn more about how to use the ```terraform/modules```.
+
+## Links
+
+- [Distributed Asynchronous Object Storage (DAOS)](https://docs.daos.io/)
+- [Google Cloud Platform (GCP)](https://cloud.google.com/)
+- [Google Cloud CLI (gcloud)](https://cloud.google.com/cli)
+- [Google Cloud Build](https://cloud.google.com/build)
+- [Cloud Shell](https://cloud.google.com/shell)
+- [Packer](https://www.packer.io/)
+- [Terraform](https://www.terraform.io/)
 
-- [git](https://git-scm.com/)
-- [Google Cloud SDK](https://cloud.google.com/sdk/docs/install)
-- [Packer](https://www.packer.io/downloads)
-- [Terraform](https://www.terraform.io/downloads)
 
 ## Development
 
-See [Development](docs/development.md)
+If you are contributing to the code in this repo, see [Development](docs/development.md)
+
+## License
 
+[Apache License Version 2.0](LICENSE)
diff --git a/docs/development.md b/docs/development.md
@@ -79,3 +79,61 @@ To run [pre-commit](https://pre-commit.com/) on all files prior to staging them
 ```shell
 pre-commit run --all-files
 ```
+
+## Updating Cloud Shell urls in documentation
+
+Several of the README.md files in this repository contain links that open tutorials in Cloud Shell.
+
+In order for these links to work properly during development the URLs must be changed to point to the correct branch.
+
+Currently Cloud Shell tutorials do not have an automatic way to detect a branch.  Therefore, the branch parameter in the URL must be updated manually.
+
+The `tools/autodoc/cloudshell_urls.sh` script should be used to update the branch parameter in all Cloud Shell URLs that are present in *.md files in this repo.
+
+### Update Cloud Shell URLs when submitting a PR
+
+If your PR changes README.md files that contain Cloud Shell URLs, then prior to requesting a review you should run the following command and push any changes to your dev branch.
+
+```bash
+tools/autodoc/cloudshell_urls.sh --repo-url <your_forked_repo_url> --branch <your_dev_branch_name>
+```
+
+This will allow the reviewers to run Cloud Shell tutorials from your PR branch.
+
+### Update Cloud Shell URLs before merging to a branch
+
+If you are merging changes to `*.md` files with Cloud Shell URLs in them you need to ensure that the URLs are updated with the name of the target branch before you merge.
+
+This is not ideal but it's the only way we can think of doing things for now.
+
+Let's say that you have a PR that has been approved and you want to merge it to the `develop` branch.
+
+Prior to merging you need to run
+
+```bash
+tools/autodoc/cloudshell_urls.sh --repo-url <your_forked_repo_url> --branch develop
+```
+
+And then commit the changes in your dev branch.
+
+Once that is done you can then merge to the `develop` branch.
+
+Now let's say that you want to merge the `develop` branch into the `main` branch.
+
+You will need to check out the https://github.com/daos-stack/google-cloud-daos `develop` branch and run
+
+```bash
+tools/autodoc/cloudshell_urls.sh --repo-url https://github.com/daos-stack/google-cloud-daos --branch main
+```
+
+Commit the changes and push them to the develop branch. After doing this you can merge the `develop` branch to `main`.
+
+Now you will need to set the URLs back to the develop branch.
+
+```bash
+tools/autodoc/cloudshell_urls.sh --repo-url https://github.com/daos-stack/google-cloud-daos --branch develop
+```
+
+Commit the changes and push them to the develop branch.
+
+This is very tedious. We will continue to seek out a better solution for maintaining the Cloud Shell URLs.
diff --git a/docs/tutorials/daosgcp_setup.md b/docs/tutorials/daosgcp_setup.md
@@ -0,0 +1,159 @@
+# DAOS GCP Setup
+
+In this walkthrough you will
+
+1. Set defaults for Google Cloud CLI (```gcloud```)
+2. Create a Packer image in your GCP project
+3. Build DAOS Server and Client images with Packer in Cloud Build
+
+After completing this walkthrough you will be able to run Terraform to deploy DAOS Server and Client instances.
+
+## Project Selection
+
+Select the project that you would like to use for deploying DAOS in GCP.
+
+<walkthrough-project-setup billing="true"></walkthrough-project-billing-setup>
+
+> Note that when running scripts and examples from the [google-cloud-daos](https://github.com/daos-stack/google-cloud-daos) repository, charges will be
+> incurred within the selected project.
+>
+> Always be sure to run ```terraform destroy``` when you no longer need your instances.
+
+Click **Start** to continue
+
+## Set ```gcloud``` defaults
+
+Many of the scripts and examples in the [google-cloud-daos](https://github.com/daos-stack/google-cloud-daos) repository use three default configuration settings in your Google Cloud CLI (```gcloud```) configuration.
+
+The default settings are
+
+1. project
+2. region
+3. zone
+
+
+### Set Default Project
+
+To set the default project run
+
+```bash
+gcloud config set project {{project-id}}
+```
+
+### Set Default Region
+
+To set the default region run
+
+```bash
+gcloud config set compute/region us-central1
+```
+
+The ```us-central1``` region is recommended but feel free to change as needed.
+
+### Set Default Zone
+
+To set the default zone run
+
+```bash
+gcloud config set compute/zone us-central1-f
+```
+
+The ```us-central1-f``` zone is recommended but feel free to change as needed.
+
+### Defaults Set!
+
+You have now set the necessary defaults required for the scripts and examples in the [google-cloud-daos](https://github.com/daos-stack/google-cloud-daos) repository.
+
+Click **Next** to continue
+
+## Create Packer Image
+
+DAOS images are built using [Packer](https://www.packer.io/) in [Cloud Build](https://cloud.google.com/build).
+
+In order to run Packer in Cloud Build you need to provision an instance from an image that has Packer installed.
+
+Therfore, in order to build DAOS images with Packer in Cloud Build, your GCP project must contain a Packer image.
+
+Creating the Packer image only needs to be done once in the GCP project.
+
+### Enable APIs
+
+To enable the necessary APIs for Cloud Build run:
+
+```bash
+gcloud services enable sourcerepo.googleapis.com
+gcloud services enable compute.googleapis.com
+gcloud services enable servicemanagement.googleapis.com
+gcloud services enable storage-api.googleapis.com
+```
+
+### Required IAM permissions
+
+The Cloud Build service account requires the editor role.
+
+To grant the editor role to the service account run:
+
+```bash
+CLOUD_BUILD_ACCOUNT=$(gcloud projects get-iam-policy {{project-id}} --filter="(bindings.role:roles/cloudbuild.builds.builder)" --flatten="bindings[].members" --format="value(bindings.members[])")
+
+gcloud projects add-iam-policy-binding {{project-id}} \
+  --member "${CLOUD_BUILD_ACCOUNT}" \
+  --role roles/compute.instanceAdmin
+```
+
+### Create the Packer Image
+
+Cloud Build provides the [Packer community builder image](https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/packer).
+
+To build the Packer image run:
+
+```bash
+pushd .
+cd ~/
+git clone https://github.com/GoogleCloudPlatform/cloud-builders-community.git
+cd cloud-builders-community/packer
+gcloud builds submit .
+rm -rf ~/cloud-builders-community
+popd
+```
+
+<br>
+
+You have completed the necessary steps to create your Packer image which will be used to build DAOS images with Packer in Cloud Build.
+
+Click **Next** to continue
+
+## Build DAOS Server and Client images
+
+In order to use Terraform to provision DAOS Server and Client instances you need to build images that have DAOS pre-installed.
+
+To build the DAOS Server and Client instances run:
+
+```bash
+pushd .
+cd images
+./build_images.sh --type all
+popd
+```
+
+It will take a few minutes for the images to build.
+
+Wait for the image build to complete.
+
+Click **Next** to continue
+
+## DAOS GCP Setup Complete
+
+<walkthrough-conclusion-trophy></walkthrough-conclusion-trophy>
+
+You can now begin using Terraform to provision DAOS Server and Client instances in the **{{project-id}}** project!
+
+**Next Steps**
+
+- Read the <walkthrough-editor-open-file filePath="terraform/modules/daos_client/README.md">terraform/modules/daos_client/README.md</walkthrough-editor-open-file> file
+- Read the <walkthrough-editor-open-file filePath="terraform/modules/daos_server/README.md">terraform/modules/daos_server/README.md</walkthrough-editor-open-file> file
+- View the files in the ```terraform/examples/daos_cluster``` directory
+- Open a tutorial that walks you through the process of deploying a DAOS cluster using the ```terraform/examples/daos_cluster``` example.
+   ```bash
+   cloudshell launch-tutorial ./docs/tutorials/example_daos_cluster.md
+   ```