opsman.prolific

What is Ops Manager?

### What?
Ops Manager is a web application that you use to streamline deploying and managing Tanzu Platform Cloud Foundry (TPCF) deployments.

Ops Manager can deploy many different Tanzu software products such as Cloud Foundry, Gemfire, and Kubernetes (TKGI) as well as partner products like Jenkins. These products are represented as Tiles in Ops Manager.
A Tile is a zip file with a `.pivotal` extension containing:
- one or more BOSH release tarballs containing the product's source code and dependencies
- a metadata file used to generate a BOSH manifest and Ops Manager form elements for that product
- migration files to automatically handle property changes when upgrading to a newer Tile

## UI Walkthrough

The Ops Manager Dashboard looks like the following:

Visit https://pcf.dhaka.cf-app.com/ to login, click the "Broadcom Okta" link at the bottom ans sign in as usual. You should have read-only access to the Opsman UI.

The Ops Manager shown here has many Tiles installed:
- The BOSH Director tile comes pre-installed with your Ops Manager. Ops Manager will deploy a BOSH Director on your chosen IaaS to manage your deployments as it's first task on install and upgrade.
- The Small Footprint VMware Tanzu Application Service Tile is the Cloud Foundry deployment. This is roughly equivalent to the Cloud Foundry you deployed in earlier stories with cf-deployment. Small Footprint is used to reduce IAAS costs as it puts all the different CF vms on a smaller set of VMs as containers. It should not be use in production.
- Many of these tiles provide services (like a Mysql DB) for CF apps to use.

More information about the Dashboard UI can be found [here](hhttps://docs.vmware.com/en/VMware-Tanzu-Operations-Manager/3.0/vmware-tanzu-ops-manager/pcf-interface.html).
Tiles can be downloaded from Broadcom. We won't have entitlements to see the same customer portal but going through ERP Release Management should allow you to download tiles.

Clicking on the "BOSH Director for GCP" tile, it looks like the following:

![Tile UI](https://github.com/pivotal/cf-onboarding/raw/master/images/director-tile.png)

Tiles provide multiple configuration panes that let you select configuration values needed for the product.
For example, "Google Config", "Director Config", and "Create Available Zones" are all forms to select your Director and IaaS configuration.
After saving these forms and clicking Apply Changes on the Dashboard, Ops Manager will deploy a BOSH Director using the provided values.
All other Tiles will be deployed as a separate BOSH deployment using that Director.
For example, filling out the forms on the TAS for VMs Tile will generate a BOSH manifest and clicking Apply Changes will then deploy that manifest using the BOSH Director.
After filling out the Director forms and clicking Apply Changes, Ops Manager will run `bosh upload-release`, `bosh deploy`, and other BOSH commands you've run in previous stories and stream the output to the UI.

In addition to the UI, Ops Manager provides an [API](https://docs.vmware.com/en/VMware-Tanzu-Operations-Manager/3.0/vmware-tanzu-ops-manager/install-ops-man-api.html) to allow users to configure Tiles via scripts.
The UI provides a nice onboarding experience, but we still encourage customers to setup Concourse pipelines to automatically configure their tiles.
This ensures configuration is checked into source control and is reproducible in case something goes wrong later on.
The [om CLI](https://github.com/pivotal-cf/om) can be used to programmatically perform operations in Ops Manager rather than clicking around the UI. For example, you can get current product config, make some changes, and update Ops Manager to reflect those changes using the following commands:

```
# This will output a STAGED-CONFIG.yml
om -u USERNAME -p PASSWORD -k -t https://OPS-MAN-URL-OR-IP/uaa staged-config -p TILE-NAME

# Whatever the file ended up being named in the previous command
vim STAGED-CONFIG.yml

# Re-upload it!
om -u USERNAME -p PASSWORD -k -t https://OPS-MAN-URL-OR-IP/uaa configure-product -c STAGED-CONFIG.yml
```

## One Platform

TPCF refers to your entire IT platform, not just the Cloud Foundry deployment.
A company might require some apps to run on CF and others on Kubernetes.
These apps might need Databases, logging, and other services to be deployed as well.
Ops Manager provides a single Dashboard to manage this wide array of components.

Links:
- [TPCF Docs](https://docs.vmware.com/en/VMware-Tanzu-Application-Service/index.html)

L: opsman, onboarding-lite

---

Differences between PCF and cf-deployment

### CF vs PCF

In the stories earlier in onboarding week, you used `bbl up` to create some GCP infrastructure and BOSH + cf-deployment to deploy a Cloud Foundry.
Here's a quick rundown of the similarities and differences between Ops Manager + TAS for VMs Tile and cf-deployment.

### Technical Similarities and Differences

Similarities:
- Both use BOSH under the hood to deploy software
- Many products can be deployed with either OSS tooling or Ops Manager, e.g. Cloud Foundry or MySQL
- The App Developer CLI experience such as `cf push` is the same, although TPCF adds web apps like [Apps Manager](https://docs.vmware.com/en/VMware-Tanzu-Application-Service/6.0/tas-for-vms/toc-using-apps-manager-index.html) to streamline this experience
- We would encourage both TPCF and OSS users to configure their environments with a Concourse pipeline

Differences:
- Ops Manager uses UI forms or API calls to configure each product while cf-deployment uses a list of ops-files
- Ops Manager Tiles include migrations to automatically handle changes like BOSH property renames when upgrading to newer versions
  - cf-deployment users would need to carefully read release notes to handle any manifest changes
- Ops Manager provides input verifiers to perform input validation prior to deploying, such as ensuring DNS records exist or Database credentials are valid
- PCF provides long-term support (LTS) for Pivotal customers
  - Many customers have complicated change management processes which require new features to be carefully vetted before deploying to production
  - These customers often stay on older versions of PCF products for many months after newer versions have been released
  - LTS means that Tanzu will backport security fixes to previously released versions of TPCF products, while new features are only added to the latest release version
  - cf-deployment users will need to upgrade to the latest release in order to get these security fixes
- Ops Manager provides a consistent experience to install many different products
  - While lots of work has gone into making deploying Cloud Foundry with cf-deployment a nice experience, the quality of documentation on deploying other BOSH releases is highly variable
- Ops Manager attempts to abstract away the BOSH implementation details, although you will still interact with BOSH when debugging
  - This added layer of abstraction can make debugging more difficult
- Clicking Apply Changes in Ops Manager is typically slower than running `bosh deploy`
  - Ops Manager performs many safety checks to ensure everything is in a good state, but this takes additional time
- Tile authors often only expose a subset of BOSH properties via the UI
  - A smaller set of properties to configure makes it easier to get something working initially, but can prevent you from configuring more advanced features
- Ops Manager does not allow you to change the deployment topology (which jobs are located on which VMs)
  - In cf-deployment you can add an ops-file to co-locate a new job on an existing VM, Ops Manager requires the Tile author to make changes to their Tile to allow this
  - Some Tiles like TAS for VMs provide both a full HA version and a "Small Footprint" version to co-locate jobs on a smaller number of VMs

### Ecosystem Similarities and Differences

Similarities:
- Most (if not all) features of core CF components like Diego, UAA, and CAPI are available in both TAS for VMs and cf-deployment
- Some customers use a mix of cf-deployment and Ops Manager across their organization
- Engineers often contribute to both cf-deployment and Ops Manager Tiles
  - e.g. a Diego engineer might submit a PR to add an ops-file to cf-deployment to toggle on a new feature, then submit a PR to the TAS for VMs Tile to add a checkbox to toggle that same feature

Differences:
- cf-deployment and associated tooling (e.g. bbl) are open-source while Ops Manager and most Tiles are closed-source
- The Cloud Foundry Foundation owns cf-deployment (along with all other projects in the `cloudfoundry` GitHub org) while Broadcom owns Ops Manager and many Tiles
- Some Cloud Foundry teams have engineers who are employed by other companies within the CF Foundation like IBM and SAP. These engineers will contribute to open-source components, but generally will not contribute to Broadcom-owned software
- Users download open-source BOSH releases and stemcells from [bosh.io](https://bosh.io) while Broadcom commercial customers will download tiles and stemcells from the Broadcom Portal (RMT).
  - The Broadcom Portal (RMT) also contains Partner Services which are Tiles jointly developed by Broadcom and partner companies like Datadog and New Relic

L: opsman, onboarding-lite

---

You decide: Claim a deployed Ops Manager or deploy one yourself

## Decision time!

**Note:** You may have already created a Toolsmiths environment as part of the **Basic BOSH Knowledge** story. If so, feel free to use that one!

By this point you're probably ready to start playing around with an Ops Manager environment.
You have two options for getting one:
- Deploy one from scratch OR
- Claim an already deployed environment

If you want to experience the workflow a customer would go through and want learn a bit more about creating GCP environments, try setting up an environment manually.
The next few stories in the backlog will guide you through this process.
If you've had enough of the IaaS stuff for this week, the following steps will show you how to grab an already deployed Ops Manager with TAS for VMs installed.

## Optional: Claim a Toolsmiths environment

**Warning!** To keep costs down, Toolsmiths environments are automatically destroyed after 24 hours

The CF Toolsmiths team ([#cf-toolsmiths slack](https://pivotal.slack.com/messages/C0563B53F/)) maintains a pool of already deployed Ops Manager + TAS for VMs environments for teams across the org to grab for testing.
You can see a dashboard of all available environments [here](https://environments.toolsmiths.cf-app.com/pooled_gcp_environments).
Note: you must be on the VPN to see this app.

The easiest way to claim an environment to play around with is to use the [smith CLI](https://github.com/pivotal/smith).
The [README](https://github.com/pivotal/smith#smith---the-community-cli-for-toolsmiths) gives instructions for getting the CLI installed and configured.
Your `.smith-token-hook.sh` file should contain:

```
#!/bin/bash
lpass show --notes 'Shared-CF SF Onboarding/Toolsmiths API Token'
```
You should have been given access to this LastPass folder at the start of the week, let your facilitators know if you still don't have access.

You should now be able to claim an environment using the steps described [here](https://github.com/pivotal/smith/blob/master/EXAMPLES.md).
We recommend grabbing the latest TAS for VMs version so you can play around with all the latest and greatest features.
Talk to your facilitators or ask the Toolsmiths team in slack for help if you run into any issues and consider giving the Toolsmiths a Thank You for providing such a valuable service!

L: opsman, onboarding-lite

---

Deploy Ops Manager Manually

### Before You Go Any Further

**This story is optional.** There are easier ways to get an Ops Manager, and you should only do this if you are terribly interested in it. If you already opted to use a Toolsmiths environment, you especially needn't trouble yourself with this and should enjoy the easy life of on-demand environment provisioning.

Okay, now that you've been warned...

### Ops Manager Interface
[Understand how an Operator uses Ops Manager](https://docs.vmware.com/en/VMware-Tanzu-Operations-Manager/3.0/vmware-tanzu-ops-manager/pcf-interface.html)

## Deploying on GCP
While we use BOSH to deploy a director and many of our services, Ops Manager is currently not deployed with BOSH.

The first step to installing Ops Manager is to set up your infrastructure.
There are some terraform scripts that will do this for you that you can download from Pivotal Network.
Go to [network.pivotal.io](https://network.pivotal.io) and make sure you're signed in to PivNet (if you don't have an account yet, click "Register" to create one.).
Next, search for `VMware Tanzu Application Service for VMs`. Once you open the product, you should see GCP Terraform templates in "Release Download Files".

Unzip the file to get the terraform templates. You can take a look at the README to see how the whole repo works, but the important bit is in the section titled "Var File". Use those instructions to build a var file for terraform. **Take care to notice some of the following gotcha's.**
1. The `project` variable needs to equal the ID of your GCP project. That's not the same thing as the _name_ of your GCP project. To find the ID, click on the project dropdown to the left of the search bar. You should say a table mapping project names to project IDs.
2. The `opsman_image_url` bears some explanation. As it's name implies, it's a URL to an OpsMan image. In the GCP case specifically, that means tha the OpsManager team has already uploaded an image in a global GCP bucket, so all you need to do is point your terraform templates at the right image. To construct the URL, you'll first need to know that all buckets have a URL of the form `https://storage.googleapis.com/<bucket name>/<file name>`. To find the bucket and file name, you'll want to find the "Pivotal Cloud Foundry Operations Manager" product on PivNet, and then download "Ops Manager for GCP". Surprise! What you actually got was a PDF that includes the bucket and file name. Note that there are different buckets for different regions -- pick the one that makes the most sense. You can test your url by running `curl --head URL` -- if you get a 200, you're in good shape.
3. `sevice_account_key` needs to contain GCP credentials that have the "owner" [Identity and Access Management (IAM) role](https://cloud.google.com/compute/docs/access/iam). If you are reusing the service account key from the OSS GCP onboarding epic then you will need to add the "owner" role to it. This can be done by adding another iam-policy-binding to it using `gcloud projects add-iam-policy-binding PROJECT_ID --member 'serviceAccount:ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com' --role 'roles/owner'`. You'll then paste the contents of the service account key file into the var file. Make sure to keep the `<<SERVICE_ACCOUNT_KEY` and `SERVICE_ACCOUNT_KEY` tokens, as those designate the beginning and end of the creds.

4. `dns_suffix` is like the `system_domain` from the OSS GCP track -- it should be some subdomain of `cf-onboarding.com` and is used to represent your particular environment.
5. For SSL configuration, let's use self-signed certificates. That means you'll set values for `ssl_cert` and `ssl_private_key`. To generate SSL keys, run `openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -nodes`. When you generate the certificates, make sure that the "Common Name" field matches your `dns_suffix`.
6. Leave `buckets_location`, `opsman_storage_bucket_count`, `create_iam_service_account_members` blank.

Finally, you'll run
```
terraform init
terraform apply
```

Terraform will not only pave your IaaS for you -- it will also deploy the OpsMan VM for you. However, before you can use the provided domain to access OpsMan, you need to do one last manual DNS hookup:
1. In the "Cloud DNS" tab in the GCP console, click on the zone that matches your full OpsMan domain.
2. You should see an `NS` record with four domains in the `Data` field. Copy them.
3. Go back to the "Cloud DNS" and navigate to the zone that matches your DNS suffix (it should look like `<env-name>.<project-name>.cf-app.com`).
4. Add a record set. It should be an `NS` record, you'll need to add each of the four domains you copied as a new name server.
5. Test that it works by running `nslookup pcf.<env_name>.<dns_suffix>`. If it returns an IP address, you've successfully set up DNS.

Congrats! You've got OpsManager up and running.

Before you can use it, you need to configure authentication. Your best bet is to use "Internal Authentication." Create a user with whichever useername, password, and decryption passphrase you want (you can leave the http proxy information blank).

### Expected Results
OpsManager is running and you can log in.

### Relevant Teams
- #ops-manager Slack
- #pas-releng Slack

### Resources
- https://docs.vmware.com/en/VMware-Tanzu-Operations-Manager/3.0/vmware-tanzu-ops-manager/gcp-index.html

L: opsman

---

Basic BOSH Knowledge

### What if I just want an environment to run BOSH CLI commands in and I don't care how it all works?

0. If you have access, visit https://environments.toolsmiths.cf-app.com/ and get yourself an environment. If you don't have access, ask the person who's guiding you through this learning to do it for you.
0. Place the `.json` file you receive from the creation of this environment under one of the `claimed` directories under `denver-locks`. You should now have access to such commands as `pcf-target` if you installed the Denver tooling via `denver-bash-it`.
0. If `pcf-target` succeeds, you should now be connected to the environment and have the ability to run BOSH CLI commands :D

### What is BOSH and why did we use it in the first place?

- [Installing and using the BOSH CLI](https://bosh.io/docs/cli-v2-install/)
- [Why Bosh? Part 1: BOSH Unique Capabilities](https://www.youtube.com/watch?v=PBiX5nRCHPs)
- [Why Bosh? Part 2: Platforms Running on BOSH](https://www.youtube.com/watch?v=HoY5KgYcx0I)

### How do I make changes to a BOSH deployment to test a fix?

- [Installing Cloud Foundry Bosh on VirtualBox.](https://www.youtube.com/watch?v=Kv0iKE2JaJ4)
- [BOSH Tutorial Part 3: Configuring BOSH](https://www.youtube.com/watch?v=BPrJiOCRWS8&list=PLu8oHHyQbBrTGvoGdF7NuZqKqU_31cKkc&index=3)
- [BOSH Tutorial Part 4: First Deployment](https://www.youtube.com/watch?v=vr1K4PWnG00&list=PLu8oHHyQbBrTGvoGdF7NuZqKqU_31cKkc&index=4)
- [BOSH Tutorial Part 5: Exploring Deployments](https://www.youtube.com/watch?v=w7LXlZeDjbs&list=PLu8oHHyQbBrTGvoGdF7NuZqKqU_31cKkc&index=5)

### What if I want to automate a BOSH deployment?

- [BOSH Tutorial part 13: Automating BOSH deployments](https://www.youtube.com/watch?v=NcEbgVSLTYg&list=PLu8oHHyQbBrTGvoGdF7NuZqKqU_31cKkc&index=13)

### What if I want a comprehensive explanation of BOSH?

- [Full BOSH Tutorial](https://www.youtube.com/playlist?list=PLu8oHHyQbBrTGvoGdF7NuZqKqU_31cKkc)
- [Bob's talk about CF, BOSH, and Healthwatch](https://vmware.zoom.us/rec/share/AyhV-PDvaUafOGJI6VoEQuQi2T17-s5GavFwF3EwousomEa7I_D77bTH6GitBKQD.dDt82PKHG6v2m_Fv)
- [Ultimate Guide to BOSH](https://ultimateguidetobosh.com/)
- https://bosh.io/docs/bosh-components/

L: bosh

---

Deploy BOSH Director using Ops Manager

**Warning:** This story is not relevant if you already opted for the Toolsmiths environment instead of manual creation. Okay, now that you've been warned...

Configure the BOSH GCP tile using this [guide](https://docs.vmware.com/en/VMware-Tanzu-Operations-Manager/3.0/vmware-tanzu-ops-manager/install-ops-man-api.html). Some tips to keep you from getting stuck:
1. When you get to "Create Availability Zones", you'll want to use the same AZs that you specified in your terraform.
2. Make sure to read in the instructions carefully when you configure the networks. Especially be careful when setting the network names: they have the form `<network-name>/<subnet-name>/<region>`

After you've configured the BOSH Director tile (it should be green to indicate that you're finished configuring the tile), you can visit the **Review Pending Changes** page and click **Apply Changes**. You'll see the same logs as if you used `bbl` or the `bosh` CLI.
Once the deployment is done, you can visit `/change_log` to review the logs of past deployment.

After the deploy succeeds, click around and see if you can find information about the BOSH Director's health. Can you find the BOSH Director's IP address? How much memory/disk/CPU is it using?

### Expected Results
OpsMan successfully deploys a BOSH Director.

### Resources
- [Manually installing Ops Manager on GCP](https://docs.vmware.com/en/VMware-Tanzu-Operations-Manager/3.0/vmware-tanzu-ops-manager/gcp-index.html)

L: opsman

---

Deploy some tiles

## What?
OpsManager uses "tiles" to deploy different products. You can find tiles by searching on [PivNet](https://network.pivotal.io). Then you can download them, upload them to your OpsManager, and finally configure and deploy them.

Some tiles have dependencies on other tiles. You should find information about that in the product description on PivNet.

To understand the workflow of an actual TPCF/OpsMan customer, choose a tile or two to install and follow the appropriate docs.

## How?
Let's start by installing the mother of them all: TAS for VMs. Go to ERP, search for "VMware Tanzu Application Service for VMs", and download the "Small Footprint TAS for VMs" -- we'll use this to save some time during the deploy. Your best bet is to choose the latest stable version (avoid releases with `alpha` or `rc` in it).

And yes, the file is rather large (as of this writing, it's 13 GB), so feel free to take a ping pong break while it's downloading.

Use the official docs as guide to configure and deploy TAS for VMs. You can the docs for TAS for VMs 2.3 [here](https://docs.pivotal.io/pivotalcf/customizing/gcp-er-config.html), but if you're using a new version than that, you can use the dropdown next to "Pivotal Cloud Foundry" to find docs for the appropriate version.

## Expected Result
You've got a running TAS for VMs.

L: opsman, onboarding-lite

---

[RELEASE] OpsManager ⇧