Skip to content

Commit

Permalink
Revert "Porting stashed changes from the merged branch onto main (med…
Browse files Browse the repository at this point in the history 
…-diag-adoc)"
  • Loading branch information
@abhatt-rh
abhatt-rh authored Jul 21, 2023
1 parent fdede1e commit 8de1c85
Show file tree
Hide file tree
Showing 7 changed files with 351 additions and 179 deletions.
2 changes: 1 addition & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,4 @@ Gemfile.lock
.vscode
.idea
.vale
modules/.vale.ini
modules/.vale.ini
67 changes: 27 additions & 40 deletions content/patterns/medical-diagnosis/_index.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,67 +24,55 @@ ci: medicaldiag
:_content-type: ASSEMBLY
include::modules/comm-attributes.adoc[]

//Module to be included
//:_content-type: CONCEPT
//:imagesdir: ../../images
[id="about-med-diag-pattern"]
= About the {med-pattern}
== Background

Background::
This Validated Pattern is based on a demo implementation of an automated data pipeline for chest Xray
analysis previously developed by Red Hat. The original demo can be found link:https://github.com/red-hat-data-services/jumpstart-library[here]. It was developed for the US Department of Veteran Affairs.

This validated pattern is based on a demo implementation of an automated data pipeline for chest Xray analysis previously developed by Red Hat. You can find the original demonstration link:https://github.com/red-hat-data-services/jumpstart-library[here]. It was developed for the US Department of Veteran Affairs.
This validated pattern includes the same functionality as the original demonstration. The difference is
that we use the _GitOps_ framework to deploy the pattern including operators, creation of namespaces,
and cluster configuration. Using GitOps provides a much more efficient means of doing continuous deployment.

This validated pattern includes the same functionality as the original demonstration. The difference is that we use the GitOps framework to deploy the pattern including Operators, creation of namespaces, and cluster configuration. Using GitOps provides an efficient means of implementing continuous deployment.
What does this pattern do?:

Workflow::

* Ingest chest Xrays from a simulated Xray machine and puts them into an `objectStore` based on Ceph.
* The `objectStore` sends a notification to a Kafka topic.
* Ingest chest Xrays from a simulated Xray machine and puts them into an objectStore based on Ceph.
* The objectStore sends a notification to a Kafka topic.
* A KNative Eventing Listener to the topic triggers a KNative Serving function.
* An ML-trained model running in a container makes a risk assessment of Pneumonia for incoming images.
* A Grafana dashboard displays the pipeline in real time, along with images incoming, processed, and anonymized, as well as full metrics collected from Prometheus.
* A Grafana dashboard displays the pipeline in real time, along with images incoming, processed and anonymized, as well as full metrics collected from Prometheus.

This pipeline is showcased link:https://www.youtube.com/watch?v=zja83FVsm14[in this video].

image::medical-edge/dashboard.png[link="/images/medical-edge/dashboard.png"]

[WARNING]
====
This validated pattern is still under development. If you have any questions or concerns contact mailto:jrickard@redhat.com[Jonny Rickard] or mailto:claudiol@redhat.com[Lester Claudio]
====
This validated pattern is still under development. Any questions or concerns
please contact mailto:jrickard@redhat.com[Jonny Rickard] or mailto:claudiol@redhat.com[Lester Claudio].

[id="about-solution-med"]
== About the solution elements
=== Solution elements

The solution aids the understanding of the following:
* How to use a GitOps approach to keep in control of configuration and operations.
* How to use a GitOps approach to keep in control of configuration and operations
* How to deploy AI/ML technologies for medical diagnosis using GitOps.

The {med-pattern} uses the following Red Hat products and technologies:
=== Red Hat Technologies

* {rh-ocp} for container orchestration
* {rh-gitops}, a GitOps continuous delivery (CD) product for Kubernetes.
* {rh-amq-first}, an event streaming platform based on the Apache Kafka
* {rh-serverless-first} for event-driven applications
* {rh-ocp-data-first} for cloud native storage capabilities
* {grafana-op} to manage and share Grafana dashboards, datasources, and so on
* {rh-ocp} (Kubernetes)
* {rh-gitops} (ArgoCD)
* Red Hat AMQ Streams (Apache Kafka Event Broker)
* Red Hat OpenShift Serverless (Knative Eventing, Knative Serving)
* Red Hat OpenShift Data Foundations (Cloud Native storage)
* Grafana dashboard (OpenShift Grafana Operator)
* Open Data Hub
* S3 storage

[id="about-architecture-med"]
== About the architecture
== Architecture

[IMPORTANT]
====
There is no edge component in this iteration of the pattern. Edge deployment capabilities are planned part of the pattern architecture for a future release.
====
In this iteration of the pattern *there is no edge component* . Future releases have planned Edge deployment capabilities as part of the pattern architecture.

image::medical-edge/edge-medical-diagnosis-marketing-slide.png[link="/images/medical-edge/edge-medical-diagnosis-marketing-slide.png"]

Components are running on OpenShift either at the data center or at the medical facility (or public cloud running OpenShift).

[id="about-physical-schema-med"]
=== About the physical schema
=== Physical Schema

The diagram below shows the components that are deployed with the various networks that connect them.

Expand All @@ -94,12 +82,11 @@ The diagram below shows the components that are deployed with the the data flows

image::medical-edge/physical-dataflow.png[link="/images/medical-edge/physical-dataflow.png"]

== Recorded demo
== Recorded Demo

link:/videos/xray-deployment.svg[image:/videos/xray-deployment.svg[Demo\]]

[id="next-steps_med-diag-index"]
== Next steps
== What Next

* Getting started link:getting-started[Deploy the Pattern]
//We have relevant links on the patterns page
* Visit the link:https://github.com/hybrid-cloud-patterns/medical-diagnosis[repository]
219 changes: 207 additions & 12 deletions content/patterns/medical-diagnosis/cluster-sizing.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,21 +7,48 @@ aliases: /medical-diagnosis/cluster-sizing/
:toc:
:imagesdir: /images
:_content-type: ASSEMBLY
include::modules/comm-attributes.adoc[]

//AI: Removed the following since we have CI status linked on the patterns page
//[id="tested-platforms-cluster-sizing"]
//== Tested Platforms
[id="tested-platforms-cluster-sizing"]
== Tested Platforms

//AI: Removed the following in favor of the link to OCP docs
//[id="general-openshift-minimum-requirements-cluster-sizing"]
//== General OpenShift Minimum Requirements
The *Medical Diagnosis* pattern has been tested in the following Certified Cloud Providers.

|===
| *Certified Cloud Providers* | 4.8 | 4.9 | 4.10 | 4.11

| Amazon Web Services
| Tested
| Tested
| Tested
| Tested

| Google Compute
|
|
|
|

| Microsoft Azure
|
|
|
|
|===

To know the minimum requirements for an {ocp} cluster, for example on bare-metal, see link: https://docs.openshift.com/container-platform/4.13/installing/installing_bare_metal/installing-bare-metal.html#installation-minimum-resource-requirements_installing-bare-metal[Minimum resource requirements for cluster installation].

[id="general-openshift-minimum-requirements-cluster-sizing"]
== General OpenShift Minimum Requirements

[id="about-med-diagnosis-components"]
=== About {med-pattern} components
OpenShift 4 has the following minimum requirements for sizing of nodes:

* *Minimum 4 vCPU* (additional are strongly recommended).
* *Minimum 16 GB RAM* (additional memory is strongly recommended, especially if etcd is colocated on Control Planes).
* *Minimum 40 GB* hard disk space for the file system containing /var/.
* *Minimum 1 GB* hard disk space for the file system containing /usr/local/bin/.


[id="medical-diagnosis-pattern-components-cluster-sizing"]
=== Medical Diagnosis Pattern Components

Here's an inventory of what gets deployed by the *Medical Diagnosis* pattern:

Expand Down Expand Up @@ -57,7 +84,7 @@ Here's an inventory of what gets deployed by the *Medical Diagnosis* pattern:


[id="medical-diagnosis-pattern-openshift-cluster-size-cluster-sizing"]
=== About {med-pattern} OpenShift Cluster Size
=== Medical Diagnosis Pattern OpenShift Cluster Size

The Medical Diagnosis pattern has been tested with a defined set of configurations that represent the most common combinations that Red Hat OpenShift Container Platform (OCP) customers are using or deploying for the x86_64 architecture.

Expand Down Expand Up @@ -85,4 +112,172 @@ The OpenShift cluster is a standard deployment of 3 control plane nodes and 3 or
| Standard_D8s_v3
|===

//Removed section for instance types as we did for MCG
[id="aws-instance-types-cluster-sizing"]
=== AWS Instance Types

The *Medical Diagnosis* pattern was tested with the highlighted AWS instances in *bold*. The OpenShift installer will let you know if the instance type meets the minimum requirements for a cluster.

The message that the openshift installer will give you will be similar to this message

[,text]
----
INFO Credentials loaded from default AWS environment variables
FATAL failed to fetch Metadata: failed to load asset "Install Config": [controlPlane.platform.aws.type: Invalid value: "m4.large": instance type does not meet minimum resource requirements of 4 vCPUs, controlPlane.platform.aws.type: Invalid value: "m4.large": instance type does not meet minimum resource requirements of 16384 MiB Memory]
----

Below you can find a list of the AWS instance types that can be used to deploy the *Medical Diagnosis* pattern.

[cols="^,^,^,^,^"]
|===
| Instance type | Default vCPUs | Memory (GiB) | Hub | Factory/Edge

|
|
|
| 3x3 OCP Cluster
| 3 Node OCP Cluster

| m4.xlarge
| 4
| 16
| N
| N

| *m4.2xlarge*
| 8
| 32
| Y
| Y

| m4.4xlarge
| 16
| 64
| Y
| Y

| m4.10xlarge
| 40
| 160
| Y
| Y

| m4.16xlarge
| 64
| 256
| Y
| Y

| *m5.xlarge*
| 4
| 16
| Y
| N

| *m5.2xlarge*
| 8
| 32
| Y
| Y

| *m5.4xlarge*
| 16
| 64
| Y
| Y

| m5.8xlarge
| 32
| 128
| Y
| Y

| m5.12xlarge
| 48
| 192
| Y
| Y

| m5.16xlarge
| 64
| 256
| Y
| Y

| m5.24xlarge
| 96
| 384
| Y
| Y
|===

The OpenShift cluster is made of 3 Control Plane nodes and 3 Workers. For the node sizes we used the *m5.4xlarge* on AWS and this instance type met the minimum requirements to deploy the *Medical Diagnosis* pattern successfully.

To understand better what types of nodes you can use on other Cloud Providers we provide some of the details below.

[id="azure-instance-types-cluster-sizing"]
=== Azure Instance Types

The *Medical Diagnosis* pattern was also deployed on Azure using the *Standard_D8s_v3* VM size. Below is a table of different VM sizes available for Azure. Keep in mind that due to limited access to Azure we only used the *Standard_D8s_v3* VM size.

The OpenShift cluster is made of 3 Control Plane nodes and 3 Workers.

|===
| Type | Sizes | Description

| https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-general[General purpose]
| B, Dsv3, Dv3, Dasv4, Dav4, DSv2, Dv2, Av2, DC, DCv2, Dv4, Dsv4, Ddv4, Ddsv4
| Balanced CPU-to-memory ratio. Ideal for testing and development, small to medium databases, and low to medium traffic web servers.

| https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-compute[Compute optimized]
| F, Fs, Fsv2, FX
| High CPU-to-memory ratio. Good for medium traffic web servers, network appliances, batch processes, and application servers.

| https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-memory[Memory optimized]
| Esv3, Ev3, Easv4, Eav4, Ev4, Esv4, Edv4, Edsv4, Mv2, M, DSv2, Dv2
| High memory-to-CPU ratio. Great for relational database servers, medium to large caches, and in-memory analytics.

| https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-storage[Storage optimized]
| Lsv2
| High disk throughput and IO ideal for Big Data, SQL, NoSQL databases, data warehousing and large transactional databases.

| https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-gpu[GPU]
| NC, NCv2, NCv3, NCasT4_v3, ND, NDv2, NV, NVv3, NVv4
| Specialized virtual machines targeted for heavy graphic rendering and video editing, as well as model training and inferencing (ND) with deep learning. Available with single or multiple GPUs.

| https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-hpc[High performance compute]
| HB, HBv2, HBv3, HC, H
| Our fastest and most powerful CPU virtual machines with optional high-throughput network interfaces (RDMA).
|===

For more information please refer to the https://docs.microsoft.com/en-us/azure/virtual-machines/sizes[Azure VM Size Page].

[id="google-cloud-gcp-instance-types-cluster-sizing"]
=== Google Cloud (GCP) Instance Types

The *Medical Diagnosis* pattern was also deployed on GCP using the *n1-standard-8* VM size. Below is a table of different VM sizes available for GCP. Keep in mind that due to limited access to GCP we only used the *n1-standard-8* VM size.

The OpenShift cluster is made of 3 Control Plane and 3 Workers cluster.

The following table provides VM recommendations for different workloads.

|===
| *General purpose* | *Workload optimized* | | | |

| Cost-optimized | Balanced | Scale-out optimized | Memory-optimized | Compute-optimized | Accelerator-optimized

| E2
| N2, N2D, N1
| T2D
| M2, M1
| C2
| A2

| Day-to-day computing at a lower cost
| Balanced price/performance across a wide range of VM shapes
| Best performance/cost for scale-out workloads
| Ultra high-memory workloads
| Ultra high performance for compute-intensive workloads
| Optimized for high performance computing workloads
|===

For more information please refer to the https://cloud.google.com/compute/docs/machine-types[GCP VM Size Page].
Loading

0 comments on commit 8de1c85

Please sign in to comment.