From bd51a8f16ecbee15be2ca23dca34383696e86045 Mon Sep 17 00:00:00 2001 From: hacktobeer Date: Fri, 2 Aug 2024 19:08:59 +0200 Subject: [PATCH] Turbinia codelab 101 updates (#1523) * typo * remove $ in front of command for easy copy-pasting * remove comment * up skaffold deadline to 90 seconds to give redis enough time to stabelize * clearer sentence * add httpx to cli dependencies * use turbinia-client upload command * upload command output * typos and links * more verbose explanation of debugger * text test updates * updated test code --- docs/developer/develop-minikube.md | 54 ++++++++++++--------- docs/developer/turbinia-codelab-analyser.md | 33 +++++-------- skaffold.yaml | 2 +- turbinia/api/cli/pyproject.toml | 1 + 4 files changed, 45 insertions(+), 45 deletions(-) diff --git a/docs/developer/develop-minikube.md b/docs/developer/develop-minikube.md index 9d4379ee3..ece5bfe25 100644 --- a/docs/developer/develop-minikube.md +++ b/docs/developer/develop-minikube.md @@ -14,7 +14,7 @@ NOTE: This setup has been tested by the Turbinia developers in the following con * If you want to develop with a full VSCode web interface you can use the [GCP Cloud Shell Editor](https://shell.cloud.google.com/) * This setup has all the dependecies and VSCode extensions neeeded for Turbinia development pre-installed and configured. You can continue the setup at the section [here](#start-minkube-cluster). * Take note of the limitations [here](https://cloud.google.com/shell/docs/quotas-limits) - * If the limitations are an issue for you, have a look into at the (paid) version called [Google Cloud Workstation](https://cloud.google.com/workstations/). + * If the limitations are an issue for you, have a look into the (paid) version called [Google Cloud Workstation](https://cloud.google.com/workstations/). ## Components ### Minikube @@ -64,35 +64,35 @@ We will now start the minikube k8s cluster NOTE: If you want to change the default cluster CPU and Memory usage, you can set those before starting the cluster - $ minikube config set cpus 4 - $ minikube config set memory 16384 + minikube config set cpus 4 + minikube config set memory 16384 ### Turbinia source and deployment code Now we have VSCode setup we are going to get a copy of the Turbinia source and deployment code. Clone the [Turbinia repository](https://github.com/google/turbinia) by forking the Turbinia repository into your own Github account and clone it locally from there. - $ git clone ssh://git@github.com:[YOUR_GITHUB_ACCOUNT]/turbinia.git + git clone ssh://git@github.com:[YOUR_GITHUB_ACCOUNT]/turbinia.git Let's get the helm charts for the Turbinia deployment. In your cloned turbinia repository - $ mkdir charts && cd charts - $ helm repo add osdfir-charts https://google.github.io/osdfir-infrastructure - $ helm pull osdfir-charts/turbinia --untar && cd .. + mkdir charts && cd charts + helm repo add osdfir-charts https://google.github.io/osdfir-infrastructure + helm pull osdfir-charts/turbinia --untar && cd .. Open up the turbinia folder in VSCode. ### Prepare Cluster Open a terminal (inside VSCode is the easiest, but any terminal will do) and let's configure skaffold, the local cluster and the additional helm repository for Redis. - $ skaffold config set --global local-cluster true - $ eval $(minikube -p minikube docker-env) - $ helm repo add bitnami https://charts.bitnami.com/bitnami - $ helm repo add kube-prometheus-stack https://prometheus-community.github.io/helm-charts + skaffold config set --global local-cluster true + eval $(minikube -p minikube docker-env) + helm repo add bitnami https://charts.bitnami.com/bitnami + helm repo add kube-prometheus-stack https://prometheus-community.github.io/helm-charts ### Verify Setup Execute a build with skaffold (from the root of the cloned Turbinia Github repository) - $ skaffold build + skaffold build This will build a Turbinia Server container image succesfully if skaffold has been correctlty setup and configured as described above. @@ -101,11 +101,11 @@ This will build a Turbinia Server container image succesfully if skaffold has be ### Install the Turbinia Client We will install the Turbinia client into a Python virtual environment to be able to control Turbinia during our development workflow. - $ python -m venv .venv (or use your favorite virtual env manager) - $ source .venv/bin/activate - $ pip install poetry - $ cd turbinia/api/cli - $ poetry install + python -m venv .venv + source .venv/bin/activate + pip install poetry + cd turbinia/api/cli + poetry install Create the Turbinia Client configuration file in `$HOME/.turbinia_api_config.json` using the base configuration from [here]( https://pypi.org/project/turbinia-client/). @@ -115,7 +115,7 @@ https://pypi.org/project/turbinia-client/). ### Run Now we are ready to run the development cluster of Turbinia. This will startup the Turbinia API server, the worker and the server in the local minikube k8s cluster. - $ skaffold dev + skaffold dev NOTE: if one of the services fails to deploy, try again. Sometimes a time-out occurs due to Redis starting too slow. @@ -142,25 +142,31 @@ NOTE: As python hot reloading of code into an already running process is tricky Keep in mind that hot-reloading: * will load a changed Python source code file as Python byte code in memory of a running process * will not re-execute any change code automatically -* means that you need to execute something to trigger the code path and code you changed and was hot reloaded +* means that you need to execute something to trigger the code path and code you changed ## Test Run Let's test the whole setup by executing a request with a disk image located at `test_data/artifact_disk.dd`. -Copy the disk to one of the containers in the shared `/mnt/turbiniavolume` folder. +Upload the disk to the shared `/mnt/turbiniavolume` folder using the `evidence upload` command. - $ kubectl cp artifact_disk.dd dev-release-turbinia-server-6d6:/mnt/turbiniavolume/ + turbinia-client evidence upload -p test_data/artifact_disk.dd 12345 + + # /home/user/turbinia/test_data/artifact_disk.dd: + * Original Name: artifact_disk.dd + * File Name: artifact_disk_2024-08-01T06:46:51.610412Z.dd + * File Path: /mnt/turbiniavolume/upload/12345/artifact_disk_2024-08-01T06:46:51.610412Z.dd + * Size: 20971520 Start a Turbinia rawdisk request. - $ turbinia-client submit rawdisk --source_path /mnt/turbiniavolume/artifact_disk.dd + turbinia-client submit rawdisk --source_path /mnt/turbiniavolume/upload/12345/artifact_disk_2024-08-01T06:46:51.610412Z.dd - Sending request: {'evidence': {'type': 'RawDisk', 'source_path': '/mnt/turbiniavolume/artifact_disk.dd'}, 'request_options': {}} + Sending request: {'evidence': {'type': 'RawDisk', 'source_path': '/mnt/turbiniavolume/upload/12345/artifact_disk_2024-08-01T06:46:51.610412Z.dd'}, 'request_options': {}} Received response: {'request_id': '4d76df84849c484a835d37fbc7668122'} You can check the Turbinia WebUI at http://localhost:8000 or use the turbinia-client to verify the status of the request. - $ turbinia-client status request 4d76df84849c484a835d37fbc7668122 + turbinia-client status request 4d76df84849c484a835d37fbc7668122 ### Next diff --git a/docs/developer/turbinia-codelab-analyser.md b/docs/developer/turbinia-codelab-analyser.md index ba17ced45..d2f265cd2 100644 --- a/docs/developer/turbinia-codelab-analyser.md +++ b/docs/developer/turbinia-codelab-analyser.md @@ -44,15 +44,11 @@ $ minikube start We will add our new analyser name to the template configuration file. Add the following to `turbinia/config/turbinia_config_tmpl.py` ``` - -... }, { 'job': 'OSInfoAnalysisJob', 'programs': ['grep'], 'docker_image': None, 'timeout': 3600 -}, -... ``` We will import our new job into the job management init code. Add the following to `​​turbinia/jobs/__init__.py` @@ -61,23 +57,16 @@ We will import our new job into the job management init code. Add the following from turbinia.jobs import os_info ``` -We will add our new task name and import that task in the task management code. Add the task name to `turbinia/taskutils.py` +We will add our new task name and import that task in the task management code. Add the task name to `TASK_LIST` in turbinia/taskutils.py` ``` -TASK_LIST = [ - ... - 'OSInfoAnalysisTask' -... -] + 'OSInfoAnalysisTask', ``` and in the same file in the function `gettask()` add ``` -# Late imports to minimize what loads all Tasks -... from turbinia.workers.analysis.os_info import OSInfoAnalysisTask -... ``` Create the boilerplate code for the job. Create a new file at `turbinia/jobs/os_info.py` @@ -265,6 +254,7 @@ Your Turbinia stack is now deployed with your new Job and Task. The stack can be ``` $ kubectl get pods + NAME READY STATUS RESTARTS AGE dev-release-redis-master-0 1/1 Running 0 45s dev-release-turbinia-api-7dfd8988b8-bprxd 1/1 Running 0 45s @@ -296,14 +286,11 @@ $ turbinia-client config download > charts/turbinia/turbinia.conf Add the new `OSInfoAnalyserJob` configuration to that file as well. ``` -... }, { 'job': 'OSInfoAnalysisJob', 'programs': ['grep'], 'docker_image': None, 'timeout': 3600 -}, -... ``` ## Run, debug and fix analyser @@ -314,6 +301,7 @@ Now upload this evidence file to Turbinia giving a fake ticket ID `12345`. ``` $ turbinia-client evidence upload -p osrelease.tgz 12345 + # /home/user/osrelease.tgz: * Original Name: osrelease.tgz * File Name: osrelease_2024-07-24T09:36:58.045255Z.tgz @@ -337,6 +325,7 @@ We can check the status of the request in the Turbinia WebUI or via the cli. ``` $ turbinia-client status request 42d715d298c24e22ba1f4e88d25d4a44 + ## Request ID: 42d715d298c24e22ba1f4e88d25d4a44 * Last Update: 2024-07-24T10:22:10.792302 * Requester: user_unspecified @@ -364,7 +353,7 @@ Error opening OSRelease file: [Errno 2] No such file or directory: '/tmp/42d715d We seem to have incorrectly constructed the path to the `/etc/os-release` file. -Let's set two breakpoints (why only do 1, when you can have 2 ;) ) in our code to see what is going on and attach to the running worker. +Let's set two breakpoints (why only do 1, when you can have 2 ;) ) in our code to see what is going on and attach the debugger to the running worker using the VSCode debug panel. ![Attach Debugger](../images/codelab-debug.png) @@ -378,6 +367,8 @@ We made a typo in `osrelease_path`, it should be `'etc/os-release'`. Fixing and Now let's submit a new request again. +NOTE: make sure to exit past the breakpoints in the debugger to enable the worker to continue before submitting the new request. + ``` $ turbinia-client submit compresseddirectory --jobs_allowlist OSInfoAnalysisJob --source_path /mnt/turbiniavolume/12345/osrelease_2024-07-24T09:36:58.045255Z.tgz Sending request: {'evidence': {'type': 'CompressedDirectory', 'source_path': '/mnt/turbiniavolume/12345/osrelease_2024-07-24T09:36:58.045255Z.tgz'}, 'request_options': {'jobs_allowlist': ['OSInfoAnalysisJob']}} @@ -427,7 +418,7 @@ class OSInfoAnalysisTaskTest(unittest.TestCase): @mock.patch( "builtins.open", new=mock.mock_open(read_data=_OS_RELEASE_CONTENT), create=True) - @mock.patch('os.path.join', return_value='/etc/os-release') + @mock.patch('os.path.join', return_value='etc/os-release') def test_run(self, os_join_mock): """Test OSInfoAnalysisTask task run.""" task = os_info.OSInfoAnalysisTask() @@ -443,9 +434,11 @@ if __name__ == '__main__': unittest.main() ``` -Now let's run this specific test from the root turbinia folder. +Now let's install the test requirements and run this specific test from the root turbinia folder. ``` +$ pip install -f requirements-test.txt $ PYTHONPATH=. python -m unittest turbinia.workers.analysis.os_info_test + Using fallback source config Copy turbinia/config/turbinia_config_tmpl.py to ~/.turbiniarc or /etc/turbinia/turbinia.conf, edit, and re-run. .... ---------------------------------------------------------------------- @@ -456,6 +449,6 @@ OK ## Tips & Tricks -### Unexplainable reloads +### Unexplainable reloads or failed hot-reloads of code Restart deployment with `skaffold dev` (ctrl-c the running instance) diff --git a/skaffold.yaml b/skaffold.yaml index 520daeffe..d108be8ed 100644 --- a/skaffold.yaml +++ b/skaffold.yaml @@ -32,7 +32,7 @@ build: infer: - turbinia/** deploy: - statusCheckDeadlineSeconds: 30 + statusCheckDeadlineSeconds: 90 helm: releases: - name: dev-release diff --git a/turbinia/api/cli/pyproject.toml b/turbinia/api/cli/pyproject.toml index 16fab92f1..dbb3337eb 100644 --- a/turbinia/api/cli/pyproject.toml +++ b/turbinia/api/cli/pyproject.toml @@ -18,6 +18,7 @@ click = "^8.1.7" turbinia-api-lib = "^1.0.3" google-auth-oauthlib = "^1.1.0" pandas = "^2.1.0" +httpx = "^0.27.0" [build-system] requires = ["poetry-core"]