Skip to content

Commit

Permalink
Turbinia codelab 101 updates (#1523)
Browse files Browse the repository at this point in the history 
* typo

* remove $ in front of command for easy copy-pasting

* remove comment

* up skaffold deadline to 90 seconds to give redis enough time to stabelize

* clearer sentence

* add httpx to cli dependencies

* use turbinia-client upload command

* upload command output

* typos and links

* more verbose explanation of debugger

* text test updates

* updated test code
  • Loading branch information
@hacktobeer
hacktobeer authored Aug 2, 2024
1 parent b88a5dd commit bd51a8f
Show file tree
Hide file tree
Showing 4 changed files with 45 additions and 45 deletions.
54 changes: 30 additions & 24 deletions docs/developer/develop-minikube.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ NOTE: This setup has been tested by the Turbinia developers in the following con
* If you want to develop with a full VSCode web interface you can use the [GCP Cloud Shell Editor](https://shell.cloud.google.com/)
* This setup has all the dependecies and VSCode extensions neeeded for Turbinia development pre-installed and configured. You can continue the setup at the section [here](#start-minkube-cluster).
* Take note of the limitations [here](https://cloud.google.com/shell/docs/quotas-limits)
* If the limitations are an issue for you, have a look into at the (paid) version called [Google Cloud Workstation](https://cloud.google.com/workstations/).
* If the limitations are an issue for you, have a look into the (paid) version called [Google Cloud Workstation](https://cloud.google.com/workstations/).

## Components
### Minikube
Expand Down Expand Up @@ -64,35 +64,35 @@ We will now start the minikube k8s cluster

NOTE: If you want to change the default cluster CPU and Memory usage, you can set those before starting the cluster

$ minikube config set cpus 4
$ minikube config set memory 16384
minikube config set cpus 4
minikube config set memory 16384

### Turbinia source and deployment code
Now we have VSCode setup we are going to get a copy of the Turbinia source and deployment code.
Clone the [Turbinia repository](https://github.com/google/turbinia) by forking the Turbinia repository into your own Github account and clone it locally from there.

$ git clone ssh://git@github.com:[YOUR_GITHUB_ACCOUNT]/turbinia.git
git clone ssh://git@github.com:[YOUR_GITHUB_ACCOUNT]/turbinia.git

Let's get the helm charts for the Turbinia deployment. In your cloned turbinia repository

$ mkdir charts && cd charts
$ helm repo add osdfir-charts https://google.github.io/osdfir-infrastructure
$ helm pull osdfir-charts/turbinia --untar && cd ..
mkdir charts && cd charts
helm repo add osdfir-charts https://google.github.io/osdfir-infrastructure
helm pull osdfir-charts/turbinia --untar && cd ..

Open up the turbinia folder in VSCode.

### Prepare Cluster
Open a terminal (inside VSCode is the easiest, but any terminal will do) and let's configure skaffold, the local cluster and the additional helm repository for Redis.

$ skaffold config set --global local-cluster true
$ eval $(minikube -p minikube docker-env)
$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm repo add kube-prometheus-stack https://prometheus-community.github.io/helm-charts
skaffold config set --global local-cluster true
eval $(minikube -p minikube docker-env)
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add kube-prometheus-stack https://prometheus-community.github.io/helm-charts

### Verify Setup
Execute a build with skaffold (from the root of the cloned Turbinia Github repository)

$ skaffold build
skaffold build

This will build a Turbinia Server container image succesfully if skaffold has been correctlty setup and configured as described above.

Expand All @@ -101,11 +101,11 @@ This will build a Turbinia Server container image succesfully if skaffold has be
### Install the Turbinia Client
We will install the Turbinia client into a Python virtual environment to be able to control Turbinia during our development workflow.

$ python -m venv .venv (or use your favorite virtual env manager)
$ source .venv/bin/activate
$ pip install poetry
$ cd turbinia/api/cli
$ poetry install
python -m venv .venv
source .venv/bin/activate
pip install poetry
cd turbinia/api/cli
poetry install

Create the Turbinia Client configuration file in `$HOME/.turbinia_api_config.json` using the base configuration from [here](
https://pypi.org/project/turbinia-client/).
Expand All @@ -115,7 +115,7 @@ https://pypi.org/project/turbinia-client/).
### Run
Now we are ready to run the development cluster of Turbinia. This will startup the Turbinia API server, the worker and the server in the local minikube k8s cluster.

$ skaffold dev
skaffold dev

NOTE: if one of the services fails to deploy, try again. Sometimes a time-out occurs due to Redis starting too slow.

Expand All @@ -142,25 +142,31 @@ NOTE: As python hot reloading of code into an already running process is tricky
Keep in mind that hot-reloading:
* will load a changed Python source code file as Python byte code in memory of a running process
* will not re-execute any change code automatically
* means that you need to execute something to trigger the code path and code you changed and was hot reloaded
* means that you need to execute something to trigger the code path and code you changed

## Test Run
Let's test the whole setup by executing a request with a disk image located at `test_data/artifact_disk.dd`.

Copy the disk to one of the containers in the shared `/mnt/turbiniavolume` folder.
Upload the disk to the shared `/mnt/turbiniavolume` folder using the `evidence upload` command.

$ kubectl cp artifact_disk.dd dev-release-turbinia-server-6d6:/mnt/turbiniavolume/
turbinia-client evidence upload -p test_data/artifact_disk.dd 12345

# /home/user/turbinia/test_data/artifact_disk.dd:
* Original Name: artifact_disk.dd
* File Name: artifact_disk_2024-08-01T06:46:51.610412Z.dd
* File Path: /mnt/turbiniavolume/upload/12345/artifact_disk_2024-08-01T06:46:51.610412Z.dd
* Size: 20971520

Start a Turbinia rawdisk request.

$ turbinia-client submit rawdisk --source_path /mnt/turbiniavolume/artifact_disk.dd
turbinia-client submit rawdisk --source_path /mnt/turbiniavolume/upload/12345/artifact_disk_2024-08-01T06:46:51.610412Z.dd

Sending request: {'evidence': {'type': 'RawDisk', 'source_path': '/mnt/turbiniavolume/artifact_disk.dd'}, 'request_options': {}}
Sending request: {'evidence': {'type': 'RawDisk', 'source_path': '/mnt/turbiniavolume/upload/12345/artifact_disk_2024-08-01T06:46:51.610412Z.dd'}, 'request_options': {}}
Received response: {'request_id': '4d76df84849c484a835d37fbc7668122'}

You can check the Turbinia WebUI at http://localhost:8000 or use the turbinia-client to verify the status of the request.

$ turbinia-client status request 4d76df84849c484a835d37fbc7668122
turbinia-client status request 4d76df84849c484a835d37fbc7668122


### Next
Expand Down
33 changes: 13 additions & 20 deletions docs/developer/turbinia-codelab-analyser.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,15 +44,11 @@ $ minikube start
We will add our new analyser name to the template configuration file. Add the following to `turbinia/config/turbinia_config_tmpl.py`

```
...
}, {
'job': 'OSInfoAnalysisJob',
'programs': ['grep'],
'docker_image': None,
'timeout': 3600
},
...
```

We will import our new job into the job management init code. Add the following to `​​turbinia/jobs/__init__.py`
Expand All @@ -61,23 +57,16 @@ We will import our new job into the job management init code. Add the following
from turbinia.jobs import os_info
```

We will add our new task name and import that task in the task management code. Add the task name to `turbinia/taskutils.py`
We will add our new task name and import that task in the task management code. Add the task name to `TASK_LIST` in turbinia/taskutils.py`

```
TASK_LIST = [
...
'OSInfoAnalysisTask'
...
]
'OSInfoAnalysisTask',
```

and in the same file in the function `gettask()` add

```
# Late imports to minimize what loads all Tasks
...
from turbinia.workers.analysis.os_info import OSInfoAnalysisTask
...
```

Create the boilerplate code for the job. Create a new file at `turbinia/jobs/os_info.py`
Expand Down Expand Up @@ -265,6 +254,7 @@ Your Turbinia stack is now deployed with your new Job and Task. The stack can be

```
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
dev-release-redis-master-0 1/1 Running 0 45s
dev-release-turbinia-api-7dfd8988b8-bprxd 1/1 Running 0 45s
Expand Down Expand Up @@ -296,14 +286,11 @@ $ turbinia-client config download > charts/turbinia/turbinia.conf
Add the new `OSInfoAnalyserJob` configuration to that file as well.

```
...
}, {
'job': 'OSInfoAnalysisJob',
'programs': ['grep'],
'docker_image': None,
'timeout': 3600
},
...
```

## Run, debug and fix analyser
Expand All @@ -314,6 +301,7 @@ Now upload this evidence file to Turbinia giving a fake ticket ID `12345`.

```
$ turbinia-client evidence upload -p osrelease.tgz 12345
# /home/user/osrelease.tgz:
* Original Name: osrelease.tgz
* File Name: osrelease_2024-07-24T09:36:58.045255Z.tgz
Expand All @@ -337,6 +325,7 @@ We can check the status of the request in the Turbinia WebUI or via the cli.

```
$ turbinia-client status request 42d715d298c24e22ba1f4e88d25d4a44
## Request ID: 42d715d298c24e22ba1f4e88d25d4a44
* Last Update: 2024-07-24T10:22:10.792302
* Requester: user_unspecified
Expand Down Expand Up @@ -364,7 +353,7 @@ Error opening OSRelease file: [Errno 2] No such file or directory: '/tmp/42d715d

We seem to have incorrectly constructed the path to the `/etc/os-release` file.

Let's set two breakpoints (why only do 1, when you can have 2 ;) ) in our code to see what is going on and attach to the running worker.
Let's set two breakpoints (why only do 1, when you can have 2 ;) ) in our code to see what is going on and attach the debugger to the running worker using the VSCode debug panel.

![Attach Debugger](../images/codelab-debug.png)

Expand All @@ -378,6 +367,8 @@ We made a typo in `osrelease_path`, it should be `'etc/os-release'`. Fixing and

Now let's submit a new request again.

NOTE: make sure to exit past the breakpoints in the debugger to enable the worker to continue before submitting the new request.

```
$ turbinia-client submit compresseddirectory --jobs_allowlist OSInfoAnalysisJob --source_path /mnt/turbiniavolume/12345/osrelease_2024-07-24T09:36:58.045255Z.tgz
Sending request: {'evidence': {'type': 'CompressedDirectory', 'source_path': '/mnt/turbiniavolume/12345/osrelease_2024-07-24T09:36:58.045255Z.tgz'}, 'request_options': {'jobs_allowlist': ['OSInfoAnalysisJob']}}
Expand Down Expand Up @@ -427,7 +418,7 @@ class OSInfoAnalysisTaskTest(unittest.TestCase):
@mock.patch(
"builtins.open", new=mock.mock_open(read_data=_OS_RELEASE_CONTENT),
create=True)
@mock.patch('os.path.join', return_value='/etc/os-release')
@mock.patch('os.path.join', return_value='etc/os-release')
def test_run(self, os_join_mock):
"""Test OSInfoAnalysisTask task run."""
task = os_info.OSInfoAnalysisTask()
Expand All @@ -443,9 +434,11 @@ if __name__ == '__main__':
unittest.main()
```

Now let's run this specific test from the root turbinia folder.
Now let's install the test requirements and run this specific test from the root turbinia folder.
```
$ pip install -f requirements-test.txt
$ PYTHONPATH=. python -m unittest turbinia.workers.analysis.os_info_test
Using fallback source config Copy turbinia/config/turbinia_config_tmpl.py to ~/.turbiniarc or /etc/turbinia/turbinia.conf, edit, and re-run.
....
----------------------------------------------------------------------
Expand All @@ -456,6 +449,6 @@ OK

## Tips & Tricks

### Unexplainable reloads
### Unexplainable reloads or failed hot-reloads of code

Restart deployment with `skaffold dev` (ctrl-c the running instance)
2 changes: 1 addition & 1 deletion skaffold.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ build:
infer:
- turbinia/**
deploy:
statusCheckDeadlineSeconds: 30
statusCheckDeadlineSeconds: 90
helm:
releases:
- name: dev-release
Expand Down
1 change: 1 addition & 0 deletions turbinia/api/cli/pyproject.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ click = "^8.1.7"
turbinia-api-lib = "^1.0.3"
google-auth-oauthlib = "^1.1.0"
pandas = "^2.1.0"
httpx = "^0.27.0"

[build-system]
requires = ["poetry-core"]
Expand Down

0 comments on commit bd51a8f

Please sign in to comment.