Skip to content

Commit

Permalink
List buckets pages (#3)
Browse files Browse the repository at this point in the history
* remove leftovers

* script to export the execution of a spec as a page

* improve readme
  • Loading branch information
fczuardi authored Oct 25, 2024
1 parent 31de38e commit 4f10365
Show file tree
Hide file tree
Showing 10 changed files with 149 additions and 96 deletions.
1 change: 1 addition & 0 deletions AUTHORS
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fabricio Campos Zuardi <@fczuardi>
16 changes: 10 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,14 @@ that are "S3-compatible".

## Contribute

This is an open project that and you can contribute with it by suggesting or writing
This is an open project and you can contribute with it by suggesting or writing
new specifications.

### Suggesting Specifications

Just open a new issue on the [issues page](), explaining what feature, scenarios and steps you are
interesting in see described as an executable specification on this collection.
Just open a new issue on the [issues page](https://github.com/marmotitude/s3-specs/issues),
explaining what feature, scenarios and steps you are interesting in see described as an
executable specification on this collection.

### Writing Specifications

Expand All @@ -28,16 +29,19 @@ Follow these steps to submit a contribution:
- write a new .ipynb document, or duplicate an existing one
- run your cells and check that the expected results are ok
- parametrize accordingly, secrets and other arguments should be variables to be filled by **papermill**
- include a link to the new spec on the main page (index.md)
- add your name to the AUTHORS file
- generate new pages with the `./run_spec.sh` script
- include a link to the new pages on the main page (index.md)
- add your name to the [AUTHORS](./AUTHORS) file
- open a pull request

## Use S3-Specs as a testing tool on a terminal

You can run any specification from the `docs` directly using the papermill CLI tool,
You can run specifications from the `docs` directly using the papermill CLI tool,
see [executing a notebook](https://github.com/nteract/papermill?tab=readme-ov-file#executing-a-notebook)
for details, for example:

```
poetry run papermill docs/list-buckets.ipynb /tmp/result.ipynb -y "profile_name: br-se1"
```


75 changes: 0 additions & 75 deletions docs/.ipynb_checkpoints/placeholder-test-checkpoint.ipynb

This file was deleted.

14 changes: 0 additions & 14 deletions docs/.ipynb_checkpoints/placeholder-test-checkpoint.md

This file was deleted.

5 changes: 4 additions & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
@@ -1 +1,4 @@
# There be dragons
# Latest Executed Specifications

- [list-buckets with br-ne1 params](./list-buckets_params_br-ne1.md)
- [list-buckets with br-se1 params](./list-buckets_params_br-se1.md)
55 changes: 55 additions & 0 deletions docs/list-buckets_params_br-ne1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# List buckets
List all buckets from a profile[<sup>1</sup>](./glossary#profile)

## Setup


```python
# Parameters
profile_name = "default"
```


```python
# Parameters
profile_name = "br-ne1"

```


```python
import datetime
print(f'execution started at {datetime.datetime.now()}')
```

execution started at 2024-10-25 10:00:07.887638



```python
# Client instantiation
import boto3
session = boto3.Session(profile_name=profile_name)
s3_client = session.client('s3')
```

## Example


```python
response = s3_client.list_buckets()
buckets = response.get('Buckets')
print(f"Profile '{profile_name}' has {len(buckets)} buckets.")

import random
print(f"One of those buckets is named {random.choice(buckets).get('Name')}")
```

Profile 'br-ne1' has 19 buckets.
One of those buckets is named test-br-ne1-1729485828


## References

- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/list_buckets.html

55 changes: 55 additions & 0 deletions docs/list-buckets_params_br-se1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# List buckets
List all buckets from a profile[<sup>1</sup>](./glossary#profile)

## Setup


```python
# Parameters
profile_name = "default"
```


```python
# Parameters
profile_name = "br-se1"

```


```python
import datetime
print(f'execution started at {datetime.datetime.now()}')
```

execution started at 2024-10-25 10:10:36.457622



```python
# Client instantiation
import boto3
session = boto3.Session(profile_name=profile_name)
s3_client = session.client('s3')
```

## Example


```python
response = s3_client.list_buckets()
buckets = response.get('Buckets')
print(f"Profile '{profile_name}' has {len(buckets)} buckets.")

import random
print(f"One of those buckets is named {random.choice(buckets).get('Name')}")
```

Profile 'br-se1' has 4 buckets.
One of those buckets is named benchmark


## References

- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/list_buckets.html

1 change: 1 addition & 0 deletions params_br-ne1.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
profile_name: "br-ne1"
1 change: 1 addition & 0 deletions params_br-se1.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
profile_name: "br-se1"
22 changes: 22 additions & 0 deletions run-spec.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/bin/bash

# Check if the correct number of arguments are provided
if [ "$#" -ne 3 ]; then
echo "Usage: $0 <notebook_path> <yaml_params_path> <output_format>"
exit 1
fi

NOTEBOOK_PATH=$1
YAML_PARAMS=$2
OUTPUT_FORMAT=$3
EXECUTION_NAME=$(basename "$YAML_PARAMS" .yaml)
PAPERMILL_OUTPUT_FOLDER="/tmp"
OUTPUT_FOLDER="docs"

# Step 1: Run the notebook with Papermill using the YAML file for parameters
EXECUTED_NOTEBOOK="${PAPERMILL_OUTPUT_FOLDER}/$(basename "$NOTEBOOK_PATH" .ipynb)_${EXECUTION_NAME}.ipynb"
papermill $NOTEBOOK_PATH $EXECUTED_NOTEBOOK -y "$(cat $YAML_PARAMS)"

# Step 2: Convert the executed notebook to the specified format
jupyter nbconvert --to $OUTPUT_FORMAT $EXECUTED_NOTEBOOK --output-dir $OUTPUT_FOLDER

0 comments on commit 4f10365

Please sign in to comment.