Skip to content

Commit

Permalink
First boto3 spec (MagaluCloud#2)
Browse files Browse the repository at this point in the history
* disable poetry package mode

* improve README

* add papermill

* add boto3

* add post-install script to create a poetry env for jupyter

* remove placeholder spec

* boto3 list-buckets example

* improve docs
  • Loading branch information
fczuardi authored Oct 25, 2024
1 parent 47365c2 commit 31de38e
Show file tree
Hide file tree
Showing 7 changed files with 913 additions and 104 deletions.
38 changes: 38 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,41 @@
A growing collection of **executable specifications** that can be used to test
**Object Storage** implementations similar to AWS's S3, for example, cloud products
that are "S3-compatible".

## Contribute

This is an open project that and you can contribute with it by suggesting or writing
new specifications.

### Suggesting Specifications

Just open a new issue on the [issues page](), explaining what feature, scenarios and steps you are
interesting in see described as an executable specification on this collection.

### Writing Specifications

Specifications are written in the format of Notebooks, we use **jupyter**, parametrized with
**papermill** and exported to markdown (to become pages) by **nbconvert**.

Follow these steps to submit a contribution:

- install the dependencies
- `poetry install`
- launch the interactive environment
- `poetry run jupyter docs`
- write a new .ipynb document, or duplicate an existing one
- run your cells and check that the expected results are ok
- parametrize accordingly, secrets and other arguments should be variables to be filled by **papermill**
- include a link to the new spec on the main page (index.md)
- add your name to the AUTHORS file
- open a pull request

## Use S3-Specs as a testing tool on a terminal

You can run any specification from the `docs` directly using the papermill CLI tool,
see [executing a notebook](https://github.com/nteract/papermill?tab=readme-ov-file#executing-a-notebook)
for details, for example:

```
poetry run papermill docs/list-buckets.ipynb /tmp/result.ipynb -y "profile_name: br-se1"
```
3 changes: 3 additions & 0 deletions docs/glossary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Glossary

<span id="profile">profile</span>: A profile is a set of configs, such as region and endpoint, and credentials, such as api_key_id and api_secret_key. On aws cli and sdks, the profile have a name and the data lives on user config directories like `~/.aws/config` and `~/.aws/credentials`. Those tools may accept an environment variable `AWS_PROFILE` or an argument `--profile`
152 changes: 152 additions & 0 deletions docs/list-buckets.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "4766f865-3d49-40ff-a12d-633d687203e6",
"metadata": {},
"source": [
"# List buckets\n",
"List all buckets from a profile[<sup>1</sup>](./glossary#profile)"
]
},
{
"cell_type": "markdown",
"id": "bad9760b-0b1b-46d1-b193-c9d5d433fd9d",
"metadata": {
"editable": true,
"jp-MarkdownHeadingCollapsed": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"source": [
"## Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "908caaf7-3fe6-42de-a1fb-fafdc7732383",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": [
"parameters"
]
},
"outputs": [],
"source": [
"# Parameters\n",
"profile_name = \"default\""
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "0d2cecec-ab01-46b6-a608-4e8f9e297ebb",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"execution started at 2024-10-24 23:27:45.201904\n"
]
}
],
"source": [
"import datetime\n",
"print(f'execution started at {datetime.datetime.now()}')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fe6d5256-8f0f-4f00-988d-6f9845b51a69",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [],
"source": [
"# Client instantiation\n",
"import boto3\n",
"session = boto3.Session(profile_name=profile_name)\n",
"s3_client = session.client('s3')"
]
},
{
"cell_type": "markdown",
"id": "599ec07a-a806-4ee4-ac7c-07137ad6be74",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"source": [
"## Example"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7fa5c156-2f2c-41a6-84c0-e4ec6a628ce6",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [],
"source": [
"response = s3_client.list_buckets()\n",
"buckets = response.get('Buckets')\n",
"print(f\"Profile '{profile_name}' has {len(buckets)} buckets.\")\n",
"\n",
"import random\n",
"print(f\"One of those buckets is named {random.choice(buckets).get('Name')}\")"
]
},
{
"cell_type": "markdown",
"id": "edb44a40-de96-472f-b8b8-a6596f7f1aa2",
"metadata": {},
"source": [
"## References\n",
"\n",
"- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/list_buckets.html\n",
" "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python (Poetry)",
"language": "python",
"name": "my-poetry-env"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
75 changes: 0 additions & 75 deletions docs/placeholder-test.ipynb

This file was deleted.

27 changes: 0 additions & 27 deletions docs/placeholder-test.md

This file was deleted.

Loading

0 comments on commit 31de38e

Please sign in to comment.