Skip to content

Commit

Permalink
Unique bucket spec (#4)
Browse files Browse the repository at this point in the history
* add uuid as a dependency

* utility lib with shared helper functions

* improve readme

* improve gitignore

* use shared helper functions on list-buckets spec

* add docs dir on pythonpath for the page generation script

* new specification: unique-bucket

* include new executions on the index
  • Loading branch information
fczuardi authored Oct 25, 2024
1 parent 4f10365 commit fd58864
Show file tree
Hide file tree
Showing 11 changed files with 545 additions and 29 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
docs/.ipynb_checkpoints/
docs/__pycache__/
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ Follow these steps to submit a contribution:
- install the dependencies
- `poetry install`
- launch the interactive environment
- `poetry run jupyter docs`
- `poetry shell`
- `jupyter lab docs/list-buckets.ipynb`
- write a new .ipynb document, or duplicate an existing one
- run your cells and check that the expected results are ok
- parametrize accordingly, secrets and other arguments should be variables to be filled by **papermill**
Expand Down
2 changes: 2 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# Latest Executed Specifications

- [unique-bucket-name with br-ne1 params](./unique-bucket-name_params_br-ne1.md)
- [unique-bucket-name with br-se1 params](./unique-bucket-name_params_br-se1.md)
- [list-buckets with br-ne1 params](./list-buckets_params_br-ne1.md)
- [list-buckets with br-se1 params](./list-buckets_params_br-se1.md)
84 changes: 57 additions & 27 deletions docs/list-buckets.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,13 @@
{
"cell_type": "markdown",
"id": "4766f865-3d49-40ff-a12d-633d687203e6",
"metadata": {},
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"source": [
"# List buckets\n",
"List all buckets from a profile[<sup>1</sup>](./glossary#profile)"
Expand All @@ -26,7 +32,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 1,
"id": "908caaf7-3fe6-42de-a1fb-fafdc7732383",
"metadata": {
"editable": true,
Expand All @@ -45,75 +51,99 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 2,
"id": "0d2cecec-ab01-46b6-a608-4e8f9e297ebb",
"metadata": {},
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"execution started at 2024-10-24 23:27:45.201904\n"
"execution started at 2024-10-25 11:58:51.538200\n"
]
}
],
"source": [
"import datetime\n",
"print(f'execution started at {datetime.datetime.now()}')"
"# Import shared functions\n",
"from s3_helpers import print_timestamp, create_s3_client\n",
"\n",
"print_timestamp()\n",
"\n",
"# Create S3 client\n",
"s3_client = create_s3_client(profile_name)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fe6d5256-8f0f-4f00-988d-6f9845b51a69",
"cell_type": "markdown",
"id": "599ec07a-a806-4ee4-ac7c-07137ad6be74",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [],
"source": [
"# Client instantiation\n",
"import boto3\n",
"session = boto3.Session(profile_name=profile_name)\n",
"s3_client = session.client('s3')"
"## Example"
]
},
{
"cell_type": "markdown",
"id": "599ec07a-a806-4ee4-ac7c-07137ad6be74",
"cell_type": "code",
"execution_count": 3,
"id": "a2857eb2-b842-4bd2-b1cf-6ba609f84a14",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Profile 'default' has 1 buckets.\n"
]
}
],
"source": [
"## Example"
"response = s3_client.list_buckets()\n",
"buckets = response.get('Buckets')\n",
"buckets_count = len(buckets)\n",
"print(f\"Profile '{profile_name}' has {buckets_count} buckets.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7fa5c156-2f2c-41a6-84c0-e4ec6a628ce6",
"execution_count": 4,
"id": "f5a38db0-cd7f-4e7a-8759-7f116fadb274",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"One of those buckets is named foo\n"
]
}
],
"source": [
"response = s3_client.list_buckets()\n",
"buckets = response.get('Buckets')\n",
"print(f\"Profile '{profile_name}' has {len(buckets)} buckets.\")\n",
"\n",
"import random\n",
"print(f\"One of those buckets is named {random.choice(buckets).get('Name')}\")"
"if buckets_count > 0:\n",
" import random\n",
" print(f\"One of those buckets is named {random.choice(buckets).get('Name')}\")"
]
},
{
Expand Down
14 changes: 14 additions & 0 deletions docs/s3_helpers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import boto3
import datetime
import uuid

def print_timestamp():
print(f'execution started at {datetime.datetime.now()}')

def create_s3_client(profile_name):
session = boto3.Session(profile_name=profile_name)
return session.client('s3')

def generate_unique_bucket_name(base_name="my-unique-bucket"):
unique_id = uuid.uuid4().hex[:6] # Short unique suffix
return f"{base_name}-{unique_id}"
Loading

0 comments on commit fd58864

Please sign in to comment.