Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resync #20

Merged
merged 60 commits into from
May 20, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
b7e1cb0
Create starlight.yml
lmangani Apr 13, 2023
f8ff67f
Update README.md
lmangani Apr 13, 2023
b5b152f
Update README.md
lmangani Apr 13, 2023
bbf2ee8
Update README.md
lmangani Apr 13, 2023
d30ce59
query presets
lmangani Apr 13, 2023
f663dcc
Merge pull request #11 from metrico/query-presets
lmangani Apr 13, 2023
73e88ac
Update README.md
lmangani Apr 14, 2023
df4e133
Add examples
lmangani Apr 14, 2023
aeb9113
Update play.html
lmangani Apr 14, 2023
185d511
Merge pull request #12 from metrico/examples
lmangani Apr 14, 2023
229d563
Update README.md
lmangani Apr 14, 2023
ae86443
Update README.md
lmangani Apr 14, 2023
3eb2980
Fix GET parameter handling
lmangani Apr 14, 2023
49636bd
CH Aliases
lmangani Apr 18, 2023
4075f91
Update aliases.sql
lmangani Apr 18, 2023
e9b9a93
Update README.md
lmangani May 5, 2023
0f25f3d
Update README.md
lmangani May 5, 2023
e528e50
Update release.yml
lmangani Jun 2, 2023
e293ff3
Update release.yml
lmangani Jun 2, 2023
679617c
Update Dockerfile
lmangani Jun 30, 2023
fdd0271
Update Dockerfile
lmangani Jun 30, 2023
ed438c9
ClickHouse UDF
lmangani Jul 2, 2023
3e20e0d
Update README.md
lmangani Jul 2, 2023
f2c7e17
Update README.md
lmangani Jul 2, 2023
a316f2c
Update README.md
lmangani Jul 2, 2023
22cf3d0
Update README.md
lmangani Jul 2, 2023
82f4309
Update README.md
lmangani Jul 2, 2023
6876b81
Update README.md
lmangani Jul 2, 2023
229ec97
Update README.md
lmangani Jul 2, 2023
e74c525
Update release.yml
lmangani Jul 8, 2023
ef2bccb
Update release.yml
lmangani Jul 8, 2023
a411393
update to latest duckdb
lmangani Aug 5, 2023
f115405
Update README.md
lmangani Sep 26, 2023
8736805
DuckDB 0.9.1
lmangani Oct 17, 2023
e82dd68
Update go.mod
lmangani Oct 17, 2023
8738346
Merge pull request #14 from metrico/duckdb_091
lmangani Oct 17, 2023
cfd4c49
Update README.md
mamonu Oct 18, 2023
1c30a3b
Merge pull request #15 from mamonu/patch-1
lmangani Oct 18, 2023
3fc5b95
Update README.md
lmangani Oct 18, 2023
73c626b
Update README.md
lmangani Oct 19, 2023
6d5d0a2
Preload ClickHouse Aliases
lmangani Oct 19, 2023
bcc76d4
compact aliases.sql
lmangani Oct 19, 2023
aad550f
Merge pull request #16 from metrico/preload-aliases
lmangani Oct 19, 2023
93814eb
Update README.md
lmangani Oct 21, 2023
f1ccf7a
Update README.md
lmangani Dec 8, 2023
84721df
Update release.yml
lmangani Mar 27, 2024
55b19e2
Update release.yml
lmangani Mar 27, 2024
9c462b3
Update release.yml
lmangani Mar 27, 2024
dcc5ac5
Update release.yml
lmangani Mar 27, 2024
af27bda
Update release.yml
lmangani Mar 27, 2024
66afabd
go-duckdb v1.6.2
lmangani Mar 27, 2024
39c244f
Update README.md
lmangani Mar 28, 2024
7ff9f5f
Update README.md
lmangani Apr 11, 2024
936b055
Update README.md
lmangani Apr 11, 2024
7920b51
Create dependabot.yml
lmangani Apr 23, 2024
3df1ed6
Bump github.com/marcboeker/go-duckdb from 1.6.2 to 1.6.3
dependabot[bot] Apr 23, 2024
93c38b8
Merge pull request #17 from metrico/dependabot/go_modules/github.com/…
lmangani Apr 23, 2024
c54525c
Update go.mod
lmangani May 20, 2024
465da1f
resync
lmangani May 20, 2024
0677c55
Merge branch 'storage-hash' into main
lmangani May 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for all configuration options:
# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file

version: 2
updates:
- package-ecosystem: "gomod" # See documentation for possible values
directory: "/" # Location of package manifests
schedule:
interval: "weekly"
11 changes: 6 additions & 5 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
goarch: [amd64]

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4.1.1
- uses: lmangani/go-release-action@v1.37-ubuntu
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
Expand All @@ -30,23 +30,24 @@ jobs:
executable_compression: upx
compress_assets: OFF
build_flags: -buildvcs=false
# ldflags: "-linkmode external -extldflags -static"
#ldflags: "-linkmode external -extldflags -static"
extra_files: LICENSE README.md

- name: Log in to the Container registry
uses: docker/login-action@v2.1.0
uses: docker/login-action@v3.1.0
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v4.3.0
uses: docker/metadata-action@v5.5.1
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

- name: Build and push Docker image
uses: docker/build-push-action@v4.0.0
uses: docker/build-push-action@v5.3.0
with:
context: .
push: true
Expand Down
16 changes: 16 additions & 0 deletions .github/workflows/starlight.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: Starring Partner
on:
issues:
types: [opened, reopened]
jobs:
# This workflow checks if a user has starred a repository and takes actions
starcheck:
runs-on: ubuntu-latest
steps:
- name: Please Star First
uses: qxip/please-star-light@v4
with:
token: ${{ secrets.GITHUB_TOKEN }}
message: "Thanks for opening an Issue! Please star this repository to motivate developers! :star:"
label: "stargazed"
autoclose: false
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ COPY . .
RUN CGO_ENABLED=1 go build -o quackpipe quackpipe.go
RUN strip quackpipe

FROM ubuntu:20.04
FROM debian:12
COPY --from=builder /quackpipe /quackpipe
RUN echo "INSTALL httpfs; INSTALL json; INSTALL parquet; INSTALL fts;" | /quackpipe --stdin
CMD ["/quackpipe"]
54 changes: 43 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,22 @@
<a href="https://quackpipe.fly.dev" target="_blank"><img src="https://user-images.githubusercontent.com/1423657/231310060-aae46ee6-c748-44c9-905e-20a4eba0a814.png" width=220 /></a>

> _quack, motherducker!_
> _a pipe for quackheads_

# :baby_chick: quackpipe

_QuackPipe is an OLAP API built on top of DuckDB with a few extra compatibility bits. If you know, you know._
_QuackPipe is a serverless OLAP API built on top of DuckDB emulating and aliasing the ClickHouse HTTP API_

Play with DuckDB SQL and Cloud storage though a familiar API, without giving up old habits and integrations.

### Demos
:hatched_chick: try a [sample s3/parquet query](https://quackpipe.metrico.in/?user=default#U0VMRUNUCiAgICB0b3duLAogICAgZGlzdHJpY3QsCiAgICBjb3VudCgpIEFTIGMsCkZST00gcmVhZF9wYXJxdWV0KCdodHRwczovL2RhdGFzZXRzLWRvY3VtZW50YXRpb24uczMuZXUtd2VzdC0zLmFtYXpvbmF3cy5jb20vaG91c2VfcGFycXVldC9ob3VzZV8wLnBhcnF1ZXQnKQpXSEVSRSByZWFkX3BhcnF1ZXQudG93biA9PSAnTE9ORE9OJwpHUk9VUCBCWQogICAgdG93biwKICAgIGRpc3RyaWN0Ck9SREVSIEJZIGMgREVTQwpMSU1JVCAxMA==) _(deta.space free tier, AWS Lambdas)_<br>
:hatched_chick: try our [miniature playground](https://quackpipe.fly.dev) _(fly.io free tier, 1x-shared-vcpu, 256Mb)_
### :hatched_chick: Demos
:hatched_chick: try a [sample s3/parquet query](https://quackpipe.fly.dev/?user=default#U0VMRUNUCiAgICB0b3duLAogICAgZGlzdHJpY3QsCiAgICBjb3VudCgpIEFTIGMsCkZST00gcmVhZF9wYXJxdWV0KCdodHRwczovL2RhdGFzZXRzLWRvY3VtZW50YXRpb24uczMuZXUtd2VzdC0zLmFtYXpvbmF3cy5jb20vaG91c2VfcGFycXVldC9ob3VzZV8wLnBhcnF1ZXQnKQpXSEVSRSByZWFkX3BhcnF1ZXQudG93biA9PSAnTE9ORE9OJwpHUk9VUCBCWQogICAgdG93biwKICAgIGRpc3RyaWN0Ck9SREVSIEJZIGMgREVTQwpMSU1JVCAxMA==) in our [miniature playground](https://quackpipe.fly.dev) _(fly.io free tier, 1x-shared-vcpu, 256Mb)_ <br>
:hatched_chick: launch your own _free instance_ on fly.io

<a href="https://flyctl.sh/shell?repo=metrico/quackpipe" target="_blank">
<img src="https://user-images.githubusercontent.com/1423657/236479471-a1cb0484-dfd3-4dc2-8d62-121debd7faa3.png" width=300>
</a>

<br>

<br>

Expand Down Expand Up @@ -55,7 +60,7 @@ Run with `-h` for a full list of parameters
#### :point_right: Playground
Execute queries using the embedded playground

![image](https://user-images.githubusercontent.com/1423657/230783859-1c69910b-6bf2-42df-8b1d-876b94fc3419.png)
<a href="https://quackpipe.fly.dev" target=_blank><img src="https://github.com/metrico/quackpipe/assets/1423657/fa0c8b8f-7480-4bd1-b8b2-bee24ee39186" width=800></a>

#### :point_right: API
Execute queries using the POST API
Expand All @@ -73,8 +78,35 @@ hello,v0.7.1
```

### :fist_right: Extensions
Several extensions are pre-installed by default in Docker images, including _parquet, json, httpfs_<br>
When using HTTP API, _httpfs, parquet, json_ extensions are automatically pre-loaded.
Several extensions are pre-installed by default in [Docker images](https://github.com/metrico/quackpipe/blob/main/Dockerfile#L9), including _parquet, json, httpfs_<br>
When using HTTP API, _httpfs, parquet, json_ extensions are automatically pre-loaded by the wrapper.

Users can pre-install extensions and execute quackpipe using a custom parameters:
```
echo "INSTALL httpfs;" | ./quackpipe --stdin --params "?extension_directory=/tmp/"
./quackpipe --port 8123 --host 0.0.0.0 --params "?extension_directory=/tmp/"
```


### <img src="https://github.com/metrico/quackpipe/assets/1423657/f66fd8f8-a756-40a6-bee9-7979b09f2576" height=20 > ClickHouse UDF

Quackpipe can be used as [executable UDF](https://clickhouse.com/docs/en/engines/table-functions/executable) to get DuckDB data IN/OUT of ClickHouse queries:

```sql
SELECT *
FROM executable('quackpipe -stdin -format TSV', TSV, 'id UInt32, num UInt32', (
SELECT 'SELECT 1, 2'
))
Query id: dd878948-bec8-4abe-9e06-2f5813653c3a
┌─id─┬─num─┐
│ 1 │ 2 │
└────┴─────┘
1 rows in set. Elapsed: 0.155 sec.
```

🃏 What is this? Think of it as a SELECT within a SELECT with a different syntax.<br>
🃏 Format confusion? Make DuckDB SQL feel like ClickHouse with the included [ClickHouse Macro Aliases](https://github.com/metrico/quackpipe/blob/main/aliases.sql)


<br>

Expand All @@ -85,14 +117,15 @@ When using HTTP API, _httpfs, parquet, json_ extensions are automatically pre-lo
- [x] [cgo](https://github.com/marcboeker/go-duckdb) binding
- [x] Extension preloading
- [ ] Aliases Extension
- [x] REST API [^3] [^4]
- [x] REST API [^3]
- [x] CH FORMAT Emulation
- [x] CSV, CSVWithNames
- [x] TSV, TSVWithNames
- [x] JSONCompact
- [ ] Native
- [x] Web Playground _(from ClickkHouse, Apache2 Licensed)_ [^2]
- [x] STDIN Fast Query Execution
- [x] ClickHouse Executable UDF
- [x] `:memory:` mode Cloud Storage _(s3/r2/minio, httpfs, etc)_
- [x] `:file:` mode using optional _parameters_

Expand All @@ -111,7 +144,6 @@ When using HTTP API, _httpfs, parquet, json_ extensions are automatically pre-lo

###### :black_joker: Disclaimers

[^1]: DuckDB ® is a trademark of MotherDuck. No direct affiliation or endorsement.
[^1]: DuckDB ® is a trademark of DuckDB Foundation. All rights reserved by their respective owners.
[^2]: ClickHouse ® is a trademark of ClickHouse Inc. No direct affiliation or endorsement.
[^3]: Released under the MIT license. See LICENSE for details. All rights reserved by their respective owners.
[^4]: Elements of this experiments (including potential bugs) were co-authored by ChatGPT.
36 changes: 36 additions & 0 deletions aliases.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
CREATE OR REPLACE MACRO toString(expr) AS CAST(expr AS VARCHAR);
CREATE OR REPLACE MACRO toInt8(expr) AS CAST(expr AS INT8);
CREATE OR REPLACE MACRO toInt16(expr) AS CAST(expr AS INT16);
CREATE OR REPLACE MACRO toInt32(expr) AS CAST(expr AS INT32);
CREATE OR REPLACE MACRO toInt64(expr) AS CAST(expr AS INT64);
CREATE OR REPLACE MACRO toInt128(expr) AS CAST(expr AS INT128);
CREATE OR REPLACE MACRO toInt256(expr) AS CAST(expr AS HUGEINT);
CREATE OR REPLACE MACRO toInt8OrZero(expr) AS CASE WHEN TRY_CAST(expr AS INT8) THEN CAST(expr as INT8) ELSE 0 END;
CREATE OR REPLACE MACRO toInt16OrZero(expr) AS CASE WHEN TRY_CAST(expr AS INT16) THEN CAST(expr as INT16) ELSE 0 END;
CREATE OR REPLACE MACRO toInt32OrZero(expr) AS CASE WHEN TRY_CAST(expr AS INT32) THEN CAST(expr as INT32) ELSE 0 END;
CREATE OR REPLACE MACRO toInt64OrZero(expr) AS CASE WHEN TRY_CAST(expr AS INT64) THEN CAST(expr as INT64) ELSE 0 END;
CREATE OR REPLACE MACRO toInt128OrZero(expr) AS CASE WHEN TRY_CAST(expr AS INT128) THEN CAST(expr as INT128) ELSE 0 END;
CREATE OR REPLACE MACRO toInt256OrZero(expr) AS CASE WHEN TRY_CAST(expr AS HUGEINT) THEN CAST(expr as HUGEINT) ELSE 0 END;
CREATE OR REPLACE MACRO toInt8OrNull(expr) AS TRY_CAST(expr AS INT8);
CREATE OR REPLACE MACRO toInt16OrNull(expr) AS TRY_CAST(expr AS INT16);
CREATE OR REPLACE MACRO toInt32OrNull(expr) AS TRY_CAST(expr AS INT32);
CREATE OR REPLACE MACRO toInt64OrNull(expr) AS TRY_CAST(expr AS INT64);
CREATE OR REPLACE MACRO toInt128OrNull(expr) AS TRY_CAST(expr AS INT128);
CREATE OR REPLACE MACRO toInt256OrNull(expr) AS TRY_CAST(expr AS HUGEINT);
CREATE OR REPLACE MACRO toUInt8(expr) AS CAST(expr AS UTINYINT);
CREATE OR REPLACE MACRO toUInt16(expr) AS CAST(expr AS USMALLINT);
CREATE OR REPLACE MACRO toUInt32(expr) AS CAST(expr AS UINTEGER);
CREATE OR REPLACE MACRO toUInt64(expr) AS CAST(expr AS UBIGINT);
CREATE OR REPLACE MACRO toUInt8rZero(expr) AS CASE WHEN TRY_CAST(expr AS UTINYINT) THEN CAST(expr as UTINYINT) ELSE 0 END;
CREATE OR REPLACE MACRO toUInt16rZero(expr) AS CASE WHEN TRY_CAST(expr AS USMALLINT) THEN CAST(expr as USMALLINT) ELSE 0 END;
CREATE OR REPLACE MACRO toUInt32rZero(expr) AS CASE WHEN TRY_CAST(expr AS UINTEGER) THEN CAST(expr as UINTEGER) ELSE 0 END;
CREATE OR REPLACE MACRO toUInt64rZero(expr) AS CASE WHEN TRY_CAST(expr AS UBIGINT) THEN CAST(expr as UBIGINT) ELSE 0 END;
CREATE OR REPLACE MACRO toUInt8rNull(expr) AS TRY_CAST(expr AS UTINYINT);
CREATE OR REPLACE MACRO toUInt16rNull(expr) AS TRY_CAST(expr AS USMALLINT);
CREATE OR REPLACE MACRO toUInt32rNull(expr) AS TRY_CAST(expr AS UINTEGER);
CREATE OR REPLACE MACRO toUInt64rNull(expr) AS TRY_CAST(expr AS UBIGINT);
CREATE OR REPLACE MACRO toFloat(expr) AS CAST(expr AS DOUBLE);
CREATE OR REPLACE MACRO toFloatOrNull(expr) AS TRY_CAST(expr AS DOUBLE);
CREATE OR REPLACE MACRO toFloatOrZero(expr) AS CASE WHEN TRY_CAST(expr AS DOUBLE) THEN CAST(expr as DOUBLE) ELSE 0 END;
CREATE OR REPLACE MACRO intDiv(a, b) AS (a / b);
CREATE OR REPLACE MACRO match(string,token) AS string LIKE token;
19 changes: 16 additions & 3 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,7 +1,20 @@
module quackpipe

go 1.18
go 1.20

require github.com/marcboeker/go-duckdb v1.2.2
require github.com/marcboeker/go-duckdb v1.6.4

require github.com/mitchellh/mapstructure v1.5.0 // indirect
require (
github.com/apache/arrow/go/v14 v14.0.2 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/google/flatbuffers v23.5.26+incompatible // indirect
github.com/klauspost/compress v1.16.7 // indirect
github.com/klauspost/cpuid/v2 v2.2.5 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/pierrec/lz4/v4 v4.1.18 // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
golang.org/x/mod v0.13.0 // indirect
golang.org/x/sys v0.13.0 // indirect
golang.org/x/tools v0.14.0 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
)
35 changes: 31 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
@@ -1,9 +1,36 @@
github.com/apache/arrow/go/v14 v14.0.2 h1:N8OkaJEOfI3mEZt07BIkvo4sC6XDbL+48MBPWO5IONw=
github.com/apache/arrow/go/v14 v14.0.2/go.mod h1:u3fgh3EdgN/YQ8cVQRguVW3R+seMybFg8QBQ5LU+eBY=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/marcboeker/go-duckdb v1.2.2 h1:Qy5yW83qAcZgsEmGo+pkEZeZvxA2dzuQNPLh7wcorb0=
github.com/marcboeker/go-duckdb v1.2.2/go.mod h1:wm91jO2GNKa6iO9NTcjXIRsW+/ykPoJbQcHSXhdAl28=
github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
github.com/google/flatbuffers v23.5.26+incompatible h1:M9dgRyhJemaM4Sw8+66GHBu8ioaQmyPLg1b8VwK5WJg=
github.com/google/flatbuffers v23.5.26+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/uuid v1.3.1 h1:KjJaJ9iWZ3jOFZIf1Lqf4laDRCasjl0BCmnEGxkdLb4=
github.com/klauspost/compress v1.16.7 h1:2mk3MPGNzKyxErAw8YaohYh69+pa4sIQSC0fPGCFR9I=
github.com/klauspost/compress v1.16.7/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
github.com/klauspost/cpuid/v2 v2.2.5 h1:0E5MSMDEoAulmXNFquVs//DdoomxaoTY1kUhbc/qbZg=
github.com/klauspost/cpuid/v2 v2.2.5/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws=
github.com/marcboeker/go-duckdb v1.6.4 h1:p7iFopIcIWoHZStQhvQ+ffhKL3ExM/oXdXAUI62gBWE=
github.com/marcboeker/go-duckdb v1.6.4/go.mod h1:WtWeqqhZoTke/Nbd7V9lnBx7I2/A/q0SAq/urGzPCMs=
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/pierrec/lz4/v4 v4.1.18 h1:xaKrnTkyoqfh1YItXl56+6KJNVYWlEEPuAQW9xsplYQ=
github.com/pierrec/lz4/v4 v4.1.18/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
github.com/zeebo/xxh3 v1.0.2 h1:xZmwmqxHZA8AI603jOQ0tMqmBr9lPeFwGg6d+xy9DC0=
github.com/zeebo/xxh3 v1.0.2/go.mod h1:5NWz9Sef7zIDm2JHfFlcQvNekmcEl9ekUZQQKCYaDcA=
golang.org/x/exp v0.0.0-20231006140011-7918f672742d h1:jtJma62tbqLibJ5sFQz8bKtEM8rJBtfilJ2qTU199MI=
golang.org/x/mod v0.13.0 h1:I/DsJXRlw/8l/0c24sM9yb0T4z9liZTduXvdAWYiysY=
golang.org/x/mod v0.13.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/sync v0.4.0 h1:zxkM55ReGkDlKSM+Fu41A+zmbZuaPVbGMzvvdUPznYQ=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE=
golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/tools v0.14.0 h1:jvNa2pY0M4r62jkRQ6RwEZZyPcymeL9XZMLBbV7U2nc=
golang.org/x/tools v0.14.0/go.mod h1:uYBEerGOWcJyEORxN+Ek8+TT266gXkNlHdJBwexUsBg=
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 h1:H2TDz8ibqkAF6YGhCdN3jS9O0/s90v0rJh3X/OLHEUk=
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8=
gonum.org/v1/gonum v0.12.0 h1:xKuo6hzt+gMav00meVPUlXwSdoEJP46BR+wdxQEFK2o=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
18 changes: 18 additions & 0 deletions play.html
Original file line number Diff line number Diff line change
Expand Up @@ -458,6 +458,16 @@
<span class="hint">&nbsp;(Ctrl/Cmd+Enter)</span>
<span id="hourglass">⧗</span>
<span id="check-mark">✔</span>
<select class="shadow" id="dropdown" style="margin-left: 5px;">
<!-- newline: &#13;&#10; single quote: &#39; -->
<option value=""></option>
<option value='SELECT version();'>Version</option>
<option value='SELECT * &#13;&#10;FROM read_csv_auto("https://s3.us-east-1.amazonaws.com/altinity-clickhouse-data/airline/data/airports/Airports.csv")&#13;&#10;WHERE Country == &#39;Italy&#39;&#13;&#10;ORDER BY City ASC'>Scan S3/CSV</option>
<option value='SELECT town,district,count() AS c,&#13;&#10;FROM read_parquet("https://datasets-documentation.s3.eu-west-3.amazonaws.com/house_parquet/house_0.parquet")&#13;&#10;WHERE read_parquet.town == &#39;LONDON&#39;&#13;&#10;GROUP BY town,district&#13;&#10;ORDER BY c DESC&#13;&#10;LIMIT 10'>Scan/Group S3/Parquet</option>
<option value='SELECT count(*) FROM "https://shell.duckdb.org/data/tpch/0_01/parquet/lineitem.parquet";'>Count HTTP/Parquet</option>
<option value='SELECT avg(c_acctbal) FROM "https://shell.duckdb.org/data/tpch/0_01/parquet/customer.parquet";'>Avg HTTP/Parquet</option>
<option value='SELECT n_name, count(*)&#13;&#10; FROM "https://shell.duckdb.org/data/tpch/0_01/parquet/customer.parquet",&#13;&#10; "https://shell.duckdb.org/data/tpch/0_01/parquet/nation.parquet"&#13;&#10; WHERE c_nationkey = n_nationkey GROUP BY n_name;'>Join HTTP/Parquet</option>
</select>
<span id="stats"></span>
<span id="toggle-dark">🌑</span><span id="toggle-light">🌞</span>
</div>
Expand All @@ -480,6 +490,14 @@
/// This is to avoid race conditions.
let request_num = 0;

// query presets
var querybox = document.getElementById('query');
var querypresets = document.getElementById('dropdown');
querypresets.onchange = function() {
var newquery = querypresets.options[querypresets.selectedIndex].value;
querybox.value = newquery;
}

/// Save query in history only if it is different.
let previous_query = '';

Expand Down
14 changes: 11 additions & 3 deletions quackpipe.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ import (
//go:embed play.html
var staticPlay string

//go:embed aliases.sql
var staticAliases string

// params for Flags
type CommandLineFlags struct {
Host *string `json:"host"`
Expand Down Expand Up @@ -54,7 +57,11 @@ func quack(query string, stdin bool, format string, params string) (string, erro
if !stdin {
check(db.Exec("LOAD httpfs; LOAD json; LOAD parquet;"))
}


if staticAliases != "" {
check(db.Exec(staticAliases))
}

startTime := time.Now()
rows, err := db.Query(query)
if err != nil {
Expand Down Expand Up @@ -297,7 +304,8 @@ func main() {

// handle query parameter
if r.URL.Query().Get("query") != "" {
query = r.Form.Get("query")
// query = r.FormValue("query")
query = r.URL.Query().Get("query")
} else if r.Body != nil {
bodyBytes, err = ioutil.ReadAll(r.Body)
if err != nil {
Expand Down Expand Up @@ -354,7 +362,7 @@ func main() {
}
})

fmt.Printf("API Running: %s:%s\n", *appFlags.Host, *appFlags.Port)
fmt.Printf("QuackPipe API Running: %s:%s\n", *appFlags.Host, *appFlags.Port)
if err := http.ListenAndServe(*appFlags.Host+":"+*appFlags.Port, nil); err != nil {
panic(err)
}
Expand Down