Google BigQuery REST API support progress (v2/minimal)

[github-christophe-oudar]

Hello,

I'm looking forward for that support and it looks like you're actively working on it.
I would like to build a DuckDB foreign data wrapper for BigQuery. I initially intended to have a working POC with BigQuery Storage Read API but to make a proper implementation, I need to list table column details and to be able to do actual query & DML operations would be a huge win (as BQ storage API doesn't allow to read views for instance).

I tried to have a quick look at the project and it's not obvious what's ready or not.
Do you have some insights to share about the state of that v2 and if I could:

• Use a job client to run a query and maybe retrieve the result with the "query results" API?
• Have an option to retrieve the temporary table path (project.dataset.table) from a query so that I can use BQ Storage Read API to read the data?
  -Request table metadata?

If none available, any ETA would be great to see if I should try to hack something by myself.

Anyway, thanks for your work on that SDK! I'm sure it will open great opportunities to combine BigQuery with other data platform programmed in C++!

[dbolduc]

Hey,

I would like to build a DuckDB foreign data wrapper for BigQuery.

Neat.

the state of that v2

You are right that there is no available C++ library for the BigQuery REST v2 API. e.g. there is no way to perform a query.

any ETA would be great

Exposing the BigQuery REST API is a top priority for us. #14061 is the tracking bug for this work.

While I can't commit to these timelines, it should be available in the 2024-08 release. There is a non-zero chance it makes it into the 2024-07 release. So somewhere between 2 weeks and 2 months from now.

it looks like you're actively working on it.

For the record, you should avoid this code. It is not public, nor do we plan to make it public. The API we plan to expose will be different.

[github-christophe-oudar]

Great! I'm looking forward for a proper API 🙌
Since I'm fairly slow at C++ programming and it's a fun side project, it's a great timing so that I can work on preparing the other required parts (i.e. actual queries to describe tables or results and type mapping).

I'll keep you posted if you're interested with the result once the MVP is ready.

[dbolduc]

@github-christophe-oudar FYI: the BigQuery REST v2 API is available in the v2.27.0 release:

https://github.com/googleapis/google-cloud-cpp/tree/v2.27.0/google/cloud/bigquerycontrol

Note that the library is initially experimental, and that it may take some extra time to propagate through the ecosystem (e.g. to be available via vcpkg).