feat: add substrait.proto convenience module and document it

[amol-]

Add a substrait.proto module that gives access to the Substrait protocol classes
removing the need to navigate the hierarchy automatically generated by protobuf.
It also provides access to the modules without the _pb2 suffix
which is an implementation detail of the protobuf version used.

Provides examples on how to generate and read back Substrait plans
using the substrait-python module itself.

[github-actions]

ACTION NEEDED

Substrait follows the Conventional Commits
specification for
release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[danepitkin]

LGTM! Thanks @amol- this is already much cleaner for the user :)

[danepitkin]

We can go ahead and delete the test_imports() function above since the new test indirectly tests these imports.

[amol-]

I think there is a value in keeping this test to ensure backward compatibility.
substrait.proto might continue to work but import the classes from modules with different names than the current ones.

In such case we would be breaking backward compatibility for anyone directly accessing the classes from the _pb2 modules and that's something we want to catch and be aware of.

[gforsyth]

looks good overall -- two small suggestions for the tests.

[EpsilonPrime]

So the purpose of this PR is to get the protos exposed with the simpler package name without changing the fact that they are registered with the existing one?

One reason the current package name is used is to prevent conflicts when the C++ version of the protos is also loaded under ::substrait::proto. We should probably document this desire along with the fact that we aren't simply changing what package we generate the protos in.

[amol-]

So the purpose of this PR is to get the protos exposed with the simpler package name without changing the fact that they are registered with the existing one?

Correct, I wanted to retain backward compatibility for anyone importing the classes from the autogenerated modules, but also make the automatic generation an implementation detail.

One reason the current package name is used is to prevent conflicts when the C++ version of the protos is also loaded under ::substrait::proto. We should probably document this desire along with the fact that we aren't simply changing what package we generate the protos in.

Can you elaborate? How does the C++ library namespace influence the Python namespace?

[EpsilonPrime]

Depending on your options you can actually use C++ protos for the Python under the hood. But ignoring that you can still end up with two definitions for protobufs which will collide and cause an exception when the module loads. If they are in different namespaces that collision won't happen (you can pretend to have two different sets of protos).

On a side note, I have been having issues with the current protos (lint and IDE) because the attributes aren't available for the package until runtime. The solution I used for the spark connect protos was to use mypy-protobuf (add --mypy_out to the protoc call in gen_proto.sh) which created pyi files which also would need to be checked in. Would this approach work here?

[gforsyth]

Depending on your options you can actually use C++ protos for the Python under the hood. But ignoring that you can still end up with two definitions for protobufs which will collide and cause an exception when the module loads. If they are in different namespaces that collision won't happen (you can pretend to have two different sets of protos).

I don't think that renaming / aliasing the generated python functions will mess with the C++ descriptor pool, I believe it's only what is set as the proto namespace at code-generation time, but also I haven't messed with that particular intersection in a while.

[gforsyth]

thanks @amol!