Better UX for multi-namespace mode

[ecordell]

I've always thought the UX for multi-namespace operators would be much better if we did something like this:

• The operator always retains read/write access to the SpiceDBCluster object at the cluster scope (i.e via a ClusterRole/ClusterRoleBinding), we'll call it spicedbcluster-rw.
• We create a ClusterRole for managing SpiceDB-related resources in a namespace; i.e. deployment create, service create. Let's call this manage-spicedb

In the default (all namespace) installation, we'll create a ClusterRoleBinding to manage-spicedb. Then the operator can do its work in any namespace.

If you want to let the operator work in just one namespace, i.e. my-spicedbs, then you could instead create a RoleBinding to manage-spicedb in my-spicedbs for the operator's service account. Now the operator has permission to create deployments only in my-spicedbs.

Similarly, if you have dev-spicedbs, staging-spicedbs, and prod-spicedbs namespaces, you can create one RoleBinding per namespace to grant the operator permission to work.

The nice thing about this model is that, because the operator retains R/W control on the SpiceDBCluster object in the entire cluster, it will be able to do things like write a status message in the object when permission is insufficient: Error: Operator does not have permission to list Deployments in namespace foo, please grant etc etc..., with explicit instructions for users to follow (if they have sufficient permission to perform them).

There's two main issues with this:

• Starting the appropriate watches at the right time. On start, the operator should try to watch all required resources in all namespaces, but shouldn't fail if it can't. Then, it should try and start a watch on any namespaces that contain SpiceDBCluster objects as they are created, and either report an error on the object or start reconciling. We don't start and stop watches dynamically like this in the spicedb-operator right now, though https://github.com/authzed/controller-idioms supports it (and we use it in some internal controllers at authzed).
• Detecting when permission has changed. The operator won't be able to watch for changes to its own permission (or at least, this doesn't seem like a reasonable thing to add). But I think on the whole that's not a big deal; we can exponentially backoff attempts to create new watches, and if you need to re-trigger faster than that you can edit the object or delete/recreate.

We could also think about doing this type of thing for some api calls; i.e. maybe you create an operator that can't create Jobs - so it can run spicedb but will error if it tries to run a migration. There might be some cases where this is useful; if you need to pre-provision serviceaccounts or something and don't want the operator to have permission to create them at all.

Originally posted by @ecordell in #291 (comment)