Support launching devappserver with user-supplied credential (including CT4E login credentials, service accounts, etc) and project ID used for Veneer libraries

[briandealwis]

Launching a local app in the devappserver that uses Google services (e.g., BigQuery) is difficult as we don't provide means to configure the application default credential nor the project. There has been some discussion about the ADC. #429 will at least allow (UPDATED by @chanseokoh: we can do this now) setting environment variables (like GOOGLE_APPLICATION_CREDENTIALS or GOOGLE_CLOUD_PROJECT), but it would be better to have some decent UI support, especially for selecting a user credential. Perhaps in the server definitions?

Steps to repeat:

1. Create a new Google App Engine Standard project with Java 7 with Maven support; in my testing, I used groupId=com.example.bq, artifactId=bq, and javaPackage=com.example.bq
2. UPDATED by @chanseokoh: Build Path > Add Libraries... > Google API Libraries
   Add the following to the pom.xml's <dependencies> (as per the BigQuery client library installation instructions;
   UPDATED by @chanseokoh: App Engine API will be provided for App Engine projects
   note that we also have to bring in the appengine-api-1.0-sdk, discussed further below:
3. Add the following sample code to the HelloAppEngine#doGet(), cribbed from the BigQuery Using the client library code:

    // Instantiates a client
    BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();

    // Prepares a new dataset
    String datasetName = "my_new_dataset";
    Dataset dataset = null;
    DatasetInfo datasetInfo = DatasetInfo.newBuilder(datasetName).build();

    // Creates the dataset
    dataset = bigquery.create(datasetInfo);

    response.setContentType("text/plain");
    response.setCharacterEncoding("UTF-8");
    response.getWriter().printf("Dataset %s created.\n", dataset.getDatasetId().getDataset());

Launching this app will fail for two reasons:
1. Cannot find a credential for access
2. Cannot find a Project ID

Problem 1: Cannot Resolve Application Default Credential

Launching will fail with an exception that BigQueryException: Could not get the access token. It turns out that this is due to b/63123716 and a workaround exists. The app will then use gcloud's default application credential.
UPDATE by @chanseokoh: the bug is fixed in latest gcloud releases.

We may want to provide support for specifying a different credential rather than the gcloud application-default credential?

Problem 2: Unable to determine app id

Because we are not recommending setting the <application> element in appengine-web.xml, the BigQuery library is unable to determine the Project ID (stacktrace below). With the workaround in b/63123716, (UPDATED by @chanseokoh: fixed in latest gcloud releases) we can provide the default Project ID via the com.google.appengine.application.id system property (UPDATE: unfortunately the devappserver overwrites the com.google.appengine.application.id system property) GOOGLE_CLOUD_PROJECT environment variable, though this is only used by the Veneer libraries and is not used by the App Engine SDK.

com.google.cloud.bigquery.BigQueryException: Invalid project ID 'no_app_id'. Project IDs must contain 6-63 lowercase letters, digits, or dashes. IDs must start with a letter and may not end with a dash.
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:86)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:141)
	at com.google.cloud.bigquery.BigQueryImpl$1.call(BigQueryImpl.java:175)
	at com.google.cloud.bigquery.BigQueryImpl$1.call(BigQueryImpl.java:172)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:94)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54)
	at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:172)
	at com.example.bq.HelloAppEngine.doGet(HelloAppEngine.java:60)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
    ...
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400
{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "Invalid project ID 'no_app_id'. Project IDs must contain 6-63 lowercase letters, digits, or dashes. IDs must start with a letter and may not end with a dash.",
    "reason" : "invalid"
  } ],
  "message" : "Invalid project ID 'no_app_id'. Project IDs must contain 6-63 lowercase letters, digits, or dashes. IDs must start with a letter and may not end with a dash."
}
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:139)
	... 43 more

Addendum: Must explicitly bring in appengine-sdk-1.0-api

Any google-cloud-java-based services require explicitly bringing in the appengine-sdk-1.0-api jar. It might be worth adding some validation to check for this case.
UPDATED by @chanseokoh: fixed by #2453. (See #2350 if for additional contexts.)

[elharo]

The addendum is essentially #2350 in a different library. I think we'll need to set our library dependencies accordingly.

On the rest of it, we should design and document what we want to do with ADC before starting work on this. Something does need to be done here, but I'm not yet sure what.

[chanseokoh]

We could cover the dev appserver, but I wonder what we can do for non-App Engine projects that use the Google Cloud libraries.

[briandealwis]

We could provide a 'Google Cloud' tab for Java config types that provides the AccountSelector and ProjectSelector, which are wired up to store/retrieve from the launch config's environment variables.

[chanseokoh]

@briandealwis updated the title. I think the old title no longer applies; I believe the local server should have no problem picking up the application default credential. And the intention was probably to allow users to provide a different credential (mainly the CT4E login credentials but also other forms of credentials such as service accounts).

[chanseokoh]

Fixed by #2568. (BTW, we only pass a service account key, not the CT4E login cred.)