Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Upgrade the version of the cluster parameter group for the created cl…
…uster Seems like that's what determines the version of the cluster, not an explicit EngineVersion attribute. That's the only place in the template that mentioned the 5.7 version number for MySQL. Update CFN stack to refer to the Serverless v2 scaling config attribute See: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-rds-dbcluster-serverlessv2scalingconfiguration.html No 'Autopause'. MinCapacity and MaxCapacity fields with numeric values. Seems like these bookkeeping files got updated automatically when I ran the sample app Add RDSDBInstance section to CloudFormation template Not sure if the !Ref is the right idiom to use for the DB cluster identifier, since other names in the template are hardcoded. Using a hardcoded name for the moment, for consistency. Take out engine mode entirely for cluster Rely on the default. (Specifying 'provisioned' explicitly would be confusing since the point of the example is to demonstrate Serverless v2.) Flip EnableHttpEndpoint to false For testing initial creation, it's not necessary to hook up the HTTP endpoint of Data API to the cluster. Since that support is not yet available. I'll flip that property back to true when the test environment is ready to really try out Data API with Aurora Serverless v2. Oops, I had MinCapacity and MaxCapacity flipped around Reorder them to it's clear the lower number (minimum) comes first. And then switch the numbers to the intended min and max values. Add a DependsOn attribute so instance creation waits for the cluster to be created first Introduce a !Ref operator to derive the DBClusterIdentifier property We don't know the actual ID of the cluster. That's generated automatically by CloudFormation. The !Ref points to the logical name used in the template for this cluster. Flip HTTP endpoint back to true for testing in eu-central-1 Punctuation cleanup for --sql examples Use \ line continuation characters for Unix compatibility, instead of ^. Remove final ; inside the --sql strings, it's not needed. Make the cluster and associated instance be Aurora PostgreSQL version 15 Adapt README for using Serverless v2 in the demo Update wording in README to focus less on Serverless cluster notion Since with Aurora Serverless v2, the serverless aspect applies to the database instance, not the cluster. Slight wording changes w.r.t. 'serverless cluster' in comments First cut at updating the creation logic to include a Serverless v2 instance along with the cluster Changed the engine to aurora-postgresql and added engine version 15, so that the code could use the latest Data API. Enhance the code that implements "wait for cluster to be created" Make it also wait for the associated DB instance. There's an assumption that the DB instance has a predictable name, i.e. that it was auto-created by this same app and its name is a mangled version of the cluster name. Create a PostgreSQL wrapper for the library app equivalent to the MySQL one Start by copying the MySQL one verbatim. Do the code changes in subsequent commits. Change MySQL-related wrapper source to be compatible with PostgreSQL Substantive changes are mainly around auto-increment columns. Instead of adding the AUTO_INCREMENT keyword into the column definition, the autoincrement attribute in the Python source just indicates which column(s) to skip in INSERT statements. It's the responsibility of the caller to specify one of the PostgreSQL auto-increment types like SERIAL or BIGSERIAL. Bring in the PostgreSQL-compatible helper code Instead of the MySQL-compatible wrapper functions. Make a start on making fill_db_tables independently callable via command-line argument The better to be able to run that part multiple times during development and debugging of the Postgres-compatible code, without having to re-run all the cluster teardown and setup steps every time. Add a little debugging output to see what's being executed for batch SQL statements Add some debugging code around executing SQL statements Print the statements and associated placeholder parameters. Undo the attempt to pass serial directly as a data type, use another approach At the point the CREATE TABLE statement and its column list are being constructed, when a column is auto-incrementing, explicitly fill in a PostgreSQL SERIAL data type instead of any INT type that was put into the Column structure earlier. Add more debugging code around interpretation of results from batch SQL statement See if the results are coming back empty, or with a different shape than expected, or different field names. The better to figure out how best to do try/catch logic at places that are throwing exceptions during MySQL -> PostgreSQL porting. Make the insert() operation use a RETURNING * clause Since Data API for PostgreSQL doesn't include data in the generatedFields pieces in the result set. Add commentary about the foreign key relationship to add_books() In the middle of revising the add_books() function to not rely on batch statement execution The code isn't expected to work yet. I'm in the middle of adding debug output to see how to reformat the data structures into simpler layout to make a single big string argument for the VALUES clause. Committing this intermediate stage because things could go haywire from here on an I might need to revert to this spot. Make the INSERT statement for the Authors table work from a single big string Supply all the VALUES data as part of the original SQL string and submitted to ExecuteStatement, not an array of parameters used with BatchExecuteStatement. Add a little exception handling to the point in the REST library demo code that throws an error It fails to "lend" a book to a patron. Need to re-run everything end-to-end to see if that could be a side-effect of something like duplicate data from running the REST demo multiple times. Make the string quoting in the VALUES string use dollar delimiters If the VALUES string is constructed with single-quoted values, e.g. ('firstname','lastname'), then it's vulnerable to a syntax error for names like O'Quinn that contain a single quote. So I'm making the delimiters be $$first$$ and $$last$$ to avoid any possibility of collisions. Don't use location constraint for S3 bucket creation if region is us-east-1 Somehow it throws an exception. Even though the same code worked in us-west-2. This could be because us-east-1 used to be the default for all buckets. (Maybe an old API version is lurking somewhere that doesn't know about LocationConstraint.) Start taking out redundant RETURNING * clauses I added RETURNING * to INSERT wherever it might be needed. Some instances have no effect or in places that are now dead code. I commented out all occurrences. That resulted in an error during the data setup. So I re-enabled the one occurrence that seems to be on a live code path. Still testing if doing so will fix the error. Add some debugging code to unpack_insert_results function If it throws an exception, print the result set parameter that didn't have the expected metadata, and the exception details. Then re-raise the exception to halt execution. I added the same code to the MySQL and PostgreSQL helper functions. I believe the same limitation with generatedFields applies to both engines when Data API is used with Aurora Serverless v2. Add some more debugging output around submitting SQL to lend or return books Add a distinctive message to the / URL of the API Gateway I'm not sure if the API Gateway is being entirely torn down and set up again on each runthrough. I'm not seeing all the debug output I expect. So changing the top-level help message to verify one way or the other. Add more workaround code for INSERT operations Also a lot of exception/debug/tracing code to verify exactly which operations fail and what the parameters and return values are around that vicinity. There are some duplicated function names in the demo app and the API Gateway code. So it's especially important to distinguish which one is being run when an exception is encountered. More exception handling / debugging / tracing output around the lending and returning functions Some more exception handling/reporting when running a single SQL statement A little more change for debug purposes Bring back None instead of "NULL" when constructing query statements intended to have IS NULL clauses. Add some more exception handling around another place where a SQL statement is issued. Change from IS NULL to IS NOT DISTINCT FROM NULL In get_borrowed_books() and return_book(). Because the substitution at the point of 'null' isn't allowed in Postgres with JDBC protocol. Even though it is allowed in MySQL. Remove tracing output statements from library_api/chalicelib/library_data.py Leave in a few that are in exception handlers and could be useful for future debugging. Change those to logger.info() if mild, logger.exception() if serious. Take out more debugging/tracing print() calls, from aurora_rest_lending_library/library_api/app.py Remove one stray debugging print() call from library_api/chalicelib/library_data.py Remove debugging/tracing print() calls from library_api/chalicelib/postgresql_helper.py Possibly final code tweaks and removal of debug/tracing messages from library_demo.py Fix the query to decode borrowed books Be more flexible in date/time-related types that get the DATE type hint. Don't cast today's date to a string, so it's recognized as a date/time type. Return the value at the end of get_borrowed_books(). I assigned it to a variable as part of debugging but didn't actually return the variable. Set up CDK setup path to use Serverless v2 instead of Serverless v1 Create DatabaseCluster instead of ServerlessCluster. Still likely needs some more to make an instance associated with the DatabaseCluster - that wasn't needed in Serverless v1. And then the instance needs to have instance class 'db.serverless' instead of one of the provisioned instance classes like db.t4g.medium. The last piece would be to include the 'enable Data API' / 'enable HTTP endpoint' flag. However, that doesn't seem to be available yet for DatabaseCluster in the CDK. There is an open pull request to add it: aws/aws-cdk#29338 So I think the changes to setup.ts can't be 100% finalized until that pull request is merged. Although I was able to test the updated Python examples by setting up resources using the CloudFormation setup.yaml and/or 'python library_demo.py deploy_database'. Take out commented SERIAL lines from table definitions I thought the types might have to be declared serial for Postgres compatibility. But in the end the type gets switched from int to serial based on the presence of the auto_increment attribute. Remove a couple of note-to-myself comments Remove commented-out attribute from SQL query submitted to Data API I thought it might be needed for debugging, but no Remove unused code from mysql_helper module MySQL doesn't have a RETURNING clause, so the Data API / JDBC workaround for INSERT/UPDATE/DELETE result sets that applies to PostgreSQL isn't applicable for MySQL. Restore some print() calls I changed to logger.info() by mistake Reformat Lending Library Python code with 'black' linter Record in the README that I updated the CDK version This is to account for the switch from InstanceProps to writer as the attribute of the DatabaseCluster that happened in Nov 2023. Update CDK version to 2.132.1 This is to account for the switch from InstanceProps to writer as the attribute of the DatabaseCluster that happened in Nov 2023. Also the addition of the enableDataApi attribute on DatabaseCluster that only happened in Mar 2024, in 2.132.1. Whatever software versions got updated during recent npm work In particular, CDK to 2.132.1. Some other versions might have gotten updated without my doing that intentionally. Change from a ServerlessCluster to a DatabaseCluster for Aurora Serverless examples This is so the cluster can be provisioned, but in the end will use a db.serverless instance. That's the way it has to work with Aurora Serverless v2. This is an interim commit that just tries to create a provisioned instance. Because the VPC parameter is giving problems. The class declaration isn't picking up the default account ID and AWS Region from the environment variables where I have them declared locally. Also I am in the middle of switching from old-style InstanceProps to new-style 'writer' attribute. But that's hung up by the VPC business. So for the moment I have both bits of code in the source file, switching which one is commented out during different test runs. Mods for VPC and engine version that allow CDK operations to get farther than before Now it's failing because of lack of private subnets. With some more version and subnet specs, now operations like 'cdk ls' succeed It still complains about the deprecated instanceProps attribute. So next step is to switch to 'writer' and get that working. Switched to 'writer' attribute due to deprecation of 'instanceProps' Had to boost VPC and VPC subnets attributes out of the instance and up to the cluster level. Also had to explicitly ask for a provisioned or Serverless v2 instance. Just to get things working, trying with provisioned. Switched to Serverless v2 instance for the writer Commented out the instance class spec in the props. That makes the props empty. Which is OK because I don't want to be too over-detailed in this basic example. Just take defaults wherever possible. Bring back the original call to output the cluster ARN When that wasn't working, it must have been due to an earlier failure to deploy the cluster. Remove commented-out experimental or stale code from setup.ts Leave only the latest and greatest. Specs: PostgreSQL, version 15.5, and a Serverless v2 instance in the Aurora cluster. Update README instructions for Aurora Serverless v2 and PostgreSQL Apply same Serverless v2 + Data API change to INSERT statement as in other examples Turn the DDL statement into a query. Get the auto-generated ID value back from "records" instead of "generatedFields".
- Loading branch information
1 parent
badf20a
commit a78fff7