CL-880 Elastic Transcoder Preset support. #25
Conversation
There was a problem hiding this comment.
I observed the following output when testing with the elastictranscoder.sh script from https://github.com/oreillymedia/cl-tf-aws/pull/46
us-east-1 - ElasticTranscoderPipeline - 1715359467543-hqdjwi - removed
us-east-1 - S3Bucket - s3://input-bucket-e755694d74824a1d5996 - [CreationDate: "2024-05-10T16:44:23Z", Name: "input-bucket-e755694d74824a1d5996"] - removed
us-east-1 - S3Bucket - s3://output-bucket-e755694d74824a1d5996 - [CreationDate: "2024-05-10T16:44:24Z", Name: "output-bucket-e755694d74824a1d5996"] - removed
us-east-1 - ElasticTranscoderPreset - 1715359468810-f2doyc - [PresetID: "1715359468810-f2doyc"] - removed
global - IAMRolePolicyAttachment - elastictranscoder-access-role-e755694d74824a1d5996 -> AmazonElasticTranscoder_FullAccess - [PolicyArn: "arn:aws:iam::aws:policy/AmazonElasticTranscoder_FullAccess", PolicyName: "AmazonElasticTranscoder_FullAccess", RoleCreateDate: "2024-05-10T16:44:25Z", RoleLastUsed: "2024-05-10T16:44:25Z", RoleName: "elastictranscoder-access-role-e755694d74824a1d5996", RolePath: "/"] - removed
There was a problem hiding this comment.
Hi Sherd, I ran both of your scripts successfully and was able to clean up the resources.
us-east-1 - ElasticTranscoderPipeline - 1715366296024-nrnbgn - removed
us-east-1 - ElasticTranscoderPreset - 1715366207569-i7a870 - [PresetID: "1715366207569-i7a870"] - removed
us-east-1 - MQBroker - b-890183ef-6cb5-482d-af27-eea52a58f552 - removed
A few observations/questions for you:
AWS MQ
I found that there was no mention the mq user or configuration being cleaned up in the logs. I checked AWS and found no users, I assume the user is deleted when the broker is deleted? As for the configuration, I found no way of deleting the mq config using the console. Do you know if there is a way to delete old mq configs?
ElasticTranscoder
I also didn't see in the logs any mention of the transcoder jobs being deleted. I am able to see the jobs via aws elastictranscoder list-jobs-by-pipeline --pipeline-id $PIPELINE_ID" --no-cli-pager" but am not able to see them in the AWS Console. Does it matter whether users can see old jobs or not?
Finally, I noticed was that I received a bunch of these messages in aws-nuke. Do you know what this is referring to?
us-west-2 - ElasticTranscoderPreset - 1351620000001-100070 - [PresetID: "1351620000001-100070"] - cannot delete elastic transcoder system presets
us-west-2 - ElasticTranscoderPreset - 1351620000001-100080 - [PresetID: "1351620000001-100080"] - cannot delete elastic transcoder system presets
us-west-2 - ElasticTranscoderPreset - 1351620000001-100090 - [PresetID: "1351620000001-100090"] - cannot delete elastic transcoder system presets
us-west-2 - ElasticTranscoderPreset - 1351620000001-100110 - [PresetID: "1351620000001-100110"] - cannot delete elastic transcoder system presets
us-west-2 - ElasticTranscoderPreset - 1351620000001-100120 - [PresetID: "1351620000001-100120"] - cannot delete elastic transcoder system presets
us-west-2 - ElasticTranscoderPreset - 1351620000001-100130 - [PresetID: "1351620000001-100130"] - cannot delete elastic transcoder system presets
There was a problem hiding this comment.
I observed the following output when testing with mq.sh from https://github.com/oreillymedia/cl-tf-aws/pull/46/files
us-east-1 - MQBroker - b-86e397ff-c312-4bbb-a200-1fbe44f81fa4 - removed
I have the same question as Cory. Should we be making use of delete-user and delete-tags?
@corybekk The MQ users get deleted with the broker, so no additional cleanup needs to happen there. The config files aren't deletable. For the most part they are auto generated when you create a broker. They seem to be pretty generic and don't hold any data that is a concern. For the ET(ElasticTranscoder) jobs, in my testing those where tied to the pipeline and deleted when the pipeline was deleted. Did you see otherwise? For the ElasticTranscoderPreset messages, that is due to the filter I added that keeps AWS managed 'system' presets from causing cleanup errors since they can't be deleted. The message is just showing those were filtered out. @danarbaugh With the tags, you can tag any resource created for the most part. I don't think we have necessarily had to worry about cleaning up tags when all the resources are being deleted. Were you seeing a resource not deleted with tags? |
There was a problem hiding this comment.
@swhite-oreilly I did not observe any issues with deleting resources with tags.
This PR adds support for ElasticTranscoder Preset Support.
ElasticTranscoder resources were created using the setup code found here, and then AWS Nuke was used to clean these new resources up, specifying :
"ElasticTranscoderPipeline"
"ElasticTranscoderPreset"
"MQBroker"
Note
ElasticTranscoder jobs are deleted when the pipeline is deleted.
MQ Users are deleted when the broker is deleted. MQ Configurations are not able to be deleted. They are created automatically when an MQ Broker is created or you can create a custom configuration, but these don't contain user identifiable information.