Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'poll-less' icav2 data copy service #780

Open
alexiswl opened this issue Dec 17, 2024 · 0 comments
Open

Add 'poll-less' icav2 data copy service #780

alexiswl opened this issue Dec 17, 2024 · 0 comments
Assignees
Labels
feature New feature pipeline Workflow/Pipeline Manager

Comments

@alexiswl
Copy link
Member

Summary

A single event-based copy service that runs ICAv2 Copy Jobs and returns task success / failure on events

An event body might look like this

{
  "DetailType": "Icav2CopyJobSync",
  "Detail": {
    "taskToken": "{% $states.context.task.token %}"
    "sourceProjectDataObjects": [
     {
       "projectId": "abcd",
       "dataId": "fil.defg"
     }
    ],
    "destProjectDataFolderObject": {
        "projectId": "hijk",
        "dataId": "lmno"
    }    
  }
}

Part 1

Given a manifest of source project data ids and a destination data id, generate a copy job with the job id and task token

image

Part 2

Listen to ICA_JOB_001 events, send task success or task failure if the event fails, wait five seconds before performing any query since the job might exist before the part 1 lambda invoke returns

image

@alexiswl alexiswl added the feature New feature label Dec 17, 2024
@alexiswl alexiswl self-assigned this Dec 17, 2024
@victorskl victorskl added the pipeline Workflow/Pipeline Manager label Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature pipeline Workflow/Pipeline Manager
Projects
None yet
Development

No branches or pull requests

3 participants
@victorskl @alexiswl and others