-
Notifications
You must be signed in to change notification settings - Fork 0
Project Proposal : Team pika pika
The Airavata MFT project focuses on file transfer across multiple cloud providers in a secure way in chunks. It brings up a framework that helps Airavata to transfer user data from one data store to another.
Our problem statement includes the current scenario of MFT, which currently has support only for 3 protocols namely SCP, Local and S3. Moreover, the system is currently not a standalone system, as it only binds to an existing service that requires MFT rather than a user directly talking to it. Furthermore, it is not deployed in a distributed environment to enable better scalability and fault tolerance.
We plan on making the system more usable by exposing an API, through which users can seamlessly integrate with their systems. This enables MFT to act as a standalone system that can be accessed by users and developers alike.
Also, to enable fault tolerance and scalability we look to provide the service in a distributed environment.
Finally we plan to extend the functionality of MFT by bringing up support for other protocols.
We plan to expose the user API by developing an API gateway that provides HTTP/REST endpoints for the user to interact with the MFT framework. This API enables functionality for the user to submit a transfer request and view the progress of the transfer.
After that, we plan to add protocols for transferring files between other HPC clients such as Azure, Google Cloud Storage, etc which the current system lacks.
We plan to then deploy the system in an orchestration environment such as Kubernetes. On top of that, we plan to explore different service mesh providers and use the one which is best suited for the current MFT architecture.