Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Technical Report #268

Merged
merged 6 commits into from
Nov 1, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,6 @@ Please cite the repo if you use the code in this repo.
title = {Couler: Unified Machine Learning Workflow Optimization in Cloud},
year = {2023},
publisher = {GitHub},
howpublished = {\url{https://github.com/couler-proj/couler}}.
howpublished = {\url{https://github.com/couler-proj/couler/tree/master/docs/Technical-Report-of-Couler}}.
}
```
Binary file not shown.
8 changes: 7 additions & 1 deletion docs/Technical-Report-of-Couler/README.md
Original file line number Diff line number Diff line change
@@ -1 +1,7 @@
In this technical report, we delve into "Couler: Optimizing Machine Learning Workflows in Cloud", a framework designed to streamline the construction and execution of machine learning workflows. The report is segmented into three comprehensive chapters: Unified Programming Model, Implementation and Running Example.
Machine Learning (ML) has become ubiquitous, fueling data-driven applications across various organizations. Contrary to the traditional perception of ML in research, ML workflows can be complex, resource-intensive, and time-consuming.
Expanding an ML workflow to encompass a wider range of data infrastructure and data types may lead to larger workloads and increased deployment costs.
Currently, numerous workflow engines are available (with over ten being widely recognized). This variety poses a challenge for end-users in terms of mastering different engine APIs. While efforts have primarily focused on optimizing ML Operations (MLOps) for a specific workflow engine, current methods largely overlook workflow optimization across different engines.

In this work, we design and implement Couler, a system designed for unified ML workflow optimization in the cloud.
Our main insight lies in the ability to generate an ML workflow using natural language (NL) descriptions.
We integrate Large Language Models (LLMs) into workflow generation, and provide a unified programming interface for various workflow engines. This approach alleviates the need to understand various workflow engines' APIs. Moreover, Couler enhances workflow computation efficiency by introducing automated caching at multiple stages, enabling large workflow auto-parallelization and automatic hyperparameters tuning. These enhancements minimize redundant computational costs and improve fault tolerance during deep learning workflow training.
Loading