Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Image to text AI pipeline support #3202

Merged
merged 6 commits into from
Oct 23, 2024
Merged

Add Image to text AI pipeline support #3202

merged 6 commits into from
Oct 23, 2024

Conversation

mjh1
Copy link
Contributor

@mjh1 mjh1 commented Oct 11, 2024

What does this pull request do? Explain your changes. (required)

Adds support for the ai-worker pipeline which implements a image-to-text model (default https://huggingface.co/Salesforce/blip-image-captioning-large)

Corresponding AI-Worker PR:
livepeer/ai-worker#223

Specific updates (required)

How did you test each of these updates (required)

Tested by running the gateway and worker locally.

Does this pull request close any open issues?

Checklist:

@mjh1 mjh1 requested a review from rickstaa as a code owner October 11, 2024 13:55
@github-actions github-actions bot added the AI Issues and PR related to the AI-video branch. label Oct 11, 2024
Copy link

codecov bot commented Oct 22, 2024

Codecov Report

Attention: Patch coverage is 1.29032% with 153 lines in your changes missing coverage. Please review.

Project coverage is 35.92244%. Comparing base (c41f3c4) to head (ef6a2a8).
Report is 7 commits behind head on ai-video.

Files with missing lines Patch % Lines
server/ai_process.go 0.00000% 84 Missing ⚠️
server/ai_http.go 0.00000% 39 Missing ⚠️
core/ai_worker.go 0.00000% 30 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@                 Coverage Diff                 @@
##            ai-video       #3202         +/-   ##
===================================================
- Coverage   36.07820%   35.92244%   -0.15576%     
===================================================
  Files            124         124                 
  Lines          34525       34658        +133     
===================================================
- Hits           12456       12450          -6     
- Misses         21381       21520        +139     
  Partials         688         688                 
Files with missing lines Coverage Δ
core/ai.go 58.51852% <ø> (ø)
core/capabilities.go 81.69336% <100.00000%> (+0.04198%) ⬆️
server/ai_mediaserver.go 10.37037% <100.00000%> (+0.33320%) ⬆️
server/rpc.go 68.07229% <ø> (ø)
core/ai_worker.go 34.22104% <0.00000%> (-0.79258%) ⬇️
server/ai_http.go 10.32702% <0.00000%> (-0.74309%) ⬇️
server/ai_process.go 0.00000% <0.00000%> (ø)

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0ffcce6...ef6a2a8. Read the comment docs.

Files with missing lines Coverage Δ
core/ai.go 58.51852% <ø> (ø)
core/capabilities.go 81.69336% <100.00000%> (+0.04198%) ⬆️
server/ai_mediaserver.go 10.37037% <100.00000%> (+0.33320%) ⬆️
server/rpc.go 68.07229% <ø> (ø)
core/ai_worker.go 34.22104% <0.00000%> (-0.79258%) ⬇️
server/ai_http.go 10.32702% <0.00000%> (-0.74309%) ⬇️
server/ai_process.go 0.00000% <0.00000%> (ø)

... and 1 file with indirect coverage changes

@mjh1
Copy link
Contributor Author

mjh1 commented Oct 22, 2024

I've merged in the latest ai-video and resolved the conflicts etc so this is ready to review now 🙏

@rickstaa
Copy link
Contributor

@mjh1 I tested that his works on-chain. Very fast and strong pipeline (i.e. 200ms).

@rickstaa rickstaa merged commit 1bc4a6a into ai-video Oct 23, 2024
15 checks passed
@rickstaa rickstaa deleted the mh/image-to-text branch October 23, 2024 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AI Issues and PR related to the AI-video branch.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants