[TT-1435] use chatgpt to find new issues in modified Solidity files #14104
Conversation
Tofel
changed the base branch from
develop
to
tt_1435_update_Sol_Foundry_with_diff
Tofel
changed the base branch from
tt_1435_update_Sol_Foundry_with_diff
to
develop
| # throw error openai_result when is not 200 | ||
| if [ "$openai_result" != '200' ]; then | ||
| echo "::error::OpenAI API call failed with status $openai_result: $(cat prompt_response.json)" | ||
| return 1 |
There was a problem hiding this comment.
should this just be exit 1?
There was a problem hiding this comment.
wouldn't that cause the pipeline to fail? If so, that's not what I want, I want to display a warning.
There was a problem hiding this comment.
this would return 1 here no from this?
if ./contracts/scripts/ci/find_slither_report_diff.sh "$base_report" "$current_report" "$new_issues_report" "contracts/scripts/ci/prompt-difference.md" "contracts/scripts/ci/prompt-validation.md"; then
then should it be if [[ $x == 1 ]]
maybe the above should be assigned to a var and then use that in the if to make it clearer?
There was a problem hiding this comment.
my bad, changed return to exit. but I'd keep as-is, as from what I found it's the recommended way of testing command exit code, not to mention the fact that in strict mode capturing that output in a variable and then doing if [[ $? -eq 0 ]]; wouldn't work as expected :/ non-zero exit code would terminate the execution.
Tofel
force-pushed
the
tt_1435_slither_diff_chatgpt
branch
from
5d30d49 to
aac42fa
Compare
| first_report_content=$(cat "$first_report_path" | sed 's/"//g' | sed -E 's/\\+$//g' | sed -E 's/\\+ //g') | ||
| second_report_content=$(cat "$second_report_path" | sed 's/"//g' | sed -E 's/\\+$//g' | sed -E 's/\\+ //g') | ||
| openai_prompt=$(cat "$report_prompt_path" | sed 's/"/\\"/g' | sed -E 's/\\+$//g' | sed -E 's/\\+ //g') | ||
| openai_model="gpt-4o" |
There was a problem hiding this comment.
I'd suggest that we pin the mode version to the current one that you are testing with (or make it a variable.)
Otherwise you are at risk of behavioral changes as newer versions are deployed to the 4o endpoints, we have seen breaking changes in format / responses before.
| exit 1 | ||
| fi | ||
|
|
||
| if [[ -z "${OPEN_API_KEY+x}" ]]; then |
There was a problem hiding this comment.
If preferred, you can safely remove this as $openai_result below will fail with ::error:: log telling you that you haven't supplied the key and we won't be charged for the call.
There was a problem hiding this comment.
I prefer to have it, because it's running in strict mode and will error if env var is not set, when read. I prefer to return a more explicit error than what bash would.
|
|
||
| if [[ -n "$validation_prompt_path" ]]; then | ||
| echo "::debug::Validating the diff report using the validation prompt" | ||
| openai_model="gpt-4-turbo" |
There was a problem hiding this comment.
Same comment for version pinning. To check the latest model version number, see: https://platform.openai.com/docs/models
|
Following changes have been implemented:
It's not bullet-proof, but I'd still give it a go and see if for now it's good enough.
Run, when no changes were found: https://github.com/smartcontractkit/chainlink/actions/runs/10385260800?pr=14111
Run, when changes were correctly found: https://github.com/smartcontractkit/chainlink/actions/runs/10385422699