Skip to content

Commit

Permalink
Fix check_for_lfs_files.sh extglob usage
Browse files Browse the repository at this point in the history 
In check_for_lfs_files.sh, we need to query all non-SEE assets.
To do this, we have previously used an extended globbing expression in Bash
(see `man bash` for details).

However, it turns out some bash installations (specifically, some Windows
GitBash instances) don't handle this correctly. To avoid this problem,
I've replaced the extglob pattern with a git diff exclusion pattern,
which should have the same effect while working on more machines.
  • Loading branch information
@falko17
falko17 committed Oct 18, 2023
1 parent 2715cee commit d7151fe
Showing 1 changed file with 48 additions and 54 deletions.
102 changes: 48 additions & 54 deletions GitScripts/check_for_lfs_files.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,67 +15,61 @@
# blob sizes after the commit has been made, so it can't be run as a
# pre-commit hook.

# Format: awk
# Maximum allowed file size.
FILE_SIZE="10 * 2^20" # 10 MB.
FILE_SIZE="10 * 2^20" # 10 MB.
# Maximum allowed total diff size.
DIFF_SIZE="50 * 2^20" # 50 MB.

DIFF_SIZE="50 * 2^20" # 50 MB.

if [ -n "$CI" ]; then

set -ex

# Check for non-SEE Assets.
shopt -s extglob
CHANGED_ASSETS=$(git diff --name-only --diff-filter=AM origin/master -- Assets/!(Editor|Native|StreamingAssets|Resources|Scenes|SEE*|XR)/ | git check-attr --stdin filter | grep -v "filter: lfs" || exit 0)
shopt -u extglob
if [ -n "$CHANGED_ASSETS" ]; then
echo ""
echo -n "::error title=Assets not in LFS::You have committed non-SEE assets outside of LFS.%0A"
echo -n "All (bought) non-SEE assets should be moved into LFS for space saving and copyright reasons.%0A"
echo -n "Please rebase your branch and move affected assets into LFS, so that no commit%0A"
echo -n "remains which references them.%0A"
echo -n "If you need help with this, or believe that moving these into LFS is not the right thing to do, please contact @koschke or @falko1 on Mattermost.%0A"
echo -n "Files which were detected as non-SEE assets are shown below.%0A"
echo "$CHANGED_ASSETS"
exit 1
fi
set -ex

# Check for non-SEE Assets.
CHANGED_ASSETS=$(git diff --name-only --diff-filter=AM origin/master -- Assets ':!Assets/Editor' ':!Assets/Native' ':!Assets/StreamingAssets' ':!Assets/Resources' ':!Assets/Scenes' ':!Assets/SEE*' ':!Assets/XR' | git check-attr --stdin filter | grep -v "filter: lfs" || exit 0)
if [ -n "$CHANGED_ASSETS" ]; then
echo ""
echo -n "::error title=Assets not in LFS::You have committed non-SEE assets outside of LFS.%0A"
echo -n "All (bought) non-SEE assets should be moved into LFS for space saving and copyright reasons.%0A"
echo -n "Please rebase your branch and move affected assets into LFS, so that no commit%0A"
echo -n "remains which references them.%0A"
echo -n "If you need help with this, or believe that moving these into LFS is not the right thing to do, please contact @koschke or @falko1 on Mattermost.%0A"
echo -n "Files which were detected as non-SEE assets are shown below.%0A"
echo "$CHANGED_ASSETS"
exit 1
fi

# Check for single changes bigger than 10 MB.
# From: https://stackoverflow.com/a/42544963 (slightly modified)
TOO_BIG=$(git rev-list --objects HEAD ^origin/master |
git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |
sed -n 's/^blob //p' |
awk "\$2 >= $FILE_SIZE { print \$2,\$3}" |
sort --numeric-sort --key=1)
if [ -n "$TOO_BIG" ]; then
# There were files which were too big.
echo ""
echo -n "::error title=Big files not in LFS::You have committed files bigger than 10 MB.%0A"
echo -n "Such files should be moved into LFS, otherwise the normal Git repository will%0A"
echo -n "become too big and slow.%0A"
echo -n "Please rebase your branch and move affected files into LFS, so that no commit%0A"
echo -n "remains which references them.%0A"
echo -n "If you need help with this, or believe that moving these into LFS is not the right thing to do, please contact @koschke or @falko1 on Mattermost.%0A"
echo -n "Files which were above the set size limit are shown below.%0A"
echo "$TOO_BIG"
exit 2
fi
# Check for single changes bigger than 10 MB.
# From: https://stackoverflow.com/a/42544963 (slightly modified)
TOO_BIG=$(git rev-list --objects HEAD ^origin/master |
git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |
sed -n 's/^blob //p' |
awk "\$2 >= $FILE_SIZE { print \$2,\$3}" |
sort --numeric-sort --key=1)
if [ -n "$TOO_BIG" ]; then
# There were files which were too big.
echo ""
echo -n "::error title=Big files not in LFS::You have committed files bigger than 10 MB.%0A"
echo -n "Such files should be moved into LFS, otherwise the normal Git repository will%0A"
echo -n "become too big and slow.%0A"
echo -n "Please rebase your branch and move affected files into LFS, so that no commit%0A"
echo -n "remains which references them.%0A"
echo -n "If you need help with this, or believe that moving these into LFS is not the right thing to do, please contact @koschke or @falko1 on Mattermost.%0A"
echo -n "Files which were above the set size limit are shown below.%0A"
echo "$TOO_BIG"
exit 2
fi

# Check for aggregate changes (additions only) bigger than 50 MB.
TOTAL_SIZE=$(git format-patch origin/master --stdout | grep '^+' | wc -c | awk "\$1 >= $DIFF_SIZE")
if [ -n "$TOTAL_SIZE" ]; then
echo ""
echo -n "::error title=Huge diff size::Your PR has a diff size of > 50 MB.%0A"
echo -n "You should put big directories into LFS.%0A"
echo -n "Please rebase your branch and move affected directories into LFS, so that no commit%0A"
echo -n "remains which references them.%0A"
echo -n "If you need help with this, or believe that moving these into LFS is not the right thing to do, please contact @koschke or @falko1 on Mattermost.%0A"
echo "Your PR diff size in bytes: $TOTAL_SIZE"
exit 3
fi
# Check for aggregate changes (additions only) bigger than 50 MB.
TOTAL_SIZE=$(git format-patch origin/master --stdout | grep '^+' | wc -c | awk "\$1 >= $DIFF_SIZE")
if [ -n "$TOTAL_SIZE" ]; then
echo ""
echo -n "::error title=Huge diff size::Your PR has a diff size of > 50 MB.%0A"
echo -n "You should put big directories into LFS.%0A"
echo -n "Please rebase your branch and move affected directories into LFS, so that no commit%0A"
echo -n "remains which references them.%0A"
echo -n "If you need help with this, or believe that moving these into LFS is not the right thing to do, please contact @koschke or @falko1 on Mattermost.%0A"
echo "Your PR diff size in bytes: $TOTAL_SIZE"
exit 3
fi

fi

0 comments on commit d7151fe

Please sign in to comment.