Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

Given a file and a chunk size in megabytes, calculates what the Amazon S3 etag will be.

License

Notifications You must be signed in to change notification settings

GeneDx/sema4-archive-calculate_multipart_etag

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 

Repository files navigation

DESCRPTION:

	Given a file and a chunk size in megabytes, calculates what the Amazon S3 etag will be.



	Amazon uses a simple md5 sum for an etag on single part uploads, but on multipart uploads they use a scheme of:
		md5ing the chunks,
		globbing the md5s together,
		converting the md5s to binary
		md5ing the binary of the chunks
		and appending -<Number_of_chunks> to the end

	This can make comparing files in s3 to local copies without downloading them a pain.

	The process to generate them is just convoluted enough that there's a lot of confused people out there on the forums.
	Lots of speculation, not much code examples other than some bash tools.

	It works from the command line, or import the function into your own code and use it there.

	Tested on python 3.4, but should work on 2.6+ unless I missed something obvious.

	Big thanks to antespi for his bash tool that does the same thing.
		 https://github.com/antespi/s3md5



USAGE:
	python calculate_multipart_etag.py source_path chunk_size [expected]

EXAMPLES:

	# just generate the etag

	$ python ./calculate_multipart_etag.py 100meg.file 50
	"67ff3c4020afd26957fe91c1891e362e-2"
	$


	# compare it with one you have already (success)

	$ python ./calculate_multipart_etag.py 100meg.file 50 "67ff3c4020afd26957fe91c1891e362e-2"
	"67ff3c4020afd26957fe91c1891e362e-2"

	# compare it with one you have already (failure)

	$ python ./calculate_multipart_etag.py 100meg.file 50 "1234567890abcdefghijklmnopqrstuv-42"
	Traceback (most recent call last):
	  File "./calculate_multipart_etag.py", line 81, in <module>
	    print(calculate_multipart_etag(source_path,chunk_size,expected))
	  File "./calculate_multipart_etag.py", line 56, in calculate_multipart_etag
	    raise ValueError('new etag %s does not match expected %s' % (new_etag,expected))
	ValueError: new etag "67ff3c4020afd26957fe91c1891e362e-2" does not match expected "1234567890abcdefghijklmnopqrstuv-42"
	$


About

Given a file and a chunk size in megabytes, calculates what the Amazon S3 etag will be.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%