Empty ZIP file analysis

[JanVomlel]

I have created empty zip file. I think it is valid (created by python script), but siegfried identifies it as unknown format.
Why? Is it a bug in signature files? Is my file invalid?

File content:
504B0506000000000000000000000000000000000000

Siegfried output:

---
siegfried   : 1.10.1
scandate    : 2024-01-19T09:11:30Z
signature   : default.sig
created     : 2023-05-12T09:10:13Z
identifiers : 
  - name    : 'pronom'
    details : 'DROID_SignatureFile_V112.xml; container-signature-20230510.xml'
---
filename : '/var/workspace/pack2/output-files/001_m0001.zip'
filesize : 22
modified : 2024-01-19T09:11:25Z
errors   : 
matches  :
  - ns      : 'pronom'
    id      : 'UNKNOWN'
    format  : 
    version : 
    mime    : 
    class   : 
    basis   : 
    warning : 'no match; possibilities based on extension are x-fmt/263'

[ross-spencer]

Hi @JanVomlel Siegfried's ability to identify this file is driven by the PRONOM database and for zip files it expects a slightly different pattern: https://www.nationalarchives.gov.uk/PRONOM/x-fmt/263

504B01{43-65531}504B0506{18-65531}

So it wouldn't match the default signature. Perhaps it is something worth bringing up with the PRONOM team? https://github.com/digital-preservation/PRONOM_Research

Your sequence matches that in the small repo: https://github.com/mathiasbynens/small which is a good sign about it being syntactically correct. Incidentally I ran DROID against this set a while back, it didn't match a number of similar smallest possible files and I wondered if it might improve PRONOM identification in future to look at these: https://groups.google.com/g/droid-list/c/nwAtmup35Yk/m/aWn4myTFBAAJ

[Dclipsham]

Just to note re the above, that's the EOF sequence. The BOF sequence is seeking 504B0304 (the local file header) within 4 bytes of the start of the file. The sidepanel on the Wikipedia page (https://en.wikipedia.org/wiki/ZIP_(file_format)#:~:text=Magic%20number) seems to suggest 504B0506 (which otherwise marks the end of a central directory record) is specifically a magic number for an 'empty' zip so perhaps this is a special edgecase that gets its own variant signature.

[marhop]

Hi all,

I think you are right, the file is valid but the PRONOM signature doesn't cover this edge case.

To be precise, this is your file:

504B 0506 end of central directory signature, identifies this record type
0000      number of this disk (ZIP files may be split on multiple disks, like multiple floppies)
0000      number of the disk with the start of the central directory
0000      number of entries in the central directory on this disk
0000      total number of entries in the central directory (i.e., number of files in this ZIP)
0000 0000 size of the central directory, not counting this record itself
0000 0000 offset of the central directory
0000      length of the ZIP file comment (which would follow thereafter, if its length wasn't 0)

Since the ZIP file is empty it consists of only an "end of central directory record". And for the same reason most of the values in the end of central directory record are set to 0. This looks all fine to me. (See this ZIP file example if you want to know more details.)

Now if you look at the current PRONOM signature for the ZIP file format it essentially expects the following byte sequences in a ZIP file (in this order):

1. 504B0304, a local file header signature
2. 504B01, (part of) a central file header signature
3. 504B0506, the end of central directory signature

The first two sequences appear once for each file contained in a ZIP archive because they both introduce data structures that describe these files, while the third appears in every ZIP archive (empty or not) because there must always be an end of central directory record in a ZIP archive, even if the ZIP archive contains no files and thus the central directory contains no entries (see section 4.3.1 of the format specification).

You already know where that leads, right? Your ZIP contains no files, so the first two sequences don't occur in it (the third does occur, but all three are required), so the PRONOM signature doesn't match.

What can we do about that? The PRONOM signature could be restricted to require only the third sequence (the end of central directory signature) because that's the only sequence that will reliably appear in every ZIP file, even in an empty ZIP. But on the other hand, that would make the PRONOM signature less specific, possibly leading to more false positives. As always, it's a trade-off ... ;-) (@Dclipsham Or does "variant signature" imply that we could have both signatures?)

Cheers,
Martin

[Dclipsham]

Thanks Martin, this explains it perfectly.

Yes, a second signature is perfectly fine, and if either signatures match then FFID tools will return a hit. I wouldn't want to remove the existing signature as zip is also the basis for most formats that have a DROID container signature and it follows that all container formats must have at least one file, so it's a logical starting point for that route.

Naming the secondary signature variant something like 'Empty zip file' has the added bonus that Siegfried tells you which signature variant it matches against so you get that additional info without the need to create a separate PRONOM entry. Sadly DROID doesn't have that feature currently.

David

[marhop]

Sounds great, I didn't know about that feature. 😲

[JanVomlel]

Thank you for your replies @marhop , @ross-spencer @Dclipsham. I understand, that my file is valid and the droid signature files do not have entry to recognize Empty zip file. But I do not undesrstand If there is any solution for me:

• Can I file bug to droid signature files? Where? Is possible that this bug will be fixed in next signature file?
• Can I add my own signature file to my Siegfried installation and extend official signature file? Do you have any tips how to do it?
• Can I create empty zip file with different content to make it valid and recognized by droid signature files?

Thank you, Jan

[Dclipsham]

This isn't really a DROID bug, more of an enhancement to PRONOM. I've raised an issue over there for this: digital-preservation/PRONOM_Research#42. PRONOM tends to get new official releases every few months so it won't happen overnight, but hopefully you'll see this in a PRONOM release in the near future. PRONOM updates are officially announced on their Google Group, which you can subscribe to if you'd like to receive a notification when this happens: https://groups.google.com/g/pronom

Siegfried's companion project, Roy - https://github.com/richardlehane/siegfried/wiki/Building-a-signature-file-with-ROY - can be used to create custom signatures to enhance Siegfried

While I'm not familiar with the capabilties of the Python script you've used to create the empty zip, if you can create a zip with a single item in it then this should naturally end up conforming to the existing ZIP identification signature without further adjustments required.

I hope this is useful, I'm happy to answer any additional questions where I can!

David

[JanVomlel]

Thank you, Jan

[Dclipsham]

@JanVomlel if you'd like to be credited in the PRONOM release notes, please pop your name/or affliliation onto the PRONOM issue - digital-preservation/PRONOM_Research#42 (comment) - its just a way of acknowledging contributions by the PRONOM community and appears here: https://www.nationalarchives.gov.uk/aboutapps/pronom/release-notes.xml - no obligation.