What information should the basis field contain?

[ross-spencer]

Given this example from a TGA: 'extension match tga; byte match at 4261283, 18' is Siegfried saying it read 4mb to read the last 18 bytes of the file, or is it seeking 18 bytes from the end of the file? TGA uses is identified using: TRUEVISION-XFILE.<null> last 18 bytes of the file. So, only 18 bytes are needed.

Does it matter? I imagined the max offsets creating small window either side of the file payload, e.g. 1000 bytes max from BOF or 500 bytes max from EOF. Establishing through SF alone requires knowing which values may be BOF or EOF?

filename : 'MARBLES.TGA'
filesize : 4261301
modified : 2022-06-02T13:14:16+02:00
errors   :
matches  :
  - ns      : 'pronom'
    id      : 'fmt/402'
    format  : 'Truevision TGA Bitmap'
    version : '2.0'
    mime    :
    basis   : 'extension match tga; byte match at 4261283, 18'
    warning :

[richardlehane]

it didn't read 4mb to match, this pattern would have been found during an EOF scan, it is just that in the basis field all offsets are reported as BOF offsets. The second value (18) is the length of the pattern match. Sometimes a signature requires multiple patterns to match e.g. a BOF and an EOF. In those cases you'll get a list of offset, length pairs e.g. byte match at [[0 14] [1822 2]]

You can of course convert it to an EOF offset by deducting it from the file size. If you want to establish a max window of BOF/EOF offsets, perhaps you could convert to EOF and assume it is an EOF offset if lower?

[ross-spencer]

You can of course convert it to an EOF offset by deducting it from the file size. If you want to establish a max window of BOF/EOF offsets, perhaps you could convert to EOF and assume it is an EOF offset if lower?

And now I'm having deja-vu! There must be something in previous emails or demystify issues talking about this.

I'll need to improve the subroutine here. This single EOF sequence is missing from that.

Ideally, I think what I'd prefer is the indicator to be provided by the tool like Siegfried, it might be something for DROID to decide on too (related to: digital-preservation/droid#773). But conceptually, I suspect what I'd like is wrong, because I'm essentially trying to think about this in terms of signature development, where it explicitly says there's an instruction to Siegfried/DROID that something is a BOF and EOF, and maybe those should be clear to be optimized too, but as a consumer of that information, Siegfried's concerns and what it displays back to the user is different? (but because we can put a heuristic into a tool like demystify or A.N. Other tool to work it out, it doesn't matter!)

Okay thanks Richard. Thinking out loud here, and caught between two different issues. Will close this and keep thinking about it.

[richardlehane]

There's a few reasons I just return absolute offsets in the basis:

• it gives sufficient information for a concrete reason for why sf matched: it found these bytes here and these bytes here, voila! You have the location now and can go and check if you want. If you want to reconcile this with the signature, then you can go check the signature too (e.g. with roy inspect) to see what the patterns were that matched in those byte locations
• an absolute offset is translatable to other tools e.g. xxd
• BOF/EOF is pretty much a pronom thing, the other identifiers (mimeinfo and LOC) don't have eof patterns
• it keeps the information returned a simpler
• it allows me to do tricks in the implementation without distorting the basis (e.g. if a user sets both a max bof and a max eof, then roy will translate all their bof wild cards into eof wild cards to increase probability of a match - because it doesn't matter if you match a wildcard pattern scanning forwards or backwards)