Output structured diff results

[EyalLavi]

In addition to a single WER metric (#32), return the results of the diff metrics (s/d/i/c) in a structured format.

Something like:

{
"hypothesis-file": "file1.txt",
"diff-results":{"substitution":5, "deletions":10, "insertions":1, "correct":37}
}

Include vendor name? See also #33

[amessina71]

This is a good idea

[EyalLavi]

There will other metrics. Each metric should have its own structure.

{
"hypothesis": "machine-transcript.txt",
"reference":"human-transcript.txt",
"metrics":[
  {"type":"word-diff", "result":{"substitution":5, "deletions":10, "insertions":1, "correct":37}},
  {"type":"wer", "result":0.76}
]
}

[EyalLavi]

@MikeSmithEU in the future we may report metrics that are not numbers, so perhaps we should make all values strings, like:

{"type":"word-diff", "result":{"substitution":"5", "deletions":"10", "insertions":"1", "correct":"37"}},
  {"type":"wer", "result":"0.76"}

This way we leave the interpretation to the client.

[MikeSmithEU]

Indeed @EyalLavi Yet when the type fits a js type I would not pass it as a string, but the type itself.
In practice this means supporting: array, object, int, float, string, null/undefined (would use null, never undefined).

[EyalLavi]

The discussion above revolved around the summary of the diff results. I would like to add the diffs themsleves to the scope. This will allow apps to do their own processing on the results. How do we output those? Standard diff format wrap in the json? Or separately as standard diff?

[EyalLavi]

For Release 1, output the worddiffs metric. Something like this:

{
	"worddiffs":
	[{
	"ref":"hello",
	"hyp":"hello" 	         //match
	},
	{
	"ref":"world",
	"hyp":null		//deletion
	},
	{
	"ref":null,
	"hyp":"foo"		//insertion
	},
	{
	"ref":"bar",
	"hyp":"gar"		//replacement
	}]
}

For future releases, consider using the canonical transcript format.

[EyalLavi]

Discussion with @MikeSmithEU today:

• Output JSON format is metric-dependent
• Currently, -o json only

Example:

--worddiffs -o json

Produces:

[
    {"metric": "worddiffs",
    "result": [
        {
            "hypothesis": "this",
            "type": "equal",
            "reference": "this"
        },
        {
            "hypothesis": "is",
            "type": "equal",
            "reference": "is"
        },
        {
            "hypothesis": null,
            "type": "delete",
            "reference": "reference"
        }
     ]
}
]