Skip to content

Benchmark meta data file

Jump to bottom
Michael Röder edited this page Mar 10, 2017 · 5 revisions

The benchmark meta data file comprises meta data about the uploaded benchmark that is needed by the Hobbit platform. The file contains the meta data as RDF triples in the Turtle format and needs to be uploaded to the git instance of the platform, e.g., into the root directory of one of the projects that host one of the benchmark components, e.g., the benchmark controller. This article gives guidance how this meta data file can be created for a benchmark called "MyBenchmark".

First, we have to collect the necessary data:

  • The benchmark needs a unique identifier, i.e., a URI. In our example, we choose http://www.example.org/exampleBenchmark/MyOwnBenchmark.
  • The benchmark needs a name ("MyBenchmark") and a short description ("This is my own example benchmark...").
  • The name of the uploaded benchmark controller Docker image is needed ("git.project-hobbit.eu:4567/maxpower/mybenchmarkcontroller").
  • The URI of the benchmark API.
  • The version number of our benchmark implementation.
  • The parameters of the benchmark, their ranges and their default values.
  • The KPIs measured by the benchmark and their ranges.

Our example file has the following content

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix hobbit: <http://w3id.org/hobbit/vocab#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<http://www.example.org/exampleBenchmark/MyOwnBenchmark> a hobbit:Benchmark;
	rdfs:label "MyBenchmark"@en;
	rdfs:comment	"This is my own example benchmark..."@en;
	hobbit:imageName	"git.project-hobbit.eu:4567/maxpower/mybenchmarkcontroller";
	hobbit:version "v1.1"@en;
	hobbit:measuresKPI <http://www.example.org/exampleBenchmark/precision> ;
	hobbit:measuresKPI <http://www.example.org/exampleBenchmark/recall> ;
	hobbit:measuresKPI <http://www.example.org/exampleBenchmark/fmeasure> ;
	hobbit:hasAPI	<http://www.example.org/exampleBenchmark/API>;
	hobbit:hasParameter <http://www.example.org/exampleBenchmark/size>;
	hobbit:hasParameter <http://www.example.org/exampleBenchmark/queryScenario>;
	hobbit:hasParameter <http://www.example.org/exampleBenchmark/amountOfNullExamples> .

It can be seen that apart from the label, the description, a version and the image name our benchmark has an API, several KPIs and parameters assigned to it which will be explained in the following.

Note that a single benchmark.ttl file should contain only one single benchmark.

API

Inside the meta data files, the API is a URI that works as a simple identifier to be able to map systems to benchmarks. A benchmark can only be executed together with a system if both share the same API.

<http://www.example.org/exampleBenchmark/API> a hobbit:API .

KPIs

A Key Performance Indicator (KPI) is a value that shows the performance of the benchmarked system. It should have a label, a description and a range. In our example we have the three KPIs precision, recall and F-measure.

<http://www.example.org/exampleBenchmark/precision> a hobbit:KPI;
	rdfs:label "Precision"@en;
	rdfs:comment "Precision = TP / (TP + FP)"@en;
	rdfs:range xsd:float .

<http://www.example.org/exampleBenchmark/recall> a hobbit:KPI;
	rdfs:label "Recall"@en;
	rdfs:comment "Recall = TP / (TP + FN)"@en;
	rdfs:range xsd:float .

<http://www.example.org/exampleBenchmark/fmeasure> a hobbit:KPI;
	rdfs:label "F-measure"@en;
	rdfs:comment "F-measure is the harmonic mean of precision and recall."@en;
	rdfs:range xsd:float .

Parameters

There are two (overlapping) groups of parameters. configurable parameters can be configured by the user when starting a benchmark. Feature parameters are used for deeper analysis.

Every parameter should have a label, a description and a value range. Configurable parameters should have a default value that can be used by non-expert users to get the benchmark running.

<http://www.example.org/exampleBenchmark/size> a hobbit:ConfigurableParameter, hobbit:FeatureParameter;
	rdfs:label "Dataset size"@en;
	rdfs:comment "The size of the generated dataset counted in triples"@en;
	rdfs:range xsd:unsignedInt;
        hobbit:defaultValue "10000"^^xsd:unsignedInt .

Apart from parameters with simple values, HOBBIT supports parameters that you have to choose from a given set of values. To use this feature, the parameter needs to have a class as rdfs:range and the single choosable values need to be instances of that class.

<http://www.example.org/exampleBenchmark/graphPattern> a hobbit:ConfigurableParameter, hobbit:FeatureParameter;
	rdfs:label "Graph pattern"@en;
	rdfs:comment "The graph pattern used to generate the test data"@en;
	rdfs:range <http://www.example.org/exampleBenchmark/GraphPattern>;
        hobbit:defaultValue <http://www.example.org/exampleBenchmark/Star> .

<http://www.example.org/exampleBenchmark/GraphPattern> a rdfs:Class, owl:Class .

<http://www.example.org/exampleBenchmark/Star> a <http://www.example.org/exampleBenchmark/GraphPattern>;
	rdfs:label "Star graph pattern"@en;
	rdfs:comment "The graph has a central node and all other nodes have only one connection to this central node"@en .

<http://www.example.org/exampleBenchmark/Grid> a <http://www.example.org/exampleBenchmark/GraphPattern>;
	rdfs:label "Grid graph pattern"@en;
	rdfs:comment "A 2-dimensional grid of nodes"@en .

<http://www.example.org/exampleBenchmark/Clique> a <http://www.example.org/exampleBenchmark/GraphPattern>;
	rdfs:label "Clique graph pattern"@en;
	rdfs:comment "A fully connected graph"@en .

In the example, the benchmark offers different graph patterns from which the user will have to choose one.

In some cases, the benchmark might define a parameter that shouldn't be set by the user but is still interesting for later analysis, e.g., if the benchmark is based on data that is generated on the fly. This can be achieved by defining a parameter only as hobbit:FeatureParameter and not as configurable.

<http://www.example.org/exampleBenchmark/amountOfBlanknodes> a hobbit:FeatureParameter;
	rdfs:label "Amount of blank nodes"@en;
	rdfs:comment "The amount of blank nodes that have been created during the graph generation."@en;
	rdfs:range xsd:float .

In our example, the benchmark would generate a graph for benchmarking a system. After the generation and the evaluation of the systems results, the benchmark could add the amount of blank nodes that have been created to the result. Note that this is not a KPI since it is not bound to the performance of the system.

The complete example

In the following, you can find the complete example described in the section above.

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix hobbit: <http://w3id.org/hobbit/vocab#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<http://www.example.org/exampleBenchmark/MyOwnBenchmark> a hobbit:Benchmark;
	rdfs:label "MyBenchmark"@en;
	rdfs:comment	"This is my own example benchmark..."@en;
	hobbit:imageName	"git.project-hobbit.eu:4567/maxpower/mybenchmarkcontroller";
	hobbit:version "v1.1"@en;
	hobbit:measuresKPI <http://www.example.org/exampleBenchmark/precision> ;
	hobbit:measuresKPI <http://www.example.org/exampleBenchmark/recall> ;
	hobbit:measuresKPI <http://www.example.org/exampleBenchmark/fmeasure> ;
	hobbit:hasAPI	<http://www.example.org/exampleBenchmark/API>;
	hobbit:hasParameter <http://www.example.org/exampleBenchmark/size>;
	hobbit:hasParameter <http://www.example.org/exampleBenchmark/queryScenario>;
	hobbit:hasParameter <http://www.example.org/exampleBenchmark/amountOfNullExamples> .

<http://www.example.org/exampleBenchmark/API> a hobbit:API .

<http://www.example.org/exampleBenchmark/precision> a hobbit:KPI;
	rdfs:label "Precision"@en;
	rdfs:comment "Precision = TP / (TP + FP)"@en;
	rdfs:range xsd:float .

<http://www.example.org/exampleBenchmark/recall> a hobbit:KPI;
	rdfs:label "Recall"@en;
	rdfs:comment "Recall = TP / (TP + FN)"@en;
	rdfs:range xsd:float .

<http://www.example.org/exampleBenchmark/fmeasure> a hobbit:KPI;
	rdfs:label "F-measure"@en;
	rdfs:comment "F-measure is the harmonic mean of precision and recall."@en;
	rdfs:range xsd:float .

<http://www.example.org/exampleBenchmark/size> a hobbit:ConfigurableParameter, hobbit:FeatureParameter;
	rdfs:label "Dataset size"@en;
	rdfs:comment "The size of the generated dataset counted in triples"@en;
	rdfs:range xsd:unsignedInt;
        hobbit:defaultValue "10000"^^xsd:unsignedInt .

<http://www.example.org/exampleBenchmark/graphPattern> a hobbit:ConfigurableParameter, hobbit:FeatureParameter;
	rdfs:label "Graph pattern"@en;
	rdfs:comment "The graph pattern used to generate the test data"@en;
	rdfs:range <http://www.example.org/exampleBenchmark/GraphPattern>;
        hobbit:defaultValue <http://www.example.org/exampleBenchmark/Star> .

<http://www.example.org/exampleBenchmark/GraphPattern> a rdfs:Class, owl:Class .

<http://www.example.org/exampleBenchmark/Star> a <http://www.example.org/exampleBenchmark/GraphPattern>;
	rdfs:label "Star graph pattern"@en;
	rdfs:comment "The graph has a central node and all other nodes have only one connection to this central node"@en .

<http://www.example.org/exampleBenchmark/Star> a <http://www.example.org/exampleBenchmark/GraphPattern>;
	rdfs:label "Grid graph pattern"@en;
	rdfs:comment "A 2-dimensional grid of nodes"@en .

<http://www.example.org/exampleBenchmark/Star> a <http://www.example.org/exampleBenchmark/GraphPattern>;
	rdfs:label "Clique graph pattern"@en;
	rdfs:comment "A fully connected graph"@en .

<http://www.example.org/exampleBenchmark/amountOfBlanknodes> a hobbit:FeatureParameter;
	rdfs:label "Amount of blank nodes"@en;
	rdfs:comment "The amount of blank nodes that have been created during the graph generation."@en;
	rdfs:range xsd:float .
Clone this wiki locally