Skip to content
/ rdf2rml Public

RDF by Example: rdfpuml for True RDF Diagrams, rdf2rml for R2RML Generation

44 stars 5 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

VladimirAlexiev/rdf2rml

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

72 Commits
bin
bin
 
 
doc
doc
 
 
ideas
ideas
 
 
lib/RDF/Prefixes
lib/RDF/Prefixes
 
 
test
test
 
 
.gitignore
.gitignore
 
 
CITATION.bib
CITATION.bib
 
 
CITATION.cff
CITATION.cff
 
 
README.org
README.org
 
 

Repository files navigation

RDF by Example: rdfpuml for True RDF Diagrams, rdf2rml for R2RML Generation

Table of Contents

Introduction

See these publications:

  • RDF by Example: rdfpuml for True RDF Diagrams, rdf2rml for R2RML Generation. Vladimir Alexiev. In Semantic Web in Libraries 2016 (SWIB 16), Bonn, Germany, November 2016: Presentation, HTML, PDF, Video
  • Generation of Declarative Transformations from Semantic Models. Vladimir Alexiev. In European Data Conference on Reference Data and Semantics (ENDORSE 2023), Mar 2023: paper, presentation, video

RDF is a graph data model, so the best way to understand RDF data schemas (ontologies, application profiles, RDF shapes) is with a diagram. Many RDF visualization tools exist, but they either focus on large graphs (where the details are not easily visible), or the visualization results are not satisfactory, or manual tweaking of the diagrams is required.

  • rdfpuml makes true diagrams directly from Turtle examples using PlantUML and GraphViz. Diagram readability is of prime concern, and rdfpuml introduces various diagram control mechanisms using triples in the puml: namespace. Special attention is paid to inlining and visualizing various Reification mechanisms (described with PRV). We give examples from Getty CONA, Getty Museum, AAC (mappings of museum data to CIDOC CRM), Multisensor (NIF and FrameNet), EHRI (Holocaust Research into Jewish social networks), Duraspace (Portland Common Data Model for holding metadata in institutional repositories), Video annotation.

If the example instances include embedded source field names, they can describe a mapping precisely. I’ve implemented a few more tools to generate transformations:

  • rdf2rml generates R2RML transformations for RDBMS tables or SQL queries. Compared to R2RML, this saves about 15x in complexity and is competitive with the dedicated DSL YARRML
  • rdf2sparql generates OntoRefine or TARQL transformations from CSV/TSV that take the form of SPARQL UPDATE (for direct GraphDB loading) or CONSTRUCT (for conversion to RDF). (Subsumes two deprecated tools rdf2tarql and rdf2ontorefine)

See http://twitter.com/hashtag/rdfpuml for news, diagrams and announcements.

License and Citation

This work is covered by the Artistic-2.0 license.

If you use this software, please cite it as shown above.

Documentation

Related Work

Software:

  • Matjaz Rihtar (2018). https://github.com/mrihtar/rdfgraph: inspired by rdfpuml, written in Python 2.7, uses Redland’s librdf library. I worked with Matjaz in the euBusinessGraph project.

The following papers use or mention this software:

Installation

Checkout this repo and add rdf2rml/bin to your path. Install the following prerequisites:

  • both tools: Perl. Tested with version 5.22 on Windows (cygwin and Strawberry).
  • rdfpuml:
    • GraphViz
    • PlantUML. You need a recent version for new features like arrow length and color. I’m currently running 1.2018.10beta7. See in particular plantuml class diagrams.
    • Perl modules: use cpan or cpanm to install them: RDF::Trine RDF::Query Encode FindBin Carp::Always Slurp
    • RDF::Prefixes::Curie. This is my own module located in ./lib, and rdfpuml needs FindBin to locate it.
  • rdf2rml:
    • Apache Jena: riot, update. Tested with version 3.1.0 of 2016-05-10.
    • cat, grep, rm

Docker Image

If you prefer to work with Docker so you don’t need to install software manually, you can use this rdf2rml image from the public Nexus (Docker Registry) of Ontotext. To run it, use:

docker run -v <directory>:/files --rm docker-registry.ontotext.com/rdf2rml:latest`

Where <directory> is the local directory holding your .ttl files. It was made on 31 May 2023 and uses the following versions:

Note: pull request 7 of 17 Sep 2019 by Jem Rayfield (@jazzyray) dockerizes the installation, and makes extra changes related to input/output and configuration. However, it has not been merged yet

Debian Repo

Jonas Smedegaard (@jonassmedegaard, dr at jones fullstop dk) has volunteered for some of the tasks below. His development is at https://salsa.debian.org/debian/rdf2rml/branches. To adopt changes, do something like this.

  • To merge all commits in the salsa/develop branch: 
    cd rdf2rml    # i.e. your local clone of your Github project
git remote add salsa https://salsa.debian.org/debian/rdf2rml.git
git fetch salsa
git merge salsa/develop
  • To adopt only single commits from the salsa/develop branch, issue remote and fetch as above, then issue: 
    git cherry-pick $commit1 $commit2 $commit3

Change Log

2024-07-10 clarify licensing

  • Issue 31: settle on Artistic-2.0 license

2024-07-10 rdfpuml.pl: handle complex types

2023-06-07 rdf2sparql.pl: minimize binds in delete clause

Issue 27: minimize the delete clause to include only necessary binds:

  • --filterColumn variable prebind
  • templated GRAPH URL and its constituent variables

2023-06-06 rdf2sparql.pl: global --filter options

  • Issue 26: add command-line options --filterColumn, --filter that are useful for handling both initial loading and data updates.
  • See global filtering and test/graphs-crunchbase

2023-06-01 rdfpuml.pl: remove Carp::Always

  • Issue 2 remove Carp::Always since it produces a stack trace that’s too verbose

2023-05-17 rdf2sparql.pl: Conditional Nodes

  • Support “Conditional Nodes”, i.e. URLs that are conditional on the existence of some fields.
  • issue 22 fixed (2023-05-31)

2023-05-05 rdfpuml.pl: don’t mangle round brackets

  • issue 21: Round brackets in fields (eg “(name)”) and URLs (eg <type/(type)>) are not mangled to square brackets anymore

2023-04-29 rdfpuml.pl: puml:option

  • issue 18 Add puml:option for left to right direction etc

2023-04-19 rdf2sparql.pl: per-model filter, dynamic graph

  • issue 19 Implement filter function, see test/filter-content
  • issue 20 Allow dynamic graph (computed from a data column), see test/graphs-crunchbase

2022-08-23 rdf2sparql.pl: add datatype to var name instead of UPPERCASING

Datatype attachment eg strdt(?var,xsd:date) now outputs to ?var_xsd_date to avoid conflict with input field names in ALL_UPPERCASE

2022-08-23 rdfpuml.pl: handle blank-node types; add shell scripts

  • issue 10 Handle blank-node types that occur on owl:Restriction (see test/blank-node)
  • Duplicate rdfpuml.bat, puml.bat as shell scripts rdfpuml, puml for use in Makefiles across Linux and Windows

2022-08-15 rdf2sparql.pl: merge to one tool

Merge rdf2tarql and rdf2ontorefine to one tool rdf2sparql

2022-04-08 rdf2ontorefine.pl: generate OntoRefine Update queries

Add script to generate OntoRefine SPARQL Update queries from model.

2021-09-02 rdfpuml.pl: Unicode Processing

Use Perl option -C when invoking for proper Unicode processing. See doc section rdfpuml.html#Unicode

2020-09-17 rdf2rml: logicalTable

Use URL for logicalTable instead of blank node, so that R2RML generated from different models for different tables can be merged more easily. Warning: this assumes that all instances of one subjectMap use the same query.

2020-06-01 rdf2tarql.pl: generate TARQL scripts

Add rdf2tarql.pl script to generate TARQL script (CSV-RDF conversion) from model.

2020-06-01 rdf2rml: improve scripts, SQL query/table propagation

  • Improve script to abort if the first pipeline step (“update”) fails
  • Improve script to work on Cygwin (invokes the Jena tools as riot.bat and update.bat)
  • Filter out harmless warnings from Jena update’s error log for datatypes like xsd:integer, xsd:date etc since the mention of a source field doesn’t match the syntax of such literals.
  • If a node has single outgoing link and no SQL query/table (puml:label), propagate that property backward across the link into the node (previously that was done only for incoming links)

2020-05-30 rdf2rml: handle inverse edge

When an edge Y-P-X is recorded in the RDB table of X (as foreign key) or in an association table, it is awkward to specify that table in the node Y. So I added this SPARQL UPDATE clause:

  • If a node ?y has no SQL, is not Inlined, has a single outgoing edge, then add the SQL of its counterparty ?x as default

2018-11-14 rdfpuml.pl: avoid puml:stereotype class node

I often define puml:stereotype for some classes in prefixes.ttl. If the class is not used in some particular turtle, it should avoid emitting a disconnected puml class.

  • stereotypes(): Avoid emitting
  • has_statements_different_from(): Check that a node has statements other than puml:stereotype

2018-06-29 rdfpuml.pl bug: class and puml:InlineProperty

When a type is also used with puml:InlineProperty, it caused this error:

Can't locate object method "uri_value" via package "RDF::Trine::Node::Literal" at rdfpuml.pl line 261.
   main::puml_qname(RDF::Trine::Node::Literal=ARRAY(0x4fd0920)) called at rdfpuml.pl line 279
   main::puml_node2(RDF::Trine::Node::Literal=ARRAY(0x4fd0920)) called at rdfpuml.pl line 128

An inline is converted to a literal, but rdf:type is always assumed to be a URL. Test: ./test/regression/type-inlineProperty.ttl

2018-04-05 rdfpuml.pl: Arrow Attributes

Add arrow attributes (dotted, dashed, bold) and length Test: ./test/regression/arrowLen.ttl

2018-02-25 rdfpuml.pl: Arrow Color

Support arrow color (named or hex)

2017-08-25 rdfpuml.pl: decorative arrows

Fix unicode of “decorative arrows” on links going to a Reified Relation:

left => "←", right => "→", up => "↑", down => "↓"

2016-02-10 rdfpuml.pl: blank nodes, hidden links

To Do Tasks

Help needed for the following tasks. Post bugs and enhancement requests to this repo!

Near-term

Modularize and Package Better

Regression Tests

  • sort is added at various places to make the tool more deterministic, i.e. independent of order of RDF statements in the input file. However, this will interfere with the ability to control the layout, especially of disconnected components (see layout_new_line)
  • Some regression tests are added.

rdf2rml: disentangle inverse edge

In the case Y-P-X described above:

  • Also need to record ?y puml:property ?p so this prop name can be added to ?y’s subject map
  • When making ?map, take puml:property into account
  • But ?map is made many times, and copy-paste is no good…
  • Also, this should be done in some cases but not others…
  • So it’s better to record ?y puml:map ?map …

Release on CPAN

Add Unicode tests

Add ttl with non-ASCII chars: Accented, Cyrillic, French, etc.

Prefixes

Allow specifying the prefixes file

See #7

Eliminate Curie.pm

./lib/RDF/Prefixes/Curie.pm remembers @base and uses that for URL shortening. Once perlrdf#131 is fixed, eliminate this dependency (local module)

Remember prefixes from input file

rdfpuml shortens URLs using prefixes only from prefixes.ttl, but should also use prefixes defined in the individual input file.

Support more RDF Formats

Now it only supports Turtle, because it concatenates prefixes.ttl to the main file. If it can collect all prefixes from RDF files, such concatenation won’t be needed

Batch Processing

Issue #1: plantuml is slow to start up, so we’d like to process a bunch of puml files at once. The best way is to have a smarter script or Makefile that uses the following http://plantuml.com/command-line features:

  • Keep the intermediate puml files (the current Makefile doesn’t preserve them)
  • Run plantuml on a whole folder (with -r[ecurse] it can even recurse through subfolders)
  • Use -checkmetadata to skip png files that don’t need to be regenerated. (The whole puml text is stored in the png, so plantuml can quickly check that there are no changes)
  • The Makefile should start plantuml only once, if some of the puml files is newer than its respective png file

“Manual” Batching

Before I discovered the -checkmetadata option, I had the idea that rdfpuml could put several diagrams in one puml file:

@startuml file1.png
  # made from file1.ttl
@enduml
@startuml file2.png
  # made from file2.ttl
@enduml

However, this interferes with make processing that regenerates only png for changed ttl files, and makes things less modular overall.

Mid-Term

Upgrade to use Attean

Trine (Perl RDF) is end of life. Attean is the new generation

Integrate in Emacs org-mode

Write Turtle, see diagram (easy to do)

Node colors, icons, tooltips

See ./ideas

More arrow types and styles

./ideas/arrows.png ./ideas/arrows-2.png

dotted|dashed|plain|bold|hidden|norank|single|thickness

Extra Layout Options

Local layout options are described in Help on Layout:

  • “hidden” makes a constraint between two nodes, but does not draw the link (rdfpuml already implements this)
  • norank ignores a link for layout purposes (same as graphviz constraint=false)
  • “together” groups classes as if they were in the same package (i.e. puts them in a graphviz cluster)

Global options include (eg see this diagram):

And there are a lot more undocumented features: https://forum.plantuml.net/7095

Custom Reification

Ability to describe custom reification situations using the Property Reification Vocabulary (PRV)

Use MindMap/WBS for Hierarchies

Plantuml now has MindMap and WBS (or OBS) diagrams that use a simple bulleted syntax to draw hierarchies.

It would be nice to use this to draw hierarchies of individuals, in particular taxonomies.

Here are examples of the two styles:

Long-Term

rdf2soml to Generate Semantic Object Models

A new tool rdf2soml to generate Ontotext Platform SOML from RDF examples.

What’s missing? Most importantly: property cardinality and virtual inverses.

PlantUML can show arrow cardinalities, and this simple and natural PlantUML code:

X "0:1" -left-> "1:m" Y : prop/\ninvProp

Is depicted as follows:

./ideas/cardinality-and-inverse.png

We have two options how to express this in triples:

Cardinality With RDF*

##### model triples
:X :prop :Y.
##### puml triples
<< :X :prop :Y >>
  puml:arrow puml:left; # direction
  puml:min 1; puml:max puml:inf; # cardinality
  puml:inverseAlias [puml:min 0; puml:max 1; puml:name "invProp"]. # virtual inverse
  • Pros: very natural
  • Cons:
    • Perl RDF doesn’t support RDF*, and few editors support it either.
    • Annotating a triple does not assert it, so we need to assert it as well

Cardinality With Blank Node

##### model triples
:X :prop :Y.
##### puml triples
:X puml:left :Y. # direction
:X :prop [ # a puml:Cardinality; # may need this marker class to skip the node from the diagram
  puml:min 1; puml:max puml:inf; # cardinality
  puml:object :Y; # only needed if X has several relations "prop" and they need different annotations
  puml:inverseAlias [puml:min 0; puml:max 1; puml:name "invProp"] # virtual inverse
].

rdf2shape to Describe & Generate RDF Shapes

Visualize RDF Shapes (SHACL and ShEx)

Issue #8: discussion with Thomas Francart of Sparna

I developed this SHACL to PlantUML converter, in Java, based on TopQuadrant SHACL lib, and the result is at https://shacl-play.sparna.fr/play/draw and code at https://github.com/sparna-git/shacl-play/tree/master/shacl-diagram

I don’t have a strong opinion on the example you provide, an alternative idea that comes to my mind is

:node1 :link [
  rdf:value :node2;
  puml:min 1 ;
  puml:max 2 ;
]

But this changes the structure of the example graph itself, which might not be convenient

Generate transformations for other than relational sources

R2RML works great for RDBMS, but how about other sources? Extend rdf2rml to generate:

  • RML: extends R2RML to handle RDB, XML, JSON, CSV
  • XSPARQL: extends XQuery with SPARQL construct and JSON input
  • DONE tarql: handles TSV/CSV with SPARQL construct
  • DONE OntoRefine: transformation of TSV/CSV and direct loading to GraphDB with SPARQL Update

About

RDF by Example: rdfpuml for True RDF Diagrams, rdf2rml for R2RML Generation

Topics

visualization graphviz plantuml uml-diagram r2rml r2rml-mapping rdfpuml rdf2rml

Resources

Readme
Activity

Stars

44 stars

Watchers

6 watching

Forks

5 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages