Create image file values in API v2 (#1011)

* feature (sipi): Add Lua code for uploading a file to Sipi as per #998

* refactor (api-v2): Refactor file value classes.

- Remove deprecated properties isPreview and qualityLevel from API v2.

* feature (api-v2): Implement file value creation (ongoing).

* feature (api-v2): Implement file value creation (ongoing).

* test (api-v2): Test creating and updating still image file values with mock Sipi.

* feature (api-v2): Have Sipi move a temporary file to permanent storage (ongoing).

- Fix broken tests.

* feature (api-v2): Have Sipi move a temporary file to permanent storage (ongoing).

* refactor (Authenticator): Start replacing deprecated JWT library (#1044).

* refactor (Authenticator): Continue replacing deprecated JWT library (#1044).

* refactor (Authenticator): Finish replacing deprecated JWT library (#1044).

- Add more JWT tests.

* feature (sipi): Improve Lua script for Sipi file upload (ongoing).

- Delete old temporary files each time an upload is processed.
- Fix incorrect log level names in Lua scripts.

* feature (sipi): Validate JWT token in file upload.

* feature (sipi): Improve JWT token validation.

* test (api-v2): Add integration test for creating a file value with Sipi (ongoing).

- Fix some bugs.
- Fix broken authentication message package structure.

* fix (sipi): Fix lots of Lua script bugs.

- Improve error-handling in Lua scripts.
- Fix test data for knora-api:fileValueAsUrl.

* feature (api-v2): If a file value triplestore update fails, have Sipi delete the temp file (ongoing).

* feature (sipi): Make SipiImage directly from uploaded file.

- Take into account directory hashing in Lua scripts.
- Recursively clean up temporary directory.
- Clean up Lua scripts.

* test (sipi): Test deleting temp file if file value creation fails.

* test (sipi): Activate subdirs in temp dir for tests.

* refactor (sipi): Use simple form of SipiImage.new.

* test (api-v2): Add tests of file uploads.

- Test creating a resource with a file value.
- Test Knora's handling of Sipi errors.

* test (sipi): Test creating a resource with multiple file values.

- Fix upload.lua to preserve the order of uploaded files.
- Give knora-api:stillImageFileValueHasIIIFBaseUrl a type of xsd:anyURI.
- Fix test data.

* feature (sipi): Have Sipi return original filename in response to upload.

* docs (api-v2): Add API and design docs about file uploads.

* docs (release-notes): Update release notes.

* test (sipi): Use new login request format.

* docs (api-v2): Add warning about #1068.

* fix (api-v1): Adapt API v1 to ignore preview image file values in triplestore.

- Update tests.

* fix (sipi): Use a JPEG 2000 image to generate previews for testing.

* feature (sipi): Clean up a few things.

* test (api-v1): Test resource context response when there's no preview image in the triplestore.

.gitignore

@@ -35,3 +35,4 @@ docs/_format_docu_v2
 webapi/_fuseki/run
 triplestores/graphdb-se-7
 idea.vmoptions
+sipi/test

docs/src/paradox/00-release-notes/v2.x.x.md

@@ -42,5 +42,3 @@ section in the documentation for more information (@github[#1025](#1025))
 ### Bugfixes:
 
 - sipi container config / sipi not able to talk to knora (@github[#988](#994))
-
-

docs/src/paradox/00-release-notes/v3.x.x.md

@@ -5,6 +5,7 @@
 
 - [BREAKING ONTOLOGY CHANGE] The property `knora-base:username` was added and is required for `knora-base:User`. (@github[#1047](#1047))
 - [BREAKING API CHANGE] The `/admin/user` API has changed due to adding the `username` property. (@github[#1047](#1047))
-- [FIX] Incorrect standoff to XML conversion if empty tag has empty child tag (@github[#1054](#1054))
 - [FEATURE] Add default permission caching (@github[#1062](#1062))
 - [FIX] Fix unescaping in update check and reading standoff URL (@github[#1074](#1074))
+- [FIX] Incorrect standoff to XML conversion if empty tag has empty child tag (@github[#1054](#1054))
+- [FEATURE] Create image file values in API v2 (@github[#1011](#1011)). Requires Sipi with tagged commit `v1.4.1-SNAPSHOT` or later.

docs/src/paradox/03-apis/api-v2/editing-values.md

@@ -183,6 +183,91 @@ This document can then be embedded in a JSON-LD request, using the predicate `kn
 Note that quotation marks and line breaks in the XML must be escaped, and that the IRI of the mapping must be
 provided.
 
+## Creating File Values
+
+Knora supports the storage of certain types of data as files, using
+[Sipi](https://github.com/dhlab-basel/Sipi)
+(see @ref:[FileValue](../../02-knora-ontologies/knora-base.md#FileValue)).
+Knora API v2 currently supports using Sipi to store image files. Support for
+other types of files will be added in the near future.
+
+The following sections describe the steps for creating a file value.
+
+### Upload Files to Sipi
+
+The first step is to upload one or more files to Sipi, using a
+`multipart/form-data` request, where `sipihost` represents the host and
+port on which Sipi is running:
+
+```
+HTTP POST to http://sipihost/upload?token=TOKEN
+```
+
+The `token` parameter must provide the [JSON Web Token](https://jwt.io/)
+that Knora returned when the client logged in. Each body part in the request
+must contain a parameter `filename`, providing the file's original filename,
+which both Knora and Sipi will store; these filenames can be descriptive
+and need not be unique.
+
+Sipi will then convert the uploaded image files to JPEG 2000 format and store
+them in a temporary location. If this is successful, it will return a JSON
+response that looks something like this:
+
+```json
+{
+  "uploadedFiles": [{
+    "originalFilename": "manuscript-1234-page-1.tiff",
+    "internalFilename": "3UIsXH9bP0j-BV0D4sN51Xz.jp2",
+    "temporaryBaseIIIFUrl": "http://sipihost/tmp/3UIsXH9bP0j-BV0D4sN51Xz.jp2"
+  }, {
+    "originalFilename": "manuscript-1234-page-2.tiff",
+    "internalFilename": "2RvJgguglpe-B45EOk0Gx8H.jp2",
+    "temporaryBaseIIIFUrl": "http://sipihost/tmp/2RvJgguglpe-B45EOk0Gx8H.jp2"
+  }]
+}
+```
+
+In this example, we uploaded two files to Sipi, so `uploadedFiles` is an
+array with two elements. For each file, we have:
+
+- the `originalFilename`, which we submitted when uploading the file
+- the unique `internalFilename` that Sipi has randomly generated for the file
+- the `temporaryBaseIIIFUrl`, which we can use to construct a IIIF URL for
+  previewing the file
+
+The client may now wish to get a thumbnail of each uploaded image, to allow
+the user to confirm that the correct files have been uploaded. This can be done
+by adding IIIF parameters to `temporaryBaseIIIFUrl`. For example, to get
+a JPG thumbnail image that is 150 pixels wide, you would add
+`/full/150,/0/default.jpg`.
+
+### Submit A File Value to Knora
+
+After uploading a file to Sipi, you can submit a request to Knora to create
+a file value. The request submitted to Knora takes the same form as any other
+request to create a value. Instead of providing the file's complete metadata,
+you just provide the unique internal filename generated by Sipi. For example:
+
+```jsonld
+{
+  "@id" : "http://rdfh.ch/0001/a-thing-picture",
+  "@type" : "anything:ThingPicture",
+  "knora-api:hasStillImageFileValue" : {
+    "@type" : "knora-api:StillImageFileValue",
+    "knora-api:fileValueHasFilename" : "3UIsXH9bP0j-BV0D4sN51Xz.jp2"
+  },
+  "@context" : {
+    "knora-api" : "http://api.knora.org/ontology/knora-api/v2#",
+    "anything" : "http://0.0.0.0:3333/ontology/0001/anything/v2#"
+  }
+}
+```
+
+Knora then gets the rest of the file's metadata from Sipi. If the client's
+request to Knora is valid, Knora saves the file value in the triplestore and
+instructs Sipi to move the file to permanent storage. Otherwise, the
+temporary file that was stored by Sipi is deleted.
+
 ## Updating a Value
 
 To update a value, use this route:

docs/src/paradox/03-apis/api-v2/query-language.md

@@ -192,6 +192,7 @@ clauses use the following patterns, with the specified restrictions:
     argument of a comparison operator must be a query variable.
     A Knora ontology entity IRI used in a `FILTER` must be a property IRI.
   - `FILTER NOT EXISTS`
+  - `MINUS`
   - `OFFSET`: the `OFFSET` is needed for paging. It does not actually
     refer to the number of triples to be returned, but to the
     requested page of results. The default value is 0, which refers

docs/src/paradox/05-internals/design/api-v2/index.md

@@ -0,0 +1,30 @@
+<!---
+Copyright © 2015-2018 the contributors (see Contributors.md).
+
+This file is part of Knora.
+
+Knora is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published
+by the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+Knora is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public
+License along with Knora.  If not, see <http://www.gnu.org/licenses/>.
+-->
+
+# Knora API v2 Design
+
+@@toc { depth=1 }
+
+@@@ index
+
+- [API v2 Design Overview](overview.md)
+- [Knora and Sipi](sipi.md)
+- [Gravsearch](gravsearch.md)
+
+@@@

docs/src/paradox/05-internals/design/api-v2.md → docs/src/paradox/05-internals/design/api-v2/overview.md

@@ -17,7 +17,7 @@ You should have received a copy of the GNU Affero General Public
 License along with Knora.  If not, see <http://www.gnu.org/licenses/>.
 -->
 
-# Knora API v2 Design
+# API v2 Design Overview
 
 @@toc
 
@@ -57,7 +57,7 @@ provided.
 
 Each schema has its own type IRIs, which are derived from the ones used
 in the triplestore. For details of these different IRI formats, see
-@ref:[Knora IRIs](../../03-apis/api-v2/knora-iris.md).
+@ref:[Knora IRIs](../../../03-apis/api-v2/knora-iris.md).
 
 ## Implementation
 

docs/src/paradox/05-internals/design/api-v2/sipi.md

@@ -0,0 +1,120 @@
+<!---
+Copyright © 2015-2018 the contributors (see Contributors.md).
+
+This file is part of Knora.
+
+Knora is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published
+by the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+Knora is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public
+License along with Knora.  If not, see <http://www.gnu.org/licenses/>.
+-->
+
+# Knora and Sipi
+
+@@toc
+
+## Configuration
+
+The Knora-specific configuration and scripts for Sipi are in the
+`sipi` subdirectory of the Knora source tree. See the `README.md` there for
+instructions on how to start Sipi with Knora.
+
+## Lua Scripts
+
+Knora API v2 uses custom Lua scripts to control Sipi. These scripts can be
+found in `sipi/scripts` in the Knora source tree.
+
+Each of these scripts expects a [JSON Web Token](https://jwt.io/) in the
+URL parameter `token`. In all cases, the token must be signed by Knora,
+it must have an expiration date and not have expired, its issuer must be `Knora`,
+and its audience must include `Sipi`. The other contents of the expected tokens
+are described below.
+
+### upload.lua
+
+The `upload.lua` script is available at Sipi's `upload` route. It processes one
+or more file uploads submitted to Sipi. It converts uploaded images to JPEG 2000
+format, and stores them in Sipi's `tmp` directory. The usage of this script is described in
+@ref:[Creating File Values](../../../03-apis/api-v2/editing-values.md#creating-file-values).
+
+Each time `upload.lua` processes a request, it also deletes old temporary files
+from `tmp` and (recursively) from any subdirectories. The maximum allowed age of
+temporary files can be set in Sipi's configuration file, using the parameter
+`max_temp_file_age`, which takes a value in seconds, and defaults to
+86400 (1 day).
+
+### store.lua
+
+The `store.lua` script is available at Sipi's `store` route. It moves a file
+from temporary to permanent storage. It expects an HTTP `POST` request containing
+`application/x-www-form-urlencoded` data with the parameter `filename`, whose
+value is the internal Sipi-generated filename of the file to be moved.
+
+The JWT sent to this script must contain the key `knora-data`, whose value
+must be a JSON object containing:
+
+- `permission`: must be `StoreFile`
+- `filename`: must be the same as the filename submitted in the form data
+
+### delete_temp_file.lua
+
+The `delete_temp_file.lua` script is available at Sipi's `delete_temp_file` route.
+It is used only if Knora rejects a file value update request. It expects an
+HTTP `DELETE` request, with a filename as the last component of the URL.
+
+The JWT sent to this script must contain the key `knora-data`, whose value
+must be a JSON object containing:
+
+- `permission`: must be `DeleteTempFile`
+- `filename`: must be the same as the filename submitted in the URL
+
+## SipiResponderV2
+
+In Knora, the responder `SipiResponderV2` handles all communication with Sipi.
+It blocks while processing each request, to ensure that the number of
+concurrent requests to Sipi is not greater than
+`akka.actor.deployment./responderManager/sipiRouterV2.nr-of-instances`.
+If it encounters an error, it returns `SipiException`.
+
+## The Image File Upload Workflow
+
+1. The client uploads an image file to the `upload` route, which runs
+  `upload.lua`. The image is converted to JPEG 2000 and stored in Sipi's `tmp`
+  directory. In the response, the client receives the JPEG 2000's unique,
+  randomly generated filename.
+2. The client submits a JSON-LD request to a Knora route (`/v2/values` or `/v2/resources`)
+   to create or change a file value. The request includes Sipi's internal filename.
+3. During parsing of this JSON-LD request, a `StillImageFileValueContentV2`
+   is constructed to represent the file value. During the construction of this
+   object, a `GetImageMetadataRequestV2` is sent to `SipiResponderV2`, which
+   uses Sipi's built-in `knora.json` route to get the rest of the file's
+   metadata.
+4. A responder (`ResourcesResponderV2` or `ValuesResponderV2`) validates
+   the request and updates the triplestore. (If it is `ResourcesResponderV2`,
+   it asks `ValuesResponderV2` to generate SPARQL for the values.)
+5. The responder that did the update calls `ValueUtilV2.doSipiPostUpdate`.
+   If the triplestore update was successful, this method sends
+   `MoveTemporaryFileToPermanentStorageRequestV2` to `SipiResponderV2`, which
+   makes a request to Sipi's `store` route. Otherwise, the same method sends
+   `DeleteTemporaryFileRequestV2` to `SipiResponderV2`, which makes a request
+   to Sipi's `delete_temp_file` route.
+
+If the request to Knora cannot be parsed, the temporary file is not deleted
+immediately, but it will be deleted during the processing of a subsequent
+request by Sipi's `upload` route.
+
+If Sipi's `store` route fails, Knora returns the `SipiException` to the client.
+In this case, manual intervention may be necessary to restore consistency
+between Knora and Sipi.
+
+If Sipi's `delete_temp_file` route fails, the error is not returned to the client,
+because there is already a Knora error that needs to be returned to the client.
+In this case, the Sipi error is simply logged.

docs/src/paradox/05-internals/design/index.md

@@ -34,7 +34,6 @@ License along with Knora.  If not, see <http://www.gnu.org/licenses/>.
 - [Consistency Checking](consistency-checking.md)
 - [Authentication](authentication.md)
 - [Administration](administration.md)
-- [API v2](api-v2.md)
-- [Gravsearch](gravsearch.md)
+- [API v2](api-v2/index.md)
 
 @@@

docs/src/paradox/07-sipi/sipi-and-knora.md

@@ -19,6 +19,8 @@ License along with Knora.  If not, see <http://www.gnu.org/licenses/>.
 
 # Interaction Between Sipi and Knora
 
+TODO: reorganise this to make clear that it describes Knora API v1.
+
 ## General Remarks
 
 Knora and Sipi (Simple Image Presentation Interface) are two

knora-ontologies/knora-base.ttl

@@ -1781,7 +1781,7 @@
                            ] ,
                            [ rdf:type owl:Restriction ;
                              owl:onProperty :originalMimeType ;
-                             owl:maxCardinality "1"^^xsd:nonNegativeInteger
+                             owl:cardinality "1"^^xsd:nonNegativeInteger
                            ] ,
                            [ rdf:type owl:Restriction ;
                              owl:onProperty :internalFilename ;
@@ -1793,7 +1793,7 @@
                            ] ,
                            [ rdf:type owl:Restriction ;
                              owl:onProperty :originalFilename ;
-                             owl:maxCardinality "1"^^xsd:nonNegativeInteger
+                             owl:cardinality "1"^^xsd:nonNegativeInteger
                            ] .
 
 

sipi/config/sipi.init-knora.lua

@@ -47,8 +47,8 @@ function pre_flight(prefix, identifier, cookie)
     end
 
     if prefix == "tmp" then
-        -- always deny access to tmp folder
-        return 'deny'
+        -- always allow access to tmp folder
+        return 'allow', filepath
     end
 
 

sipi/config/sipi.knora-docker-config.lua

@@ -124,7 +124,8 @@ sipi = {
     knora_port = '3333',
 
     --
-    -- loglevel, one of "EMERGENCY", "ALERT", "CRITICAL", "ERROR", "WARNING", "NOTICE", "INFORMATIONAL", "DEBUG"
+    -- loglevel, one of "DEBUG", "INFO", "NOTICE", "WARNING", "ERR",
+    -- "CRIT", "ALERT", "EMERG"
     --
     loglevel = "DEBUG"
 
@@ -173,6 +174,21 @@ routes = {
         method = 'POST',
         route = '/Knora_logout',
         script = 'Knora_logout.lua'
+    },
+    {
+        method = 'POST',
+        route = '/upload',
+        script = 'upload.lua'
+    },
+    {
+        method = 'POST',
+        route = '/store',
+        script = 'store.lua'
+    },
+    {
+        method = 'DELETE',
+        route = '/delete_temp_file',
+        script = 'delete_temp_file.lua'
     }
 
 }