Impl. file upload endpoint #470

pvannierop · 2024-06-03T13:46:41Z

This PR will add a new endpoint that allows users to upload files to a storage location. This upload functionality is meant for data that is too large to be submitted tot the Kafka queue. After successful upload the upload endpoint returns the path of the newly created file. The application can then submit a new message to the Kafka queue that mentions this path.

For now, upload is only implemented for S3 storage.

The upload endpoint is disabled by default. It can be activated with the radar.file-upload.enabled application property.

Remarks

This PR add a new minio dependency. This allows file storage on Minio and on AWS S3.

yatharthranjan

Looks good. Thanks for adding this. Please find some comments.

yatharthranjan · 2024-06-18T16:05:16Z

src/main/java/org/radarbase/appserver/controller/UploadController.java

+ @PostMapping(
+ "/" + PathsUtil.PROJECT_PATH + "/" + PathsUtil.PROJECT_ID_CONSTANT +
+ "/" + PathsUtil.USER_PATH + "/" + PathsUtil.SUBJECT_ID_CONSTANT +
+ "/" + PathsUtil.TOPIC_PATH + "/" + PathsUtil.TOPIC_ID_CONSTANT +


Can you add "files" to the path too like .../files/topics to denote it is for file uploads

yatharthranjan · 2024-06-18T16:08:09Z

src/main/java/org/radarbase/appserver/controller/UploadController.java

+ "/" + PathsUtil.TOPIC_PATH + "/" + PathsUtil.TOPIC_ID_CONSTANT +
+ "/upload")
+ public ResponseEntity<?> subjectFileUpload(
+ @RequestParam("file") MultipartFile file,


Why not accept a filename parameter too, which by default can be the random name you generate below.

This is a security risk. The uploading party better not determine the filename if not needed. See OWASP Unrestricted file upload. I decided to assign server-side randomized filename for simplicity so that I did not have to do filename sanitization.

ok, good point. Can you then add the timestamp to the generated file name when like {timestamp}_{randomUUID}.{ext}? And can also go in a folder per day like 022024)

I added this feature. A timestamp is always added in front of the random filename as per your suggestion. You can opt-in for collection per day via a new property storage.s3.path.collect-per-day.

The functionality is handled by a new StoragePath.Builder class. Here is the Javadoc for reference:

* Represents path on Object Storage for uploaded files. * <p> * The storage path is constructed to include required arguments (projectId, subjectId, topicId and filename) * and optional arguments (prefix, collectPerDay, folderTimestampPattern, fileTimestampPattern). The path will follow * the format: prefix/projectId/subjectId/topicId/[day folder]/timestamp_filename.extension. * The day folder is included if collectPerDay is set to true. File extensions are converted to lowercase. * </p> * * <p> * Usage example: * </p> * <pre> * StoragePath path = StoragePath.builder() * .prefix("uploads") * .projectId("project1") * .subjectId("subjectA") * .topicId("topicX") * .collectPerDay(true) * .filename("example.txt") * .build(); * * System.out.println(path.getFullPath(); * 'uploads/project1/subjectA/topicX/20220101/20220101_example.txt' * * System.out.println(path.getPathInTopicDir() * '20220101/20220101_example.txt' * </pre>

yatharthranjan · 2024-06-26T11:31:17Z

src/main/java/org/radarbase/appserver/service/storage/S3StorageService.java

+
+ public String store(MultipartFile file, String projectId, String subjectId, String topicId) {
+ Assert.notNull(file, "File must not be null");
+ Assert.notEmpty(new String[]{projectId, subjectId, topicId}, "Project, subject and topic IDs must not be empty");


It is probably better to use specific exceptions with HTTP codes and handle them with Exception handler.

I do not quite understand what you mean here. In the Service layer it does ont make sense to me to throw HTTP-level exceptions. The Assert in the service throws an IllegalArgumentException which is fine I think. Or ar e you not proposing to send HTTP-level exceptions from the service?

These will be relayed to the app as an HTTP response, so it makes sense to have a specific error message in case they are useful. You can create new exceptions like InvalidFileDetailsException extends IllegalArgumentException and annotate with @ResponseStatus(HttpStatus.XYZ). This can then be converted to a proper response for the app in Exception handler. Alternative is to catch the assertion error in the controller and raise these exceptions there, but I prefer doing it in the service (or where the exception is raised) as you can include more context. Essentially, you are not sending HTTP responses from the service but raising usual exceptions and configuring spring to convert to responses (this avoids handling them manually and keeps all the exception handling code neatly in one place).

Ah ok, I think I understand. Could you please review my last commit to see whether this is what you intended?

yatharthranjan · 2024-06-26T11:33:47Z

src/main/java/org/radarbase/appserver/controller/UploadController.java

+ @Authorized(
+ permission = AuthPermissions.UPDATE,
+ entity = AuthEntities.SUBJECT,
+ permissionOn = PermissionOn.SUBJECT


Do you want to use CREATE MEASUREMENT permissions?

Ooops you are right! Updated...

mpgxvii

Thanks for this. Would it be possible to fix the PMD errors as well?

pvannierop · 2024-07-03T09:08:25Z

@mpgxvii @yatharthranjan I have fixed the PMD linting errors. It forced me to remove duplication of string constants to member variables. Normally a good thing, but for tests it makes the tests less easy to write and interpret. Can we not better suppress linting checks for test classes, or at least suppress the The String literal "project1" appears 9 times in this file; ... errors for tests?

And on a side note. I think that it would be an improvement to separate style checks and unit/integration tests in separate GitHub Action workflows.

Currently only implemented for S3 storage type.

pvannierop · 2024-07-10T08:01:02Z

@mpgxvii @yatharthranjan I have added another commit that fixes a CORS issue. The Location header was not exposed to the aRMT app because of CORS protection. In the commit the CORS exception is added to the upload endpoint method. I tried to add this to the MultiHttpSecurityConfig class, but this did not work.

mpgxvii · 2024-07-12T13:07:02Z

src/main/java/org/radarbase/appserver/controller/UploadController.java

+ origins = "*",
+ allowedHeaders = "*",
+ exposedHeaders = "Location", // needed to get the URI of the uploaded file in aRMT
+ methods = { RequestMethod.POST }


Since the rest are already specified in the MultiHttpSecurityConfig, should we only use exposedHeaders here?

No, we tried this and it dit not work. Leaving out origins caused the annotation to not work. It is almost as if the MultiHttpSecurityConfig does not apply to the endpoints. We also tried adding .exposedHeaders(...) to MultiHttpSecurityConfig but this also did not work. The annotation in its current form above the upload method was needed.

mpgxvii · 2024-07-12T13:46:42Z

src/main/java/org/radarbase/appserver/service/storage/S3StorageService.java

+ public String store(MultipartFile file, String projectId, String subjectId, String topicId) {
+ if (
+ file == null || projectId == null || subjectId == null || topicId == null
+ || file.isEmpty()|| projectId.isEmpty() || subjectId.isEmpty() || topicId.isEmpty()) {


If the check for blank values is being done in StoragePath, maybe we don't need the check for empty here anymore?

I think that every component should enforce its own contract/API. But this is my opinion.

src/main/java/org/radarbase/appserver/service/storage/StoragePath.java

mpgxvii

LGTM, thanks for the updates!

yatharthranjan

LGTM, thanks for the substantial effort.

Just a minor comment on adding @ResponseStatus to the exceptions.

yatharthranjan · 2024-07-15T11:26:40Z

src/main/java/org/radarbase/appserver/exception/InvalidFileDetailsException.java

@@ -0,0 +1,13 @@
+package org.radarbase.appserver.exception;
+
+public class InvalidFileDetailsException extends IllegalArgumentException {


Can you map these to HTTP codes using @ResponseStatus(HttpStatus.XYZ)? Perhaps Expectation failed?

Added! Can you check this please. I have never used this mechanism.

yes looks good, thanks

yatharthranjan · 2024-07-15T11:26:44Z

src/main/java/org/radarbase/appserver/exception/InvalidPathDetailsException.java

@@ -0,0 +1,13 @@
+package org.radarbase.appserver.exception;
+
+public class InvalidPathDetailsException extends IllegalArgumentException {


Can you map these to HTTP codes using @ResponseStatus(HttpStatus.XYZ)? Perhaps Expectation failed?

Added! Can you check this please. I have never used this mechanism.

I don't see the code being added, can you double check and add @ResponseStatus(HttpStatus.EXPECTATION_FAILED) here. Thanks.

yatharthranjan · 2024-07-15T11:41:49Z

src/main/java/org/radarbase/appserver/exception/FileStorageException.java

@@ -0,0 +1,13 @@
+package org.radarbase.appserver.exception;
+
+public class FileStorageException extends RuntimeException {


Can you map these to HTTP codes using @ResponseStatus(HttpStatus.XYZ)? Perhaps http code 500 in this case.

Added! Can you check this please. I have never used this mechanism.

yes looks good, thanks

yatharthranjan

Thanks for the changes, please find some comments below.

yatharthranjan · 2024-07-16T10:16:44Z

src/main/java/org/radarbase/appserver/exception/EmailMessageTransmitException.java

+import org.springframework.http.HttpStatus;
+import org.springframework.web.bind.annotation.ResponseStatus;
+
+@ResponseStatus(HttpStatus.INTERNAL_SERVER_ERROR)
 public class EmailMessageTransmitException extends MessageTransmitException {


Are the MessageTransmitException (s) ever unhandled? You don't need this response status here if they are caught and handled. Moreover, these are thrown when a scheduled job is run, not in response to an HTTP request, hence, no need to add these here.

yatharthranjan · 2024-07-16T10:17:04Z

src/main/java/org/radarbase/appserver/exception/FcmMessageTransmitException.java

+import org.springframework.http.HttpStatus;
+import org.springframework.web.bind.annotation.ResponseStatus;
+
+@ResponseStatus(HttpStatus.INTERNAL_SERVER_ERROR)


similarly not needed

yatharthranjan · 2024-07-16T10:17:16Z

src/main/java/org/radarbase/appserver/exception/FileStorageException.java

@@ -0,0 +1,13 @@
+package org.radarbase.appserver.exception;
+
+public class FileStorageException extends RuntimeException {


yes looks good, thanks

yatharthranjan · 2024-07-16T10:17:27Z

src/main/java/org/radarbase/appserver/exception/InvalidFileDetailsException.java

@@ -0,0 +1,13 @@
+package org.radarbase.appserver.exception;
+
+public class InvalidFileDetailsException extends IllegalArgumentException {


yes looks good, thanks

yatharthranjan · 2024-07-16T10:18:10Z

src/main/java/org/radarbase/appserver/exception/InvalidPathDetailsException.java

@@ -0,0 +1,13 @@
+package org.radarbase.appserver.exception;
+
+public class InvalidPathDetailsException extends IllegalArgumentException {


I don't see the code being added, can you double check and add @ResponseStatus(HttpStatus.EXPECTATION_FAILED) here. Thanks.

yatharthranjan · 2024-07-16T10:18:36Z

src/main/java/org/radarbase/appserver/exception/handler/ResponseEntityExceptionHandler.java

+ @ExceptionHandler(MessageTransmitException.class)
+ public final ResponseEntity<ErrorDetails> handleMessageTransmitException(Exception ex, WebRequest request) {
+ return handleEntityWithCause(ex, request);
+ }


can be removed as don't need a response for these exceptions

yatharthranjan

LGTM, thanks!

mpgxvii · 2024-07-23T16:39:16Z

@pvannierop Is this ok to merge?

Init at application start interfered with the deployment of RADAR-base where S3 services are installed after Appserver.

pvannierop requested review from yatharthranjan and mpgxvii June 3, 2024 13:46

pvannierop self-assigned this Jun 3, 2024

pvannierop force-pushed the file-upload-endpoint branch from f41cac6 to e22b66d Compare June 3, 2024 13:56

pvannierop changed the base branch from master to dev June 3, 2024 13:56

pvannierop force-pushed the file-upload-endpoint branch 2 times, most recently from 4ffc5fb to 750b7f5 Compare June 3, 2024 14:03

pvannierop marked this pull request as draft June 3, 2024 14:08

pvannierop force-pushed the file-upload-endpoint branch 7 times, most recently from 8532eeb to af7902c Compare June 10, 2024 13:52

pvannierop force-pushed the file-upload-endpoint branch from af7902c to 1e60613 Compare June 18, 2024 09:45

yatharthranjan reviewed Jun 26, 2024

View reviewed changes

mpgxvii reviewed Jun 26, 2024

View reviewed changes

pvannierop requested a review from yatharthranjan July 3, 2024 07:41

pvannierop force-pushed the file-upload-endpoint branch 3 times, most recently from 75fb66e to b81fe7e Compare July 3, 2024 08:43

pvannierop marked this pull request as ready for review July 3, 2024 09:08

pvannierop force-pushed the file-upload-endpoint branch 3 times, most recently from 9494b04 to b012df9 Compare July 3, 2024 09:54

yatharthranjan mentioned this pull request Jul 3, 2024

Add a file upload endpoint RADAR-base/RADAR-Gateway#108

Open

pvannierop force-pushed the file-upload-endpoint branch from 8eba6f8 to 634e333 Compare July 4, 2024 06:42

Impl. file upload endpoint

06b0c46

Currently only implemented for S3 storage type.

pvannierop added 8 commits July 10, 2024 09:56

Add '/files' to the REST path for file upload

9eb55a8

Change to CREATE MEASUREMENT permission for upload endpoint

aabf53a

Implement per-day folder and timestamp as part of filename

b2cb1b8

Fix PMD tests

1e4523e

Return path to bucket root in HTTP response

9cb5ee5

Throw custom exceptions handled by ResponseEntityExceptionHandler

7ec460c

Add 'radar.' prefix for storage properties

d1a9ab9

Add Location to exposed headers in CORS

f1523af

pvannierop force-pushed the file-upload-endpoint branch from 92a8768 to f1523af Compare July 10, 2024 07:56

mpgxvii reviewed Jul 12, 2024

View reviewed changes

src/main/java/org/radarbase/appserver/service/storage/StoragePath.java Outdated Show resolved Hide resolved

Refactor StoragePath

28132f4

pvannierop requested a review from mpgxvii July 15, 2024 11:01

mpgxvii approved these changes Jul 15, 2024

View reviewed changes

yatharthranjan approved these changes Jul 15, 2024

View reviewed changes

yatharthranjan reviewed Jul 16, 2024

View reviewed changes

Add @ResponseStatus on file storage exceptions

a679464

pvannierop force-pushed the file-upload-endpoint branch from 4984fa8 to a679464 Compare July 16, 2024 12:13

yatharthranjan approved these changes Jul 17, 2024

View reviewed changes

Add lazy init of S3 client

d559e0f

Init at application start interfered with the deployment of RADAR-base where S3 services are installed after Appserver.

mpgxvii merged commit 9e45eb7 into dev Jul 30, 2024
4 checks passed

mpgxvii deleted the file-upload-endpoint branch July 30, 2024 13:23

		@@ -0,0 +1,13 @@
		package org.radarbase.appserver.exception;

		public class InvalidFileDetailsException extends IllegalArgumentException {

		@@ -0,0 +1,13 @@
		package org.radarbase.appserver.exception;

		public class InvalidPathDetailsException extends IllegalArgumentException {

		@@ -0,0 +1,13 @@
		package org.radarbase.appserver.exception;

		public class FileStorageException extends RuntimeException {

Impl. file upload endpoint #470

Impl. file upload endpoint #470

Conversation

pvannierop commented Jun 3, 2024 • edited Loading

Remarks

yatharthranjan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pvannierop Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pvannierop Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mpgxvii left a comment

Choose a reason for hiding this comment

pvannierop commented Jul 3, 2024 • edited Loading

pvannierop commented Jul 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pvannierop Jul 15, 2024 • edited Loading

Choose a reason for hiding this comment

mpgxvii left a comment

Choose a reason for hiding this comment

yatharthranjan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yatharthranjan left a comment

Choose a reason for hiding this comment

yatharthranjan Jul 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yatharthranjan left a comment

Choose a reason for hiding this comment

mpgxvii commented Jul 23, 2024

pvannierop commented Jun 3, 2024 •

edited

Loading

pvannierop Jul 2, 2024 •

edited

Loading

pvannierop Jul 3, 2024 •

edited

Loading

pvannierop commented Jul 3, 2024 •

edited

Loading

pvannierop Jul 15, 2024 •

edited

Loading

yatharthranjan Jul 16, 2024 •

edited

Loading