diff --git a/NEWS.md b/NEWS.md index f421bfc72..f9bcc179b 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,9 +1,65 @@ +## Unreleased v3.4.0 + +## 2024-11-08 v3.3.1 + +[Full Changelog](https://github.com/folio-org/mod-data-export-worker/compare/v3.3.0...v3.3.1) + +### Technical tasks +* [MODEXPW-530](https://folio-org.atlassian.net/browse/MODEXPW-530) Upgrade commons-io for Ramsons +* Improvements and dependencies updating. + +## 2024-10-30 v3.3.0 + +[Full Changelog](https://github.com/folio-org/mod-data-export-worker/compare/v3.2.4...v3.3.0) + +### Technical tasks +* [MODEXPW-521](https://folio-org.atlassian.net/browse/MODEXPW-521) Verify notes permissions +* [MODEXPW-519](https://folio-org.atlassian.net/browse/MODEXPW-519) Verify eholdings permissions +* [MODEXPW-513](https://folio-org.atlassian.net/browse/MODEXPW-513) Verify consortium-search permissions +* [MODEXPW-512](https://folio-org.atlassian.net/browse/MODEXPW-512) Rename sourceRecords permission +* [MODEXPW-511](https://folio-org.atlassian.net/browse/MODEXPW-511) API version update +* [MODEXPW-509](https://folio-org.atlassian.net/browse/MODEXPW-509) Support Eureka permissions model for bulk operations (read operations) +* [MODEXPW-505](https://folio-org.atlassian.net/browse/MODEXPW-505) API version update +* [MODEXPW-503](https://folio-org.atlassian.net/browse/MODEXPW-503) Bulk-Ops: performance improvement (data-export-worker part) +* [MODEXPW-498](https://folio-org.atlassian.net/browse/MODEXPW-498) HoldingsRecordsSource schema alignment +* [MODEXPW-489](https://folio-org.atlassian.net/browse/MODEXPW-489) Remove unused updating functionality +* [MODEXPW-488](https://folio-org.atlassian.net/browse/MODEXPW-488) Central tenant view permissions handling +* [MODEXPW-487](https://folio-org.atlassian.net/browse/MODEXPW-487) S3 Env Vars optimisation +* [MODEXPW-483](https://folio-org.atlassian.net/browse/MODEXPW-483) User schema updating - new field preferredEmailCommunication +* [MODEXPW-479](https://folio-org.atlassian.net/browse/MODEXPW-479) API version update +* [MODEXPW-478](https://folio-org.atlassian.net/browse/MODEXPW-478) Add notes field to instance csv file +* [MODEXPW-477](https://folio-org.atlassian.net/browse/MODEXPW-477) Items and holdings tenant populating +* [MODEXPW-476](https://folio-org.atlassian.net/browse/MODEXPW-476) Extend entities schemas to support tenant information +* [MODEXPW-475](https://folio-org.atlassian.net/browse/MODEXPW-475) Increase multipart upload consumption size +* [MODEXPW-470](https://folio-org.atlassian.net/browse/MODEXPW-470) .mrc-file creation +* [MODEXPW-345](https://folio-org.atlassian.net/browse/MODEXPW-345) Refreshing mechanism for presigned url (Bursar) +* [FOLIO-4087](https://folio-org.atlassian.net/browse/FOLIO-4087) RMB & Spring upgrades (all modules) + +### Stories +* [MODEXPW-516](https://folio-org.atlassian.net/browse/MODEXPW-516) Update module permissions in the ModuleDescriptor +* [MODEXPW-515](https://folio-org.atlassian.net/browse/MODEXPW-515) Include Tentant in columns selection +* [MODEXPW-504](https://folio-org.atlassian.net/browse/MODEXPW-504) Include tenantId in Item's and Holdings' Notes names in ECS +* [MODEXPW-465](https://folio-org.atlassian.net/browse/MODEXPW-465) Separate Instance Notes by Note Type +* [MODEXPW-460](https://folio-org.atlassian.net/browse/MODEXPW-460) Holdings record's column names cleanup +* [MODEXPW-455](https://folio-org.atlassian.net/browse/MODEXPW-455) Item record's column names cleanup +* [MODEXPW-396](https://folio-org.atlassian.net/browse/MODEXPW-396) Allow additional item status updates + +### Bug fixes +* [MODEXPW-520](https://folio-org.atlassian.net/browse/MODEXPW-520) Bulk edit allows editing shadow users +* [MODEXPW-507](https://folio-org.atlassian.net/browse/MODEXPW-507) 500 Internal Server Error when download bulk edit job file from Export manager +* [MODEXPW-495](https://folio-org.atlassian.net/browse/MODEXPW-495) ECS | Incorrect "totalRecords" value when search Items by Former identifier on Member tenant +* [MODEXPW-493](https://folio-org.atlassian.net/browse/MODEXPW-493) Outdated preview of matched records in case of remove (add) affiliation and upload the same file with Holdings, Items identifiers in Central tenant + +### Tech debts +* [MODEXPW-499](https://folio-org.atlassian.net/browse/MODEXPW-499) Replace deprecated KafkaProperties methods + ## 2024-03-23 v3.2.1 [Full Changelog](https://github.com/folio-org/mod-data-export-worker/compare/v3.2.0...v3.2.1) ### Technical tasks -ModuleDescriptor: ERM interface version upgrading +* ModuleDescriptor: ERM interface version upgrading +* [MODEXPW-516](https://folio-org.atlassian.net/browse/MODEXPW-516) Upgrade "holdings-storage" to 8.0 ## 2024-03-19 v3.2.0 diff --git a/README.md b/README.md index 39f8d6e76..025376156 100644 --- a/README.md +++ b/README.md @@ -32,34 +32,35 @@ AWS container: memory - 3072, memory (soft limit) - 2600, cpu - 1024. This module uses separate storage of temporary (local) files for its work. These files are necessary for processing bulk-edit business flows. Any S3-compatible storage (AWS S3, Minio Server) supported by the Minio Client can be used as such storage. Thus, in addition to the AWS configuration (AWS_URL, AWS_REGION, AWS_BUCKET, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) of the permanent storage, -one need to configure the environment settings for temporary files storage (LOCAL_FS_URL, LOCAL_FS_REGION, LOCAL_FS_BUCKET, LOCAL_FS_ACCESS_KEY_ID, LOCAL_FS_SECRET_ACCESS_KEY). -Typically, these options must specify a separate storage. It should be noted that a single storage can also be used for the results of processing and storing temporary files, -but in this case it is necessary to use different buckets. -It is also necessary to specify variable LOCAL_FS_COMPOSE_WITH_AWS_SDK to determine if AWS S3 is used as files storage. By default this variable is `false` and means that MinIO server is used as files storage. +one need to configure the environment settings for s3 subpathes (S3_SUB_PATH, S3_LOCAL_SUB_PATH). +Typically, these options must specify a separate pathes. +It is also necessary to specify variable S3_IS_AWS to determine if AWS S3 is used as files storage. By default this variable is `false` and means that MinIO server is used as files storage. This value should be `true` if AWS S3 is used as storage. -| Name | Default value | Description | -|:--------------------------------------------------|:-----------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| KAFKA_HOST | localhost | Kafka broker hostname | -| KAFKA_PORT | 9092 | Kafka broker port | -| KAFKA_CONSUMER_POLL_INTERVAL | 3600000 | Max interval before next poll. If long record processing is in place and interval exceeded then consumer will be kicked out of the group and another consumer will start processing the same message. | -| ENV | folio | Environment name | -| AWS_URL | http://127.0.0.1:9000/ | AWS url | -| AWS_REGION | - | AWS region | -| AWS_BUCKET | - | AWS bucket | -| AWS_ACCESS_KEY_ID | - | AWS access key | -| AWS_SECRET_ACCESS_KEY | - | AWS secret key | -| LOCAL_FS_URL | http://127.0.0.1:9000/ | S3-compatible local files storage url | -| LOCAL_FS_REGION | - | S3-compatible local files storage region | -| LOCAL_FS_BUCKET | - | S3-compatible local files storage bucket | -| LOCAL_FS_ACCESS_KEY_ID | - | S3-compatible local files storage access key | -| LOCAL_FS_SECRET_ACCESS_KEY | - | S3-compatible local files storage secret key | -| URL_EXPIRATION_TIME | 604800 | Presigned url expiration time (in seconds) | -| DATA_EXPORT_JOB_UPDATE_TOPIC_PARTITIONS | 50 | Number of partitions for topic | -| KAFKA_CONCURRENCY_LEVEL | 30 | Concurrency level of kafka listener | -| LOCAL_FS_COMPOSE_WITH_AWS_SDK | false | Specify if AWS S3 is used as local files storage | -| E_HOLDINGS_BATCH_JOB_CHUNK_SIZE | 100 | Specify chunk size for eHoldings export job which will be used to query data from kb-ebsco, write to database, read from database and write to file | -| E_HOLDINGS_BATCH_KB_EBSCO_CHUNK_SIZE | 100 | Amount to retrieve per request to mod-kb-ebsco-java (100 is max acceptable value) | -| AUTHORITY_CONTROL_BATCH_JOB_CHUNK_SIZE | 100 | Specify chunk size for authority control export job which will be used to query data from entities-links, and write to file | -| AUTHORITY_CONTROL_BATCH_ENTITIES_LINKS_CHUNK_SIZE | 100 | Amount to retrieve per request to mod-entities-links | -| MAX_UPLOADED_FILE_SIZE | 40MB | Specifies multipart upload file size | +| Name | Default value | Description | +|:---------------------------------------------------|:------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| KAFKA_HOST | localhost | Kafka broker hostname | +| KAFKA_PORT | 9092 | Kafka broker port | +| KAFKA_CONSUMER_POLL_INTERVAL | 3600000 | Max interval before next poll. If long record processing is in place and interval exceeded then consumer will be kicked out of the group and another consumer will start processing the same message. | +| ENV | folio | Environment name | +| S3_URL | http://127.0.0.1:9000/ | AWS url | +| S3_REGION | - | AWS region | +| S3_BUCKET | - | AWS bucket | +| S3_ACCESS_KEY_ID | - | AWS access key | +| S3_SECRET_ACCESS_KEY | - | AWS secret key | +| S3_SUB_PATH | mod-data-export-worker/remote | S3 subpath for files storage | +| S3_LOCAL_SUB_PATH | mod-data-export-worker/local | S3 subpath for local files storage | +| S3_IS_AWS | false | Specify if AWS S3 is used as files storage | +| URL_EXPIRATION_TIME | 604800 | Presigned url expiration time (in seconds) | +| DATA_EXPORT_JOB_UPDATE_TOPIC_PARTITIONS | 50 | Number of partitions for topic | +| KAFKA_CONCURRENCY_LEVEL | 30 | Concurrency level of kafka listener | +| E_HOLDINGS_BATCH_JOB_CHUNK_SIZE | 100 | Specify chunk size for eHoldings export job which will be used to query data from kb-ebsco, write to database, read from database and write to file | +| E_HOLDINGS_BATCH_KB_EBSCO_CHUNK_SIZE | 100 | Amount to retrieve per request to mod-kb-ebsco-java (100 is max acceptable value) | +| AUTHORITY_CONTROL_BATCH_JOB_CHUNK_SIZE | 100 | Specify chunk size for authority control export job which will be used to query data from entities-links, and write to file | +| AUTHORITY_CONTROL_BATCH_ENTITIES_LINKS_CHUNK_SIZE | 100 | Amount to retrieve per request to mod-entities-links | +| MAX_UPLOADED_FILE_SIZE | 40MB | Specifies multipart upload file size | +| PLATFORM | okapi | Specifies if okapi or eureka platform | +| CHUNKS | 100 | Number of items being passed to write at once | +| CORE_POOL_SIZE | 10 | Maximum number of threads being created for each task before the queue is utilized | +| MAX_POOL_SIZE | 10 | Maximum number of threads that can be created after the queue is full and before rejecting the new tasks | +| BUCKET_SIZE | 50 | Size of the bucket used in partitioning parameters | diff --git a/descriptors/ModuleDescriptor-template.json b/descriptors/ModuleDescriptor-template.json index ff9174305..9f4095aa9 100644 --- a/descriptors/ModuleDescriptor-template.json +++ b/descriptors/ModuleDescriptor-template.json @@ -8,7 +8,7 @@ }, { "id": "inventory", - "version": "13.0" + "version": "13.0 14.0" }, { "id": "configuration", @@ -18,17 +18,13 @@ "id": "feesfines", "version": "18.0 19.0" }, - { - "id": "inventory", - "version": "13.0" - }, { "id": "holdings-storage", - "version": "7.0" + "version": "7.0 8.0" }, { "id": "instance-storage", - "version": "10.0" + "version": "10.0 11.0" }, { "id": "call-number-types", @@ -46,13 +42,9 @@ "id": "item-damaged-statuses", "version": "1.0" }, - { - "id": "data-export-spring", - "version": "2.0" - }, { "id": "electronic-access-relationships", - "version": "1.0" + "version": "1.1" }, { "id": "instance-authority-links-statistics", @@ -88,7 +80,7 @@ }, { "id": "eholdings", - "version": "3.1" + "version": "4.0" }, { "id": "loan-types", @@ -96,7 +88,7 @@ }, { "id": "locations", - "version": "3.0" + "version": "3.1" }, { "id": "material-types", @@ -104,7 +96,7 @@ }, { "id": "notes", - "version": "3.0" + "version": "4.0" }, { "id": "orders-storage.purchase-orders", @@ -124,7 +116,7 @@ }, { "id": "search", - "version": "1.0" + "version": "1.3" }, { "id": "service-points", @@ -135,18 +127,68 @@ "version": "1.0" }, { - "id": "service-points", - "version": "3.3" + "id": "instance-note-types", + "version": "1.0" }, { - "id": "instance-note-types", + "id": "permissions", + "version": "5.7" + }, + { + "id": "user-tenants", "version": "1.0" + }, + { + "id": "instance-formats", + "version": "2.0" + }, + { + "id": "instance-types", + "version": "2.0" + }, + { + "id": "modes-of-issuance", + "version": "1.1" + }, + { + "id": "nature-of-content-terms", + "version": "1.0" + }, + { + "id": "consortium-search", + "version": "2.1" + }, + { + "id": "source-storage-source-records", + "version": "3.2" + }, + { + "id": "circulation-logs", + "version": "1.2" + } + ], + "optional": [ + { + "id": "data-export-spring", + "version": "2.0" + }, + { + "id": "users-keycloak", + "version": "1.0" + }, + { + "id": "permissions-users", + "version": "1.0" + }, + { + "id": "consortia", + "version": "1.1" } ], "provides": [ { "id": "bulk-edit", - "version": "3.0", + "version": "4.1", "handlers": [ { "methods": [ "POST" ], @@ -204,79 +246,14 @@ "inventory-storage.nature-of-content-terms.item.get", "inventory-storage.instance-formats.item.get", "inventory-storage.instance-note-types.item.get", - "source-storage.sourceRecords.get", + "source-storage.source-records.collection.get", "user-tenants.collection.get", "consortium-search.holdings.collection.get", - "consortium-search.items.collection.get" - ] - }, - { - "methods": [ "POST" ], - "pathPattern": "/bulk-edit/{id}/item-content-update/upload", - "permissionsRequired": [ "bulk-edit.items-content-update.collection.post" ], - "modulePermissions": [ - "inventory-storage.call-number-types.item.get", - "inventory-storage.call-number-types.collection.get", - "inventory-storage.item-damaged-statuses.item.get", - "inventory-storage.item-damaged-statuses.collection.get", - "inventory-storage.electronic-access-relationships.item.get", - "inventory-storage.electronic-access-relationships.collection.get", - "inventory-storage.item-note-types.item.get", - "inventory-storage.item-note-types.collection.get", - "inventory-storage.service-points.item.get", - "inventory-storage.service-points.collection.get", - "inventory-storage.statistical-codes.item.get", - "inventory-storage.statistical-codes.collection.get", - "inventory-storage.material-types.item.get", - "inventory-storage.material-types.collection.get", - "inventory-storage.loan-types.item.get", - "inventory-storage.loan-types.collection.get", - "inventory-storage.locations.item.get", - "inventory-storage.locations.collection.get", - "inventory-storage.instances.item.get", - "inventory-storage.instances.collection.get", - "inventory-storage.holdings.item.get", - "inventory-storage.holdings.collection.get", - "configuration.entries.collection.get" - ] - }, - { - "methods": [ "POST" ], - "pathPattern": "/bulk-edit/{id}/user-content-update/upload", - "permissionsRequired": [ "bulk-edit.users-content-update.collection.post" ], - "modulePermissions": [ - "users.collection.get", - "users.item.get", - "users.item.put", - "usergroups.item.get", - "addresstypes.item.get", - "proxiesfor.item.get", - "departments.item.get", - "usergroups.collection.get", - "addresstypes.collection.get" - ] - }, - { - "methods": [ "POST" ], - "pathPattern": "/bulk-edit/{id}/holdings-content-update/upload", - "permissionsRequired": [ "bulk-edit.holdings-content-update.collection.post" ], - "modulePermissions": [ - "inventory-storage.instances.item.get", - "inventory-storage.holdings-types.collection.get", - "inventory-storage.holdings-types.item.get", - "inventory-storage.locations.collection.get", - "inventory-storage.locations.item.get", - "inventory-storage.call-number-types.collection.get", - "inventory-storage.call-number-types.item.get", - "inventory-storage.holdings-note-types.collection.get", - "inventory-storage.holdings-note-types.item.get", - "inventory-storage.ill-policies.collection.get", - "inventory-storage.ill-policies.item.get", - "inventory-storage.holdings-sources.collection.get", - "inventory-storage.holdings-sources.item.get", - "inventory-storage.statistical-codes.collection.get", - "inventory-storage.statistical-codes.item.get", - "inventory-storage.electronic-access-relationships.collection.get" + "consortium-search.holdings.batch.collection.get", + "consortium-search.items.batch.collection.get", + "consortium-search.items.collection.get", + "bulk-edit.permissions-self-check.get", + "consortia.user-tenants.collection.get" ] }, { @@ -328,81 +305,8 @@ "inventory-storage.ill-policies.item.get", "inventory-storage.holdings-sources.collection.get", "inventory-storage.holdings-sources.item.get", - "inventory-storage.holdings.item.put" - ] - }, - { - "methods": [ - "POST" - ], - "pathPattern": "/bulk-edit/{jobId}/roll-back", - "permissionsRequired": [ - "bulk-edit.roll-back.item.post" - ], - "modulePermissions": [ - "users.collection.get", - "users.item.get", - "users.item.put", - "usergroups.item.get", - "addresstypes.item.get", - "proxiesfor.item.get", - "departments.item.get", - "usergroups.collection.get", - "addresstypes.collection.get", - "data-export.job.item.get" - ] - }, - { - "methods": [ "GET" ], - "pathPattern": "/bulk-edit/{jobId}/preview/users", - "permissionsRequired": [ "bulk-edit.preview-users.collection.get" ], - "modulePermissions": [ - "users.collection.get", - "addresstypes.collection.get", - "usergroups.collection.get" - ] - }, - { - "methods": [ "GET" ], - "pathPattern": "/bulk-edit/{jobId}/preview/items", - "permissionsRequired": [ "bulk-edit.preview-items.collection.get" ], - "modulePermissions": [ - "inventory.items.collection.get", - "inventory-storage.material-types.collection.get", - "inventory-storage.statistical-codes.collection.get", - "inventory-storage.item-note-types.collection.get", - "inventory-storage.call-number-types.collection.get", - "inventory-storage.electronic-access-relationships.collection.get", - "inventory-storage.service-points.collection.get", - "users.collection.get", - "inventory-storage.locations.collection.get", - "inventory-storage.loan-types.collection.get", - "inventory-storage.item-damaged-statuses.collection.get", - "inventory-storage.instances.collection.get", - "inventory-storage.holdings.collection.get", - "configuration.entries.collection.get" - ] - }, - { - "methods": [ "GET" ], - "pathPattern": "/bulk-edit/{jobId}/preview/holdings", - "permissionsRequired": [ "bulk-edit.preview-holdings.collection.get" ], - "modulePermissions": [ - "inventory-storage.holdings.collection.get", - "inventory-storage.holdings-types.collection.get", - "inventory-storage.holdings-types.item.get", - "inventory-storage.locations.collection.get", - "inventory-storage.locations.item.get", - "inventory-storage.call-number-types.collection.get", - "inventory-storage.call-number-types.item.get", - "inventory-storage.ill-policies.collection.get", - "inventory-storage.ill-policies.item.get", - "inventory-storage.holdings-sources.collection.get", - "inventory-storage.instances.item.get", - "inventory-storage.statistical-codes.collection.get", - "inventory-storage.statistical-codes.item.get", - "inventory-storage.electronic-access-relationships.collection.get", - "inventory-storage.holdings-note-types.collection.get" + "inventory-storage.holdings.item.put", + "bulk-edit.permissions-self-check.get" ] }, { @@ -413,21 +317,12 @@ }, { "methods": [ "GET" ], - "pathPattern": "/bulk-edit/{jobId}/preview/updated-items/download", - "permissionsRequired": [ "bulk-edit.preview.updated-items.collection.get" ], - "modulePermissions": [] - }, - { - "methods": [ "GET" ], - "pathPattern": "/bulk-edit/{jobId}/preview/updated-users/download", - "permissionsRequired": [ "bulk-edit.preview.updated-users.collection.get" ], - "modulePermissions": [] - }, - { - "methods": [ "GET" ], - "pathPattern": "/bulk-edit/{jobId}/preview/updated-holdings/download", - "permissionsRequired": [ "bulk-edit.preview.updated-holdings.collection.get" ], - "modulePermissions": [] + "pathPattern": "/bulk-edit/permissions-self-check", + "permissionsRequired": ["bulk-edit.permissions-self-check.get"], + "modulePermissions": [ + "perms.users.get", + "permissions.users.item.get" + ] }, { "methods": [ "GET" ], @@ -457,7 +352,8 @@ "inventory-storage.service-points.collection.get", "instance-authority-links.authority-statistics.collection.get", "configuration.entries.collection.get", - "configuration.entries.item.post" + "configuration.entries.item.post", + "configuration.entries.item.delete" ] }, { @@ -489,66 +385,21 @@ "displayName" : "upload identifiers list", "description" : "Upload list of item identifiers" }, - { - "permissionName" : "bulk-edit.items-content-update.collection.post", - "displayName" : "upload items content update collection", - "description" : "Upload collection of item content updates" - }, - { - "permissionName" : "bulk-edit.users-content-update.collection.post", - "displayName" : "upload users content update collection", - "description" : "Upload collection of user content updates" - }, - { - "permissionName" : "bulk-edit.holdings-content-update.collection.post", - "displayName" : "upload holdings records content update collection", - "description" : "Upload collection of holdings records content updates" - }, - { - "permissionName" : "bulk-edit.preview.updated-items.collection.get", - "displayName" : "download updated items preview csv", - "description" : "Download csv of preview updated items" - }, - { - "permissionName" : "bulk-edit.preview.updated-users.collection.get", - "displayName" : "download updated users preview csv", - "description" : "Download csv with preview of updated users" - }, - { - "permissionName" : "bulk-edit.preview.updated-holdings.collection.get", - "displayName" : "download updated holdings records preview csv", - "description" : "Download csv with preview of updated holdings records" - }, { "permissionName": "bulk-edit.start.item.post", "displayName": "start update items", "description": "Start update items" }, - { - "permissionName": "bulk-edit.roll-back.item.post", - "displayName": "roll back list of uploaded items", - "description": "Roll back list of uploaded items" - }, - { - "permissionName" : "bulk-edit.preview-users.collection.get", - "displayName" : "get N users for preview", - "description" : "Get N users for preview" - }, - { - "permissionName" : "bulk-edit.preview-items.collection.get", - "displayName" : "get N items for preview", - "description" : "Get N items for preview" - }, - { - "permissionName" : "bulk-edit.preview-holdings.collection.get", - "displayName" : "get N holdings for preview", - "description" : "Get N holdings for preview" - }, { "permissionName" : "bulk-edit.errors.collection.get", "displayName" : "get errors for preview", "description" : "Get errors for preview" }, + { + "permissionName" : "bulk-edit.permissions-self-check.get", + "displayName" : "Set of users permissions for self check", + "description" : "Set of users permissions for self check" + }, { "permissionName" : "refresh-presigned-url.get", "displayName" : "Get refreshed presigned url for export file", @@ -560,18 +411,9 @@ "description" : "All permissions for bulk-edit module", "subPermissions" : [ "bulk-edit.item.post", - "bulk-edit.items-content-update.collection.post", - "bulk-edit.users-content-update.collection.post", - "bulk-edit.holdings-content-update.collection.post", - "bulk-edit.preview.updated-items.collection.get", - "bulk-edit.preview.updated-users.collection.get", - "bulk-edit.preview.updated-holdings.collection.get", "bulk-edit.start.item.post", - "bulk-edit.roll-back.item.post", - "bulk-edit.preview-users.collection.get", - "bulk-edit.preview-items.collection.get", - "bulk-edit.preview-holdings.collection.get", - "bulk-edit.errors.collection.get" + "bulk-edit.errors.collection.get", + "bulk-edit.permissions-self-check.get" ] }, { diff --git a/folio-export-common b/folio-export-common index 4aae4a204..129ece8d3 160000 --- a/folio-export-common +++ b/folio-export-common @@ -1 +1 @@ -Subproject commit 4aae4a204d0e69a2056632ad36715d774843479a +Subproject commit 129ece8d3bc1b59b915042a8fc01fcee928b4ada diff --git a/pom.xml b/pom.xml index ed379fb52..9d16a573f 100644 --- a/pom.xml +++ b/pom.xml @@ -12,7 +12,7 @@ org.folio mod-data-export-worker Data Export Worker module - 3.3.0-SNAPSHOT + 3.4.0-SNAPSHOT jar @@ -48,11 +48,10 @@ ${project.basedir}/src/main/resources/swagger.api/refresh-presigned-url.yaml - - 8.1.0 + 8.2.0 3.1.0 - 8.1.0 - 2.11.0 + 8.2.0 + 1.0.0 8.5.9 6.2.1 4.4 @@ -179,11 +178,6 @@ commons-collections4 ${commons-collections4.version} - - commons-io - commons-io - ${commons-io.version} - org.projectlombok @@ -513,110 +507,110 @@ - bursar-export-worker-schedule-parameters - - generate - - - ${bursar-export.yaml.file} - ${project.build.directory}/generated-sources - spring - ${project.groupId}.dew.domain.dto - false - false - false - true - ScheduleParameters - false - false - true - ApiUtil.java - - true - java8 - true - true - - - + bursar-export-worker-schedule-parameters + + generate + + + ${bursar-export.yaml.file} + ${project.build.directory}/generated-sources + spring + ${project.groupId}.dew.domain.dto + false + false + false + true + ScheduleParameters + false + false + true + ApiUtil.java + + true + java8 + true + true + + + - bulk-edit-worker - - generate - - - ${bulk-edit.yaml.file} - ${project.build.directory}/generated-sources - spring - ${project.groupId}.dew.domain.dto - true - false - false - true - false - true - true - - true - java - true - true - true - - - + bulk-edit-worker + + generate + + + ${bulk-edit.yaml.file} + ${project.build.directory}/generated-sources + spring + ${project.groupId}.dew.domain.dto + true + false + false + true + false + true + true + + true + java + true + true + true + + + - refresh-presigned-url - - generate - - - ${refresh-presigned-url.yaml.file} - ${project.build.directory}/generated-sources - spring - ${project.groupId}.dew.domain.dto - true - false - true - true - false - true - true - ApiUtil.java - - true - java - true - true - - - + refresh-presigned-url + + generate + + + ${refresh-presigned-url.yaml.file} + ${project.build.directory}/generated-sources + spring + ${project.groupId}.dew.domain.dto + true + false + true + true + false + true + true + ApiUtil.java + + true + java + true + true + + + - order-export-worker - - generate - - - ${order-export.yaml.file} - ${project.build.directory}/generated-sources - spring - ${project.groupId}.dew.domain.dto - true - false - false - true - false - true - true - - true - java - true - true - true - - - + order-export-worker + + generate + + + ${order-export.yaml.file} + ${project.build.directory}/generated-sources + spring + ${project.groupId}.dew.domain.dto + true + false + false + true + false + true + true + + true + java + true + true + true + + + @@ -724,6 +718,19 @@ true + + + org.folio + folio-module-descriptor-validator + ${folio-module-descriptor-validator.version} + + + + validate + + + + @@ -742,6 +749,14 @@ + + + folio-nexus + FOLIO Maven repository + https://repository.folio.org/repository/maven-folio + + + folio-nexus diff --git a/src/main/java/org/folio/dew/batch/CsvFileAssembler.java b/src/main/java/org/folio/dew/batch/CsvFileAssembler.java index d38b34010..e5b40b12c 100644 --- a/src/main/java/org/folio/dew/batch/CsvFileAssembler.java +++ b/src/main/java/org/folio/dew/batch/CsvFileAssembler.java @@ -27,7 +27,6 @@ public void aggregate(StepExecution stepExecution, Collection fin .collect(Collectors.toList()); var destCsvObject = FilenameUtils.getName( stepExecution.getJobExecution().getJobParameters().getString(JobParameterNames.TEMP_OUTPUT_FILE_PATH) + ".csv"); - try { if ("CIRCULATION_LOG".equals(stepExecution.getJobExecution().getJobInstance().getJobName())) { var csvUrl = remoteFilesStorage.composeObject(destCsvObject, csvFilePartObjectNames, null, TEXT_CSV); diff --git a/src/main/java/org/folio/dew/batch/JobCompletionNotificationListener.java b/src/main/java/org/folio/dew/batch/JobCompletionNotificationListener.java index 8fa09fdbc..79782042a 100644 --- a/src/main/java/org/folio/dew/batch/JobCompletionNotificationListener.java +++ b/src/main/java/org/folio/dew/batch/JobCompletionNotificationListener.java @@ -62,7 +62,6 @@ import org.folio.dew.error.BulkEditException; import org.folio.dew.repository.LocalFilesStorage; import org.folio.dew.repository.RemoteFilesStorage; -import org.folio.dew.service.BulkEditChangedRecordsService; import org.folio.dew.service.BulkEditProcessingErrorsService; import org.folio.dew.service.BulkEditStatisticService; import org.folio.dew.utils.CsvHelper; @@ -83,7 +82,6 @@ public class JobCompletionNotificationListener implements JobExecutionListener { private final LocalFilesStorage localFilesStorage; private final BulkEditProcessingErrorsService bulkEditProcessingErrorsService; private final BulkEditStatisticService bulkEditStatisticService; - private final BulkEditChangedRecordsService changedRecordsService; @Override public void beforeJob(JobExecution jobExecution) { @@ -111,15 +109,6 @@ private void processJobUpdate(JobExecution jobExecution, boolean after) { handleProcessingMatchedRecordsAndErrors(jobExecution, jobId); handleProcessingMarcFile(jobExecution); } - if (isBulkEditUpdateJob(jobExecution)) { - handleProcessingChangedRecords(jobExecution); - String downloadErrorLink = bulkEditProcessingErrorsService.saveErrorFileAndGetDownloadLink(jobId, jobExecution); - var isChangedRecordsLinkPresent = jobExecution.getExecutionContext().containsKey(OUTPUT_FILES_IN_STORAGE); - jobExecution.getExecutionContext().putString(OUTPUT_FILES_IN_STORAGE, - (isChangedRecordsLinkPresent ? jobExecution.getExecutionContext().getString(OUTPUT_FILES_IN_STORAGE) : EMPTY) + - PATHS_DELIMITER + (StringUtils.isNotBlank(downloadErrorLink) ? downloadErrorLink : EMPTY)); - - } processJobAfter(jobId, jobParameters); } else { if (jobExecution.getJobInstance().getJobName().contains(BULK_EDIT_UPDATE.getValue())) { @@ -142,19 +131,19 @@ private void processJobUpdate(JobExecution jobExecution, boolean after) { } var jobExecutionUpdate = createJobExecutionUpdate(jobId, jobExecution); - if (jobExecution.getJobInstance().getJobName().contains(BULK_EDIT_UPDATE.getValue()) || jobExecution.getJobInstance().getJobName().contains(BULK_EDIT_IDENTIFIERS.getValue())) { + if (jobExecution.getJobInstance().getJobName().contains(BULK_EDIT_IDENTIFIERS.getValue())) { var progress = new Progress(); if (jobExecution.getStatus() == BatchStatus.COMPLETED) { var fileName = FilenameUtils.getName(jobParameters.getString(FILE_NAME)); var errors = bulkEditProcessingErrorsService.readErrorsFromCSV(jobId, fileName, 1_000_000); - var statistic = bulkEditStatisticService.getStatistic(); - var totalRecords = statistic.getSuccess() + errors.getTotalRecords(); + var totalRecords = bulkEditStatisticService.getSuccess(jobId) + errors.getTotalRecords(); progress.setTotal(totalRecords); progress.setProcessed(totalRecords); progress.setProgress(COMPLETE_PROGRESS_VALUE); - progress.setSuccess(statistic.getSuccess()); + progress.setSuccess(bulkEditStatisticService.getSuccess(jobId)); progress.setErrors(errors.getTotalRecords()); jobExecutionUpdate.setProgress(progress); + bulkEditStatisticService.reset(jobId); } jobExecutionUpdate.setProgress(progress); } @@ -213,17 +202,13 @@ private void moveFileToStorage(String destFileName, String sourceFileName) throw private void handleProcessingMatchedRecordsAndErrors(JobExecution jobExecution, String jobId) { String downloadErrorLink = bulkEditProcessingErrorsService.saveErrorFileAndGetDownloadLink(jobId, jobExecution); - jobExecution.getExecutionContext().putString(OUTPUT_FILES_IN_STORAGE, saveResult(jobExecution, false) + PATHS_DELIMITER + (isNull(downloadErrorLink) ? EMPTY : downloadErrorLink) + PATHS_DELIMITER + saveJsonResult(jobExecution, !isBulkEditUpdateJob(jobExecution))); + jobExecution.getExecutionContext().putString(OUTPUT_FILES_IN_STORAGE, saveResult(jobExecution, false) + PATHS_DELIMITER + (isNull(downloadErrorLink) ? EMPTY : downloadErrorLink) + PATHS_DELIMITER + saveJsonResult(jobExecution, true)); } private void handleProcessingMarcFile(JobExecution jobExecution) { jobExecution.getExecutionContext().putString(OUTPUT_FILES_IN_STORAGE, jobExecution.getExecutionContext().getString(OUTPUT_FILES_IN_STORAGE) + PATHS_DELIMITER + saveMarcResult(jobExecution, false)); } - private void handleProcessingChangedRecords(JobExecution jobExecution) { - jobExecution.getExecutionContext().putString(OUTPUT_FILES_IN_STORAGE, saveResult(jobExecution, !isBulkEditUpdateJob(jobExecution))); - } - private void processJobAfter(String jobId, JobParameters jobParameters) { var tempOutputFilePath = jobParameters.getString(TEMP_OUTPUT_FILE_PATH); if (StringUtils.isBlank(tempOutputFilePath) || @@ -322,21 +307,12 @@ private boolean isBulkEditIdentifiersJob(JobExecution jobExecution) { return jobExecution.getJobInstance().getJobName().contains(BULK_EDIT_IDENTIFIERS.getValue()); } - private boolean isBulkEditUpdateJob(JobExecution jobExecution) { - return jobExecution.getJobInstance().getJobName().contains(BULK_EDIT_UPDATE.getValue()); - } - - private boolean isBulkEditContentUpdateJob(JobExecution jobExecution) { - return nonNull(jobExecution.getJobParameters().getString(UPDATED_FILE_NAME)); - } - private boolean isBulkEditQueryJob(JobExecution jobExecution) { return jobExecution.getJobInstance().getJobName().contains(BULK_EDIT_QUERY.getValue()); } private boolean isBulkEditJob(JobExecution jobExecution) { - return isBulkEditContentUpdateJob(jobExecution) || isBulkEditUpdateJob(jobExecution) || - isBulkEditIdentifiersJob(jobExecution) || isBulkEditQueryJob(jobExecution); + return isBulkEditIdentifiersJob(jobExecution) || isBulkEditQueryJob(jobExecution); } private boolean isBursarFeesFinesJob(JobExecution jobExecution) { @@ -395,13 +371,9 @@ private String saveMarcResult(JobExecution jobExecution, boolean isSourceToBeDel } private String preparePath(JobExecution jobExecution) { - if (isBulkEditContentUpdateJob(jobExecution)) { - return jobExecution.getJobParameters().getString(UPDATED_FILE_NAME); - } else if (isBulkEditUpdateJob(jobExecution)) { - return prepareChangedUsersFile(jobExecution.getJobParameters().getString(FILE_NAME), jobExecution.getJobParameters().getString(JobParameterNames.JOB_ID)); - } return jobExecution.getJobParameters().getString(TEMP_OUTPUT_FILE_PATH); } + private boolean noRecordsFound(String path) throws Exception { if (localFilesStorage.notExists(path) && !remoteFilesStorage.containsFile(path)) { log.error("Path to found records does not exist: {}", path); @@ -419,11 +391,11 @@ private boolean noRecordsFound(String path) throws Exception { } private String prepareObject(JobExecution jobExecution, String path) { - return jobExecution.getJobParameters().getString(JobParameterNames.JOB_ID) + PATH_SEPARATOR + FilenameUtils.getName(path) + (!isBulkEditUpdateJob(jobExecution) ? CSV_EXTENSION : EMPTY); + return jobExecution.getJobParameters().getString(JobParameterNames.JOB_ID) + PATH_SEPARATOR + FilenameUtils.getName(path) + CSV_EXTENSION; } private String prepareJsonObject(JobExecution jobExecution, String path) { - return jobExecution.getJobParameters().getString(JobParameterNames.JOB_ID) + PATH_SEPARATOR + FilenameUtils.getName(path) + (!isBulkEditUpdateJob(jobExecution) ? ".json" : EMPTY); + return jobExecution.getJobParameters().getString(JobParameterNames.JOB_ID) + PATH_SEPARATOR + FilenameUtils.getName(path) + ".json"; } private String prepareMrcObject(JobExecution jobExecution, String path) { @@ -450,21 +422,4 @@ private String prepareDownloadJsonFilename(JobExecution jobExecution, String pat .replace(INITIAL_PREFIX, EMPTY); } - private String prepareChangedUsersFile(String path, String jobId) { - var updatedIds = changedRecordsService.fetchChangedUserIds(jobId); - if (isNull(updatedIds) || updatedIds.isEmpty()) { - return EMPTY; - } - try { - var updatedUserFormats = CsvHelper.readRecordsFromStorage(localFilesStorage, path, UserFormat.class, true) - .stream() - .filter(userFormat -> updatedIds.contains(userFormat.getId())) - .collect(Collectors.toList()); - CsvHelper.saveRecordsToStorage(localFilesStorage, updatedUserFormats, UserFormat.class, path); - } catch (Exception e) { - log.error("Error processing file {}: {}", path, e.getMessage()); - } - return path; - } - } diff --git a/src/main/java/org/folio/dew/batch/JsonFileWriter.java b/src/main/java/org/folio/dew/batch/JsonFileWriter.java index c11dd545a..254be58b7 100644 --- a/src/main/java/org/folio/dew/batch/JsonFileWriter.java +++ b/src/main/java/org/folio/dew/batch/JsonFileWriter.java @@ -1,6 +1,7 @@ package org.folio.dew.batch; import static org.apache.commons.lang3.StringUtils.EMPTY; +import static org.folio.dew.utils.Constants.NEW_LINE; import static org.folio.dew.utils.WriterHelper.enrichHoldingsJson; import com.fasterxml.jackson.databind.ObjectMapper; @@ -43,9 +44,7 @@ public String doWrite(Chunk items) { } else { lines.append(marshaller.marshal(item.getOriginal())); } - if(iterator.hasNext()) { - lines.append('\n'); - } + lines.append(NEW_LINE); } return lines.toString(); } diff --git a/src/main/java/org/folio/dew/batch/JsonListFileWriter.java b/src/main/java/org/folio/dew/batch/JsonListFileWriter.java index c566248a2..0eee9dc8c 100644 --- a/src/main/java/org/folio/dew/batch/JsonListFileWriter.java +++ b/src/main/java/org/folio/dew/batch/JsonListFileWriter.java @@ -1,6 +1,7 @@ package org.folio.dew.batch; import static org.apache.commons.lang3.StringUtils.EMPTY; +import static org.folio.dew.utils.Constants.NEW_LINE; import com.fasterxml.jackson.databind.ObjectMapper; import org.folio.dew.domain.dto.Formatable; @@ -43,9 +44,7 @@ public String doWrite(Chunk> lists) { } else { lines.append(marshaller.marshal(item.getOriginal())); } - if(iterator.hasNext()) { - lines.append('\n'); - } + lines.append(NEW_LINE); } return lines.toString(); } diff --git a/src/main/java/org/folio/dew/batch/MarcAsListStringsWriter.java b/src/main/java/org/folio/dew/batch/MarcAsListStringsWriter.java index d17f4d45d..f620f126d 100644 --- a/src/main/java/org/folio/dew/batch/MarcAsListStringsWriter.java +++ b/src/main/java/org/folio/dew/batch/MarcAsListStringsWriter.java @@ -78,7 +78,7 @@ public void close() { private List getMarcContent(String id) throws Exception { List mrcRecords = new ArrayList<>(); - var srsRecords = srsClient.getMarc(id, "INSTANCE").get("sourceRecords"); + var srsRecords = srsClient.getMarc(id, "INSTANCE", true).get("sourceRecords"); if (srsRecords.isEmpty()) { log.warn("No SRS records found by instanceId = {}", id); return mrcRecords; diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditHoldingsProcessor.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditHoldingsProcessor.java index 0a89cbeef..c0d90a666 100644 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditHoldingsProcessor.java +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditHoldingsProcessor.java @@ -11,20 +11,21 @@ import static org.folio.dew.utils.BulkEditProcessorHelper.resolveIdentifier; import static org.folio.dew.utils.Constants.DUPLICATES_ACROSS_TENANTS; import static org.folio.dew.utils.Constants.MULTIPLE_MATCHES_MESSAGE; -import static org.folio.dew.utils.Constants.NO_HOLDING_AFFILIATION; +import static org.folio.dew.utils.Constants.NO_HOLDING_VIEW_PERMISSIONS; import static org.folio.dew.utils.Constants.NO_MATCH_FOUND_MESSAGE; import static org.folio.dew.utils.SearchIdentifierTypeResolver.getSearchIdentifierType; -import feign.FeignException; import lombok.RequiredArgsConstructor; import lombok.extern.log4j.Log4j2; import org.apache.commons.io.FilenameUtils; import org.apache.commons.lang3.StringUtils; +import org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionsValidator; import org.folio.dew.client.HoldingClient; import org.folio.dew.client.SearchClient; import org.folio.dew.client.UserClient; import org.folio.dew.domain.dto.BatchIdsDto; import org.folio.dew.domain.dto.ConsortiumHolding; +import org.folio.dew.domain.dto.EntityType; import org.folio.dew.domain.dto.ExtendedHoldingsRecord; import org.folio.dew.domain.dto.ExtendedHoldingsRecordCollection; import org.folio.dew.domain.dto.HoldingsFormat; @@ -38,15 +39,16 @@ import org.folio.dew.service.mapper.HoldingsMapper; import org.folio.spring.FolioExecutionContext; import org.folio.spring.scope.FolioExecutionContextSetter; +import org.springframework.batch.core.JobExecution; import org.springframework.batch.core.configuration.annotation.StepScope; import org.springframework.batch.item.ItemProcessor; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Component; import java.util.ArrayList; -import java.util.HashSet; import java.util.List; import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; import java.util.stream.Collectors; @Component @@ -61,7 +63,11 @@ public class BulkEditHoldingsProcessor extends FolioExecutionContextManager impl private final ConsortiaService consortiaService; private final FolioExecutionContext folioExecutionContext; private final UserClient userClient; + private final PermissionsValidator permissionsValidator; + private final TenantResolver tenantResolver; + @Value("#{stepExecution.jobExecution}") + private JobExecution jobExecution; @Value("#{jobParameters['identifierType']}") private String identifierType; @Value("#{jobParameters['jobId']}") @@ -69,22 +75,17 @@ public class BulkEditHoldingsProcessor extends FolioExecutionContextManager impl @Value("#{jobParameters['fileName']}") private String fileName; - private Set identifiersToCheckDuplication = new HashSet<>(); - private Set fetchedHoldingsIds = new HashSet<>(); + private Set identifiersToCheckDuplication = ConcurrentHashMap.newKeySet(); + private Set fetchedHoldingsIds = ConcurrentHashMap.newKeySet(); @Override - public List process(ItemIdentifier itemIdentifier) throws BulkEditException { + public synchronized List process(ItemIdentifier itemIdentifier) throws BulkEditException { if (identifiersToCheckDuplication.contains(itemIdentifier)) { throw new BulkEditException("Duplicate entry"); } identifiersToCheckDuplication.add(itemIdentifier); var holdings = getHoldingsRecords(itemIdentifier); - if (holdings.getExtendedHoldingsRecords().isEmpty()) { - log.error(NO_MATCH_FOUND_MESSAGE); - throw new BulkEditException(NO_MATCH_FOUND_MESSAGE); - } - var distinctHoldings = holdings.getExtendedHoldingsRecords().stream() .filter(holdingsRecord -> !fetchedHoldingsIds.contains(holdingsRecord.getEntity().getId())) .toList(); @@ -121,7 +122,9 @@ private ExtendedHoldingsRecordCollection getHoldingsRecords(ItemIdentifier itemI if (INSTANCEHRID != identifierTypeEnum && tenantIds.size() > 1) { throw new BulkEditException(DUPLICATES_ACROSS_TENANTS); } - tenantIds.forEach(tenantId -> { + var affiliatedPermittedTenants = tenantResolver.getAffiliatedPermittedTenantIds(EntityType.HOLDINGS_RECORD, + jobExecution, identifierType, tenantIds, itemIdentifier); + affiliatedPermittedTenants.forEach(tenantId -> { try (var context = new FolioExecutionContextSetter(refreshAndGetFolioExecutionContext(tenantId, folioExecutionContext))) { var holdingsRecordCollection = getHoldingsRecordCollection(type, itemIdentifier); extendedHoldingsRecordCollection.getExtendedHoldingsRecords().addAll( @@ -130,24 +133,32 @@ private ExtendedHoldingsRecordCollection getHoldingsRecords(ItemIdentifier itemI ); extendedHoldingsRecordCollection.setTotalRecords(extendedHoldingsRecordCollection.getTotalRecords() + holdingsRecordCollection.getTotalRecords()); } catch (Exception e) { - if (e instanceof FeignException && ((FeignException) e).status() == 401) { - var user = userClient.getUserById(folioExecutionContext.getUserId().toString()); - throw new BulkEditException(format(NO_HOLDING_AFFILIATION, user.getUsername(), resolveIdentifier(identifierType), identifier, tenantId)); - } else { - throw e; - } + log.error(e.getMessage()); + throw e; } }); - return extendedHoldingsRecordCollection; + return extendedHoldingsRecordCollection; } else { throw new BulkEditException(NO_MATCH_FOUND_MESSAGE); } } else { // Process local tenant case + checkReadPermissions(folioExecutionContext.getTenantId(), identifier); var holdingsRecordCollection = getHoldingsRecordCollection(type, itemIdentifier); - return new ExtendedHoldingsRecordCollection().extendedHoldingsRecords(holdingsRecordCollection.getHoldingsRecords().stream() + var extendedHoldingsRecordCollection = new ExtendedHoldingsRecordCollection().extendedHoldingsRecords(holdingsRecordCollection.getHoldingsRecords().stream() .map(holdingsRecord -> new ExtendedHoldingsRecord().tenantId(folioExecutionContext.getTenantId()).entity(holdingsRecord)).toList()) .totalRecords(holdingsRecordCollection.getTotalRecords()); + if (extendedHoldingsRecordCollection.getExtendedHoldingsRecords().isEmpty()) { + throw new BulkEditException(NO_MATCH_FOUND_MESSAGE); + } + return extendedHoldingsRecordCollection; + } + } + + private void checkReadPermissions(String tenantId, String identifier) { + if (!permissionsValidator.isBulkEditReadPermissionExists(tenantId, EntityType.HOLDINGS_RECORD)) { + var user = userClient.getUserById(folioExecutionContext.getUserId().toString()); + throw new BulkEditException(format(NO_HOLDING_VIEW_PERMISSIONS, user.getUsername(), resolveIdentifier(identifierType), identifier, tenantId)); } } @@ -172,5 +183,4 @@ private HoldingsRecordCollection getHoldingsRecordCollection(IdentifierType type throw new BulkEditException(format("Identifier type \"%s\" is not supported", identifierType)); } } - } diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditInstanceProcessor.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditInstanceProcessor.java index 45aa5490e..2e2e9f8e0 100644 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditInstanceProcessor.java +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditInstanceProcessor.java @@ -1,15 +1,20 @@ package org.folio.dew.batch.bulkedit.jobs; +import static java.lang.String.format; import static org.folio.dew.domain.dto.IdentifierType.ISBN; import static org.folio.dew.domain.dto.IdentifierType.ISSN; import static org.folio.dew.utils.BulkEditProcessorHelper.getMatchPattern; import static org.folio.dew.utils.BulkEditProcessorHelper.resolveIdentifier; +import static org.folio.dew.utils.Constants.NO_INSTANCE_VIEW_PERMISSIONS; import static org.folio.dew.utils.Constants.NO_MATCH_FOUND_MESSAGE; import lombok.RequiredArgsConstructor; import lombok.extern.log4j.Log4j2; import org.apache.commons.io.FilenameUtils; +import org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionsValidator; import org.folio.dew.client.InventoryInstancesClient; +import org.folio.dew.client.UserClient; +import org.folio.dew.domain.dto.EntityType; import org.folio.dew.domain.dto.IdentifierType; import org.folio.dew.domain.dto.Instance; import org.folio.dew.domain.dto.InstanceCollection; @@ -27,6 +32,7 @@ import java.util.HashSet; import java.util.List; import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; @Component @StepScope @@ -37,6 +43,8 @@ public class BulkEditInstanceProcessor implements ItemProcessor identifiersToCheckDuplication = new HashSet<>(); - private Set fetchedInstanceIds = new HashSet<>(); + private Set identifiersToCheckDuplication = ConcurrentHashMap.newKeySet(); + private Set fetchedInstanceIds = ConcurrentHashMap.newKeySet(); @Override - public List process(ItemIdentifier itemIdentifier) throws BulkEditException { + public synchronized List process(ItemIdentifier itemIdentifier) throws BulkEditException { + if (!permissionsValidator.isBulkEditReadPermissionExists(folioExecutionContext.getTenantId(), EntityType.INSTANCE)) { + var user = userClient.getUserById(folioExecutionContext.getUserId().toString()); + throw new BulkEditException(format(NO_INSTANCE_VIEW_PERMISSIONS, user.getUsername(), resolveIdentifier(identifierType), itemIdentifier.getItemId(), folioExecutionContext.getTenantId())); + } if (identifiersToCheckDuplication.contains(itemIdentifier)) { throw new BulkEditException("Duplicate entry"); } diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditItemListProcessor.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditItemListProcessor.java index fe354eeb5..b37a53fe2 100644 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditItemListProcessor.java +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditItemListProcessor.java @@ -1,18 +1,10 @@ package org.folio.dew.batch.bulkedit.jobs; -import static org.folio.dew.utils.Constants.NO_MATCH_FOUND_MESSAGE; import lombok.RequiredArgsConstructor; import lombok.extern.log4j.Log4j2; -import org.apache.commons.lang3.StringUtils; -import org.folio.dew.client.SearchClient; -import org.folio.dew.domain.dto.BatchIdsDto; -import org.folio.dew.domain.dto.ConsortiumItem; import org.folio.dew.domain.dto.ExtendedItemCollection; -import org.folio.dew.domain.dto.Item; -import org.folio.dew.domain.dto.ItemCollection; import org.folio.dew.domain.dto.ItemFormat; -import org.folio.dew.error.BulkEditException; import org.springframework.batch.core.configuration.annotation.StepScope; import org.springframework.batch.item.ItemProcessor; import org.springframework.stereotype.Component; @@ -28,10 +20,6 @@ public class BulkEditItemListProcessor implements ItemProcessor process(ExtendedItemCollection extendedItemCollection) { - if (extendedItemCollection.getExtendedItems().isEmpty()) { - log.error(NO_MATCH_FOUND_MESSAGE); - throw new BulkEditException(NO_MATCH_FOUND_MESSAGE); - } return extendedItemCollection.getExtendedItems().stream() .map(bulkEditItemProcessor::process) .toList(); diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditItemProcessor.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditItemProcessor.java index 3dc309c82..66f27a295 100644 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditItemProcessor.java +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/BulkEditItemProcessor.java @@ -157,7 +157,9 @@ private String fetchNotes(Item item, ErrorServiceArgs args, String tenantId) { .map(itemNote -> String.join(ARRAY_DELIMITER, escaper.escape(itemReferenceService.getNoteTypeNameById(itemNote.getItemNoteTypeId(), args, tenantId)), escaper.escape(itemNote.getNote()), - escaper.escape(booleanToStringNullSafe(itemNote.getStaffOnly())))) + escaper.escape(booleanToStringNullSafe(itemNote.getStaffOnly())), + tenantId, + itemNote.getItemNoteTypeId())) .collect(Collectors.joining(ITEM_DELIMITER)); } diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/TenantResolver.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/TenantResolver.java new file mode 100644 index 000000000..71a7eb574 --- /dev/null +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/TenantResolver.java @@ -0,0 +1,88 @@ +package org.folio.dew.batch.bulkedit.jobs; + +import static java.lang.String.format; +import static org.folio.dew.utils.BulkEditProcessorHelper.resolveIdentifier; +import static org.folio.dew.utils.Constants.FILE_NAME; +import static org.folio.dew.utils.Constants.NO_HOLDING_AFFILIATION; +import static org.folio.dew.utils.Constants.NO_HOLDING_VIEW_PERMISSIONS; +import static org.folio.dew.utils.Constants.NO_ITEM_AFFILIATION; +import static org.folio.dew.utils.Constants.NO_ITEM_VIEW_PERMISSIONS; + +import feign.FeignException; +import lombok.RequiredArgsConstructor; +import org.apache.commons.io.FilenameUtils; +import org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionsValidator; +import org.folio.dew.client.UserClient; +import org.folio.dew.domain.dto.EntityType; +import org.folio.dew.domain.dto.ItemIdentifier; +import org.folio.dew.domain.dto.JobParameterNames; +import org.folio.dew.error.BulkEditException; +import org.folio.dew.service.BulkEditProcessingErrorsService; +import org.folio.dew.service.ConsortiaService; +import org.folio.spring.FolioExecutionContext; +import org.springframework.batch.core.JobExecution; +import org.springframework.stereotype.Component; + +import java.util.HashSet; +import java.util.Set; + +@Component +@RequiredArgsConstructor +public class TenantResolver { + + private static final String UNSUPPORTED_ERROR_MESSAGE_FOR_AFFILIATIONS = "Unsupported entity type to get affiliation error message"; + private static final String UNSUPPORTED_ERROR_MESSAGE_FOR_PERMISSIONS = "Unsupported entity type to get permissions error message"; + + private final FolioExecutionContext folioExecutionContext; + private final ConsortiaService consortiaService; + private final PermissionsValidator permissionsValidator; + private final BulkEditProcessingErrorsService bulkEditProcessingErrorsService; + private final UserClient userClient; + + public Set getAffiliatedPermittedTenantIds(EntityType entityType, JobExecution jobExecution, String identifierType, Set tenantIds, ItemIdentifier itemIdentifier) { + var affiliatedTenants = consortiaService.getAffiliatedTenants(folioExecutionContext.getTenantId(), folioExecutionContext.getUserId().toString()); + var jobId = jobExecution.getJobParameters().getString(JobParameterNames.JOB_ID); + var fileName = FilenameUtils.getName(jobExecution.getJobParameters().getString(FILE_NAME)); + var affiliatedAndPermittedTenants = new HashSet(); + for (var tenantId : tenantIds) { + if (!affiliatedTenants.contains(tenantId)) { + var user = userClient.getUserById(folioExecutionContext.getUserId().toString()); + var errorMessage = format(getAffiliationErrorPlaceholder(entityType), user.getUsername(), + resolveIdentifier(identifierType), itemIdentifier.getItemId(), tenantId); + bulkEditProcessingErrorsService.saveErrorInCSV(jobId, itemIdentifier.getItemId(), errorMessage, fileName); + } else if (!isBulkEditReadPermissionExists(tenantId, entityType)) { + var user = userClient.getUserById(folioExecutionContext.getUserId().toString()); + var errorMessage = format(getViewPermissionErrorPlaceholder(entityType), user.getUsername(), + resolveIdentifier(identifierType), itemIdentifier.getItemId(), tenantId); + bulkEditProcessingErrorsService.saveErrorInCSV(jobId, itemIdentifier.getItemId(), errorMessage, fileName); + } else { + affiliatedAndPermittedTenants.add(tenantId); + } + } + return affiliatedAndPermittedTenants; + } + + private boolean isBulkEditReadPermissionExists(String tenantId, EntityType entityType) { + try { + return permissionsValidator.isBulkEditReadPermissionExists(tenantId, entityType); + } catch (FeignException e) { + throw new BulkEditException(e.getMessage()); + } + } + + protected String getAffiliationErrorPlaceholder(EntityType entityType) { + return switch (entityType) { + case ITEM -> NO_ITEM_AFFILIATION; + case HOLDINGS_RECORD -> NO_HOLDING_AFFILIATION; + default -> throw new UnsupportedOperationException(UNSUPPORTED_ERROR_MESSAGE_FOR_AFFILIATIONS); + }; + } + + protected String getViewPermissionErrorPlaceholder(EntityType entityType) { + return switch (entityType) { + case ITEM -> NO_ITEM_VIEW_PERMISSIONS; + case HOLDINGS_RECORD -> NO_HOLDING_VIEW_PERMISSIONS; + default -> throw new UnsupportedOperationException(UNSUPPORTED_ERROR_MESSAGE_FOR_PERMISSIONS); + }; + } +} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/PermissionEnum.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/PermissionEnum.java new file mode 100644 index 000000000..409d74902 --- /dev/null +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/PermissionEnum.java @@ -0,0 +1,18 @@ +package org.folio.dew.batch.bulkedit.jobs.permissions.check; + + +import lombok.AllArgsConstructor; +import lombok.Getter; + +@AllArgsConstructor +@Getter +public enum PermissionEnum { + BULK_EDIT_INVENTORY_VIEW_PERMISSION("bulk-operations.item.inventory.get"), + BULK_EDIT_USERS_VIEW_PERMISSION("bulk-operations.item.users.get"), + USER_ITEM_GET_PERMISSION("users.item.get"), + INVENTORY_ITEMS_ITEM_GET_PERMISSION("inventory.items.item.get"), + INVENTORY_STORAGE_HOLDINGS_ITEM_GET_PERMISSION("inventory-storage.holdings.item.get"), + INVENTORY_INSTANCES_ITEM_GET_PERMISSION("inventory.instances.item.get"); + + private final String value; +} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/PermissionsProvider.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/PermissionsProvider.java new file mode 100644 index 000000000..644db43c3 --- /dev/null +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/PermissionsProvider.java @@ -0,0 +1,26 @@ +package org.folio.dew.batch.bulkedit.jobs.permissions.check; + +import lombok.RequiredArgsConstructor; +import org.folio.dew.client.PermissionsSelfCheckClient; +import org.folio.dew.service.FolioExecutionContextManager; +import org.folio.spring.FolioExecutionContext; +import org.folio.spring.scope.FolioExecutionContextSetter; +import org.springframework.cache.annotation.Cacheable; +import org.springframework.stereotype.Component; + +import java.util.List; + +@Component +@RequiredArgsConstructor +public class PermissionsProvider extends FolioExecutionContextManager { + + private final PermissionsSelfCheckClient permissionsSelfCheckClient; + private final FolioExecutionContext folioExecutionContext; + + @Cacheable(cacheNames = "userPermissions") + public List getUserPermissions(String tenantId, String userId) { + try (var ignored = new FolioExecutionContextSetter(refreshAndGetFolioExecutionContext(tenantId, folioExecutionContext))) { + return permissionsSelfCheckClient.getUserPermissionsForSelfCheck(); + } + } +} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/PermissionsValidator.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/PermissionsValidator.java new file mode 100644 index 000000000..4f1a0f76b --- /dev/null +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/PermissionsValidator.java @@ -0,0 +1,34 @@ +package org.folio.dew.batch.bulkedit.jobs.permissions.check; + +import lombok.RequiredArgsConstructor; +import lombok.extern.log4j.Log4j2; +import org.folio.dew.domain.dto.EntityType; +import org.folio.spring.FolioExecutionContext; +import org.springframework.stereotype.Component; + +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.BULK_EDIT_INVENTORY_VIEW_PERMISSION; +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.BULK_EDIT_USERS_VIEW_PERMISSION; + +@Component +@RequiredArgsConstructor +@Log4j2 +public class PermissionsValidator { + + private final PermissionsProvider permissionsProvider; + private final RequiredPermissionResolver requiredPermissionResolver; + private final FolioExecutionContext folioExecutionContext; + + public boolean isBulkEditReadPermissionExists(String tenantId, EntityType entityType) { + var readPermissionForEntity = requiredPermissionResolver.getReadPermission(entityType); + var userPermissions = permissionsProvider.getUserPermissions(tenantId, folioExecutionContext.getUserId().toString()); + var isReadPermissionsExist = false; + if (entityType == EntityType.USER) { + isReadPermissionsExist = userPermissions.contains(readPermissionForEntity.getValue()) && userPermissions.contains(BULK_EDIT_USERS_VIEW_PERMISSION.getValue()); + } else { + isReadPermissionsExist = userPermissions.contains(readPermissionForEntity.getValue()) && userPermissions.contains(BULK_EDIT_INVENTORY_VIEW_PERMISSION.getValue()); + } + log.info("isBulkEditReadPermissionExists:: user {} has read permissions {} for {} in tenant {}", folioExecutionContext.getUserId(), + isReadPermissionsExist, entityType, tenantId); + return isReadPermissionsExist; + } +} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/RequiredPermissionResolver.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/RequiredPermissionResolver.java new file mode 100644 index 000000000..bbc0ea46b --- /dev/null +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/RequiredPermissionResolver.java @@ -0,0 +1,22 @@ +package org.folio.dew.batch.bulkedit.jobs.permissions.check; + +import org.folio.dew.domain.dto.EntityType; +import org.springframework.stereotype.Component; + +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.INVENTORY_INSTANCES_ITEM_GET_PERMISSION; +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.INVENTORY_ITEMS_ITEM_GET_PERMISSION; +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.INVENTORY_STORAGE_HOLDINGS_ITEM_GET_PERMISSION; +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.USER_ITEM_GET_PERMISSION; + +@Component +public class RequiredPermissionResolver { + + public PermissionEnum getReadPermission(EntityType entityType) { + return switch (entityType) { + case USER -> USER_ITEM_GET_PERMISSION; + case ITEM -> INVENTORY_ITEMS_ITEM_GET_PERMISSION; + case HOLDINGS_RECORD -> INVENTORY_STORAGE_HOLDINGS_ITEM_GET_PERMISSION; + case INSTANCE -> INVENTORY_INSTANCES_ITEM_GET_PERMISSION; + }; + } +} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/UserPermissions.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/UserPermissions.java new file mode 100644 index 000000000..c2a2c2938 --- /dev/null +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/UserPermissions.java @@ -0,0 +1,25 @@ +package org.folio.dew.batch.bulkedit.jobs.permissions.check; + +import com.fasterxml.jackson.annotation.JsonProperty; +import lombok.AllArgsConstructor; +import lombok.Builder; +import lombok.Data; +import lombok.NoArgsConstructor; +import lombok.With; + +import java.util.ArrayList; +import java.util.List; + +@Data +@With +@Builder(toBuilder = true) +@NoArgsConstructor +@AllArgsConstructor +public class UserPermissions { + + @JsonProperty("permissionNames") + private List permissionNames = new ArrayList<>(); + + @JsonProperty("permissions") + private List permissions = new ArrayList<>(); +} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditHoldingsIdentifiersJobConfig.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditHoldingsIdentifiersJobConfig.java index ba1c1b2b2..69fb38867 100644 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditHoldingsIdentifiersJobConfig.java +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditHoldingsIdentifiersJobConfig.java @@ -2,7 +2,6 @@ import static org.folio.dew.domain.dto.EntityType.HOLDINGS_RECORD; import static org.folio.dew.domain.dto.JobParameterNames.TEMP_LOCAL_FILE_PATH; -import static org.folio.dew.utils.Constants.CHUNKS; import static org.folio.dew.utils.Constants.JOB_NAME_POSTFIX_SEPARATOR; import lombok.RequiredArgsConstructor; @@ -23,12 +22,14 @@ import org.springframework.batch.core.launch.support.RunIdIncrementer; import org.springframework.batch.core.repository.JobRepository; import org.springframework.batch.core.step.builder.StepBuilder; -import org.springframework.batch.item.file.FlatFileItemReader; import org.springframework.batch.item.support.CompositeItemWriter; +import org.springframework.batch.item.support.SynchronizedItemStreamReader; +import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.core.io.FileSystemResource; +import org.springframework.core.task.TaskExecutor; import org.springframework.transaction.PlatformTransactionManager; import java.util.Arrays; @@ -41,6 +42,9 @@ public class BulkEditHoldingsIdentifiersJobConfig { private final BulkEditSkipListener bulkEditSkipListener; private final LocalFilesStorage localFilesStorage; + @Value("${application.chunks}") + private int chunks; + @Bean public Job bulkEditProcessHoldingsIdentifiersJob(JobCompletionNotificationListener listener, JobRepository jobRepository, @@ -54,12 +58,12 @@ public Job bulkEditProcessHoldingsIdentifiersJob(JobCompletionNotificationListen } @Bean - public Step bulkEditHoldingsStep(FlatFileItemReader csvItemIdentifierReader, - CompositeItemWriter> compositeHoldingsListWriter, - ListIdentifiersWriteListener listIdentifiersWriteListener, JobRepository jobRepository, - PlatformTransactionManager transactionManager) { + public Step bulkEditHoldingsStep(SynchronizedItemStreamReader csvItemIdentifierReader, + CompositeItemWriter> compositeHoldingsListWriter, + ListIdentifiersWriteListener listIdentifiersWriteListener, JobRepository jobRepository, + PlatformTransactionManager transactionManager, @Qualifier("asyncTaskExecutorBulkEdit") TaskExecutor taskExecutor) { return new StepBuilder("bulkEditHoldingsStep", jobRepository) - .> chunk(CHUNKS, transactionManager) + .> chunk(chunks, transactionManager) .reader(csvItemIdentifierReader) .processor(bulkEditHoldingsProcessor) .faultTolerant() @@ -69,6 +73,7 @@ public Step bulkEditHoldingsStep(FlatFileItemReader csvItemIdent .listener(bulkEditSkipListener) .writer(compositeHoldingsListWriter) .listener(listIdentifiersWriteListener) + .taskExecutor(taskExecutor) .build(); } diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditInstanceIdentifiersJobConfig.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditInstanceIdentifiersJobConfig.java index 986cd5890..d829a75ad 100644 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditInstanceIdentifiersJobConfig.java +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditInstanceIdentifiersJobConfig.java @@ -21,12 +21,14 @@ import org.springframework.batch.core.launch.support.RunIdIncrementer; import org.springframework.batch.core.repository.JobRepository; import org.springframework.batch.core.step.builder.StepBuilder; -import org.springframework.batch.item.file.FlatFileItemReader; import org.springframework.batch.item.support.CompositeItemWriter; +import org.springframework.batch.item.support.SynchronizedItemStreamReader; +import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.core.io.FileSystemResource; +import org.springframework.core.task.TaskExecutor; import org.springframework.transaction.PlatformTransactionManager; import java.util.Arrays; @@ -35,7 +37,6 @@ import static org.folio.dew.domain.dto.EntityType.INSTANCE; import static org.folio.dew.domain.dto.JobParameterNames.TEMP_LOCAL_FILE_PATH; import static org.folio.dew.domain.dto.JobParameterNames.TEMP_LOCAL_MARC_PATH; -import static org.folio.dew.utils.Constants.CHUNKS; import static org.folio.dew.utils.Constants.JOB_NAME_POSTFIX_SEPARATOR; @Configuration @@ -46,6 +47,9 @@ public class BulkEditInstanceIdentifiersJobConfig { private final SrsClient srsClient; private final JsonToMarcConverter jsonToMarcConverter; + @Value("${application.chunks}") + private int chunks; + @Bean public Job bulkEditProcessInstanceIdentifiersJob(JobCompletionNotificationListener listener, Step bulkEditInstanceStep, JobRepository jobRepository) { @@ -58,12 +62,12 @@ public Job bulkEditProcessInstanceIdentifiersJob(JobCompletionNotificationListen } @Bean - public Step bulkEditInstanceStep(FlatFileItemReader csvItemIdentifierReader, - CompositeItemWriter> compositeInstanceListWriter, - ListIdentifiersWriteListener listIdentifiersWriteListener, JobRepository jobRepository, - PlatformTransactionManager transactionManager) { + public Step bulkEditInstanceStep(SynchronizedItemStreamReader csvItemIdentifierReader, + CompositeItemWriter> compositeInstanceListWriter, + ListIdentifiersWriteListener listIdentifiersWriteListener, JobRepository jobRepository, + PlatformTransactionManager transactionManager, @Qualifier("asyncTaskExecutorBulkEdit") TaskExecutor taskExecutor) { return new StepBuilder("bulkEditInstanceStep", jobRepository) - .> chunk(CHUNKS, transactionManager) + .> chunk(chunks, transactionManager) .reader(csvItemIdentifierReader) .processor(bulkEditInstanceProcessor) .faultTolerant() @@ -73,6 +77,7 @@ public Step bulkEditInstanceStep(FlatFileItemReader csvItemIdent .listener(bulkEditSkipListener) .writer(compositeInstanceListWriter) .listener(listIdentifiersWriteListener) + .taskExecutor(taskExecutor) .build(); } diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditItemIdentifiersJobConfig.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditItemIdentifiersJobConfig.java index 1f3f9746e..a71578f5b 100644 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditItemIdentifiersJobConfig.java +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditItemIdentifiersJobConfig.java @@ -2,7 +2,6 @@ import static org.folio.dew.domain.dto.EntityType.ITEM; import static org.folio.dew.domain.dto.JobParameterNames.TEMP_LOCAL_FILE_PATH; -import static org.folio.dew.utils.Constants.CHUNKS; import static org.folio.dew.utils.Constants.JOB_NAME_POSTFIX_SEPARATOR; import lombok.RequiredArgsConstructor; @@ -22,13 +21,15 @@ import org.springframework.batch.core.launch.support.RunIdIncrementer; import org.springframework.batch.core.repository.JobRepository; import org.springframework.batch.core.step.builder.StepBuilder; -import org.springframework.batch.item.file.FlatFileItemReader; import org.springframework.batch.item.support.CompositeItemProcessor; import org.springframework.batch.item.support.CompositeItemWriter; +import org.springframework.batch.item.support.SynchronizedItemStreamReader; +import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.core.io.FileSystemResource; +import org.springframework.core.task.TaskExecutor; import org.springframework.transaction.PlatformTransactionManager; import java.util.Arrays; @@ -41,6 +42,9 @@ public class BulkEditItemIdentifiersJobConfig { private final ItemFetcher itemFetcher; private final BulkEditSkipListener bulkEditSkipListener; + @Value("${application.chunks}") + private int chunks; + @Bean public Job bulkEditProcessItemIdentifiersJob(JobCompletionNotificationListener listener, Step bulkEditItemStep, JobRepository jobRepository) { @@ -53,12 +57,12 @@ public Job bulkEditProcessItemIdentifiersJob(JobCompletionNotificationListener l } @Bean - public Step bulkEditItemStep(FlatFileItemReader csvItemIdentifierReader, - CompositeItemWriter> compositeItemListWriter, - ListIdentifiersWriteListener listIdentifiersWriteListener, JobRepository jobRepository, - PlatformTransactionManager transactionManager) { + public Step bulkEditItemStep(SynchronizedItemStreamReader csvItemIdentifierReader, + CompositeItemWriter> compositeItemListWriter, + ListIdentifiersWriteListener listIdentifiersWriteListener, JobRepository jobRepository, + PlatformTransactionManager transactionManager, @Qualifier("asyncTaskExecutorBulkEdit") TaskExecutor taskExecutor) { return new StepBuilder("bulkEditItemStep", jobRepository) - .> chunk(CHUNKS, transactionManager) + .> chunk(chunks, transactionManager) .reader(csvItemIdentifierReader) .processor(identifierItemProcessor()) .faultTolerant() @@ -68,6 +72,7 @@ public Step bulkEditItemStep(FlatFileItemReader csvItemIdentifie .listener(bulkEditSkipListener) .writer(compositeItemListWriter) .listener(listIdentifiersWriteListener) + .taskExecutor(taskExecutor) .build(); } diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditUserIdentifiersJobConfig.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditUserIdentifiersJobConfig.java index 1b719c6c7..a2a6bf678 100644 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditUserIdentifiersJobConfig.java +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/BulkEditUserIdentifiersJobConfig.java @@ -4,7 +4,6 @@ import static org.folio.dew.domain.dto.JobParameterNames.TEMP_LOCAL_FILE_PATH; import static org.folio.dew.domain.dto.UserFormat.getUserColumnHeaders; import static org.folio.dew.domain.dto.UserFormat.getUserFieldsArray; -import static org.folio.dew.utils.Constants.CHUNKS; import static org.folio.dew.utils.Constants.JOB_NAME_POSTFIX_SEPARATOR; import lombok.RequiredArgsConstructor; @@ -24,13 +23,15 @@ import org.springframework.batch.core.launch.support.RunIdIncrementer; import org.springframework.batch.core.repository.JobRepository; import org.springframework.batch.core.step.builder.StepBuilder; -import org.springframework.batch.item.file.FlatFileItemReader; import org.springframework.batch.item.support.CompositeItemProcessor; import org.springframework.batch.item.support.CompositeItemWriter; +import org.springframework.batch.item.support.SynchronizedItemStreamReader; +import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.core.io.FileSystemResource; +import org.springframework.core.task.TaskExecutor; import org.springframework.transaction.PlatformTransactionManager; import java.util.Arrays; @@ -42,6 +43,9 @@ public class BulkEditUserIdentifiersJobConfig { private final UserFetcher userFetcher; private final BulkEditSkipListener bulkEditSkipListener; + @Value("${application.chunks}") + private int chunks; + @Bean public Job bulkEditProcessUserIdentifiersJob(JobCompletionNotificationListener listener, Step bulkEditUserStep, JobRepository jobRepository) { @@ -54,12 +58,12 @@ public Job bulkEditProcessUserIdentifiersJob(JobCompletionNotificationListener l } @Bean - public Step bulkEditUserStep(FlatFileItemReader csvItemIdentifierReader, - CompositeItemWriter compositeItemWriter, - IdentifiersWriteListener identifiersWriteListener, JobRepository jobRepository, - PlatformTransactionManager transactionManager) { + public Step bulkEditUserStep(SynchronizedItemStreamReader csvItemIdentifierReader, + CompositeItemWriter compositeItemWriter, + IdentifiersWriteListener identifiersWriteListener, JobRepository jobRepository, + PlatformTransactionManager transactionManager, @Qualifier("asyncTaskExecutorBulkEdit") TaskExecutor taskExecutor) { return new StepBuilder("bulkEditUserStep", jobRepository) - . chunk(CHUNKS, transactionManager) + . chunk(chunks, transactionManager) .reader(csvItemIdentifierReader) .processor(identifierUserProcessor()) .faultTolerant() @@ -69,6 +73,7 @@ public Step bulkEditUserStep(FlatFileItemReader csvItemIdentifie .listener(bulkEditSkipListener) .writer(compositeItemWriter) .listener(identifiersWriteListener) + .taskExecutor(taskExecutor) .build(); } diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/IdentifiersConfig.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/IdentifiersConfig.java index ea6645d04..1d2424edc 100644 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/IdentifiersConfig.java +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/IdentifiersConfig.java @@ -10,6 +10,7 @@ import org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper; import org.springframework.batch.item.file.mapping.DefaultLineMapper; import org.springframework.batch.item.file.transform.DelimitedLineTokenizer; +import org.springframework.batch.item.support.SynchronizedItemStreamReader; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @@ -19,7 +20,16 @@ public class IdentifiersConfig { @Bean @StepScope - public FlatFileItemReader csvItemIdentifierReader( + public SynchronizedItemStreamReader csvItemIdentifierReader( + @Value("#{jobParameters['" + TEMP_IDENTIFIERS_FILE_NAME + "']}") String uploadedFileName) { + var reader = new SynchronizedItemStreamReader(); + reader.setDelegate(synchronizedCsvItemIdentifierReader1(uploadedFileName)); + return reader; + } + + @Bean + @StepScope + public FlatFileItemReader synchronizedCsvItemIdentifierReader1( @Value("#{jobParameters['" + TEMP_IDENTIFIERS_FILE_NAME + "']}") String uploadedFileName) { return new FlatFileItemReaderBuilder() .name("userItemIdentifierReader") diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/IdentifiersWriteListener.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/IdentifiersWriteListener.java index ce9100e28..01f946d1c 100644 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/IdentifiersWriteListener.java +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/IdentifiersWriteListener.java @@ -3,7 +3,6 @@ import static java.util.Objects.isNull; import static java.util.Objects.nonNull; import static org.folio.dew.domain.dto.JobParameterNames.JOB_ID; -import static org.folio.dew.utils.Constants.CHUNKS; import static org.folio.dew.utils.Constants.NUMBER_OF_WRITTEN_RECORDS; import static org.folio.dew.utils.Constants.TOTAL_CSV_LINES; @@ -18,7 +17,7 @@ import org.springframework.batch.core.BatchStatus; import org.springframework.batch.core.ItemWriteListener; import org.springframework.batch.core.JobExecution; -import org.springframework.batch.core.configuration.annotation.JobScope; +import org.springframework.batch.core.configuration.annotation.StepScope; import org.springframework.batch.item.Chunk; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Component; @@ -29,15 +28,18 @@ import java.util.concurrent.atomic.AtomicLong; @Component -@JobScope +@StepScope @RequiredArgsConstructor @Log4j2 public class IdentifiersWriteListener implements ItemWriteListener { private final KafkaService kafka; - @Value("#{jobExecution}") + @Value("#{stepExecution.jobExecution}") private JobExecution jobExecution; + @Value("${application.chunks}") + private int chunks; + private AtomicInteger processedRecords = new AtomicInteger(); private AtomicLong processedIdentifiers = new AtomicLong(); @@ -50,9 +52,9 @@ public void beforeWrite(Chunk list) { @Override public void afterWrite(Chunk list) { - bulkEditStatisticService.incrementSuccess(list.size()); var job = new Job(); job.setId(UUID.fromString(jobExecution.getJobParameters().getString(JOB_ID))); + bulkEditStatisticService.incrementSuccess(job.getId().toString(), list.size()); job.setType(ExportType.BULK_EDIT_IDENTIFIERS); job.setEntityType(EntityType.fromValue(jobExecution.getJobInstance().getJobName().split("-")[1])); job.setBatchStatus(BatchStatus.STARTED); @@ -61,7 +63,7 @@ public void afterWrite(Chunk list) { log.info("afterWrite:: update job by id {} after write for identifiers", job.getId()); var totalCsvLines = jobExecution.getJobParameters().getLong(TOTAL_CSV_LINES); - var processed = processedIdentifiers.addAndGet(CHUNKS); + var processed = processedIdentifiers.addAndGet(chunks); if (nonNull(totalCsvLines) && processed > totalCsvLines) { processed = totalCsvLines; } @@ -69,7 +71,7 @@ public void afterWrite(Chunk list) { progress.setTotal(isNull(totalCsvLines) ? 0 : totalCsvLines.intValue()); progress.setProcessed((int) processed); progress.setProgress(isNull(totalCsvLines) ? 0 : calculateProgress(processed, totalCsvLines)); - progress.setSuccess(bulkEditStatisticService.getStatistic().getSuccess()); + progress.setSuccess(bulkEditStatisticService.getSuccess(job.getId().toString())); job.setProgress(progress); jobExecution.getExecutionContext().putLong(NUMBER_OF_WRITTEN_RECORDS, processedRecords.longValue()); @@ -78,7 +80,7 @@ public void afterWrite(Chunk list) { } private int calculateProgress(long processed, long total) { - if (total <= CHUNKS) { + if (total <= chunks) { return 90; } var res = (double) processed / total * 100; diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/InstanceFetcher.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/InstanceFetcher.java index 54fc3ba6f..9421c880c 100644 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/InstanceFetcher.java +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/InstanceFetcher.java @@ -1,20 +1,26 @@ package org.folio.dew.batch.bulkedit.jobs.processidentifiers; +import static java.lang.String.format; import static org.folio.dew.domain.dto.IdentifierType.HOLDINGS_RECORD_ID; import static org.folio.dew.utils.BulkEditProcessorHelper.getMatchPattern; import static org.folio.dew.utils.BulkEditProcessorHelper.resolveIdentifier; import static org.folio.dew.utils.Constants.MULTIPLE_MATCHES_MESSAGE; +import static org.folio.dew.utils.Constants.NO_INSTANCE_VIEW_PERMISSIONS; import feign.codec.DecodeException; import lombok.RequiredArgsConstructor; import lombok.extern.log4j.Log4j2; +import org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionsValidator; import org.folio.dew.client.InventoryInstancesClient; +import org.folio.dew.client.UserClient; +import org.folio.dew.domain.dto.EntityType; import org.folio.dew.domain.dto.IdentifierType; import org.folio.dew.domain.dto.InstanceCollection; import org.folio.dew.domain.dto.ItemIdentifier; import org.folio.dew.error.BulkEditException; import org.folio.dew.service.InstanceReferenceService; import org.folio.dew.utils.ExceptionHelper; +import org.folio.spring.FolioExecutionContext; import org.jetbrains.annotations.NotNull; import org.springframework.batch.core.configuration.annotation.StepScope; import org.springframework.batch.item.ItemProcessor; @@ -31,6 +37,9 @@ public class InstanceFetcher implements ItemProcessor { private final InventoryInstancesClient inventoryInstancesClient; private final InstanceReferenceService instanceReferenceService; + private final FolioExecutionContext folioExecutionContext; + private final PermissionsValidator permissionsValidator; + private final UserClient userClient; @Value("#{jobParameters['identifierType']}") private String identifierType; @@ -39,6 +48,10 @@ public class InstanceFetcher implements ItemProcessor identifiersToCheckDuplication = new HashSet<>(); + private Set identifiersToCheckDuplication = ConcurrentHashMap.newKeySet(); @Override - public ExtendedItemCollection process(ItemIdentifier itemIdentifier) throws BulkEditException { + public synchronized ExtendedItemCollection process(ItemIdentifier itemIdentifier) throws BulkEditException { if (identifiersToCheckDuplication.contains(itemIdentifier)) { throw new BulkEditException("Duplicate entry"); } @@ -90,7 +95,9 @@ public ExtendedItemCollection process(ItemIdentifier itemIdentifier) throws Bulk if (HOLDINGSRECORDID != identifierTypeEnum && tenantIds.size() > 1) { throw new BulkEditException(DUPLICATES_ACROSS_TENANTS); } - tenantIds.forEach(tenantId -> { + var affiliatedPermittedTenants = tenantResolver.getAffiliatedPermittedTenantIds(EntityType.ITEM, + jobExecution, identifierType, tenantIds, itemIdentifier); + affiliatedPermittedTenants.forEach(tenantId -> { try (var context = new FolioExecutionContextSetter(refreshAndGetFolioExecutionContext(tenantId, folioExecutionContext))) { var url = format(getMatchPattern(identifierType), idType, identifier); var itemCollection = inventoryClient.getItemByQuery(url, Integer.MAX_VALUE); @@ -103,12 +110,8 @@ public ExtendedItemCollection process(ItemIdentifier itemIdentifier) throws Bulk ); extendedItemCollection.setTotalRecords(extendedItemCollection.getTotalRecords() + itemCollection.getTotalRecords()); } catch (Exception e) { - if (e instanceof FeignException && ((FeignException) e).status() == 401) { - var user = userClient.getUserById(folioExecutionContext.getUserId().toString()); - throw new BulkEditException(format(NO_ITEM_AFFILIATION, user.getUsername(), idType, identifier, tenantId)); - } else { - throw e; - } + log.error(e.getMessage()); + throw e; } }); } else { @@ -116,6 +119,7 @@ public ExtendedItemCollection process(ItemIdentifier itemIdentifier) throws Bulk } } else { // Process local tenant case + checkReadPermissions(folioExecutionContext.getTenantId(), identifier); var url = format(getMatchPattern(identifierType), idType, identifier); var currentTenantId = folioExecutionContext.getTenantId(); var itemCollection = inventoryClient.getItemByQuery(url, Integer.MAX_VALUE); @@ -126,6 +130,10 @@ public ExtendedItemCollection process(ItemIdentifier itemIdentifier) throws Bulk extendedItemCollection.setExtendedItems(itemCollection.getItems().stream() .map(item -> new ExtendedItem().tenantId(folioExecutionContext.getTenantId()).entity(item)).toList()); extendedItemCollection.setTotalRecords(itemCollection.getTotalRecords()); + if (extendedItemCollection.getExtendedItems().isEmpty()) { + log.error(NO_MATCH_FOUND_MESSAGE); + throw new BulkEditException(NO_MATCH_FOUND_MESSAGE); + } } return extendedItemCollection; } catch (DecodeException e) { @@ -133,6 +141,13 @@ public ExtendedItemCollection process(ItemIdentifier itemIdentifier) throws Bulk } } + private void checkReadPermissions(String tenantId, String identifier) { + if (!permissionsValidator.isBulkEditReadPermissionExists(tenantId, EntityType.ITEM)) { + var user = userClient.getUserById(folioExecutionContext.getUserId().toString()); + throw new BulkEditException(format(NO_ITEM_VIEW_PERMISSIONS, user.getUsername(), resolveIdentifier(identifierType), identifier, tenantId)); + } + } + private boolean isCurrentTenantCentral(String centralTenantId) { return StringUtils.isNotEmpty(centralTenantId) && centralTenantId.equals(folioExecutionContext.getTenantId()); } diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/UserFetcher.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/UserFetcher.java index bd0aa4ba5..414e93ba1 100644 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/UserFetcher.java +++ b/src/main/java/org/folio/dew/batch/bulkedit/jobs/processidentifiers/UserFetcher.java @@ -1,45 +1,62 @@ package org.folio.dew.batch.bulkedit.jobs.processidentifiers; +import static java.lang.String.format; import static org.folio.dew.utils.BulkEditProcessorHelper.resolveIdentifier; import static org.folio.dew.utils.Constants.MULTIPLE_MATCHES_MESSAGE; import static org.folio.dew.utils.Constants.NO_MATCH_FOUND_MESSAGE; +import static org.folio.dew.utils.Constants.NO_USER_VIEW_PERMISSIONS; import feign.codec.DecodeException; import lombok.RequiredArgsConstructor; import lombok.extern.log4j.Log4j2; +import org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionsValidator; import org.folio.dew.client.UserClient; +import org.folio.dew.domain.dto.EntityType; import org.folio.dew.domain.dto.ItemIdentifier; import org.folio.dew.domain.dto.User; import org.folio.dew.error.BulkEditException; import org.folio.dew.utils.ExceptionHelper; +import org.folio.spring.FolioExecutionContext; import org.springframework.batch.core.configuration.annotation.StepScope; import org.springframework.batch.item.ItemProcessor; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Component; -import java.util.HashSet; import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; @Component @StepScope @RequiredArgsConstructor @Log4j2 public class UserFetcher implements ItemProcessor { + private static final String USER_SEARCH_QUERY = "(cql.allRecords=1 NOT type=\"\" or type<>\"shadow\") and %s==\"%s\""; + private final UserClient userClient; @Value("#{jobParameters['identifierType']}") private String identifierType; - private Set identifiersToCheckDuplication = new HashSet<>(); + private Set identifiersToCheckDuplication = ConcurrentHashMap.newKeySet(); + private final FolioExecutionContext folioExecutionContext; + private final PermissionsValidator permissionsValidator; @Override - public User process(ItemIdentifier itemIdentifier) throws BulkEditException { + public synchronized User process(ItemIdentifier itemIdentifier) throws BulkEditException { + if (!permissionsValidator.isBulkEditReadPermissionExists(folioExecutionContext.getTenantId(), EntityType.USER)) { + var user = userClient.getUserById(folioExecutionContext.getUserId().toString()); + throw new BulkEditException(format(NO_USER_VIEW_PERMISSIONS, user.getUsername(), resolveIdentifier(identifierType), itemIdentifier.getItemId(), folioExecutionContext.getTenantId())); + } if (identifiersToCheckDuplication.contains(itemIdentifier)) { throw new BulkEditException("Duplicate entry"); } identifiersToCheckDuplication.add(itemIdentifier); try { var limit = 1; - var userCollection = userClient.getUserByQuery(String.format("%s==\"%s\"", resolveIdentifier(identifierType), itemIdentifier.getItemId()), limit); + var userCollection = userClient.getUserByQuery( + String.format(USER_SEARCH_QUERY, resolveIdentifier(identifierType), itemIdentifier.getItemId()), + limit + ); + if (userCollection.getUsers().isEmpty()) { throw new BulkEditException(NO_MATCH_FOUND_MESSAGE); } else if (userCollection.getTotalRecords() > limit) { diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditFilterUserRecordsForRollBackProcessor.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditFilterUserRecordsForRollBackProcessor.java deleted file mode 100644 index 3ea4fb2ee..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditFilterUserRecordsForRollBackProcessor.java +++ /dev/null @@ -1,34 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.rollbackjob; - -import lombok.RequiredArgsConstructor; -import lombok.Setter; -import org.folio.dew.domain.dto.User; -import org.folio.dew.domain.dto.UserFormat; -import org.folio.dew.service.BulkEditParseService; -import org.folio.dew.service.BulkEditRollBackService; -import org.springframework.batch.core.configuration.annotation.JobScope; -import org.springframework.batch.item.ItemProcessor; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.stereotype.Component; - -import java.util.UUID; - -@Component -@JobScope -@RequiredArgsConstructor -public class BulkEditFilterUserRecordsForRollBackProcessor implements ItemProcessor { - - @Value("#{jobParameters['jobId']}") - @Setter - private String jobId; - private final BulkEditRollBackService bulkEditRollBackService; - private final BulkEditParseService bulkEditParseService; - - @Override - public User process(UserFormat userFormat) { - if (bulkEditRollBackService.isUserBeRollBack(userFormat.getId(), UUID.fromString(jobId))) { - return bulkEditParseService.mapUserFormatToUser(userFormat); - } - return null; - } -} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditUpdateUserRecordsAfterRollBackJobConfig.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditUpdateUserRecordsAfterRollBackJobConfig.java deleted file mode 100644 index b5ac4967c..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditUpdateUserRecordsAfterRollBackJobConfig.java +++ /dev/null @@ -1,71 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.rollbackjob; - -import org.folio.dew.batch.bulkedit.jobs.JobConfigReaderHelper; -import org.folio.dew.domain.dto.User; -import org.folio.dew.domain.dto.UserFormat; -import org.folio.dew.repository.LocalFilesStorage; -import org.folio.dew.repository.S3CompatibleResource; -import org.springframework.batch.core.Job; -import org.springframework.batch.core.Step; -import org.springframework.batch.core.configuration.annotation.StepScope; -import org.springframework.batch.core.job.builder.JobBuilder; -import org.springframework.batch.core.launch.support.RunIdIncrementer; -import org.springframework.batch.core.repository.JobRepository; -import org.springframework.batch.core.step.builder.StepBuilder; -import org.springframework.batch.item.ItemProcessor; -import org.springframework.batch.item.ItemReader; -import org.springframework.batch.item.ItemWriter; -import org.springframework.batch.item.file.FlatFileItemReader; -import org.springframework.batch.item.file.builder.FlatFileItemReaderBuilder; -import org.springframework.beans.factory.annotation.Qualifier; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.context.annotation.Bean; -import org.springframework.context.annotation.Configuration; -import org.springframework.transaction.PlatformTransactionManager; - -@Configuration -public class BulkEditUpdateUserRecordsAfterRollBackJobConfig { - - @Bean - public Job bulkEditRollBackJob( - BulkEditUpdateUserRecordsAfterRollBackListener listener, - Step bulkEditRollBackRecordsStep, - JobRepository jobRepository) { - return new JobBuilder("BULK_EDIT_ROLL_BACK", jobRepository) - .incrementer(new RunIdIncrementer()) - .listener(listener) - .flow(bulkEditRollBackRecordsStep) - .end() - .build(); - } - - @Bean - public Step bulkEditRollBackRecordsStep( - @Qualifier("bulkEditRollBackReader") - ItemReader reader, - @Qualifier("bulkEditFilterUserRecordsForRollBackProcessor") - ItemProcessor processor, - @Qualifier("bulkEditUpdateUserRecordsForRollBackWriter") - ItemWriter writer, - JobRepository jobRepository, - PlatformTransactionManager transactionManager) { - return new StepBuilder("bulkEditRollBackRecordsStep", jobRepository) - .chunk(1, transactionManager) - .reader(reader) - .processor(processor) - .writer(writer) - .build(); - } - - @Bean - @StepScope - public FlatFileItemReader bulkEditRollBackReader(@Value("#{jobParameters['fileName']}") String fileName, LocalFilesStorage localFilesStorage) { - var userLineMapper = JobConfigReaderHelper.createLineMapper(UserFormat.class, UserFormat.getUserFieldsArray()); - return new FlatFileItemReaderBuilder() - .name("bulkEditRollBackReader") - .resource(new S3CompatibleResource<>(fileName, localFilesStorage)) - .linesToSkip(1) - .lineMapper(userLineMapper) - .build(); - } -} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditUpdateUserRecordsAfterRollBackListener.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditUpdateUserRecordsAfterRollBackListener.java deleted file mode 100644 index e7e0f7700..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditUpdateUserRecordsAfterRollBackListener.java +++ /dev/null @@ -1,34 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.rollbackjob; - -import lombok.RequiredArgsConstructor; -import org.apache.commons.io.FileUtils; -import org.folio.dew.service.BulkEditRollBackService; -import org.springframework.batch.core.JobExecution; -import org.springframework.batch.core.JobExecutionListener; -import org.springframework.batch.core.configuration.annotation.JobScope; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.stereotype.Component; - -import java.io.File; -import java.util.UUID; - -@Component -@JobScope -@RequiredArgsConstructor -public class BulkEditUpdateUserRecordsAfterRollBackListener implements JobExecutionListener { - - @Value("#{jobParameters['jobId']}") - private String jobId; - @Value("#{jobParameters['fileName']}") - private String fileName; - private final BulkEditRollBackService bulkEditRollBackService; - - @Override - public void beforeJob(JobExecution jobExecution) {} - - @Override - public void afterJob(JobExecution jobExecution) { - bulkEditRollBackService.cleanJobData(UUID.fromString(jobId)); - FileUtils.deleteQuietly(new File(fileName)); - } -} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditUpdateUserRecordsForRollBackWriter.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditUpdateUserRecordsForRollBackWriter.java deleted file mode 100644 index ed9873738..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditUpdateUserRecordsForRollBackWriter.java +++ /dev/null @@ -1,34 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.rollbackjob; - -import lombok.RequiredArgsConstructor; -import lombok.extern.log4j.Log4j2; -import org.folio.dew.client.UserClient; -import org.folio.dew.domain.dto.User; -import org.springframework.batch.core.configuration.annotation.JobScope; -import org.springframework.batch.item.Chunk; -import org.springframework.batch.item.ItemWriter; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.stereotype.Component; - -@Component -@JobScope -@RequiredArgsConstructor -@Log4j2 -public class BulkEditUpdateUserRecordsForRollBackWriter implements ItemWriter { - - private final UserClient userClient; - @Value("#{jobParameters['jobId']}") - private String jobId; - - @Override - public void write(Chunk items) throws Exception { - items.forEach(user -> { - try { - userClient.updateUser(user, user.getId()); - log.info("Rollback user with id - {} from updating by job {}", user.getId(), jobId); - } catch (Exception e) { - log.info("Cannot rollback user with id {}. Reason: {}", user.getId(), e.getMessage()); - } - }); - } -} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateHoldingsRecordsJobConfig.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateHoldingsRecordsJobConfig.java deleted file mode 100644 index 50df4463e..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateHoldingsRecordsJobConfig.java +++ /dev/null @@ -1,74 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.updatejob; - -import org.folio.dew.batch.JobCompletionNotificationListener; -import org.folio.dew.batch.bulkedit.jobs.JobConfigReaderHelper; -import org.folio.dew.domain.dto.HoldingsFormat; -import org.folio.dew.domain.dto.HoldingsRecord; -import org.folio.dew.repository.RemoteFilesStorage; -import org.springframework.batch.core.ItemWriteListener; -import org.springframework.batch.core.Job; -import org.springframework.batch.core.Step; -import org.springframework.batch.core.configuration.annotation.StepScope; -import org.springframework.batch.core.job.builder.JobBuilder; -import org.springframework.batch.core.launch.support.RunIdIncrementer; -import org.springframework.batch.core.repository.JobRepository; -import org.springframework.batch.core.step.builder.StepBuilder; -import org.springframework.batch.item.ItemProcessor; -import org.springframework.batch.item.ItemReader; -import org.springframework.batch.item.ItemWriter; -import org.springframework.batch.item.file.FlatFileItemReader; -import org.springframework.batch.item.file.builder.FlatFileItemReaderBuilder; -import org.springframework.beans.factory.annotation.Qualifier; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.context.annotation.Bean; -import org.springframework.context.annotation.Configuration; -import org.springframework.core.io.InputStreamResource; -import org.springframework.transaction.PlatformTransactionManager; - -import java.io.IOException; - -import static org.folio.dew.domain.dto.EntityType.HOLDINGS_RECORD; -import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_UPDATE; -import static org.folio.dew.domain.dto.JobParameterNames.UPDATED_FILE_NAME; -import static org.folio.dew.utils.Constants.JOB_NAME_POSTFIX_SEPARATOR; - -@Configuration public class BulkEditUpdateHoldingsRecordsJobConfig { - - @Bean public Job bulkEditUpdateHoldingsRecordsJob(Step bulkEditUpdateHoldingsRecordsStep, JobRepository jobRepository, - JobCompletionNotificationListener completionListener) { - return new JobBuilder(BULK_EDIT_UPDATE.getValue() + JOB_NAME_POSTFIX_SEPARATOR + HOLDINGS_RECORD.getValue(), jobRepository) - .incrementer(new RunIdIncrementer()) - .listener(completionListener) - .flow(bulkEditUpdateHoldingsRecordsStep) - .end() - .build(); - } - - @Bean public Step bulkEditUpdateHoldingsRecordsStep(ItemReader csvHoldingsRecordsReader, - @Qualifier("bulkEditUpdateHoldingsRecordsProcessor") ItemProcessor processor, - @Qualifier("updateHoldingsRecordsWriter") ItemWriter writer, - @Qualifier("updateRecordWriteListener") ItemWriteListener updateRecordWriteListener, - JobRepository jobRepository, PlatformTransactionManager transactionManager) { - return new StepBuilder("bulkEditUpdateHoldingsRecordsStep", jobRepository) - .chunk(10, transactionManager) - .reader(csvHoldingsRecordsReader) - .processor(processor) - .writer(writer) - .listener(updateRecordWriteListener) - .build(); - } - - @Bean - @StepScope - public FlatFileItemReader csvHoldingsRecordsReader( - @Value("#{jobParameters['" + UPDATED_FILE_NAME + "']}") String updatedFileName, - RemoteFilesStorage remoteFilesStorage) - throws IOException { - var holdingsLineMapper = JobConfigReaderHelper.createLineMapper(HoldingsFormat.class, HoldingsFormat.getHoldingsFieldsArray()); - return new FlatFileItemReaderBuilder().name("holdingsReader") - .resource(new InputStreamResource(remoteFilesStorage.newInputStream(updatedFileName))) - .linesToSkip(1) - .lineMapper(holdingsLineMapper) - .build(); - } -} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateHoldingsRecordsProcessor.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateHoldingsRecordsProcessor.java deleted file mode 100644 index 62ff51a57..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateHoldingsRecordsProcessor.java +++ /dev/null @@ -1,50 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.updatejob; - -import static org.folio.dew.utils.Constants.FILE_NAME; - -import lombok.RequiredArgsConstructor; -import lombok.extern.log4j.Log4j2; -import org.apache.commons.io.FilenameUtils; -import org.folio.dew.domain.dto.HoldingsFormat; -import org.folio.dew.domain.dto.HoldingsRecord; -import org.folio.dew.error.BulkEditException; -import org.folio.dew.service.BulkEditProcessingErrorsService; -import org.folio.dew.service.mapper.HoldingsMapper; -import org.springframework.batch.core.JobExecution; -import org.springframework.batch.core.configuration.annotation.JobScope; -import org.springframework.batch.item.ItemProcessor; -import org.springframework.beans.factory.annotation.Qualifier; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.stereotype.Component; - -@Log4j2 -@Component -@Qualifier("updateHoldingsRecordsProcessor") -@RequiredArgsConstructor -@JobScope -public class BulkEditUpdateHoldingsRecordsProcessor implements ItemProcessor { - - @Value("#{jobParameters['jobId']}") - private String jobId; - - @Value("#{jobParameters['identifierType']}") - private String identifierType; - - @Value("#{jobExecution}") - private JobExecution jobExecution; - - private final HoldingsMapper holdingsMapper; - private final BulkEditProcessingErrorsService bulkEditProcessingErrorsService; - - @Override - public HoldingsRecord process(HoldingsFormat holdingsFormat) throws Exception { - try { - return holdingsMapper.mapToHoldingsRecord(holdingsFormat); - } catch (Exception e) { - log.error("Error process holdings format {} : {}", holdingsFormat.getIdentifier(identifierType), e.getMessage()); - bulkEditProcessingErrorsService.saveErrorInCSV(jobId, holdingsFormat.getIdentifier(identifierType), new BulkEditException(e.getMessage()), FilenameUtils.getName(jobExecution.getJobParameters().getString(FILE_NAME))); - return null; - } - } - -} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateHoldingsRecordsWriter.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateHoldingsRecordsWriter.java deleted file mode 100644 index 7d060c0d5..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateHoldingsRecordsWriter.java +++ /dev/null @@ -1,50 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.updatejob; - -import lombok.RequiredArgsConstructor; -import lombok.extern.log4j.Log4j2; -import org.apache.commons.io.FilenameUtils; -import org.folio.dew.client.HoldingClient; -import org.folio.dew.domain.dto.HoldingsRecord; -import org.folio.dew.error.BulkEditException; -import org.folio.dew.service.BulkEditProcessingErrorsService; -import org.folio.dew.service.BulkEditStatisticService; -import org.springframework.batch.core.JobExecution; -import org.springframework.batch.core.configuration.annotation.JobScope; -import org.springframework.batch.item.Chunk; -import org.springframework.batch.item.ItemWriter; -import org.springframework.beans.factory.annotation.Qualifier; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.stereotype.Component; - -import static org.folio.dew.utils.Constants.FILE_NAME; - -@Log4j2 -@Component -@Qualifier("updateHoldingsRecordsWriter") -@RequiredArgsConstructor -@JobScope -public class BulkEditUpdateHoldingsRecordsWriter implements ItemWriter { - - @Value("#{jobParameters['jobId']}") - private String jobId; - @Value("#{jobExecution}") - private JobExecution jobExecution; - - private final HoldingClient holdingClient; - private final BulkEditProcessingErrorsService bulkEditProcessingErrorsService; - private final BulkEditStatisticService bulkEditStatisticService; - - @Override - public void write(Chunk holdingsRecords) throws Exception { - holdingsRecords.forEach(holdingsRecord -> { - try { - holdingClient.updateHoldingsRecord(holdingsRecord, holdingsRecord.getId()); - bulkEditStatisticService.incrementSuccess(); - log.info("Update holdings record with id - {} by job id {}", holdingsRecord.getId(), jobId); - } catch (Exception e) { - log.info("Cannot update holdings record with id {}. Reason: {}", holdingsRecord.getId(), e.getMessage()); - bulkEditProcessingErrorsService.saveErrorInCSV(jobId, holdingsRecord.getId(), new BulkEditException(e.getMessage()), FilenameUtils.getName(jobExecution.getJobParameters().getString(FILE_NAME))); - } - }); - } -} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateItemRecordsJobConfig.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateItemRecordsJobConfig.java deleted file mode 100644 index f709c0f46..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateItemRecordsJobConfig.java +++ /dev/null @@ -1,74 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.updatejob; - -import org.folio.dew.batch.JobCompletionNotificationListener; -import org.folio.dew.batch.bulkedit.jobs.JobConfigReaderHelper; -import org.folio.dew.domain.dto.Item; -import org.folio.dew.domain.dto.ItemFormat; -import org.folio.dew.repository.LocalFilesStorage; -import org.folio.dew.repository.S3CompatibleResource; -import org.springframework.batch.core.ItemWriteListener; -import org.springframework.batch.core.Job; -import org.springframework.batch.core.Step; -import org.springframework.batch.core.configuration.annotation.StepScope; -import org.springframework.batch.core.job.builder.JobBuilder; -import org.springframework.batch.core.launch.support.RunIdIncrementer; -import org.springframework.batch.core.repository.JobRepository; -import org.springframework.batch.core.step.builder.StepBuilder; -import org.springframework.batch.item.ItemProcessor; -import org.springframework.batch.item.ItemReader; -import org.springframework.batch.item.ItemWriter; -import org.springframework.batch.item.file.FlatFileItemReader; -import org.springframework.batch.item.file.LineMapper; -import org.springframework.batch.item.file.builder.FlatFileItemReaderBuilder; -import org.springframework.beans.factory.annotation.Qualifier; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.context.annotation.Bean; -import org.springframework.context.annotation.Configuration; -import org.springframework.transaction.PlatformTransactionManager; - -import static org.apache.commons.lang3.ObjectUtils.isEmpty; -import static org.folio.dew.domain.dto.EntityType.ITEM; -import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_UPDATE; -import static org.folio.dew.domain.dto.JobParameterNames.UPDATED_FILE_NAME; -import static org.folio.dew.utils.Constants.FILE_NAME; -import static org.folio.dew.utils.Constants.JOB_NAME_POSTFIX_SEPARATOR; - -@Configuration public class BulkEditUpdateItemRecordsJobConfig { - - @Bean public Job bulkEditUpdateItemRecordsJob(Step bulkEditUpdateItemRecordsStep, JobRepository jobRepository, - JobCompletionNotificationListener completionListener) { - return new JobBuilder(BULK_EDIT_UPDATE.getValue() + JOB_NAME_POSTFIX_SEPARATOR + ITEM.getValue(), jobRepository) - .incrementer(new RunIdIncrementer()) - .listener(completionListener) - .flow(bulkEditUpdateItemRecordsStep) - .end() - .build(); - } - - @Bean public Step bulkEditUpdateItemRecordsStep(ItemReader csvItemRecordsReader, - @Qualifier("bulkEditUpdateItemRecordsProcessor") ItemProcessor processor, - @Qualifier("updateItemRecordsWriter") ItemWriter writer, - @Qualifier("updateRecordWriteListener") ItemWriteListener updateRecordWriteListener, - JobRepository jobRepository, - PlatformTransactionManager transactionManager) { - return new StepBuilder("bulkEditUpdateRecordsStep", jobRepository) - .chunk(10, transactionManager) - .reader(csvItemRecordsReader) - .processor(processor) - .writer(writer) - .listener(updateRecordWriteListener) - .build(); - } - - @Bean @StepScope public FlatFileItemReader csvItemRecordsReader( - @Value("#{jobParameters['" + FILE_NAME + "']}") String fileName, - @Value("#{jobParameters['" + UPDATED_FILE_NAME + "']}") String updatedFileName, - LocalFilesStorage localFilesStorage) { - LineMapper itemLineMapper = JobConfigReaderHelper.createLineMapper(ItemFormat.class, ItemFormat.getItemFieldsArray()); - return new FlatFileItemReaderBuilder().name("itemReader") - .resource(new S3CompatibleResource<>(isEmpty(updatedFileName) ? fileName : updatedFileName, localFilesStorage)) - .linesToSkip(1) - .lineMapper(itemLineMapper) - .build(); - } -} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateItemRecordsProcessor.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateItemRecordsProcessor.java deleted file mode 100644 index ff7789d92..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateItemRecordsProcessor.java +++ /dev/null @@ -1,50 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.updatejob; - -import static org.folio.dew.utils.Constants.FILE_NAME; - -import lombok.RequiredArgsConstructor; -import lombok.extern.log4j.Log4j2; -import org.apache.commons.io.FilenameUtils; -import org.folio.dew.domain.dto.Item; -import org.folio.dew.domain.dto.ItemFormat; -import org.folio.dew.error.BulkEditException; -import org.folio.dew.service.BulkEditParseService; -import org.folio.dew.service.BulkEditProcessingErrorsService; -import org.springframework.batch.core.JobExecution; -import org.springframework.batch.core.configuration.annotation.JobScope; -import org.springframework.batch.item.ItemProcessor; -import org.springframework.beans.factory.annotation.Qualifier; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.stereotype.Component; - -@Log4j2 -@Component -@Qualifier("updateItemRecordsProcessor") -@RequiredArgsConstructor -@JobScope -public class BulkEditUpdateItemRecordsProcessor implements ItemProcessor { - - @Value("#{jobParameters['jobId']}") - private String jobId; - - @Value("#{jobParameters['identifierType']}") - private String identifierType; - - @Value("#{jobExecution}") - private JobExecution jobExecution; - - private final BulkEditParseService bulkEditParseService; - private final BulkEditProcessingErrorsService bulkEditProcessingErrorsService; - - @Override - public Item process(ItemFormat itemFormat) throws Exception { - try { - return bulkEditParseService.mapItemFormatToItem(itemFormat); - } catch (Exception e) { - log.error("Error process item format {} : {}", itemFormat.getIdentifier(identifierType), e.getMessage()); - bulkEditProcessingErrorsService.saveErrorInCSV(jobId, itemFormat.getIdentifier(identifierType), new BulkEditException(e.getMessage()), FilenameUtils.getName(jobExecution.getJobParameters().getString(FILE_NAME))); - return null; - } - } - -} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateItemRecordsWriter.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateItemRecordsWriter.java deleted file mode 100644 index e2a5f6ab5..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateItemRecordsWriter.java +++ /dev/null @@ -1,50 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.updatejob; - -import lombok.RequiredArgsConstructor; -import lombok.extern.log4j.Log4j2; -import org.apache.commons.io.FilenameUtils; -import org.folio.dew.client.InventoryClient; -import org.folio.dew.domain.dto.Item; -import org.folio.dew.error.BulkEditException; -import org.folio.dew.service.BulkEditProcessingErrorsService; -import org.folio.dew.service.BulkEditStatisticService; -import org.springframework.batch.core.JobExecution; -import org.springframework.batch.core.configuration.annotation.JobScope; -import org.springframework.batch.item.Chunk; -import org.springframework.batch.item.ItemWriter; -import org.springframework.beans.factory.annotation.Qualifier; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.stereotype.Component; - -import static org.folio.dew.utils.Constants.FILE_NAME; - -@Log4j2 -@Component -@Qualifier("updateItemRecordsWriter") -@RequiredArgsConstructor -@JobScope -public class BulkEditUpdateItemRecordsWriter implements ItemWriter { - - @Value("#{jobParameters['jobId']}") - private String jobId; - @Value("#{jobExecution}") - private JobExecution jobExecution; - - private final InventoryClient inventoryClient; - private final BulkEditProcessingErrorsService bulkEditProcessingErrorsService; - private final BulkEditStatisticService bulkEditStatisticService; - - @Override - public void write(Chunk items) throws Exception { - items.forEach(item -> { - try { - inventoryClient.updateItem(item, item.getId()); - bulkEditStatisticService.incrementSuccess(); - log.info("Update item with id - {} by job id {}", item.getId(), jobId); - } catch (Exception e) { - log.info("Cannot update item with id {}. Reason: {}", item.getId(), e.getMessage()); - bulkEditProcessingErrorsService.saveErrorInCSV(jobId, item.getId(), new BulkEditException(e.getMessage()), FilenameUtils.getName(jobExecution.getJobParameters().getString(FILE_NAME))); - } - }); - } -} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateUserRecordsJobConfig.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateUserRecordsJobConfig.java deleted file mode 100644 index b89923873..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateUserRecordsJobConfig.java +++ /dev/null @@ -1,90 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.updatejob; - -import org.folio.dew.batch.JobCompletionNotificationListener; -import org.folio.dew.batch.bulkedit.jobs.JobConfigReaderHelper; -import org.folio.dew.batch.bulkedit.jobs.updatejob.listeners.BulkEditUpdateUserRecordsListener; -import org.folio.dew.domain.dto.User; -import org.folio.dew.domain.dto.UserFormat; -import org.folio.dew.repository.LocalFilesStorage; -import org.folio.dew.repository.RemoteFilesStorage; -import org.folio.dew.repository.S3CompatibleResource; -import org.springframework.batch.core.ItemWriteListener; -import org.springframework.batch.core.Job; -import org.springframework.batch.core.Step; -import org.springframework.batch.core.configuration.annotation.StepScope; -import org.springframework.batch.core.job.builder.JobBuilder; -import org.springframework.batch.core.launch.support.RunIdIncrementer; -import org.springframework.batch.core.repository.JobRepository; -import org.springframework.batch.core.step.builder.StepBuilder; -import org.springframework.batch.item.ItemProcessor; -import org.springframework.batch.item.ItemReader; -import org.springframework.batch.item.ItemWriter; -import org.springframework.batch.item.file.FlatFileItemReader; -import org.springframework.batch.item.file.builder.FlatFileItemReaderBuilder; -import org.springframework.beans.factory.annotation.Qualifier; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.context.annotation.Bean; -import org.springframework.context.annotation.Configuration; -import org.springframework.core.io.InputStreamResource; -import org.springframework.transaction.PlatformTransactionManager; - -import java.io.IOException; - -import static org.apache.commons.lang3.ObjectUtils.isEmpty; -import static org.folio.dew.domain.dto.EntityType.USER; -import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_UPDATE; -import static org.folio.dew.domain.dto.JobParameterNames.UPDATED_FILE_NAME; -import static org.folio.dew.utils.Constants.FILE_NAME; -import static org.folio.dew.utils.Constants.JOB_NAME_POSTFIX_SEPARATOR; - -@Configuration -public class BulkEditUpdateUserRecordsJobConfig { - - @Bean - public Job bulkEditUpdateUserRecordsJob( - Step bulkEditUpdateUserRecordsStep, - JobRepository jobRepository, - BulkEditUpdateUserRecordsListener updateUserRecordsListener, - JobCompletionNotificationListener completionListener) { - return new JobBuilder(BULK_EDIT_UPDATE.getValue() + JOB_NAME_POSTFIX_SEPARATOR + USER.getValue(), jobRepository) - .incrementer(new RunIdIncrementer()) - .listener(updateUserRecordsListener) - .listener(completionListener) - .flow(bulkEditUpdateUserRecordsStep) - .end() - .build(); - } - - @Bean - public Step bulkEditUpdateUserRecordsStep( - ItemReader csvUserRecordsReader, - @Qualifier("bulkEditUpdateUserRecordsProcessor") - ItemProcessor processor, - @Qualifier("updateUserRecordsWriter") ItemWriter writer, - @Qualifier("updateRecordWriteListener") ItemWriteListener updateRecordWriteListener, - JobRepository jobRepository, PlatformTransactionManager transactionManager) { - return new StepBuilder("bulkEditUpdateRecordsStep", jobRepository) - .chunk(10, transactionManager) - .reader(csvUserRecordsReader) - .processor(processor) - .writer(writer) - .listener(updateRecordWriteListener) - .build(); - } - - @Bean - @StepScope - public FlatFileItemReader csvUserRecordsReader( - @Value("#{jobParameters['" + FILE_NAME + "']}") String fileName, - @Value("#{jobParameters['" + UPDATED_FILE_NAME + "']}") String updatedFileName, - LocalFilesStorage localFilesStorage, RemoteFilesStorage remoteFilesStorage) - throws IOException { - var userLineMapper = JobConfigReaderHelper.createLineMapper(UserFormat.class, UserFormat.getUserFieldsArray()); - return new FlatFileItemReaderBuilder() - .name("userReader") - .resource(isEmpty(updatedFileName) ? new S3CompatibleResource<>(fileName, localFilesStorage) : new InputStreamResource(remoteFilesStorage.newInputStream(updatedFileName))) - .linesToSkip(1) - .lineMapper(userLineMapper) - .build(); - } -} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateUserRecordsProcessor.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateUserRecordsProcessor.java deleted file mode 100644 index 69faca304..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateUserRecordsProcessor.java +++ /dev/null @@ -1,51 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.updatejob; - -import static org.folio.dew.utils.Constants.FILE_NAME; - -import org.apache.commons.io.FilenameUtils; -import org.folio.dew.domain.dto.User; -import org.folio.dew.domain.dto.UserFormat; -import org.folio.dew.error.BulkEditException; -import org.folio.dew.service.BulkEditChangedRecordsService; -import org.folio.dew.service.BulkEditParseService; -import org.folio.dew.service.BulkEditProcessingErrorsService; -import org.springframework.batch.core.JobExecution; -import org.springframework.batch.core.configuration.annotation.JobScope; -import org.springframework.batch.item.ItemProcessor; -import org.springframework.beans.factory.annotation.Qualifier; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.stereotype.Component; - -import lombok.RequiredArgsConstructor; -import lombok.extern.log4j.Log4j2; - -@Log4j2 -@Component -@Qualifier("updateUserRecordsProcessor") -@RequiredArgsConstructor -@JobScope -public class BulkEditUpdateUserRecordsProcessor implements ItemProcessor { - - @Value("#{jobParameters['jobId']}") - private String jobId; - @Value("#{jobExecution}") - private JobExecution jobExecution; - - private final BulkEditParseService bulkEditParseService; - private final BulkEditProcessingErrorsService bulkEditProcessingErrorsService; - private final BulkEditChangedRecordsService changedRecordsService; - - @Override - public User process(UserFormat userFormat) throws Exception { - try { - var user = bulkEditParseService.mapUserFormatToUser(userFormat); - changedRecordsService.addUserId(user.getId(), jobId); - return user; - } catch (Exception e) { - log.info("Error process user format {} : {}", userFormat.getId(), e.getMessage()); - bulkEditProcessingErrorsService.saveErrorInCSV(jobId, userFormat.getBarcode(), new BulkEditException(e.getMessage()), FilenameUtils.getName(jobExecution.getJobParameters().getString(FILE_NAME))); - return null; - } - } - -} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateUserRecordsWriter.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateUserRecordsWriter.java deleted file mode 100644 index 889413505..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/BulkEditUpdateUserRecordsWriter.java +++ /dev/null @@ -1,68 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.updatejob; - -import static org.folio.dew.utils.Constants.FILE_NAME; -import static org.folio.dew.utils.Constants.NO_CHANGE_MESSAGE; - -import java.util.List; -import java.util.UUID; - -import lombok.RequiredArgsConstructor; -import lombok.extern.log4j.Log4j2; -import org.apache.commons.io.FilenameUtils; -import org.folio.dew.client.UserClient; -import org.folio.dew.domain.dto.User; -import org.folio.dew.error.BulkEditException; -import org.folio.dew.service.BulkEditChangedRecordsService; -import org.folio.dew.service.BulkEditProcessingErrorsService; -import org.folio.dew.service.BulkEditRollBackService; -import org.folio.dew.service.BulkEditStatisticService; -import org.springframework.batch.core.JobExecution; -import org.springframework.batch.core.configuration.annotation.JobScope; -import org.springframework.batch.item.Chunk; -import org.springframework.batch.item.ItemWriter; -import org.springframework.beans.factory.annotation.Qualifier; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.stereotype.Component; - -@Log4j2 -@Component -@Qualifier("updateUserRecordsWriter") -@RequiredArgsConstructor -@JobScope -public class BulkEditUpdateUserRecordsWriter implements ItemWriter { - - @Value("#{jobParameters['jobId']}") - private String jobId; - @Value("#{jobExecution}") - private JobExecution jobExecution; - - private final UserClient userClient; - private final BulkEditRollBackService bulkEditRollBackService; - private final BulkEditProcessingErrorsService bulkEditProcessingErrorsService; - private final BulkEditStatisticService bulkEditStatisticService; - private final BulkEditChangedRecordsService changedRecordsService; - - @Override - public void write(Chunk users) throws Exception { - users.forEach(user -> { - try { - var initialUser = userClient.getUserById(user.getId()); - initialUser.setMetadata(null); - if (initialUser.equals(user)) { - log.info("User with barcode={}: {}", user.getBarcode(), NO_CHANGE_MESSAGE); - bulkEditProcessingErrorsService.saveErrorInCSV(jobId, initialUser.getBarcode(), new BulkEditException(NO_CHANGE_MESSAGE), FilenameUtils.getName(jobExecution.getJobParameters().getString(FILE_NAME))); - changedRecordsService.removeUserId(user.getId(), jobId); - } else { - userClient.updateUser(user, user.getId()); - log.info("Update user with barcode={} by job id {}", user.getBarcode(), jobId); - bulkEditStatisticService.incrementSuccess(); - bulkEditRollBackService.putUserIdForJob(user.getId(), UUID.fromString(jobId)); - } - } catch (Exception e) { - log.error("Cannot update user with barcode={}. Reason: {}", user.getBarcode(), e.getMessage()); - bulkEditProcessingErrorsService.saveErrorInCSV(jobId, user.getBarcode(), new BulkEditException(e.getMessage()), FilenameUtils.getName(jobExecution.getJobParameters().getString(FILE_NAME))); - changedRecordsService.removeUserId(user.getId(), jobId); - } - }); - } -} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/listeners/BulkEditUpdateUserRecordsListener.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/listeners/BulkEditUpdateUserRecordsListener.java deleted file mode 100644 index db32ad022..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/listeners/BulkEditUpdateUserRecordsListener.java +++ /dev/null @@ -1,29 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.updatejob.listeners; - -import lombok.RequiredArgsConstructor; -import org.folio.dew.service.BulkEditRollBackService; -import org.springframework.batch.core.JobExecution; -import org.springframework.batch.core.JobExecutionListener; -import org.springframework.batch.core.configuration.annotation.JobScope; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.stereotype.Component; - -import java.util.UUID; - -@Component -@JobScope -@RequiredArgsConstructor -public class BulkEditUpdateUserRecordsListener implements JobExecutionListener { - - @Value("#{jobParameters['jobId']}") - private String jobId; - private final BulkEditRollBackService bulkEditRollBackService; - - @Override - public void beforeJob(JobExecution jobExecution) {} - - @Override - public void afterJob(JobExecution jobExecution) { - bulkEditRollBackService.cleanJobData(jobExecution.getExitStatus().getExitCode(), UUID.fromString(jobId)); - } -} diff --git a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/listeners/UpdateRecordWriteListener.java b/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/listeners/UpdateRecordWriteListener.java deleted file mode 100644 index a5bab8247..000000000 --- a/src/main/java/org/folio/dew/batch/bulkedit/jobs/updatejob/listeners/UpdateRecordWriteListener.java +++ /dev/null @@ -1,84 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.updatejob.listeners; - -import static org.folio.dew.domain.dto.JobParameterNames.JOB_ID; -import static org.folio.dew.domain.dto.JobParameterNames.TOTAL_RECORDS; - -import java.util.Date; -import java.util.UUID; -import java.util.concurrent.atomic.AtomicInteger; - -import lombok.extern.log4j.Log4j2; -import org.folio.de.entity.Job; -import org.folio.dew.config.kafka.KafkaService; -import org.folio.dew.domain.dto.EntityType; -import org.folio.dew.domain.dto.ExportType; -import org.folio.dew.domain.dto.Progress; -import org.folio.dew.service.BulkEditStatisticService; -import org.springframework.batch.core.BatchStatus; -import org.springframework.batch.core.ItemWriteListener; -import org.springframework.batch.core.JobExecution; -import org.springframework.batch.core.configuration.annotation.JobScope; -import org.springframework.batch.item.Chunk; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.stereotype.Component; - -import lombok.RequiredArgsConstructor; - -@Component -@JobScope -@RequiredArgsConstructor -@Log4j2 -public class UpdateRecordWriteListener implements ItemWriteListener { - - private static final int BATCH_SIZE = 10; - - private final KafkaService kafka; - - @Value("#{jobExecution}") - private JobExecution jobExecution; - - private AtomicInteger processedRecords = new AtomicInteger(); - private final BulkEditStatisticService bulkEditUpdateStatisticService; - - @Override - public void afterWrite(Chunk items) { - var job = prepareJobWithProgress(); - kafka.send(KafkaService.Topic.JOB_UPDATE, job.getId().toString(), job); - } - - private Job prepareJobWithProgress() { - var totalRecords = jobExecution.getExecutionContext().getInt(TOTAL_RECORDS); - if (totalRecords < BATCH_SIZE) { - processedRecords.addAndGet(totalRecords); - } else { - processedRecords.addAndGet(BATCH_SIZE); - } - var job = new Job(); - job.setId(UUID.fromString(jobExecution.getJobParameters().getString(JOB_ID))); - job.setType(ExportType.BULK_EDIT_UPDATE); - job.setEntityType(EntityType.fromValue(jobExecution.getJobInstance().getJobName().split("-")[1])); - job.setBatchStatus(BatchStatus.STARTED); - job.setStartTime(new Date()); - job.setCreatedDate(new Date()); - job.setEndTime(new Date()); - job.setUpdatedDate(new Date()); - - Progress progress = new Progress(); - progress.setTotal(totalRecords); - progress.setProcessed(processedRecords.get()); - progress.setProgress(getProgressBarValue(processedRecords.get(), totalRecords)); - - var statistic = bulkEditUpdateStatisticService.getStatistic(); - progress.setSuccess(statistic.getSuccess()); - job.setProgress(progress); - return job; - } - - private int getProgressBarValue(int processed, int totalRecords) { - if (totalRecords < BATCH_SIZE) { - return 90; - } - var progress = ((double) processed / totalRecords) * 100; - return progress < 100 ? (int) progress : 99; - } -} diff --git a/src/main/java/org/folio/dew/client/ConfigurationClient.java b/src/main/java/org/folio/dew/client/ConfigurationClient.java index 4c404a788..0997afdb1 100644 --- a/src/main/java/org/folio/dew/client/ConfigurationClient.java +++ b/src/main/java/org/folio/dew/client/ConfigurationClient.java @@ -5,6 +5,7 @@ import org.folio.dew.domain.dto.ModelConfiguration; import org.springframework.cloud.openfeign.FeignClient; import org.springframework.http.MediaType; +import org.springframework.web.bind.annotation.DeleteMapping; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.PathVariable; import org.springframework.web.bind.annotation.PostMapping; @@ -22,4 +23,7 @@ public interface ConfigurationClient { @PostMapping ModelConfiguration postConfiguration(@RequestBody ModelConfiguration config); + + @DeleteMapping(path = "/{entryId}") + void deleteConfiguration(@PathVariable String entryId); } diff --git a/src/main/java/org/folio/dew/client/ConsortiumClient.java b/src/main/java/org/folio/dew/client/ConsortiumClient.java new file mode 100644 index 000000000..0551d5511 --- /dev/null +++ b/src/main/java/org/folio/dew/client/ConsortiumClient.java @@ -0,0 +1,19 @@ +package org.folio.dew.client; + +import org.folio.dew.domain.bean.ConsortiaCollection; +import org.folio.dew.domain.dto.UserTenantCollection; +import org.springframework.cloud.openfeign.FeignClient; +import org.springframework.http.MediaType; +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.PathVariable; +import org.springframework.web.bind.annotation.RequestParam; + +@FeignClient(name = "consortia") +public interface ConsortiumClient { + + @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) + ConsortiaCollection getConsortia(); + + @GetMapping(value = "/{consortiumId}/user-tenants", produces = MediaType.APPLICATION_JSON_VALUE) + UserTenantCollection getConsortiaUserTenants(@PathVariable String consortiumId, @RequestParam String userId, @RequestParam int limit); +} diff --git a/src/main/java/org/folio/dew/client/DataExportSpringClient.java b/src/main/java/org/folio/dew/client/DataExportSpringClient.java index 6ceceaaae..fadd1af96 100644 --- a/src/main/java/org/folio/dew/client/DataExportSpringClient.java +++ b/src/main/java/org/folio/dew/client/DataExportSpringClient.java @@ -1,19 +1,14 @@ package org.folio.dew.client; import org.folio.dew.domain.dto.ExportConfigCollection; -import org.folio.dew.domain.dto.Job; import org.springframework.cloud.openfeign.FeignClient; import org.springframework.http.MediaType; import org.springframework.web.bind.annotation.GetMapping; -import org.springframework.web.bind.annotation.PathVariable; import org.springframework.web.bind.annotation.RequestParam; @FeignClient(name = "data-export-spring") public interface DataExportSpringClient { - @GetMapping(value = "/jobs/{jobId}", produces = MediaType.APPLICATION_JSON_VALUE) - Job getJobById(@PathVariable String jobId); - @GetMapping(value = "/configs", produces = MediaType.APPLICATION_JSON_VALUE) ExportConfigCollection getExportConfigs(@RequestParam("query") String query); diff --git a/src/main/java/org/folio/dew/client/EurekaUserPermissionsClient.java b/src/main/java/org/folio/dew/client/EurekaUserPermissionsClient.java new file mode 100644 index 000000000..f2cc295f1 --- /dev/null +++ b/src/main/java/org/folio/dew/client/EurekaUserPermissionsClient.java @@ -0,0 +1,18 @@ +package org.folio.dew.client; + +import org.folio.dew.batch.bulkedit.jobs.permissions.check.UserPermissions; +import org.folio.dew.config.feign.FeignClientConfiguration; +import org.springframework.cloud.openfeign.FeignClient; +import org.springframework.http.MediaType; +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.PathVariable; +import org.springframework.web.bind.annotation.RequestParam; + +import java.util.List; + +@FeignClient(name = "users-keycloak/users", configuration = FeignClientConfiguration.class) +public interface EurekaUserPermissionsClient { + + @GetMapping(value = "/{userId}/permissions", produces = MediaType.APPLICATION_JSON_VALUE) + UserPermissions getPermissions(@PathVariable String userId, @RequestParam List desiredPermissions); +} diff --git a/src/main/java/org/folio/dew/client/FeefineactionsClient.java b/src/main/java/org/folio/dew/client/FeefineactionsClient.java deleted file mode 100644 index 77a9dcfc4..000000000 --- a/src/main/java/org/folio/dew/client/FeefineactionsClient.java +++ /dev/null @@ -1,14 +0,0 @@ -package org.folio.dew.client; - -import org.folio.dew.domain.dto.FeefineactionCollection; -import org.springframework.cloud.openfeign.FeignClient; -import org.springframework.http.MediaType; -import org.springframework.web.bind.annotation.GetMapping; -import org.springframework.web.bind.annotation.RequestParam; - -@FeignClient(name = "feefineactions") -public interface FeefineactionsClient { - - @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) - FeefineactionCollection getFeefineactions(@RequestParam String query, @RequestParam long limit); -} diff --git a/src/main/java/org/folio/dew/client/GroupClient.java b/src/main/java/org/folio/dew/client/GroupClient.java index 283e97dd0..5bf5e8aca 100644 --- a/src/main/java/org/folio/dew/client/GroupClient.java +++ b/src/main/java/org/folio/dew/client/GroupClient.java @@ -2,12 +2,10 @@ import org.folio.dew.config.feign.FeignClientConfiguration; import org.folio.dew.domain.dto.UserGroup; -import org.folio.dew.domain.dto.UserGroupCollection; import org.springframework.cloud.openfeign.FeignClient; import org.springframework.http.MediaType; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.PathVariable; -import org.springframework.web.bind.annotation.RequestParam; @FeignClient(name = "groups", configuration = FeignClientConfiguration.class) public interface GroupClient { @@ -15,6 +13,4 @@ public interface GroupClient { @GetMapping(value = "/{groupId}", produces = MediaType.APPLICATION_JSON_VALUE) UserGroup getGroupById(@PathVariable String groupId); - @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) - UserGroupCollection getGroupByQuery(@RequestParam String query); } diff --git a/src/main/java/org/folio/dew/client/HoldingClient.java b/src/main/java/org/folio/dew/client/HoldingClient.java index f22bdc581..4dcc49ab0 100644 --- a/src/main/java/org/folio/dew/client/HoldingClient.java +++ b/src/main/java/org/folio/dew/client/HoldingClient.java @@ -1,9 +1,7 @@ package org.folio.dew.client; -import io.swagger.v3.oas.annotations.parameters.RequestBody; import org.folio.dew.config.feign.FeignClientConfiguration; import org.folio.dew.config.feign.FeignEncoderConfiguration; -import org.folio.dew.domain.dto.BriefHoldingsRecordCollection; import org.folio.dew.domain.dto.HoldingsRecord; import org.folio.dew.domain.dto.HoldingsRecordCollection; import org.springframework.cloud.openfeign.FeignClient; @@ -12,7 +10,6 @@ import org.springframework.web.bind.annotation.PathVariable; import com.fasterxml.jackson.databind.JsonNode; -import org.springframework.web.bind.annotation.PutMapping; import org.springframework.web.bind.annotation.RequestParam; @FeignClient(name = "holdings-storage/holdings", configuration = { FeignClientConfiguration.class, FeignEncoderConfiguration.class }) @@ -23,15 +20,9 @@ public interface HoldingClient { @GetMapping(value = "/{id}", produces = MediaType.APPLICATION_JSON_VALUE) HoldingsRecord getHoldingsRecordById(@PathVariable String id); - @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) - BriefHoldingsRecordCollection getBriefHoldingsByQuery(@RequestParam String query); - @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) HoldingsRecordCollection getHoldingsByQuery(@RequestParam String query); @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) HoldingsRecordCollection getHoldingsByQuery(@RequestParam String query, @RequestParam long limit); - - @PutMapping(value = "/{holdingsId}", consumes = MediaType.APPLICATION_JSON_VALUE) - void updateHoldingsRecord(@RequestBody HoldingsRecord holdingsRecord, @PathVariable String holdingsId); } diff --git a/src/main/java/org/folio/dew/client/HoldingsNoteTypeClient.java b/src/main/java/org/folio/dew/client/HoldingsNoteTypeClient.java index 2e71a9815..c42681041 100644 --- a/src/main/java/org/folio/dew/client/HoldingsNoteTypeClient.java +++ b/src/main/java/org/folio/dew/client/HoldingsNoteTypeClient.java @@ -16,7 +16,4 @@ public interface HoldingsNoteTypeClient { @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) HoldingsNoteTypeCollection getByQuery(@RequestParam String query); - - @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) - HoldingsNoteTypeCollection getNoteTypes(@RequestParam("limit") int limit); } diff --git a/src/main/java/org/folio/dew/client/InventoryClient.java b/src/main/java/org/folio/dew/client/InventoryClient.java index 0ecb56d22..b0b20fc97 100644 --- a/src/main/java/org/folio/dew/client/InventoryClient.java +++ b/src/main/java/org/folio/dew/client/InventoryClient.java @@ -1,30 +1,17 @@ package org.folio.dew.client; -import io.swagger.v3.oas.annotations.parameters.RequestBody; import org.folio.dew.config.feign.FeignEncoderConfiguration; -import org.folio.dew.domain.dto.Item; import org.folio.dew.domain.dto.ItemCollection; import org.springframework.cloud.openfeign.FeignClient; import org.springframework.http.MediaType; import org.springframework.web.bind.annotation.GetMapping; -import org.springframework.web.bind.annotation.PathVariable; -import org.springframework.web.bind.annotation.PutMapping; import org.springframework.web.bind.annotation.RequestParam; @FeignClient(name = "inventory/items", configuration = FeignEncoderConfiguration.class) public interface InventoryClient { - @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) - ItemCollection getItemByQuery(@RequestParam String query); - - @GetMapping(value = "/{itemId}", produces = MediaType.APPLICATION_JSON_VALUE) - Item getItemById(@PathVariable String itemId); - @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) ItemCollection getItemByQuery(@RequestParam("query") String query, @RequestParam long limit); @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) ItemCollection getItemByQuery(@RequestParam("query") String query, @RequestParam long offset, @RequestParam long limit); - - @PutMapping(value = "/{itemId}", consumes = MediaType.APPLICATION_JSON_VALUE) - void updateItem(@RequestBody Item item, @PathVariable String itemId); } diff --git a/src/main/java/org/folio/dew/client/InventoryInstancesClient.java b/src/main/java/org/folio/dew/client/InventoryInstancesClient.java index 4a097da82..ffc15716d 100644 --- a/src/main/java/org/folio/dew/client/InventoryInstancesClient.java +++ b/src/main/java/org/folio/dew/client/InventoryInstancesClient.java @@ -1,14 +1,10 @@ package org.folio.dew.client; -import io.swagger.v3.oas.annotations.parameters.RequestBody; import org.folio.dew.config.feign.FeignEncoderConfiguration; -import org.folio.dew.domain.dto.Instance; import org.folio.dew.domain.dto.InstanceCollection; import org.springframework.cloud.openfeign.FeignClient; import org.springframework.http.MediaType; import org.springframework.web.bind.annotation.GetMapping; -import org.springframework.web.bind.annotation.PathVariable; -import org.springframework.web.bind.annotation.PutMapping; import org.springframework.web.bind.annotation.RequestParam; @FeignClient(name = "inventory/instances", configuration = FeignEncoderConfiguration.class) @@ -16,15 +12,7 @@ public interface InventoryInstancesClient { @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) InstanceCollection getInstanceByQuery(@RequestParam String query); - @GetMapping(value = "/{instanceId}", produces = MediaType.APPLICATION_JSON_VALUE) - Instance getInstanceById(@PathVariable String instanceId); - @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) InstanceCollection getInstanceByQuery(@RequestParam("query") String query, @RequestParam long limit); - @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) - InstanceCollection getInstanceByQuery(@RequestParam("query") String query, @RequestParam long offset, @RequestParam long limit); - - @PutMapping(value = "/{instanceId}", consumes = MediaType.APPLICATION_JSON_VALUE) - void updateInstance(@RequestBody Instance instance, @PathVariable String instanceId); } diff --git a/src/main/java/org/folio/dew/client/OkapiUserPermissionsClient.java b/src/main/java/org/folio/dew/client/OkapiUserPermissionsClient.java new file mode 100644 index 000000000..ce6040755 --- /dev/null +++ b/src/main/java/org/folio/dew/client/OkapiUserPermissionsClient.java @@ -0,0 +1,15 @@ +package org.folio.dew.client; + +import org.folio.dew.batch.bulkedit.jobs.permissions.check.UserPermissions; +import org.folio.dew.config.feign.FeignClientConfiguration; +import org.springframework.cloud.openfeign.FeignClient; +import org.springframework.http.MediaType; +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.PathVariable; + +@FeignClient(name = "perms/users", configuration = FeignClientConfiguration.class) +public interface OkapiUserPermissionsClient { + + @GetMapping(value = "/{userId}/permissions?expanded=true&indexField=userId", produces = MediaType.APPLICATION_JSON_VALUE) + UserPermissions getPermissions(@PathVariable String userId); +} diff --git a/src/main/java/org/folio/dew/client/OwnersClient.java b/src/main/java/org/folio/dew/client/OwnersClient.java deleted file mode 100644 index 1959d1340..000000000 --- a/src/main/java/org/folio/dew/client/OwnersClient.java +++ /dev/null @@ -1,15 +0,0 @@ -package org.folio.dew.client; - -import org.folio.dew.domain.dto.Ownerdatacollection; -import org.springframework.cloud.openfeign.FeignClient; -import org.springframework.http.MediaType; -import org.springframework.web.bind.annotation.GetMapping; -import org.springframework.web.bind.annotation.RequestParam; - -@FeignClient(name = "owners") -public interface OwnersClient { - - @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) - Ownerdatacollection get(@RequestParam String query, @RequestParam long limit); - -} diff --git a/src/main/java/org/folio/dew/client/PermissionsSelfCheckClient.java b/src/main/java/org/folio/dew/client/PermissionsSelfCheckClient.java new file mode 100644 index 000000000..79a1b60bd --- /dev/null +++ b/src/main/java/org/folio/dew/client/PermissionsSelfCheckClient.java @@ -0,0 +1,15 @@ +package org.folio.dew.client; + +import org.folio.dew.config.feign.FeignClientConfiguration; +import org.springframework.cloud.openfeign.FeignClient; +import org.springframework.http.MediaType; +import org.springframework.web.bind.annotation.GetMapping; + +import java.util.List; + +@FeignClient(name = "bulk-edit", configuration = FeignClientConfiguration.class) +public interface PermissionsSelfCheckClient { + + @GetMapping(value = "/permissions-self-check", produces = MediaType.APPLICATION_JSON_VALUE) + List getUserPermissionsForSelfCheck(); +} diff --git a/src/main/java/org/folio/dew/client/ProxiesForClient.java b/src/main/java/org/folio/dew/client/ProxiesForClient.java deleted file mode 100644 index 0fd65538a..000000000 --- a/src/main/java/org/folio/dew/client/ProxiesForClient.java +++ /dev/null @@ -1,20 +0,0 @@ -package org.folio.dew.client; - -import org.folio.dew.config.feign.FeignClientConfiguration; -import org.folio.dew.domain.dto.ProxyFor; -import org.folio.dew.domain.dto.ProxyForCollection; -import org.springframework.cloud.openfeign.FeignClient; -import org.springframework.http.MediaType; -import org.springframework.web.bind.annotation.GetMapping; -import org.springframework.web.bind.annotation.PathVariable; -import org.springframework.web.bind.annotation.RequestParam; - -@FeignClient(name = "proxiesfor", configuration = FeignClientConfiguration.class) -public interface ProxiesForClient { - - @GetMapping(value = "/{id}", produces = MediaType.APPLICATION_JSON_VALUE) - ProxyFor getProxiesForById(@PathVariable String id); - - @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) - ProxyForCollection getProxiesForByQuery(@RequestParam String query); -} diff --git a/src/main/java/org/folio/dew/client/PurchaseOrderLineClient.java b/src/main/java/org/folio/dew/client/PurchaseOrderLineClient.java deleted file mode 100644 index 860cb8379..000000000 --- a/src/main/java/org/folio/dew/client/PurchaseOrderLineClient.java +++ /dev/null @@ -1,21 +0,0 @@ -package org.folio.dew.client; - -import org.folio.dew.domain.dto.CompositePoLine; -import org.folio.dew.domain.dto.PoLineCollection; -import org.springframework.cloud.openfeign.FeignClient; -import org.springframework.http.MediaType; -import org.springframework.web.bind.annotation.GetMapping; -import org.springframework.web.bind.annotation.PathVariable; -import org.springframework.web.bind.annotation.RequestParam; - -@FeignClient(name = "orders/order-lines") -public interface PurchaseOrderLineClient { - - @GetMapping(value = "/{id}", produces = MediaType.APPLICATION_JSON_VALUE) - CompositePoLine getCompositePoLineById(@PathVariable String id); - - - @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) - PoLineCollection getPoLineByQuery(@RequestParam("query") String query); - -} diff --git a/src/main/java/org/folio/dew/client/SearchClient.java b/src/main/java/org/folio/dew/client/SearchClient.java index 91dcfdab5..7c6218afd 100644 --- a/src/main/java/org/folio/dew/client/SearchClient.java +++ b/src/main/java/org/folio/dew/client/SearchClient.java @@ -4,21 +4,11 @@ import org.folio.dew.domain.dto.ConsortiumHoldingCollection; import org.folio.dew.domain.dto.ConsortiumItemCollection; import org.springframework.cloud.openfeign.FeignClient; -import org.springframework.core.io.InputStreamResource; -import org.springframework.http.ResponseEntity; -import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.RequestBody; -import org.springframework.web.bind.annotation.RequestParam; @FeignClient(name = "search") public interface SearchClient { - @GetMapping(value = "/instances/ids", headers = {"Accept=text/plain"}) - ResponseEntity getInstanceIds(@RequestParam String query); - - @GetMapping(value = "/holdings/ids", headers = {"Accept=text/plain"}) - ResponseEntity getHoldingIds(@RequestParam String query); - @PostMapping(value = "/consortium/batch/items", headers = {"Accept=application/json"}) ConsortiumItemCollection getConsortiumItemCollection(@RequestBody BatchIdsDto batchIdsDto); diff --git a/src/main/java/org/folio/dew/client/SrsClient.java b/src/main/java/org/folio/dew/client/SrsClient.java index af54d8347..f80b6cd6d 100644 --- a/src/main/java/org/folio/dew/client/SrsClient.java +++ b/src/main/java/org/folio/dew/client/SrsClient.java @@ -11,5 +11,5 @@ public interface SrsClient { @GetMapping(value = "/source-records", produces = MediaType.APPLICATION_JSON_VALUE) - JsonNode getMarc(@RequestParam("instanceId") String instanceId, @RequestParam("idType") String idType); + JsonNode getMarc(@RequestParam("instanceId") String instanceId, @RequestParam("idType") String idType, @RequestParam("deleted") boolean deleted); } diff --git a/src/main/java/org/folio/dew/client/UserClient.java b/src/main/java/org/folio/dew/client/UserClient.java index 29267746d..f316a1653 100644 --- a/src/main/java/org/folio/dew/client/UserClient.java +++ b/src/main/java/org/folio/dew/client/UserClient.java @@ -7,11 +7,8 @@ import org.springframework.http.MediaType; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.PathVariable; -import org.springframework.web.bind.annotation.PutMapping; import org.springframework.web.bind.annotation.RequestParam; -import io.swagger.v3.oas.annotations.parameters.RequestBody; - @FeignClient(name = "users", configuration = FeignClientConfiguration.class) public interface UserClient { @@ -24,9 +21,4 @@ public interface UserClient { @GetMapping(value = "/{userId}", produces = MediaType.APPLICATION_JSON_VALUE) User getUserById(@PathVariable String userId); - @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE) - UserCollection getUserByQuery(@RequestParam String query); - - @PutMapping(value = "/{userId}", consumes = MediaType.APPLICATION_JSON_VALUE) - void updateUser(@RequestBody User user, @PathVariable String userId); } diff --git a/src/main/java/org/folio/dew/config/AsyncConfig.java b/src/main/java/org/folio/dew/config/AsyncConfig.java index 709a75a1e..2b79b2acd 100644 --- a/src/main/java/org/folio/dew/config/AsyncConfig.java +++ b/src/main/java/org/folio/dew/config/AsyncConfig.java @@ -5,6 +5,7 @@ import org.springframework.batch.core.launch.support.TaskExecutorJobLauncher; import org.springframework.batch.core.repository.JobRepository; import org.springframework.beans.factory.annotation.Qualifier; +import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.core.task.TaskExecutor; @@ -18,6 +19,12 @@ public class AsyncConfig { private static final int TASK_EXECUTOR_CORE_POOL_SIZE = 10; private static final int TASK_EXECUTOR_MAX_POOL_SIZE = 10; + @Value("${application.core-pool-size}") + private int corePoolSize; + + @Value("${application.max-pool-size}") + private int maxPoolSize; + @Bean(name = "asyncJobLauncher") public JobLauncher getAsyncJobLauncher( JobRepository jobRepository, @Qualifier("asyncTaskExecutor") TaskExecutor taskExecutor) { @@ -37,4 +44,14 @@ public TaskExecutor getAsyncTaskExecutor() { return threadPoolTaskExecutor; } + @Bean(name = "asyncTaskExecutorBulkEdit") + public TaskExecutor getAsyncTaskExecutorBulkEdit() { + var threadPoolTaskExecutor = new ThreadPoolTaskExecutor(); + threadPoolTaskExecutor.setCorePoolSize(corePoolSize); + threadPoolTaskExecutor.setMaxPoolSize(maxPoolSize); + threadPoolTaskExecutor.setTaskDecorator( + FolioExecutionScopeExecutionContextManager::getRunnableWithCurrentFolioContext); + return threadPoolTaskExecutor; + } + } diff --git a/src/main/java/org/folio/dew/config/properties/MinioClientProperties.java b/src/main/java/org/folio/dew/config/properties/MinioClientProperties.java index 8484ad0d8..574a36061 100644 --- a/src/main/java/org/folio/dew/config/properties/MinioClientProperties.java +++ b/src/main/java/org/folio/dew/config/properties/MinioClientProperties.java @@ -41,6 +41,11 @@ public class MinioClientProperties { */ private boolean forcePathStyle; + /** + * Path in s3 bucket. + */ + private String subPath; + /** * Presigned url expiration time (in seconds). */ diff --git a/src/main/java/org/folio/dew/controller/BulkEditController.java b/src/main/java/org/folio/dew/controller/BulkEditController.java index 86220ac2d..3bfb8f8a6 100644 --- a/src/main/java/org/folio/dew/controller/BulkEditController.java +++ b/src/main/java/org/folio/dew/controller/BulkEditController.java @@ -2,93 +2,47 @@ import static java.lang.String.format; import static java.util.Objects.isNull; -import static java.util.Objects.nonNull; import static java.util.Optional.ofNullable; -import static org.apache.commons.lang3.StringUtils.EMPTY; -import static org.folio.dew.domain.dto.EntityType.HOLDINGS_RECORD; -import static org.folio.dew.domain.dto.EntityType.ITEM; -import static org.folio.dew.domain.dto.EntityType.USER; import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_IDENTIFIERS; -import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_UPDATE; -import static org.folio.dew.domain.dto.JobParameterNames.PREVIEW_FILE_NAME; -import static org.folio.dew.domain.dto.JobParameterNames.QUERY; import static org.folio.dew.domain.dto.JobParameterNames.TEMP_LOCAL_FILE_PATH; import static org.folio.dew.domain.dto.JobParameterNames.TEMP_OUTPUT_FILE_PATH; import static org.folio.dew.domain.dto.JobParameterNames.TEMP_OUTPUT_MARC_PATH; import static org.folio.dew.domain.dto.JobParameterNames.TEMP_LOCAL_MARC_PATH; -import static org.folio.dew.domain.dto.JobParameterNames.UPDATED_FILE_NAME; -import static org.folio.dew.utils.BulkEditProcessorHelper.getMatchPattern; -import static org.folio.dew.utils.BulkEditProcessorHelper.resolveIdentifier; import static org.folio.dew.utils.Constants.BULKEDIT_DIR_NAME; -import static org.folio.dew.utils.Constants.CSV_EXTENSION; import static org.folio.dew.utils.Constants.EXPORT_TYPE; import static org.folio.dew.utils.Constants.FILE_NAME; import static org.folio.dew.utils.Constants.FILE_UPLOAD_ERROR; -import static org.folio.dew.utils.Constants.IDENTIFIER_TYPE; -import static org.folio.dew.utils.Constants.INITIAL_PREFIX; import static org.folio.dew.utils.Constants.MARC_RECORDS; import static org.folio.dew.utils.Constants.MATCHED_RECORDS; import static org.folio.dew.utils.Constants.PATH_SEPARATOR; import static org.folio.dew.utils.Constants.TEMP_IDENTIFIERS_FILE_NAME; import static org.folio.dew.utils.Constants.TOTAL_CSV_LINES; -import static org.folio.dew.utils.Constants.PREVIEW_PREFIX; import static org.folio.dew.utils.Constants.getWorkingDirectory; import static org.folio.dew.utils.CsvHelper.countLines; import static org.folio.dew.utils.SystemHelper.getTempDirWithSeparatorSuffix; import static org.folio.spring.scope.FolioExecutionScopeExecutionContextManager.getRunnableWithCurrentFolioContext; -import com.opencsv.CSVReader; import io.swagger.annotations.ApiParam; import jakarta.annotation.PostConstruct; -import jakarta.validation.Valid; -import jakarta.validation.constraints.NotNull; import java.io.IOException; -import java.io.InputStream; -import java.io.InputStreamReader; -import java.io.Reader; import java.nio.file.Files; import java.nio.file.Path; import java.time.LocalDate; -import java.util.Arrays; -import java.util.Collections; import java.util.List; import java.util.UUID; -import java.util.stream.Collectors; + +import jakarta.validation.Valid; +import jakarta.validation.constraints.NotNull; import lombok.RequiredArgsConstructor; import lombok.extern.log4j.Log4j2; import org.apache.commons.io.FilenameUtils; -import org.apache.commons.lang3.StringUtils; import org.folio.de.entity.JobCommand; import org.folio.dew.batch.ExportJobManagerSync; -import org.folio.dew.client.HoldingClient; -import org.folio.dew.client.InventoryClient; -import org.folio.dew.client.UserClient; import org.folio.dew.domain.dto.Errors; -import org.folio.dew.domain.dto.HoldingsContentUpdateCollection; -import org.folio.dew.domain.dto.HoldingsFormat; -import org.folio.dew.domain.dto.HoldingsRecordCollection; -import org.folio.dew.domain.dto.IdentifierType; -import org.folio.dew.domain.dto.ItemCollection; -import org.folio.dew.domain.dto.ItemContentUpdateCollection; -import org.folio.dew.domain.dto.ItemFormat; -import org.folio.dew.domain.dto.UserCollection; -import org.folio.dew.domain.dto.UserContentUpdateCollection; -import org.folio.dew.domain.dto.UserFormat; -import org.folio.dew.error.FileOperationException; -import org.folio.dew.error.NonSupportedEntityException; import org.folio.dew.error.NotFoundException; import org.folio.dew.repository.LocalFilesStorage; -import org.folio.dew.repository.RemoteFilesStorage; -import org.folio.dew.service.BulkEditItemContentUpdateService; -import org.folio.dew.service.BulkEditParseService; import org.folio.dew.service.BulkEditProcessingErrorsService; -import org.folio.dew.service.BulkEditRollBackService; import org.folio.dew.service.JobCommandsReceiverService; -import org.folio.dew.service.UpdatesResult; -import org.folio.dew.service.mapper.HoldingsMapper; -import org.folio.dew.service.update.BulkEditHoldingsContentUpdateService; -import org.folio.dew.service.update.BulkEditUserContentUpdateService; -import org.folio.dew.utils.CsvHelper; import org.folio.spring.FolioExecutionContext; import org.folio.spring.FolioModuleMetadata; import org.openapitools.api.JobIdApi; @@ -97,15 +51,9 @@ import org.springframework.batch.core.JobParametersBuilder; import org.springframework.batch.integration.launch.JobLaunchRequest; import org.springframework.beans.factory.annotation.Value; -import org.springframework.core.io.ByteArrayResource; -import org.springframework.core.io.DescriptiveResource; -import org.springframework.core.io.Resource; -import org.springframework.http.HttpHeaders; import org.springframework.http.HttpStatus; -import org.springframework.http.MediaType; import org.springframework.http.ResponseEntity; import org.springframework.web.bind.annotation.PathVariable; -import org.springframework.web.bind.annotation.RequestBody; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestParam; import org.springframework.web.bind.annotation.RestController; @@ -119,24 +67,13 @@ public class BulkEditController implements JobIdApi { private static final String JOB_COMMAND_NOT_FOUND_ERROR = "JobCommand with id %s doesn't exist."; - private static final String FAILED_TO_READ_FILE_ERROR = "Failed to read %s for job id %s, reason: %s"; private static final boolean JOB_PARAMETER_DEFAULT_IDENTIFYING_VALUE = false; - private final UserClient userClient; - private final InventoryClient inventoryClient; - private final HoldingClient holdingClient; private final JobCommandsReceiverService jobCommandsReceiverService; private final ExportJobManagerSync exportJobManagerSync; - private final BulkEditRollBackService bulkEditRollBackService; - private final BulkEditProcessingErrorsService bulkEditProcessingErrorsService; private final List jobs; - private final BulkEditItemContentUpdateService itemContentUpdateService; - private final BulkEditUserContentUpdateService userContentUpdateService; - private final BulkEditHoldingsContentUpdateService holdingsContentUpdateService; - private final BulkEditParseService bulkEditParseService; - private final HoldingsMapper holdingsMapper; - private final RemoteFilesStorage remoteFilesStorage; private final LocalFilesStorage localFilesStorage; + private final BulkEditProcessingErrorsService bulkEditProcessingErrorsService; private final FolioModuleMetadata folioModuleMetadata; private final FolioExecutionContext folioExecutionContext; @@ -149,175 +86,6 @@ public void postConstruct() { workDir = getWorkingDirectory(springApplicationName, BULKEDIT_DIR_NAME); } - @Override - public ResponseEntity postItemContentUpdates(@ApiParam(value = "UUID of the JobCommand",required=true) @PathVariable("jobId") UUID jobId,@ApiParam(value = "" ,required=true ) @Valid @RequestBody ItemContentUpdateCollection contentUpdateCollection,@ApiParam(value = "The numbers of records to return") @Valid @RequestParam(value = "limit", required = false) Integer limit) { - var jobCommand = prepareForContentUpdates(jobId); - log.info("postItemContentUpdates: jobCommand={}.", jobCommand); - var updatesResult = itemContentUpdateService.processContentUpdates(jobCommand, contentUpdateCollection); - log.info("postItemContentUpdates: jobCommand={}; updatesResult size={}.", jobCommand, updatesResult.getEntitiesForPreview().size()); - jobCommandsReceiverService.updateJobCommand(jobCommand); - return new ResponseEntity<>(prepareItemContentUpdateResponse(updatesResult, limit), HttpStatus.OK); - } - - @Override - public ResponseEntity postUserContentUpdates(@ApiParam(value = "UUID of the JobCommand",required=true) @PathVariable("jobId") UUID jobId, @ApiParam(value = "" ,required=true ) @Valid @RequestBody UserContentUpdateCollection contentUpdateCollection, @ApiParam(value = "The numbers of records to return") @Valid @RequestParam(value = "limit", required = false) Integer limit) { - var jobCommand = prepareForContentUpdates(jobId); - log.info("postUserContentUpdates: jobCommand={}.", jobCommand); - var updatesResult = userContentUpdateService.process(jobCommand, contentUpdateCollection); - log.info("postUserContentUpdate: {} users", updatesResult.getEntitiesForPreview().size()); - jobCommandsReceiverService.updateJobCommand(jobCommand); - return new ResponseEntity<>(prepareUserContentUpdateResponse(updatesResult, limit), HttpStatus.OK); - } - - @Override - public ResponseEntity postHoldingsContentUpdates(@ApiParam(value = "UUID of the JobCommand",required=true) @PathVariable("jobId") UUID jobId, @ApiParam(value = "" ,required=true ) @Valid @RequestBody HoldingsContentUpdateCollection contentUpdateCollection, @ApiParam(value = "The numbers of records to return") @Valid @RequestParam(value = "limit", required = false) Integer limit) { - var jobCommand = prepareForContentUpdates(jobId); - log.info("postHoldingsContentUpdates: jobCommand={}.", jobCommand); - var updatesResult = holdingsContentUpdateService.process(jobCommand, contentUpdateCollection); - log.info("postHoldingsContentUpdates: jobCommand={}; updatesResult size={}.", jobCommand, updatesResult.getEntitiesForPreview().size()); - jobCommandsReceiverService.updateJobCommand(jobCommand); - return new ResponseEntity<>(prepareHoldingsContentUpdateResponse(updatesResult, limit), HttpStatus.OK); - } - - private JobCommand prepareForContentUpdates(UUID jobId) { - bulkEditProcessingErrorsService.removeTemporaryErrorStorage(); - var jobCommand = getJobCommandById(jobId.toString()); - if (nonNull(jobCommand.getIdentifierType())) { - jobCommand.setJobParameters(new JobParametersBuilder(jobCommand.getJobParameters()) - .addString(IDENTIFIER_TYPE, jobCommand.getIdentifierType().getValue()) - .toJobParameters()); - } - return jobCommand; - } - - @Override - public ResponseEntity getPreviewUsersByJobId(@ApiParam(value = "UUID of the JobCommand", required = true) @PathVariable("jobId") UUID jobId, @NotNull @ApiParam(value = "The numbers of items to return", required = true) @Valid @RequestParam(value = "limit") Integer limit) { - var jobCommand = getJobCommandById(jobId.toString()); - log.info("getPreviewUsersByJobId:: with jobExportType={}", jobCommand.getExportType()); - if (BULK_EDIT_IDENTIFIERS == jobCommand.getExportType()) { - var fileName = jobCommand.getId() + PATH_SEPARATOR + FilenameUtils.getName(jobCommand.getJobParameters().getString(TEMP_OUTPUT_FILE_PATH)) + CSV_EXTENSION; - try { - var userFormats = CsvHelper.readRecordsFromRemoteFilesStorage(remoteFilesStorage, fileName, limit, UserFormat.class); - log.info("getPreviewUsersByJobId:: Reading of file {} complete, number of userFormats: {}", fileName, userFormats.size()); - var users = userFormats.stream() - .map(bulkEditParseService::mapUserFormatToUser) - .collect(Collectors.toList()); - return new ResponseEntity<>(new UserCollection().users(users).totalRecords(users.size()), HttpStatus.OK); - } catch (Exception e) { - var msg = String.format(FAILED_TO_READ_FILE_ERROR, fileName, jobCommand.getId(), e.getMessage()); - log.error(msg); - return new ResponseEntity<>(new UserCollection().users(Collections.emptyList()).totalRecords(0), HttpStatus.OK); - } - } else { - return new ResponseEntity<>(userClient.getUserByQuery(buildPreviewUsersQueryFromJobCommand(jobCommand, limit), limit), HttpStatus.OK); - } - } - - @Override public ResponseEntity getPreviewItemsByJobId(UUID jobId, Integer limit) { - var jobCommand = getJobCommandById(jobId.toString()); - log.info("getPreviewItemsByJobId:: with jobExportType={}", jobCommand.getExportType()); - if (BULK_EDIT_IDENTIFIERS == jobCommand.getExportType()) { - var fileName = jobCommand.getId() + PATH_SEPARATOR + FilenameUtils.getName(jobCommand.getJobParameters().getString(TEMP_OUTPUT_FILE_PATH)) + CSV_EXTENSION; - try { - var itemsFormat = CsvHelper.readRecordsFromRemoteFilesStorage(remoteFilesStorage, fileName, limit, ItemFormat.class); - log.info("getPreviewItemsByJobId:: Reading of file {} complete, number of itemsFormat: {}", fileName, itemsFormat.size()); - var items = itemsFormat.stream() - .map(bulkEditParseService::mapItemFormatToItem) - .collect(Collectors.toList()); - return new ResponseEntity<>(new ItemCollection().items(items).totalRecords(items.size()), HttpStatus.OK); - } catch (Exception e) { - var msg = String.format(FAILED_TO_READ_FILE_ERROR, fileName, jobCommand.getId(), e.getMessage()); - log.error(msg); - return new ResponseEntity<>(new ItemCollection().items(Collections.emptyList()).totalRecords(0), HttpStatus.OK); - } - } else { - return new ResponseEntity<>(inventoryClient.getItemByQuery(buildPreviewQueryFromJobCommand(jobCommand, limit), limit), HttpStatus.OK); - } - } - - @Override - public ResponseEntity getPreviewHoldingsByJobId(UUID jobId, Integer limit) { - var jobCommand = getJobCommandById(jobId.toString()); - log.info("getPreviewHoldingsByJobId:: with jobExportType={}", jobCommand.getExportType()); - if (BULK_EDIT_IDENTIFIERS == jobCommand.getExportType()) { - var fileName = jobCommand.getId() + PATH_SEPARATOR + FilenameUtils.getName(jobCommand.getJobParameters().getString(TEMP_OUTPUT_FILE_PATH)) + CSV_EXTENSION; - try { - var holdingsFormat = CsvHelper.readRecordsFromRemoteFilesStorage(remoteFilesStorage, fileName, limit, HoldingsFormat.class); - log.info("getPreviewHoldingsByJobId:: Reading of file {} complete, number of holdingsFormat: {}", fileName, holdingsFormat.size()); - var holdings = holdingsFormat.stream() - .map(holdingsMapper::mapToHoldingsRecord) - .collect(Collectors.toList()); - return new ResponseEntity<>(new HoldingsRecordCollection().holdingsRecords(holdings).totalRecords(holdings.size()), HttpStatus.OK); - } catch (Exception e) { - var msg = String.format(FAILED_TO_READ_FILE_ERROR, fileName, jobCommand.getId(), e.getMessage()); - log.error(msg); - return new ResponseEntity<>(new HoldingsRecordCollection().holdingsRecords(Collections.emptyList()).totalRecords(0), HttpStatus.OK); - } - } else { - return new ResponseEntity<>(holdingClient.getHoldingsByQuery(buildPreviewQueryFromJobCommand(jobCommand, limit), limit), HttpStatus.OK); - } - } - - @Override - public ResponseEntity downloadItemsPreviewByJobId(@ApiParam(value = "UUID of the JobCommand", required = true) @PathVariable("jobId") UUID jobId) { - var jobCommand = getJobCommandById(jobId.toString()); - HttpHeaders headers = new HttpHeaders(); - headers.setContentType(MediaType.APPLICATION_OCTET_STREAM); - try { - var updatedFileName = FilenameUtils.getName(jobCommand.getJobParameters().getString(PREVIEW_FILE_NAME)); - var updatedFullPath = FilenameUtils.getFullPath(jobCommand.getJobParameters().getString(PREVIEW_FILE_NAME)); - var updatedFilePath = updatedFullPath + updatedFileName; - var content = localFilesStorage.readAllBytes(updatedFilePath); - ByteArrayResource updatedFileResource = new ByteArrayResource(content); - headers.setContentLength(content.length); - headers.setContentDispositionFormData(updatedFileName, updatedFileName); - return ResponseEntity.ok().headers(headers).body(updatedFileResource); - } catch (Exception e) { - log.error("Something went wrong during downloadItemsPreviewByJobId."); - return ResponseEntity.internalServerError().body(new DescriptiveResource(e.getMessage())); - } - } - - @Override - public ResponseEntity downloadUsersPreviewByJobId(@ApiParam(value = "UUID of the JobCommand", required = true) @PathVariable("jobId") UUID jobId) { - return downloadPreviewByJobId(jobId); - } - - @Override - public ResponseEntity downloadHoldingsPreviewByJobId(@ApiParam(value = "UUID of the JobCommand", required = true) @PathVariable("jobId") UUID jobId) { - return downloadPreviewByJobId(jobId); - } - - private ResponseEntity downloadPreviewByJobId(UUID jobId) { - var jobCommand = getJobCommandById(jobId.toString()); - var fileName = jobCommand.getJobParameters().getString(PREVIEW_FILE_NAME); - if (nonNull(fileName)) { - HttpHeaders headers = new HttpHeaders(); - headers.setContentType(MediaType.APPLICATION_OCTET_STREAM); - try (InputStream is = remoteFilesStorage.newInputStream(fileName)) { - var updatedUsersResource = new ByteArrayResource(is.readAllBytes()); - headers.setContentLength(updatedUsersResource.contentLength()); - headers.setContentDispositionFormData(fileName, fileName); - return ResponseEntity.ok().headers(headers).body(updatedUsersResource); - } catch (Exception e) { - log.error("Something went wrong during downloadPreviewByJobId."); - return ResponseEntity.internalServerError().body(new DescriptiveResource(e.getMessage())); - } - } - log.error("Preview is not available"); - throw new NotFoundException("Preview is not available"); - } - - @Override - public ResponseEntity getErrorsPreviewByJobId(@ApiParam(value = "UUID of the JobCommand", required = true) @PathVariable("jobId") UUID jobId, @NotNull @ApiParam(value = "The numbers of users to return", required = true) @Valid @RequestParam(value = "limit") Integer limit) { - var jobCommand = getJobCommandById(jobId.toString()); - var fileName = jobCommand.getId() + PATH_SEPARATOR + FilenameUtils.getName(jobCommand.getJobParameters().getString(FILE_NAME)); - log.info("downloadHoldingsPreviewByJobId:: fileName={}", fileName); - - var errors = bulkEditProcessingErrorsService.readErrorsFromCSV(jobId.toString(), fileName, limit); - return new ResponseEntity<>(errors, HttpStatus.OK); - } - @Override public ResponseEntity uploadCsvFile(UUID jobId, MultipartFile file) { log.info("uploadCsvFile:: for jobId={} ", jobId); @@ -339,23 +107,18 @@ public ResponseEntity uploadCsvFile(UUID jobId, MultipartFile file) { } prepareJobParameters(jobCommand, uploadedPath, tempIdentifiersFile); jobCommandsReceiverService.updateJobCommand(jobCommand); - if (isBulkEditUpdate(jobCommand) && jobCommand.getEntityType() == USER) { - localFilesStorage.write(workDir + jobId + PATH_SEPARATOR + INITIAL_PREFIX + file.getOriginalFilename(), file.getBytes()); - } log.info("File {} has been uploaded successfully.", file.getOriginalFilename()); - if (!isBulkEditUpdate(jobCommand)) { - var job = getBulkEditJob(jobCommand); - var jobLaunchRequest = new JobLaunchRequest(job, jobCommand.getJobParameters()); - log.info("Launching bulk edit identifiers job."); - new Thread(getRunnableWithCurrentFolioContext(() -> { - try { - exportJobManagerSync.launchJob(jobLaunchRequest); - } catch (JobExecutionException e) { - String errorMessage = format(FILE_UPLOAD_ERROR, e.getMessage()); - log.error(errorMessage); - } - })).start(); - } + var job = getBulkEditJob(jobCommand); + var jobLaunchRequest = new JobLaunchRequest(job, jobCommand.getJobParameters()); + log.info("Launching bulk edit identifiers job."); + new Thread(getRunnableWithCurrentFolioContext(() -> { + try { + exportJobManagerSync.launchJob(jobLaunchRequest); + } catch (JobExecutionException e) { + String errorMessage = format(FILE_UPLOAD_ERROR, e.getMessage()); + log.error(errorMessage); + } + })).start(); var numberOfLines = jobCommand.getJobParameters().getLong(TOTAL_CSV_LINES); return new ResponseEntity<>(Long.toString(isNull(numberOfLines) ? 0 : numberOfLines), HttpStatus.OK); } catch (Exception e) { @@ -376,13 +139,6 @@ private String saveTemporaryIdentifiersFile(UUID jobId, MultipartFile file) thro return tempFilePath; } - @Override - public ResponseEntity rollBackCsvFile(UUID jobId) { - log.info("rollBackCsvFile:: for jobId={}", jobId.toString()); - var message = bulkEditRollBackService.stopAndRollBackJobExecutionByJobId(jobId); - return new ResponseEntity<>(message, HttpStatus.OK); - } - @Override public ResponseEntity startJob(UUID jobId) { var jobCommand = getJobCommandById(jobId.toString()); @@ -405,8 +161,18 @@ public ResponseEntity startJob(UUID jobId) { return new ResponseEntity<>(HttpStatus.OK); } + @Override + public ResponseEntity getErrorsPreviewByJobId(@ApiParam(value = "UUID of the JobCommand", required = true) @PathVariable("jobId") UUID jobId, @NotNull @ApiParam(value = "The numbers of users to return", required = true) @Valid @RequestParam(value = "limit") Integer limit) { + var jobCommand = getJobCommandById(jobId.toString()); + var fileName = jobCommand.getId() + PATH_SEPARATOR + FilenameUtils.getName(jobCommand.getJobParameters().getString(FILE_NAME)); + log.info("downloadHoldingsPreviewByJobId:: fileName={}", fileName); + + var errors = bulkEditProcessingErrorsService.readErrorsFromCSV(jobId.toString(), fileName, limit); + return new ResponseEntity<>(errors, HttpStatus.OK); + } + private Job getBulkEditJob(JobCommand jobCommand) { - var jobName = BULK_EDIT_IDENTIFIERS == jobCommand.getExportType() || BULK_EDIT_UPDATE == jobCommand.getExportType() ? + var jobName = BULK_EDIT_IDENTIFIERS == jobCommand.getExportType() ? jobCommand.getExportType().getValue() + "-" + jobCommand.getEntityType() : jobCommand.getExportType().getValue(); return jobs.stream() @@ -419,12 +185,12 @@ private void prepareJobParameters(JobCommand jobCommand, String uploadedPath, St var paramsBuilder = new JobParametersBuilder(jobCommand.getJobParameters()); ofNullable(tempIdentifiersFile).ifPresent(path -> paramsBuilder.addString(TEMP_IDENTIFIERS_FILE_NAME, path, JOB_PARAMETER_DEFAULT_IDENTIFYING_VALUE)); paramsBuilder.addString(FILE_NAME, uploadedPath, JOB_PARAMETER_DEFAULT_IDENTIFYING_VALUE); - paramsBuilder.addLong(TOTAL_CSV_LINES, countLines(localFilesStorage, uploadedPath, isBulkEditUpdate(jobCommand)), JOB_PARAMETER_DEFAULT_IDENTIFYING_VALUE); - var fileName = jobCommand.getId() + PATH_SEPARATOR + (isBulkEditUpdate(jobCommand) ? EMPTY : LocalDate.now() + MATCHED_RECORDS) + FilenameUtils.getBaseName(uploadedPath); + paramsBuilder.addLong(TOTAL_CSV_LINES, countLines(localFilesStorage, uploadedPath), JOB_PARAMETER_DEFAULT_IDENTIFYING_VALUE); + var fileName = jobCommand.getId() + PATH_SEPARATOR + LocalDate.now() + MATCHED_RECORDS + FilenameUtils.getBaseName(uploadedPath); paramsBuilder.addString(TEMP_OUTPUT_FILE_PATH, workDir + fileName, JOB_PARAMETER_DEFAULT_IDENTIFYING_VALUE); paramsBuilder.addString(TEMP_LOCAL_FILE_PATH, getTempDirWithSeparatorSuffix() + springApplicationName + PATH_SEPARATOR + fileName, JOB_PARAMETER_DEFAULT_IDENTIFYING_VALUE); paramsBuilder.addString(EXPORT_TYPE, jobCommand.getExportType().getValue(), JOB_PARAMETER_DEFAULT_IDENTIFYING_VALUE); - var marcFileName = jobCommand.getId() + PATH_SEPARATOR + (isBulkEditUpdate(jobCommand) ? EMPTY : LocalDate.now() + MARC_RECORDS) + FilenameUtils.getBaseName(uploadedPath); + var marcFileName = jobCommand.getId() + PATH_SEPARATOR + LocalDate.now() + MARC_RECORDS + FilenameUtils.getBaseName(uploadedPath); paramsBuilder.addString(TEMP_OUTPUT_MARC_PATH, workDir + marcFileName); paramsBuilder.addString(TEMP_LOCAL_MARC_PATH, getTempDirWithSeparatorSuffix() + springApplicationName + PATH_SEPARATOR + marcFileName, JOB_PARAMETER_DEFAULT_IDENTIFYING_VALUE); ofNullable(jobCommand.getIdentifierType()).ifPresent(type -> @@ -434,10 +200,6 @@ private void prepareJobParameters(JobCommand jobCommand, String uploadedPath, St jobCommand.setJobParameters(paramsBuilder.toJobParameters()); } - private boolean isBulkEditUpdate(JobCommand jobCommand) { - return jobCommand.getExportType() == BULK_EDIT_UPDATE; - } - private JobCommand getJobCommandById(String jobId) { var jobCommandOptional = jobCommandsReceiverService.getBulkEditJobCommandById(jobId); if (jobCommandOptional.isEmpty()) { @@ -448,136 +210,4 @@ private JobCommand getJobCommandById(String jobId) { return jobCommandOptional.get(); } - private ItemCollection prepareItemContentUpdateResponse(UpdatesResult updatesResult, Integer limit) { - var items = updatesResult.getEntitiesForPreview().stream() - .limit(isNull(limit) ? Integer.MAX_VALUE : limit) - .map(bulkEditParseService::mapItemFormatToItem) - .collect(Collectors.toList()); - return new ItemCollection().items(items).totalRecords(updatesResult.getTotal()); - } - - private UserCollection prepareUserContentUpdateResponse(UpdatesResult updatesResult, Integer limit) { - var users = updatesResult.getEntitiesForPreview().stream() - .limit(isNull(limit) ? Integer.MAX_VALUE : limit) - .map(bulkEditParseService::mapUserFormatToUser) - .collect(Collectors.toList()); - return new UserCollection().users(users).totalRecords(updatesResult.getTotal()); - } - - private HoldingsRecordCollection prepareHoldingsContentUpdateResponse(UpdatesResult updatesResult, Integer limit) { - var holdingsRecords = updatesResult.getEntitiesForPreview().stream() - .limit(isNull(limit) ? Integer.MAX_VALUE : limit) - .map(holdingsMapper::mapToHoldingsRecord) - .collect(Collectors.toList()); - return new HoldingsRecordCollection().holdingsRecords(holdingsRecords).totalRecords(updatesResult.getTotal()); - } - - private String buildPreviewUsersQueryFromJobCommand(JobCommand jobCommand, int limit) { - if (isBulkEditUpdate(jobCommand)) { - ofNullable(jobCommand.getJobParameters().getString(FILE_NAME)).ifPresent(filename -> { - var basename = FilenameUtils.getBaseName(filename); - if (!basename.startsWith(INITIAL_PREFIX)) { - jobCommand.setJobParameters(new JobParametersBuilder(jobCommand.getJobParameters()).addString(FILE_NAME, - filename.replace(basename, INITIAL_PREFIX + basename)).toJobParameters()); - } - }); - } - return buildPreviewQueryFromJobCommand(jobCommand, limit); - } - - private String buildPreviewQueryFromJobCommand(JobCommand jobCommand, int limit) { - switch(jobCommand.getExportType()) { - case BULK_EDIT_UPDATE: - var query = buildPreviewQueryFromCsv(jobCommand, limit); - return query.replace("()", "(default)"); - case BULK_EDIT_QUERY: - return jobCommand.getJobParameters().getString(QUERY); - default: - throw new NonSupportedEntityException(format("Non-supported export type: %s", jobCommand.getExportType())); - } - } - - private String buildPreviewQueryFromCsv(JobCommand jobCommand, int limit) { - var fileName = extractFileName(jobCommand); - if (StringUtils.isEmpty(fileName)) throw new FileOperationException("File for preview is not present or was not uploaded"); - if (!fileName.contains(CSV_EXTENSION)) fileName += CSV_EXTENSION; - try { - Reader inputReader; - var minioFileName = nonNull(jobCommand.getJobParameters().getString(UPDATED_FILE_NAME)) ? jobCommand.getId() + PATH_SEPARATOR + FilenameUtils.getName(fileName) : PREVIEW_PREFIX + FilenameUtils.getName(fileName); - if (remoteFilesStorage.containsFile(minioFileName)) { - inputReader = new InputStreamReader(remoteFilesStorage.newInputStream(minioFileName)); - } else { - inputReader = new InputStreamReader(localFilesStorage.newInputStream(fileName)); - } - try (var reader = new CSVReader(inputReader)) { - var values = reader.readAll().stream() - .skip(getNumberOfLinesToSkip(jobCommand)) - .limit(limit) - .map(line -> extractIdentifiersFromLine(line, jobCommand)) - .map(identifier -> String.format("\"%s\"", identifier)) - .collect(Collectors.joining(" OR ", "(", ")")); - var identifierType = getIdentifierType(jobCommand); - return format(getMatchPattern(identifierType), resolveIdentifier(identifierType), values); - } - - } catch (Exception e) { - throw new FileOperationException(format("Failed to read %s file, reason: %s", fileName, e.getMessage())); - } - } - - private String getIdentifierType(JobCommand jobCommand) { - if (jobCommand.getEntityType() == HOLDINGS_RECORD) { - return IdentifierType.ID.getValue(); - } - return jobCommand.getIdentifierType().getValue(); - } - - private String extractIdentifiersFromLine(String[] line, JobCommand jobCommand) { - var identifierIndex = getIdentifierIndex(jobCommand); - if (line.length > identifierIndex + 1) { - return line[identifierIndex]; - } else if (line.length == 1) { - return line[0]; - } - return EMPTY; - } - - private int getNumberOfLinesToSkip(JobCommand jobCommand) { - if (BULK_EDIT_UPDATE == jobCommand.getExportType()) { - return nonNull(jobCommand.getJobParameters().getString(UPDATED_FILE_NAME)) ? 1 : 0; - } - return 0; - } - - private String extractFileName(JobCommand jobCommand) { - if (isItemUpdatePreview(jobCommand) || isHoldingUpdatePreview(jobCommand)) { - return jobCommand.getJobParameters().getString(UPDATED_FILE_NAME); - } - var fileProperty = isUserUpdatePreview(jobCommand) ? TEMP_OUTPUT_FILE_PATH : FILE_NAME; - return jobCommand.getJobParameters().getString(fileProperty); - } - - private boolean isHoldingUpdatePreview(JobCommand jobCommand) { - return jobCommand.getExportType() == BULK_EDIT_UPDATE && jobCommand.getEntityType() == HOLDINGS_RECORD; - } - - private boolean isUserUpdatePreview(JobCommand jobCommand) { - return jobCommand.getExportType() == BULK_EDIT_UPDATE && jobCommand.getEntityType() == USER; - } - - private boolean isItemUpdatePreview(JobCommand jobCommand) { - return jobCommand.getExportType() == BULK_EDIT_UPDATE && jobCommand.getEntityType() == ITEM; - } - - private int getIdentifierIndex(JobCommand jobCommand) { - if (USER == jobCommand.getEntityType()) { - return Arrays.asList(UserFormat.getUserFieldsArray()).indexOf(resolveIdentifier(jobCommand.getIdentifierType().getValue())); - } else if (ITEM == jobCommand.getEntityType()) { - return Arrays.asList(ItemFormat.getItemFieldsArray()).indexOf(resolveIdentifier(jobCommand.getIdentifierType().getValue())); - } else if (HOLDINGS_RECORD == jobCommand.getEntityType()) { - return Arrays.asList(HoldingsFormat.getHoldingsFieldsArray()).indexOf(IdentifierType.ID.getValue().toLowerCase()); - } else { - throw new NonSupportedEntityException(format("Non-supported entity type: %s", jobCommand.getEntityType())); - } - } } diff --git a/src/main/java/org/folio/dew/controller/PermissionsSelfCheckController.java b/src/main/java/org/folio/dew/controller/PermissionsSelfCheckController.java new file mode 100644 index 000000000..4937becf9 --- /dev/null +++ b/src/main/java/org/folio/dew/controller/PermissionsSelfCheckController.java @@ -0,0 +1,24 @@ +package org.folio.dew.controller; + +import lombok.RequiredArgsConstructor; +import org.folio.dew.service.UserPermissionsService; +import org.openapitools.api.PermissionsSelfCheckApi; +import org.springframework.http.HttpStatus; +import org.springframework.http.ResponseEntity; +import org.springframework.web.bind.annotation.RequestMapping; +import org.springframework.web.bind.annotation.RestController; + +import java.util.List; + +@RestController +@RequestMapping("/bulk-edit") +@RequiredArgsConstructor +public class PermissionsSelfCheckController implements PermissionsSelfCheckApi { + + private final UserPermissionsService userPermissionsService; + + @Override + public ResponseEntity> getUsersPermissions() { + return new ResponseEntity<>(userPermissionsService.getPermissions(), HttpStatus.OK); + } +} diff --git a/src/main/java/org/folio/dew/domain/bean/Consortia.java b/src/main/java/org/folio/dew/domain/bean/Consortia.java new file mode 100644 index 000000000..1a5d1730e --- /dev/null +++ b/src/main/java/org/folio/dew/domain/bean/Consortia.java @@ -0,0 +1,12 @@ +package org.folio.dew.domain.bean; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; + +import lombok.Data; + +@Data +@JsonIgnoreProperties(ignoreUnknown = true) +public class Consortia { + + private String id; +} diff --git a/src/main/java/org/folio/dew/domain/bean/ConsortiaCollection.java b/src/main/java/org/folio/dew/domain/bean/ConsortiaCollection.java new file mode 100644 index 000000000..ce6f51ca3 --- /dev/null +++ b/src/main/java/org/folio/dew/domain/bean/ConsortiaCollection.java @@ -0,0 +1,18 @@ +package org.folio.dew.domain.bean; + +import java.util.ArrayList; +import java.util.List; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonProperty; + +import lombok.Data; +import org.folio.dew.domain.bean.Consortia; + +@Data +@JsonIgnoreProperties(ignoreUnknown = true) +public class ConsortiaCollection { + + @JsonProperty("consortia") + private List consortia = new ArrayList<>(); +} diff --git a/src/main/java/org/folio/dew/domain/dto/HoldingsFormat.java b/src/main/java/org/folio/dew/domain/dto/HoldingsFormat.java index bcc146a0f..21d1c3fc7 100644 --- a/src/main/java/org/folio/dew/domain/dto/HoldingsFormat.java +++ b/src/main/java/org/folio/dew/domain/dto/HoldingsFormat.java @@ -19,7 +19,6 @@ @AllArgsConstructor public class HoldingsFormat implements Formatable { private HoldingsRecord original; - private String tenantId; @CsvBindByName(column = "Holdings UUID") @CsvBindByPosition(position = 0) @@ -141,6 +140,10 @@ public class HoldingsFormat implements Formatable { @CsvBindByPosition(position = 29) private String tags; + @CsvBindByName(column = "Tenant") + @CsvBindByPosition(position = 30) + private String tenantId; + private String instanceHrid; private String itemBarcode; diff --git a/src/main/java/org/folio/dew/domain/dto/ItemFormat.java b/src/main/java/org/folio/dew/domain/dto/ItemFormat.java index 66901edb9..748f2c932 100644 --- a/src/main/java/org/folio/dew/domain/dto/ItemFormat.java +++ b/src/main/java/org/folio/dew/domain/dto/ItemFormat.java @@ -20,7 +20,6 @@ public class ItemFormat implements Formatable { private org.folio.dew.domain.dto.Item original; - private String tenantId; @CsvBindByName(column = "Item UUID") @CsvBindByPosition(position = 0) @@ -198,6 +197,10 @@ public class ItemFormat implements Formatable { @CsvBindByPosition(position = 43) private String holdingsRecordId; + @CsvBindByName(column = "Tenant") + @CsvBindByPosition(position = 44) + private String tenantId; + public static String[] getItemFieldsArray() { return FieldUtils.getFieldsListWithAnnotation(ItemFormat.class, CsvBindByName.class).stream() .map(Field::getName) diff --git a/src/main/java/org/folio/dew/error/BulkEditSkipListener.java b/src/main/java/org/folio/dew/error/BulkEditSkipListener.java index d32968e97..c8bdb6cd3 100644 --- a/src/main/java/org/folio/dew/error/BulkEditSkipListener.java +++ b/src/main/java/org/folio/dew/error/BulkEditSkipListener.java @@ -9,6 +9,7 @@ import org.springframework.batch.core.JobExecution; import org.springframework.batch.core.annotation.OnSkipInProcess; import org.springframework.batch.core.configuration.annotation.JobScope; +import org.springframework.batch.core.configuration.annotation.StepScope; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Component; @@ -16,12 +17,12 @@ @Log4j2 @RequiredArgsConstructor -@JobScope +@StepScope @Component public class BulkEditSkipListener { private final BulkEditProcessingErrorsService bulkEditProcessingErrorsService; - @Value("#{jobExecution}") + @Value("#{stepExecution.jobExecution}") private JobExecution jobExecution; @OnSkipInProcess diff --git a/src/main/java/org/folio/dew/repository/BaseFilesStorage.java b/src/main/java/org/folio/dew/repository/BaseFilesStorage.java index e6f377305..18b2fa899 100644 --- a/src/main/java/org/folio/dew/repository/BaseFilesStorage.java +++ b/src/main/java/org/folio/dew/repository/BaseFilesStorage.java @@ -57,14 +57,18 @@ import static io.minio.ObjectWriteArgs.MIN_MULTIPART_SIZE; import static java.lang.String.format; +import static org.folio.dew.utils.Constants.PATH_SEPARATOR; @Log4j2 public class BaseFilesStorage implements S3CompatibleStorage { + private static final String SET_VALUE = ""; + private static final String NOT_SET_VALUE = ""; private final MinioClient client; private S3Client s3Client; private final String bucket; private final String region; + private final String subPath; private final boolean isComposeWithAwsSdk; @@ -74,10 +78,12 @@ public BaseFilesStorage(MinioClientProperties properties) { final String regionName = properties.getRegion(); final String bucketName = properties.getBucket(); final String secretKey = properties.getSecretKey(); + subPath = properties.getSubPath(); isComposeWithAwsSdk = properties.isComposeWithAwsSdk(); final boolean isForcePathStyle = properties.isForcePathStyle(); - log.info("Creating MinIO client endpoint {},region {},bucket {},accessKey {},secretKey {}, isComposedWithAwsSdk {}.", endpoint, regionName, bucketName, - StringUtils.isNotBlank(accessKey) ? "" : "", StringUtils.isNotBlank(secretKey) ? "" : "", isComposeWithAwsSdk); + log.info("Creating MinIO client endpoint {},region {},bucket {},accessKey {},secretKey {}, subPath {}, isComposedWithAwsSdk {}.", endpoint, regionName, bucketName, + StringUtils.isNotBlank(accessKey) ? SET_VALUE : NOT_SET_VALUE, StringUtils.isNotBlank(secretKey) ? SET_VALUE : NOT_SET_VALUE, + StringUtils.isNotBlank(subPath) ? SET_VALUE : NOT_SET_VALUE, isComposeWithAwsSdk); var builder = MinioClient.builder().endpoint(endpoint); if (StringUtils.isNotBlank(regionName)) { @@ -149,6 +155,7 @@ public void createBucketIfNotExists() { * @throws IOException - if an I/O error occurs */ public String upload(String path, String filename) throws IOException { + path = getS3Path(path); try { return client.uploadObject(UploadObjectArgs.builder() .bucket(bucket) @@ -172,7 +179,7 @@ public String upload(String path, String filename) throws IOException { * @throws IOException - if an I/O error occurs */ public String write(String path, byte[] bytes, Map headers) throws IOException { - + path = getS3Path(path); if (isComposeWithAwsSdk) { log.info("Writing with using AWS SDK client"); s3Client.putObject(PutObjectRequest.builder().bucket(bucket) @@ -211,7 +218,7 @@ public String write(String path, byte[] bytes) throws IOException { * @throws IOException - if an I/O error occurs */ public String writeFile(String path, Path inputPath, Map headers) throws IOException { - + path = getS3Path(path); if (isComposeWithAwsSdk) { log.info("Writing file using AWS SDK client"); s3Client.putObject(PutObjectRequest.builder().bucket(bucket) @@ -248,6 +255,7 @@ public String writeFile(String destPath, Path inputPath) throws IOException { * @throws IOException if an I/O error occurs */ public void append(String path, byte[] bytes) throws IOException { + path = getS3Path(path); try { if (notExists(path)) { log.info("Appending non-existing file"); @@ -349,6 +357,7 @@ public void append(String path, byte[] bytes) throws IOException { * @throws FileOperationException if an I/O error occurs */ public void delete(String path) { + path = getS3Path(path); try { var paths = walk(path).collect(Collectors.toList()); @@ -377,7 +386,7 @@ public void delete(String path) { * @throws FileOperationException if an I/O error occurs */ public Stream walk(String path) { - return getInternalStructure(path, true); + return getInternalStructure(getS3Path(path), true); } /** @@ -387,6 +396,7 @@ public Stream walk(String path) { * @return true if file exists, otherwise - false */ public boolean exists(String path) { + path = getS3Path(path); var iterator = client.listObjects(ListObjectsArgs.builder() .bucket(bucket) .region(region) @@ -419,6 +429,7 @@ public boolean notExists(String path) { * @throws IOException - if an I/O error occurs reading from the file */ public InputStream newInputStream(String path) throws IOException { + path = getS3Path(path); try { return client.getObject(GetObjectArgs.builder() .bucket(bucket) @@ -546,4 +557,14 @@ private Stream getInternalStructure(String path, boolean isRecursive) { return null; } } + + public String getS3Path(String path) { + if (StringUtils.isBlank(subPath) || StringUtils.startsWith(path, subPath + PATH_SEPARATOR)) { + return path; + } + if (path.startsWith(PATH_SEPARATOR)) { + return subPath + path; + } + return subPath + PATH_SEPARATOR + path; + } } diff --git a/src/main/java/org/folio/dew/repository/RemoteFilesStorage.java b/src/main/java/org/folio/dew/repository/RemoteFilesStorage.java index 345a22990..065546ae1 100644 --- a/src/main/java/org/folio/dew/repository/RemoteFilesStorage.java +++ b/src/main/java/org/folio/dew/repository/RemoteFilesStorage.java @@ -2,7 +2,6 @@ import io.minio.ComposeObjectArgs; import io.minio.ComposeSource; -import io.minio.GetObjectArgs; import io.minio.GetPresignedObjectUrlArgs; import io.minio.ListObjectsArgs; import io.minio.MinioClient; @@ -33,7 +32,6 @@ import org.apache.commons.lang3.StringUtils; import org.folio.dew.config.properties.RemoteFilesStorageProperties; import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.beans.factory.annotation.Value; import org.springframework.http.HttpHeaders; import org.springframework.stereotype.Repository; @@ -75,16 +73,11 @@ public String uploadObject(String object, String filename, String downloadFilena return result; } - public void downloadObject(String objectToGet, String fileToSave) throws IOException, InvalidKeyException, - InvalidResponseException, InsufficientDataException, NoSuchAlgorithmException, ServerException, - InternalException, XmlParserException, ErrorResponseException { - localFilesStorage.write(fileToSave, client.getObject(GetObjectArgs.builder().bucket(bucket).object(objectToGet).build()).readAllBytes()); - } - public boolean containsFile(String fileName) throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException { - for (Result itemResult : client.listObjects(ListObjectsArgs.builder().bucket(bucket).prefix(fileName).build())) { + fileName = getS3Path(fileName); + for (Result itemResult : client.listObjects(ListObjectsArgs.builder().bucket(bucket).prefix(getS3Path(fileName)).build())) { if (fileName.equals(itemResult.get().objectName())) { return true; } @@ -96,8 +89,9 @@ public String composeObject(String destObject, List sourceObjects, Strin String contentType) throws IOException, InvalidKeyException, InvalidResponseException, InsufficientDataException, NoSuchAlgorithmException, ServerException, InternalException, XmlParserException, ErrorResponseException { + destObject = getS3Path(destObject); List sources = sourceObjects.stream() - .map(so -> ComposeSource.builder().bucket(bucket).object(so).build()) + .map(so -> ComposeSource.builder().bucket(bucket).object(getS3Path(so)).build()) .collect(Collectors.toList()); log.info("Composing object {},sources [{}],downloadFilename {},contentType {}.", destObject, sources.stream().map(s -> String.format("bucket %s,object %s", s.bucket(), s.object())).collect(Collectors.joining(",")), @@ -114,7 +108,7 @@ public Iterable> removeObjects(List objects) { log.info("Deleting objects [{}].", StringUtils.join(objects, ",")); return client.removeObjects(RemoveObjectsArgs.builder() .bucket(bucket) - .objects(objects.stream().map(DeleteObject::new).collect(Collectors.toList())) + .objects(objects.stream().map(this::getS3Path).map(DeleteObject::new).toList()) .build()); } @@ -124,7 +118,7 @@ public String objectToPresignedObjectUrl(String object) String result = client.getPresignedObjectUrl(GetPresignedObjectUrlArgs.builder() .method(Method.GET) .bucket(bucket) - .object(object) + .object(getS3Path(object)) .region(region) .expiry(urlExpirationTimeInSeconds, TimeUnit.SECONDS) .build()); diff --git a/src/main/java/org/folio/dew/service/BulkEditChangedRecordsService.java b/src/main/java/org/folio/dew/service/BulkEditChangedRecordsService.java deleted file mode 100644 index 28db145b6..000000000 --- a/src/main/java/org/folio/dew/service/BulkEditChangedRecordsService.java +++ /dev/null @@ -1,27 +0,0 @@ -package org.folio.dew.service; - -import org.springframework.stereotype.Service; - -import java.util.HashSet; -import java.util.Map; -import java.util.Set; -import java.util.concurrent.ConcurrentHashMap; - -@Service -public class BulkEditChangedRecordsService { - private final Map> changedIdsMap = new ConcurrentHashMap<>(); - - public void addUserId(String userId, String jobId) { - var ids = changedIdsMap.computeIfAbsent(jobId, key -> new HashSet<>()); - ids.add(userId); - } - - public void removeUserId(String userId, String jobId) { - var ids = changedIdsMap.getOrDefault(jobId, new HashSet<>()); - ids.remove(userId); - } - - public Set fetchChangedUserIds(String jobId) { - return changedIdsMap.remove(jobId); - } -} diff --git a/src/main/java/org/folio/dew/service/BulkEditConfigurationService.java b/src/main/java/org/folio/dew/service/BulkEditConfigurationService.java index 5872bbdb0..64498a396 100644 --- a/src/main/java/org/folio/dew/service/BulkEditConfigurationService.java +++ b/src/main/java/org/folio/dew/service/BulkEditConfigurationService.java @@ -2,6 +2,7 @@ import static org.folio.dew.domain.dto.InventoryItemStatus.NameEnum.AVAILABLE; import static org.folio.dew.domain.dto.InventoryItemStatus.NameEnum.INTELLECTUAL_ITEM; +import static org.folio.dew.domain.dto.InventoryItemStatus.NameEnum.IN_PROCESS; import static org.folio.dew.domain.dto.InventoryItemStatus.NameEnum.IN_PROCESS_NON_REQUESTABLE_; import static org.folio.dew.domain.dto.InventoryItemStatus.NameEnum.LONG_MISSING; import static org.folio.dew.domain.dto.InventoryItemStatus.NameEnum.MISSING; @@ -40,6 +41,7 @@ public class BulkEditConfigurationService { allowedStatuses.put(AVAILABLE, Arrays.asList(MISSING, WITHDRAWN, + IN_PROCESS, IN_PROCESS_NON_REQUESTABLE_, INTELLECTUAL_ITEM, LONG_MISSING, @@ -118,14 +120,18 @@ public class BulkEditConfigurationService { LONG_MISSING, RESTRICTED, UNAVAILABLE)); + allowedStatuses.put(IN_PROCESS, + Arrays.asList(MISSING, + WITHDRAWN)); } - public void checkBulkEditConfiguration() { + public void updateBulkEditConfiguration() { var configurations = configurationClient.getConfigurations(String.format(BULK_EDIT_CONFIGURATIONS_QUERY_TEMPLATE, MODULE_NAME, STATUSES_CONFIG_NAME)); - if (configurations.getConfigs().isEmpty()) { - log.info("Bulk-edit configuration was not found, uploading default"); - configurationClient.postConfiguration(buildDefaultConfig()); + if (!configurations.getConfigs().isEmpty()) { + log.info("Deleting old bulk edit statuses configuration"); + configurationClient.deleteConfiguration(configurations.getConfigs().get(0).getId()); } + configurationClient.postConfiguration(buildDefaultConfig()); } @SneakyThrows diff --git a/src/main/java/org/folio/dew/service/BulkEditItemContentUpdateService.java b/src/main/java/org/folio/dew/service/BulkEditItemContentUpdateService.java deleted file mode 100644 index 78666ab05..000000000 --- a/src/main/java/org/folio/dew/service/BulkEditItemContentUpdateService.java +++ /dev/null @@ -1,264 +0,0 @@ -package org.folio.dew.service; - -import static java.time.ZoneOffset.UTC; -import static java.util.Objects.isNull; -import static java.util.Objects.nonNull; -import static org.apache.commons.lang3.ObjectUtils.isEmpty; -import static org.apache.commons.lang3.StringUtils.EMPTY; -import static org.folio.dew.domain.dto.ItemContentUpdate.ActionEnum.CLEAR_FIELD; -import static org.folio.dew.domain.dto.ItemContentUpdate.ActionEnum.REPLACE_WITH; -import static org.folio.dew.domain.dto.ItemContentUpdate.OptionEnum.PERMANENT_LOAN_TYPE; -import static org.folio.dew.domain.dto.ItemContentUpdate.OptionEnum.PERMANENT_LOCATION; -import static org.folio.dew.domain.dto.ItemContentUpdate.OptionEnum.STATUS; -import static org.folio.dew.domain.dto.ItemContentUpdate.OptionEnum.TEMPORARY_LOCATION; -import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_UPDATE; -import static org.folio.dew.domain.dto.JobParameterNames.PREVIEW_FILE_NAME; -import static org.folio.dew.domain.dto.JobParameterNames.TEMP_OUTPUT_FILE_PATH; -import static org.folio.dew.domain.dto.JobParameterNames.UPDATED_FILE_NAME; -import static org.folio.dew.utils.BulkEditProcessorHelper.dateToString; -import static org.folio.dew.utils.Constants.ARRAY_DELIMITER; -import static org.folio.dew.utils.Constants.COMMA; -import static org.folio.dew.utils.Constants.BULKEDIT_DIR_NAME; -import static org.folio.dew.utils.Constants.CSV_EXTENSION; -import static org.folio.dew.utils.Constants.FILE_NAME; -import static org.folio.dew.utils.Constants.IDENTIFIER_TYPE; -import static org.folio.dew.utils.Constants.NO_CHANGE_MESSAGE; -import static org.folio.dew.utils.Constants.PATH_SEPARATOR; -import static org.folio.dew.utils.Constants.PREVIEW_PREFIX; -import static org.folio.dew.utils.Constants.STATUS_FIELD_CAN_NOT_CLEARED; -import static org.folio.dew.utils.Constants.STATUS_VALUE_NOT_ALLOWED; -import static org.folio.dew.utils.Constants.UPDATED_PREFIX; -import static org.folio.dew.utils.Constants.getWorkingDirectory; - -import lombok.RequiredArgsConstructor; -import lombok.extern.log4j.Log4j2; -import org.apache.commons.io.FilenameUtils; -import org.folio.de.entity.JobCommand; -import org.folio.dew.domain.dto.ItemContentUpdate; -import org.folio.dew.domain.dto.ItemContentUpdateCollection; -import org.folio.dew.domain.dto.ItemFormat; -import org.folio.dew.error.FileOperationException; -import org.folio.dew.repository.LocalFilesStorage; -import org.folio.dew.repository.RemoteFilesStorage; -import org.folio.dew.utils.CsvHelper; -import org.springframework.batch.core.JobParametersBuilder; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.stereotype.Component; - -import jakarta.annotation.PostConstruct; -import java.time.LocalDateTime; -import java.util.Date; -import java.util.List; -import java.util.Objects; - -@Component -@RequiredArgsConstructor -@Log4j2 -public class BulkEditItemContentUpdateService { - private String workdir; - - @Value("${spring.application.name}") - private String springApplicationName; - - private final ItemReferenceService itemReferenceService; - private final RemoteFilesStorage remoteFilesStorage; - private final LocalFilesStorage localFilesStorage; - private final BulkEditProcessingErrorsService errorsService; - - @PostConstruct - public void postConstruct() { - workdir = getWorkingDirectory(springApplicationName, BULKEDIT_DIR_NAME); - } - - public UpdatesResult processContentUpdates(JobCommand jobCommand, ItemContentUpdateCollection contentUpdates) { - var outputFileName = workdir + jobCommand.getId() + PATH_SEPARATOR + UPDATED_PREFIX + FilenameUtils.getName(jobCommand.getJobParameters().getString(TEMP_OUTPUT_FILE_PATH)) + CSV_EXTENSION; - try { - log.info("Processing content updates for job id {}", jobCommand.getId()); - localFilesStorage.delete(outputFileName); - remoteFilesStorage.downloadObject(jobCommand.getId() + PATH_SEPARATOR + FilenameUtils.getName(jobCommand.getJobParameters().getString(TEMP_OUTPUT_FILE_PATH)) + CSV_EXTENSION, outputFileName); - var updateResult = new UpdatesResult(); - var records = CsvHelper.readRecordsFromStorage(localFilesStorage, outputFileName, ItemFormat.class, true); - log.info("Reading of file {} complete, number of itemFormats: {}", outputFileName, records.size()); - updateResult.setTotal(records.size()); - var contentUpdated = applyContentUpdates(records, contentUpdates, jobCommand); - log.info("Finished processing content updates: {} records, {} preview", contentUpdated.getUpdated().size(), contentUpdated.getPreview().size()); - updateResult.setEntitiesForPreview(contentUpdated.getPreview()); - var previewOutputFileName = workdir + jobCommand.getId() + PATH_SEPARATOR + PREVIEW_PREFIX + FilenameUtils.getName(jobCommand.getJobParameters().getString(TEMP_OUTPUT_FILE_PATH)) + CSV_EXTENSION; - saveResultToFile(contentUpdated.getPreview(), jobCommand, previewOutputFileName, PREVIEW_FILE_NAME); - saveResultToFile(contentUpdated.getUpdated(), jobCommand, outputFileName, UPDATED_FILE_NAME); - jobCommand.setExportType(BULK_EDIT_UPDATE); - return updateResult; - } catch (Exception e) { - var msg = String.format("Failed to read %s item records file for job id %s, reason: %s", outputFileName, jobCommand.getId(), e.getMessage()); - log.error(msg); - throw new FileOperationException(msg); - } - } - - private void saveResultToFile(List itemFormats, JobCommand jobCommand, String outputFileName, String propertyValue) { - try { - CsvHelper.saveRecordsToStorage(localFilesStorage, itemFormats, ItemFormat.class, outputFileName); - log.info("Saved {}", outputFileName); - jobCommand.setJobParameters(new JobParametersBuilder(jobCommand.getJobParameters()) - .addString(propertyValue, outputFileName) - .toJobParameters()); - } catch (Exception e) { - var msg = String.format("Failed to write %s item records file for job id %s, reason: %s", outputFileName, jobCommand.getId(), e.getMessage()); - log.error(msg); - throw new FileOperationException(msg); - } - } - - private ContentUpdateRecords applyContentUpdates(List itemFormats, ItemContentUpdateCollection contentUpdates, JobCommand jobCommand) { - var result = new ContentUpdateRecords(); - var errorStringBuilder = new StringBuilder(); - for (ItemFormat itemFormat: itemFormats) { - var updatedItemFormat = itemFormat; - var errorMessage = new ErrorMessage(); - - for (ItemContentUpdate contentUpdate: contentUpdates.getItemContentUpdates()) { - updatedItemFormat = applyContentUpdate(updatedItemFormat, contentUpdate, errorMessage); - } - - if (!Objects.equals(itemFormat, updatedItemFormat)) { - if (isLocationChange(contentUpdates)) { - updateEffectiveLocation(updatedItemFormat); - } - result.addToUpdated(updatedItemFormat); - result.addToPreview(applyUpdatesForPreview(contentUpdates, updatedItemFormat)); - } else { - var previewItemFormat = applyUpdatesForPreview(contentUpdates, itemFormat); - result.addToPreview(previewItemFormat); - if (Objects.equals(itemFormat, previewItemFormat)) { - errorMessage.setValue(NO_CHANGE_MESSAGE); - } - } - - if (errorMessage.getValue() != null) { - errorStringBuilder - .append(itemFormat.getIdentifier(jobCommand.getJobParameters().getString(IDENTIFIER_TYPE))) - .append(COMMA) - .append(errorMessage.getValue()) - .append(System.lineSeparator()); - } - } - if (!errorStringBuilder.toString().isEmpty()) { - errorsService.saveErrorInCSV(jobCommand.getId().toString(), errorStringBuilder.toString(), FilenameUtils.getName(jobCommand.getJobParameters().getString(FILE_NAME))); - } - return result; - } - - - private ItemFormat applyContentUpdate(ItemFormat itemFormat, ItemContentUpdate contentUpdate, ErrorMessage errorMessage) { - if (REPLACE_WITH == contentUpdate.getAction()) { - return applyReplaceWith(itemFormat, contentUpdate, errorMessage); - } else if (CLEAR_FIELD == contentUpdate.getAction()) { - if (STATUS == contentUpdate.getOption()) { - errorMessage.setValue(STATUS_FIELD_CAN_NOT_CLEARED); - } else if (PERMANENT_LOAN_TYPE == contentUpdate.getOption()) { - errorMessage.setValue("Permanent loan type cannot be cleared"); - } else { - return applyClearField(itemFormat, contentUpdate); - } - } - return itemFormat; - } - - private ItemFormat applyReplaceWith(ItemFormat itemFormat, ItemContentUpdate contentUpdate, ErrorMessage errorMessage) { - var newValue = isEmpty(contentUpdate.getValue()) ? EMPTY : contentUpdate.getValue().toString(); - switch (contentUpdate.getOption()) { - case PERMANENT_LOAN_TYPE: - return replacePermanentLoanTypeIfAllowed(itemFormat, newValue, errorMessage); - case TEMPORARY_LOAN_TYPE: - return itemFormat.withTemporaryLoanType(newValue); - case TEMPORARY_LOCATION: - return itemFormat.withTemporaryLocation(newValue); - case PERMANENT_LOCATION: - return itemFormat.withPermanentLocation(newValue); - case STATUS: - return replaceStatusIfAllowed(itemFormat, newValue, errorMessage); - default: - return itemFormat; - } - } - - private ItemFormat replacePermanentLoanTypeIfAllowed(ItemFormat itemFormat, String newValue, ErrorMessage errorMessage) { - if (newValue.isEmpty()) { - errorMessage.setValue("Permanent loan type value cannot be empty"); - return itemFormat; - } else { - return itemFormat.withPermanentLoanType(newValue); - } - } - - private ItemFormat applyClearField(ItemFormat itemFormat, ItemContentUpdate contentUpdate) { - switch (contentUpdate.getOption()) { - case PERMANENT_LOCATION: - return itemFormat.withPermanentLocation(EMPTY); - case TEMPORARY_LOCATION: - return itemFormat.withTemporaryLocation(EMPTY); - case TEMPORARY_LOAN_TYPE: - return itemFormat.withTemporaryLoanType(EMPTY); - default: - return itemFormat; - } - } - - private void updateEffectiveLocation(ItemFormat itemFormat) { - if (isEmpty(itemFormat.getTemporaryLocation())) { - itemFormat.setEffectiveLocation(isEmpty(itemFormat.getPermanentLocation()) ? - itemReferenceService.getHoldingEffectiveLocationCodeById(itemFormat.getHoldingsRecordId()) : - itemFormat.getPermanentLocation()); - } else { - itemFormat.setEffectiveLocation(itemFormat.getTemporaryLocation()); - } - } - - private boolean isLocationChange(ItemContentUpdateCollection contentUpdates) { - return contentUpdates.getItemContentUpdates().stream() - .anyMatch(update -> TEMPORARY_LOCATION == update.getOption() || PERMANENT_LOCATION == update.getOption()); - } - - private ItemFormat replaceStatusIfAllowed(ItemFormat itemFormat, String newStatus, ErrorMessage errorMessage) { - var currentStatus = extractStatusName(itemFormat.getStatus()); - if (!currentStatus.equals(newStatus)) { - if (itemReferenceService.getAllowedStatuses(currentStatus).contains(newStatus)) { - return itemFormat.withStatus(String.join(ARRAY_DELIMITER, newStatus, dateToString(Date.from(LocalDateTime.now().atZone(UTC).toInstant())))); - } else { - var msg = String.format(STATUS_VALUE_NOT_ALLOWED, newStatus); - errorMessage.setValue(msg); - } - } - return itemFormat; - } - - private String extractStatusName(String s) { - var tokens = s.split(ARRAY_DELIMITER, -1); - return tokens.length > 0 ? tokens[0] : EMPTY; - } - - private ItemFormat applyUpdatesForPreview(ItemContentUpdateCollection contentUpdates, ItemFormat itemFormat) { - var updatedItemFormat = applyStatusUpdateForPreview(contentUpdates, itemFormat); - return applyLoanTypeUpdateForPreview(contentUpdates, updatedItemFormat); - } - - private ItemFormat applyStatusUpdateForPreview(ItemContentUpdateCollection contentUpdates, ItemFormat itemFormat) { - var statusUpdate = contentUpdates.getItemContentUpdates().stream() - .filter(contentUpdate -> contentUpdate.getOption() == STATUS) - .findFirst(); - if (statusUpdate.isPresent() && nonNull(statusUpdate.get().getValue()) && !extractStatusName(itemFormat.getStatus()).equals(statusUpdate.get().getValue())) { - return itemFormat.withStatus(String.join(ARRAY_DELIMITER, statusUpdate.get().getValue().toString(), dateToString(Date.from(LocalDateTime.now().atZone(UTC).toInstant())))); - } - return itemFormat; - } - - private ItemFormat applyLoanTypeUpdateForPreview(ItemContentUpdateCollection contentUpdates, ItemFormat itemFormat) { - var update = contentUpdates.getItemContentUpdates().stream() - .filter(contentUpdate -> contentUpdate.getOption() == PERMANENT_LOAN_TYPE) - .findFirst(); - if (update.isPresent() && !Objects.equals(update.get().getValue(), itemFormat.getPermanentLoanType())) { - return itemFormat.withPermanentLoanType(isNull(update.get().getValue()) ? EMPTY : update.get().getValue().toString()); - } - return itemFormat; - } -} diff --git a/src/main/java/org/folio/dew/service/BulkEditParseService.java b/src/main/java/org/folio/dew/service/BulkEditParseService.java deleted file mode 100644 index e62953fc2..000000000 --- a/src/main/java/org/folio/dew/service/BulkEditParseService.java +++ /dev/null @@ -1,445 +0,0 @@ -package org.folio.dew.service; - -import static org.apache.commons.lang3.StringUtils.isBlank; -import static org.apache.commons.lang3.StringUtils.isEmpty; -import static org.apache.commons.lang3.StringUtils.isNotEmpty; -import static org.folio.dew.utils.BulkEditProcessorHelper.dateFromString; -import static org.folio.dew.utils.Constants.ARRAY_DELIMITER; -import static org.folio.dew.utils.Constants.ITEM_DELIMITER_PATTERN; -import static org.folio.dew.utils.Constants.KEY_VALUE_DELIMITER; -import static org.folio.dew.utils.Constants.LINE_BREAK; -import static org.folio.dew.utils.Constants.LINE_BREAK_REPLACEMENT; - -import java.net.URI; -import java.util.Arrays; -import java.util.Collections; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Objects; -import java.util.Set; -import java.util.UUID; -import java.util.stream.Collectors; - -import lombok.extern.log4j.Log4j2; -import org.apache.commons.lang3.StringUtils; -import org.apache.commons.lang3.tuple.Pair; -import org.folio.dew.domain.dto.Address; -import org.folio.dew.domain.dto.CirculationNote; -import org.folio.dew.domain.dto.ContributorName; -import org.folio.dew.domain.dto.CustomField; - -import org.folio.dew.domain.dto.InventoryItemStatus; -import org.folio.dew.domain.dto.Item; -import org.folio.dew.domain.dto.ItemFormat; -import org.folio.dew.domain.dto.ItemLocation; -import org.folio.dew.domain.dto.ItemNote; -import org.folio.dew.domain.dto.LastCheckIn; -import org.folio.dew.domain.dto.LoanType; -import org.folio.dew.domain.dto.MaterialType; -import org.folio.dew.domain.dto.Personal; -import org.folio.dew.domain.dto.Source; -import org.folio.dew.domain.dto.Tags; -import org.folio.dew.domain.dto.User; -import org.folio.dew.domain.dto.UserFormat; -import org.folio.dew.error.BulkEditException; -import org.springframework.stereotype.Component; - -import lombok.RequiredArgsConstructor; - -@Component -@RequiredArgsConstructor -@Log4j2 -public class BulkEditParseService { - - private final UserReferenceService userReferenceService; - private final ItemReferenceService itemReferenceService; - - private final ElectronicAccessService electronicAccessService; - private final SpecialCharacterEscaper escaper; - - private static final int ADDRESS_ID = 0; - private static final int ADDRESS_COUNTRY_ID = 1; - private static final int ADDRESS_LINE_1 = 2; - private static final int ADDRESS_LINE_2 = 3; - private static final int ADDRESS_CITY = 4; - private static final int ADDRESS_REGION = 5; - private static final int ADDRESS_POSTAL_CODE = 6; - private static final int ADDRESS_PRIMARY_ADDRESS = 7; - private static final int ADDRESS_TYPE = 8; - private static final int NUMBER_OF_ITEM_NOTE_COMPONENTS = 3; - private static final int NOTE_TYPE_NAME_INDEX = 0; - private static final int NOTE_INDEX = 1; - private static final int STAFF_ONLY_OFFSET = 1; - - private static final int NUMBER_OF_CIRCULATION_NOTE_COMPONENTS = 8; - private static final int CIRC_NOTE_ID_INDEX = 0; - private static final int CIRC_NOTE_TYPE_INDEX = 1; - private static final int CIRC_NOTE_NOTE_INDEX = 2; - private static final int CIRC_NOTE_STAFF_ONLY_OFFSET = 5; - private static final int CIRC_NOTE_SOURCE_ID_OFFSET = 4; - private static final int CIRC_NOTE_LAST_NAME_OFFSET = 3; - private static final int CIRC_NOTE_FIRST_NAME_OFFSET = 2; - private static final int CIRC_NOTE_DATE_OFFSET = 1; - - private static final int NUMBER_OF_STATUS_COMPONENTS = 2; - private static final int STATUS_NAME_INDEX = 0; - private static final int STATUS_DATE_INDEX = 1; - private static final int NUMBER_OF_LAST_CHECK_IN_COMPONENTS = 3; - private static final int LAST_CHECK_IN_SERVICE_POINT_NAME_INDEX = 0; - private static final int LAST_CHECK_IN_USERNAME_INDEX = 1; - private static final int LAST_CHECK_IN_DATE_TIME_INDEX = 2; - - private static final String START_ARRAY = "["; - private static final String END_ARRAY = "]"; - - public User mapUserFormatToUser(UserFormat userFormat) { - User user = new User(); - populateUserFields(user, userFormat); - return user; - } - - private void populateUserFields(User user, UserFormat userFormat) { - user.setId(userFormat.getId()); - user.setUsername(isEmpty(userFormat.getUsername()) ? null : userFormat.getUsername()); - user.setExternalSystemId(isBlank(userFormat.getExternalSystemId()) ? null : userFormat.getExternalSystemId()); - user.setBarcode(isBlank(userFormat.getBarcode()) ? null : userFormat.getBarcode()); - user.setActive(getIsActive(userFormat)); - user.setType(userFormat.getType()); - user.setPatronGroup(userReferenceService.getPatronGroupIdByName(userFormat.getPatronGroup())); - user.setDepartments(new HashSet<>(getUserDepartments(userFormat))); - user.setProxyFor(isEmpty(userFormat.getProxyFor()) ? Collections.emptyList() : Arrays.asList(userFormat.getProxyFor().split(ARRAY_DELIMITER))); - user.setPersonal(getUserPersonalInfo(userFormat)); - user.setEnrollmentDate(dateFromString(userFormat.getEnrollmentDate())); - user.setExpirationDate(dateFromString(userFormat.getExpirationDate())); - user.setTags(getTags(userFormat)); - user.setCustomFields(getCustomFields(userFormat)); - user.setPreferredEmailCommunication(getPreferredEmailCommunication(userFormat)); - } - - private boolean getIsActive(UserFormat userFormat) { - String value = userFormat.getActive(); - if (value.matches("true") || value.matches("false")) { - return Boolean.parseBoolean(value); - } - //TODO in MODBULKED-14 save error that filed has a wrong value instead of returning false - return false; - } - - private List getUserDepartments(UserFormat userFormat) { - String[] departmentNames = userFormat.getDepartments().split(ARRAY_DELIMITER); - if (departmentNames.length > 0) { - return Arrays.stream(departmentNames).parallel() - .filter(StringUtils::isNotEmpty) - .map(escaper::restore) - .map(userReferenceService::getDepartmentIdByName) - .map(UUID::fromString) - .collect(Collectors.toList()); - } - return Collections.emptyList(); - } - - private Personal getUserPersonalInfo(UserFormat userFormat) { - Personal personal = new Personal(); - personal.setLastName(userFormat.getLastName()); - personal.setFirstName(userFormat.getFirstName()); - personal.setMiddleName(userFormat.getMiddleName()); - personal.setPreferredFirstName(userFormat.getPreferredFirstName()); - personal.setEmail(userFormat.getEmail()); - personal.setPhone(userFormat.getPhone()); - personal.setMobilePhone(userFormat.getMobilePhone()); - personal.setDateOfBirth(dateFromString(userFormat.getDateOfBirth())); - personal.setAddresses(getUserAddresses(userFormat)); - personal.setPreferredContactTypeId(isEmpty(userFormat.getPreferredContactTypeId()) ? null : userFormat.getPreferredContactTypeId()); - personal.setProfilePictureLink(isEmpty(userFormat.getProfilePictureLink()) ? null : URI.create(userFormat.getProfilePictureLink())); - return personal; - } - - private List
getUserAddresses(UserFormat userFormat) { - String[] addresses = userFormat.getAddresses().split(ITEM_DELIMITER_PATTERN); - if (addresses.length > 0) { - return Arrays.stream(addresses) - .parallel() - .filter(StringUtils::isNotEmpty) - .map(this::getAddressFromString) - .collect(Collectors.toList()); - } - return Collections.emptyList(); - } - - private Address getAddressFromString(String stringAddress) { - Address address = new Address(); - List addressFields = escaper.restore(Arrays.asList(stringAddress.split(ARRAY_DELIMITER))); - address.setId(addressFields.get(ADDRESS_ID)); - address.setCountryId(addressFields.get(ADDRESS_COUNTRY_ID)); - address.setAddressLine1(addressFields.get(ADDRESS_LINE_1)); - address.setAddressLine2(addressFields.get(ADDRESS_LINE_2)); - address.setCity(addressFields.get(ADDRESS_CITY)); - address.setRegion(addressFields.get(ADDRESS_REGION)); - address.setPostalCode(addressFields.get(ADDRESS_POSTAL_CODE)); - address.setPrimaryAddress(Boolean.valueOf(addressFields.get(ADDRESS_PRIMARY_ADDRESS))); - address.setAddressTypeId(userReferenceService.getAddressTypeIdByDesc(addressFields.get(ADDRESS_TYPE))); - return address; - } - - private Tags getTags(UserFormat userFormat) { - if (isNotEmpty(userFormat.getTags())) { - Tags tags = new Tags(); - List tagList = escaper.restore(Arrays.asList(userFormat.getTags().split(ARRAY_DELIMITER))); - return tags.tagList(tagList); - } - return null; - } - - private Map getCustomFields(UserFormat userFormat) { - if (isNotEmpty(userFormat.getCustomFields())) { - return Arrays.stream(userFormat.getCustomFields().split(ITEM_DELIMITER_PATTERN)) - .map(this::restoreCustomFieldValue) - .collect(Collectors.toMap(Pair::getKey, Pair::getValue)); - } - return Collections.emptyMap(); - } - - private Set getPreferredEmailCommunication(UserFormat userFormat) { - if (isNotEmpty(userFormat.getPreferredEmailCommunication())) { - return Arrays.stream(userFormat.getPreferredEmailCommunication().split(ARRAY_DELIMITER)) - .map(User.PreferredEmailCommunicationEnum::fromValue) - .collect(Collectors.toSet()); - } - return Collections.emptySet(); - } - - private Pair restoreCustomFieldValue(String s) { - var valuePair = stringToPair(s); - var fieldName = valuePair.getKey(); - var fieldValue = valuePair.getValue(); - var customField = userReferenceService.getCustomFieldByName(fieldName); - switch (customField.getType()) { - case DATE_PICKER: - return Pair.of(customField.getRefId(), fieldValue); - case SINGLE_CHECKBOX: - return Pair.of(customField.getRefId(), Boolean.parseBoolean(fieldValue)); - case TEXTBOX_LONG: - case TEXTBOX_SHORT: - return Pair.of(customField.getRefId(), fieldValue.replace(LINE_BREAK_REPLACEMENT, LINE_BREAK)); - case SINGLE_SELECT_DROPDOWN: - case RADIO_BUTTON: - return Pair.of(customField.getRefId(), restoreValueId(customField, fieldValue)); - case MULTI_SELECT_DROPDOWN: - return Pair.of(customField.getRefId(), restoreValueIds(customField, fieldValue)); - default: - throw new BulkEditException("Invalid custom field: " + s); - } - } - - private Pair stringToPair(String value) { - var tokens = value.split(KEY_VALUE_DELIMITER, -1); - if (tokens.length == 2) { - return Pair.of(escaper.restore(tokens[0]), escaper.restore(tokens[1])); - } else { - var msg = "Invalid key/value pair: " + value; - log.error(msg); - throw new BulkEditException(msg); - } - } - - private List restoreValueIds(CustomField customField, String values) { - return isEmpty(values) ? - Collections.emptyList() : - Arrays.stream(values.split(ARRAY_DELIMITER)) - .map(token -> restoreValueId(customField, token)) - .collect(Collectors.toList()); - } - - private String restoreValueId(CustomField customField, String value) { - var optionalValue = customField.getSelectField().getOptions().getValues().stream() - .filter(selectFieldOption -> Objects.equals(value, selectFieldOption.getValue())) - .findFirst(); - if (optionalValue.isPresent()) { - return optionalValue.get().getId(); - } else { - var msg = "Invalid custom field value: " + value; - log.error(msg); - throw new BulkEditException(msg); - } - } - - public Item mapItemFormatToItem(ItemFormat itemFormat) { - return new Item() - .id(itemFormat.getId()) - .hrid(itemFormat.getHrid()) - .holdingsRecordId(itemFormat.getHoldingsRecordId()) - .formerIds(restoreListValue(itemFormat.getFormerIds())) - .discoverySuppress(isEmpty(itemFormat.getDiscoverySuppress()) ? null : Boolean.valueOf(itemFormat.getDiscoverySuppress())) - .title(itemFormat.getTitle()) - .barcode(restoreStringValue(itemFormat.getBarcode())) - .effectiveShelvingOrder(restoreStringValue(itemFormat.getEffectiveShelvingOrder())) - .accessionNumber(restoreStringValue(itemFormat.getAccessionNumber())) - .itemLevelCallNumber(restoreStringValue(itemFormat.getItemLevelCallNumber())) - .itemLevelCallNumberPrefix(restoreStringValue(itemFormat.getItemLevelCallNumberPrefix())) - .itemLevelCallNumberSuffix(restoreStringValue(itemFormat.getItemLevelCallNumberSuffix())) - .itemLevelCallNumberTypeId(restoreItemLevelCallNumberTypeId(itemFormat.getItemLevelCallNumberType())) - .volume(restoreStringValue(itemFormat.getVolume())) - .enumeration(restoreStringValue(itemFormat.getEnumeration())) - .chronology(restoreStringValue(itemFormat.getChronology())) - .yearCaption(restoreListValue(itemFormat.getYearCaption())) - .itemIdentifier(restoreStringValue(itemFormat.getItemIdentifier())) - .copyNumber(restoreStringValue(itemFormat.getCopyNumber())) - .numberOfPieces(restoreStringValue(itemFormat.getNumberOfPieces())) - .descriptionOfPieces(restoreStringValue(itemFormat.getDescriptionOfPieces())) - .numberOfMissingPieces(restoreStringValue(itemFormat.getNumberOfMissingPieces())) - .missingPieces(restoreStringValue(itemFormat.getMissingPieces())) - .missingPiecesDate(restoreStringValue(itemFormat.getMissingPiecesDate())) - .itemDamagedStatusId(restoreItemDamagedStatusId(itemFormat.getItemDamagedStatus())) - .itemDamagedStatusDate(restoreStringValue(itemFormat.getItemDamagedStatusDate())) - .administrativeNotes(restoreListValue(itemFormat.getAdministrativeNotes())) - .notes(restoreItemNotes(itemFormat.getNotes())) - .circulationNotes(restoreCirculationNotes(itemFormat.getCheckInNotes() + ITEM_DELIMITER_PATTERN + itemFormat.getCheckOutNotes())) - .status(restoreStatus(itemFormat.getStatus())) - .materialType(restoreMaterialType(itemFormat.getMaterialType())) - .permanentLoanType(restoreLoanType(itemFormat.getPermanentLoanType())) - .temporaryLoanType(restoreLoanType(itemFormat.getTemporaryLoanType())) - .permanentLocation(restoreLocation(itemFormat.getPermanentLocation())) - .temporaryLocation(restoreLocation(itemFormat.getTemporaryLocation())) - .effectiveLocation(restoreLocation(itemFormat.getEffectiveLocation())) - .electronicAccess(electronicAccessService.restoreElectronicAccess(itemFormat.getElectronicAccess())) - .statisticalCodeIds(restoreStatisticalCodeIds(itemFormat.getStatisticalCodes())) - .tags(isEmpty(itemFormat.getTags()) ? new Tags().tagList(Collections.emptyList()) : new Tags().tagList(restoreListValue(itemFormat.getTags()))); - } - - private String restoreStringValue(String s) { - return isEmpty(s) || "null".equalsIgnoreCase(s) ? null : s; - } - - private List restoreListValue(String s) { - return isEmpty(s) ? - Collections.emptyList() : - escaper.restore(Arrays.asList(s.split(ARRAY_DELIMITER))); - } - - private List restoreContributorNames(String s) { - return isEmpty(s) ? - Collections.emptyList() : - Arrays.stream(s.split(ARRAY_DELIMITER)) - .map(escaper::restore) - .map(token -> new ContributorName().name(token)) - .collect(Collectors.toList()); - } - - private String restoreItemLevelCallNumberTypeId(String name) { - return itemReferenceService.getCallNumberTypeIdByName(name); - } - - private String restoreItemDamagedStatusId(String name) { - return itemReferenceService.getDamagedStatusIdByName(name); - } - - private List restoreItemNotes(String s) { - return isEmpty(s) ? Collections.emptyList() : - Arrays.stream(s.split(ITEM_DELIMITER_PATTERN)) - .map(this::restoreItemNote) - .filter(Objects::nonNull) - .collect(Collectors.toList()); - } - - private ItemNote restoreItemNote(String s) { - if (isNotEmpty(s)) { - var tokens = s.split(ARRAY_DELIMITER, -1); - if (tokens.length < NUMBER_OF_ITEM_NOTE_COMPONENTS) { - throw new BulkEditException(String.format("Illegal number of item note elements: %d, expected: %d", tokens.length, NUMBER_OF_ITEM_NOTE_COMPONENTS)); - } - - return new ItemNote() - .itemNoteTypeId(itemReferenceService.getNoteTypeIdByName(escaper.restore(tokens[NOTE_TYPE_NAME_INDEX]))) - .note(Arrays.stream(tokens, NOTE_INDEX, tokens.length - STAFF_ONLY_OFFSET) - .map(escaper::restore) - .collect(Collectors.joining(";"))) - .staffOnly(Boolean.valueOf(tokens[tokens.length - STAFF_ONLY_OFFSET])); - - } - return null; - } - - private List restoreCirculationNotes(String s) { - return isEmpty(s) ? Collections.emptyList() : - Arrays.stream(s.split(ITEM_DELIMITER_PATTERN)) - .map(this::restoreCirculationNote) - .filter(Objects::nonNull) - .collect(Collectors.toList()); - } - - private CirculationNote restoreCirculationNote(String s) { - if (isNotEmpty(s)) { - var tokens = s.split(ARRAY_DELIMITER, -1); - if (tokens.length < NUMBER_OF_CIRCULATION_NOTE_COMPONENTS) { - throw new BulkEditException(String.format("Illegal number of circulation note elements: %d, expected: %d", tokens.length, NUMBER_OF_CIRCULATION_NOTE_COMPONENTS)); - } - return new CirculationNote() - .id(tokens[CIRC_NOTE_ID_INDEX]) - .noteType(CirculationNote.NoteTypeEnum.fromValue(tokens[CIRC_NOTE_TYPE_INDEX])) - .note(Arrays.stream(tokens, CIRC_NOTE_NOTE_INDEX, tokens.length - CIRC_NOTE_STAFF_ONLY_OFFSET) - .map(escaper::restore) - .collect(Collectors.joining(";"))) - .staffOnly(Boolean.valueOf(tokens[tokens.length - CIRC_NOTE_STAFF_ONLY_OFFSET])) - .source(new Source() - .id(tokens[tokens.length - CIRC_NOTE_SOURCE_ID_OFFSET]) - .personal(new Personal() - .lastName(escaper.restore(tokens[tokens.length - CIRC_NOTE_LAST_NAME_OFFSET])) - .firstName(escaper.restore(tokens[tokens.length - CIRC_NOTE_FIRST_NAME_OFFSET])))) - .date(dateFromString(tokens[tokens.length - CIRC_NOTE_DATE_OFFSET])); - } - return null; - } - - private InventoryItemStatus restoreStatus(String s) { - if (isNotEmpty(s)) { - var tokens = s.split(ARRAY_DELIMITER, -1); - if (NUMBER_OF_STATUS_COMPONENTS == tokens.length) { - return new InventoryItemStatus() - .name(InventoryItemStatus.NameEnum.fromValue(tokens[STATUS_NAME_INDEX])) - .date(dateFromString(tokens[STATUS_DATE_INDEX])); - } - throw new BulkEditException(String.format("Illegal number of item status elements: %d, expected: %d", tokens.length, NUMBER_OF_STATUS_COMPONENTS)); - } - return null; - } - - private MaterialType restoreMaterialType(String s) { - return isEmpty(s) ? null : itemReferenceService.getMaterialTypeByName(s); - } - - private LoanType restoreLoanType(String s) { - return isEmpty(s) ? null : itemReferenceService.getLoanTypeByName(s); - } - - private ItemLocation restoreLocation(String s) { - return isEmpty(s) ? null : itemReferenceService.getLocationByName(s); - } - - private String restoreServicePointId(String s) { - return itemReferenceService.getServicePointIdByName(s); - } - - private List restoreStatisticalCodeIds(String s) { - return isEmpty(s) ? Collections.emptyList() : - Arrays.stream(s.split(ARRAY_DELIMITER)) - .map(escaper::restore) - .map(itemReferenceService::getStatisticalCodeIdByCode) - .filter(Objects::nonNull) - .collect(Collectors.toList()); - } - - private LastCheckIn restoreLastCheckIn(String s) { - if (isNotEmpty(s)) { - var tokens = s.split(ARRAY_DELIMITER, -1); - if (NUMBER_OF_LAST_CHECK_IN_COMPONENTS == tokens.length) { - return new LastCheckIn() - .servicePointId(itemReferenceService.getServicePointIdByName(escaper.restore(tokens[LAST_CHECK_IN_SERVICE_POINT_NAME_INDEX]))) - .staffMemberId(itemReferenceService.getUserIdByUserName(escaper.restore(tokens[LAST_CHECK_IN_USERNAME_INDEX]))) - .dateTime(restoreStringValue(tokens[LAST_CHECK_IN_DATE_TIME_INDEX])); - } - throw new BulkEditException(String.format("Illegal number of last check in elements: %d, expected: %d", tokens.length, NUMBER_OF_LAST_CHECK_IN_COMPONENTS)); - } - return null; - } -} diff --git a/src/main/java/org/folio/dew/service/BulkEditProcessingErrorsService.java b/src/main/java/org/folio/dew/service/BulkEditProcessingErrorsService.java index 3e64edccc..5a7b18854 100644 --- a/src/main/java/org/folio/dew/service/BulkEditProcessingErrorsService.java +++ b/src/main/java/org/folio/dew/service/BulkEditProcessingErrorsService.java @@ -39,6 +39,7 @@ public class BulkEditProcessingErrorsService { public static final DateTimeFormatter CSV_NAME_DATE_FORMAT = DateTimeFormatter.ISO_LOCAL_DATE; public static final String STORAGE = "storage"; + private static final String FAILED_TO_SAVE_ERROR_FILE_PLACEHOLDER = "Failed to save {} error file with job id {} cause {}"; private static final String STORAGE_TEMPLATE = "E" + File.separator + STORAGE + File.separator + "%s"; private static final String CSV_FILE_TEMPLATE = STORAGE_TEMPLATE + File.separator + "%s"; private static final String CONTENT_TYPE = "text/csv"; @@ -49,9 +50,7 @@ public class BulkEditProcessingErrorsService { private final LocalFilesStorage localFilesStorage; - - - public void saveErrorInCSV(String jobId, String affectedIdentifier, Throwable reasonForError, String fileName) { + public synchronized void saveErrorInCSV(String jobId, String affectedIdentifier, Throwable reasonForError, String fileName) { if (isNull(jobId) || isNull(affectedIdentifier) || isNull(reasonForError) || isNull(fileName)) { log.error("Some of the parameters is null, jobId: {}, affectedIdentifier: {}, reasonForError: {}, fileName: {}", jobId, affectedIdentifier, reasonForError, fileName); return; @@ -64,18 +63,23 @@ public void saveErrorInCSV(String jobId, String affectedIdentifier, Throwable re try { localFilesStorage.append(pathToCSVFile, errorLine.getBytes(StandardCharsets.UTF_8)); } catch (IOException ioException) { - log.error("Failed to save {} error file with job id {} cause {}", csvFileName, jobId, ioException); + log.error(FAILED_TO_SAVE_ERROR_FILE_PLACEHOLDER, csvFileName, jobId, ioException); } } } - public void saveErrorInCSV(String jobId, String errorString, String fileName) { + public synchronized void saveErrorInCSV(String jobId, String affectedIdentifier, String errorMessage, String fileName) { + if (isNull(jobId) || isNull(affectedIdentifier) || isNull(errorMessage) || isNull(fileName)) { + log.error("Some of the parameters is null, jobId: {}, affectedIdentifier: {}, reasonForError: {}, fileName: {}", jobId, affectedIdentifier, errorMessage, fileName); + return; + } var csvFileName = getCsvFileName(jobId, fileName); - var pathToCSVFile = getPathToCsvFile(jobId, getCsvFileName(jobId, fileName)); + var errorLine = affectedIdentifier + COMMA_SEPARATOR + errorMessage + System.lineSeparator(); + var pathToCSVFile = getPathToCsvFile(jobId, csvFileName); try { - localFilesStorage.append(pathToCSVFile, errorString.getBytes(StandardCharsets.UTF_8)); + localFilesStorage.append(pathToCSVFile, errorLine.getBytes(StandardCharsets.UTF_8)); } catch (IOException ioException) { - log.error("Failed to save {} error file with job id {} cause {}", csvFileName, jobId, ioException); + log.error(FAILED_TO_SAVE_ERROR_FILE_PLACEHOLDER, csvFileName, jobId, ioException); } } diff --git a/src/main/java/org/folio/dew/service/BulkEditRollBackJobLauncher.java b/src/main/java/org/folio/dew/service/BulkEditRollBackJobLauncher.java deleted file mode 100644 index 0b7f76a32..000000000 --- a/src/main/java/org/folio/dew/service/BulkEditRollBackJobLauncher.java +++ /dev/null @@ -1,30 +0,0 @@ -package org.folio.dew.service; - -import org.springframework.batch.core.Job; -import org.springframework.batch.core.JobExecution; -import org.springframework.batch.core.JobParameters; -import org.springframework.batch.core.JobParametersInvalidException; -import org.springframework.batch.core.launch.support.SimpleJobLauncher; -import org.springframework.batch.core.repository.JobExecutionAlreadyRunningException; -import org.springframework.batch.core.repository.JobInstanceAlreadyCompleteException; -import org.springframework.batch.core.repository.JobRepository; -import org.springframework.batch.core.repository.JobRestartException; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; -import org.springframework.transaction.annotation.Propagation; -import org.springframework.transaction.annotation.Transactional; - -@Component -public class BulkEditRollBackJobLauncher extends SimpleJobLauncher { - - @Autowired - public BulkEditRollBackJobLauncher(JobRepository jobRepository) { - this.setJobRepository(jobRepository); - } - - @Override - @Transactional(propagation = Propagation.NOT_SUPPORTED) - public JobExecution run(Job job, JobParameters jobParameters) throws JobParametersInvalidException, JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException { - return super.run(job, jobParameters); - } -} diff --git a/src/main/java/org/folio/dew/service/BulkEditRollBackService.java b/src/main/java/org/folio/dew/service/BulkEditRollBackService.java deleted file mode 100644 index c3c557fcc..000000000 --- a/src/main/java/org/folio/dew/service/BulkEditRollBackService.java +++ /dev/null @@ -1,142 +0,0 @@ -package org.folio.dew.service; - -import jakarta.annotation.PostConstruct; -import lombok.RequiredArgsConstructor; -import lombok.SneakyThrows; -import lombok.extern.log4j.Log4j2; -import org.apache.commons.lang3.StringUtils; -import org.folio.dew.client.DataExportSpringClient; -import org.folio.dew.error.BulkEditException; -import org.folio.dew.repository.RemoteFilesStorage; -import org.folio.dew.utils.Constants; -import org.springframework.batch.core.ExitStatus; -import org.springframework.batch.core.Job; -import org.springframework.batch.core.JobParameters; -import org.springframework.batch.core.JobParametersBuilder; -import org.springframework.batch.core.launch.JobOperator; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.beans.factory.annotation.Qualifier; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.stereotype.Service; - -import java.util.HashSet; -import java.util.Map; -import java.util.Set; -import java.util.UUID; -import java.util.concurrent.ConcurrentHashMap; - -import static org.folio.dew.utils.Constants.JOB_ID_SEPARATOR; -import static org.folio.dew.utils.Constants.PATH_SEPARATOR; -import static org.folio.dew.utils.Constants.TMP_DIR_PROPERTY; - -@Service -@RequiredArgsConstructor -@Log4j2 -public class BulkEditRollBackService { - - private static final String ROLLBACK_ERROR_MESSAGE = "Rollback error"; - private static final String ROLLBACK_DONE_MESSAGE = "Rollback has been done"; - - private final Map executionIdPerJobId = new ConcurrentHashMap<>(); - private final Map> usersIdsToRollBackForJobId = new ConcurrentHashMap<>(); - - private String workDir; - @Value("${spring.application.name}") - private String springApplicationName; - private final JobOperator jobOperator; - @Autowired - @Qualifier("bulkEditRollBackJob") - private Job job; - private final BulkEditRollBackJobLauncher rollBackJobLauncher; - private final DataExportSpringClient dataExportSpringClient; - private final RemoteFilesStorage remoteFilesStorage; - - @PostConstruct - public void postConstruct() { - workDir = System.getProperty(TMP_DIR_PROPERTY) + PATH_SEPARATOR + springApplicationName + PATH_SEPARATOR; - } - - public String stopAndRollBackJobExecutionByJobId(UUID jobId) { - try { - log.info("Rollback for jobId {} is started", jobId.toString()); - if (executionIdPerJobId.containsKey(jobId)) { - jobOperator.stop(executionIdPerJobId.get(jobId)); - } - rollBackByJobId(jobId); - return ROLLBACK_DONE_MESSAGE; - } catch (Exception e) { - log.error(e.getMessage()); - } - return ROLLBACK_ERROR_MESSAGE; - } - - public void putExecutionInfoPerJob(long executionId, UUID jobId) { - executionIdPerJobId.put(jobId, executionId); - } - - public void putUserIdForJob(String userId, UUID jobId) { - var existUsersIds = usersIdsToRollBackForJobId.computeIfAbsent(jobId, key -> new HashSet<>()); - existUsersIds.add(userId); - } - - public boolean isUserBeRollBack(String userId, UUID jobId) { - return !executionIdPerJobId.containsKey(jobId) || (usersIdsToRollBackForJobId.get(jobId) != null - && usersIdsToRollBackForJobId.get(jobId).remove(userId)); - } - - public boolean isExecutionIdExistForJob(UUID jobId) { - return executionIdPerJobId.containsKey(jobId); - } - public void cleanJobData(String exitCode, UUID jobId) { - if (!ExitStatus.STOPPED.getExitCode().equals(exitCode)) { - executionIdPerJobId.remove(jobId); - usersIdsToRollBackForJobId.remove(jobId); - } - } - - public void cleanJobData(UUID jobId) { - executionIdPerJobId.remove(jobId); - usersIdsToRollBackForJobId.remove(jobId); - } - - @SneakyThrows - public String getFileForRollBackFromMinIO(String fileUploadName) { - var jobId = getJobIdFromFileName(fileUploadName); - var files = dataExportSpringClient.getJobById(jobId).getFiles(); - if (files.isEmpty()) { - var error = "Rollback file does not exist for job " + jobId; - throw new BulkEditException(error); - } - return files.get(0); - } - - private String getJobIdFromFileName(String fileUploadName) { - return StringUtils.substringBefore(fileUploadName, JOB_ID_SEPARATOR); - } - - @SneakyThrows - private void rollBackByJobId(UUID jobId) { - var fileForRollBack = workDir + jobId.toString() + "_rollBack.csv"; - var files = dataExportSpringClient.getJobById(jobId.toString()).getFiles(); - if (files.isEmpty()) { - var error = "Rollback file does not exist for job " + jobId.toString(); - throw new BulkEditException(error); - } - var fileForRollBackMinIOPath = files.get(0); - var objectName = getObjectName(fileForRollBackMinIOPath); - remoteFilesStorage.downloadObject(objectName, fileForRollBack); - rollBackJobLauncher.run(job, getRollBackParameters(jobId.toString(), fileForRollBack)); - } - - private JobParameters getRollBackParameters(String jobId, String fileToRollBack) { - var jobParametersBuilder = new JobParametersBuilder(); - jobParametersBuilder.addString(Constants.JOB_ID, jobId); - jobParametersBuilder.addString(Constants.FILE_NAME, fileToRollBack); - return jobParametersBuilder.toJobParameters(); - } - - private String getObjectName(String path) { - String[] splitted = path.split("/"); - return StringUtils.substringBeforeLast(splitted[splitted.length - 1], ".csv") + ".csv"; - } -} diff --git a/src/main/java/org/folio/dew/service/BulkEditStatisticService.java b/src/main/java/org/folio/dew/service/BulkEditStatisticService.java index e97dcf0c2..1190e91d8 100644 --- a/src/main/java/org/folio/dew/service/BulkEditStatisticService.java +++ b/src/main/java/org/folio/dew/service/BulkEditStatisticService.java @@ -1,23 +1,29 @@ package org.folio.dew.service; -import org.springframework.batch.core.configuration.annotation.JobScope; import org.springframework.stereotype.Service; -@JobScope +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.LongAdder; + @Service public class BulkEditStatisticService { - private BulkEditStatistic statistic = new BulkEditStatistic(); + private final Map success; + + public BulkEditStatisticService() { + success = new ConcurrentHashMap<>(); + } - public void incrementSuccess() { - statistic.setSuccess(statistic.getSuccess() + 1); + public void incrementSuccess(String jobId, int value) { + success.computeIfAbsent(jobId, val -> new LongAdder()).add(value); } - public void incrementSuccess(int value) { - statistic.setSuccess(statistic.getSuccess() + value); + public int getSuccess(String jobId) { + return success.get(jobId).intValue(); } - public BulkEditStatistic getStatistic() { - return statistic; + public void reset(String jobId) { + success.remove(jobId); } } diff --git a/src/main/java/org/folio/dew/service/ConsortiaService.java b/src/main/java/org/folio/dew/service/ConsortiaService.java index 688bb8d07..9f806858a 100644 --- a/src/main/java/org/folio/dew/service/ConsortiaService.java +++ b/src/main/java/org/folio/dew/service/ConsortiaService.java @@ -4,15 +4,21 @@ import lombok.extern.log4j.Log4j2; import org.apache.commons.lang3.StringUtils; import org.folio.dew.client.ConsortiaClient; +import org.folio.dew.client.ConsortiumClient; +import org.folio.dew.domain.dto.UserTenant; import org.springframework.cache.annotation.Cacheable; import org.springframework.stereotype.Service; +import java.util.ArrayList; +import java.util.List; + @Service @RequiredArgsConstructor @Log4j2 public class ConsortiaService { private final ConsortiaClient consortiaClient; + private final ConsortiumClient consortiumClient; @Cacheable(value = "centralTenantCache") public String getCentralTenantId() { @@ -23,4 +29,15 @@ public String getCentralTenantId() { } return StringUtils.EMPTY; } + + @Cacheable(value = "affiliatedTenantsCache") + public List getAffiliatedTenants(String currentTenantId, String userId) { + var consortia = consortiumClient.getConsortia(); + var consortiaList = consortia.getConsortia(); + if (!consortiaList.isEmpty()) { + var userTenants = consortiumClient.getConsortiaUserTenants(consortiaList.get(0).getId(), userId, Integer.MAX_VALUE); + return userTenants.getUserTenants().stream().map(UserTenant::getTenantId).toList(); + } + return new ArrayList<>(); + } } diff --git a/src/main/java/org/folio/dew/service/ContentUpdateRecords.java b/src/main/java/org/folio/dew/service/ContentUpdateRecords.java deleted file mode 100644 index 47493629b..000000000 --- a/src/main/java/org/folio/dew/service/ContentUpdateRecords.java +++ /dev/null @@ -1,21 +0,0 @@ -package org.folio.dew.service; - -import lombok.Getter; - -import java.util.ArrayList; -import java.util.List; - -@Getter -public class ContentUpdateRecords { - - private List updated = new ArrayList<>(); - private List preview = new ArrayList<>(); - - public void addToUpdated(T entity) { - updated.add(entity); - } - - public void addToPreview(T entity) { - preview.add(entity); - } -} diff --git a/src/main/java/org/folio/dew/service/FolioExecutionContextManager.java b/src/main/java/org/folio/dew/service/FolioExecutionContextManager.java index d8d375a28..8689044f8 100644 --- a/src/main/java/org/folio/dew/service/FolioExecutionContextManager.java +++ b/src/main/java/org/folio/dew/service/FolioExecutionContextManager.java @@ -1,8 +1,10 @@ package org.folio.dew.service; +import org.apache.commons.lang3.SerializationUtils; import org.folio.spring.DefaultFolioExecutionContext; import org.folio.spring.FolioExecutionContext; +import java.util.Collection; import java.util.HashMap; import java.util.List; @@ -10,8 +12,8 @@ public class FolioExecutionContextManager { public static final String X_OKAPI_TENANT = "x-okapi-tenant"; public DefaultFolioExecutionContext refreshAndGetFolioExecutionContext(String tenantId, FolioExecutionContext folioExecutionContext) { - var headers = new HashMap<>(folioExecutionContext.getAllHeaders()); - headers.replace(X_OKAPI_TENANT, List.of(tenantId)); - return new DefaultFolioExecutionContext(folioExecutionContext.getFolioModuleMetadata(), headers); + var headersCopy = SerializationUtils.clone((HashMap>) folioExecutionContext.getAllHeaders()); + headersCopy.replace(X_OKAPI_TENANT, List.of(tenantId)); + return new DefaultFolioExecutionContext(folioExecutionContext.getFolioModuleMetadata(), headersCopy); } } diff --git a/src/main/java/org/folio/dew/service/FolioTenantService.java b/src/main/java/org/folio/dew/service/FolioTenantService.java index 38d3f7673..a6c53a8df 100644 --- a/src/main/java/org/folio/dew/service/FolioTenantService.java +++ b/src/main/java/org/folio/dew/service/FolioTenantService.java @@ -40,7 +40,7 @@ protected void afterTenantUpdate(TenantAttributes tenantAttributes) { try { kafkaService.createKafkaTopics(); kafkaService.restartEventListeners(); - configurationService.checkBulkEditConfiguration(); + configurationService.updateBulkEditConfiguration(); } catch (Exception e) { log.error(e.getMessage(), e); } diff --git a/src/main/java/org/folio/dew/service/HoldingsReferenceService.java b/src/main/java/org/folio/dew/service/HoldingsReferenceService.java index eb8be4373..2f0e39189 100644 --- a/src/main/java/org/folio/dew/service/HoldingsReferenceService.java +++ b/src/main/java/org/folio/dew/service/HoldingsReferenceService.java @@ -22,7 +22,6 @@ import org.folio.dew.client.StatisticalCodeClient; import org.folio.dew.domain.dto.ErrorServiceArgs; import org.folio.dew.domain.dto.HoldingsRecord; -import org.folio.dew.domain.dto.ItemLocation; import org.folio.dew.error.BulkEditException; import org.folio.dew.error.NotFoundException; import org.folio.spring.FolioExecutionContext; @@ -137,17 +136,6 @@ public String getLocationNameById(String id, String tenantId) { } } - @Cacheable(cacheNames = "holdingsLocations") - public ItemLocation getLocationByName(String name) { - var locations = locationClient.getLocationByQuery(String.format(QUERY_PATTERN_NAME, name)); - if (locations.getLocations().isEmpty()) { - var msg = "Location not found by name=" + name; - log.error(msg); - throw new BulkEditException(msg); - } - return locations.getLocations().get(0); - } - @Cacheable(cacheNames = "holdingsCallNumberTypesNames") public String getCallNumberTypeNameById(String id, ErrorServiceArgs args, String tenantId) { try (var context = new FolioExecutionContextSetter(refreshAndGetFolioExecutionContext(tenantId, folioExecutionContext))) { diff --git a/src/main/java/org/folio/dew/service/ItemReferenceService.java b/src/main/java/org/folio/dew/service/ItemReferenceService.java index de4866ea9..c930243bc 100644 --- a/src/main/java/org/folio/dew/service/ItemReferenceService.java +++ b/src/main/java/org/folio/dew/service/ItemReferenceService.java @@ -25,11 +25,8 @@ import org.folio.dew.client.StatisticalCodeClient; import org.folio.dew.client.UserClient; import org.folio.dew.domain.dto.ErrorServiceArgs; -import org.folio.dew.domain.dto.ItemLocation; import org.folio.dew.domain.dto.ItemLocationCollection; -import org.folio.dew.domain.dto.LoanType; import org.folio.dew.domain.dto.LoanTypeCollection; -import org.folio.dew.domain.dto.MaterialType; import org.folio.dew.domain.dto.MaterialTypeCollection; import org.folio.dew.error.BulkEditException; import org.folio.dew.error.ConfigurationException; @@ -48,16 +45,12 @@ @RequiredArgsConstructor public class ItemReferenceService extends FolioExecutionContextManager { private static final String QUERY_PATTERN_NAME = "name==\"%s\""; - private static final String QUERY_PATTERN_CODE = "code==\"%s\""; - private static final String QUERY_PATTERN_USERNAME = "username==\"%s\""; public static final String EFFECTIVE_LOCATION_ID = "effectiveLocationId"; private final CallNumberTypeClient callNumberTypeClient; private final DamagedStatusClient damagedStatusClient; private final ItemNoteTypeClient itemNoteTypeClient; - private final ServicePointClient servicePointClient; private final StatisticalCodeClient statisticalCodeClient; - private final UserClient userClient; private final LocationClient locationClient; private final MaterialTypeClient materialTypeClient; private final HoldingClient holdingClient; @@ -77,18 +70,6 @@ public String getCallNumberTypeNameById(String callNumberTypeId, ErrorServiceArg } } - @Cacheable(cacheNames = "callNumberTypeIds") - public String getCallNumberTypeIdByName(String name) { - if (isEmpty(name)) { - return null; - } - var response = callNumberTypeClient.getByQuery(String.format(QUERY_PATTERN_NAME, name)); - if (response.getCallNumberTypes().isEmpty()) { - return name; - } - return response.getCallNumberTypes().get(0).getId(); - } - @Cacheable(cacheNames = "damagedStatusNames") public String getDamagedStatusNameById(String damagedStatusId, ErrorServiceArgs args, String tenantId) { try (var context = new FolioExecutionContextSetter(refreshAndGetFolioExecutionContext(tenantId, folioExecutionContext))) { @@ -99,18 +80,6 @@ public String getDamagedStatusNameById(String damagedStatusId, ErrorServiceArgs } } - @Cacheable(cacheNames = "damagedStatusIds") - public String getDamagedStatusIdByName(String name) { - if (isEmpty(name)) { - return null; - } - var response = damagedStatusClient.getByQuery(String.format(QUERY_PATTERN_NAME, name)); - if (response.getItemDamageStatuses().isEmpty()) { - return name; - } - return response.getItemDamageStatuses().get(0).getId(); - } - @Cacheable(cacheNames = "noteTypeNames") public String getNoteTypeNameById(String noteTypeId, ErrorServiceArgs args, String tenantId) { try (var context = new FolioExecutionContextSetter(refreshAndGetFolioExecutionContext(tenantId, folioExecutionContext))) { @@ -121,30 +90,6 @@ public String getNoteTypeNameById(String noteTypeId, ErrorServiceArgs args, Stri } } - @Cacheable(cacheNames = "noteTypeIds") - public String getNoteTypeIdByName(String name) { - if (isEmpty(name)) { - return null; - } - var response = itemNoteTypeClient.getByQuery(String.format(QUERY_PATTERN_NAME, name)); - if (response.getItemNoteTypes().isEmpty()) { - return name; - } - return response.getItemNoteTypes().get(0).getId(); - } - - @Cacheable(cacheNames = "servicePointIds") - public String getServicePointIdByName(String name) { - if (isEmpty(name)) { - return null; - } - var response = servicePointClient.get(String.format(QUERY_PATTERN_NAME, name), 1L); - if (response.getServicepoints().isEmpty()) { - return name; - } - return response.getServicepoints().get(0).getId(); - } - @Cacheable(cacheNames = "statisticalCodeNames") public String getStatisticalCodeById(String statisticalCodeId, ErrorServiceArgs args, String tenantId) { try (var context = new FolioExecutionContextSetter(refreshAndGetFolioExecutionContext(tenantId, folioExecutionContext))) { @@ -155,69 +100,21 @@ public String getStatisticalCodeById(String statisticalCodeId, ErrorServiceArgs } } - @Cacheable(cacheNames = "statisticalCodeIds") - public String getStatisticalCodeIdByCode(String code) { - if (isEmpty(code)) { - return null; - } - var response = statisticalCodeClient.getByQuery(String.format(QUERY_PATTERN_CODE, code)); - if (response.getStatisticalCodes().isEmpty()) { - return code; - } - return response.getStatisticalCodes().get(0).getId(); - } - - @Cacheable(cacheNames = "userIds") - public String getUserIdByUserName(String name) { - if (isEmpty(name)) { - return null; - } - var response = userClient.getUserByQuery(String.format(QUERY_PATTERN_USERNAME, name)); - if (response.getUsers().isEmpty()) { - return name; - } - return response.getUsers().get(0).getId(); - } - @Cacheable(cacheNames = "locations") public ItemLocationCollection getItemLocationsByName(String name) { return locationClient.getLocationByQuery(String.format(QUERY_PATTERN_NAME, name)); } - public ItemLocation getLocationByName(String name) { - var locations = getItemLocationsByName(name); - if (locations.getLocations().isEmpty()) { - throw new BulkEditException("Location not found: " + name); - } - return locations.getLocations().get(0); - } - @Cacheable(cacheNames = "materialTypes") public MaterialTypeCollection getMaterialTypesByName(String name) { return materialTypeClient.getByQuery(String.format(QUERY_PATTERN_NAME, name)); } - public MaterialType getMaterialTypeByName(String name) { - var types = getMaterialTypesByName(name); - if (types.getMtypes().isEmpty()) { - throw new BulkEditException("Material type not found: " + name); - } - return types.getMtypes().get(0); - } - @Cacheable(cacheNames = "loanTypes") public LoanTypeCollection getLoanTypesByName(String name) { return loanTypeClient.getByQuery(String.format(QUERY_PATTERN_NAME, name)); } - public LoanType getLoanTypeByName(String name) { - var loanTypes = getLoanTypesByName(name); - if (loanTypes.getLoantypes().isEmpty()) { - throw new BulkEditException("Loan type not found: " + name); - } - return loanTypes.getLoantypes().get(0); - } - @Cacheable(cacheNames = "holdings") public String getHoldingEffectiveLocationCodeById(String id) { var holdingJson = holdingClient.getHoldingById(id); diff --git a/src/main/java/org/folio/dew/service/JobCommandsReceiverService.java b/src/main/java/org/folio/dew/service/JobCommandsReceiverService.java index 3d65c218e..fbc8fe0c1 100644 --- a/src/main/java/org/folio/dew/service/JobCommandsReceiverService.java +++ b/src/main/java/org/folio/dew/service/JobCommandsReceiverService.java @@ -3,7 +3,6 @@ import static java.util.Objects.nonNull; import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_IDENTIFIERS; import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_QUERY; -import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_UPDATE; import static org.folio.dew.domain.dto.ExportType.EDIFACT_ORDERS_EXPORT; import static org.folio.dew.utils.Constants.BULKEDIT_DIR_NAME; import static org.folio.dew.utils.Constants.CSV_EXTENSION; @@ -107,9 +106,9 @@ public void receiveStartJobCommand(@Payload JobCommand jobCommand, @Headers Map< prepareJobParameters(jobCommand); - if (Set.of(BULK_EDIT_IDENTIFIERS, BULK_EDIT_QUERY, BULK_EDIT_UPDATE).contains(jobCommand.getExportType())) { + if (Set.of(BULK_EDIT_IDENTIFIERS, BULK_EDIT_QUERY).contains(jobCommand.getExportType())) { addBulkEditJobCommand(jobCommand); - if (BULK_EDIT_IDENTIFIERS.equals(jobCommand.getExportType()) || BULK_EDIT_UPDATE.equals(jobCommand.getExportType())) { + if (BULK_EDIT_IDENTIFIERS.equals(jobCommand.getExportType())) { return; } } @@ -137,35 +136,6 @@ private String resolveJobKey(JobCommand jobCommand) { private void prepareJobParameters(JobCommand jobCommand) { var paramsBuilder = new JobParametersBuilder(jobCommand.getJobParameters()); - // TODO enrich exportType.json with value MARC_EXPORT - if ("MARC_EXPORT".equals(jobCommand.getExportType().getValue())) { - var uploadedFilePath = jobCommand.getJobParameters().getString(FILE_NAME); - if (nonNull(uploadedFilePath) && FilenameUtils.isExtension(uploadedFilePath, "cql")) { - var tempIdentifiersFileName = workDir + FilenameUtils.getBaseName(uploadedFilePath) + CSV_EXTENSION; - try (var lines = localFilesStorage.lines(uploadedFilePath); - var outputStream = new FileOutputStream(tempIdentifiersFileName)) { - var query = lines.collect(Collectors.joining()); - // TODO enrich entityType.json with values INSTANCE, HOLDINGS - InputStreamResource resource = null; - if ("INSTANCE".equals(jobCommand.getEntityType().getValue())) { - resource = searchClient.getInstanceIds(query).getBody(); - } else if ("HOLDINGS".equals(jobCommand.getEntityType().getValue())) { - resource = searchClient.getHoldingIds(query).getBody(); - } - if (nonNull(resource)) { - resource.getInputStream().transferTo(outputStream); - } - var identifiersUrl = remoteFilesStorage.objectToPresignedObjectUrl( - remoteFilesStorage.uploadObject(FilenameUtils.getName(tempIdentifiersFileName), tempIdentifiersFileName, null, "text/csv", true)); - paramsBuilder.addString(FILE_NAME, identifiersUrl, JOB_PARAMETER_DEFAULT_IDENTIFYING_VALUE); - } catch (Exception e) { - var msg = String.format("Failed to read %s, reason: %s", FilenameUtils.getBaseName(uploadedFilePath), e.getMessage()); - log.error(msg); - throw new FileOperationException(msg); - } - } - } - var jobId = jobCommand.getId().toString(); var outputFileName = fileNameResolver.resolve(jobCommand, workDir, jobId); diff --git a/src/main/java/org/folio/dew/service/UserPermissionsService.java b/src/main/java/org/folio/dew/service/UserPermissionsService.java new file mode 100644 index 000000000..c103e0a4e --- /dev/null +++ b/src/main/java/org/folio/dew/service/UserPermissionsService.java @@ -0,0 +1,52 @@ +package org.folio.dew.service; + +import lombok.RequiredArgsConstructor; +import lombok.Setter; +import lombok.extern.log4j.Log4j2; +import org.apache.commons.lang3.StringUtils; +import org.folio.dew.client.EurekaUserPermissionsClient; +import org.folio.dew.client.OkapiUserPermissionsClient; +import org.folio.spring.FolioExecutionContext; +import org.springframework.beans.factory.annotation.Value; +import org.springframework.stereotype.Service; + +import java.util.List; + +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.BULK_EDIT_INVENTORY_VIEW_PERMISSION; +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.BULK_EDIT_USERS_VIEW_PERMISSION; +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.INVENTORY_INSTANCES_ITEM_GET_PERMISSION; +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.INVENTORY_ITEMS_ITEM_GET_PERMISSION; +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.INVENTORY_STORAGE_HOLDINGS_ITEM_GET_PERMISSION; +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.USER_ITEM_GET_PERMISSION; + + +@RequiredArgsConstructor +@Log4j2 +@Service +public class UserPermissionsService { + + public static final String EUREKA_PLATFORM = "eureka"; + public static final String OKAPI_PLATFORM = "okapi"; + + @Setter + @Value("${application.platform}") + private String platform; + + private final FolioExecutionContext folioExecutionContext; + private final OkapiUserPermissionsClient okapiUserPermissionsClient; + private final EurekaUserPermissionsClient eurekaUserPermissionsClient; + + public List getPermissions() { + if (StringUtils.equals(EUREKA_PLATFORM, platform)) { + var desiredPermissions = getDesiredPermissions(); + return eurekaUserPermissionsClient.getPermissions(folioExecutionContext.getUserId().toString(), + desiredPermissions).getPermissions(); + } + return okapiUserPermissionsClient.getPermissions(folioExecutionContext.getUserId().toString()).getPermissionNames(); + } + + private List getDesiredPermissions() { + return List.of(BULK_EDIT_INVENTORY_VIEW_PERMISSION.getValue(), BULK_EDIT_USERS_VIEW_PERMISSION.getValue(), USER_ITEM_GET_PERMISSION.getValue(), + INVENTORY_ITEMS_ITEM_GET_PERMISSION.getValue(), INVENTORY_STORAGE_HOLDINGS_ITEM_GET_PERMISSION.getValue(), INVENTORY_INSTANCES_ITEM_GET_PERMISSION.getValue()); + } +} diff --git a/src/main/java/org/folio/dew/service/UserReferenceService.java b/src/main/java/org/folio/dew/service/UserReferenceService.java index 56f0217c1..d850fd8dd 100644 --- a/src/main/java/org/folio/dew/service/UserReferenceService.java +++ b/src/main/java/org/folio/dew/service/UserReferenceService.java @@ -94,20 +94,6 @@ public String getPatronGroupNameById(String id, ErrorServiceArgs args) { } } - @Cacheable(cacheNames = "patronGroupIds") - public String getPatronGroupIdByName(String name) { - if (isEmpty(name)) { - throw new BulkEditException("Patron group can not be empty"); - } - var response = groupClient.getGroupByQuery(String.format("group==\"%s\"", name)); - if (response.getUsergroups().isEmpty()) { - var msg = "Invalid patron group value: " + name; - log.error(msg); - throw new BulkEditException(msg); - } - return response.getUsergroups().get(0).getId(); - } - @Cacheable(cacheNames = "customFields") public CustomField getCustomFieldByRefId(String refId) { return customFieldsClient.getCustomFieldsByQuery(getModuleId(MOD_USERS),String.format("refId==\"%s\"", refId)) @@ -116,14 +102,6 @@ public CustomField getCustomFieldByRefId(String refId) { .orElseThrow(() -> new BulkEditException(format("Custom field with refId=%s not found", refId))); } - @Cacheable(cacheNames = "customFields") - public CustomField getCustomFieldByName(String name) { - return customFieldsClient.getCustomFieldsByQuery(getModuleId(MOD_USERS), String.format("name==\"%s\"", name)) - .getCustomFields().stream().filter(customField -> customField.getName().equals(name)) - .findFirst() - .orElseThrow(() -> new BulkEditException(format("Custom field with name=%s not found", name))); - } - @Cacheable(cacheNames = "moduleIds") public String getModuleId(String moduleName) { var tenantId = folioExecutionContext.getTenantId(); diff --git a/src/main/java/org/folio/dew/service/mapper/HoldingsMapper.java b/src/main/java/org/folio/dew/service/mapper/HoldingsMapper.java index 2fa5fe0f3..e008a2463 100644 --- a/src/main/java/org/folio/dew/service/mapper/HoldingsMapper.java +++ b/src/main/java/org/folio/dew/service/mapper/HoldingsMapper.java @@ -78,7 +78,9 @@ private String notesToString(List notes, ErrorServiceArgs args, St .map(note -> String.join(ARRAY_DELIMITER, escaper.escape(holdingsReferenceService.getNoteTypeNameById(note.getHoldingsNoteTypeId(), args, tenantId)), escaper.escape(note.getNote()), - booleanToStringNullSafe(note.getStaffOnly()))) + booleanToStringNullSafe(note.getStaffOnly()), + tenantId, + note.getHoldingsNoteTypeId())) .collect(Collectors.joining(ITEM_DELIMITER)); } diff --git a/src/main/java/org/folio/dew/service/mapper/InstanceMapper.java b/src/main/java/org/folio/dew/service/mapper/InstanceMapper.java index 6254ec171..8863bd4fa 100644 --- a/src/main/java/org/folio/dew/service/mapper/InstanceMapper.java +++ b/src/main/java/org/folio/dew/service/mapper/InstanceMapper.java @@ -68,7 +68,7 @@ private String fetchInstanceFormats(List instanceFormats, ErrorServiceAr .collect(Collectors.joining(ITEM_DELIMITER_SPACED)); } - private String fetchNatureOfContentTerms(Set natureOfContentTermIds, ErrorServiceArgs errorServiceArgs) { + private String fetchNatureOfContentTerms(List natureOfContentTermIds, ErrorServiceArgs errorServiceArgs) { return isEmpty(natureOfContentTermIds) ? EMPTY : natureOfContentTermIds.stream() .map(natId -> instanceReferenceService.getNatureOfContentTermNameById(natId, errorServiceArgs)) diff --git a/src/main/java/org/folio/dew/service/mapper/MapperHelper.java b/src/main/java/org/folio/dew/service/mapper/MapperHelper.java deleted file mode 100644 index 89bbe5bfa..000000000 --- a/src/main/java/org/folio/dew/service/mapper/MapperHelper.java +++ /dev/null @@ -1,13 +0,0 @@ -package org.folio.dew.service.mapper; - -import static org.apache.commons.lang3.StringUtils.isEmpty; - -import lombok.experimental.UtilityClass; - -@UtilityClass -public class MapperHelper { - public static String restoreStringValue(String s) { - return isEmpty(s) || "null".equalsIgnoreCase(s) ? null : s; - } - -} diff --git a/src/main/java/org/folio/dew/service/update/BulkEditHoldingsContentUpdateService.java b/src/main/java/org/folio/dew/service/update/BulkEditHoldingsContentUpdateService.java deleted file mode 100644 index 5841a224a..000000000 --- a/src/main/java/org/folio/dew/service/update/BulkEditHoldingsContentUpdateService.java +++ /dev/null @@ -1,111 +0,0 @@ -package org.folio.dew.service.update; - -import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_UPDATE; -import static org.folio.dew.domain.dto.JobParameterNames.PREVIEW_FILE_NAME; -import static org.folio.dew.domain.dto.JobParameterNames.TEMP_OUTPUT_FILE_PATH; -import static org.folio.dew.domain.dto.JobParameterNames.UPDATED_FILE_NAME; -import static org.folio.dew.utils.Constants.COMMA; -import static org.folio.dew.utils.Constants.CSV_EXTENSION; -import static org.folio.dew.utils.Constants.FILE_NAME; -import static org.folio.dew.utils.Constants.IDENTIFIER_TYPE; -import static org.folio.dew.utils.Constants.NO_CHANGE_MESSAGE; -import static org.folio.dew.utils.Constants.PATH_SEPARATOR; -import static org.folio.dew.utils.Constants.PREVIEW_PREFIX; -import static org.folio.dew.utils.Constants.UPDATED_PREFIX; - -import lombok.RequiredArgsConstructor; -import lombok.extern.log4j.Log4j2; -import org.apache.commons.io.FilenameUtils; -import org.folio.de.entity.JobCommand; -import org.folio.dew.domain.dto.HoldingsContentUpdate; -import org.folio.dew.domain.dto.HoldingsContentUpdateCollection; -import org.folio.dew.domain.dto.HoldingsFormat; -import org.folio.dew.error.BulkEditException; -import org.folio.dew.error.FileOperationException; -import org.folio.dew.repository.RemoteFilesStorage; -import org.folio.dew.service.BulkEditProcessingErrorsService; -import org.folio.dew.service.ContentUpdateRecords; -import org.folio.dew.service.UpdatesResult; -import org.folio.dew.service.validation.HoldingsContentUpdateValidatorService; -import org.folio.dew.utils.CsvHelper; -import org.springframework.batch.core.JobParametersBuilder; -import org.springframework.stereotype.Component; - -import java.util.List; -import java.util.Objects; - -@Component -@RequiredArgsConstructor -@Log4j2 -public class BulkEditHoldingsContentUpdateService { - private final RemoteFilesStorage remoteFilesStorage; - private final BulkEditProcessingErrorsService errorsService; - private final HoldingsLocationUpdateStrategy locationUpdateStrategy; - private final HoldingsContentUpdateValidatorService validatorService; - - public UpdatesResult process(JobCommand jobCommand, HoldingsContentUpdateCollection contentUpdates) { - validatorService.validateContentUpdateCollection(contentUpdates); - try { - var fileName = FilenameUtils.getName(jobCommand.getJobParameters().getString(TEMP_OUTPUT_FILE_PATH)) + CSV_EXTENSION; - var updatedFileName = jobCommand.getId() + PATH_SEPARATOR + UPDATED_PREFIX + fileName; - var previewFileName = jobCommand.getId() + PATH_SEPARATOR + PREVIEW_PREFIX + fileName; - var holdingsFormats = CsvHelper.readRecordsFromStorage(remoteFilesStorage, jobCommand.getId() + PATH_SEPARATOR + fileName, HoldingsFormat.class, true); - var updatedHoldings = applyContentUpdates(holdingsFormats, contentUpdates, jobCommand); - CsvHelper.saveRecordsToStorage(remoteFilesStorage, updatedHoldings.getUpdated(), HoldingsFormat.class, updatedFileName); - CsvHelper.saveRecordsToStorage(remoteFilesStorage, updatedHoldings.getPreview(), HoldingsFormat.class, previewFileName); - jobCommand.setJobParameters(new JobParametersBuilder(jobCommand.getJobParameters()) - .addString(UPDATED_FILE_NAME, updatedFileName) - .addString(PREVIEW_FILE_NAME, previewFileName) - .toJobParameters()); - jobCommand.setExportType(BULK_EDIT_UPDATE); - return new UpdatesResult().withTotal(holdingsFormats.size()).withEntitiesForPreview(updatedHoldings.getPreview()); - } catch (Exception e) { - var msg = String.format("I/O exception for job id %s, reason: %s", jobCommand.getId(), e.getMessage()); - log.error(msg); - throw new FileOperationException(msg); - } - } - - private ContentUpdateRecords applyContentUpdates(List holdingsFormats, HoldingsContentUpdateCollection contentUpdateCollection, JobCommand jobCommand) { - var updateResult = new ContentUpdateRecords(); - var errorStringBuilder = new StringBuilder(); - holdingsFormats.forEach(holdingsFormat -> { - var updatedHoldingsRecord = holdingsFormat; - if ("MARC".equals(holdingsFormat.getSource())) { - errorStringBuilder - .append(holdingsFormat.getIdentifier(jobCommand.getJobParameters().getString(IDENTIFIER_TYPE))) - .append(COMMA) - .append("Holdings records that have source \"MARC\" cannot be changed") - .append(System.lineSeparator()); - } else { - for (HoldingsContentUpdate contentUpdate: contentUpdateCollection.getHoldingsContentUpdates()) { - updatedHoldingsRecord = resolveUpdateStrategy(contentUpdate).applyUpdate(updatedHoldingsRecord, contentUpdate); - } - if (!Objects.equals(updatedHoldingsRecord, holdingsFormat)) { - updateResult.addToUpdated(updatedHoldingsRecord); - } else { - errorStringBuilder - .append(holdingsFormat.getIdentifier(jobCommand.getJobParameters().getString(IDENTIFIER_TYPE))) - .append(COMMA) - .append(NO_CHANGE_MESSAGE) - .append(System.lineSeparator()); - } - } - updateResult.addToPreview(updatedHoldingsRecord); - }); - if (!errorStringBuilder.toString().isEmpty()) { - errorsService.saveErrorInCSV(jobCommand.getId().toString(), errorStringBuilder.toString(), FilenameUtils.getName(jobCommand.getJobParameters().getString(FILE_NAME))); - } - return updateResult; - } - - private UpdateStrategy resolveUpdateStrategy(HoldingsContentUpdate update) { - switch (update.getOption()) { - case PERMANENT_LOCATION: - case TEMPORARY_LOCATION: - return locationUpdateStrategy; - default: - throw new BulkEditException(String.format("Content updates for %s not implemented", update.getOption())); - } - } -} diff --git a/src/main/java/org/folio/dew/service/update/BulkEditUserContentUpdateService.java b/src/main/java/org/folio/dew/service/update/BulkEditUserContentUpdateService.java deleted file mode 100644 index a955a05c5..000000000 --- a/src/main/java/org/folio/dew/service/update/BulkEditUserContentUpdateService.java +++ /dev/null @@ -1,108 +0,0 @@ -package org.folio.dew.service.update; - -import static java.lang.String.format; -import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_UPDATE; -import static org.folio.dew.domain.dto.JobParameterNames.PREVIEW_FILE_NAME; -import static org.folio.dew.domain.dto.JobParameterNames.TEMP_OUTPUT_FILE_PATH; -import static org.folio.dew.domain.dto.JobParameterNames.UPDATED_FILE_NAME; -import static org.folio.dew.utils.Constants.CSV_EXTENSION; -import static org.folio.dew.utils.Constants.FILE_NAME; -import static org.folio.dew.utils.Constants.IDENTIFIER_TYPE; -import static org.folio.dew.utils.Constants.NO_CHANGE_MESSAGE; -import static org.folio.dew.utils.Constants.PATH_SEPARATOR; -import static org.folio.dew.utils.Constants.PREVIEW_PREFIX; -import static org.folio.dew.utils.Constants.UPDATED_PREFIX; - -import lombok.RequiredArgsConstructor; -import lombok.extern.log4j.Log4j2; -import org.apache.commons.io.FilenameUtils; -import org.folio.de.entity.JobCommand; -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.domain.dto.UserContentUpdateCollection; -import org.folio.dew.domain.dto.UserFormat; -import org.folio.dew.error.BulkEditException; -import org.folio.dew.error.FileOperationException; -import org.folio.dew.repository.RemoteFilesStorage; -import org.folio.dew.service.BulkEditProcessingErrorsService; -import org.folio.dew.service.ContentUpdateRecords; -import org.folio.dew.service.UpdatesResult; -import org.folio.dew.service.validation.UserContentUpdateValidatorService; -import org.folio.dew.utils.CsvHelper; -import org.springframework.batch.core.JobParametersBuilder; -import org.springframework.stereotype.Component; - -import java.util.List; -import java.util.Objects; - -@Component -@RequiredArgsConstructor -@Log4j2 -public class BulkEditUserContentUpdateService { - private final RemoteFilesStorage remoteFilesStorage; - private final BulkEditProcessingErrorsService errorsService; - private final EmailUpdateStrategy emailUpdateStrategy; - private final ExpirationDateUpdateStrategy expirationDateUpdateStrategy; - private final PatronGroupUpdateStrategy patronGroupUpdateStrategy; - private final UserContentUpdateValidatorService validatorService; - - public UpdatesResult process(JobCommand jobCommand, UserContentUpdateCollection contentUpdates) { - validatorService.validateContentUpdateCollection(contentUpdates); - try { - log.info("process:: Processing content updates for job id {}", jobCommand.getId()); - var fileName = FilenameUtils.getName(jobCommand.getJobParameters().getString(TEMP_OUTPUT_FILE_PATH)) + CSV_EXTENSION; - var updatedFileName = jobCommand.getId() + PATH_SEPARATOR + UPDATED_PREFIX + fileName; - var previewFileName = jobCommand.getId() + PATH_SEPARATOR + PREVIEW_PREFIX + fileName; - var userFormats = CsvHelper.readRecordsFromStorage(remoteFilesStorage, jobCommand.getId() + PATH_SEPARATOR + fileName, UserFormat.class, true); - log.info("process:: Reading of file {} complete, number of userFormats: {}", fileName, userFormats.size()); - var contentUpdatedUsers = applyContentUpdates(userFormats, contentUpdates, jobCommand); - log.info("process:: Finished processing content updates: {} records, {} preview", contentUpdatedUsers.getUpdated().size(), contentUpdatedUsers.getPreview().size()); - CsvHelper.saveRecordsToStorage(remoteFilesStorage, contentUpdatedUsers.getUpdated(), UserFormat.class, updatedFileName); - CsvHelper.saveRecordsToStorage(remoteFilesStorage, contentUpdatedUsers.getPreview(), UserFormat.class, previewFileName); - jobCommand.setJobParameters(new JobParametersBuilder(jobCommand.getJobParameters()) - .addString(UPDATED_FILE_NAME, updatedFileName) - .addString(PREVIEW_FILE_NAME, previewFileName) - .toJobParameters()); - jobCommand.setExportType(BULK_EDIT_UPDATE); - return new UpdatesResult().withTotal(userFormats.size()).withEntitiesForPreview(contentUpdatedUsers.getPreview()); - } catch (Exception e) { - var msg = String.format("I/O exception for job id %s, reason: %s", jobCommand.getId(), e.getMessage()); - log.error(msg); - throw new FileOperationException(msg); - } - } - - private ContentUpdateRecords applyContentUpdates(List userFormats, UserContentUpdateCollection contentUpdateCollection, JobCommand jobCommand) { - var updateResult = new ContentUpdateRecords(); - userFormats.forEach(userFormat -> { - var updatedUser = userFormat; - for (UserContentUpdate contentUpdate: contentUpdateCollection.getUserContentUpdates()) { - try { - updatedUser = resolveUpdateStrategy(contentUpdate).applyUpdate(updatedUser, contentUpdate); - } catch (BulkEditException e) { - log.error("User content update {} was not applied for user {}, reason: {}", contentUpdate.getOption(), userFormat.getIdentifier(jobCommand.getJobParameters().getString(IDENTIFIER_TYPE)), e.getMessage()); - errorsService.saveErrorInCSV(jobCommand.getId().toString(), userFormat.getIdentifier(jobCommand.getJobParameters().getString(IDENTIFIER_TYPE)), e, FilenameUtils.getName(jobCommand.getJobParameters().getString(FILE_NAME))); - } - } - if (!Objects.equals(updatedUser, userFormat)) { - updateResult.addToUpdated(updatedUser); - } else { - errorsService.saveErrorInCSV(jobCommand.getId().toString(), userFormat.getIdentifier(jobCommand.getJobParameters().getString(IDENTIFIER_TYPE)), new BulkEditException(NO_CHANGE_MESSAGE), FilenameUtils.getName(jobCommand.getJobParameters().getString(FILE_NAME))); - } - updateResult.addToPreview(updatedUser); - }); - return updateResult; - } - - private UpdateStrategy resolveUpdateStrategy(UserContentUpdate update) { - switch (update.getOption()) { - case PATRON_GROUP: - return patronGroupUpdateStrategy; - case EXPIRATION_DATE: - return expirationDateUpdateStrategy; - case EMAIL_ADDRESS: - return emailUpdateStrategy; - default: - throw new BulkEditException(format("Content updates for %s not implemented", update.getOption())); - } - } -} diff --git a/src/main/java/org/folio/dew/service/update/EmailUpdateStrategy.java b/src/main/java/org/folio/dew/service/update/EmailUpdateStrategy.java deleted file mode 100644 index 3f2c11f73..000000000 --- a/src/main/java/org/folio/dew/service/update/EmailUpdateStrategy.java +++ /dev/null @@ -1,21 +0,0 @@ -package org.folio.dew.service.update; - -import lombok.extern.log4j.Log4j2; -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.domain.dto.UserFormat; -import org.folio.dew.error.BulkEditException; -import org.springframework.stereotype.Component; - -@Component -@Log4j2 -public class EmailUpdateStrategy implements UpdateStrategy { - @Override - public UserFormat applyUpdate(UserFormat userFormat, UserContentUpdate update) { - var findValue = update.getActions().get(0).getValue().toString(); - var replaceWithValue = update.getActions().get(1).getValue().toString(); - if (userFormat.getEmail().contains(findValue)) { - return userFormat.withEmail(userFormat.getEmail().replace(findValue, replaceWithValue)); - } - throw new BulkEditException("Email does not match find criteria"); - } -} diff --git a/src/main/java/org/folio/dew/service/update/ExpirationDateUpdateStrategy.java b/src/main/java/org/folio/dew/service/update/ExpirationDateUpdateStrategy.java deleted file mode 100644 index c4ff55708..000000000 --- a/src/main/java/org/folio/dew/service/update/ExpirationDateUpdateStrategy.java +++ /dev/null @@ -1,20 +0,0 @@ -package org.folio.dew.service.update; - -import static org.folio.dew.utils.BulkEditProcessorHelper.dateFromString; - -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.domain.dto.UserFormat; -import org.springframework.stereotype.Component; - -import java.util.Date; - -@Component -public class ExpirationDateUpdateStrategy implements UpdateStrategy { - @Override - public UserFormat applyUpdate(UserFormat userFormat, UserContentUpdate update) { - var action = update.getActions().get(0); - return userFormat - .withExpirationDate(action.getValue().toString()) - .withActive(Boolean.toString(dateFromString(action.getValue().toString()).after(new Date()))); - } -} diff --git a/src/main/java/org/folio/dew/service/update/HoldingsLocationUpdateStrategy.java b/src/main/java/org/folio/dew/service/update/HoldingsLocationUpdateStrategy.java deleted file mode 100644 index 0c743ba50..000000000 --- a/src/main/java/org/folio/dew/service/update/HoldingsLocationUpdateStrategy.java +++ /dev/null @@ -1,41 +0,0 @@ -package org.folio.dew.service.update; - -import static org.apache.commons.lang3.StringUtils.EMPTY; - -import org.folio.dew.domain.dto.HoldingsContentUpdate; -import org.folio.dew.domain.dto.HoldingsFormat; -import org.springframework.stereotype.Component; - -@Component -public class HoldingsLocationUpdateStrategy implements UpdateStrategy { - @Override - public HoldingsFormat applyUpdate(HoldingsFormat holdingsFormat, HoldingsContentUpdate update) { - switch (update.getAction()) { - case REPLACE_WITH: - return replaceLocation(holdingsFormat, update); - case CLEAR_FIELD: - return clearLocation(holdingsFormat); - default: - return holdingsFormat; - } - } - - private HoldingsFormat replaceLocation(HoldingsFormat holdingsFormat, HoldingsContentUpdate update) { - var newLocation = update.getValue().toString(); - switch (update.getOption()) { - case PERMANENT_LOCATION: - return holdingsFormat - .withPermanentLocation(newLocation); - case TEMPORARY_LOCATION: - return holdingsFormat - .withTemporaryLocation(newLocation); - default: - return holdingsFormat; - } - } - - private HoldingsFormat clearLocation(HoldingsFormat holdingsFormat) { - return holdingsFormat - .withTemporaryLocation(EMPTY); - } -} diff --git a/src/main/java/org/folio/dew/service/update/PatronGroupUpdateStrategy.java b/src/main/java/org/folio/dew/service/update/PatronGroupUpdateStrategy.java deleted file mode 100644 index eddfbe888..000000000 --- a/src/main/java/org/folio/dew/service/update/PatronGroupUpdateStrategy.java +++ /dev/null @@ -1,13 +0,0 @@ -package org.folio.dew.service.update; - -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.domain.dto.UserFormat; -import org.springframework.stereotype.Component; - -@Component -public class PatronGroupUpdateStrategy implements UpdateStrategy { - @Override - public UserFormat applyUpdate(UserFormat userFormat, UserContentUpdate update) { - return userFormat.withPatronGroup(update.getActions().get(0).getValue().toString()); - } -} diff --git a/src/main/java/org/folio/dew/service/update/UpdateStrategy.java b/src/main/java/org/folio/dew/service/update/UpdateStrategy.java deleted file mode 100644 index 583b033bf..000000000 --- a/src/main/java/org/folio/dew/service/update/UpdateStrategy.java +++ /dev/null @@ -1,5 +0,0 @@ -package org.folio.dew.service.update; - -public interface UpdateStrategy { - T applyUpdate(T entity, U update); -} diff --git a/src/main/java/org/folio/dew/service/validation/ContentUpdateValidator.java b/src/main/java/org/folio/dew/service/validation/ContentUpdateValidator.java deleted file mode 100644 index 4f06f8552..000000000 --- a/src/main/java/org/folio/dew/service/validation/ContentUpdateValidator.java +++ /dev/null @@ -1,5 +0,0 @@ -package org.folio.dew.service.validation; - -public interface ContentUpdateValidator { - boolean isValid(T update); -} diff --git a/src/main/java/org/folio/dew/service/validation/EmailUpdateValidator.java b/src/main/java/org/folio/dew/service/validation/EmailUpdateValidator.java deleted file mode 100644 index cdf15fe4d..000000000 --- a/src/main/java/org/folio/dew/service/validation/EmailUpdateValidator.java +++ /dev/null @@ -1,37 +0,0 @@ -package org.folio.dew.service.validation; - -import static java.util.Objects.nonNull; -import static org.apache.commons.lang3.ObjectUtils.isEmpty; -import static org.folio.dew.domain.dto.UserContentUpdateAction.NameEnum.FIND; -import static org.folio.dew.domain.dto.UserContentUpdateAction.NameEnum.REPLACE_WITH; - -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.error.ContentUpdateValidationException; -import org.springframework.stereotype.Component; - -import java.util.Objects; - -@Component -public class EmailUpdateValidator implements ContentUpdateValidator { - @Override - public boolean isValid(UserContentUpdate update) { - String errorMessage = null; - if (update.getActions().size() == 2) { - var findAction = update.getActions().get(0); - var replaceWithAction = update.getActions().get(1); - if (FIND != findAction.getName() || REPLACE_WITH != replaceWithAction.getName()) { - errorMessage = "Email update must contain FIND action followed by REPLACE_WITH action"; - } else if (isEmpty(findAction.getValue()) || isEmpty(replaceWithAction.getValue())) { - errorMessage = "FIND and REPLACE_WITH values cannot be null or empty"; - } else if (Objects.equals(findAction.getValue(), replaceWithAction.getValue())) { - errorMessage = "FIND and REPLACE_WITH values cannot be equal"; - } - } else { - errorMessage = "Email update must contain FIND action followed by REPLACE_WITH action"; - } - if (nonNull(errorMessage)) { - throw new ContentUpdateValidationException(errorMessage); - } - return true; - } -} diff --git a/src/main/java/org/folio/dew/service/validation/ExpirationDateUpdateValidator.java b/src/main/java/org/folio/dew/service/validation/ExpirationDateUpdateValidator.java deleted file mode 100644 index 1929da053..000000000 --- a/src/main/java/org/folio/dew/service/validation/ExpirationDateUpdateValidator.java +++ /dev/null @@ -1,42 +0,0 @@ -package org.folio.dew.service.validation; - -import static java.util.Objects.isNull; -import static java.util.Objects.nonNull; -import static org.apache.commons.lang3.ObjectUtils.isEmpty; -import static org.folio.dew.domain.dto.UserContentUpdateAction.NameEnum.REPLACE_WITH; -import static org.folio.dew.utils.Constants.DATE_TIME_PATTERN; - -import lombok.extern.log4j.Log4j2; -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.error.ContentUpdateValidationException; -import org.springframework.stereotype.Component; - -import java.time.LocalDateTime; -import java.time.format.DateTimeFormatter; -import java.time.format.DateTimeParseException; - -@Component -@Log4j2 -public class ExpirationDateUpdateValidator implements ContentUpdateValidator { - @Override - public boolean isValid(UserContentUpdate update) { - String errorMessage = null; - var action = update.getActions().size() == 1 ? update.getActions().get(0) : null; - if (isNull(action) || REPLACE_WITH != action.getName()) { - errorMessage = "Expiration date update should consist of single REPLACE_WITH action"; - } else if (isEmpty(action.getValue())) { - errorMessage = "Value cannot be empty"; - } else { - try { - LocalDateTime.parse(action.getValue().toString(), DateTimeFormatter.ofPattern(DATE_TIME_PATTERN)); - } catch (DateTimeParseException e) { - errorMessage = String.format("Invalid date format: %s, expected yyyy-MM-dd HH:mm:ss.SSSX", action.getValue().toString()); - log.error(errorMessage); - } - } - if (nonNull(errorMessage)) { - throw new ContentUpdateValidationException(errorMessage); - } - return true; - } -} diff --git a/src/main/java/org/folio/dew/service/validation/HoldingsContentUpdateValidatorService.java b/src/main/java/org/folio/dew/service/validation/HoldingsContentUpdateValidatorService.java deleted file mode 100644 index 40ce0bb2f..000000000 --- a/src/main/java/org/folio/dew/service/validation/HoldingsContentUpdateValidatorService.java +++ /dev/null @@ -1,32 +0,0 @@ -package org.folio.dew.service.validation; - -import lombok.RequiredArgsConstructor; -import org.folio.dew.domain.dto.HoldingsContentUpdate; -import org.folio.dew.domain.dto.HoldingsContentUpdateCollection; -import org.folio.dew.error.ContentUpdateValidationException; -import org.springframework.stereotype.Service; - -@Service -@RequiredArgsConstructor -public class HoldingsContentUpdateValidatorService { - private final HoldingsLocationUpdateValidator locationUpdateValidator; - - public boolean validateContentUpdateCollection(HoldingsContentUpdateCollection contentUpdateCollection) { - return contentUpdateCollection.getHoldingsContentUpdates().stream() - .allMatch(this::isValidContentUpdate); - } - - private boolean isValidContentUpdate(HoldingsContentUpdate update) { - return resolveValidator(update).isValid(update); - } - - private ContentUpdateValidator resolveValidator(HoldingsContentUpdate update) { - switch (update.getOption()) { - case TEMPORARY_LOCATION: - case PERMANENT_LOCATION: - return locationUpdateValidator; - default: - throw new ContentUpdateValidationException(update.getOption() + " update is not supported"); - } - } -} diff --git a/src/main/java/org/folio/dew/service/validation/HoldingsLocationUpdateValidator.java b/src/main/java/org/folio/dew/service/validation/HoldingsLocationUpdateValidator.java deleted file mode 100644 index 22ce5afb8..000000000 --- a/src/main/java/org/folio/dew/service/validation/HoldingsLocationUpdateValidator.java +++ /dev/null @@ -1,40 +0,0 @@ -package org.folio.dew.service.validation; - -import static java.util.Objects.isNull; -import static java.util.Objects.nonNull; -import static org.folio.dew.domain.dto.HoldingsContentUpdate.ActionEnum.REPLACE_WITH; -import static org.folio.dew.domain.dto.HoldingsContentUpdate.OptionEnum.PERMANENT_LOCATION; - -import lombok.RequiredArgsConstructor; -import org.folio.dew.domain.dto.HoldingsContentUpdate; -import org.folio.dew.error.BulkEditException; -import org.folio.dew.error.ContentUpdateValidationException; -import org.folio.dew.service.HoldingsReferenceService; -import org.springframework.stereotype.Component; - -@Component -@RequiredArgsConstructor -public class HoldingsLocationUpdateValidator implements ContentUpdateValidator { - private final HoldingsReferenceService referenceService; - @Override - public boolean isValid(HoldingsContentUpdate update) { - String errorMessage = null; - if (REPLACE_WITH == update.getAction()) { - try { - if (isNull(update.getValue())) { - errorMessage = "Location name cannot be empty"; - } - var locationName = update.getValue().toString(); - referenceService.getLocationByName(locationName); - } catch (BulkEditException e) { - errorMessage = "Location does not exist"; - } - } else if (PERMANENT_LOCATION == update.getOption()) { - errorMessage = "Permanent location cannot be cleared"; - } - if (nonNull(errorMessage)) { - throw new ContentUpdateValidationException(errorMessage); - } - return true; - } -} diff --git a/src/main/java/org/folio/dew/service/validation/PatronGroupUpdateValidator.java b/src/main/java/org/folio/dew/service/validation/PatronGroupUpdateValidator.java deleted file mode 100644 index bcc9453cd..000000000 --- a/src/main/java/org/folio/dew/service/validation/PatronGroupUpdateValidator.java +++ /dev/null @@ -1,37 +0,0 @@ -package org.folio.dew.service.validation; - -import static java.util.Objects.nonNull; -import static org.folio.dew.domain.dto.UserContentUpdateAction.NameEnum.REPLACE_WITH; - -import lombok.RequiredArgsConstructor; -import org.apache.commons.lang3.ObjectUtils; -import org.folio.dew.client.GroupClient; -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.error.ContentUpdateValidationException; -import org.springframework.stereotype.Component; - -@Component -@RequiredArgsConstructor -public class PatronGroupUpdateValidator implements ContentUpdateValidator { - private final GroupClient groupClient; - @Override - public boolean isValid(UserContentUpdate update) { - String errorMessage = null; - if (update.getActions().size() != 1) { - errorMessage = "Patron group update should consist of single REPLACE_WITH action"; - } else { - var action = update.getActions().get(0); - if (REPLACE_WITH != action.getName()) { - errorMessage = action.getName() + " cannot be applied to Patron group"; - } else if (ObjectUtils.isEmpty(action.getValue())) { - errorMessage = "REPLACE_WITH value cannot be null or empty"; - } else if (groupClient.getGroupByQuery(String.format("group==\"%s\"", action.getValue().toString())).getUsergroups().isEmpty()) { - errorMessage = "Non-existing patron group: " + action.getValue().toString(); - } - } - if (nonNull(errorMessage)) { - throw new ContentUpdateValidationException(errorMessage); - } - return true; - } -} diff --git a/src/main/java/org/folio/dew/service/validation/UserContentUpdateValidatorService.java b/src/main/java/org/folio/dew/service/validation/UserContentUpdateValidatorService.java deleted file mode 100644 index dd0d8ce85..000000000 --- a/src/main/java/org/folio/dew/service/validation/UserContentUpdateValidatorService.java +++ /dev/null @@ -1,37 +0,0 @@ -package org.folio.dew.service.validation; - -import lombok.RequiredArgsConstructor; -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.domain.dto.UserContentUpdateCollection; -import org.folio.dew.error.ContentUpdateValidationException; -import org.springframework.stereotype.Service; - -@Service -@RequiredArgsConstructor -public class UserContentUpdateValidatorService { - private final ExpirationDateUpdateValidator expirationDateUpdateValidator; - private final PatronGroupUpdateValidator patronGroupUpdateValidator; - private final EmailUpdateValidator emailUpdateValidator; - - public boolean validateContentUpdateCollection(UserContentUpdateCollection contentUpdateCollection) { - return contentUpdateCollection.getUserContentUpdates().stream() - .allMatch(this::isValidContentUpdate); - } - - private boolean isValidContentUpdate(UserContentUpdate update) { - return resolveValidator(update).isValid(update); - } - - private ContentUpdateValidator resolveValidator(UserContentUpdate update) { - switch (update.getOption()) { - case EXPIRATION_DATE: - return expirationDateUpdateValidator; - case PATRON_GROUP: - return patronGroupUpdateValidator; - case EMAIL_ADDRESS: - return emailUpdateValidator; - default: - throw new ContentUpdateValidationException(update.getOption() + " update is not supported"); - } - } -} diff --git a/src/main/java/org/folio/dew/utils/Constants.java b/src/main/java/org/folio/dew/utils/Constants.java index 99a3ea3d5..b0d6d571e 100644 --- a/src/main/java/org/folio/dew/utils/Constants.java +++ b/src/main/java/org/folio/dew/utils/Constants.java @@ -10,7 +10,6 @@ @UtilityClass public class Constants { - public static final int CHUNKS = 100; public static final String ROLLBACK_FILE = "rollBackFile"; public static final String TMP_DIR_PROPERTY = "java.io.tmpdir"; public static final String PATH_SEPARATOR = "/"; @@ -55,6 +54,7 @@ public class Constants { public static final String COMMA = ","; public static final String QUOTE = "\""; public static final String QUOTE_REPLACEMENT = "\"\""; + public static final char NEW_LINE = '\n'; public static final String NO_MATCH_FOUND_MESSAGE = "No match found"; public static final String NO_CHANGE_MESSAGE = "No change in value needed"; @@ -63,7 +63,11 @@ public class Constants { public static final String MULTIPLE_MATCHES_MESSAGE = "Multiple matches for the same identifier."; public static final String NO_MARC_CONTENT = "Cannot get marc content for record with id = %s, reason: %s"; public static final String NO_ITEM_AFFILIATION = "User %s does not have required affiliation to view the item record - %s=%s on the tenant %s"; + public static final String NO_ITEM_VIEW_PERMISSIONS = "User %s does not have required permission to view the item record - %s=%s on the tenant %s"; public static final String NO_HOLDING_AFFILIATION = "User %s does not have required affiliation to view the holdings record - %s=%s on the tenant %s"; + public static final String NO_HOLDING_VIEW_PERMISSIONS = "User %s does not have required permission to view the holdings record - %s=%s on the tenant %s"; + public static final String NO_INSTANCE_VIEW_PERMISSIONS = "User %s does not have required permission to view the instance record - %s=%s on the tenant %s"; + public static final String NO_USER_VIEW_PERMISSIONS = "User %s does not have required permission to view the user record - %s=%s on the tenant %s"; public static final String DUPLICATES_ACROSS_TENANTS = "Duplicates across tenants"; public static final String MODULE_NAME = "BULKEDIT"; diff --git a/src/main/java/org/folio/dew/utils/CsvHelper.java b/src/main/java/org/folio/dew/utils/CsvHelper.java index 4d0b2405a..c2d1aa3ae 100644 --- a/src/main/java/org/folio/dew/utils/CsvHelper.java +++ b/src/main/java/org/folio/dew/utils/CsvHelper.java @@ -82,9 +82,9 @@ public static void saveRecordsToStorage(R storag } } - public static long countLines(R storage, String path, boolean skipHeaders) throws IOException { + public static long countLines(R storage, String path) throws IOException { try (var lines = storage.lines(path)) { - return skipHeaders ? lines.count() - 1 : lines.count(); + return lines.count(); } } } diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml index b933e57fb..6d797b7d6 100644 --- a/src/main/resources/application.yml +++ b/src/main/resources/application.yml @@ -79,20 +79,22 @@ application: topic-pattern: ${ENV:folio}.(.*\.)?data-export.job.command group-id: ${ENV:folio}-mod-data-export-worker-events-group minio-remote: - endpoint: ${AWS_URL:http://127.0.0.1:9000/} - region: ${AWS_REGION:} - bucket: ${AWS_BUCKET:} - accessKey: ${AWS_ACCESS_KEY_ID:} - secretKey: ${AWS_SECRET_ACCESS_KEY:} - composeWithAwsSdk: ${LOCAL_FS_COMPOSE_WITH_AWS_SDK:false} + endpoint: ${S3_URL:http://127.0.0.1:9000/} + region: ${S3_REGION:} + bucket: ${S3_BUCKET:} + accessKey: ${S3_ACCESS_KEY_ID:} + secretKey: ${S3_SECRET_ACCESS_KEY:} + composeWithAwsSdk: ${S3_IS_AWS:false} + subPath: ${S3_SUB_PATH:mod-data-export-worker/remote} url-expiration-time-in-seconds: ${URL_EXPIRATION_TIME:604800} # 7 days minio-local: - endpoint: ${LOCAL_FS_URL:http://127.0.0.1:9000/} - region: ${LOCAL_FS_REGION:} - bucket: ${LOCAL_FS_BUCKET:} - accessKey: ${LOCAL_FS_ACCESS_KEY_ID:} - secretKey: ${LOCAL_FS_SECRET_ACCESS_KEY:} - composeWithAwsSdk: ${LOCAL_FS_COMPOSE_WITH_AWS_SDK:false} + endpoint: ${S3_URL:http://127.0.0.1:9000/} + region: ${S3_REGION:} + bucket: ${S3_BUCKET:} + accessKey: ${S3_ACCESS_KEY_ID:} + secretKey: ${S3_SECRET_ACCESS_KEY:} + composeWithAwsSdk: ${S3_IS_AWS:false} + subPath: ${S3_LOCAL_SUB_PATH:mod-data-export-worker/local} url-expiration-time-in-seconds: ${URL_EXPIRATION_TIME:604800} # 7 days ftp: bufferSize: 1048576 #that 1024 * 1024 @@ -107,6 +109,10 @@ application: authority-control-batch: job-chunk-size: ${AUTHORITY_CONTROL_BATCH_JOB_CHUNK_SIZE:100} entities-links-chunk-size: ${AUTHORITY_CONTROL_BATCH_ENTITIES_LINKS_CHUNK_SIZE:100} + chunks: ${CHUNKS:100} + core-pool-size: ${CORE_POOL_SIZE:10} + max-pool-size: ${MAX_POOL_SIZE:10} + platform: ${PLATFORM:okapi} folio: tenant: diff --git a/src/main/resources/swagger.api/bulk-edit.yaml b/src/main/resources/swagger.api/bulk-edit.yaml index 15299b86e..8fb09ef6e 100644 --- a/src/main/resources/swagger.api/bulk-edit.yaml +++ b/src/main/resources/swagger.api/bulk-edit.yaml @@ -5,64 +5,10 @@ info: servers: - url: /bulk-edit/ paths: - /{jobId}/item-content-update/upload: - post: - description: Upload item content updates - operationId: postItemContentUpdates - parameters: - - name: jobId - in: path - required: true - description: UUID of the JobCommand - schema: - $ref: "#/components/schemas/UUID" - - in: query - name: limit - required: false - schema: - type: integer - description: The numbers of records to return - requestBody: - required: true - content: - application/json: - schema: - $ref: "#/components/schemas/itemContentUpdateCollection" - responses: - "200": - description: Collection of items for preview - content: - application/json: - schema: - $ref: "#/components/schemas/itemCollection" - "400": - description: Bad request - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - "404": - description: Not found - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - "500": - description: Internal server errors, e.g. due to misconfiguration - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - /{jobId}/user-content-update/upload: + /{jobId}/upload: post: - description: Upload user content updates - operationId: postUserContentUpdates + description: Upload csv file + operationId: uploadCsvFile parameters: - name: jobId in: path @@ -70,35 +16,24 @@ paths: description: UUID of the JobCommand schema: $ref: "#/components/schemas/UUID" - - in: query - name: limit - required: false - schema: - type: integer - description: The numbers of records to return requestBody: - required: true content: - application/json: + multipart/form-data: schema: - $ref: "#/components/schemas/userContentUpdateCollection" + type: object + properties: + file: + type: string + format: binary responses: - "200": - description: Collection of users for preview + "201": + description: File uploaded content: application/json: schema: - $ref: "#/components/schemas/userCollection" + type: string "400": - description: Bad request - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - "404": - description: Not found + description: Bad Request content: application/json: example: @@ -113,10 +48,10 @@ paths: $ref: "#/components/examples/errors" schema: $ref: "#/components/schemas/errors" - /{jobId}/holdings-content-update/upload: + /{jobId}/start: post: - description: Upload holdings record content updates - operationId: postHoldingsContentUpdates + description: Start job + operationId: startJob parameters: - name: jobId in: path @@ -124,113 +59,15 @@ paths: description: UUID of the JobCommand schema: $ref: "#/components/schemas/UUID" - - in: query - name: limit - required: false - schema: - type: integer - description: The numbers of records to return - requestBody: - required: true - content: - application/json: - schema: - $ref: "#/components/schemas/holdingsContentUpdateCollection" responses: "200": - description: Collection of holdings records for preview - content: - application/json: - schema: - $ref: "#/components/schemas/holdingsRecordCollection" - "400": - description: Bad request - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - "404": - description: Not found - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - "500": - description: Internal server errors, e.g. due to misconfiguration - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - /{jobId}/preview/updated-items/download: - get: - description: Download updated items preview as csv-file - operationId: downloadItemsPreviewByJobId - parameters: - - name: jobId - in: path - required: true - description: UUID of the JobCommand - schema: - $ref: "#/components/schemas/UUID" - responses: - '200': - description: Preview of updated items to download - content: - text/csv: - schema: - type: string - format: binary - "400": - description: Bad request - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - "404": - description: Not found - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - "500": - description: Internal server errors, e.g. due to misconfiguration + description: Job is started content: application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - /{jobId}/preview/updated-users/download: - get: - description: Download updated users preview as csv-file - operationId: downloadUsersPreviewByJobId - parameters: - - name: jobId - in: path - required: true - description: UUID of the JobCommand - schema: - $ref: "#/components/schemas/UUID" - responses: - '200': - description: Preview of updated items to download - content: - text/csv: schema: type: string - format: binary "400": - description: Bad request + description: Bad Request content: application/json: example: @@ -238,56 +75,12 @@ paths: schema: $ref: "#/components/schemas/errors" "404": - description: Not found - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - "500": - description: Internal server errors, e.g. due to misconfiguration - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - /{jobId}/preview/updated-holdings/download: - get: - description: Download updated holdings records preview as csv-file - operationId: downloadHoldingsPreviewByJobId - parameters: - - name: jobId - in: path - required: true - description: UUID of the JobCommand - schema: - $ref: "#/components/schemas/UUID" - responses: - '200': - description: Preview of updated holdings to download + description: Bad Request content: - text/csv: + text/plain: schema: type: string - format: binary - "400": - description: Bad request - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - "404": - description: Not found - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" + example: Job not found "500": description: Internal server errors, e.g. due to misconfiguration content: @@ -296,150 +89,6 @@ paths: $ref: "#/components/examples/errors" schema: $ref: "#/components/schemas/errors" - /{jobId}/preview/users: - get: - description: Get a list of users for preview - operationId: getPreviewUsersByJobId - parameters: - - name: jobId - in: path - required: true - description: UUID of the JobCommand - schema: - $ref: "#/components/schemas/UUID" - - in: query - name: limit - required: true - schema: - type: integer - description: The numbers of users to return - responses: - '200': - description: Collection of users for preview - content: - application/json: - schema: - $ref: "#/components/schemas/userCollection" - '400': - description: Bad Request - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - '404': - description: Bad Request - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - '500': - description: Internal server errors, e.g. due to misconfiguration - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - /{jobId}/preview/items: - get: - description: Get a list of items for preview - operationId: getPreviewItemsByJobId - parameters: - - name: jobId - in: path - required: true - description: UUID of the JobCommand - schema: - $ref: "#/components/schemas/UUID" - - in: query - name: limit - required: true - schema: - type: integer - description: The numbers of items to return - responses: - '200': - description: Collection of items for preview - content: - application/json: - schema: - $ref: "#/components/schemas/itemCollection" - '400': - description: Bad Request - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - '404': - description: Not found - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - '500': - description: Internal server errors, e.g. due to misconfiguration - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - /{jobId}/preview/holdings: - get: - description: Get a list of holdings for preview - operationId: getPreviewHoldingsByJobId - parameters: - - name: jobId - in: path - required: true - description: UUID of the JobCommand - schema: - $ref: "#/components/schemas/UUID" - - in: query - name: limit - required: true - schema: - type: integer - description: The numbers of holdings to return - responses: - '200': - description: Collection of holdings for preview - content: - application/json: - schema: - $ref: "#/components/schemas/holdingsRecordCollection" - '400': - description: Bad Request - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - '404': - description: Not found - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - '500': - description: Internal server errors, e.g. due to misconfiguration - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" /{jobId}/errors: get: description: Get a list of errors for preview @@ -480,118 +129,37 @@ paths: schema: type: string example: Internal server error - /{jobId}/upload: - post: - description: Upload csv file - operationId: uploadCsvFile - parameters: - - name: jobId - in: path - required: true - description: UUID of the JobCommand - schema: - $ref: "#/components/schemas/UUID" - requestBody: - content: - multipart/form-data: - schema: - type: object - properties: - file: - type: string - format: binary - responses: - "201": - description: File uploaded - content: - application/json: - schema: - type: string - "400": - description: Bad Request - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - "500": - description: Internal server errors, e.g. due to misconfiguration - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - /{jobId}/start: - post: - description: Start job - operationId: startJob - parameters: - - name: jobId - in: path - required: true - description: UUID of the JobCommand - schema: - $ref: "#/components/schemas/UUID" + /permissions-self-check: + get: + description: Get users permissions + operationId: getUsersPermissions responses: - "200": - description: Job is started - content: - application/json: - schema: - type: string - "400": - description: Bad Request + '200': + description: List of actual users permissions for current tenant content: application/json: - example: - $ref: "#/components/examples/errors" schema: - $ref: "#/components/schemas/errors" - "404": - description: Bad Request + type: array + items: + $ref: "#/components/schemas/userPermission" + '404': + description: No found content: text/plain: schema: type: string - example: Job not found - "500": + example: Desired permissions not found + '500': description: Internal server errors, e.g. due to misconfiguration - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" - /{jobId}/roll-back: - post: - description: Roll back csv file - operationId: rollBackCsvFile - parameters: - - name: jobId - in: path - required: true - description: UUID of the JobCommand - schema: - $ref: "#/components/schemas/UUID" - responses: - "200": - description: Csv file roll back uploaded content: text/plain: schema: type: string - "500": - description: Internal server errors, e.g. due to misconfiguration - content: - application/json: - example: - $ref: "#/components/examples/errors" - schema: - $ref: "#/components/schemas/errors" + example: Internal server error components: schemas: + userPermission: + type: string UUID: type: string format: uuid @@ -641,10 +209,6 @@ components: $ref: '../../../../folio-export-common/schemas/inventory/electronicAccessRelationshipCollection.json#/ElectronicAccessRelationshipCollection' statisticalCodes: $ref: '../../../../folio-export-common/schemas/inventory/statisticalCodeCollection.json#/StatisticalCodeCollection' - itemContentUpdateCollection: - $ref: '../../../../folio-export-common/schemas/bulk-edit/itemContentUpdateCollection.json#/ItemContentUpdateCollection' - userContentUpdateCollection: - $ref: '../../../../folio-export-common/schemas/bulk-edit/userContentUpdateCollection.json#/UserContentUpdateCollection' customFieldCollection: $ref: '../../../../folio-export-common/schemas/user/customFieldCollection.json#/CustomFieldCollection' holdingsRecordCollection: diff --git a/src/test/java/org/folio/dew/AuthorityControlTest.java b/src/test/java/org/folio/dew/AuthorityControlTest.java index 245090064..18b03d814 100644 --- a/src/test/java/org/folio/dew/AuthorityControlTest.java +++ b/src/test/java/org/folio/dew/AuthorityControlTest.java @@ -50,7 +50,7 @@ class AuthorityControlTest extends BaseBatchTest { "src/test/resources/output/authority_control/auth_heading_update.csv"; private static final String EXPECTED_AUTH_HEADING_UPDATE_EMPTY_OUTPUT = "src/test/resources/output/authority_control/auth_heading_update_empty.csv"; - private static final String FILE_PATH = "mod-data-export-worker/authority_control_export/diku/"; + private static final String EXPECTED_S3_FILE_PATH = "remote/mod-data-export-worker/authority_control_export/diku/"; @Autowired private Job getAuthHeadingJob; @Autowired @@ -129,7 +129,7 @@ private void verifyFile(JobExecution jobExecution, String expectedFile) throws E final String fileInStorage = executionContext.getString(OUTPUT_FILES_IN_STORAGE); final String fileName = executionContext.getString(AUTHORITY_CONTROL_FILE_NAME); - assertEquals(FILE_PATH + fileName, fileInStorage); + assertEquals(EXPECTED_S3_FILE_PATH + fileName, fileInStorage); verifyFileOutput(fileInStorage, expectedFile); } @@ -148,7 +148,7 @@ private void verifyJobEvent() { final String filePath = job.getFiles().get(0); final String fileName = job.getFileNames().get(0); - assertEquals(FILE_PATH + fileName, filePath); + assertEquals(EXPECTED_S3_FILE_PATH + fileName, filePath); } @SneakyThrows diff --git a/src/test/java/org/folio/dew/BaseBatchTest.java b/src/test/java/org/folio/dew/BaseBatchTest.java index 7204fdbe6..cbd47e5cc 100644 --- a/src/test/java/org/folio/dew/BaseBatchTest.java +++ b/src/test/java/org/folio/dew/BaseBatchTest.java @@ -28,6 +28,7 @@ import org.folio.dew.batch.ExportJobManagerSync; import org.folio.dew.client.ConsortiaClient; import org.folio.dew.client.SearchClient; +import org.folio.dew.client.OkapiUserPermissionsClient; import org.folio.dew.domain.dto.BatchIdsDto; import org.folio.dew.domain.dto.ConsortiumHolding; import org.folio.dew.domain.dto.ConsortiumHoldingCollection; @@ -115,6 +116,8 @@ public abstract class BaseBatchTest { private SearchClient searchClient; @MockBean private ConsortiaClient consortiaClient; + @MockBean + protected OkapiUserPermissionsClient okapiUserPermissionsClient; static { postgreDBContainer.start(); diff --git a/src/test/java/org/folio/dew/BulkEditTest.java b/src/test/java/org/folio/dew/BulkEditTest.java index f398f041e..cbad9a740 100644 --- a/src/test/java/org/folio/dew/BulkEditTest.java +++ b/src/test/java/org/folio/dew/BulkEditTest.java @@ -16,18 +16,13 @@ import org.folio.dew.domain.dto.IdentifierType; import org.folio.dew.domain.dto.JobParameterNames; import org.folio.dew.repository.LocalFilesStorage; -import org.folio.dew.service.BulkEditProcessingErrorsService; -import org.folio.dew.service.update.BulkEditHoldingsContentUpdateService; -import org.folio.dew.utils.Constants; import org.folio.spring.FolioExecutionContext; -import org.junit.jupiter.api.Disabled; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.ExtendWith; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.CsvSource; import org.junit.jupiter.params.provider.EnumSource; -import org.junit.jupiter.params.provider.ValueSource; import org.mockito.ArgumentCaptor; import org.mockito.Mock; import org.mockito.Mockito; @@ -52,9 +47,9 @@ import java.nio.charset.Charset; import java.nio.file.Files; import java.nio.file.Path; +import java.util.Arrays; import java.util.Collections; import java.util.List; -import java.util.Map; import java.util.Objects; import java.util.UUID; @@ -79,7 +74,6 @@ import static org.folio.dew.domain.dto.JobParameterNames.TEMP_LOCAL_MARC_PATH; import static org.folio.dew.domain.dto.JobParameterNames.TEMP_OUTPUT_FILE_PATH; import static org.folio.dew.domain.dto.JobParameterNames.TEMP_OUTPUT_MARC_PATH; -import static org.folio.dew.service.FolioExecutionContextManager.X_OKAPI_TENANT; import static org.folio.dew.utils.Constants.BULKEDIT_DIR_NAME; import static org.folio.dew.utils.Constants.ENTITY_TYPE; import static org.folio.dew.utils.Constants.EXPORT_TYPE; @@ -110,12 +104,12 @@ class BulkEditTest extends BaseBatchTest { private static final String EXPECTED_HOLDINGS_OUTPUT_BY_ITEM_BARCODE_CSV = "src/test/resources/output/bulk_edit_holdings_records_by_item_barcode.csv"; private static final String ITEM_IDENTIFIERS_BAD_REFERENCE_IDS_CSV = "src/test/resources/upload/item_identifiers_bad_reference.csv"; private static final String EXPECTED_ITEMS_OUTPUT_BAD_REFERENCE_CSV = "src/test/resources/output/bulk_edit_items_reference_not_found.csv"; - private final static String EXPECTED_ITEM_OUTPUT_BAD_REFERENCE_ERRORS = "src/test/resources/output/bulk_edit_items_bad_reference_errors.csv"; + private static final String EXPECTED_ITEM_OUTPUT_BAD_REFERENCE_ERRORS = "src/test/resources/output/bulk_edit_items_bad_reference_errors.csv"; private static final String ITEM_IDENTIFIERS_EMPTY_REFERENCE_IDS_CSV = "src/test/resources/upload/item_identifiers_empty_reference.csv"; private static final String EXPECTED_ITEM_OUTPUT_EMPTY_REFERENCE_CSV = "src/test/resources/output/bulk_edit_items_empty_reference.csv"; private static final String USER_IDENTIFIERS_BAD_REFERENCE_IDS_CSV = "src/test/resources/upload/user_identifiers_bad_reference.csv"; private static final String EXPECTED_USER_OUTPUT_BAD_REFERENCE_CSV = "src/test/resources/output/bulk_edit_users_reference_not_found.csv"; - private final static String EXPECTED_USER_OUTPUT_BAD_REFERENCE_ERRORS = "src/test/resources/output/bulk_edit_users_bad_reference_errors.csv"; + private static final String EXPECTED_USER_OUTPUT_BAD_REFERENCE_ERRORS = "src/test/resources/output/bulk_edit_users_bad_reference_errors.csv"; private static final String USER_IDENTIFIERS_EMPTY_REFERENCE_IDS_CSV = "src/test/resources/upload/user_identifiers_empty_reference.csv"; private static final String EXPECTED_USER_OUTPUT_EMPTY_REFERENCE_CSV = "src/test/resources/output/bulk_edit_users_empty_reference.csv"; private static final String BARCODES_CSV = "src/test/resources/upload/barcodes.csv"; @@ -130,17 +124,9 @@ class BulkEditTest extends BaseBatchTest { private static final String INSTANCE_ISSN_ISBN_CSV = "src/test/resources/upload/instance_ISSN_ISBN.csv"; private static final String ITEM_BARCODES_DOUBLE_QOUTES_CSV = "src/test/resources/upload/item_barcodes_double_qoutes.csv"; private static final String ITEM_HOLDINGS_CSV = "src/test/resources/upload/item_holdings.csv"; - private static final String USER_RECORD_CSV = "src/test/resources/upload/bulk_edit_user_record.csv"; - private static final String USER_RECORD_CSV_NOT_FOUND = "src/test/resources/upload/bulk_edit_user_record_not_found.csv"; - private static final String USER_RECORD_CSV_BAD_CONTENT = "src/test/resources/upload/bulk_edit_user_record_bad_content.csv"; - private static final String USER_RECORD_CSV_BAD_CUSTOM_FIELD = "src/test/resources/upload/bulk_edit_user_record_bad_custom_field.csv"; - private static final String USER_RECORD_CSV_EMPTY_PATRON_GROUP = "src/test/resources/upload/bulk_edit_user_record_empty_patron_group.csv"; - private static final String USER_RECORD_ROLLBACK_CSV = "test-directory/bulk_edit_rollback.csv"; private static final String BARCODES_SOME_NOT_FOUND = "src/test/resources/upload/barcodesSomeNotFound.csv"; private static final String ITEM_BARCODES_SOME_NOT_FOUND = "src/test/resources/upload/item_barcodes_some_not_found.csv"; private static final String INSTANCE_HRIDS_SOME_NOT_FOUND = "src/test/resources/upload/instance_hrids_some_not_found.csv"; - private static final String USERS_QUERY_FILE_PATH = "src/test/resources/upload/users_by_group.cql"; - private static final String ITEMS_QUERY_FILE_PATH = "src/test/resources/upload/items_by_barcode.cql"; private static final String QUERY_NO_GROUP_FILE_PATH = "src/test/resources/upload/active_no_group.cql"; private static final String EXPECTED_BULK_EDIT_USER_OUTPUT = "src/test/resources/output/bulk_edit_user_identifiers_output.csv"; private static final String EXPECTED_BULK_EDIT_USER_PREFERRED_EMAIL_OUTPUT = "src/test/resources/output/bulk_edit_user_identifiers_preferred_email_output.csv"; @@ -153,7 +139,6 @@ class BulkEditTest extends BaseBatchTest { private static final String EXPECTED_BULK_EDIT_INSTANCE_JSON_OUTPUT = "src/test/resources/output/bulk_edit_instance_identifiers_json_output.json"; private static final String EXPECTED_BULK_EDIT_INSTANCE_BY_ISSN_JSON_OUTPUT = "src/test/resources/output/bulk_edit_instance_by_issn_json_output.json"; private static final String EXPECTED_BULK_EDIT_INSTANCE_BY_ISBN_JSON_OUTPUT = "src/test/resources/output/bulk_edit_instance_by_isbn_json_output.json"; - private static final String EXPECTED_BULK_EDIT_ITEM_QUERY_JSON_OUTPUT = "src/test/resources/output/bulk_edit_item_query_json_output.json"; private static final String EXPECTED_BULK_EDIT_HOLDINGS_OUTPUT = "src/test/resources/output/bulk_edit_holdings_records_output.csv"; private static final String EXPECTED_BULK_EDIT_HOLDINGS_JSON_OUTPUT = "src/test/resources/output/bulk_edit_holdings_records_json_output.json"; @@ -162,18 +147,17 @@ class BulkEditTest extends BaseBatchTest { private static final String EXPECTED_BULK_EDIT_HOLDINGS_OUTPUT_ITEM_BARCODE = "src/test/resources/output/bulk_edit_holdings_records_output_item_barcode.csv"; private static final String EXPECTED_BULK_EDIT_ITEM_OUTPUT_ESCAPED = "src/test/resources/output/bulk_edit_item_identifiers_output_escaped.csv"; private static final String EXPECTED_NO_GROUP_OUTPUT = "src/test/resources/output/bulk_edit_no_group_output.csv"; - private static final String EXPECTED_ITEMS_QUERY_OUTPUT = "src/test/resources/output/bulk_edit_item_query_output.csv"; - private final static String EXPECTED_BULK_EDIT_OUTPUT_SOME_NOT_FOUND = "src/test/resources/output/bulk_edit_user_identifiers_output_some_not_found.csv"; - private final static String EXPECTED_BULK_EDIT_ITEM_OUTPUT_SOME_NOT_FOUND = "src/test/resources/output/bulk_edit_item_identifiers_output_some_not_found.csv"; - private final static String EXPECTED_BULK_EDIT_INSTANCE_OUTPUT_SOME_NOT_FOUND = "src/test/resources/output/bulk_edit_instance_identifiers_output_some_not_found.csv"; - private final static String EXPECTED_BULK_EDIT_OUTPUT_ERRORS = "src/test/resources/output/bulk_edit_user_identifiers_errors_output.csv"; - private final static String EXPECTED_BULK_EDIT_ITEM_OUTPUT_ERRORS = "src/test/resources/output/bulk_edit_item_identifiers_errors_output.csv"; - private final static String EXPECTED_BULK_EDIT_INSTANCE_OUTPUT_ERRORS = "src/test/resources/output/bulk_edit_instance_identifiers_errors_output.csv"; - private final static String EXPECTED_BULK_EDIT_HOLDINGS_ERRORS = "src/test/resources/output/bulk_edit_holdings_records_errors_output.csv"; - private final static String EXPECTED_BULK_EDIT_HOLDINGS_BAD_REFERENCE_IDS_ERRORS = "src/test/resources/output/bulk_edit_holdings_records_bad_reference_ids_errors_output.csv"; - private final static String EXPECTED_BULK_EDIT_HOLDINGS_ERRORS_INST_HRID = "src/test/resources/output/bulk_edit_holdings_records_errors_output_inst_hrid.csv"; + private static final String EXPECTED_BULK_EDIT_OUTPUT_SOME_NOT_FOUND = "src/test/resources/output/bulk_edit_user_identifiers_output_some_not_found.csv"; + private static final String EXPECTED_BULK_EDIT_ITEM_OUTPUT_SOME_NOT_FOUND = "src/test/resources/output/bulk_edit_item_identifiers_output_some_not_found.csv"; + private static final String EXPECTED_BULK_EDIT_INSTANCE_OUTPUT_SOME_NOT_FOUND = "src/test/resources/output/bulk_edit_instance_identifiers_output_some_not_found.csv"; + private static final String EXPECTED_BULK_EDIT_OUTPUT_ERRORS = "src/test/resources/output/bulk_edit_user_identifiers_errors_output.csv"; + private static final String EXPECTED_BULK_EDIT_ITEM_OUTPUT_ERRORS = "src/test/resources/output/bulk_edit_item_identifiers_errors_output.csv"; + private static final String EXPECTED_BULK_EDIT_INSTANCE_OUTPUT_ERRORS = "src/test/resources/output/bulk_edit_instance_identifiers_errors_output.csv"; + private static final String EXPECTED_BULK_EDIT_HOLDINGS_ERRORS = "src/test/resources/output/bulk_edit_holdings_records_errors_output.csv"; + private static final String EXPECTED_BULK_EDIT_HOLDINGS_BAD_REFERENCE_IDS_ERRORS = "src/test/resources/output/bulk_edit_holdings_records_bad_reference_ids_errors_output.csv"; + private static final String EXPECTED_BULK_EDIT_HOLDINGS_ERRORS_INST_HRID = "src/test/resources/output/bulk_edit_holdings_records_errors_output_inst_hrid.csv"; private static final String EXPECTED_BULK_EDIT_HOLDINGS_ERRORS_ITEM_BARCODE = "src/test/resources/output/bulk_edit_holdings_records_errors_output_item_barcode.csv"; - private final static String EXPECTED_BULK_EDIT_ITEM_IDENTIFIERS_HOLDINGS_ERRORS_OUTPUT = "src/test/resources/output/bulk_edit_item_identifiers_holdings_errors_output.csv"; + private static final String EXPECTED_BULK_EDIT_ITEM_IDENTIFIERS_HOLDINGS_ERRORS_OUTPUT = "src/test/resources/output/bulk_edit_item_identifiers_holdings_errors_output.csv"; @Autowired @@ -190,19 +174,7 @@ class BulkEditTest extends BaseBatchTest { @Autowired private Job bulkEditItemCqlJob; @Autowired - private Job bulkEditUpdateUserRecordsJob; - @Autowired - private Job bulkEditUpdateItemRecordsJob; - @Autowired - private Job bulkEditRollBackJob; - @Autowired - private Job bulkEditUpdateHoldingsRecordsJob; - @Autowired - private BulkEditProcessingErrorsService bulkEditProcessingErrorsService; - @Autowired private LocalFilesStorage localFilesStorage; - @Autowired - private BulkEditHoldingsContentUpdateService bulkEditHoldingsContentUpdateService; @MockBean private InstanceClient instanceClient; @MockBean @@ -248,8 +220,8 @@ void shouldUpdateProgressUponUserIdentifiersJob() throws Exception { var jobCaptor = ArgumentCaptor.forClass(org.folio.de.entity.Job.class); - // expected 4 events: 1st - job started, 2nd, 3rd - updates after each chunk (100 identifiers), 4th - job completed - Mockito.verify(kafkaService, Mockito.times(4)).send(any(), any(), jobCaptor.capture()); + // expected 4 events: 1st - job started, 2nd, 3rd, 4th, 5th - updates after each chunk (100 identifiers) from more than 1 thread, 6th - job completed + Mockito.verify(kafkaService, Mockito.times(6)).send(any(), any(), jobCaptor.capture()); verifyJobProgressUpdates(jobCaptor); } @@ -268,8 +240,8 @@ void shouldUpdateProgressUponItemIdentifiersJob() throws Exception { var jobCaptor = ArgumentCaptor.forClass(org.folio.de.entity.Job.class); - // expected 4 events: 1st - job started, 2nd, 3rd - updates after each chunk (100 identifiers), 4th - job completed - Mockito.verify(kafkaService, Mockito.times(4)).send(any(), any(), jobCaptor.capture()); + // expected 4 events: 1st - job started, 2nd, 3rd, 4th, 5th - updates after each chunk (100 identifiers) from more than 1 thread, 6th - job completed + Mockito.verify(kafkaService, Mockito.times(6)).send(any(), any(), jobCaptor.capture()); verifyJobProgressUpdates(jobCaptor); } @@ -334,7 +306,7 @@ void uploadMarcInstanceIdentifiersJobTest(String identifierType, String path) th FilenameUtils.removeExtension((new File(path)).getName()) + "E" + FilenameUtils.getExtension(path); parametersBuilder.addString(FILE_NAME, file); localFilesStorage.write(file, Files.readAllBytes(of)); - parametersBuilder.addLong(TOTAL_CSV_LINES, countLines(localFilesStorage, file, false), false); + parametersBuilder.addLong(TOTAL_CSV_LINES, countLines(localFilesStorage, file), false); var tempDir = getTempDirWithSeparatorSuffix() + springApplicationName + PATH_SEPARATOR + jobId; var tempFile = tempDir + PATH_SEPARATOR + of.getFileName(); @@ -385,7 +357,7 @@ void uploadMarcInstanceIdentifiersInvalidContentJobTest() throws Exception { FilenameUtils.removeExtension((new File(path)).getName()) + "E" + FilenameUtils.getExtension(path); parametersBuilder.addString(FILE_NAME, file); localFilesStorage.write(file, Files.readAllBytes(of)); - parametersBuilder.addLong(TOTAL_CSV_LINES, countLines(localFilesStorage, file, false), false); + parametersBuilder.addLong(TOTAL_CSV_LINES, countLines(localFilesStorage, file), false); var tempDir = getTempDirWithSeparatorSuffix() + springApplicationName + PATH_SEPARATOR + jobId; var tempFile = tempDir + PATH_SEPARATOR + of.getFileName(); @@ -481,10 +453,47 @@ void shouldSkipEmptyUserReferenceData() throws Exception { assertThat(jobExecution.getExitStatus()).isEqualTo(ExitStatus.COMPLETED); } - @ParameterizedTest - @EnumSource(value = IdentifierType.class, names = {"ID", "HRID", "INSTANCE_HRID", "ITEM_BARCODE"}, mode = EnumSource.Mode.INCLUDE) - @DisplayName("Run bulk-edit (holdings records identifiers) successfully") - void uploadHoldingsIdentifiersJobTest(IdentifierType identifierType) throws Exception { + @Test + void uploadHoldingsIdentifiersIdJobTest() throws Exception { + var identifierType = ID; + mockInstanceClient(); + mockInstanceClientForHrid(); + + JobLauncherTestUtils testLauncher = createTestLauncher(bulkEditProcessHoldingsIdentifiersJob); + + final JobParameters jobParameters = prepareJobParameters(BULK_EDIT_IDENTIFIERS, HOLDINGS_RECORD, identifierType, HOLDINGS_IDENTIFIERS_CSV); + JobExecution jobExecution = testLauncher.launchJob(jobParameters); + + var expectedErrorsOutputFilePath = EXPECTED_BULK_EDIT_HOLDINGS_ERRORS; + verifyFilesOutput(jobExecution, EXPECTED_BULK_EDIT_HOLDINGS_OUTPUT, expectedErrorsOutputFilePath); + verifyCsvAndJsonOutput(jobExecution, EXPECTED_BULK_EDIT_HOLDINGS_OUTPUT, EXPECTED_BULK_EDIT_HOLDINGS_JSON_OUTPUT); + + + assertThat(jobExecution.getExitStatus()).isEqualTo(ExitStatus.COMPLETED); + } + + @Test + void uploadHoldingsIdentifiersHridJobTest() throws Exception { + var identifierType = HRID; + mockInstanceClient(); + mockInstanceClientForHrid(); + + JobLauncherTestUtils testLauncher = createTestLauncher(bulkEditProcessHoldingsIdentifiersJob); + + final JobParameters jobParameters = prepareJobParameters(BULK_EDIT_IDENTIFIERS, HOLDINGS_RECORD, identifierType, HOLDINGS_IDENTIFIERS_CSV); + JobExecution jobExecution = testLauncher.launchJob(jobParameters); + + var expectedErrorsOutputFilePath = EXPECTED_BULK_EDIT_HOLDINGS_ERRORS; + verifyFilesOutput(jobExecution, EXPECTED_BULK_EDIT_HOLDINGS_OUTPUT, expectedErrorsOutputFilePath); + verifyCsvAndJsonOutput(jobExecution, EXPECTED_BULK_EDIT_HOLDINGS_OUTPUT, EXPECTED_BULK_EDIT_HOLDINGS_JSON_OUTPUT); + + + assertThat(jobExecution.getExitStatus()).isEqualTo(ExitStatus.COMPLETED); + } + + @Test + void uploadHoldingsIdentifiersInstanceHridJobTest() throws Exception { + var identifierType = INSTANCE_HRID; mockInstanceClient(); mockInstanceClientForHrid(); @@ -494,17 +503,26 @@ void uploadHoldingsIdentifiersJobTest(IdentifierType identifierType) throws Exce JobExecution jobExecution = testLauncher.launchJob(jobParameters); String expectedErrorsOutputFilePath; - if (INSTANCE_HRID == identifierType) { - expectedErrorsOutputFilePath = EXPECTED_BULK_EDIT_HOLDINGS_ERRORS_INST_HRID; - verifyFilesOutput(jobExecution, EXPECTED_BULK_EDIT_HOLDINGS_OUTPUT_INST_HRID, expectedErrorsOutputFilePath); - } else if (ITEM_BARCODE == identifierType) { - expectedErrorsOutputFilePath = EXPECTED_BULK_EDIT_HOLDINGS_ERRORS_ITEM_BARCODE; - verifyFilesOutput(jobExecution, EXPECTED_BULK_EDIT_HOLDINGS_OUTPUT_ITEM_BARCODE, expectedErrorsOutputFilePath); - } else { - expectedErrorsOutputFilePath = EXPECTED_BULK_EDIT_HOLDINGS_ERRORS; - verifyFilesOutput(jobExecution, EXPECTED_BULK_EDIT_HOLDINGS_OUTPUT, expectedErrorsOutputFilePath); - verifyCsvAndJsonOutput(jobExecution, EXPECTED_BULK_EDIT_HOLDINGS_OUTPUT, EXPECTED_BULK_EDIT_HOLDINGS_JSON_OUTPUT); - } + expectedErrorsOutputFilePath = EXPECTED_BULK_EDIT_HOLDINGS_ERRORS_INST_HRID; + verifyFilesOutput(jobExecution, EXPECTED_BULK_EDIT_HOLDINGS_OUTPUT_INST_HRID, expectedErrorsOutputFilePath); + + assertThat(jobExecution.getExitStatus()).isEqualTo(ExitStatus.COMPLETED); + } + + @Test + void uploadHoldingsIdentifiersItemBarcodeJobTest() throws Exception { + var identifierType = ITEM_BARCODE; + mockInstanceClient(); + mockInstanceClientForHrid(); + + JobLauncherTestUtils testLauncher = createTestLauncher(bulkEditProcessHoldingsIdentifiersJob); + + final JobParameters jobParameters = prepareJobParameters(BULK_EDIT_IDENTIFIERS, HOLDINGS_RECORD, identifierType, HOLDINGS_IDENTIFIERS_CSV); + JobExecution jobExecution = testLauncher.launchJob(jobParameters); + + String expectedErrorsOutputFilePath; + expectedErrorsOutputFilePath = EXPECTED_BULK_EDIT_HOLDINGS_ERRORS_ITEM_BARCODE; + verifyFilesOutput(jobExecution, EXPECTED_BULK_EDIT_HOLDINGS_OUTPUT_ITEM_BARCODE, expectedErrorsOutputFilePath); assertThat(jobExecution.getExitStatus()).isEqualTo(ExitStatus.COMPLETED); } @@ -613,19 +631,6 @@ void bulkEditInstanceJobTestWithErrors() throws Exception { assertThat(jobExecution.getExitStatus()).isEqualTo(ExitStatus.COMPLETED); } - @Test - @DisplayName("Run bulk-edit (user query) successfully") - void bulkEditUserQueryJobTest() throws Exception { - JobLauncherTestUtils testLauncher = createTestLauncher(bulkEditUserCqlJob); - - final JobParameters jobParameters = prepareJobParameters(ExportType.BULK_EDIT_QUERY, USER, BARCODE, USERS_QUERY_FILE_PATH); - JobExecution jobExecution = testLauncher.launchJob(jobParameters); - - verifyCsvAndJsonOutput(jobExecution, EXPECTED_BULK_EDIT_USER_OUTPUT, EXPECTED_BULK_EDIT_USER_JSON_OUTPUT); - - assertThat(jobExecution.getExitStatus()).isEqualTo(ExitStatus.COMPLETED); - } - @Test @DisplayName("Process users without patron group id successfully") void shouldProcessUsersWithoutPatronGroupIdSuccessfully() throws Exception { @@ -639,65 +644,6 @@ void shouldProcessUsersWithoutPatronGroupIdSuccessfully() throws Exception { assertThat(jobExecution.getExitStatus()).isEqualTo(ExitStatus.COMPLETED); } - @Test - @Deprecated - @Disabled - @DisplayName("Run bulk-edit (item query) successfully") - void bulkEditItemQueryJobTest() throws Exception { - mockInstanceClient(); - - JobLauncherTestUtils testLauncher = createTestLauncher(bulkEditItemCqlJob); - when(folioExecutionContext.getAllHeaders()).thenReturn(Map.of(X_OKAPI_TENANT, List.of("original"))); - - final JobParameters jobParameters = prepareJobParameters(ExportType.BULK_EDIT_QUERY, ITEM, BARCODE, ITEMS_QUERY_FILE_PATH); - JobExecution jobExecution = testLauncher.launchJob(jobParameters); - - verifyCsvAndJsonOutput(jobExecution, EXPECTED_ITEMS_QUERY_OUTPUT, EXPECTED_BULK_EDIT_ITEM_QUERY_JSON_OUTPUT); - - assertThat(jobExecution.getExitStatus()).isEqualTo(ExitStatus.COMPLETED); - } - - @Disabled - // TODO uncomment when resolved - @ParameterizedTest - @ValueSource(strings = {USER_RECORD_CSV, USER_RECORD_CSV_NOT_FOUND, USER_RECORD_CSV_BAD_CONTENT, USER_RECORD_CSV_BAD_CUSTOM_FIELD, USER_RECORD_CSV_EMPTY_PATRON_GROUP}) - @DisplayName("Run update user records w/ and w/o errors") - void uploadUserRecordsJobTest(String csvFileName) throws Exception { - JobLauncherTestUtils testLauncher = createTestLauncher(bulkEditUpdateUserRecordsJob); - final JobParameters jobParameters = prepareJobParameters(BULK_EDIT_UPDATE, USER, BARCODE, csvFileName); - JobExecution jobExecution = testLauncher.launchJob(jobParameters); - - assertThat(jobExecution.getExitStatus()).isEqualTo(ExitStatus.COMPLETED); - - var errors = bulkEditProcessingErrorsService.readErrorsFromCSV(jobExecution.getJobParameters().getString("jobId"), csvFileName, 10); - - if (!USER_RECORD_CSV.equals(csvFileName)) { - if (USER_RECORD_CSV_BAD_CUSTOM_FIELD.equals(csvFileName)) { - assertThat(errors.getErrors()).hasSize(2); - } else { - assertThat(errors.getErrors()).hasSize(1); - } - assertThat(jobExecution.getExecutionContext().getString(OUTPUT_FILES_IN_STORAGE)).isNotEmpty(); - } else { - assertThat(jobExecution.getExecutionContext().getString(OUTPUT_FILES_IN_STORAGE)).isNotEmpty(); - assertThat(errors.getErrors()).isEmpty(); - } - } - - @Disabled - // TODO uncomment when resolved - @Test - @DisplayName("Run rollback user records successfully") - void rollBackUserRecordsJobTest() throws Exception { - JobLauncherTestUtils testLauncher = createTestLauncher(bulkEditRollBackJob); - localFilesStorage.write(USER_RECORD_ROLLBACK_CSV, Files.readAllBytes(new File(USER_RECORD_CSV).toPath())); - var parametersBuilder = new JobParametersBuilder(); - parametersBuilder.addString(Constants.JOB_ID, "74914e57-3406-4757-938b-9a3f718d0ee6"); - parametersBuilder.addString(FILE_NAME, USER_RECORD_ROLLBACK_CSV); - JobExecution jobExecution = testLauncher.launchJob(parametersBuilder.toJobParameters()); - assertThat(jobExecution.getExitStatus()).isEqualTo(ExitStatus.COMPLETED); - } - @Test @DisplayName("Double quotes in data should be escaped") @SneakyThrows @@ -724,12 +670,18 @@ private void verifyFilesOutput(JobExecution jobExecution, String output) { if (StringUtils.isNotEmpty(errorInStorage)){ final FileSystemResource actualResultWithErrors = actualFileOutput(errorInStorage); final FileSystemResource expectedResultWithErrors = getExpectedResourceByJobName(jobExecution.getJobInstance().getJobName()); - assertEquals(new String(expectedResultWithErrors.getContentAsByteArray()), new String(actualResultWithErrors.getContentAsByteArray())); + assertEquals(getSortedOutput(expectedResultWithErrors), getSortedOutput(actualResultWithErrors)); } } final FileSystemResource actualResult = actualFileOutput(fileInStorage); FileSystemResource expectedCharges = new FileSystemResource(output); - assertEquals(new String(expectedCharges.getContentAsByteArray()), new String(actualResult.getContentAsByteArray())); + assertEquals(getSortedOutput(expectedCharges), getSortedOutput(actualResult)); + } + + private String getSortedOutput(FileSystemResource resource) throws IOException { + var lines = Files.readAllLines(Path.of(resource.getPath())); + Collections.sort(lines); + return lines.toString(); } private FileSystemResource getExpectedResourceByJobName(String jobName) { @@ -753,7 +705,7 @@ private void verifyCsvAndJsonOutput(JobExecution jobExecution, String output, St final FileSystemResource actualResult = actualFileOutput(fileInStorage); FileSystemResource expectedCharges = new FileSystemResource(output); - assertEquals(new String(expectedCharges.getContentAsByteArray()), new String(actualResult.getContentAsByteArray())); + assertEquals(getSortedOutput(expectedCharges), getSortedOutput(actualResult)); } private void assertFileEqualsIgnoringCreatedAndUpdatedDate(FileSystemResource expectedJsonFile, FileSystemResource actualJsonResult) @@ -761,7 +713,9 @@ private void assertFileEqualsIgnoringCreatedAndUpdatedDate(FileSystemResource ex var expectedContent = IOUtils.toString(expectedJsonFile.getInputStream(), Charset.forName("UTF-8")); var actualContent = IOUtils.toString(actualJsonResult.getInputStream(), Charset.forName("UTF-8")); String actualUpdated = ""; - for (String json : actualContent.split("\n")) { + var jsons = actualContent.split("\n"); + Arrays.sort(jsons); + for (String json : jsons) { var actualJsonItem = new JSONObject(json); actualJsonItem.remove("createdDate"); actualJsonItem.remove("updatedDate"); @@ -782,14 +736,14 @@ private void verifyFilesOutput(JobExecution jobExecution, String output, String String errorInStorage = links[1]; final FileSystemResource actualResultWithErrors = actualFileOutput(errorInStorage); final FileSystemResource expectedResultWithErrors = new FileSystemResource(expectedErrorOutput); - assertEquals(new String(expectedResultWithErrors.getContentAsByteArray()), new String(actualResultWithErrors.getContentAsByteArray())); + assertEquals(getSortedOutput(expectedResultWithErrors), getSortedOutput(actualResultWithErrors)); } if (isEmpty(fileInStorage)) { assertTrue(isEmpty(output)); } else { final FileSystemResource actualResult = actualFileOutput(fileInStorage); FileSystemResource expectedCharges = new FileSystemResource(output); - assertEquals(new String(expectedCharges.getContentAsByteArray()), new String(actualResult.getContentAsByteArray())); + assertEquals(getSortedOutput(expectedCharges), getSortedOutput(actualResult)); } } @@ -820,7 +774,7 @@ private JobParameters prepareJobParameters(ExportType exportType, EntityType ent var file = getWorkingDirectory("mod-data-export-worker", BULKEDIT_DIR_NAME) + FileNameUtils.getBaseName(path) + "E" + FileNameUtils.getExtension(path); parametersBuilder.addString(FILE_NAME, file); localFilesStorage.write(file, Files.readAllBytes(of)); - parametersBuilder.addLong(TOTAL_CSV_LINES, countLines(localFilesStorage, file, false), false); + parametersBuilder.addLong(TOTAL_CSV_LINES, countLines(localFilesStorage, file), false); } var tempDir = getTempDirWithSeparatorSuffix() + springApplicationName + PATH_SEPARATOR + jobId; @@ -845,21 +799,7 @@ private String readQueryString(String path) { } private void verifyJobProgressUpdates(ArgumentCaptor jobCaptor) { - var job = jobCaptor.getAllValues().get(1); - assertThat(job.getProgress().getTotal()).isEqualTo(179); - assertThat(job.getProgress().getProcessed()).isEqualTo(100); - assertThat(job.getProgress().getProgress()).isEqualTo(45); - assertThat(job.getProgress().getSuccess()).isEqualTo(80); - assertThat(job.getProgress().getErrors()).isZero(); - - job = jobCaptor.getAllValues().get(2); - assertThat(job.getProgress().getTotal()).isEqualTo(179); - assertThat(job.getProgress().getProcessed()).isEqualTo(179); - assertThat(job.getProgress().getProgress()).isEqualTo(90); - assertThat(job.getProgress().getSuccess()).isEqualTo(144); - assertThat(job.getProgress().getErrors()).isZero(); - - job = jobCaptor.getAllValues().get(3); + var job = jobCaptor.getAllValues().get(5); assertThat(job.getProgress().getTotal()).isEqualTo(179); assertThat(job.getProgress().getProcessed()).isEqualTo(179); assertThat(job.getProgress().getProgress()).isEqualTo(100); diff --git a/src/test/java/org/folio/dew/CirculationLogTest.java b/src/test/java/org/folio/dew/CirculationLogTest.java index 9205a11ca..b2566749c 100644 --- a/src/test/java/org/folio/dew/CirculationLogTest.java +++ b/src/test/java/org/folio/dew/CirculationLogTest.java @@ -9,7 +9,6 @@ import org.springframework.batch.core.ExitStatus; import org.springframework.batch.core.Job; import org.springframework.batch.core.JobExecution; -import org.springframework.batch.core.JobParameter; import org.springframework.batch.core.JobParameters; import org.springframework.batch.core.JobParametersBuilder; import org.springframework.batch.item.ExecutionContext; @@ -78,11 +77,12 @@ private void verifyFileOutput(JobExecution jobExecution) throws Exception { final ExecutionContext executionContext = jobExecution.getExecutionContext(); final String fileInStorage = (String) executionContext.get("outputFilesInStorage"); final String fileName = executionContext.getString(CIRCULATION_LOG_FILE_NAME); + final String expectedNameInStorage = "remote/" + fileName; final FileSystemResource actualChargeFeesFinesOutput = actualFileOutput(fileInStorage); FileSystemResource expectedCharges = new FileSystemResource(EXPECTED_CIRCULATION_OUTPUT); assertFileEquals(expectedCharges, actualChargeFeesFinesOutput); - assertEquals(fileName, fileInStorage); + assertEquals(expectedNameInStorage, fileInStorage); } private JobParameters prepareJobParameters() { diff --git a/src/test/java/org/folio/dew/EHoldingsTest.java b/src/test/java/org/folio/dew/EHoldingsTest.java index 4d3ce473f..db696bca2 100644 --- a/src/test/java/org/folio/dew/EHoldingsTest.java +++ b/src/test/java/org/folio/dew/EHoldingsTest.java @@ -80,7 +80,7 @@ class EHoldingsTest extends BaseBatchTest { "src/test/resources/output/eholdings_package_export_with_3_titles.csv"; private final static String EXPECTED_PACKAGE_WITH_SAME_TITLE_NAMES_OUTPUT = "src/test/resources/output/eholdings_package_export_with_same_title_names.csv"; - private static final String FILE_PATH = "mod-data-export-worker/e_holdings_export/diku/"; + private static final String FILE_PATH = "remote/mod-data-export-worker/e_holdings_export/diku/"; @Test @DisplayName("Run EHoldingsJob export resource without provider load successfully") diff --git a/src/test/java/org/folio/dew/batch/bulkedit/jobs/BulkEditProcessorsTest.java b/src/test/java/org/folio/dew/batch/bulkedit/jobs/BulkEditProcessorsTest.java index e6d656902..0fb5f77f0 100644 --- a/src/test/java/org/folio/dew/batch/bulkedit/jobs/BulkEditProcessorsTest.java +++ b/src/test/java/org/folio/dew/batch/bulkedit/jobs/BulkEditProcessorsTest.java @@ -6,10 +6,14 @@ import static org.folio.dew.utils.Constants.MULTIPLE_MATCHES_MESSAGE; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.ArgumentMatchers.isA; import static org.mockito.Mockito.when; import lombok.SneakyThrows; import org.folio.dew.BaseBatchTest; +import org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionsValidator; import org.folio.dew.batch.bulkedit.jobs.processidentifiers.InstanceFetcher; import org.folio.dew.batch.bulkedit.jobs.processidentifiers.ItemFetcher; import org.folio.dew.batch.bulkedit.jobs.processidentifiers.UserFetcher; @@ -17,6 +21,7 @@ import org.folio.dew.client.InventoryClient; import org.folio.dew.client.InventoryInstancesClient; import org.folio.dew.client.UserClient; +import org.folio.dew.domain.dto.EntityType; import org.folio.dew.domain.dto.ExtendedItem; import org.folio.dew.domain.dto.HoldingsRecord; import org.folio.dew.domain.dto.HoldingsRecordCollection; @@ -27,6 +32,7 @@ import org.folio.dew.domain.dto.User; import org.folio.dew.domain.dto.UserCollection; import org.folio.dew.error.BulkEditException; +import org.folio.spring.FolioExecutionContext; import org.junit.jupiter.api.Test; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.ValueSource; @@ -37,6 +43,7 @@ import org.springframework.batch.test.StepScopeTestUtils; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.mock.mockito.MockBean; +import org.springframework.boot.test.mock.mockito.SpyBean; import java.nio.file.Path; import java.util.Collections; @@ -65,6 +72,12 @@ class BulkEditProcessorsTest extends BaseBatchTest { private BulkEditHoldingsProcessor holdingsProcessor; @MockBean private HoldingClient holdingClient; + @MockBean + private PermissionsValidator permissionsValidator; + @MockBean + private TenantResolver tenantResolver; + @SpyBean + private FolioExecutionContext folioExecutionContext; @Test @SneakyThrows @@ -73,7 +86,7 @@ void shouldIgnoreListsWithNullsAndNullObjectsForItems() { StepExecution stepExecution = MetaDataInstanceFactory.createStepExecution(new JobParameters()); StepScopeTestUtils.doInStepScope(stepExecution, () -> { var itemFormat = bulkEditItemProcessor.process(item); - assertEquals("0e40884c-3523-4c6d-8187-d578e3d2794e;note;|0e40884c-3523-4c6d-8187-d578e3d2794e;note;false", itemFormat.getNotes()); + assertEquals("0e40884c-3523-4c6d-8187-d578e3d2794e;note;;tenant;0e40884c-3523-4c6d-8187-d578e3d2794e|0e40884c-3523-4c6d-8187-d578e3d2794e;note;false;tenant;0e40884c-3523-4c6d-8187-d578e3d2794e", itemFormat.getNotes()); assertEquals("check in (staff only) | check in", itemFormat.getCheckInNotes()); assertEquals("books;be53b4c9-6eb8-4bdf-a785-904cccd04146", itemFormat.getStatisticalCodes()); assertEquals("hrid;hrid;title|;;", itemFormat.getBoundWithTitles()); @@ -100,7 +113,9 @@ void shouldIgnoreListsWithNullsAndNullObjectsForUsers() { @ValueSource(strings = {"ID", "HRID"}) @SneakyThrows void shouldNotIncludeDuplicatedInstances(String identifierType) { + when(permissionsValidator.isBulkEditReadPermissionExists(isA(String.class), eq(EntityType.INSTANCE))).thenReturn(true); when(inventoryInstancesClient.getInstanceByQuery(String.format("%s==duplicateIdentifier", resolveIdentifier(identifierType)), 1)).thenReturn(new InstanceCollection().instances(Collections.emptyList()).totalRecords(2)); + StepExecution stepExecution = MetaDataInstanceFactory.createStepExecution(new JobParameters(Collections.singletonMap("identifierType", new JobParameter<>(identifierType, String.class)))); StepScopeTestUtils.doInStepScope(stepExecution, () -> { var identifier = new ItemIdentifier("duplicateIdentifier"); @@ -114,7 +129,12 @@ void shouldNotIncludeDuplicatedInstances(String identifierType) { @ValueSource(strings = {"ID", "HRID", "EXTERNAL_SYSTEM_ID", "USER_NAME"}) @SneakyThrows void shouldNotIncludeDuplicatedUsers(String identifierType) { - when(userClient.getUserByQuery(String.format("%s==\"duplicateIdentifier\"", resolveIdentifier(identifierType)), 1)).thenReturn(new UserCollection().users(Collections.singletonList(new User())).totalRecords(2)); + when(permissionsValidator.isBulkEditReadPermissionExists(isA(String.class), eq(EntityType.USER))).thenReturn(true); + when(userClient.getUserByQuery( + String.format("(cql.allRecords=1 NOT type=\"\" or type<>\"shadow\") and %s==\"duplicateIdentifier\"", resolveIdentifier(identifierType)), + 1 + )).thenReturn(new UserCollection().users(Collections.singletonList(new User())).totalRecords(2)); + StepExecution stepExecution = MetaDataInstanceFactory.createStepExecution(new JobParameters(Collections.singletonMap("identifierType", new JobParameter<>(identifierType, String.class)))); StepScopeTestUtils.doInStepScope(stepExecution, () -> { var identifier = new ItemIdentifier("duplicateIdentifier"); @@ -130,6 +150,8 @@ void shouldNotIncludeDuplicatedUsers(String identifierType) { void shouldNotIncludeDuplicatedItems(String identifierType) { when(inventoryClient.getItemByQuery(String.format(getMatchPattern(identifierType), resolveIdentifier(identifierType), "duplicateIdentifier"), Integer.MAX_VALUE)).thenReturn(new ItemCollection().items(List.of(new Item(), new Item())).totalRecords(2)); when(inventoryClient.getItemByQuery("barcode==\"duplicateIdentifier\"", Integer.MAX_VALUE)).thenReturn(new ItemCollection().items(List.of(new Item(), new Item())).totalRecords(2)); + when(permissionsValidator.isBulkEditReadPermissionExists(isA(String.class), eq(EntityType.ITEM))).thenReturn(true); + StepExecution stepExecution = MetaDataInstanceFactory.createStepExecution(new JobParameters(Collections.singletonMap("identifierType", new JobParameter<>(identifierType, String.class)))); StepScopeTestUtils.doInStepScope(stepExecution, () -> { var identifier = new ItemIdentifier("duplicateIdentifier"); @@ -144,6 +166,8 @@ void shouldNotIncludeDuplicatedItems(String identifierType) { @SneakyThrows void shouldNotIncludeDuplicatedHoldings(String identifierType) { when(holdingClient.getHoldingsByQuery(String.format("%s==duplicateIdentifier", resolveIdentifier(identifierType)))).thenReturn(new HoldingsRecordCollection().holdingsRecords(Collections.singletonList(new HoldingsRecord())).totalRecords(2)); + when(permissionsValidator.isBulkEditReadPermissionExists(isA(String.class), eq(EntityType.HOLDINGS_RECORD))).thenReturn(true); + StepExecution stepExecution = MetaDataInstanceFactory.createStepExecution(new JobParameters(Collections.singletonMap("identifierType", new JobParameter<>(identifierType, String.class)))); StepScopeTestUtils.doInStepScope(stepExecution, () -> { var identifier = new ItemIdentifier("duplicateIdentifier"); @@ -152,4 +176,82 @@ void shouldNotIncludeDuplicatedHoldings(String identifierType) { return null; }); } + + @Test + @SneakyThrows + void shouldProvideBulkEditExceptionWithNoInstanceViewPermissionMessage() { + var user = new User(); + user.setUsername("userName"); + + when(permissionsValidator.isBulkEditReadPermissionExists(isA(String.class), eq(EntityType.HOLDINGS_RECORD))).thenReturn(false); + when(userClient.getUserById(any())).thenReturn(user); + + StepExecution stepExecution = MetaDataInstanceFactory.createStepExecution(new JobParameters(Collections.singletonMap("identifierType", new JobParameter<>("HRID", String.class)))); + var expectedErrorMessage = "User userName does not have required permission to view the instance record - hrid=hrid on the tenant diku"; + StepScopeTestUtils.doInStepScope(stepExecution, () -> { + var identifier = new ItemIdentifier("hrid"); + var throwable = assertThrows(BulkEditException.class, () -> instanceFetcher.process(identifier)); + assertEquals(expectedErrorMessage, throwable.getMessage()); + return null; + }); + } + + @Test + @SneakyThrows + void shouldProvideBulkEditExceptionWithNoUserViewPermissionMessage() { + var user = new User(); + user.setUsername("userName"); + + when(permissionsValidator.isBulkEditReadPermissionExists(isA(String.class), eq(EntityType.USER))).thenReturn(false); + when(userClient.getUserById(any())).thenReturn(user); + + StepExecution stepExecution = MetaDataInstanceFactory.createStepExecution(new JobParameters(Collections.singletonMap("identifierType", new JobParameter<>("HRID", String.class)))); + var expectedErrorMessage = "User userName does not have required permission to view the user record - hrid=hrid on the tenant diku"; + StepScopeTestUtils.doInStepScope(stepExecution, () -> { + var identifier = new ItemIdentifier("hrid"); + var throwable = assertThrows(BulkEditException.class, () -> userFetcher.process(identifier)); + assertEquals(expectedErrorMessage, throwable.getMessage()); + return null; + }); + } + + @Test + @SneakyThrows + void shouldProvideBulkEditExceptionWithNoItemViewPermissionMessageForLocal() { + var user = new User(); + user.setUsername("userName"); + + when(permissionsValidator.isBulkEditReadPermissionExists(isA(String.class), eq(EntityType.HOLDINGS_RECORD))).thenReturn(false); + when(userClient.getUserById(any())).thenReturn(user); + + StepExecution stepExecution = MetaDataInstanceFactory.createStepExecution(new JobParameters(Collections.singletonMap("identifierType", new JobParameter<>("HRID", String.class)))); + var expectedErrorMessage = "User userName does not have required permission to view the item record - hrid=hrid on the tenant diku"; + StepScopeTestUtils.doInStepScope(stepExecution, () -> { + var identifier = new ItemIdentifier("hrid"); + var throwable = assertThrows(BulkEditException.class, () -> itemFetcher.process(identifier)); + assertEquals(expectedErrorMessage, throwable.getMessage()); + return null; + }); + } + + @Test + @SneakyThrows + void shouldProvideBulkEditExceptionWithNoHoldingsViewPermissionMessageForLocal() { + var user = new User(); + user.setUsername("userName"); + + when(permissionsValidator.isBulkEditReadPermissionExists(isA(String.class), eq(EntityType.HOLDINGS_RECORD))).thenReturn(false); + when(userClient.getUserById(any())).thenReturn(user); + + StepExecution stepExecution = MetaDataInstanceFactory.createStepExecution(new JobParameters(Collections.singletonMap("identifierType", new JobParameter<>("HRID", String.class)))); + var expectedErrorMessage = "User userName does not have required permission to view the holdings record - hrid=hrid on the tenant diku"; + StepScopeTestUtils.doInStepScope(stepExecution, () -> { + var identifier = new ItemIdentifier("hrid"); + var throwable = assertThrows(BulkEditException.class, () -> holdingsProcessor.process(identifier)); + assertEquals(expectedErrorMessage, throwable.getMessage()); + return null; + }); + } + + } diff --git a/src/test/java/org/folio/dew/batch/bulkedit/jobs/TenantResolverTest.java b/src/test/java/org/folio/dew/batch/bulkedit/jobs/TenantResolverTest.java new file mode 100644 index 000000000..651e5354f --- /dev/null +++ b/src/test/java/org/folio/dew/batch/bulkedit/jobs/TenantResolverTest.java @@ -0,0 +1,102 @@ +package org.folio.dew.batch.bulkedit.jobs; + +import static org.folio.dew.utils.Constants.FILE_NAME; +import static org.folio.dew.utils.Constants.NO_HOLDING_AFFILIATION; +import static org.folio.dew.utils.Constants.NO_HOLDING_VIEW_PERMISSIONS; +import static org.folio.dew.utils.Constants.NO_ITEM_AFFILIATION; +import static org.folio.dew.utils.Constants.NO_ITEM_VIEW_PERMISSIONS; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionsValidator; +import org.folio.dew.client.UserClient; +import org.folio.dew.domain.dto.EntityType; +import org.folio.dew.domain.dto.ItemIdentifier; +import org.folio.dew.domain.dto.JobParameterNames; +import org.folio.dew.domain.dto.User; +import org.folio.dew.service.BulkEditProcessingErrorsService; +import org.folio.dew.service.ConsortiaService; +import org.folio.spring.FolioExecutionContext; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.InjectMocks; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.springframework.batch.core.JobExecution; +import org.springframework.batch.core.JobParametersBuilder; + +import java.util.List; +import java.util.Set; +import java.util.UUID; + +@ExtendWith(MockitoExtension.class) +class TenantResolverTest { + + @Mock + private FolioExecutionContext folioExecutionContext; + @Mock + private ConsortiaService consortiaService; + @Mock + private PermissionsValidator permissionsValidator; + @Mock + private BulkEditProcessingErrorsService bulkEditProcessingErrorsService; + @Mock + private UserClient userClient; + @Mock + private JobExecution jobExecution; + + @InjectMocks + private TenantResolver tenantResolver; + + @Test + void testGetAffiliatedPermittedTenantIdsRecords() { + var user = new User(); + user.setUsername("userName"); + var tenantsIds = Set.of("member1", "member2", "member3"); + + var hrid = "hrid"; + var itemIdentifier = new ItemIdentifier(hrid); + + var builder = new JobParametersBuilder(); + var jobId = "jobId"; + builder.addString(JobParameterNames.JOB_ID, jobId); + var fileName = "fileName"; + builder.addString(FILE_NAME, fileName); + var jobParameters = builder.toJobParameters(); + + when(folioExecutionContext.getUserId()).thenReturn(UUID.randomUUID()); + when(userClient.getUserById(any())).thenReturn(user); + when(consortiaService.getAffiliatedTenants(any(), any())).thenReturn(List.of("member2", "member3")); + when(permissionsValidator.isBulkEditReadPermissionExists("member2", EntityType.HOLDINGS_RECORD)).thenReturn(false); + when(permissionsValidator.isBulkEditReadPermissionExists("member3", EntityType.HOLDINGS_RECORD)).thenReturn(true); + when(jobExecution.getJobParameters()).thenReturn(jobParameters); + + var affiliatedAndPermittedTenants = tenantResolver.getAffiliatedPermittedTenantIds(EntityType.HOLDINGS_RECORD, jobExecution, "HRID", tenantsIds, itemIdentifier); + + var expectedAffiliationError = "User userName does not have required affiliation to view the holdings record - hrid=hrid on the tenant member1"; + var expectedPermissionError = "User userName does not have required permission to view the holdings record - hrid=hrid on the tenant member2"; + verify(bulkEditProcessingErrorsService).saveErrorInCSV(jobId, itemIdentifier.getItemId(), expectedAffiliationError, fileName); + verify(bulkEditProcessingErrorsService).saveErrorInCSV(jobId, itemIdentifier.getItemId(), expectedPermissionError, fileName); + + assertEquals(1, affiliatedAndPermittedTenants.size()); + } + + @Test + void testGetAffiliationErrorPlaceholder() { + assertEquals(NO_ITEM_AFFILIATION, tenantResolver.getAffiliationErrorPlaceholder(EntityType.ITEM)); + assertEquals(NO_HOLDING_AFFILIATION, tenantResolver.getAffiliationErrorPlaceholder(EntityType.HOLDINGS_RECORD)); + + assertThrows(UnsupportedOperationException.class, () -> tenantResolver.getAffiliationErrorPlaceholder(EntityType.INSTANCE)); + } + + @Test + void testGetViewPermissionErrorPlaceholder() { + assertEquals(NO_ITEM_VIEW_PERMISSIONS, tenantResolver.getViewPermissionErrorPlaceholder(EntityType.ITEM)); + assertEquals(NO_HOLDING_VIEW_PERMISSIONS, tenantResolver.getViewPermissionErrorPlaceholder(EntityType.HOLDINGS_RECORD)); + + assertThrows(UnsupportedOperationException.class, () -> tenantResolver.getViewPermissionErrorPlaceholder(EntityType.INSTANCE)); + } +} diff --git a/src/test/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/PermissionsValidatorTest.java b/src/test/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/PermissionsValidatorTest.java new file mode 100644 index 000000000..ea3f011ef --- /dev/null +++ b/src/test/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/PermissionsValidatorTest.java @@ -0,0 +1,58 @@ +package org.folio.dew.batch.bulkedit.jobs.permissions.check; + + +import org.folio.dew.domain.dto.EntityType; +import org.folio.spring.FolioExecutionContext; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.InjectMocks; +import org.mockito.Mock; +import org.mockito.Spy; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.util.List; +import java.util.UUID; + +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.BULK_EDIT_INVENTORY_VIEW_PERMISSION; +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.BULK_EDIT_USERS_VIEW_PERMISSION; +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.INVENTORY_ITEMS_ITEM_GET_PERMISSION; +import static org.folio.dew.batch.bulkedit.jobs.permissions.check.PermissionEnum.USER_ITEM_GET_PERMISSION; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class PermissionsValidatorTest { + + @Mock + private PermissionsProvider permissionsProvider; + @Spy + private RequiredPermissionResolver requiredPermissionResolver; + @Mock + private FolioExecutionContext folioExecutionContext; + + @InjectMocks + private PermissionsValidator permissionsValidator; + + @Test + void testIsBulkEditReadPermissionExistsForInventoryRecords() { + when(permissionsProvider.getUserPermissions(eq("tenant1"), any())).thenReturn(List.of(INVENTORY_ITEMS_ITEM_GET_PERMISSION.getValue(), "not_read_permission", BULK_EDIT_INVENTORY_VIEW_PERMISSION.getValue())); + when(permissionsProvider.getUserPermissions(eq("tenant2"), any())).thenReturn(List.of("not_read_permission")); + when(folioExecutionContext.getUserId()).thenReturn(UUID.randomUUID()); + + assertTrue(permissionsValidator.isBulkEditReadPermissionExists("tenant1", EntityType.ITEM)); + assertFalse(permissionsValidator.isBulkEditReadPermissionExists("tenant2", EntityType.ITEM)); + } + + @Test + void testIsBulkEditReadPermissionExistsForUsers() { + when(permissionsProvider.getUserPermissions(eq("tenant1"), any())).thenReturn(List.of(USER_ITEM_GET_PERMISSION.getValue(), "not_read_permission", BULK_EDIT_USERS_VIEW_PERMISSION.getValue(), BULK_EDIT_USERS_VIEW_PERMISSION.getValue())); + when(permissionsProvider.getUserPermissions(eq("tenant2"), any())).thenReturn(List.of("not_read_permission")); + when(folioExecutionContext.getUserId()).thenReturn(UUID.randomUUID()); + + assertTrue(permissionsValidator.isBulkEditReadPermissionExists("tenant1", EntityType.USER)); + assertFalse(permissionsValidator.isBulkEditReadPermissionExists("tenant2", EntityType.USER)); + } +} diff --git a/src/test/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/RequiredPermissionResolverTest.java b/src/test/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/RequiredPermissionResolverTest.java new file mode 100644 index 000000000..bc677dbc2 --- /dev/null +++ b/src/test/java/org/folio/dew/batch/bulkedit/jobs/permissions/check/RequiredPermissionResolverTest.java @@ -0,0 +1,18 @@ +package org.folio.dew.batch.bulkedit.jobs.permissions.check; + +import org.folio.dew.domain.dto.EntityType; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +class RequiredPermissionResolverTest { + + @Test + void testGetReadPermission() { + var requiredPermissionResolver = new RequiredPermissionResolver(); + assertEquals("users.item.get", requiredPermissionResolver.getReadPermission(EntityType.USER).getValue()); + assertEquals("inventory.items.item.get", requiredPermissionResolver.getReadPermission(EntityType.ITEM).getValue()); + assertEquals("inventory-storage.holdings.item.get", requiredPermissionResolver.getReadPermission(EntityType.HOLDINGS_RECORD).getValue()); + assertEquals("inventory.instances.item.get", requiredPermissionResolver.getReadPermission(EntityType.INSTANCE).getValue()); + } +} diff --git a/src/test/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditFilterUserRecordsForRollBackProcessorTest.java b/src/test/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditFilterUserRecordsForRollBackProcessorTest.java deleted file mode 100644 index 211f55693..000000000 --- a/src/test/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditFilterUserRecordsForRollBackProcessorTest.java +++ /dev/null @@ -1,61 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.rollbackjob; - -import org.folio.dew.domain.dto.User; -import org.folio.dew.domain.dto.UserFormat; -import org.folio.dew.service.BulkEditParseService; -import org.folio.dew.service.BulkEditRollBackService; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.extension.ExtendWith; -import org.mockito.InjectMocks; -import org.mockito.Mock; -import org.mockito.junit.jupiter.MockitoExtension; - -import java.util.UUID; - -import static org.junit.jupiter.api.Assertions.assertNotNull; -import static org.junit.jupiter.api.Assertions.assertNull; -import static org.mockito.ArgumentMatchers.isA; -import static org.mockito.Mockito.times; -import static org.mockito.Mockito.verify; -import static org.mockito.Mockito.when; - -@ExtendWith(MockitoExtension.class) -class BulkEditFilterUserRecordsForRollBackProcessorTest { - - @Mock - private BulkEditRollBackService bulkEditRollBackService; - @Mock - private BulkEditParseService bulkEditParseService; - - @InjectMocks - private BulkEditFilterUserRecordsForRollBackProcessor processor; - - @Test - void testWriteIfUserBeRollBack() { - var userFormat = new UserFormat(); - userFormat.setId("userId"); - var user= new User(); - user.setId("userId"); - - when(bulkEditRollBackService.isUserBeRollBack(isA(String.class), isA(UUID.class))).thenReturn(true); - when(bulkEditParseService.mapUserFormatToUser(isA(UserFormat.class))).thenReturn(user); - - processor.setJobId("edd30136-9a7b-4226-9e82-83024dbeac4a"); - var processedUser = processor.process(userFormat); - verify(bulkEditParseService, times(1)).mapUserFormatToUser(isA(UserFormat.class)); - assertNotNull(processedUser); - } - - @Test - void testWriteIfUserNotBeRollBack() { - var userFormat = new UserFormat(); - userFormat.setId("userId"); - - when(bulkEditRollBackService.isUserBeRollBack(isA(String.class), isA(UUID.class))).thenReturn(false); - - processor.setJobId("edd30136-9a7b-4226-9e82-83024dbeac4a"); - var user = processor.process(userFormat); - verify(bulkEditParseService, times(0)).mapUserFormatToUser(isA(UserFormat.class)); - assertNull(user); - } -} diff --git a/src/test/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditUpdateUserRecordsForRollBackCsvWriterTest.java b/src/test/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditUpdateUserRecordsForRollBackCsvWriterTest.java deleted file mode 100644 index ebd4e7354..000000000 --- a/src/test/java/org/folio/dew/batch/bulkedit/jobs/rollbackjob/BulkEditUpdateUserRecordsForRollBackCsvWriterTest.java +++ /dev/null @@ -1,39 +0,0 @@ -package org.folio.dew.batch.bulkedit.jobs.rollbackjob; - -import org.folio.dew.client.UserClient; -import org.folio.dew.domain.dto.User; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.extension.ExtendWith; -import org.mockito.InjectMocks; -import org.mockito.Mock; -import org.mockito.junit.jupiter.MockitoExtension; -import org.springframework.batch.item.Chunk; - -import java.util.List; - -import static org.mockito.ArgumentMatchers.isA; -import static org.mockito.Mockito.doNothing; -import static org.mockito.Mockito.times; -import static org.mockito.Mockito.verify; - -@ExtendWith(MockitoExtension.class) -class BulkEditUpdateUserRecordsForRollBackCsvWriterTest { - - @Mock - private UserClient userClient; - @InjectMocks - private BulkEditUpdateUserRecordsForRollBackWriter writer; - - @Test - void writeTest() throws Exception { - var user1 = new User(); - user1.setId("1"); - var user2 = new User(); - user2.setId("2"); - var users = List.of(user1, user2); - - doNothing().when(userClient).updateUser(isA(User.class), isA(String.class)); - writer.write(new Chunk<>(users)); - verify(userClient, times(2)).updateUser(isA(User.class), isA(String.class)); - } -} diff --git a/src/test/java/org/folio/dew/controller/BulkEditControllerTest.java b/src/test/java/org/folio/dew/controller/BulkEditControllerTest.java index 4f767b208..480936c4e 100644 --- a/src/test/java/org/folio/dew/controller/BulkEditControllerTest.java +++ b/src/test/java/org/folio/dew/controller/BulkEditControllerTest.java @@ -1,38 +1,25 @@ package org.folio.dew.controller; import static java.lang.String.format; -import static org.apache.commons.lang3.StringUtils.EMPTY; -import static org.folio.dew.domain.dto.EntityType.HOLDINGS_RECORD; import static org.folio.dew.domain.dto.EntityType.USER; import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_IDENTIFIERS; -import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_QUERY; import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_UPDATE; import static org.folio.dew.domain.dto.IdentifierType.BARCODE; -import static org.folio.dew.domain.dto.JobParameterNames.PREVIEW_FILE_NAME; import static org.folio.dew.domain.dto.JobParameterNames.TEMP_OUTPUT_FILE_PATH; import static org.folio.dew.domain.dto.JobParameterNames.UPDATED_FILE_NAME; -import static org.folio.dew.domain.dto.UserContentUpdateAction.NameEnum.FIND; -import static org.folio.dew.domain.dto.UserContentUpdateAction.NameEnum.REPLACE_WITH; -import static org.folio.dew.utils.Constants.CSV_EXTENSION; import static org.folio.dew.utils.Constants.FILE_NAME; -import static org.folio.dew.utils.Constants.PATH_SEPARATOR; import static org.folio.dew.utils.Constants.UPDATED_PREFIX; import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.empty; import static org.hamcrest.Matchers.equalTo; -import static org.hamcrest.Matchers.hasSize; import static org.hamcrest.Matchers.is; -import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.hamcrest.Matchers.hasSize; import static org.mockito.ArgumentMatchers.any; -import static org.mockito.Mockito.isA; -import static org.mockito.Mockito.timeout; import static org.mockito.Mockito.times; import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; -import static org.springframework.batch.test.AssertFile.assertFileEquals; import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get; import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.multipart; -import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post; import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.content; import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status; @@ -40,10 +27,6 @@ import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; -import java.nio.file.Paths; -import java.util.Collection; -import java.util.Date; -import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import java.util.Optional; @@ -54,129 +37,46 @@ import org.folio.de.entity.JobCommand; import org.folio.de.entity.JobCommandType; import org.folio.dew.BaseBatchTest; -import org.folio.dew.client.ConfigurationClient; -import org.folio.dew.client.GroupClient; -import org.folio.dew.client.InventoryClient; import org.folio.dew.client.UserClient; import org.folio.dew.domain.dto.EntityType; +import org.folio.dew.domain.dto.Errors; import org.folio.dew.domain.dto.ExportType; -import org.folio.dew.domain.dto.HoldingsRecordCollection; import org.folio.dew.domain.dto.IdentifierType; -import org.folio.dew.domain.dto.Item; -import org.folio.dew.domain.dto.ItemCollection; -import org.folio.dew.domain.dto.JobParameterNames; import org.folio.dew.domain.dto.Metadata; import org.folio.dew.domain.dto.Personal; import org.folio.dew.domain.dto.User; -import org.folio.dew.domain.dto.UserCollection; -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.domain.dto.UserContentUpdateAction; -import org.folio.dew.domain.dto.UserContentUpdateCollection; -import org.folio.dew.domain.dto.UserGroup; -import org.folio.dew.domain.dto.UserGroupCollection; import org.folio.dew.error.BulkEditException; import org.folio.dew.error.FileOperationException; -import org.folio.dew.repository.JobCommandRepository; import org.folio.dew.repository.LocalFilesStorage; -import org.folio.dew.repository.RemoteFilesStorage; import org.folio.dew.service.BulkEditProcessingErrorsService; -import org.folio.dew.service.BulkEditRollBackService; import org.folio.dew.service.JobCommandsReceiverService; -import org.folio.spring.DefaultFolioExecutionContext; -import org.folio.spring.FolioExecutionContext; -import org.folio.spring.FolioModuleMetadata; -import org.folio.spring.integration.XOkapiHeaders; -import org.folio.spring.scope.FolioExecutionContextSetter; -import org.folio.tenant.domain.dto.Errors; -import org.junit.jupiter.api.Disabled; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; -import org.junit.jupiter.params.ParameterizedTest; -import org.junit.jupiter.params.provider.CsvSource; -import org.junit.jupiter.params.provider.EnumSource; -import org.junit.jupiter.params.provider.ValueSource; -import org.mockito.ArgumentCaptor; -import org.springframework.batch.core.JobExecution; -import org.springframework.batch.core.JobExecutionException; -import org.springframework.batch.core.JobParameters; import org.springframework.batch.core.JobParametersBuilder; -import org.springframework.batch.integration.launch.JobLaunchRequest; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.mock.mockito.MockBean; -import org.springframework.core.io.FileSystemResource; import org.springframework.http.MediaType; import org.springframework.mock.web.MockMultipartFile; class BulkEditControllerTest extends BaseBatchTest { - private static final String UPLOAD_URL_TEMPLATE = "/bulk-edit/%s/upload"; - private static final String START_URL_TEMPLATE = "/bulk-edit/%s/start"; - private static final String PREVIEW_USERS_URL_TEMPLATE = "/bulk-edit/%s/preview/users"; - private static final String PREVIEW_ITEMS_URL_TEMPLATE = "/bulk-edit/%s/preview/items"; - private static final String PREVIEW_HOLDINGS_RECORD_URL_TEMPLATE = "/bulk-edit/%s/preview/holdings"; private static final String ERRORS_URL_TEMPLATE = "/bulk-edit/%s/errors"; - private static final String ITEMS_CONTENT_UPDATE_UPLOAD_URL_TEMPLATE = "/bulk-edit/%s/item-content-update/upload"; - private static final String USERS_CONTENT_UPDATE_UPLOAD_URL_TEMPLATE = "/bulk-edit/%s/user-content-update/upload"; - private static final String HOLDINGS_CONTENT_UPDATE_UPLOAD_URL_TEMPLATE = "/bulk-edit/%s/holdings-content-update/upload"; - private static final String ITEMS_CONTENT_PREVIEW_DOWNLOAD_URL_TEMPLATE = "/bulk-edit/%s/preview/updated-items/download"; - private static final String USERS_CONTENT_PREVIEW_DOWNLOAD_URL_TEMPLATE = "/bulk-edit/%s/preview/updated-users/download"; - private static final String ITEMS_FOR_LOCATION_UPDATE = "src/test/resources/upload/bulk_edit_items_for_location_update.csv"; - private static final String ITEMS_FOR_LOAN_TYPE_UPDATE = "src/test/resources/upload/bulk_edit_items_for_loan_type_update.csv"; - private static final String ITEMS_FOR_STATUS_UPDATE = "src/test/resources/upload/bulk_edit_items_for_status_update.csv"; - private static final String ITEM_FOR_STATUS_UPDATE_ERROR = "src/test/resources/upload/bulk_edit_item_for_status_update_error.csv"; - private static final String ITEMS_FOR_NOTHING_UPDATE = "src/test/resources/upload/bulk_edit_items_for_nothing_update.csv"; + private static final String UPLOAD_URL_TEMPLATE = "/bulk-edit/%s/upload"; private static final String USER_DATA = "src/test/resources/upload/user_data.csv"; private static final String ITEM_DATA = "src/test/resources/upload/item_data.csv"; - private static final String PREVIEW_USER_DATA = "src/test/resources/upload/preview_user_data.csv"; - private static final String PREVIEW_ITEM_DATA = "src/test/resources/upload/preview_item_data.csv"; - private static final String PREVIEW_HOLDINGS_RECORD_DATA = "src/test/resources/upload/preview_holdings_record_data.csv"; - private static final String EXPECTED_ERRORS_FOR_CLEAR_PATRON_GROUP = "src/test/resources/output/expected_errors_for_clear_patron_group.json"; - private static final String EXPECTED_USER_CONTENT_UPDATE_OUTPUT = "src/test/resources/output/bulk_edit_user_content_updates_expected_output.csv"; - private static final String HOLDINGS_RECORDS_FOR_UPDATE = "src/test/resources/output/bulk_edit_holdings_records_output.csv"; - private static final String UPDATED_HOLDINGS_RECORDS_JSON = "src/test/resources/output/bulk_edit_updated_holdings_records_output.json"; - private static final UUID JOB_ID = UUID.randomUUID(); public static final String LIMIT = "limit"; - - @MockBean - private BulkEditRollBackService bulkEditRollBackService; + private static final UUID JOB_ID = UUID.randomUUID(); @MockBean private UserClient userClient; - @MockBean - private InventoryClient inventoryClient; - - @MockBean - private JobCommandRepository jobCommandRepository; - @MockBean private JobCommandsReceiverService jobCommandsReceiverService; - @MockBean - private ConfigurationClient configurationClient; - - @Autowired - private BulkEditProcessingErrorsService bulkEditProcessingErrorsService; - - @Autowired - private RemoteFilesStorage remoteFilesStorage; - @Autowired private LocalFilesStorage localFilesStorage; @Autowired - private BulkEditProcessingErrorsService errorsService; - - @Autowired - private org.springframework.batch.core.Job bulkEditProcessUserIdentifiersJob; - - @Autowired - private FolioModuleMetadata folioModuleMetadata; - - @Autowired - private FolioExecutionContext folioExecutionContext; - - @MockBean - private GroupClient groupClient; + private BulkEditProcessingErrorsService bulkEditProcessingErrorsService; @Test void shouldReturnErrorsPreview() throws Exception { @@ -214,8 +114,6 @@ void shouldReturnEmptyErrorsForErrorsPreview() throws Exception { var jobCommand = createBulkEditJobRequest(jobId, BULK_EDIT_IDENTIFIERS, USER, BARCODE); when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - var expectedErrorMsg = format("errors file for job id %s", jobId); - var headers = defaultHeaders(); var response = mockMvc.perform(get(format(ERRORS_URL_TEMPLATE, jobId)) @@ -244,285 +142,6 @@ void shouldReturnErrorsFileNotFoundErrorForErrorsPreview() throws Exception { .andExpect(content().json(expectedJson)); } - @SneakyThrows - @Test - void shouldReturnCompleteUserPreviewForQuery() { - var query = "(patronGroup==\"3684a786-6671-4268-8ed0-9db82ebca60b\") sortby personal.lastName"; - when(userClient.getUserByQuery(query, 3)).thenReturn(buildUserCollection()); - - var jobId = UUID.randomUUID(); - var jobCommand = createBulkEditJobRequest(jobId, BULK_EDIT_QUERY, USER, BARCODE); - - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - - var headers = defaultHeaders(); - - var response = mockMvc.perform(get(format(PREVIEW_USERS_URL_TEMPLATE, jobId)) - .headers(headers) - .queryParam(LIMIT, String.valueOf(3))) - .andExpect(status().isOk()); - - var users = objectMapper.readValue(response.andReturn().getResponse().getContentAsString(), UserCollection.class); - assertThat(users.getTotalRecords(), equalTo(3)); - assertThat(users.getUsers(), hasSize(3)); - } - - @Disabled - // TODO uncomment when resolved - @SneakyThrows - @ParameterizedTest - @EnumSource(value = IdentifierType.class, - names = {"ID", "BARCODE", "EXTERNAL_SYSTEM_ID", "USER_NAME"}, - mode = EnumSource.Mode.INCLUDE) - void shouldReturnCompleteUserPreviewForAnyIdentifier(IdentifierType identifierType) { - when(groupClient.getGroupByQuery("group==\"PatronGroup\"")) - .thenReturn(new UserGroupCollection().usergroups(List.of(new UserGroup().group("PatronGroup") - .desc("Staff Member").id("3684a786-6671-4268-8ed0-9db82ebca60b").expirationOffsetInDays(730))).totalRecords(1)); - remoteFilesStorage.upload(FilenameUtils.getName(PREVIEW_USER_DATA), PREVIEW_USER_DATA); - var jobId = UUID.randomUUID(); - var jobCommand = new JobCommand(); - jobCommand.setId(jobId); - jobCommand.setExportType(BULK_EDIT_IDENTIFIERS); - jobCommand.setEntityType(USER); - jobCommand.setIdentifierType(identifierType); - jobCommand.setJobParameters(new JobParametersBuilder() - .addString(TEMP_OUTPUT_FILE_PATH, "test/path/" + PREVIEW_USER_DATA.replace(CSV_EXTENSION, EMPTY)) - .toJobParameters()); - - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - - var headers = defaultHeaders(); - var response = mockMvc.perform(get(format(PREVIEW_USERS_URL_TEMPLATE, jobId)) - .headers(headers) - .queryParam(LIMIT, String.valueOf(3))) - .andExpect(status().isOk()); - - var users = objectMapper.readValue(response.andReturn().getResponse().getContentAsString(), UserCollection.class); - assertThat(users.getTotalRecords(), equalTo(3)); - assertThat(users.getUsers(), hasSize(3)); - } - - @ParameterizedTest - @EnumSource(value = IdentifierType.class, names = {"ID", "HOLDINGS_RECORD_ID", "INSTANCE_HRID", "ITEM_BARCODE"}) - @SneakyThrows - void shouldReturnCompleteHoldingsRecordPreview(IdentifierType identifierType) { - var jobId = UUID.randomUUID(); - remoteFilesStorage.upload(jobId + PATH_SEPARATOR + FilenameUtils.getName(PREVIEW_HOLDINGS_RECORD_DATA), PREVIEW_HOLDINGS_RECORD_DATA); - - var jobCommand = new JobCommand(); - jobCommand.setId(jobId); - jobCommand.setExportType(BULK_EDIT_IDENTIFIERS); - jobCommand.setEntityType(HOLDINGS_RECORD); - jobCommand.setIdentifierType(identifierType); - jobCommand.setJobParameters(new JobParametersBuilder() - .addString(TEMP_OUTPUT_FILE_PATH, "test/path/" + PREVIEW_HOLDINGS_RECORD_DATA.replace(CSV_EXTENSION, EMPTY)) - .toJobParameters()); - - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - - var headers = defaultHeaders(); - var response = mockMvc.perform(get(format(PREVIEW_HOLDINGS_RECORD_URL_TEMPLATE, jobId)) - .headers(headers) - .queryParam(LIMIT, String.valueOf(1))) - .andExpect(status().isOk()); - - var holdingsRecord = objectMapper.readValue(response.andReturn().getResponse().getContentAsString(), HoldingsRecordCollection.class); - assertThat(holdingsRecord.getTotalRecords(), equalTo(1)); - assertThat(holdingsRecord.getHoldingsRecords(), hasSize(1)); - } - - @ParameterizedTest - @EnumSource(value = IdentifierType.class, names = {"ID", "HOLDINGS_RECORD_ID", "INSTANCE_HRID", "ITEM_BARCODE"}) - @SneakyThrows - void shouldReturnChangedHoldingsRecordPreview(IdentifierType identifierType) { - var jobId = UUID.randomUUID(); - remoteFilesStorage.upload(jobId + PATH_SEPARATOR + FilenameUtils.getName(PREVIEW_HOLDINGS_RECORD_DATA), PREVIEW_HOLDINGS_RECORD_DATA); - - var jobCommand = new JobCommand(); - jobCommand.setId(jobId); - jobCommand.setExportType(BULK_EDIT_UPDATE); - jobCommand.setEntityType(HOLDINGS_RECORD); - jobCommand.setIdentifierType(identifierType); - jobCommand.setJobParameters(new JobParametersBuilder() - .addString(UPDATED_FILE_NAME, jobId + PATH_SEPARATOR + FilenameUtils.getName(PREVIEW_HOLDINGS_RECORD_DATA).replace(CSV_EXTENSION, EMPTY)) - .toJobParameters()); - - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - - var headers = defaultHeaders(); - var response = mockMvc.perform(get(format(PREVIEW_HOLDINGS_RECORD_URL_TEMPLATE, jobId)) - .headers(headers) - .queryParam(LIMIT, String.valueOf(1))) - .andExpect(status().isOk()); - - var holdingsRecord = objectMapper.readValue(response.andReturn().getResponse().getContentAsString(), HoldingsRecordCollection.class); - assertThat(holdingsRecord.getTotalRecords(), equalTo(1)); - assertThat(holdingsRecord.getHoldingsRecords(), hasSize(1)); - } - - @SneakyThrows - @ParameterizedTest - @EnumSource(value = IdentifierType.class, - names = {"ID", "BARCODE", "EXTERNAL_SYSTEM_ID", "USER_NAME"}, - mode = EnumSource.Mode.INCLUDE) - void shouldReturnEmptyUserPreviewIfNoRecordsAvailable(IdentifierType identifierType) { - var jobId = UUID.randomUUID(); - var jobCommand = new JobCommand(); - jobCommand.setId(jobId); - jobCommand.setExportType(BULK_EDIT_IDENTIFIERS); - jobCommand.setEntityType(USER); - jobCommand.setIdentifierType(identifierType); - jobCommand.setJobParameters(new JobParametersBuilder() - .addString(TEMP_OUTPUT_FILE_PATH, "test/path/no_file") - .toJobParameters()); - - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - - var headers = defaultHeaders(); - var response = mockMvc.perform(get(format(PREVIEW_USERS_URL_TEMPLATE, jobId)) - .headers(headers) - .queryParam(LIMIT, String.valueOf(3))) - .andExpect(status().isOk()); - - var users = objectMapper.readValue(response.andReturn().getResponse().getContentAsString(), UserCollection.class); - assertThat(users.getTotalRecords(), equalTo(0)); - assertThat(users.getUsers(), hasSize(0)); - } - - @ParameterizedTest - @EnumSource(value = IdentifierType.class, names = {"EXTERNAL_SYSTEM_ID", "USER_NAME"}, mode = EnumSource.Mode.EXCLUDE) - @SneakyThrows - void shouldReturnEmptyItemPreviewIfNoRecordsAvailable(IdentifierType identifierType) { - var jobId = UUID.randomUUID(); - var jobCommand = new JobCommand(); - jobCommand.setId(jobId); - jobCommand.setExportType(BULK_EDIT_IDENTIFIERS); - jobCommand.setEntityType(USER); - jobCommand.setIdentifierType(identifierType); - jobCommand.setJobParameters(new JobParametersBuilder() - .addString(TEMP_OUTPUT_FILE_PATH, "test/path/no_file") - .toJobParameters()); - - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - - var headers = defaultHeaders(); - var response = mockMvc.perform(get(format(PREVIEW_ITEMS_URL_TEMPLATE, jobId)) - .headers(headers) - .queryParam(LIMIT, String.valueOf(3))) - .andExpect(status().isOk()); - - var items = objectMapper.readValue(response.andReturn().getResponse().getContentAsString(), ItemCollection.class); - assertThat(items.getTotalRecords(), equalTo(0)); - assertThat(items.getItems(), hasSize(0)); - } - - @ParameterizedTest - @EnumSource(value = IdentifierType.class, names = {"ID", "HOLDINGS_RECORD_ID", "INSTANCE_HRID", "ITEM_BARCODE"}) - @SneakyThrows - void shouldReturnEmptyHoldingsRecordPreviewIfNoRecordsAvailable(IdentifierType identifierType) { - var jobId = UUID.randomUUID(); - var jobCommand = new JobCommand(); - jobCommand.setId(jobId); - jobCommand.setExportType(BULK_EDIT_IDENTIFIERS); - jobCommand.setEntityType(HOLDINGS_RECORD); - jobCommand.setIdentifierType(identifierType); - jobCommand.setJobParameters(new JobParametersBuilder() - .addString(TEMP_OUTPUT_FILE_PATH, "test/path/no_file") - .toJobParameters()); - - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - - var headers = defaultHeaders(); - var response = mockMvc.perform(get(format(PREVIEW_HOLDINGS_RECORD_URL_TEMPLATE, jobId)) - .headers(headers) - .queryParam(LIMIT, String.valueOf(1))) - .andExpect(status().isOk()); - - var holdingsRecord = objectMapper.readValue(response.andReturn().getResponse().getContentAsString(), HoldingsRecordCollection.class); - assertThat(holdingsRecord.getTotalRecords(), equalTo(0)); - assertThat(holdingsRecord.getHoldingsRecords(), hasSize(0)); - } - - @SneakyThrows - @ParameterizedTest - @CsvSource({"BULK_EDIT_UPDATE,barcode==(\"123\" OR \"456\")", - "BULK_EDIT_QUERY,(patronGroup==\"3684a786-6671-4268-8ed0-9db82ebca60b\") sortby personal.lastName"}) - void shouldReturnCompleteUserPreviewWithLimitControl(String exportType, String query) { - - var jobId = UUID.randomUUID(); - var jobCommand = createBulkEditJobRequest(jobId, ExportType.fromValue(exportType), USER, BARCODE); - - when(userClient.getUserByQuery(query, 2)).thenReturn(buildUserCollection()); - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - - var headers = defaultHeaders(); - - var response = mockMvc.perform(get(format(PREVIEW_USERS_URL_TEMPLATE, jobId)) - .headers(headers) - .queryParam(LIMIT, String.valueOf(2))) - .andExpect(status().isOk()); - - ArgumentCaptor queryCaptor = ArgumentCaptor.forClass(String.class); - ArgumentCaptor limitCaptor = ArgumentCaptor.forClass(Long.class); - verify(userClient).getUserByQuery(queryCaptor.capture(), limitCaptor.capture()); - assertThat(query, equalTo(queryCaptor.getValue())); - assertThat(2L, equalTo(limitCaptor.getValue())); - } - - @Disabled("The source code is unsupported, so the test is invalid.") - @SneakyThrows - @ParameterizedTest - @EnumSource(value = EntityType.class, names = {"USER", "ITEM"}, mode = EnumSource.Mode.INCLUDE) - void shouldReturnCompleteUpdatePreviewWithLimitControl(EntityType entityType) { - var query = "barcode==(\"123\" OR \"456\")"; - var jobId = UUID.randomUUID(); - var jobCommand = createBulkEditJobRequest(jobId, BULK_EDIT_UPDATE, entityType, BARCODE); - when(inventoryClient.getItemByQuery(query, 2)).thenReturn(buildItemCollection()); - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - - var headers = defaultHeaders(); - - var response = mockMvc.perform(get(format(PREVIEW_ITEMS_URL_TEMPLATE, jobId)) - .headers(headers) - .queryParam(LIMIT, String.valueOf(2))) - .andExpect(status().isOk()); - - ArgumentCaptor queryCaptor = ArgumentCaptor.forClass(String.class); - ArgumentCaptor limitCaptor = ArgumentCaptor.forClass(Long.class); - verify(inventoryClient).getItemByQuery(queryCaptor.capture(), limitCaptor.capture()); - assertThat(query, equalTo(queryCaptor.getValue())); - assertThat(2L, equalTo(limitCaptor.getValue())); - } - - @SneakyThrows - @Test - void shouldReturnErrorForInvalidExportType() { - - var jobId = UUID.randomUUID(); - - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())) - .thenReturn(Optional.of(createBulkEditJobRequest(jobId, ExportType.ORDERS_EXPORT, USER, BARCODE))); - - var headers = defaultHeaders(); - - mockMvc.perform(get(format(PREVIEW_USERS_URL_TEMPLATE, jobId)) - .headers(headers) - .queryParam(LIMIT, String.valueOf(2))) - .andExpect(status().isBadRequest()); - } - - @SneakyThrows - @Test - void shouldReturnErrorJobNotFound() { - - var headers = defaultHeaders(); - - mockMvc.perform(get(format(PREVIEW_USERS_URL_TEMPLATE, UUID.randomUUID())) - .headers(headers) - .queryParam(LIMIT, String.valueOf(2))) - .andExpect(status().isNotFound()); - } - @Test @DisplayName("Launch job on upload file with identifiers successfully") @SneakyThrows @@ -548,27 +167,6 @@ void shouldLaunchJobAndReturnNumberOfRecordsOnIdentifiersFileUpload() { verify(exportJobManagerSync, times(1)).launchJob(any()); } - @Test - @DisplayName("Skip headers while counting records for update") - @SneakyThrows - void shouldSkipHeadersWhileCountingRecordsForUpdate() { - var jobId = UUID.randomUUID(); - var jobCommand = createBulkEditJobRequest(jobId, ExportType.BULK_EDIT_UPDATE, USER, BARCODE); - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - - var headers = defaultHeaders(); - - var bytes = new FileInputStream("src/test/resources/upload/bulk_edit_user_record.csv").readAllBytes(); - var file = new MockMultipartFile("file", "bulk_edit_user_record.csv", MediaType.TEXT_PLAIN_VALUE, bytes); - - var result = mockMvc.perform(multipart(format(UPLOAD_URL_TEMPLATE, jobId)) - .file(file) - .headers(headers)) - .andExpect(status().isOk()) - .andReturn(); - assertThat(result.getResponse().getContentAsString(), equalTo("2")); - } - @Test @DisplayName("Upload empty file - BAD REQUEST") @SneakyThrows @@ -603,113 +201,6 @@ void shouldReturnNotFoundIfJobDoesNotExist() { .andExpect(status().isNotFound()); } - @Test - @DisplayName("Start update job test") - @SneakyThrows - void startUpdateJobTest() { - var jobId = UUID.fromString("edd30136-9a7b-4226-9e82-83024dbeac4a"); - var jobCommand = new JobCommand(); - jobCommand.setExportType(ExportType.BULK_EDIT_UPDATE); - jobCommand.setEntityType(USER); - jobCommand.setJobParameters(new JobParametersBuilder().toJobParameters()); - var executionId = 0l; - var jobExecution = new JobExecution(executionId); - - var headers = defaultHeaders(); - - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - when(exportJobManagerSync.launchJob(isA(JobLaunchRequest.class))).thenReturn(jobExecution); - - mockMvc.perform(multipart(format(START_URL_TEMPLATE, jobId)) - .headers(headers)) - .andExpect(status().isOk()); - - verify(jobCommandsReceiverService, times(1)).getBulkEditJobCommandById(jobId.toString()); - verify(exportJobManagerSync, timeout(1000).times(1)).launchJob(isA(JobLaunchRequest.class)); - } - - @Test - @DisplayName("Job doesn't exist - NOT FOUND") - @SneakyThrows - void startUpdateJobReturnNotFoundTest() { - var jobId = UUID.fromString("edd30136-9a7b-4226-9e82-83024dbeac4a"); - var headers = defaultHeaders(); - - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.empty()); - - mockMvc.perform(multipart(format(START_URL_TEMPLATE, jobId)) - .headers(headers)) - .andExpect(status().isNotFound()); - } - - @Test - @DisplayName("Start update job - INTERNAL SERVER ERROR") - @SneakyThrows - void startUpdateJobReturnInternalServerErrorTest() { - var jobId = UUID.fromString("edd30136-9a7b-4226-9e82-83024dbeac4a"); - var jobCommand = new JobCommand(); - jobCommand.setExportType(ExportType.BULK_EDIT_UPDATE); - jobCommand.setEntityType(USER); - jobCommand.setJobParameters(new JobParametersBuilder().toJobParameters()); - - var headers = defaultHeaders(); - - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - when(exportJobManagerSync.launchJob(isA(JobLaunchRequest.class))).thenThrow(new JobExecutionException("Execution exception")); - - mockMvc.perform(multipart(format(START_URL_TEMPLATE, jobId)) - .headers(headers)) - .andExpect(status().isOk()); - } - - @ParameterizedTest - @ValueSource(strings = {"/bulk-edit/%s/preview/updated-users/download", "/bulk-edit/%s/preview/updated-holdings/download"}) - @SneakyThrows - @DisplayName("Download users/holdings preview when preview is not available - NOT FOUND") - void shouldReturnNotFoundIfPreviewIsNotAvailable(String urlTemplate) { - var jobCommand = new JobCommand(); - var jobId = UUID.randomUUID(); - jobCommand.setId(jobId); - jobCommand.setJobParameters(new JobParameters()); - - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - - mockMvc.perform(get(format(urlTemplate, jobId)) - .headers(defaultHeaders())) - .andExpect(status().isNotFound()); - } - - @ParameterizedTest - @CsvSource({"src/test/resources/upload/preview_user_data.csv,/bulk-edit/%s/preview/updated-users/download", - "src/test/resources/output/bulk_edit_holdings_records_output.csv,/bulk-edit/%s/preview/updated-holdings/download"}) - @SneakyThrows - @DisplayName("Download users/holdings preview - successful") - void shouldDownloadPreviewAfterUserContentUpdate(String previewPath, String urlTemplate) { - var fileName = FilenameUtils.getName(previewPath); - remoteFilesStorage.upload(fileName, previewPath); - var jobCommand = new JobCommand(); - var jobId = UUID.randomUUID(); - jobCommand.setId(jobId); - jobCommand.setJobParameters(new JobParametersBuilder().addString(PREVIEW_FILE_NAME, fileName).toJobParameters()); - - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - - var response = mockMvc.perform(get(format(urlTemplate, jobId)) - .headers(defaultHeaders())) - .andExpect(status().isOk()) - .andReturn(); - - var expectedCsv = new FileSystemResource(previewPath); - var actualCsvByteArr = response.getResponse().getContentAsByteArray(); - Path actualDownloadedCsvTmp = Paths.get("actualDownloaded.csv"); - Files.write(actualDownloadedCsvTmp, actualCsvByteArr); - var actualCsv = new FileSystemResource(actualDownloadedCsvTmp); - - assertFileEquals(expectedCsv, actualCsv); - - Files.delete(actualDownloadedCsvTmp); - } - @Test @SneakyThrows void shouldReturnNumberOfRowsInCSVFile() { @@ -747,151 +238,6 @@ void shouldReturnNumberOfRowsInCSVFile() { // 3 lines loaded (+1 header because of BULK_EDIT_IDENTIFIERS job). assertThat(result.getResponse().getContentAsString(), equalTo("4")); - - jobId = UUID.randomUUID(); - var jobCommand2 = createBulkEditJobRequest(jobId, BULK_EDIT_UPDATE, USER, BARCODE); - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand2)); - - headers = defaultHeaders(); - - bytes = new FileInputStream("src/test/resources/upload/bulk_edit_user_record_3_lines_edited_1_line.csv").readAllBytes(); - file = new MockMultipartFile("file", "bulk_edit_user_record_3_lines_edited_1_line.csv", MediaType.TEXT_PLAIN_VALUE, bytes); - - result = mockMvc.perform(multipart(format(UPLOAD_URL_TEMPLATE, jobId)) - .file(file) - .headers(headers)) - .andExpect(status().isOk()) - .andReturn(); - - // Edited only 1 line. - assertThat(result.getResponse().getContentAsString(), equalTo("3")); - - jobId = UUID.randomUUID(); - var jobCommand3 = createBulkEditJobRequest(jobId, BULK_EDIT_UPDATE, USER, BARCODE); - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand3)); - - headers = defaultHeaders(); - - // Load initial file with no edited lines. - bytes = new FileInputStream("src/test/resources/upload/bulk_edit_user_record_3_lines.csv").readAllBytes(); - file = new MockMultipartFile("file", "bulk_edit_user_record_3_lines.csv", MediaType.TEXT_PLAIN_VALUE, bytes); - - result = mockMvc.perform(multipart(format(UPLOAD_URL_TEMPLATE, jobId)) - .file(file) - .headers(headers)) - .andExpect(status().isOk()) - .andReturn(); - - // Edited 0 lines. - assertThat(result.getResponse().getContentAsString(), equalTo("3")); - - jobId = UUID.randomUUID(); - var jobCommand4 = createBulkEditJobRequest(jobId, BULK_EDIT_UPDATE, USER, BARCODE); - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand4)); - - headers = defaultHeaders(); - - // Load edited file with incorrect number of tokens. - bytes = new FileInputStream("src/test/resources/upload/invalid-user-records-incorrect-number-of-tokens.csv").readAllBytes(); - file = new MockMultipartFile("file", "invalid-user-records-incorrect-number-of-tokens.csv", MediaType.TEXT_PLAIN_VALUE, bytes); - - result = mockMvc.perform(multipart(format(UPLOAD_URL_TEMPLATE, jobId)) - .file(file) - .headers(headers)) - .andExpect(status().isOk()) - .andReturn(); - - // Keep all 3 lines to delegate them into SkipListener. - assertThat(result.getResponse().getContentAsString(), equalTo("3")); - } - - @Test - @DisplayName("Post user content update to replace email") - @SneakyThrows - void shouldReplaceEmailAddress() { - when(userClient.getUserByQuery("barcode==\"123\"", 1)) - .thenReturn(new UserCollection().addUsersItem(new User().barcode("123").active(true).personal(new Personal().email("123@example1.com")) - .expirationDate(new Date()).patronGroup("3684a786-6671-4268-8ed0-9db82ebca60b")).totalRecords(1)); - when(userClient.getUserByQuery("barcode==\"456\"", 1)) - .thenReturn(new UserCollection().addUsersItem(new User().barcode("456").active(true).personal(new Personal().email("456@example2.com")) - .expirationDate(new Date()).patronGroup("3684a786-6671-4268-8ed0-9db82ebca60b")).totalRecords(1)); - when(userClient.getUserByQuery("barcode==\"789\"", 1)) - .thenReturn(new UserCollection().addUsersItem(new User().barcode("789").active(true).personal(new Personal().email("789@example3.com")) - .expirationDate(new Date()).patronGroup("3684a786-6671-4268-8ed0-9db82ebca60b")).totalRecords(1)); - UserGroup userGroup; - when(groupClient.getGroupById("3684a786-6671-4268-8ed0-9db82ebca60b")) - .thenReturn(userGroup = new UserGroup().group("some group").id("3684a786-6671-4268-8ed0-9db82ebca60b")); - when(groupClient.getGroupByQuery("group==\"some group\"")) - .thenReturn(new UserGroupCollection().usergroups(List.of(userGroup))); - - var jobId = UUID.randomUUID(); - var jobCommand = new JobCommand(); - jobCommand.setId(jobId); - jobCommand.setExportType(BULK_EDIT_IDENTIFIERS); - jobCommand.setEntityType(USER); - jobCommand.setIdentifierType(BARCODE); - jobCommand.setJobParameters(new JobParametersBuilder().addString(JobParameterNames.JOB_ID, jobId.toString()).toJobParameters()); - - when(jobCommandsReceiverService.getBulkEditJobCommandById(jobId.toString())).thenReturn(Optional.of(jobCommand)); - - var bytes = new FileInputStream("src/test/resources/upload/barcodes.csv").readAllBytes(); - var file = new MockMultipartFile("file", "barcodes.csv", MediaType.TEXT_PLAIN_VALUE, bytes); - - var responseUpload = mockMvc.perform(multipart(format(UPLOAD_URL_TEMPLATE, jobId)) - .file(file) - .headers(defaultHeaders())) - .andExpect(status().isOk()) - .andReturn(); - - Map> okapiHeaders = new LinkedHashMap<>(); - okapiHeaders.put(XOkapiHeaders.TENANT, List.of(TENANT)); - var defaultFolioExecutionContext = new DefaultFolioExecutionContext(folioModuleMetadata, okapiHeaders); - try (var context = new FolioExecutionContextSetter(defaultFolioExecutionContext)) { - - createTestLauncher(bulkEditProcessUserIdentifiersJob).launchJob(jobCommand.getJobParameters()); - - assertThat(responseUpload.getResponse().getContentAsString(), equalTo("3")); - - var updates = objectMapper.writeValueAsString(new UserContentUpdateCollection() - .userContentUpdates(List.of(new UserContentUpdate() - .option(UserContentUpdate.OptionEnum.EMAIL_ADDRESS) - .actions(List.of( - new UserContentUpdateAction().name(FIND).value("example1.com"), - new UserContentUpdateAction().name(REPLACE_WITH).value("NEWexample1.com"))), - new UserContentUpdate() - .option(UserContentUpdate.OptionEnum.EMAIL_ADDRESS) - .actions(List.of( - new UserContentUpdateAction().name(FIND).value("example2.com"), - new UserContentUpdateAction().name(REPLACE_WITH).value("NEWexample2.com"))), - new UserContentUpdate() - .option(UserContentUpdate.OptionEnum.EMAIL_ADDRESS) - .actions(List.of( - new UserContentUpdateAction().name(FIND).value("example3.com"), - new UserContentUpdateAction().name(REPLACE_WITH).value("NEWexample3.com"))))) - .totalRecords(3)); - - var responseContentUpdateUpload = mockMvc.perform(post(format(USERS_CONTENT_UPDATE_UPLOAD_URL_TEMPLATE, jobId)) - .headers(defaultHeaders()) - .content(updates)) - .andExpect(status().isOk()) - .andReturn(); - var actualUsers = objectMapper.readValue(responseContentUpdateUpload.getResponse().getContentAsString(), - UserCollection.class); - actualUsers.getUsers().forEach(u -> { - if (u.getBarcode().equals("123")) { - assertEquals("123@NEWexample1.com", u.getPersonal().getEmail()); - } else if (u.getBarcode().equals("456")) { - assertEquals("456@NEWexample2.com", u.getPersonal().getEmail()); - } else if (u.getBarcode().equals("789")) { - assertEquals("789@NEWexample3.com", u.getPersonal().getEmail()); - } - }); - mockMvc.perform(get(format(PREVIEW_USERS_URL_TEMPLATE, jobId)) - .headers(defaultHeaders()) - .queryParam(LIMIT, String.valueOf(10))) - .andExpect(status().isOk()) - .andReturn(); - } } private JobCommand createBulkEditJobRequest(UUID id, ExportType exportType, EntityType entityType, IdentifierType identifierType) { @@ -936,57 +282,4 @@ private JobCommand createBulkEditJobRequest(UUID id, ExportType exportType, Enti return jobCommand; } - private UserCollection buildUserCollection() { - return new UserCollection() - .addUsersItem(new User().barcode("123").patronGroup("3684a786-6671-4268-8ed0-9db82ebca60b")) - .addUsersItem(new User().barcode("456").patronGroup("3684a786-6671-4268-8ed0-9db82ebca60b")) - .addUsersItem(new User().barcode("789").patronGroup("3684a786-6671-4268-8ed0-9db82ebca60b")) - .totalRecords(3); - } - - private ItemCollection buildItemCollection() { - return new ItemCollection() - .addItemsItem(new Item().barcode("123")) - .addItemsItem(new Item().barcode("456")) - .addItemsItem(new Item().barcode("789")) - .totalRecords(3); - } - - private void verifyLocationUpdate(ItemCollection expectedItems, ItemCollection actualItems) { - assertThat(expectedItems.getItems(), hasSize(actualItems.getItems().size())); - for (int i = 0; i < expectedItems.getItems().size(); i++) { - assertThat(expectedItems.getItems().get(i).getId(), equalTo(actualItems.getItems().get(i).getId())); - assertThat(expectedItems.getItems().get(i).getPermanentLocation(), equalTo(actualItems.getItems().get(i).getPermanentLocation())); - assertThat(expectedItems.getItems().get(i).getTemporaryLocation(), equalTo(actualItems.getItems().get(i).getTemporaryLocation())); - assertThat(expectedItems.getItems().get(i).getEffectiveLocation(), equalTo(actualItems.getItems().get(i).getEffectiveLocation())); - } - } - - private void verifyLoanTypeUpdate(ItemCollection expectedItems, ItemCollection actualItems) { - assertThat(expectedItems.getItems(), hasSize(actualItems.getItems().size())); - for (int i = 0; i < expectedItems.getItems().size(); i++) { - assertThat(expectedItems.getItems().get(i).getId(), equalTo(actualItems.getItems().get(i).getId())); - assertThat(expectedItems.getItems().get(i).getPermanentLoanType(), equalTo(actualItems.getItems().get(i).getPermanentLoanType())); - assertThat(expectedItems.getItems().get(i).getTemporaryLoanType(), equalTo(actualItems.getItems().get(i).getTemporaryLoanType())); - } - } - - private String getIdentifier(IdentifierType identifierType) { - switch (identifierType) { - case ID: - return "b7a9718a-0c26-4d43-ace9-52234ff74ad8"; - case HRID: - return "it00000000002"; - case HOLDINGS_RECORD_ID: - return "4929e3d5-8de5-4bb2-8818-3c23695e7505"; - case BARCODE: - return "0001"; - case FORMER_IDS: - return "former_id"; - case ACCESSION_NUMBER: - return "accession_number"; - default: - return null; - } - } } diff --git a/src/test/java/org/folio/dew/controller/ItemsContentUpdateTestData.java b/src/test/java/org/folio/dew/controller/ItemsContentUpdateTestData.java deleted file mode 100644 index b4daf0324..000000000 --- a/src/test/java/org/folio/dew/controller/ItemsContentUpdateTestData.java +++ /dev/null @@ -1,85 +0,0 @@ -package org.folio.dew.controller; - -import static org.folio.dew.domain.dto.ItemContentUpdate.ActionEnum.CLEAR_FIELD; -import static org.folio.dew.domain.dto.ItemContentUpdate.ActionEnum.REPLACE_WITH; -import static org.folio.dew.domain.dto.ItemContentUpdate.OptionEnum.PERMANENT_LOAN_TYPE; -import static org.folio.dew.domain.dto.ItemContentUpdate.OptionEnum.PERMANENT_LOCATION; -import static org.folio.dew.domain.dto.ItemContentUpdate.OptionEnum.STATUS; -import static org.folio.dew.domain.dto.ItemContentUpdate.OptionEnum.TEMPORARY_LOAN_TYPE; -import static org.folio.dew.domain.dto.ItemContentUpdate.OptionEnum.TEMPORARY_LOCATION; - -import lombok.AllArgsConstructor; -import lombok.Getter; -import org.folio.dew.domain.dto.ItemContentUpdate; - -@AllArgsConstructor -@Getter -public enum ItemsContentUpdateTestData { - REPLACE_WITH_PERMANENT_LOCATION(PERMANENT_LOCATION, REPLACE_WITH, "Annex", - "src/test/resources/output/expected_items_with_updated_permanent_location.json", - "src/test/resources/output/expected_items_with_updated_permanent_location.csv", - "src/test/resources/output/expected_preview_items_with_updated_permanent_location.csv"), - REPLACE_WITH_TEMPORARY_LOCATION(TEMPORARY_LOCATION, REPLACE_WITH, "Annex", - "src/test/resources/output/expected_items_with_updated_temporary_location.json", - "src/test/resources/output/expected_items_with_updated_temporary_location.csv", - "src/test/resources/output/expected_preview_items_with_updated_temporary_location.csv"), - REPLACE_WITH_NULL_PERMANENT_LOCATION(PERMANENT_LOCATION, REPLACE_WITH, null, - "src/test/resources/output/expected_items_with_deleted_permanent_location.json", - "src/test/resources/output/expected_items_with_deleted_permanent_location.csv", - "src/test/resources/output/expected_preview_items_with_deleted_permanent_location.csv"), - REPLACE_WITH_NULL_TEMPORARY_LOCATION(TEMPORARY_LOCATION, REPLACE_WITH, null, - "src/test/resources/output/expected_items_with_deleted_temporary_location.json", - "src/test/resources/output/expected_items_with_deleted_temporary_location.csv", - "src/test/resources/output/expected_preview_items_with_deleted_temporary_location.csv"), - REPLACE_WITH_PERMANENT_LOAN_TYPE(PERMANENT_LOAN_TYPE, REPLACE_WITH, "Reading room", - "src/test/resources/output/expected_items_with_updated_permanent_loan_type.json", - "src/test/resources/output/expected_items_with_updated_permanent_loan_type.csv", - "src/test/resources/output/expected_items_with_updated_permanent_loan_type.csv"), - REPLACE_WITH_TEMPORARY_LOAN_TYPE(TEMPORARY_LOAN_TYPE, REPLACE_WITH, "Reading room", - "src/test/resources/output/expected_items_with_updated_temporary_loan_type.json", - "src/test/resources/output/expected_items_with_updated_temporary_loan_type.csv", - "src/test/resources/output/expected_items_with_updated_temporary_loan_type.csv"), - REPLACE_WITH_NULL_PERMANENT_LOAN_TYPE(PERMANENT_LOAN_TYPE, REPLACE_WITH, null, - "src/test/resources/output/expected_items_with_deleted_permanent_loan_type.json", - "src/test/resources/output/expected_items_with_deleted_permanent_loan_type.csv", - "src/test/resources/output/expected_preview_items_with_deleted_permanent_loan_type.csv"), - REPLACE_WITH_NULL_TEMPORARY_LOAN_TYPE(TEMPORARY_LOAN_TYPE, REPLACE_WITH, null, - "src/test/resources/output/expected_items_with_deleted_temporary_loan_type.json", - "src/test/resources/output/expected_items_with_deleted_temporary_loan_type.csv", - "src/test/resources/output/expected_preview_items_with_deleted_temporary_loan_type.csv"), - REPLACE_WITH_ALLOWED_STATUS(STATUS, REPLACE_WITH, "Missing", - "src/test/resources/output/expected_items_with_updated_status.json", - null, null), - REPLACE_WITH_NOT_ALLOWED_STATUS(STATUS, REPLACE_WITH, "Aged to lost", - "src/test/resources/output/expected_items_with_non_updated_status.json", - null, null), - REPLACE_WITH_EMPTY_STATUS(STATUS, REPLACE_WITH, null, - "src/test/resources/output/expected_items_with_non_updated_status.json", - null, null), - CLEAR_FIELD_PERMANENT_LOCATION(PERMANENT_LOCATION, CLEAR_FIELD, null, - "src/test/resources/output/expected_items_with_deleted_permanent_location.json", - "src/test/resources/output/expected_items_with_deleted_permanent_location.csv", - "src/test/resources/output/expected_preview_items_with_deleted_permanent_location.csv"), - CLEAR_FIELD_TEMPORARY_LOCATION(TEMPORARY_LOCATION, CLEAR_FIELD, null, - "src/test/resources/output/expected_items_with_deleted_temporary_location.json", - "src/test/resources/output/expected_items_with_deleted_temporary_location.csv", - "src/test/resources/output/expected_preview_items_with_deleted_temporary_location.csv"), - CLEAR_FIELD_STATUS(STATUS, CLEAR_FIELD, null, - "src/test/resources/output/expected_items_with_non_updated_status.json", - null, null), - CLEAR_FIELD_PERMANENT_LOAN_TYPE(PERMANENT_LOAN_TYPE, CLEAR_FIELD, null, - "src/test/resources/output/expected_items_with_deleted_permanent_loan_type.json", - "src/test/resources/output/expected_items_with_deleted_permanent_loan_type.csv", - "src/test/resources/output/expected_preview_items_with_deleted_permanent_loan_type.csv"), - CLEAR_FIELD_TEMPORARY_LOAN_TYPE(TEMPORARY_LOAN_TYPE, CLEAR_FIELD, null, - "src/test/resources/output/expected_items_with_deleted_temporary_loan_type.json", - "src/test/resources/output/expected_items_with_deleted_temporary_loan_type.csv", - "src/test/resources/output/expected_preview_items_with_deleted_temporary_loan_type.csv"); - - final ItemContentUpdate.OptionEnum option; - final ItemContentUpdate.ActionEnum action; - final String value; - final String expectedPreviewJsonPath; - final String expectedCsvPath; - final String expectedPreviewCsvPath; -} diff --git a/src/test/java/org/folio/dew/controller/PermissionsSelfCheckControllerTest.java b/src/test/java/org/folio/dew/controller/PermissionsSelfCheckControllerTest.java new file mode 100644 index 000000000..9c661b877 --- /dev/null +++ b/src/test/java/org/folio/dew/controller/PermissionsSelfCheckControllerTest.java @@ -0,0 +1,29 @@ +package org.folio.dew.controller; + +import lombok.SneakyThrows; +import org.folio.dew.BaseBatchTest; +import org.folio.dew.batch.bulkedit.jobs.permissions.check.UserPermissions; +import org.junit.jupiter.api.Test; + +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.mockito.ArgumentMatchers.isA; +import static org.mockito.Mockito.when; +import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get; +import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status; + +class PermissionsSelfCheckControllerTest extends BaseBatchTest { + + @Test + @SneakyThrows + void shouldReturnDesiredPermissions() { + var headers = defaultHeaders(); + when(okapiUserPermissionsClient.getPermissions(isA(String.class))) + .thenReturn(UserPermissions.builder().permissionNames(List.of("permission-1", "permission-2")).build()); + var result = mockMvc.perform(get("/bulk-edit/permissions-self-check") + .headers(headers)).andExpect(status().isOk()).andReturn(); + + assertEquals("[\"permission-1\",\"permission-2\"]", result.getResponse().getContentAsString()); + } +} diff --git a/src/test/java/org/folio/dew/controller/PresignedUrlControllerTest.java b/src/test/java/org/folio/dew/controller/PresignedUrlControllerTest.java index c48b2d9e0..cc42b0c80 100644 --- a/src/test/java/org/folio/dew/controller/PresignedUrlControllerTest.java +++ b/src/test/java/org/folio/dew/controller/PresignedUrlControllerTest.java @@ -32,14 +32,4 @@ void shouldRetrievePresignedUrl() throws Exception { .queryParam(FILE_PATH, filePath)) .andExpect(status().isOk()); } - - @Test - void shouldReturnErrorWhenRetrievingPresignedUrlFailed() throws Exception { - var headers = defaultHeaders(); - - mockMvc.perform(get(REFRESH_PRESIGNED_URL) - .headers(headers) - .queryParam(FILE_PATH, "")) - .andExpect(status().is5xxServerError()); - } } diff --git a/src/test/java/org/folio/dew/repository/LocalFilesStorageAwsSdkComposingTest.java b/src/test/java/org/folio/dew/repository/LocalFilesStorageAwsSdkComposingTest.java index a5f9486be..8527182f6 100644 --- a/src/test/java/org/folio/dew/repository/LocalFilesStorageAwsSdkComposingTest.java +++ b/src/test/java/org/folio/dew/repository/LocalFilesStorageAwsSdkComposingTest.java @@ -19,6 +19,7 @@ import lombok.extern.log4j.Log4j2; import static java.util.stream.Collectors.toList; +import static org.folio.dew.utils.Constants.PATH_SEPARATOR; import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.is; import static org.junit.jupiter.api.Assertions.assertTrue; @@ -29,6 +30,8 @@ @EnableConfigurationProperties class LocalFilesStorageAwsSdkComposingTest { + @Autowired + private LocalFilesStorageProperties localFilesStorageProperties; @Autowired private LocalFilesStorage localFilesStorage; @@ -39,8 +42,9 @@ void testWriteReadPatchDelete(int size) throws IOException { byte[] original = getRandomBytes(size); var remoteFilePath = "CSV_Data.csv"; + var expectedS3Path = localFilesStorageProperties.getSubPath() + PATH_SEPARATOR + remoteFilePath; - assertThat(localFilesStorage.write(remoteFilePath, original), is(remoteFilePath)); + assertThat(localFilesStorage.write(remoteFilePath, original), is(expectedS3Path)); assertTrue(localFilesStorage.exists(remoteFilePath)); assertTrue(Objects.deepEquals(localFilesStorage.readAllBytes(remoteFilePath), original)); diff --git a/src/test/java/org/folio/dew/repository/LocalFilesStorageTest.java b/src/test/java/org/folio/dew/repository/LocalFilesStorageTest.java index bb80c5471..8205a7d6e 100644 --- a/src/test/java/org/folio/dew/repository/LocalFilesStorageTest.java +++ b/src/test/java/org/folio/dew/repository/LocalFilesStorageTest.java @@ -12,7 +12,6 @@ import org.springframework.boot.test.context.SpringBootTest; import java.io.BufferedWriter; -import java.io.ByteArrayInputStream; import java.io.IOException; import java.io.InputStream; import java.util.List; @@ -21,6 +20,7 @@ import static java.util.List.of; import static java.util.stream.Collectors.toList; +import static org.folio.dew.utils.Constants.PATH_SEPARATOR; import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.is; import static org.junit.jupiter.api.Assertions.assertEquals; @@ -35,20 +35,25 @@ class LocalFilesStorageTest { private static final String NON_EXISTING_PATH = "non-existing-path"; + @Autowired + private LocalFilesStorageProperties localFilesStorageProperties; @Autowired private LocalFilesStorage localFilesStorage; + @ParameterizedTest @ValueSource(ints = {1024, ObjectWriteArgs.MIN_MULTIPART_SIZE + 1 }) @DisplayName("Create files and read internal objects structure") void testWriteRead(int size) throws IOException { + var subPath = localFilesStorageProperties.getSubPath() + PATH_SEPARATOR; byte[] content = getRandomBytes(size); var original = of("directory_1/CSV_Data_1.csv", "directory_1/directory_2/CSV_Data_2.csv", "directory_1/directory_2/directory_3/CSV_Data_3.csv"); - - List expected; + var expectedS3Pathes = of(subPath + "directory_1/CSV_Data_1.csv", subPath + "directory_1/directory_2/CSV_Data_2.csv", + subPath + "directory_1/directory_2/directory_3/CSV_Data_3.csv"); + List actual; try { - expected = original.stream() + actual = original.stream() .map(p -> { try { return localFilesStorage.write(p, content); @@ -61,16 +66,16 @@ void testWriteRead(int size) throws IOException { throw new IOException(e); } - assertTrue(Objects.deepEquals(original, expected)); + assertTrue(Objects.deepEquals(expectedS3Pathes, actual)); - assertTrue(Objects.deepEquals(localFilesStorage.walk("directory_1/") + assertTrue(Objects.deepEquals(localFilesStorage.walk(subPath + "directory_1/") .collect(toList()), - of("directory_1/CSV_Data_1.csv", "directory_1/directory_2/CSV_Data_2.csv", - "directory_1/directory_2/directory_3/CSV_Data_3.csv"))); + of(subPath + "directory_1/CSV_Data_1.csv", subPath + "directory_1/directory_2/CSV_Data_2.csv", + subPath + "directory_1/directory_2/directory_3/CSV_Data_3.csv"))); - assertTrue(Objects.deepEquals(localFilesStorage.walk("directory_1/directory_2/") + assertTrue(Objects.deepEquals(localFilesStorage.walk(subPath + "directory_1/directory_2/") .collect(toList()), - of("directory_1/directory_2/CSV_Data_2.csv", "directory_1/directory_2/directory_3/CSV_Data_3.csv"))); + of(subPath + "directory_1/directory_2/CSV_Data_2.csv", subPath + "directory_1/directory_2/directory_3/CSV_Data_3.csv"))); original.forEach(p -> assertTrue(localFilesStorage.exists(p))); @@ -111,8 +116,9 @@ void testWriteReadPatchDelete(int size) throws IOException { byte[] original = getRandomBytes(size); byte[] patch = getRandomBytes(size); var remoteFilePath = "directory_1/directory_2/CSV_Data.csv"; + var expectedS3FilePath = localFilesStorageProperties.getSubPath() + PATH_SEPARATOR + remoteFilePath; - assertThat(localFilesStorage.write(remoteFilePath, original), is(remoteFilePath)); + assertThat(localFilesStorage.write(remoteFilePath, original), is(expectedS3FilePath)); assertTrue(localFilesStorage.exists(remoteFilePath)); assertTrue(Objects.deepEquals(localFilesStorage.readAllBytes(remoteFilePath), original)); diff --git a/src/test/java/org/folio/dew/repository/RemoteFilesStorageTest.java b/src/test/java/org/folio/dew/repository/RemoteFilesStorageTest.java new file mode 100644 index 000000000..bf4ceaf99 --- /dev/null +++ b/src/test/java/org/folio/dew/repository/RemoteFilesStorageTest.java @@ -0,0 +1,29 @@ +package org.folio.dew.repository; + + +import lombok.SneakyThrows; +import org.folio.dew.BaseBatchTest; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class RemoteFilesStorageTest extends BaseBatchTest { + + @Autowired + private RemoteFilesStorage remoteFilesStorage; + + @Test + @SneakyThrows + void testContainsFile() { + byte[] content = "content".getBytes(); + var path = "directory/data.csv"; + + var uploadedPath = remoteFilesStorage.write(path, content); + + assertEquals("remote/directory/data.csv", uploadedPath); + assertTrue(remoteFilesStorage.containsFile(path)); + assertTrue(remoteFilesStorage.containsFile(uploadedPath)); + } +} diff --git a/src/test/java/org/folio/dew/service/BulkEditParseServiceTest.java b/src/test/java/org/folio/dew/service/BulkEditParseServiceTest.java deleted file mode 100644 index feedbdef6..000000000 --- a/src/test/java/org/folio/dew/service/BulkEditParseServiceTest.java +++ /dev/null @@ -1,58 +0,0 @@ -package org.folio.dew.service; - -import org.folio.dew.BaseBatchTest; -import org.folio.dew.domain.dto.*; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.params.ParameterizedTest; -import org.junit.jupiter.params.provider.ValueSource; -import org.springframework.beans.factory.annotation.Autowired; - -import java.util.Collections; -import java.util.Set; -import java.util.UUID; - -import static org.assertj.core.api.Assertions.assertThat; -import static org.junit.jupiter.api.Assertions.assertEquals; - -class BulkEditParseServiceTest extends BaseBatchTest { - @Autowired - private BulkEditParseService bulkEditParseService; - - @ParameterizedTest - @ValueSource(strings = {"", " "}) - void shouldIgnoreBlankBarcodeAndExternalSystemId(String val) { - var userFormat = UserFormat.builder() - .patronGroup("PatronGroup") - .externalSystemId(val) - .barcode(val) - .active("true") - .departments("") - .proxyFor("") - .addresses("") - .build(); - - assertThat(bulkEditParseService.mapUserFormatToUser(userFormat).getExternalSystemId()).isNull(); - assertThat(bulkEditParseService.mapUserFormatToUser(userFormat).getBarcode()).isNull(); - } - - @Test - void shouldReturnInitialIdsForWrongReferenceIdsWhenMappingToUser() { - var userFormat = UserFormat.builder() - .active("true") - .patronGroup("staff") - .addresses(";;;;;;;false;db541cda-fcc7-403b-8077-3613f3244901") - .preferredContactTypeId("002") - .departments("103aee0f-c5f6-44de-94aa-74093f0e45d9") - .build(); - - var expectedUser = new User() - .personal(new Personal().addresses(Collections.singletonList(new Address().addressTypeId("db541cda-fcc7-403b-8077-3613f3244901")))) - .patronGroup("3684a786-6671-4268-8ed0-9db82ebca60b") - .departments(Set.of(UUID.fromString("103aee0f-c5f6-44de-94aa-74093f0e45d9"))); - - var actualUser = bulkEditParseService.mapUserFormatToUser(userFormat); - - assertEquals(expectedUser.getPersonal().getAddresses().get(0).getAddressTypeId(), actualUser.getPersonal().getAddresses().get(0).getAddressTypeId()); - assertEquals(expectedUser.getDepartments(), actualUser.getDepartments()); - } -} diff --git a/src/test/java/org/folio/dew/service/BulkEditProcessingErrorsServiceTest.java b/src/test/java/org/folio/dew/service/BulkEditProcessingErrorsServiceTest.java index 05869f958..2a93749f6 100644 --- a/src/test/java/org/folio/dew/service/BulkEditProcessingErrorsServiceTest.java +++ b/src/test/java/org/folio/dew/service/BulkEditProcessingErrorsServiceTest.java @@ -55,6 +55,35 @@ void saveErrorInCSVTestSuccessTest() throws IOException { removeStorage(); } + @Test + @DisplayName("Show that error message is stored in error file") + void saveErrorMessageInCSVTestSuccessTest() throws IOException { + var jobId = UUID.randomUUID().toString(); + var affectedIdentifier = "ID"; + var errorMessage = "Record not found"; + var fileName = "userUUIDs.csv"; + var csvFileName = LocalDate.now().format(CSV_NAME_DATE_FORMAT) + "-Matching-Records-Errors-" + fileName; + var pathToCsvFile = "E" + File.separator + BulkEditProcessingErrorsService.STORAGE + File.separator + jobId + File.separator + csvFileName; + bulkEditProcessingErrorsService.saveErrorInCSV(jobId, affectedIdentifier, errorMessage, fileName); + assertTrue(localFilesStorage.exists(pathToCsvFile)); + List lines = localFilesStorage.readAllLines(pathToCsvFile); + String expectedLine = affectedIdentifier + "," + errorMessage; + assertEquals(expectedLine, lines.get(0)); + assertThat(lines, hasSize(1)); + } + + @Test + @DisplayName("Show that error message is not stored in error file") + void saveErrorNullMessageInCSVTestSuccessTest() { + var jobId = UUID.randomUUID().toString(); + var affectedIdentifier = "ID"; + var fileName = "userUUIDs.csv"; + var csvFileName = LocalDate.now().format(CSV_NAME_DATE_FORMAT) + "-Matching-Records-Errors-" + fileName; + var pathToCsvFile = "E" + File.separator + BulkEditProcessingErrorsService.STORAGE + File.separator + jobId + File.separator + csvFileName; + bulkEditProcessingErrorsService.saveErrorInCSV(jobId, affectedIdentifier, (String) null, fileName); + assertFalse(localFilesStorage.exists(pathToCsvFile)); + } + @Test @DisplayName("Show that error file is not created if at lease one of the parameter is null") void saveErrorInCSVTestFailedTest() { @@ -68,7 +97,7 @@ void saveErrorInCSVTestFailedTest() { assertFalse(Files.exists(pathToCsvFile)); bulkEditProcessingErrorsService.saveErrorInCSV(jobId, null, reasonForError, fileName); assertFalse(Files.exists(pathToCsvFile)); - bulkEditProcessingErrorsService.saveErrorInCSV(jobId, affectedIdentifier, null, fileName); + bulkEditProcessingErrorsService.saveErrorInCSV(jobId, affectedIdentifier, new BulkEditException("error message"), fileName); assertFalse(Files.exists(pathToCsvFile)); bulkEditProcessingErrorsService.saveErrorInCSV(jobId, affectedIdentifier, reasonForError, null); assertFalse(Files.exists(pathToCsvFile)); diff --git a/src/test/java/org/folio/dew/service/BulkEditRollBackServiceTest.java b/src/test/java/org/folio/dew/service/BulkEditRollBackServiceTest.java deleted file mode 100644 index 12687e4a5..000000000 --- a/src/test/java/org/folio/dew/service/BulkEditRollBackServiceTest.java +++ /dev/null @@ -1,127 +0,0 @@ -package org.folio.dew.service; - -import org.folio.dew.client.DataExportSpringClient; -import org.folio.dew.repository.RemoteFilesStorage; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.extension.ExtendWith; -import org.mockito.InjectMocks; -import org.mockito.Mock; -import org.mockito.junit.jupiter.MockitoExtension; -import org.springframework.batch.core.ExitStatus; -import org.springframework.batch.core.JobExecution; -import org.springframework.batch.core.JobParameters; -import org.springframework.batch.core.launch.JobOperator; - -import java.util.List; -import java.util.UUID; - -import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertFalse; -import static org.junit.jupiter.api.Assertions.assertTrue; -import static org.mockito.ArgumentMatchers.any; -import static org.mockito.ArgumentMatchers.isA; -import static org.mockito.Mockito.doNothing; -import static org.mockito.Mockito.times; -import static org.mockito.Mockito.verify; -import static org.mockito.Mockito.when; - -@ExtendWith(MockitoExtension.class) -class BulkEditRollBackServiceTest { - - @Mock - private JobOperator jobOperator; - @Mock - private BulkEditRollBackJobLauncher rollBackJobLauncher; - @Mock - private DataExportSpringClient dataExportSpringClient; - @Mock - private RemoteFilesStorage remoteFilesStorage; - - @InjectMocks - private BulkEditRollBackService bulkEditRollBackService; - - @Test - void stopAndRollBackJobExecutionByJobIdTest() throws Exception { - var jobId = UUID.fromString("edd30136-9a7b-4226-9e82-83024dbeac4a"); - var jobIdWithRollBackFile = jobId.toString(); - var executionId = 0l; - var job = new org.folio.dew.domain.dto.Job(); - job.setFiles(List.of("minio/path/" + jobIdWithRollBackFile + "_file.csv")); - - bulkEditRollBackService.putExecutionInfoPerJob(executionId, jobId); - when(dataExportSpringClient.getJobById(isA(String.class))).thenReturn(job); - doNothing().when(remoteFilesStorage).downloadObject(isA(String.class), isA(String.class)); - when(rollBackJobLauncher.run(any(), isA(JobParameters.class))).thenReturn(new JobExecution(1l)); - - bulkEditRollBackService.stopAndRollBackJobExecutionByJobId(jobId); - - verify(jobOperator, times(1)).stop(executionId); - verify(dataExportSpringClient, times(1)).getJobById(jobIdWithRollBackFile); - verify(remoteFilesStorage, times(1)).downloadObject(isA(String.class), isA(String.class)); - verify(rollBackJobLauncher, times(1)).run(any(), isA(JobParameters.class)); - } - - @Test - void getFileForRollBackFromMinIO() { - var jobIdWithRollBackFile = "74914e57-3406-4757-938b-9a3f718d0ee6"; - var fileUploadName = "/some/file/" + jobIdWithRollBackFile + "_file.csv"; - var job = new org.folio.dew.domain.dto.Job(); - job.setFiles(List.of("minio/path/" + jobIdWithRollBackFile + "_file.csv")); - - when(dataExportSpringClient.getJobById(isA(String.class))).thenReturn(job); - - var actual = bulkEditRollBackService.getFileForRollBackFromMinIO(fileUploadName); - assertEquals("minio/path/74914e57-3406-4757-938b-9a3f718d0ee6_file.csv", actual); - } - - @Test - void cleanJobDataTest() { - var jobId = UUID.fromString("edd30136-9a7b-4226-9e82-83024dbeac4a"); - var jobIdWithRollBackFile = "74914e57-3406-4757-938b-9a3f718d0ee6"; - var executionId = 0l; - var userId = "userId"; - - bulkEditRollBackService.putExecutionInfoPerJob(executionId, jobId); - bulkEditRollBackService.putUserIdForJob(userId, jobId); - assertTrue(bulkEditRollBackService.isExecutionIdExistForJob(jobId)); - assertTrue(bulkEditRollBackService.isUserBeRollBack(userId, jobId)); - - bulkEditRollBackService.cleanJobData(jobId); - assertFalse(bulkEditRollBackService.isExecutionIdExistForJob(jobId)); - assertTrue(bulkEditRollBackService.isUserBeRollBack(userId, jobId)); - } - - @Test - void cleanJobDataWithCompletedExitCodeTest() { - var jobId = UUID.fromString("edd30136-9a7b-4226-9e82-83024dbeac4a"); - var jobIdWithRollBackFile = "74914e57-3406-4757-938b-9a3f718d0ee6"; - var executionId = 0l; - var userId = "userId"; - - bulkEditRollBackService.putExecutionInfoPerJob(executionId, jobId); - bulkEditRollBackService.putUserIdForJob(userId, jobId); - assertTrue(bulkEditRollBackService.isExecutionIdExistForJob(jobId)); - assertTrue(bulkEditRollBackService.isUserBeRollBack(userId, jobId)); - - bulkEditRollBackService.cleanJobData(ExitStatus.COMPLETED.getExitCode(), jobId); - assertFalse(bulkEditRollBackService.isExecutionIdExistForJob(jobId)); - assertTrue(bulkEditRollBackService.isUserBeRollBack(userId, jobId)); - } - - @Test - void cleanJobDataWithStoppedExitCodeTest() { - var jobId = UUID.fromString("edd30136-9a7b-4226-9e82-83024dbeac4a"); - var jobIdWithRollBackFile = "74914e57-3406-4757-938b-9a3f718d0ee6"; - var executionId = 0l; - var userId = "userId"; - - bulkEditRollBackService.putExecutionInfoPerJob(executionId, jobId); - bulkEditRollBackService.putUserIdForJob(userId, jobId); - assertTrue(bulkEditRollBackService.isExecutionIdExistForJob(jobId)); - - bulkEditRollBackService.cleanJobData(ExitStatus.STOPPED.getExitCode(), jobId); - assertTrue(bulkEditRollBackService.isExecutionIdExistForJob(jobId)); - assertTrue(bulkEditRollBackService.isUserBeRollBack(userId, jobId)); - assertFalse(bulkEditRollBackService.isUserBeRollBack(userId, jobId)); - } -} diff --git a/src/test/java/org/folio/dew/service/BulkEditUserContentUpdateServiceTest.java b/src/test/java/org/folio/dew/service/BulkEditUserContentUpdateServiceTest.java deleted file mode 100644 index 1c0b8b40f..000000000 --- a/src/test/java/org/folio/dew/service/BulkEditUserContentUpdateServiceTest.java +++ /dev/null @@ -1,77 +0,0 @@ -package org.folio.dew.service; - -import static org.apache.commons.lang3.StringUtils.EMPTY; -import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_IDENTIFIERS; -import static org.folio.dew.domain.dto.ExportType.BULK_EDIT_UPDATE; -import static org.folio.dew.domain.dto.JobParameterNames.PREVIEW_FILE_NAME; -import static org.folio.dew.domain.dto.JobParameterNames.TEMP_OUTPUT_FILE_PATH; -import static org.folio.dew.domain.dto.JobParameterNames.UPDATED_FILE_NAME; -import static org.folio.dew.utils.Constants.CSV_EXTENSION; -import static org.folio.dew.utils.Constants.PATH_SEPARATOR; -import static org.folio.dew.utils.Constants.PREVIEW_PREFIX; -import static org.folio.dew.utils.Constants.UPDATED_PREFIX; -import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.equalTo; -import static org.hamcrest.Matchers.hasSize; -import static org.hamcrest.Matchers.is; - -import lombok.SneakyThrows; -import org.apache.commons.io.FilenameUtils; -import org.folio.de.entity.JobCommand; -import org.folio.dew.BaseBatchTest; -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.domain.dto.UserContentUpdateAction; -import org.folio.dew.domain.dto.UserContentUpdateCollection; -import org.folio.dew.service.update.BulkEditUserContentUpdateService; -import org.junit.jupiter.api.Test; -import org.springframework.batch.core.JobParametersBuilder; -import org.springframework.beans.factory.annotation.Autowired; - -import java.util.Collections; -import java.util.UUID; - -class BulkEditUserContentUpdateServiceTest extends BaseBatchTest { - private static final String USER_DATA = "src/test/resources/upload/user_data.csv"; - - @Autowired - private BulkEditUserContentUpdateService contentUpdateService; - - @Test - @SneakyThrows - void shouldCreateUpdatedAndPreviewFilesAndUpdateJobCommand() { - var jobId = UUID.randomUUID(); - var uploadedFileName = FilenameUtils.getName(USER_DATA); - var updatedFileName = jobId + PATH_SEPARATOR + UPDATED_PREFIX + uploadedFileName; - var previewFileName = jobId + PATH_SEPARATOR + PREVIEW_PREFIX + uploadedFileName; - remoteFilesStorage.upload(jobId + PATH_SEPARATOR + uploadedFileName, USER_DATA); - - var jobCommand = new JobCommand(); - jobCommand.setId(jobId); - jobCommand.setExportType(BULK_EDIT_IDENTIFIERS); - jobCommand.setJobParameters(new JobParametersBuilder() - .addString(TEMP_OUTPUT_FILE_PATH, "test/path/" + USER_DATA.replace(CSV_EXTENSION, EMPTY)) - .toJobParameters()); - - jobCommandsReceiverService.addBulkEditJobCommand(jobCommand); - - var contentUpdates = new UserContentUpdateCollection() - .userContentUpdates(Collections.singletonList(new UserContentUpdate() - .option(UserContentUpdate.OptionEnum.PATRON_GROUP) - .actions(Collections.singletonList(new UserContentUpdateAction() - .name(UserContentUpdateAction.NameEnum.REPLACE_WITH) - .value("PatronGroup"))))) - .totalRecords(1); - - var res = contentUpdateService.process(jobCommand, contentUpdates); - - assertThat(res.getEntitiesForPreview(), hasSize(2)); - assertThat(res.getEntitiesForPreview().stream().allMatch(userFormat -> "PatronGroup".equals(userFormat.getPatronGroup())), is(true)); - - assertThat(remoteFilesStorage.containsFile(updatedFileName), is(true)); - assertThat(remoteFilesStorage.containsFile(previewFileName), is(true)); - - assertThat(jobCommand.getExportType(), equalTo(BULK_EDIT_UPDATE)); - assertThat(jobCommand.getJobParameters().getString(UPDATED_FILE_NAME), equalTo(updatedFileName)); - assertThat(jobCommand.getJobParameters().getString(PREVIEW_FILE_NAME), equalTo(previewFileName)); - } -} diff --git a/src/test/java/org/folio/dew/service/ConsortiaServiceTest.java b/src/test/java/org/folio/dew/service/ConsortiaServiceTest.java new file mode 100644 index 000000000..86e171614 --- /dev/null +++ b/src/test/java/org/folio/dew/service/ConsortiaServiceTest.java @@ -0,0 +1,64 @@ +package org.folio.dew.service; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.when; + +import org.folio.dew.client.ConsortiumClient; +import org.folio.dew.domain.bean.Consortia; +import org.folio.dew.domain.bean.ConsortiaCollection; +import org.folio.dew.domain.dto.UserTenant; +import org.folio.dew.domain.dto.UserTenantCollection; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.InjectMocks; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.util.List; + +@ExtendWith(MockitoExtension.class) +class ConsortiaServiceTest { + + @Mock + private ConsortiumClient consortiumClient; + + @InjectMocks + private ConsortiaService consortiaService; + + @Test + void testGetAffiliatedTenants() { + var userId = "userId"; + + var consortia = new Consortia(); + consortia.setId("consortia"); + var consortiaCollection = new ConsortiaCollection(); + consortiaCollection.setConsortia(List.of(consortia)); + + var userTenant = new UserTenant(); + userTenant.setTenantId("member"); + var userTenantCollection = new UserTenantCollection(); + userTenantCollection.setUserTenants(List.of(userTenant)); + + when(consortiumClient.getConsortia()).thenReturn(consortiaCollection); + when(consortiumClient.getConsortiaUserTenants(consortia.getId(), userId, Integer.MAX_VALUE)).thenReturn(userTenantCollection); + + var affiliatedTenants = consortiaService.getAffiliatedTenants("currentTenantId", userId); + + assertFalse(affiliatedTenants.isEmpty()); + assertEquals("member", affiliatedTenants.get(0)); + } + + @Test + void testGetAffiliatedTenantsIfConsortiaDoesNotExist() { + var userId = "userId"; + var consortiaCollection = new ConsortiaCollection(); + consortiaCollection.setConsortia(List.of()); + + when(consortiumClient.getConsortia()).thenReturn(consortiaCollection); + + var affiliatedTenants = consortiaService.getAffiliatedTenants("currentTenantId", userId); + assertTrue(affiliatedTenants.isEmpty()); + } +} diff --git a/src/test/java/org/folio/dew/service/ElectronicAccessServiceTest.java b/src/test/java/org/folio/dew/service/ElectronicAccessServiceTest.java index 906b04472..a91b72126 100644 --- a/src/test/java/org/folio/dew/service/ElectronicAccessServiceTest.java +++ b/src/test/java/org/folio/dew/service/ElectronicAccessServiceTest.java @@ -16,6 +16,8 @@ import org.mockito.junit.jupiter.MockitoExtension; import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.UUID; @@ -116,7 +118,9 @@ void getRelationshipNameAndIdByIdTest() { @Test void getRelationshipNameAndIdByIdNotFoundExceptionTest() { - when(folioExecutionContext.getAllHeaders()).thenReturn(Map.of(X_OKAPI_TENANT, List.of("original"))); + Map> headers = new HashMap<>(); + headers.put(X_OKAPI_TENANT, List.of("original")); + when(folioExecutionContext.getAllHeaders()).thenReturn(headers); var id = UUID.randomUUID().toString(); var electronicAccessRelationship = new ElectronicAccessRelationship(); electronicAccessRelationship.setId(id); diff --git a/src/test/java/org/folio/dew/service/HoldingsReferenceServiceTest.java b/src/test/java/org/folio/dew/service/HoldingsReferenceServiceTest.java index a31771a49..8ef8cd16f 100644 --- a/src/test/java/org/folio/dew/service/HoldingsReferenceServiceTest.java +++ b/src/test/java/org/folio/dew/service/HoldingsReferenceServiceTest.java @@ -17,6 +17,8 @@ import org.mockito.junit.jupiter.MockitoExtension; import java.io.File; +import java.util.Collection; +import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.UUID; @@ -38,11 +40,13 @@ class HoldingsReferenceServiceTest { "titlePublisherDate.json;Sample title. Publisher, 2023"}, delimiter = ';') @SneakyThrows void shouldFormatInstanceTitle(String fileName, String expected) { + Map> headers = new HashMap<>(); + headers.put(X_OKAPI_TENANT, List.of("original")); var mapper = new ObjectMapper(); var path = "src/test/resources/samples/" + fileName; var resultJson = mapper.readTree(new File(path)); - when(folioExecutionContext.getAllHeaders()).thenReturn(Map.of(X_OKAPI_TENANT, List.of("original"))); + when(folioExecutionContext.getAllHeaders()).thenReturn(headers); when(instanceClient.getInstanceJsonById(anyString())).thenReturn(resultJson); var actual = service.getInstanceTitleById(UUID.randomUUID().toString(), "tenant"); diff --git a/src/test/java/org/folio/dew/service/UserPermissionsServiceTest.java b/src/test/java/org/folio/dew/service/UserPermissionsServiceTest.java new file mode 100644 index 000000000..7deacc936 --- /dev/null +++ b/src/test/java/org/folio/dew/service/UserPermissionsServiceTest.java @@ -0,0 +1,54 @@ +package org.folio.dew.service; + +import org.folio.dew.batch.bulkedit.jobs.permissions.check.UserPermissions; +import org.folio.dew.client.EurekaUserPermissionsClient; +import org.folio.dew.client.OkapiUserPermissionsClient; +import org.folio.spring.FolioExecutionContext; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.InjectMocks; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.util.UUID; + +import static org.folio.dew.service.UserPermissionsService.EUREKA_PLATFORM; +import static org.folio.dew.service.UserPermissionsService.OKAPI_PLATFORM; +import static org.mockito.ArgumentMatchers.anyList; +import static org.mockito.ArgumentMatchers.isA; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class UserPermissionsServiceTest { + + @Mock + private FolioExecutionContext folioExecutionContext; + @Mock + private OkapiUserPermissionsClient okapiUserPermissionsClient; + @Mock + private EurekaUserPermissionsClient eurekaUserPermissionsClient; + + @InjectMocks + private UserPermissionsService userPermissionsService; + + @Test + void getPermissionsTest() { + when(folioExecutionContext.getUserId()).thenReturn(UUID.randomUUID()); + when(okapiUserPermissionsClient.getPermissions(isA(String.class))).thenReturn(new UserPermissions()); + + userPermissionsService.setPlatform(OKAPI_PLATFORM); + userPermissionsService.getPermissions(); + verify(okapiUserPermissionsClient).getPermissions(isA(String.class)); + } + + @Test + void getPermissionsIfEurekaTest() { + when(folioExecutionContext.getUserId()).thenReturn(UUID.randomUUID()); + when(eurekaUserPermissionsClient.getPermissions(isA(String.class), anyList())).thenReturn(new UserPermissions()); + + userPermissionsService.setPlatform(EUREKA_PLATFORM); + userPermissionsService.getPermissions(); + verify(eurekaUserPermissionsClient).getPermissions(isA(String.class), anyList()); + } +} diff --git a/src/test/java/org/folio/dew/service/update/EmailUpdateStrategyTest.java b/src/test/java/org/folio/dew/service/update/EmailUpdateStrategyTest.java deleted file mode 100644 index 4172a8270..000000000 --- a/src/test/java/org/folio/dew/service/update/EmailUpdateStrategyTest.java +++ /dev/null @@ -1,77 +0,0 @@ -package org.folio.dew.service.update; - -import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.equalTo; -import static org.junit.jupiter.api.Assertions.assertThrows; - -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.domain.dto.UserContentUpdateAction; -import org.folio.dew.domain.dto.UserFormat; -import org.folio.dew.error.BulkEditException; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.params.ParameterizedTest; -import org.junit.jupiter.params.provider.CsvSource; -import org.junit.jupiter.params.provider.ValueSource; - -import java.util.List; - -class EmailUpdateStrategyTest { - private final EmailUpdateStrategy updateStrategy = new EmailUpdateStrategy(); - - @Test - void shouldUpdateEmailIfDomainMatch() { - var userFormat = new UserFormat().withEmail("test@somedomain.com"); - var contentUpdate = new UserContentUpdate() - .option(UserContentUpdate.OptionEnum.EMAIL_ADDRESS) - .actions(List.of(new UserContentUpdateAction() - .name(UserContentUpdateAction.NameEnum.FIND) - .value("somedomain.com"), - new UserContentUpdateAction() - .name(UserContentUpdateAction.NameEnum.REPLACE_WITH) - .value("newdomain.com") - )); - - var updatedUserFormat = updateStrategy.applyUpdate(userFormat, contentUpdate); - - assertThat(updatedUserFormat.getEmail(), equalTo("test@newdomain.com")); - } - - @ParameterizedTest - @CsvSource({"test,org@somedomain.com", "te,orgst@somedomain.com", "st,teorg@somedomain.com", "es,torgt@somedomain.com", - "@,testorgsomedomain.com", "t@,tesorgsomedomain.com", "@s,testorgomedomain.com", "some,test@orgdomain.com", - "med,test@soorgomain.com", "e,torgst@somorgdomain.com", ".com,test@somedomainorg", ".,test@somedomainorgcom", - "o,test@sorgmedorgmain.corgm", "co,test@somedomain.orgm", "om,test@sorgedorgain.corg"}) - void shouldUpdateAnyPartOfEmail(String findValue, String expected) { - var userFormat = new UserFormat().withEmail("test@somedomain.com"); - var contentUpdate = new UserContentUpdate() - .option(UserContentUpdate.OptionEnum.EMAIL_ADDRESS) - .actions(List.of(new UserContentUpdateAction() - .name(UserContentUpdateAction.NameEnum.FIND) - .value(findValue), - new UserContentUpdateAction() - .name(UserContentUpdateAction.NameEnum.REPLACE_WITH) - .value("org") - )); - - var updatedUserFormat = updateStrategy.applyUpdate(userFormat, contentUpdate); - - assertThat(updatedUserFormat.getEmail(), equalTo(expected)); - } - - @ParameterizedTest - @ValueSource(strings = {"another.com", "SomeDomain.com"}) - void shouldThrowExceptionIfDomainDoesNotMatch(String findValue) { - var userFormat = new UserFormat().withEmail("test@somedomain.com"); - var contentUpdate = new UserContentUpdate() - .option(UserContentUpdate.OptionEnum.EMAIL_ADDRESS) - .actions(List.of(new UserContentUpdateAction() - .name(UserContentUpdateAction.NameEnum.FIND) - .value(findValue), - new UserContentUpdateAction() - .name(UserContentUpdateAction.NameEnum.REPLACE_WITH) - .value("newdomain.com") - )); - - assertThrows(BulkEditException.class, () -> updateStrategy.applyUpdate(userFormat, contentUpdate)); - } -} diff --git a/src/test/java/org/folio/dew/service/update/ExpirationDateUpdateStrategyTest.java b/src/test/java/org/folio/dew/service/update/ExpirationDateUpdateStrategyTest.java deleted file mode 100644 index c9426a826..000000000 --- a/src/test/java/org/folio/dew/service/update/ExpirationDateUpdateStrategyTest.java +++ /dev/null @@ -1,54 +0,0 @@ -package org.folio.dew.service.update; - -import static java.time.ZoneOffset.UTC; -import static org.folio.dew.utils.BulkEditProcessorHelper.dateToString; -import static org.folio.dew.utils.Constants.DATE_TIME_PATTERN; -import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.equalTo; - -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.domain.dto.UserContentUpdateAction; -import org.folio.dew.domain.dto.UserFormat; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.params.ParameterizedTest; -import org.junit.jupiter.params.provider.ValueSource; - -import java.time.LocalDateTime; -import java.time.format.DateTimeFormatter; -import java.util.Collections; -import java.util.Date; - -class ExpirationDateUpdateStrategyTest { - private final static String CURRENT_DATE = dateToString(new Date()); - private final ExpirationDateUpdateStrategy updateStrategy = new ExpirationDateUpdateStrategy(); - - @ParameterizedTest - @ValueSource(booleans = {true, false}) - void shouldUpdateExpirationDateAndActiveValue(boolean isActive) { - var userFormat = new UserFormat().withExpirationDate(CURRENT_DATE); - var newDate = isActive ? - dateToString(Date.from(LocalDateTime.now().plusDays(1).atZone(UTC).toInstant())) : - dateToString(Date.from(LocalDateTime.now().minusDays(1).atZone(UTC).toInstant())); - var contentUpdate = new UserContentUpdate() - .option(UserContentUpdate.OptionEnum.EXPIRATION_DATE) - .actions(Collections.singletonList(new UserContentUpdateAction() - .name(UserContentUpdateAction.NameEnum.REPLACE_WITH) - .value(newDate))); - var updatedUserFormat = updateStrategy.applyUpdate(userFormat, contentUpdate); - - assertThat(updatedUserFormat.getExpirationDate(), equalTo(newDate)); - assertThat(updatedUserFormat.getActive(), equalTo(Boolean.toString(isActive))); - } - - @Test - void shouldNotUpdateExpirationDateIfValuesAreEqual() { - var userFormat = new UserFormat().withActive("false").withExpirationDate(CURRENT_DATE); - var contentUpdate = new UserContentUpdate() - .option(UserContentUpdate.OptionEnum.EXPIRATION_DATE) - .actions(Collections.singletonList(new UserContentUpdateAction() - .name(UserContentUpdateAction.NameEnum.REPLACE_WITH) - .value(CURRENT_DATE))); - - assertThat(userFormat, equalTo(updateStrategy.applyUpdate(userFormat, contentUpdate))); - } -} diff --git a/src/test/java/org/folio/dew/service/update/PatronGroupUpdateStrategyTest.java b/src/test/java/org/folio/dew/service/update/PatronGroupUpdateStrategyTest.java deleted file mode 100644 index e1da22c60..000000000 --- a/src/test/java/org/folio/dew/service/update/PatronGroupUpdateStrategyTest.java +++ /dev/null @@ -1,42 +0,0 @@ -package org.folio.dew.service.update; - -import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.equalTo; - -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.domain.dto.UserContentUpdateAction; -import org.folio.dew.domain.dto.UserFormat; -import org.junit.jupiter.api.Test; - -import java.util.Collections; - -class PatronGroupUpdateStrategyTest { - private final static String NEW_VALUE = "New patron group"; - private final PatronGroupUpdateStrategy updateStrategy = new PatronGroupUpdateStrategy(); - - @Test - void shouldUpdatePatronGroup() { - var userFormat = new UserFormat().withPatronGroup("Patron group"); - var contentUpdate = new UserContentUpdate() - .option(UserContentUpdate.OptionEnum.PATRON_GROUP) - .actions(Collections.singletonList(new UserContentUpdateAction() - .name(UserContentUpdateAction.NameEnum.REPLACE_WITH) - .value(NEW_VALUE))); - - var updatedUserFormat = updateStrategy.applyUpdate(userFormat, contentUpdate); - - assertThat(updatedUserFormat.getPatronGroup(), equalTo(NEW_VALUE)); - } - - @Test - void shouldNotUpdatePatronGroupIfValuesAreEqual() { - var userFormat = new UserFormat().withPatronGroup(NEW_VALUE); - var contentUpdate = new UserContentUpdate() - .option(UserContentUpdate.OptionEnum.PATRON_GROUP) - .actions(Collections.singletonList(new UserContentUpdateAction() - .name(UserContentUpdateAction.NameEnum.REPLACE_WITH) - .value(NEW_VALUE))); - - assertThat(userFormat, equalTo(updateStrategy.applyUpdate(userFormat, contentUpdate))); - } -} diff --git a/src/test/java/org/folio/dew/service/validation/HoldingsContentUpdateValidatorServiceTest.java b/src/test/java/org/folio/dew/service/validation/HoldingsContentUpdateValidatorServiceTest.java deleted file mode 100644 index dfa309ff1..000000000 --- a/src/test/java/org/folio/dew/service/validation/HoldingsContentUpdateValidatorServiceTest.java +++ /dev/null @@ -1,107 +0,0 @@ -package org.folio.dew.service.validation; - -import static org.folio.dew.domain.dto.HoldingsContentUpdate.ActionEnum.CLEAR_FIELD; -import static org.folio.dew.domain.dto.HoldingsContentUpdate.ActionEnum.REPLACE_WITH; -import static org.folio.dew.domain.dto.HoldingsContentUpdate.OptionEnum.PERMANENT_LOCATION; -import static org.folio.dew.domain.dto.HoldingsContentUpdate.OptionEnum.TEMPORARY_LOCATION; -import static org.junit.jupiter.api.Assertions.assertThrows; -import static org.junit.jupiter.api.Assertions.assertTrue; - -import lombok.AllArgsConstructor; -import lombok.Getter; -import org.folio.dew.BaseBatchTest; -import org.folio.dew.domain.dto.HoldingsContentUpdate; -import org.folio.dew.domain.dto.HoldingsContentUpdateCollection; -import org.folio.dew.error.ContentUpdateValidationException; -import org.junit.jupiter.params.ParameterizedTest; -import org.junit.jupiter.params.provider.EnumSource; -import org.springframework.beans.factory.annotation.Autowired; - -import java.util.Collections; -import java.util.List; - -class HoldingsContentUpdateValidatorServiceTest extends BaseBatchTest { - @Autowired - private HoldingsContentUpdateValidatorService validatorService; - - @ParameterizedTest - @EnumSource(HoldingsContentUpdateValidTestData.class) - void shouldAllowValidContentUpdateData(HoldingsContentUpdateValidTestData testData) { - assertTrue(validatorService.validateContentUpdateCollection(testData.getUpdateCollection())); - } - - @ParameterizedTest - @EnumSource(HoldingsContentUpdateInvalidTestData.class) - void shouldRejectInvalidContentUpdateData(HoldingsContentUpdateInvalidTestData testData) { - var updates = testData.getUpdateCollection(); - assertThrows(ContentUpdateValidationException.class, () -> validatorService.validateContentUpdateCollection(updates)); - } - - @AllArgsConstructor - @Getter - enum HoldingsContentUpdateValidTestData { - REPLACE_WITH_PERMANENT_LOCATION(new HoldingsContentUpdateCollection() - .holdingsContentUpdates(Collections.singletonList(new HoldingsContentUpdate() - .option(PERMANENT_LOCATION) - .action(REPLACE_WITH) - .value("Annex"))) - .totalRecords(1)), - REPLACE_WITH_TEMPORARY_LOCATION(new HoldingsContentUpdateCollection() - .holdingsContentUpdates(Collections.singletonList(new HoldingsContentUpdate() - .option(TEMPORARY_LOCATION) - .action(REPLACE_WITH) - .value("Annex"))) - .totalRecords(1)), - CLEAR_FIELD_TEMPORARY_LOCATION(new HoldingsContentUpdateCollection() - .holdingsContentUpdates(Collections.singletonList(new HoldingsContentUpdate() - .option(TEMPORARY_LOCATION) - .action(CLEAR_FIELD))) - .totalRecords(1)), - REPLACE_PERMANENT_LOCATION_CLEAR_TEMPORARY_LOCATION(new HoldingsContentUpdateCollection() - .holdingsContentUpdates(List.of(new HoldingsContentUpdate() - .option(PERMANENT_LOCATION) - .action(REPLACE_WITH) - .value("Annex"), - new HoldingsContentUpdate() - .option(TEMPORARY_LOCATION) - .action(CLEAR_FIELD))) - .totalRecords(2)); - - final HoldingsContentUpdateCollection updateCollection; - } - - @AllArgsConstructor - @Getter - enum HoldingsContentUpdateInvalidTestData { - REPLACE_WITH_NON_EXISTING_PERMANENT_LOCATION(new HoldingsContentUpdateCollection() - .holdingsContentUpdates(Collections.singletonList(new HoldingsContentUpdate() - .option(PERMANENT_LOCATION) - .action(REPLACE_WITH) - .value("Abc"))) - .totalRecords(1)), - REPLACE_WITH_NON_EXISTING_TEMPORARY_LOCATION(new HoldingsContentUpdateCollection() - .holdingsContentUpdates(Collections.singletonList(new HoldingsContentUpdate() - .option(TEMPORARY_LOCATION) - .action(REPLACE_WITH) - .value("Abc"))) - .totalRecords(1)), - CLEAR_FIELD_PERMANENT_LOCATION(new HoldingsContentUpdateCollection() - .holdingsContentUpdates(Collections.singletonList(new HoldingsContentUpdate() - .option(PERMANENT_LOCATION) - .action(CLEAR_FIELD))) - .totalRecords(1)), - REPLACE_TEMPORARY_LOCATION_CLEAR_PERMANENT_LOCATION(new HoldingsContentUpdateCollection() - .holdingsContentUpdates(List.of(new HoldingsContentUpdate() - .option(TEMPORARY_LOCATION) - .action(REPLACE_WITH) - .value("Annex"), - new HoldingsContentUpdate() - .option(PERMANENT_LOCATION) - .action(CLEAR_FIELD))) - .totalRecords(2)); - - final HoldingsContentUpdateCollection updateCollection; - } -} - - diff --git a/src/test/java/org/folio/dew/service/validation/UserContentUpdateInvalidTestData.java b/src/test/java/org/folio/dew/service/validation/UserContentUpdateInvalidTestData.java deleted file mode 100644 index 239ed8242..000000000 --- a/src/test/java/org/folio/dew/service/validation/UserContentUpdateInvalidTestData.java +++ /dev/null @@ -1,192 +0,0 @@ -package org.folio.dew.service.validation; - -import static org.folio.dew.domain.dto.UserContentUpdate.OptionEnum.EMAIL_ADDRESS; -import static org.folio.dew.domain.dto.UserContentUpdate.OptionEnum.EXPIRATION_DATE; -import static org.folio.dew.domain.dto.UserContentUpdate.OptionEnum.PATRON_GROUP; -import static org.folio.dew.domain.dto.UserContentUpdateAction.NameEnum.ADD_TO_EXISTING; -import static org.folio.dew.domain.dto.UserContentUpdateAction.NameEnum.CLEAR_FIELD; -import static org.folio.dew.domain.dto.UserContentUpdateAction.NameEnum.FIND; -import static org.folio.dew.domain.dto.UserContentUpdateAction.NameEnum.FIND_AND_REMOVE_THESE; -import static org.folio.dew.domain.dto.UserContentUpdateAction.NameEnum.REPLACE_WITH; - -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.domain.dto.UserContentUpdateAction; - -import java.util.Collections; -import java.util.List; - -public enum UserContentUpdateInvalidTestData { - PATRON_GROUP_REPLACE_WITH_EMPTY_VALUE( - new UserContentUpdate() - .option(PATRON_GROUP) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(REPLACE_WITH)))), - PATRON_GROUP_REPLACE_WITH_NON_EXISTING_VALUE( - new UserContentUpdate() - .option(PATRON_GROUP) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(REPLACE_WITH) - .value("non-existing group") - )) - ), - PATRON_GROUP_CLEAR_FIELD( - new UserContentUpdate() - .option(PATRON_GROUP) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(CLEAR_FIELD)))), - PATRON_GROUP_FIND( - new UserContentUpdate() - .option(PATRON_GROUP) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(FIND) - .value("find value")))), - PATRON_GROUP_FIND_AND_REPLACE_WITH( - new UserContentUpdate() - .option(PATRON_GROUP) - .actions(List.of( - new UserContentUpdateAction() - .name(FIND) - .value("find value"), - new UserContentUpdateAction() - .name(REPLACE_WITH) - .value("new value")))), - PATRON_GROUP_ADD_TO_EXISTING( - new UserContentUpdate() - .option(PATRON_GROUP) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(ADD_TO_EXISTING) - .value("find value")))), - PATRON_GROUP_FIND_AND_REMOVE_THESE( - new UserContentUpdate() - .option(PATRON_GROUP) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(FIND_AND_REMOVE_THESE) - .value("find value")))), - EXPIRATION_DATE_REPLACE_WITH_INVALID_DATE( - new UserContentUpdate() - .option(EXPIRATION_DATE) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(REPLACE_WITH) - .value("2022/01/01")))), - EXPIRATION_DATE_REPLACE_WITH_EMPTY_VALUE( - new UserContentUpdate() - .option(EXPIRATION_DATE) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(REPLACE_WITH)))), - EXPIRATION_DATE_CLEAR_FIELD( - new UserContentUpdate() - .option(EXPIRATION_DATE) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(CLEAR_FIELD)))), - EXPIRATION_DATE_FIND( - new UserContentUpdate() - .option(EXPIRATION_DATE) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(FIND) - .value("find value")))), - EXPIRATION_DATE_FIND_AND_REPLACE_WITH( - new UserContentUpdate() - .option(EXPIRATION_DATE) - .actions(List.of( - new UserContentUpdateAction() - .name(FIND) - .value("find value"), - new UserContentUpdateAction() - .name(REPLACE_WITH) - .value("new value")))), - EXPIRATION_DATE_ADD_TO_EXISTING( - new UserContentUpdate() - .option(EXPIRATION_DATE) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(ADD_TO_EXISTING) - .value("find value")))), - EXPIRATION_DATE_AND_REMOVE_THESE( - new UserContentUpdate() - .option(EXPIRATION_DATE) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(FIND_AND_REMOVE_THESE) - .value("find value")))), - EMAIL_REPLACE_WITH_EMPTY_VALUE( - new UserContentUpdate() - .option(EMAIL_ADDRESS) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(REPLACE_WITH)))), - EMAIL_CLEAR_FIELD( - new UserContentUpdate() - .option(EMAIL_ADDRESS) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(CLEAR_FIELD)))), - EMAIL_FIND( - new UserContentUpdate() - .option(EMAIL_ADDRESS) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(FIND) - .value("find value")))), - EMAIL_ADD_TO_EXISTING( - new UserContentUpdate() - .option(EMAIL_ADDRESS) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(ADD_TO_EXISTING) - .value("find value")))), - EMAIL_FIND_AND_REMOVE_THESE( - new UserContentUpdate() - .option(EMAIL_ADDRESS) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(FIND_AND_REMOVE_THESE) - .value("find value")))), - EMAIL_FIND_AND_AND_REPLACE_WITH_EMPTY_FIND( - new UserContentUpdate() - .option(EMAIL_ADDRESS) - .actions(List.of( - new UserContentUpdateAction() - .name(FIND), - new UserContentUpdateAction() - .name(REPLACE_WITH) - .value("replace value")))), - EMAIL_FIND_AND_AND_REPLACE_WITH_EMPTY_REPLACE( - new UserContentUpdate() - .option(EMAIL_ADDRESS) - .actions(List.of( - new UserContentUpdateAction() - .name(FIND) - .value("find value"), - new UserContentUpdateAction() - .name(REPLACE_WITH)))), - EMAIL_FIND_AND_AND_REPLACE_WITH_EQUAL_VALUES( - new UserContentUpdate() - .option(EMAIL_ADDRESS) - .actions(List.of( - new UserContentUpdateAction() - .name(FIND) - .value("some value"), - new UserContentUpdateAction() - .name(REPLACE_WITH) - .value("some value")))); - - private UserContentUpdate update; - - UserContentUpdateInvalidTestData(UserContentUpdate update) { - this.update = update; - } - - public UserContentUpdate getUpdate() { - return update; - } -} diff --git a/src/test/java/org/folio/dew/service/validation/UserContentUpdateValidTestData.java b/src/test/java/org/folio/dew/service/validation/UserContentUpdateValidTestData.java deleted file mode 100644 index aa8de6c88..000000000 --- a/src/test/java/org/folio/dew/service/validation/UserContentUpdateValidTestData.java +++ /dev/null @@ -1,51 +0,0 @@ -package org.folio.dew.service.validation; - -import static org.folio.dew.domain.dto.UserContentUpdate.OptionEnum.EMAIL_ADDRESS; -import static org.folio.dew.domain.dto.UserContentUpdate.OptionEnum.EXPIRATION_DATE; -import static org.folio.dew.domain.dto.UserContentUpdate.OptionEnum.PATRON_GROUP; -import static org.folio.dew.domain.dto.UserContentUpdateAction.NameEnum.CLEAR_FIELD; -import static org.folio.dew.domain.dto.UserContentUpdateAction.NameEnum.FIND; -import static org.folio.dew.domain.dto.UserContentUpdateAction.NameEnum.REPLACE_WITH; - -import org.folio.dew.domain.dto.UserContentUpdate; -import org.folio.dew.domain.dto.UserContentUpdateAction; - -import java.util.Collections; -import java.util.List; - -public enum UserContentUpdateValidTestData { - EXPIRATION_DATE_REPLACE_WITH( - new UserContentUpdate() - .option(EXPIRATION_DATE) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(REPLACE_WITH) - .value("2022-01-01 12:00:00.123Z")))), - PATRON_GROUP_REPLACE_WITH( - new UserContentUpdate() - .option(PATRON_GROUP) - .actions(Collections.singletonList( - new UserContentUpdateAction() - .name(REPLACE_WITH) - .value("PatronGroup")))), - EMAIL_FIND_AND_REPLACE_WITH( - new UserContentUpdate() - .option(EMAIL_ADDRESS) - .actions(List.of( - new UserContentUpdateAction() - .name(FIND) - .value("find value"), - new UserContentUpdateAction() - .name(REPLACE_WITH) - .value("new value")))); - - private UserContentUpdate update; - - UserContentUpdateValidTestData(UserContentUpdate update) { - this.update = update; - } - - public UserContentUpdate getUpdate() { - return update; - } -} diff --git a/src/test/java/org/folio/dew/service/validation/UserContentUpdateValidatorServiceTest.java b/src/test/java/org/folio/dew/service/validation/UserContentUpdateValidatorServiceTest.java deleted file mode 100644 index 3d0e68c43..000000000 --- a/src/test/java/org/folio/dew/service/validation/UserContentUpdateValidatorServiceTest.java +++ /dev/null @@ -1,37 +0,0 @@ -package org.folio.dew.service.validation; - -import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.is; -import static org.junit.jupiter.api.Assertions.assertThrows; - -import java.util.Collections; -import org.folio.dew.BaseBatchTest; -import org.folio.dew.domain.dto.UserContentUpdateCollection; -import org.folio.dew.error.ContentUpdateValidationException; -import org.junit.jupiter.params.ParameterizedTest; -import org.junit.jupiter.params.provider.EnumSource; -import org.springframework.beans.factory.annotation.Autowired; - -class UserContentUpdateValidatorServiceTest extends BaseBatchTest { - @Autowired - private UserContentUpdateValidatorService validatorService; - @ParameterizedTest - @EnumSource(UserContentUpdateValidTestData.class) - void shouldAllowValidContentUpdates(UserContentUpdateValidTestData update) { - var updateCollection = new UserContentUpdateCollection() - .userContentUpdates(Collections.singletonList(update.getUpdate())) - .totalRecords(1); - - assertThat(validatorService.validateContentUpdateCollection(updateCollection), is(true)); - } - - @ParameterizedTest - @EnumSource(UserContentUpdateInvalidTestData.class) - void shouldRejectInvalidContentUpdates(UserContentUpdateInvalidTestData update) { - var updateCollection = new UserContentUpdateCollection() - .userContentUpdates(Collections.singletonList(update.getUpdate())) - .totalRecords(1); - - assertThrows(ContentUpdateValidationException.class, () -> validatorService.validateContentUpdateCollection(updateCollection)); - } -} diff --git a/src/test/resources/application.yml b/src/test/resources/application.yml index 241add083..34d5a2878 100644 --- a/src/test/resources/application.yml +++ b/src/test/resources/application.yml @@ -29,19 +29,21 @@ application: size: ${BUCKET_SIZE:50} minio-remote: endpoint: http://${embedded.minio.host}:${embedded.minio.port}/ - region: ${AWS_REGION:region} - bucket: ${AWS_BUCKET:remote-files-test} - accessKey: ${AWS_ACCESS_KEY_ID:AKIAIOSFODNN7EXAMPLE} - secretKey: ${AWS_SECRET_ACCESS_KEY:wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY} - composeWithAwsSdk: ${LOCAL_FS_COMPOSE_WITH_AWS_SDK:false} + region: ${S3_REGION:region} + bucket: ${S3_BUCKET:files-test} + accessKey: ${S3_ACCESS_KEY_ID:AKIAIOSFODNN7EXAMPLE} + secretKey: ${S3_SECRET_ACCESS_KEY:wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY} + composeWithAwsSdk: ${S3_IS_AWS:false} + subPath: ${S3_SUB_PATH:remote} url-expiration-time-in-seconds: ${URL_EXPIRATION_TIME:604800} # 7 days minio-local: endpoint: http://${embedded.minio.host}:${embedded.minio.port}/ - region: ${LOCAL_FS_REGION:region} - bucket: ${LOCAL_FS_BUCKET:local-files-test} - accessKey: ${LOCAL_FS_ACCESS_KEY_ID:AKIAIOSFODNN7EXAMPLE} - secretKey: ${LOCAL_FS_SECRET_ACCESS_KEY:wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY} - composeWithAwsSdk: ${LOCAL_FS_COMPOSE_WITH_AWS_SDK:false} + region: ${S3_REGION:region} + bucket: ${S3_BUCKET:files-test} + accessKey: ${S3_ACCESS_KEY_ID:AKIAIOSFODNN7EXAMPLE} + secretKey: ${S3_SECRET_ACCESS_KEY:wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY} + composeWithAwsSdk: ${S3_IS_AWS:false} + subPath: ${S3_LOCAL_SUB_PATH:local} url-expiration-time-in-seconds: ${URL_EXPIRATION_TIME:604800} # 7 days e-holdings-batch: job-chunk-size: 2 @@ -49,6 +51,11 @@ application: authority-control-batch: job-chunk-size: 2 entities-links-chunk-size: 2 + chunks: ${CHUNKS:100} + core-pool-size: ${CORE_POOL_SIZE:10} + max-pool-size: ${MAX_POOL_SIZE:10} + platform: ${PLATFORM:okapi} + folio: tenant: validation: diff --git a/src/test/resources/mapper/holdings_record_full.json b/src/test/resources/mapper/holdings_record_full.json index 20e87064e..464b6afe9 100644 --- a/src/test/resources/mapper/holdings_record_full.json +++ b/src/test/resources/mapper/holdings_record_full.json @@ -1 +1 @@ -{"id":"0b1e3760-f689-493e-a98e-9cc9dadb7e83","_version":1,"hrid":"ho13","holdingsTypeId":"0c422f92-0f4d-4d32-8cbe-390ebc33a3e5","formerIds":["d1670310-ceac-47d9-aaba-aaeeb890bc07"],"instanceId":"5bf370e0-8cca-4d9c-82e4-5170ab2a0a39","permanentLocationId":"53cf956f-c1df-410b-8bea-27f712cca7c0","permanentLocation":null,"temporaryLocationId":"fcd64ce1-6995-48f0-840e-89ffa2288371","effectiveLocationId":"fcd64ce1-6995-48f0-840e-89ffa2288371","electronicAccess":[{"uri":"www.someurl.com","linkText":"link text","materialsSpecification":"material","publicNote":"www.someurl.com","relationshipId":"3b430592-2e09-4b48-9a0c-0636d66b9fb3"}],"callNumberTypeId":"512173a7-bd09-490e-b773-17d83f2b63fe","callNumberPrefix":"prefix","callNumber":"call number","callNumberSuffix":"suffix","shelvingTitle":"shelving title","acquisitionFormat":"order format","acquisitionMethod":"aquisition method","receiptStatus":"receipt status","administrativeNotes":["Test administrative note"],"notes":[{"holdingsNoteTypeId":"b160f13a-ddba-4053-b9c4-60ec5ea45d56","holdingsNoteType":null,"note":"a note","staffOnly":false},{"holdingsNoteTypeId":"b160f13a-ddba-4053-b9c4-60ec5ea45d56","holdingsNoteType":null,"note":"a note","staffOnly":true}],"illPolicyId":"9e49924b-f649-4b36-ab57-e66e639a9b0e","illPolicy":null,"retentionPolicy":"retention policy","digitizationPolicy":"digitisation policy","holdingsStatements":[{"statement":"statement","note":"statement public note","staffNote":"statement staff note"}],"holdingsStatementsForIndexes":[{"statement":"statement for indexes","note":"statement for indexes public note","staffNote":"statement for indexes staff note"}],"holdingsStatementsForSupplements":[{"statement":"statement for supplements","note":"statement for supplements public note","staffNote":"statement for supplements staff note"}],"copyNumber":"copy number","numberOfItems":"10","receivingHistory":{"displayType":null,"entries":[{"publicDisplay":true,"enumeration":"enum","chronology":"chronology"},{"publicDisplay":null,"enumeration":"enum2","chronology":"chronology2"}]},"discoverySuppress":null,"statisticalCodeIds":["b5968c9e-cddc-4576-99e3-8e60aed8b0dd"],"tags":null,"metadata":null,"sourceId":"f32d531e-df79-46b3-8932-cdd35f7a2264"} +{"id":"0b1e3760-f689-493e-a98e-9cc9dadb7e83","_version":1,"hrid":"ho13","holdingsTypeId":"0c422f92-0f4d-4d32-8cbe-390ebc33a3e5","formerIds":["d1670310-ceac-47d9-aaba-aaeeb890bc07"],"instanceId":"5bf370e0-8cca-4d9c-82e4-5170ab2a0a39","permanentLocationId":"53cf956f-c1df-410b-8bea-27f712cca7c0","permanentLocation":null,"temporaryLocationId":"fcd64ce1-6995-48f0-840e-89ffa2288371","effectiveLocationId":"fcd64ce1-6995-48f0-840e-89ffa2288371","electronicAccess":[{"uri":"www.someurl.com","linkText":"link text","materialsSpecification":"material","publicNote":"www.someurl.com","relationshipId":"3b430592-2e09-4b48-9a0c-0636d66b9fb3"}],"callNumberTypeId":"512173a7-bd09-490e-b773-17d83f2b63fe","callNumberPrefix":"prefix","callNumber":"call number","callNumberSuffix":"suffix","shelvingTitle":"shelving title","acquisitionFormat":"order format","acquisitionMethod":"aquisition method","receiptStatus":"receipt status","administrativeNotes":["Test administrative note"],"notes":[{"holdingsNoteTypeId":"b160f13a-ddba-4053-b9c4-60ec5ea45d56","holdingsNoteType":null,"note":"a note","staffOnly":false},{"holdingsNoteTypeId":"b160f13a-ddba-4053-b9c4-60ec5ea45d56","holdingsNoteType":null,"note":"a note","staffOnly":true}],"illPolicyId":"9e49924b-f649-4b36-ab57-e66e639a9b0e","illPolicy":null,"retentionPolicy":"retention policy","digitizationPolicy":"digitisation policy","holdingsStatements":[{"statement":"statement","note":"statement public note","staffNote":"statement staff note"}],"holdingsStatementsForIndexes":[{"statement":"statement for indexes","note":"statement for indexes public note","staffNote":"statement for indexes staff note"}],"holdingsStatementsForSupplements":[{"statement":"statement for supplements","note":"statement for supplements public note","staffNote":"statement for supplements staff note"}],"copyNumber":"copy number","numberOfItems":"10","receivingHistory":{"displayType":null,"entries":[{"publicDisplay":true,"enumeration":"enum","chronology":"chronology"},{"publicDisplay":null,"enumeration":"enum2","chronology":"chronology2"}]},"discoverySuppress":null,"statisticalCodeIds":["b5968c9e-cddc-4576-99e3-8e60aed8b0dd"],"tags":null,"metadata":null} diff --git a/src/test/resources/mapper/holdings_record_min.json b/src/test/resources/mapper/holdings_record_min.json index ec14ec117..6b2702cf4 100644 --- a/src/test/resources/mapper/holdings_record_min.json +++ b/src/test/resources/mapper/holdings_record_min.json @@ -1 +1 @@ -{"id":"855d4693-4087-4339-8c14-c25c350e5e59","_version":null,"hrid":"ho14","holdingsTypeId":null,"formerIds":[],"instanceId":"5bf370e0-8cca-4d9c-82e4-5170ab2a0a39","permanentLocationId":"fcd64ce1-6995-48f0-840e-89ffa2288371","permanentLocation":null,"temporaryLocationId":null,"effectiveLocationId":"fcd64ce1-6995-48f0-840e-89ffa2288371","electronicAccess":[],"callNumberTypeId":null,"callNumberPrefix":null,"callNumber":null,"callNumberSuffix":null,"shelvingTitle":null,"acquisitionFormat":null,"acquisitionMethod":null,"receiptStatus":null,"administrativeNotes":[],"notes":[],"illPolicyId":null,"illPolicy":null,"retentionPolicy":null,"digitizationPolicy":null,"holdingsStatements":[],"holdingsStatementsForIndexes":[],"holdingsStatementsForSupplements":[],"copyNumber":null,"numberOfItems":null,"receivingHistory":null,"discoverySuppress":null,"statisticalCodeIds":[],"tags":null,"metadata":null,"sourceId":"f32d531e-df79-46b3-8932-cdd35f7a2264"} +{"id":"855d4693-4087-4339-8c14-c25c350e5e59","_version":null,"hrid":"ho14","holdingsTypeId":null,"formerIds":[],"instanceId":"5bf370e0-8cca-4d9c-82e4-5170ab2a0a39","permanentLocationId":"fcd64ce1-6995-48f0-840e-89ffa2288371","permanentLocation":null,"temporaryLocationId":null,"effectiveLocationId":"fcd64ce1-6995-48f0-840e-89ffa2288371","electronicAccess":[],"callNumberTypeId":null,"callNumberPrefix":null,"callNumber":null,"callNumberSuffix":null,"shelvingTitle":null,"acquisitionFormat":null,"acquisitionMethod":null,"receiptStatus":null,"administrativeNotes":[],"notes":[],"illPolicyId":null,"illPolicy":null,"retentionPolicy":null,"digitizationPolicy":null,"holdingsStatements":[],"holdingsStatementsForIndexes":[],"holdingsStatementsForSupplements":[],"copyNumber":null,"numberOfItems":null,"receivingHistory":null,"discoverySuppress":null,"statisticalCodeIds":[],"tags":null,"metadata":null} diff --git a/src/test/resources/mappings/instances-query.json b/src/test/resources/mappings/instances-query.json index 990dc524f..c1025977e 100644 --- a/src/test/resources/mappings/instances-query.json +++ b/src/test/resources/mappings/instances-query.json @@ -7,7 +7,7 @@ }, "response": { "status": 200, - "body": "{\n \"instances\": [\n {\n \"id\": \"549fad9e-7f8e-4d8e-9a71-00d251817866\",\n \"_version\": \"2\",\n \"hrid\": \"inst000000000028\",\n \"source\": \"FOLIO\",\n \"title\": \"Agile Organisation, Risiko- und Change Management Harald Augustin (Hrsg.)\",\n \"administrativeNotes\": [],\n \"parentInstances\": [\n {\n \"id\": \"1b449f40-5ae8-47df-9113-3c0a958b5ce8\",\n \"superInstanceId\": \"a317b304-528c-424f-961c-39174933b454\",\n \"instanceRelationshipTypeId\": \"a17daf0a-f057-43b3-9997-13d0724cdf51\"\n }\n ],\n \"childInstances\": [],\n \"isBoundWith\": false,\n \"alternativeTitles\": [],\n \"editions\": [],\n \"series\": [\n {\n \"authorityId\": null,\n \"value\": \"Umsetzung der DIN EN ISO 9001:2015 / Harald Augustin (Hrsg.) ; Band 2\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Berichte aus der Betriebswirtschaft\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Umsetzung der DIN EN ISO 9001:2015 Band 2\"\n }\n ],\n \"identifiers\": [\n {\n \"identifierTypeId\": \"8261054f-be78-422d-bd51-4ed9f33c3422\",\n \"value\": \"3844057439\"\n },\n {\n \"identifierTypeId\": \"8261054f-be78-422d-bd51-4ed9f33c3422\",\n \"value\": \"9783844057430\"\n },\n {\n \"identifierTypeId\": \"2e8b3b6c-0e7d-4e48-bca2-b0b23b376af5\",\n \"value\": \"9783844057430\"\n },\n {\n \"identifierTypeId\": \"7e591197-f335-4afb-bc6d-a6d76ca3bace\",\n \"value\": \"(OCoLC)1024128245\"\n },\n {\n \"identifierTypeId\": \"7e591197-f335-4afb-bc6d-a6d76ca3bace\",\n \"value\": \"(DE-101)1150176652\"\n },\n {\n \"identifierTypeId\": \"7e591197-f335-4afb-bc6d-a6d76ca3bace\",\n \"value\": \"(DE-599)DNB1150176652\"\n }\n ],\n \"contributors\": [\n {\n \"authorityId\": null,\n \"contributorNameTypeId\": \"2b94c631-fca9-4892-a730-03ee529ffe2a\",\n \"name\": \"Augustin, Harald\",\n \"contributorTypeId\": null,\n \"contributorTypeText\": null,\n \"primary\": null\n },\n {\n \"authorityId\": null,\n \"contributorNameTypeId\": \"2e48e713-17f3-4c13-a9f8-23845bb210aa\",\n \"name\": \"Shaker Verlag GmbH\",\n \"contributorTypeId\": null,\n \"contributorTypeText\": null,\n \"primary\": null\n }\n ],\n \"subjects\": [\n {\n \"authorityId\": null,\n \"value\": \"DIN EN ISO 9001:2015\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Standard\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Risikomanagement\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Einführung\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Organisatorischer Wandel\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Industrie\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Deutschland\"\n }\n ],\n \"classifications\": [\n {\n \"classificationNumber\": \"658.4013\",\n \"classificationTypeId\": \"42471af9-7d25-4f3a-bf78-60d29dcf463b\"\n },\n {\n \"classificationNumber\": \"650\",\n \"classificationTypeId\": \"42471af9-7d25-4f3a-bf78-60d29dcf463b\"\n }\n ],\n \"publication\": [\n {\n \"publisher\": \"Shaker Verlag\",\n \"place\": \"Aachen\",\n \"dateOfPublication\": \"2017\",\n \"role\": null\n }\n ],\n \"publicationFrequency\": [],\n \"publicationRange\": [],\n \"electronicAccess\": [\n {\n \"uri\": \"http://d-nb.info/1150176652/04\",\n \"linkText\": \"Electronic resource (PDF)\",\n \"materialsSpecification\": null,\n \"publicNote\": \"Address for accessing the table of content. PDF file\",\n \"relationshipId\": null\n }\n ],\n \"instanceTypeId\": \"6312d172-f0cf-40f6-b27d-9fa8feaf332f\",\n \"instanceFormatIds\": [],\n \"physicalDescriptions\": [\n \"x, 148 Seiten Illustrationen 21 cm, 188 g\"\n ],\n \"languages\": [\n \"ger\"\n ],\n \"notes\": [],\n \"previouslyHeld\": false,\n \"staffSuppress\": true,\n \"discoverySuppress\": false,\n \"statisticalCodeIds\": [],\n \"statusUpdatedDate\": \"2023-11-10T11:54:16.884+0000\",\n \"metadata\": {\n \"createdDate\": \"2023-11-10T11:54:16.883+00:00\",\n \"createdByUserId\": \"cffb2565-07fc-470b-86c6-17d8ce14432e\",\n \"updatedDate\": \"2023-12-14T14:33:51.734+00:00\",\n \"updatedByUserId\": \"ca6022de-2644-46fe-b6f2-78df15483721\"\n },\n \"tags\": {\n \"tagList\": []\n },\n \"natureOfContentTermIds\": [],\n \"publicationPeriod\": {\n \"start\": 2017\n },\n \"precedingTitles\": [],\n \"succeedingTitles\": []\n }\n ],\n \"totalRecords\": 1\n}", + "body": "{\n \"instances\": [\n {\n \"id\": \"549fad9e-7f8e-4d8e-9a71-00d251817866\",\n \"_version\": \"2\",\n \"hrid\": \"inst000000000028\",\n \"source\": \"FOLIO\",\n \"title\": \"Agile Organisation, Risiko- und Change Management Harald Augustin (Hrsg.)\",\n \"administrativeNotes\": [],\n \"parentInstances\": [\n {\n \"id\": \"1b449f40-5ae8-47df-9113-3c0a958b5ce8\",\n \"superInstanceId\": \"a317b304-528c-424f-961c-39174933b454\",\n \"instanceRelationshipTypeId\": \"a17daf0a-f057-43b3-9997-13d0724cdf51\"\n }\n ],\n \"childInstances\": [],\n \"isBoundWith\": false,\n \"alternativeTitles\": [],\n \"editions\": [],\n \"series\": [\n {\n \"authorityId\": null,\n \"value\": \"Umsetzung der DIN EN ISO 9001:2015 / Harald Augustin (Hrsg.) ; Band 2\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Berichte aus der Betriebswirtschaft\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Umsetzung der DIN EN ISO 9001:2015 Band 2\"\n }\n ],\n \"identifiers\": [\n {\n \"identifierTypeId\": \"8261054f-be78-422d-bd51-4ed9f33c3422\",\n \"value\": \"3844057439\"\n },\n {\n \"identifierTypeId\": \"8261054f-be78-422d-bd51-4ed9f33c3422\",\n \"value\": \"9783844057430\"\n },\n {\n \"identifierTypeId\": \"2e8b3b6c-0e7d-4e48-bca2-b0b23b376af5\",\n \"value\": \"9783844057430\"\n },\n {\n \"identifierTypeId\": \"7e591197-f335-4afb-bc6d-a6d76ca3bace\",\n \"value\": \"(OCoLC)1024128245\"\n },\n {\n \"identifierTypeId\": \"7e591197-f335-4afb-bc6d-a6d76ca3bace\",\n \"value\": \"(DE-101)1150176652\"\n },\n {\n \"identifierTypeId\": \"7e591197-f335-4afb-bc6d-a6d76ca3bace\",\n \"value\": \"(DE-599)DNB1150176652\"\n }\n ],\n \"contributors\": [\n {\n \"authorityId\": null,\n \"contributorNameTypeId\": \"2b94c631-fca9-4892-a730-03ee529ffe2a\",\n \"name\": \"Augustin, Harald\",\n \"contributorTypeId\": null,\n \"contributorTypeText\": null,\n \"primary\": null\n },\n {\n \"authorityId\": null,\n \"contributorNameTypeId\": \"2e48e713-17f3-4c13-a9f8-23845bb210aa\",\n \"name\": \"Shaker Verlag GmbH\",\n \"contributorTypeId\": null,\n \"contributorTypeText\": null,\n \"primary\": null\n }\n ],\n \"subjects\": [\n {\n \"authorityId\": null,\n \"value\": \"DIN EN ISO 9001:2015\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Standard\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Risikomanagement\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Einführung\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Organisatorischer Wandel\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Industrie\"\n },\n {\n \"authorityId\": null,\n \"value\": \"Deutschland\"\n }\n ],\n \"classifications\": [\n {\n \"classificationNumber\": \"658.4013\",\n \"classificationTypeId\": \"42471af9-7d25-4f3a-bf78-60d29dcf463b\"\n },\n {\n \"classificationNumber\": \"650\",\n \"classificationTypeId\": \"42471af9-7d25-4f3a-bf78-60d29dcf463b\"\n }\n ],\n \"publication\": [\n {\n \"publisher\": \"Shaker Verlag\",\n \"place\": \"Aachen\",\n \"dateOfPublication\": \"2017\",\n \"role\": null\n }\n ],\n \"publicationFrequency\": [],\n \"publicationRange\": [],\n \"electronicAccess\": [\n {\n \"uri\": \"http://d-nb.info/1150176652/04\",\n \"linkText\": \"Electronic resource (PDF)\",\n \"materialsSpecification\": null,\n \"publicNote\": \"Address for accessing the table of content. PDF file\",\n \"relationshipId\": null\n }\n ],\n \"instanceTypeId\": \"6312d172-f0cf-40f6-b27d-9fa8feaf332f\",\n \"instanceFormatIds\": [],\n \"physicalDescriptions\": [\n \"x, 148 Seiten Illustrationen 21 cm, 188 g\"\n ],\n \"languages\": [\n \"ger\"\n ],\n \"notes\": [],\n \"previouslyHeld\": false,\n \"staffSuppress\": true,\n \"discoverySuppress\": false,\n \"statisticalCodeIds\": [],\n \"statusUpdatedDate\": \"2023-11-10T11:54:16.884+0000\",\n \"metadata\": {\n \"createdDate\": \"2023-11-10T11:54:16.883+00:00\",\n \"createdByUserId\": \"cffb2565-07fc-470b-86c6-17d8ce14432e\",\n \"updatedDate\": \"2023-12-14T14:33:51.734+00:00\",\n \"updatedByUserId\": \"ca6022de-2644-46fe-b6f2-78df15483721\"\n },\n \"tags\": {\n \"tagList\": []\n },\n \"natureOfContentTermIds\": [],\n \"precedingTitles\": [],\n \"succeedingTitles\": []\n }\n ],\n \"totalRecords\": 1\n}", "headers": { "Content-Type": "application/json" } diff --git a/src/test/resources/mappings/permissionsSelfCheck.json b/src/test/resources/mappings/permissionsSelfCheck.json new file mode 100644 index 000000000..fae73cc9b --- /dev/null +++ b/src/test/resources/mappings/permissionsSelfCheck.json @@ -0,0 +1,17 @@ +{ + "mappings": [ + { + "request": { + "method": "GET", + "url": "/bulk-edit/permissions-self-check" + }, + "response": { + "status": 200, + "body": "[\"users.item.get\", \"inventory.items.item.get\", \"inventory-storage.holdings.item.get\", \"inventory.instances.item.get\", \"bulk-operations.item.inventory.get\", \"bulk-operations.item.users.get\"]", + "headers": { + "Content-Type": "application/json" + } + } + } + ] +} diff --git a/src/test/resources/mappings/srs-record-invalid-content.json b/src/test/resources/mappings/srs-record-invalid-content.json index 53d41aa15..74b2beb49 100644 --- a/src/test/resources/mappings/srs-record-invalid-content.json +++ b/src/test/resources/mappings/srs-record-invalid-content.json @@ -3,7 +3,7 @@ { "request": { "method": "GET", - "url": "/source-storage/source-records?instanceId=7772796a-b88b-4991-a9f7-2e368217c487&idType=INSTANCE" + "url": "/source-storage/source-records?instanceId=7772796a-b88b-4991-a9f7-2e368217c487&idType=INSTANCE&deleted=true" }, "response": { "status": 200, diff --git a/src/test/resources/mappings/srs-records.json b/src/test/resources/mappings/srs-records.json index 1c8c96768..354688be8 100644 --- a/src/test/resources/mappings/srs-records.json +++ b/src/test/resources/mappings/srs-records.json @@ -3,7 +3,7 @@ { "request": { "method": "GET", - "url": "/source-storage/source-records?instanceId=6662796a-b88b-4991-a9f7-2e368217c487&idType=INSTANCE" + "url": "/source-storage/source-records?instanceId=6662796a-b88b-4991-a9f7-2e368217c487&idType=INSTANCE&deleted=true" }, "response": { "status": 200, diff --git a/src/test/resources/mappings/users.json b/src/test/resources/mappings/users.json index 3e4ba8576..30b0b8f24 100644 --- a/src/test/resources/mappings/users.json +++ b/src/test/resources/mappings/users.json @@ -68,7 +68,7 @@ { "request": { "method": "GET", - "url": "/users?query=barcode%3D%3D%22123%22&limit=1" + "url": "/users?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20barcode%3D%3D%22123%22&limit=1" }, "response": { "status": 200, @@ -81,7 +81,7 @@ { "request": { "method": "GET", - "url": "/users?query=username%3D%3D%22morty%22&limit=1" + "url": "/users?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20username%3D%3D%22morty%22&limit=1" }, "response": { "status": 200, @@ -94,7 +94,7 @@ { "request": { "method": "GET", - "urlPattern": "/users\\?query=barcode%3D%3D%22bar\\d*%22&limit=1" + "urlPattern": "/users\\?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20barcode%3D%3D%22bar\\d*%22&limit=1" }, "response": { "status": 200, @@ -107,7 +107,7 @@ { "request": { "method": "GET", - "urlPattern": "/users\\?query=barcode%3D%3D%22foo\\d*%22&limit=1" + "urlPattern": "/users\\?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20barcode%3D%3D%22foo\\d*%22&limit=1" }, "response": { "status": 200, @@ -133,7 +133,7 @@ { "request": { "method": "GET", - "url": "/users?query=barcode%3D%3D%22456%22&limit=1" + "url": "/users?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20barcode%3D%3D%22456%22&limit=1" }, "response": { "status": 200, @@ -146,7 +146,7 @@ { "request": { "method": "GET", - "url": "/users?query=username%3D%3D%22rick%22&limit=1" + "url": "/users?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20username%3D%3D%22rick%22&limit=1" }, "response": { "status": 200, @@ -159,7 +159,7 @@ { "request": { "method": "GET", - "url": "/users?query=username%3D%3D%22rickemail%22&limit=1" + "url": "/users?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20username%3D%3D%22rickemail%22&limit=1" }, "response": { "status": 200, @@ -172,7 +172,7 @@ { "request": { "method": "GET", - "url": "/users?query=barcode%3D%3D%22455%22&limit=1" + "url": "/users?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20barcode%3D%3D%22455%22&limit=1" }, "response": { "status": 200, @@ -185,7 +185,7 @@ { "request": { "method": "GET", - "url": "/users?query=barcode%3D%3D%22invalid_date%22&limit=1" + "url": "/users?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20barcode%3D%3D%22invalid_date%22&limit=1" }, "response": { "status": 200, @@ -211,7 +211,7 @@ { "request": { "method": "GET", - "url": "/users?query=barcode%3D%3D%22789%22&limit=1" + "url": "/users?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20barcode%3D%3D%22789%22&limit=1" }, "response": { "status": 200, @@ -224,7 +224,7 @@ { "request": { "method": "GET", - "url": "/users?query=username%3D%3D%22sheldon%22&limit=1" + "url": "/users?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20username%3D%3D%22sheldon%22&limit=1" }, "response": { "status": 200, diff --git a/src/test/resources/mappings/users_bad_reference.json b/src/test/resources/mappings/users_bad_reference.json index e03b5cf4c..1e757c75a 100644 --- a/src/test/resources/mappings/users_bad_reference.json +++ b/src/test/resources/mappings/users_bad_reference.json @@ -3,7 +3,7 @@ { "request": { "method": "GET", - "url": "/users?query=barcode%3D%3D%22emptyAddressType%22&limit=1" + "url": "/users?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20barcode%3D%3D%22emptyAddressType%22&limit=1" }, "response": { "status": 200, @@ -16,7 +16,7 @@ { "request": { "method": "GET", - "url": "/users?query=barcode%3D%3D%22badAddressType%22&limit=1" + "url": "/users?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20barcode%3D%3D%22badAddressType%22&limit=1" }, "response": { "status": 200, @@ -29,7 +29,7 @@ { "request": { "method": "GET", - "url": "/users?query=barcode%3D%3D%22emptyPreferredContactTypeId%22&limit=1" + "url": "/users?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20barcode%3D%3D%22emptyPreferredContactTypeId%22&limit=1" }, "response": { "status": 200, @@ -42,7 +42,7 @@ { "request": { "method": "GET", - "url": "/users?query=barcode%3D%3D%22emptyDepartments%22&limit=1" + "url": "/users?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20barcode%3D%3D%22emptyDepartments%22&limit=1" }, "response": { "status": 200, @@ -55,7 +55,7 @@ { "request": { "method": "GET", - "url": "/users?query=barcode%3D%3D%22badDepartments%22&limit=1" + "url": "/users?query=%28cql.allRecords%3D1%20NOT%20type%3D%22%22%20or%20type%3C%3E%22shadow%22%29%20and%20barcode%3D%3D%22badDepartments%22&limit=1" }, "response": { "status": 200, diff --git a/src/test/resources/output/bulk_edit_holdings_records_by_item_barcode.csv b/src/test/resources/output/bulk_edit_holdings_records_by_item_barcode.csv index f70a3b5ca..a2c77020f 100644 --- a/src/test/resources/output/bulk_edit_holdings_records_by_item_barcode.csv +++ b/src/test/resources/output/bulk_edit_holdings_records_by_item_barcode.csv @@ -1,3 +1,3 @@ -Holdings UUID,"Instance (Title, Publisher, Publication date)",Suppress from discovery,Holdings HRID,Source,Former holdings Id,Holdings type,Statistical codes,Administrative note,Holdings permanent location,Holdings temporary location,Shelving title,Holdings copy number,Holdings level call number type,Holdings level call number prefix,Holdings level call number,Holdings level call number suffix,Number of items,Holdings statement,Holdings statement for supplements,Holdings statement for indexes,ILL policy,Digitization policy,Retention policy,Notes,Electronic access,Acquisition method,Order format,Receipt status,Tags -0b1e3760-f689-493e-a98e-9cc9dadb7e83,title,,ho13,FOLIO,d1670310-ceac-47d9-aaba-aaeeb890bc07,Physical,"Book, print (books)",Test administrative note,Annex,Main Library,shelving title,copy number,LC Modified,prefix,call number,suffix,10,statement;statement public note;statement staff note,statement for supplements;statement for supplements public note;statement for supplements staff note,statement for indexes;statement for indexes public note;statement for indexes staff note,Limited lending policy,digitisation policy,retention policy,Note;a note;false|Note;a note;true,"URL relationship;URI;Link text;Materials specified;URL public note -Version of resource;www.someurl.com;link text;material;www.someurl.com",aquisition method,order format,receipt status, +Holdings UUID,"Instance (Title, Publisher, Publication date)",Suppress from discovery,Holdings HRID,Source,Former holdings Id,Holdings type,Statistical codes,Administrative note,Holdings permanent location,Holdings temporary location,Shelving title,Holdings copy number,Holdings level call number type,Holdings level call number prefix,Holdings level call number,Holdings level call number suffix,Number of items,Holdings statement,Holdings statement for supplements,Holdings statement for indexes,ILL policy,Digitization policy,Retention policy,Notes,Electronic access,Acquisition method,Order format,Receipt status,Tags,Tenant +0b1e3760-f689-493e-a98e-9cc9dadb7e83,title,,ho13,FOLIO,d1670310-ceac-47d9-aaba-aaeeb890bc07,Physical,"Book, print (books)",Test administrative note,Annex,Main Library,shelving title,copy number,LC Modified,prefix,call number,suffix,10,statement;statement public note;statement staff note,statement for supplements;statement for supplements public note;statement for supplements staff note,statement for indexes;statement for indexes public note;statement for indexes staff note,Limited lending policy,digitisation policy,retention policy,Note;a note;false;diku;b160f13a-ddba-4053-b9c4-60ec5ea45d56|Note;a note;true;diku;b160f13a-ddba-4053-b9c4-60ec5ea45d56,"URL relationship;URI;Link text;Materials specified;URL public note +Version of resource;www.someurl.com;link text;material;www.someurl.com",aquisition method,order format,receipt status,,diku diff --git a/src/test/resources/output/bulk_edit_holdings_records_empty_reference.csv b/src/test/resources/output/bulk_edit_holdings_records_empty_reference.csv index 87d65e790..25706e748 100644 --- a/src/test/resources/output/bulk_edit_holdings_records_empty_reference.csv +++ b/src/test/resources/output/bulk_edit_holdings_records_empty_reference.csv @@ -1,6 +1,6 @@ -Holdings UUID,"Instance (Title, Publisher, Publication date)",Suppress from discovery,Holdings HRID,Source,Former holdings Id,Holdings type,Statistical codes,Administrative note,Holdings permanent location,Holdings temporary location,Shelving title,Holdings copy number,Holdings level call number type,Holdings level call number prefix,Holdings level call number,Holdings level call number suffix,Number of items,Holdings statement,Holdings statement for supplements,Holdings statement for indexes,ILL policy,Digitization policy,Retention policy,Notes,Electronic access,Acquisition method,Order format,Receipt status,Tags -855d4693-4087-4339-8c14-c25c350e5e59,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,,,,,, -ae4288c6-4999-492a-82f4-eceb3a6e1ec6,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,,,,,, -59b36165-fcf2-49d2-bf7f-25fedbc07e44,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,;a note;false,,,,, -b431e730-f711-41bc-9860-96cab69dd313,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,,,,,, -7f788fa2-3d25-4c00-ae9c-57b28c803da4,title,,ho14,,,,,,Main Library,,,,,,,,,,,,,,,,,,,, +Holdings UUID,"Instance (Title, Publisher, Publication date)",Suppress from discovery,Holdings HRID,Source,Former holdings Id,Holdings type,Statistical codes,Administrative note,Holdings permanent location,Holdings temporary location,Shelving title,Holdings copy number,Holdings level call number type,Holdings level call number prefix,Holdings level call number,Holdings level call number suffix,Number of items,Holdings statement,Holdings statement for supplements,Holdings statement for indexes,ILL policy,Digitization policy,Retention policy,Notes,Electronic access,Acquisition method,Order format,Receipt status,Tags,Tenant +855d4693-4087-4339-8c14-c25c350e5e59,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,,,,,,,diku +ae4288c6-4999-492a-82f4-eceb3a6e1ec6,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,,,,,,,diku +59b36165-fcf2-49d2-bf7f-25fedbc07e44,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,;a note;false;diku;null,,,,,,diku +b431e730-f711-41bc-9860-96cab69dd313,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,,,,,,,diku +7f788fa2-3d25-4c00-ae9c-57b28c803da4,title,,ho14,,,,,,Main Library,,,,,,,,,,,,,,,,,,,,,diku diff --git a/src/test/resources/output/bulk_edit_holdings_records_output.csv b/src/test/resources/output/bulk_edit_holdings_records_output.csv index cebe90d80..df3e7a902 100644 --- a/src/test/resources/output/bulk_edit_holdings_records_output.csv +++ b/src/test/resources/output/bulk_edit_holdings_records_output.csv @@ -1,4 +1,4 @@ -Holdings UUID,"Instance (Title, Publisher, Publication date)",Suppress from discovery,Holdings HRID,Source,Former holdings Id,Holdings type,Statistical codes,Administrative note,Holdings permanent location,Holdings temporary location,Shelving title,Holdings copy number,Holdings level call number type,Holdings level call number prefix,Holdings level call number,Holdings level call number suffix,Number of items,Holdings statement,Holdings statement for supplements,Holdings statement for indexes,ILL policy,Digitization policy,Retention policy,Notes,Electronic access,Acquisition method,Order format,Receipt status,Tags -0b1e3760-f689-493e-a98e-9cc9dadb7e83,title,,ho13,FOLIO,d1670310-ceac-47d9-aaba-aaeeb890bc07,Physical,"Book, print (books)",Test administrative note,Annex,Main Library,shelving title,copy number,LC Modified,prefix,call number,suffix,10,statement;statement public note;statement staff note,statement for supplements;statement for supplements public note;statement for supplements staff note,statement for indexes;statement for indexes public note;statement for indexes staff note,Limited lending policy,digitisation policy,retention policy,Note;a note;false|Note;a note;true,"URL relationship;URI;Link text;Materials specified;URL public note -Version of resource;www.someurl.com;link text;material;www.someurl.com",aquisition method,order format,receipt status, -855d4693-4087-4339-8c14-c25c350e5e59,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,,,,,, +Holdings UUID,"Instance (Title, Publisher, Publication date)",Suppress from discovery,Holdings HRID,Source,Former holdings Id,Holdings type,Statistical codes,Administrative note,Holdings permanent location,Holdings temporary location,Shelving title,Holdings copy number,Holdings level call number type,Holdings level call number prefix,Holdings level call number,Holdings level call number suffix,Number of items,Holdings statement,Holdings statement for supplements,Holdings statement for indexes,ILL policy,Digitization policy,Retention policy,Notes,Electronic access,Acquisition method,Order format,Receipt status,Tags,Tenant +0b1e3760-f689-493e-a98e-9cc9dadb7e83,title,,ho13,FOLIO,d1670310-ceac-47d9-aaba-aaeeb890bc07,Physical,"Book, print (books)",Test administrative note,Annex,Main Library,shelving title,copy number,LC Modified,prefix,call number,suffix,10,statement;statement public note;statement staff note,statement for supplements;statement for supplements public note;statement for supplements staff note,statement for indexes;statement for indexes public note;statement for indexes staff note,Limited lending policy,digitisation policy,retention policy,Note;a note;false;diku;b160f13a-ddba-4053-b9c4-60ec5ea45d56|Note;a note;true;diku;b160f13a-ddba-4053-b9c4-60ec5ea45d56,"URL relationship;URI;Link text;Materials specified;URL public note +Version of resource;www.someurl.com;link text;material;www.someurl.com",aquisition method,order format,receipt status,,diku +855d4693-4087-4339-8c14-c25c350e5e59,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,,,,,,,diku diff --git a/src/test/resources/output/bulk_edit_holdings_records_output_instance_hrid.csv b/src/test/resources/output/bulk_edit_holdings_records_output_instance_hrid.csv index cebe90d80..df3e7a902 100644 --- a/src/test/resources/output/bulk_edit_holdings_records_output_instance_hrid.csv +++ b/src/test/resources/output/bulk_edit_holdings_records_output_instance_hrid.csv @@ -1,4 +1,4 @@ -Holdings UUID,"Instance (Title, Publisher, Publication date)",Suppress from discovery,Holdings HRID,Source,Former holdings Id,Holdings type,Statistical codes,Administrative note,Holdings permanent location,Holdings temporary location,Shelving title,Holdings copy number,Holdings level call number type,Holdings level call number prefix,Holdings level call number,Holdings level call number suffix,Number of items,Holdings statement,Holdings statement for supplements,Holdings statement for indexes,ILL policy,Digitization policy,Retention policy,Notes,Electronic access,Acquisition method,Order format,Receipt status,Tags -0b1e3760-f689-493e-a98e-9cc9dadb7e83,title,,ho13,FOLIO,d1670310-ceac-47d9-aaba-aaeeb890bc07,Physical,"Book, print (books)",Test administrative note,Annex,Main Library,shelving title,copy number,LC Modified,prefix,call number,suffix,10,statement;statement public note;statement staff note,statement for supplements;statement for supplements public note;statement for supplements staff note,statement for indexes;statement for indexes public note;statement for indexes staff note,Limited lending policy,digitisation policy,retention policy,Note;a note;false|Note;a note;true,"URL relationship;URI;Link text;Materials specified;URL public note -Version of resource;www.someurl.com;link text;material;www.someurl.com",aquisition method,order format,receipt status, -855d4693-4087-4339-8c14-c25c350e5e59,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,,,,,, +Holdings UUID,"Instance (Title, Publisher, Publication date)",Suppress from discovery,Holdings HRID,Source,Former holdings Id,Holdings type,Statistical codes,Administrative note,Holdings permanent location,Holdings temporary location,Shelving title,Holdings copy number,Holdings level call number type,Holdings level call number prefix,Holdings level call number,Holdings level call number suffix,Number of items,Holdings statement,Holdings statement for supplements,Holdings statement for indexes,ILL policy,Digitization policy,Retention policy,Notes,Electronic access,Acquisition method,Order format,Receipt status,Tags,Tenant +0b1e3760-f689-493e-a98e-9cc9dadb7e83,title,,ho13,FOLIO,d1670310-ceac-47d9-aaba-aaeeb890bc07,Physical,"Book, print (books)",Test administrative note,Annex,Main Library,shelving title,copy number,LC Modified,prefix,call number,suffix,10,statement;statement public note;statement staff note,statement for supplements;statement for supplements public note;statement for supplements staff note,statement for indexes;statement for indexes public note;statement for indexes staff note,Limited lending policy,digitisation policy,retention policy,Note;a note;false;diku;b160f13a-ddba-4053-b9c4-60ec5ea45d56|Note;a note;true;diku;b160f13a-ddba-4053-b9c4-60ec5ea45d56,"URL relationship;URI;Link text;Materials specified;URL public note +Version of resource;www.someurl.com;link text;material;www.someurl.com",aquisition method,order format,receipt status,,diku +855d4693-4087-4339-8c14-c25c350e5e59,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,,,,,,,diku diff --git a/src/test/resources/output/bulk_edit_holdings_records_output_item_barcode.csv b/src/test/resources/output/bulk_edit_holdings_records_output_item_barcode.csv index cebe90d80..df3e7a902 100644 --- a/src/test/resources/output/bulk_edit_holdings_records_output_item_barcode.csv +++ b/src/test/resources/output/bulk_edit_holdings_records_output_item_barcode.csv @@ -1,4 +1,4 @@ -Holdings UUID,"Instance (Title, Publisher, Publication date)",Suppress from discovery,Holdings HRID,Source,Former holdings Id,Holdings type,Statistical codes,Administrative note,Holdings permanent location,Holdings temporary location,Shelving title,Holdings copy number,Holdings level call number type,Holdings level call number prefix,Holdings level call number,Holdings level call number suffix,Number of items,Holdings statement,Holdings statement for supplements,Holdings statement for indexes,ILL policy,Digitization policy,Retention policy,Notes,Electronic access,Acquisition method,Order format,Receipt status,Tags -0b1e3760-f689-493e-a98e-9cc9dadb7e83,title,,ho13,FOLIO,d1670310-ceac-47d9-aaba-aaeeb890bc07,Physical,"Book, print (books)",Test administrative note,Annex,Main Library,shelving title,copy number,LC Modified,prefix,call number,suffix,10,statement;statement public note;statement staff note,statement for supplements;statement for supplements public note;statement for supplements staff note,statement for indexes;statement for indexes public note;statement for indexes staff note,Limited lending policy,digitisation policy,retention policy,Note;a note;false|Note;a note;true,"URL relationship;URI;Link text;Materials specified;URL public note -Version of resource;www.someurl.com;link text;material;www.someurl.com",aquisition method,order format,receipt status, -855d4693-4087-4339-8c14-c25c350e5e59,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,,,,,, +Holdings UUID,"Instance (Title, Publisher, Publication date)",Suppress from discovery,Holdings HRID,Source,Former holdings Id,Holdings type,Statistical codes,Administrative note,Holdings permanent location,Holdings temporary location,Shelving title,Holdings copy number,Holdings level call number type,Holdings level call number prefix,Holdings level call number,Holdings level call number suffix,Number of items,Holdings statement,Holdings statement for supplements,Holdings statement for indexes,ILL policy,Digitization policy,Retention policy,Notes,Electronic access,Acquisition method,Order format,Receipt status,Tags,Tenant +0b1e3760-f689-493e-a98e-9cc9dadb7e83,title,,ho13,FOLIO,d1670310-ceac-47d9-aaba-aaeeb890bc07,Physical,"Book, print (books)",Test administrative note,Annex,Main Library,shelving title,copy number,LC Modified,prefix,call number,suffix,10,statement;statement public note;statement staff note,statement for supplements;statement for supplements public note;statement for supplements staff note,statement for indexes;statement for indexes public note;statement for indexes staff note,Limited lending policy,digitisation policy,retention policy,Note;a note;false;diku;b160f13a-ddba-4053-b9c4-60ec5ea45d56|Note;a note;true;diku;b160f13a-ddba-4053-b9c4-60ec5ea45d56,"URL relationship;URI;Link text;Materials specified;URL public note +Version of resource;www.someurl.com;link text;material;www.someurl.com",aquisition method,order format,receipt status,,diku +855d4693-4087-4339-8c14-c25c350e5e59,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,,,,,,,diku diff --git a/src/test/resources/output/bulk_edit_holdings_records_reference_not_found.csv b/src/test/resources/output/bulk_edit_holdings_records_reference_not_found.csv index f73c5c48a..ca372d669 100644 --- a/src/test/resources/output/bulk_edit_holdings_records_reference_not_found.csv +++ b/src/test/resources/output/bulk_edit_holdings_records_reference_not_found.csv @@ -1,7 +1,7 @@ -Holdings UUID,"Instance (Title, Publisher, Publication date)",Suppress from discovery,Holdings HRID,Source,Former holdings Id,Holdings type,Statistical codes,Administrative note,Holdings permanent location,Holdings temporary location,Shelving title,Holdings copy number,Holdings level call number type,Holdings level call number prefix,Holdings level call number,Holdings level call number suffix,Number of items,Holdings statement,Holdings statement for supplements,Holdings statement for indexes,ILL policy,Digitization policy,Retention policy,Notes,Electronic access,Acquisition method,Order format,Receipt status,Tags -855d4693-4087-4339-8c14-c25c350e5e59,title,,ho14,FOLIO,,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,,Main Library,,,,,,,,,,,,,,,,,,,, -f836ca1d-c55a-4689-9b57-bb0de0c4d43d,title,,ho14,FOLIO,,,,,Main Library,,,,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,,,,,,,,,,,,,,, -c10bb70f-b0fd-4623-a8cf-25e5198087ad,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3;a note;false,,,,, -e5bfbcfa-1fe5-4aab-8220-0698455875f3,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,,,,,,, -b86cf4ba-3cb4-4e47-8e98-c051146af965,title,,ho14,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,,,,Main Library,,,,,,,,,,,,,,,,,,,, -53e62b1e-8280-47f3-b640-60d0a4850c36,title,,ho14,FOLIO,,,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,Main Library,,,,,,,,,,,,,,,,,,,, +Holdings UUID,"Instance (Title, Publisher, Publication date)",Suppress from discovery,Holdings HRID,Source,Former holdings Id,Holdings type,Statistical codes,Administrative note,Holdings permanent location,Holdings temporary location,Shelving title,Holdings copy number,Holdings level call number type,Holdings level call number prefix,Holdings level call number,Holdings level call number suffix,Number of items,Holdings statement,Holdings statement for supplements,Holdings statement for indexes,ILL policy,Digitization policy,Retention policy,Notes,Electronic access,Acquisition method,Order format,Receipt status,Tags,Tenant +855d4693-4087-4339-8c14-c25c350e5e59,title,,ho14,FOLIO,,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,,Main Library,,,,,,,,,,,,,,,,,,,,,diku +f836ca1d-c55a-4689-9b57-bb0de0c4d43d,title,,ho14,FOLIO,,,,,Main Library,,,,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,,,,,,,,,,,,,,,,diku +c10bb70f-b0fd-4623-a8cf-25e5198087ad,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,,,,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3;a note;false;diku;1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,,,,,diku +e5bfbcfa-1fe5-4aab-8220-0698455875f3,title,,ho14,FOLIO,,,,,Main Library,,,,,,,,,,,,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,,,,,,,,diku +b86cf4ba-3cb4-4e47-8e98-c051146af965,title,,ho14,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,,,,Main Library,,,,,,,,,,,,,,,,,,,,,diku +53e62b1e-8280-47f3-b640-60d0a4850c36,title,,ho14,FOLIO,,,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,Main Library,,,,,,,,,,,,,,,,,,,,,diku diff --git a/src/test/resources/output/bulk_edit_instance_by_isbn_json_output.json b/src/test/resources/output/bulk_edit_instance_by_isbn_json_output.json index 9d6e20420..21707a3c2 100644 --- a/src/test/resources/output/bulk_edit_instance_by_isbn_json_output.json +++ b/src/test/resources/output/bulk_edit_instance_by_isbn_json_output.json @@ -1 +1 @@ -{"tenantId":"diku","entity":{"id":"7fbd5d84-62d1-44c6-9c45-6cb173998bbd","_version":31,"hrid":"inst000000000006","matchKey":null,"source":"FOLIO","title":"Bridget Jones's Baby: the diaries","indexTitle":null,"alternativeTitles":[],"editions":["First American Edition"],"series":[],"identifiers":[{"value":"12345","identifierTypeId":"913300b2-03ed-469a-8179-c1092c991227","identifierTypeObject":null},{"value":"12345","identifierTypeId":"8261054f-be78-422d-bd51-4ed9f33c3422","identifierTypeObject":null}],"contributors":[{"name":"Fielding, Helen","contributorTypeId":null,"contributorTypeText":null,"contributorNameTypeId":"2b94c631-fca9-4892-a730-03ee529ffe2a","authorityId":null,"contributorNameType":null,"primary":null}],"subjects":[],"classifications":[],"publication":[],"publicationFrequency":["A frequency description"],"publicationRange":["A publication range"],"publicationPeriod":null,"electronicAccess":[],"instanceTypeId":"6312d172-f0cf-40f6-b27d-9fa8feaf332f","instanceFormatIds":[],"instanceFormats":null,"physicalDescriptions":["219 pages ; 20 cm."],"languages":["eng"],"notes":[],"administrativeNotes":["Cataloging data"],"modeOfIssuanceId":null,"catalogedDate":"2024-01-26","previouslyHeld":true,"staffSuppress":false,"discoverySuppress":false,"statisticalCodeIds":[],"sourceRecordFormat":null,"statusId":null,"statusUpdatedDate":null,"tags":{"tagList":[]},"metadata":{"createdDate":1699617257074,"createdByUserId":"cffb2565-07fc-470b-86c6-17d8ce14432e","createdByUsername":null,"updatedDate":1704810384610,"updatedByUserId":"ca6022de-2644-46fe-b6f2-78df15483721","updatedByUsername":null},"holdingsRecords2":null,"natureOfContentTermIds":[],"precedingTitles":[],"succeedingTitles":[],"ISSN":null,"ISBN":"12345"}} +{"tenantId":"diku","entity":{"id":"7fbd5d84-62d1-44c6-9c45-6cb173998bbd","_version":31,"hrid":"inst000000000006","matchKey":null,"source":"FOLIO","parentInstances":[],"childInstances":[],"title":"Bridget Jones's Baby: the diaries","indexTitle":null,"alternativeTitles":[],"editions":["First American Edition"],"series":[],"identifiers":[{"value":"12345","identifierTypeId":"913300b2-03ed-469a-8179-c1092c991227","identifierTypeObject":null},{"value":"12345","identifierTypeId":"8261054f-be78-422d-bd51-4ed9f33c3422","identifierTypeObject":null}],"contributors":[{"name":"Fielding, Helen","contributorTypeId":null,"contributorTypeText":null,"contributorNameTypeId":"2b94c631-fca9-4892-a730-03ee529ffe2a","authorityId":null,"contributorNameType":null,"primary":null}],"subjects":[],"classifications":[],"publication":[],"publicationFrequency":["A frequency description"],"publicationRange":["A publication range"],"electronicAccess":[],"dates":null,"instanceTypeId":"6312d172-f0cf-40f6-b27d-9fa8feaf332f","instanceFormatIds":[],"instanceFormats":null,"physicalDescriptions":["219 pages ; 20 cm."],"languages":["eng"],"notes":[],"administrativeNotes":["Cataloging data"],"modeOfIssuanceId":null,"catalogedDate":"2024-01-26","previouslyHeld":true,"staffSuppress":false,"discoverySuppress":false,"statisticalCodeIds":[],"sourceRecordFormat":null,"statusId":null,"statusUpdatedDate":null,"tags":{"tagList":[]},"metadata":{"createdDate":1699617257074,"createdByUserId":"cffb2565-07fc-470b-86c6-17d8ce14432e","createdByUsername":null,"updatedDate":1704810384610,"updatedByUserId":"ca6022de-2644-46fe-b6f2-78df15483721","updatedByUsername":null},"holdingsRecords2":null,"natureOfContentTermIds":[],"isBoundWith":false,"precedingTitles":[],"succeedingTitles":[],"ISSN":null,"ISBN":"12345"}} diff --git a/src/test/resources/output/bulk_edit_instance_by_issn_json_output.json b/src/test/resources/output/bulk_edit_instance_by_issn_json_output.json index 9885951f4..b5000c80d 100644 --- a/src/test/resources/output/bulk_edit_instance_by_issn_json_output.json +++ b/src/test/resources/output/bulk_edit_instance_by_issn_json_output.json @@ -1 +1 @@ -{"tenantId":"diku","entity":{"id":"7fbd5d84-62d1-44c6-9c45-6cb173998bbd","_version":31,"hrid":"inst000000000006","matchKey":null,"source":"FOLIO","title":"Bridget Jones's Baby: the diaries","indexTitle":null,"alternativeTitles":[],"editions":["First American Edition"],"series":[],"identifiers":[{"value":"12345","identifierTypeId":"913300b2-03ed-469a-8179-c1092c991227","identifierTypeObject":null},{"value":"12345","identifierTypeId":"8261054f-be78-422d-bd51-4ed9f33c3422","identifierTypeObject":null}],"contributors":[{"name":"Fielding, Helen","contributorTypeId":null,"contributorTypeText":null,"contributorNameTypeId":"2b94c631-fca9-4892-a730-03ee529ffe2a","authorityId":null,"contributorNameType":null,"primary":null}],"subjects":[],"classifications":[],"publication":[],"publicationFrequency":["A frequency description"],"publicationRange":["A publication range"],"publicationPeriod":null,"electronicAccess":[],"instanceTypeId":"6312d172-f0cf-40f6-b27d-9fa8feaf332f","instanceFormatIds":[],"instanceFormats":null,"physicalDescriptions":["219 pages ; 20 cm."],"languages":["eng"],"notes":[],"administrativeNotes":["Cataloging data"],"modeOfIssuanceId":null,"catalogedDate":"2024-01-26","previouslyHeld":true,"staffSuppress":false,"discoverySuppress":false,"statisticalCodeIds":[],"sourceRecordFormat":null,"statusId":null,"statusUpdatedDate":null,"tags":{"tagList":[]},"metadata":{"createdDate":1699617257074,"createdByUserId":"cffb2565-07fc-470b-86c6-17d8ce14432e","createdByUsername":null,"updatedDate":1704810384610,"updatedByUserId":"ca6022de-2644-46fe-b6f2-78df15483721","updatedByUsername":null},"holdingsRecords2":null,"natureOfContentTermIds":[],"precedingTitles":[],"succeedingTitles":[],"ISSN":"12345","ISBN":null}} +{"tenantId":"diku","entity":{"id":"7fbd5d84-62d1-44c6-9c45-6cb173998bbd","_version":31,"hrid":"inst000000000006","matchKey":null,"source":"FOLIO","parentInstances":[],"childInstances":[],"title":"Bridget Jones's Baby: the diaries","indexTitle":null,"alternativeTitles":[],"editions":["First American Edition"],"series":[],"identifiers":[{"value":"12345","identifierTypeId":"913300b2-03ed-469a-8179-c1092c991227","identifierTypeObject":null},{"value":"12345","identifierTypeId":"8261054f-be78-422d-bd51-4ed9f33c3422","identifierTypeObject":null}],"contributors":[{"name":"Fielding, Helen","contributorTypeId":null,"contributorTypeText":null,"contributorNameTypeId":"2b94c631-fca9-4892-a730-03ee529ffe2a","authorityId":null,"contributorNameType":null,"primary":null}],"subjects":[],"classifications":[],"publication":[],"publicationFrequency":["A frequency description"],"publicationRange":["A publication range"],"electronicAccess":[],"dates":null,"instanceTypeId":"6312d172-f0cf-40f6-b27d-9fa8feaf332f","instanceFormatIds":[],"instanceFormats":null,"physicalDescriptions":["219 pages ; 20 cm."],"languages":["eng"],"notes":[],"administrativeNotes":["Cataloging data"],"modeOfIssuanceId":null,"catalogedDate":"2024-01-26","previouslyHeld":true,"staffSuppress":false,"discoverySuppress":false,"statisticalCodeIds":[],"sourceRecordFormat":null,"statusId":null,"statusUpdatedDate":null,"tags":{"tagList":[]},"metadata":{"createdDate":1699617257074,"createdByUserId":"cffb2565-07fc-470b-86c6-17d8ce14432e","createdByUsername":null,"updatedDate":1704810384610,"updatedByUserId":"ca6022de-2644-46fe-b6f2-78df15483721","updatedByUsername":null},"holdingsRecords2":null,"natureOfContentTermIds":[],"isBoundWith":false,"precedingTitles":[],"succeedingTitles":[],"ISSN":"12345","ISBN":null}} diff --git a/src/test/resources/output/bulk_edit_instance_identifiers_json_output.json b/src/test/resources/output/bulk_edit_instance_identifiers_json_output.json index c5babbeb9..3b6521e9f 100644 --- a/src/test/resources/output/bulk_edit_instance_identifiers_json_output.json +++ b/src/test/resources/output/bulk_edit_instance_identifiers_json_output.json @@ -1,2 +1,2 @@ -{"tenantId":"diku","entity":{"id":"00f10ab9-d845-4334-92d2-ff55862bf4f9","_version":3,"hrid":"inst000000000002","matchKey":null,"source":"FOLIO","title":"American Bar Association journal.","indexTitle":"American Bar Association journal.","alternativeTitles":[],"editions":[],"series":[],"identifiers":[],"contributors":[{"name":"American Bar Association","contributorTypeId":"6e09d47d-95e2-4d8a-831b-f777b8ef6d81","contributorTypeText":"","contributorNameTypeId":"d376e36c-b759-4fed-8502-7130d1eeff39","authorityId":null,"contributorNameType":null,"primary":null},{"name":"American Bar Association. Journal","contributorTypeId":"06b2cbd8-66bf-4956-9d90-97c9776365a4","contributorTypeText":"","contributorNameTypeId":"d376e36c-b759-4fed-8502-7130d1eeff39","authorityId":null,"contributorNameType":null,"primary":null}],"subjects":[],"classifications":[],"publication":[],"publicationFrequency":["Monthly, 1921-83","Quarterly, 1915-20"],"publicationRange":["Began with vol. 1, no. 1 (Jan. 1915); ceased with v. 69, [no.12] (Dec. 1983)"],"publicationPeriod":null,"electronicAccess":[],"instanceTypeId":"6312d172-f0cf-40f6-b27d-9fa8feaf332f","instanceFormatIds":["5cb91d15-96b1-4b8a-bf60-ec310538da66"],"instanceFormats":null,"physicalDescriptions":["69 v. : ill. ; 23-30 cm."],"languages":["eng"],"notes":[],"administrativeNotes":[],"modeOfIssuanceId":null,"catalogedDate":null,"previouslyHeld":false,"staffSuppress":null,"discoverySuppress":false,"statisticalCodeIds":[],"sourceRecordFormat":null,"statusId":null,"statusUpdatedDate":null,"tags":{"tagList":[]},"metadata":{"createdDate":1699617256187,"createdByUserId":"cffb2565-07fc-470b-86c6-17d8ce14432e","createdByUsername":null,"updatedDate":1704448346651,"updatedByUserId":"ca6022de-2644-46fe-b6f2-78df15483721","updatedByUsername":null},"holdingsRecords2":null,"natureOfContentTermIds":[],"precedingTitles":[],"succeedingTitles":[],"ISSN":null,"ISBN":null}} -{"tenantId":"diku","entity":{"id":"549fad9e-7f8e-4d8e-9a71-00d251817866","_version":2,"hrid":"inst000000000028","matchKey":null,"source":"FOLIO","title":"Agile Organisation, Risiko- und Change Management Harald Augustin (Hrsg.)","indexTitle":null,"alternativeTitles":[],"editions":[],"series":[{"value":"Umsetzung der DIN EN ISO 9001:2015 / Harald Augustin (Hrsg.) ; Band 2","authorityId":null},{"value":"Berichte aus der Betriebswirtschaft","authorityId":null},{"value":"Umsetzung der DIN EN ISO 9001:2015 Band 2","authorityId":null}],"identifiers":[{"value":"3844057439","identifierTypeId":"8261054f-be78-422d-bd51-4ed9f33c3422","identifierTypeObject":null},{"value":"9783844057430","identifierTypeId":"8261054f-be78-422d-bd51-4ed9f33c3422","identifierTypeObject":null},{"value":"9783844057430","identifierTypeId":"2e8b3b6c-0e7d-4e48-bca2-b0b23b376af5","identifierTypeObject":null},{"value":"(OCoLC)1024128245","identifierTypeId":"7e591197-f335-4afb-bc6d-a6d76ca3bace","identifierTypeObject":null},{"value":"(DE-101)1150176652","identifierTypeId":"7e591197-f335-4afb-bc6d-a6d76ca3bace","identifierTypeObject":null},{"value":"(DE-599)DNB1150176652","identifierTypeId":"7e591197-f335-4afb-bc6d-a6d76ca3bace","identifierTypeObject":null}],"contributors":[{"name":"Augustin, Harald","contributorTypeId":null,"contributorTypeText":null,"contributorNameTypeId":"2b94c631-fca9-4892-a730-03ee529ffe2a","authorityId":null,"contributorNameType":null,"primary":null},{"name":"Shaker Verlag GmbH","contributorTypeId":null,"contributorTypeText":null,"contributorNameTypeId":"2e48e713-17f3-4c13-a9f8-23845bb210aa","authorityId":null,"contributorNameType":null,"primary":null}],"subjects":[{"value":"DIN EN ISO 9001:2015","authorityId":null},{"value":"Standard","authorityId":null},{"value":"Risikomanagement","authorityId":null},{"value":"Einführung","authorityId":null},{"value":"Organisatorischer Wandel","authorityId":null},{"value":"Industrie","authorityId":null},{"value":"Deutschland","authorityId":null}],"classifications":[{"classificationNumber":"658.4013","classificationTypeId":"42471af9-7d25-4f3a-bf78-60d29dcf463b","classificationType":null},{"classificationNumber":"650","classificationTypeId":"42471af9-7d25-4f3a-bf78-60d29dcf463b","classificationType":null}],"publication":[{"publisher":"Shaker Verlag","place":"Aachen","dateOfPublication":"2017","role":null}],"publicationFrequency":[],"publicationRange":[],"publicationPeriod":{"start":2017,"end":null},"electronicAccess":[{"uri":"http://d-nb.info/1150176652/04","linkText":"Electronic resource (PDF)","materialsSpecification":null,"publicNote":"Address for accessing the table of content. PDF file","relationshipId":null}],"instanceTypeId":"6312d172-f0cf-40f6-b27d-9fa8feaf332f","instanceFormatIds":[],"instanceFormats":null,"physicalDescriptions":["x, 148 Seiten Illustrationen 21 cm, 188 g"],"languages":["ger"],"notes":[],"administrativeNotes":[],"modeOfIssuanceId":null,"catalogedDate":null,"previouslyHeld":false,"staffSuppress":true,"discoverySuppress":false,"statisticalCodeIds":[],"sourceRecordFormat":null,"statusId":null,"statusUpdatedDate":"2023-11-10T11:54:16.884+0000","tags":{"tagList":[]},"metadata":{"createdDate":1699617256883,"createdByUserId":"cffb2565-07fc-470b-86c6-17d8ce14432e","createdByUsername":null,"updatedDate":1702564431734,"updatedByUserId":"ca6022de-2644-46fe-b6f2-78df15483721","updatedByUsername":null},"holdingsRecords2":null,"natureOfContentTermIds":[],"precedingTitles":[],"succeedingTitles":[],"ISSN":null,"ISBN":null}} +{"tenantId":"diku","entity":{"id":"00f10ab9-d845-4334-92d2-ff55862bf4f9","_version":3,"hrid":"inst000000000002","matchKey":null,"source":"FOLIO","parentInstances":[],"childInstances":[],"title":"American Bar Association journal.","indexTitle":"American Bar Association journal.","alternativeTitles":[],"editions":[],"series":[],"identifiers":[],"contributors":[{"name":"American Bar Association","contributorTypeId":"6e09d47d-95e2-4d8a-831b-f777b8ef6d81","contributorTypeText":"","contributorNameTypeId":"d376e36c-b759-4fed-8502-7130d1eeff39","authorityId":null,"contributorNameType":null,"primary":null},{"name":"American Bar Association. Journal","contributorTypeId":"06b2cbd8-66bf-4956-9d90-97c9776365a4","contributorTypeText":"","contributorNameTypeId":"d376e36c-b759-4fed-8502-7130d1eeff39","authorityId":null,"contributorNameType":null,"primary":null}],"subjects":[],"classifications":[],"publication":[],"publicationFrequency":["Monthly, 1921-83","Quarterly, 1915-20"],"publicationRange":["Began with vol. 1, no. 1 (Jan. 1915); ceased with v. 69, [no.12] (Dec. 1983)"],"electronicAccess":[],"dates":null,"instanceTypeId":"6312d172-f0cf-40f6-b27d-9fa8feaf332f","instanceFormatIds":["5cb91d15-96b1-4b8a-bf60-ec310538da66"],"instanceFormats":null,"physicalDescriptions":["69 v. : ill. ; 23-30 cm."],"languages":["eng"],"notes":[],"administrativeNotes":[],"modeOfIssuanceId":null,"catalogedDate":null,"previouslyHeld":false,"staffSuppress":null,"discoverySuppress":false,"statisticalCodeIds":[],"sourceRecordFormat":null,"statusId":null,"statusUpdatedDate":null,"tags":{"tagList":[]},"metadata":{"createdDate":1699617256187,"createdByUserId":"cffb2565-07fc-470b-86c6-17d8ce14432e","createdByUsername":null,"updatedDate":1704448346651,"updatedByUserId":"ca6022de-2644-46fe-b6f2-78df15483721","updatedByUsername":null},"holdingsRecords2":null,"natureOfContentTermIds":[],"isBoundWith":false,"precedingTitles":[],"succeedingTitles":[],"ISSN":null,"ISBN":null}} +{"tenantId":"diku","entity":{"id":"549fad9e-7f8e-4d8e-9a71-00d251817866","_version":2,"hrid":"inst000000000028","matchKey":null,"source":"FOLIO","parentInstances":[{"id":"1b449f40-5ae8-47df-9113-3c0a958b5ce8","superInstanceId":"a317b304-528c-424f-961c-39174933b454","instanceRelationshipTypeId":"a17daf0a-f057-43b3-9997-13d0724cdf51"}],"childInstances":[],"title":"Agile Organisation, Risiko- und Change Management Harald Augustin (Hrsg.)","indexTitle":null,"alternativeTitles":[],"editions":[],"series":[{"value":"Umsetzung der DIN EN ISO 9001:2015 / Harald Augustin (Hrsg.) ; Band 2","authorityId":null},{"value":"Berichte aus der Betriebswirtschaft","authorityId":null},{"value":"Umsetzung der DIN EN ISO 9001:2015 Band 2","authorityId":null}],"identifiers":[{"value":"3844057439","identifierTypeId":"8261054f-be78-422d-bd51-4ed9f33c3422","identifierTypeObject":null},{"value":"9783844057430","identifierTypeId":"8261054f-be78-422d-bd51-4ed9f33c3422","identifierTypeObject":null},{"value":"9783844057430","identifierTypeId":"2e8b3b6c-0e7d-4e48-bca2-b0b23b376af5","identifierTypeObject":null},{"value":"(OCoLC)1024128245","identifierTypeId":"7e591197-f335-4afb-bc6d-a6d76ca3bace","identifierTypeObject":null},{"value":"(DE-101)1150176652","identifierTypeId":"7e591197-f335-4afb-bc6d-a6d76ca3bace","identifierTypeObject":null},{"value":"(DE-599)DNB1150176652","identifierTypeId":"7e591197-f335-4afb-bc6d-a6d76ca3bace","identifierTypeObject":null}],"contributors":[{"name":"Augustin, Harald","contributorTypeId":null,"contributorTypeText":null,"contributorNameTypeId":"2b94c631-fca9-4892-a730-03ee529ffe2a","authorityId":null,"contributorNameType":null,"primary":null},{"name":"Shaker Verlag GmbH","contributorTypeId":null,"contributorTypeText":null,"contributorNameTypeId":"2e48e713-17f3-4c13-a9f8-23845bb210aa","authorityId":null,"contributorNameType":null,"primary":null}],"subjects":[{"value":"DIN EN ISO 9001:2015","authorityId":null,"sourceId":null,"typeId":null},{"value":"Standard","authorityId":null,"sourceId":null,"typeId":null},{"value":"Risikomanagement","authorityId":null,"sourceId":null,"typeId":null},{"value":"Einführung","authorityId":null,"sourceId":null,"typeId":null},{"value":"Organisatorischer Wandel","authorityId":null,"sourceId":null,"typeId":null},{"value":"Industrie","authorityId":null,"sourceId":null,"typeId":null},{"value":"Deutschland","authorityId":null,"sourceId":null,"typeId":null}],"classifications":[{"classificationNumber":"658.4013","classificationTypeId":"42471af9-7d25-4f3a-bf78-60d29dcf463b","classificationType":null},{"classificationNumber":"650","classificationTypeId":"42471af9-7d25-4f3a-bf78-60d29dcf463b","classificationType":null}],"publication":[{"publisher":"Shaker Verlag","place":"Aachen","dateOfPublication":"2017","role":null}],"publicationFrequency":[],"publicationRange":[],"electronicAccess":[{"uri":"http://d-nb.info/1150176652/04","linkText":"Electronic resource (PDF)","materialsSpecification":null,"publicNote":"Address for accessing the table of content. PDF file","relationshipId":null}],"dates":null,"instanceTypeId":"6312d172-f0cf-40f6-b27d-9fa8feaf332f","instanceFormatIds":[],"instanceFormats":null,"physicalDescriptions":["x, 148 Seiten Illustrationen 21 cm, 188 g"],"languages":["ger"],"notes":[],"administrativeNotes":[],"modeOfIssuanceId":null,"catalogedDate":null,"previouslyHeld":false,"staffSuppress":true,"discoverySuppress":false,"statisticalCodeIds":[],"sourceRecordFormat":null,"statusId":null,"statusUpdatedDate":"2023-11-10T11:54:16.884+0000","tags":{"tagList":[]},"metadata":{"createdDate":1699617256883,"createdByUserId":"cffb2565-07fc-470b-86c6-17d8ce14432e","createdByUsername":null,"updatedDate":1702564431734,"updatedByUserId":"ca6022de-2644-46fe-b6f2-78df15483721","updatedByUsername":null},"holdingsRecords2":null,"natureOfContentTermIds":[],"isBoundWith":false,"precedingTitles":[],"succeedingTitles":[],"ISSN":null,"ISBN":null}} diff --git a/src/test/resources/output/bulk_edit_item_identifiers_output.csv b/src/test/resources/output/bulk_edit_item_identifiers_output.csv index 2b3df501e..f14f33b20 100644 --- a/src/test/resources/output/bulk_edit_item_identifiers_output.csv +++ b/src/test/resources/output/bulk_edit_item_identifiers_output.csv @@ -1,5 +1,5 @@ -Item UUID,"Instance (Title, Publisher, Publication date)","Holdings (Location, Call number)",Item effective location,Effective call number,Suppress from discovery,Item HRID,Barcode,Accession number,Item identifier,Former identifier,Statistical codes,Administrative note,Material type,Copy number,Shelving order,Item level call number type,Item level call number prefix,Item level call number,Item level call number suffix,Number of pieces,Description of pieces,Enumeration,Chronology,Volume,"Year, caption",Number of missing pieces,Missing pieces,Missing pieces date,Item damaged status,Item damaged status date,Notes,Permanent loan type,Temporary loan type,Status,Check in note,Check out note,Item permanent location,Item temporary location,Electronic access,Is bound with,Bound with titles,Tags,Holdings UUID -100d10bf-2f06-4aa0-be15-0b95b2d9f9e3,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Annex,TK5105.88815 . A58 2004 FT MEADE,,item000000000015,90000,,,,books,,book,,TK 45105.88815 A58 42004 FT MEADE,,,TK5105.88815 . A58 2004 FT MEADE,,,,,,,,,,,Not Damaged,2022-03-15,Note;Sample note;false,Can circulate,,Paged,Check in note,,Main Library,Annex,"URL relationship;URI;Link text;Materials specified;URL public note -Version of resource;http://www.loc.gov/catdir/toc/ecip0718/2007020429.html;Links available;Table of contents;Table of contents only",false,,,e3ff6133-b9a2-4d4c-a1c9-dc1867d4df19 +Item UUID,"Instance (Title, Publisher, Publication date)","Holdings (Location, Call number)",Item effective location,Effective call number,Suppress from discovery,Item HRID,Barcode,Accession number,Item identifier,Former identifier,Statistical codes,Administrative note,Material type,Copy number,Shelving order,Item level call number type,Item level call number prefix,Item level call number,Item level call number suffix,Number of pieces,Description of pieces,Enumeration,Chronology,Volume,"Year, caption",Number of missing pieces,Missing pieces,Missing pieces date,Item damaged status,Item damaged status date,Notes,Permanent loan type,Temporary loan type,Status,Check in note,Check out note,Item permanent location,Item temporary location,Electronic access,Is bound with,Bound with titles,Tags,Holdings UUID,Tenant +100d10bf-2f06-4aa0-be15-0b95b2d9f9e3,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Annex,TK5105.88815 . A58 2004 FT MEADE,,item000000000015,90000,,,,books,,book,,TK 45105.88815 A58 42004 FT MEADE,,,TK5105.88815 . A58 2004 FT MEADE,,,,,,,,,,,Not Damaged,2022-03-15,Note;Sample note;false;diku;8d0a5eca-25de-4391-81a9-236eeefdd20b,Can circulate,,Paged,Check in note,,Main Library,Annex,"URL relationship;URI;Link text;Materials specified;URL public note +Version of resource;http://www.loc.gov/catdir/toc/ecip0718/2007020429.html;Links available;Table of contents;Table of contents only",false,,,e3ff6133-b9a2-4d4c-a1c9-dc1867d4df19,diku 7212ba6a-8dcf-45a1-be9a-ffaa847c4423,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,TK5105.88815 . A58 2004 FT MEADE,,item000000000014,10101,,,,books,,book,Copy 2,TK 45105.88815 A58 42004 FT MEADE COPY 12,,,TK5105.88815 . A58 2004 FT MEADE,,,,,,,,,,,,,,Can circulate,,Paged,,,Main Library,,"URL relationship;URI;Link text;Materials specified;URL public note -Version of resource;http://www.loc.gov/catdir/toc/ecip0718/2007020429.html;Links available;Table of contents;Table of contents only",false,,,e3ff6133-b9a2-4d4c-a1c9-dc1867d4df19 +Version of resource;http://www.loc.gov/catdir/toc/ecip0718/2007020429.html;Links available;Table of contents;Table of contents only",false,,,e3ff6133-b9a2-4d4c-a1c9-dc1867d4df19,diku diff --git a/src/test/resources/output/bulk_edit_item_identifiers_output_escaped.csv b/src/test/resources/output/bulk_edit_item_identifiers_output_escaped.csv index 8260a2a97..097351a5f 100644 --- a/src/test/resources/output/bulk_edit_item_identifiers_output_escaped.csv +++ b/src/test/resources/output/bulk_edit_item_identifiers_output_escaped.csv @@ -1,3 +1,3 @@ -Item UUID,"Instance (Title, Publisher, Publication date)","Holdings (Location, Call number)",Item effective location,Effective call number,Suppress from discovery,Item HRID,Barcode,Accession number,Item identifier,Former identifier,Statistical codes,Administrative note,Material type,Copy number,Shelving order,Item level call number type,Item level call number prefix,Item level call number,Item level call number suffix,Number of pieces,Description of pieces,Enumeration,Chronology,Volume,"Year, caption",Number of missing pieces,Missing pieces,Missing pieces date,Item damaged status,Item damaged status date,Notes,Permanent loan type,Temporary loan type,Status,Check in note,Check out note,Item permanent location,Item temporary location,Electronic access,Is bound with,Bound with titles,Tags,Holdings UUID +Item UUID,"Instance (Title, Publisher, Publication date)","Holdings (Location, Call number)",Item effective location,Effective call number,Suppress from discovery,Item HRID,Barcode,Accession number,Item identifier,Former identifier,Statistical codes,Administrative note,Material type,Copy number,Shelving order,Item level call number type,Item level call number prefix,Item level call number,Item level call number suffix,Number of pieces,Description of pieces,Enumeration,Chronology,Volume,"Year, caption",Number of missing pieces,Missing pieces,Missing pieces date,Item damaged status,Item damaged status date,Notes,Permanent loan type,Temporary loan type,Status,Check in note,Check out note,Item permanent location,Item temporary location,Electronic access,Is bound with,Bound with titles,Tags,Holdings UUID,Tenant 7212ba6a-8dcf-45a1-be9a-ffaa847c4423,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,TK5105.88815 . A58 2004 FT MEADE,,item000000000014,10101,,,,books,,book,Copy 2,TK 45105.88815 A58 42004 FT MEADE COPY 12,,,TK5105.88815 . A58 2004 FT MEADE,,,,,,,,,,,,,,Can circulate,,Paged,,,Main Library,,"URL relationship;URI;Link text;Materials specified;URL public note -Version of resource;http://www.loc.gov/catdir/toc/ecip0718/2007020429.html;Links available;Table of contents;Table of contents only",false,,,e3ff6133-b9a2-4d4c-a1c9-dc1867d4df19 +Version of resource;http://www.loc.gov/catdir/toc/ecip0718/2007020429.html;Links available;Table of contents;Table of contents only",false,,,e3ff6133-b9a2-4d4c-a1c9-dc1867d4df19,diku diff --git a/src/test/resources/output/bulk_edit_item_identifiers_output_some_not_found.csv b/src/test/resources/output/bulk_edit_item_identifiers_output_some_not_found.csv index 3ad984c9c..8a3230ffc 100644 --- a/src/test/resources/output/bulk_edit_item_identifiers_output_some_not_found.csv +++ b/src/test/resources/output/bulk_edit_item_identifiers_output_some_not_found.csv @@ -1,3 +1,3 @@ -Item UUID,"Instance (Title, Publisher, Publication date)","Holdings (Location, Call number)",Item effective location,Effective call number,Suppress from discovery,Item HRID,Barcode,Accession number,Item identifier,Former identifier,Statistical codes,Administrative note,Material type,Copy number,Shelving order,Item level call number type,Item level call number prefix,Item level call number,Item level call number suffix,Number of pieces,Description of pieces,Enumeration,Chronology,Volume,"Year, caption",Number of missing pieces,Missing pieces,Missing pieces date,Item damaged status,Item damaged status date,Notes,Permanent loan type,Temporary loan type,Status,Check in note,Check out note,Item permanent location,Item temporary location,Electronic access,Is bound with,Bound with titles,Tags,Holdings UUID -100d10bf-2f06-4aa0-be15-0b95b2d9f9e3,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Annex,TK5105.88815 . A58 2004 FT MEADE,,item000000000015,90000,,,,books,,book,,TK 45105.88815 A58 42004 FT MEADE,,,TK5105.88815 . A58 2004 FT MEADE,,,,,,,,,,,Not Damaged,2022-03-15,Note;Sample note;false,Can circulate,,Paged,Check in note,,Main Library,Annex,"URL relationship;URI;Link text;Materials specified;URL public note -Version of resource;http://www.loc.gov/catdir/toc/ecip0718/2007020429.html;Links available;Table of contents;Table of contents only",false,,,e3ff6133-b9a2-4d4c-a1c9-dc1867d4df19 +Item UUID,"Instance (Title, Publisher, Publication date)","Holdings (Location, Call number)",Item effective location,Effective call number,Suppress from discovery,Item HRID,Barcode,Accession number,Item identifier,Former identifier,Statistical codes,Administrative note,Material type,Copy number,Shelving order,Item level call number type,Item level call number prefix,Item level call number,Item level call number suffix,Number of pieces,Description of pieces,Enumeration,Chronology,Volume,"Year, caption",Number of missing pieces,Missing pieces,Missing pieces date,Item damaged status,Item damaged status date,Notes,Permanent loan type,Temporary loan type,Status,Check in note,Check out note,Item permanent location,Item temporary location,Electronic access,Is bound with,Bound with titles,Tags,Holdings UUID,Tenant +100d10bf-2f06-4aa0-be15-0b95b2d9f9e3,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Annex,TK5105.88815 . A58 2004 FT MEADE,,item000000000015,90000,,,,books,,book,,TK 45105.88815 A58 42004 FT MEADE,,,TK5105.88815 . A58 2004 FT MEADE,,,,,,,,,,,Not Damaged,2022-03-15,Note;Sample note;false;diku;8d0a5eca-25de-4391-81a9-236eeefdd20b,Can circulate,,Paged,Check in note,,Main Library,Annex,"URL relationship;URI;Link text;Materials specified;URL public note +Version of resource;http://www.loc.gov/catdir/toc/ecip0718/2007020429.html;Links available;Table of contents;Table of contents only",false,,,e3ff6133-b9a2-4d4c-a1c9-dc1867d4df19,diku diff --git a/src/test/resources/output/bulk_edit_items_empty_reference.csv b/src/test/resources/output/bulk_edit_items_empty_reference.csv index bfffb9c6e..85fd19fb6 100644 --- a/src/test/resources/output/bulk_edit_items_empty_reference.csv +++ b/src/test/resources/output/bulk_edit_items_empty_reference.csv @@ -1,6 +1,6 @@ -Item UUID,"Instance (Title, Publisher, Publication date)","Holdings (Location, Call number)",Item effective location,Effective call number,Suppress from discovery,Item HRID,Barcode,Accession number,Item identifier,Former identifier,Statistical codes,Administrative note,Material type,Copy number,Shelving order,Item level call number type,Item level call number prefix,Item level call number,Item level call number suffix,Number of pieces,Description of pieces,Enumeration,Chronology,Volume,"Year, caption",Number of missing pieces,Missing pieces,Missing pieces date,Item damaged status,Item damaged status date,Notes,Permanent loan type,Temporary loan type,Status,Check in note,Check out note,Item permanent location,Item temporary location,Electronic access,Is bound with,Bound with titles,Tags,Holdings UUID -8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,emptyCallNumberType,,,,,,book,,,,,,,,,,,,,,,,,,,Can circulate,,Available,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683 -8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,emptyDamagedStatus,,,,,,book,,,,,,,,,,,,,,,,,2022-12-02,,Can circulate,,Available,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683 -8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,emptyNoteType,,,,,,book,,,,,,,,,,,,,,,,,2022-12-02,;Note;false,Can circulate,,Available,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683 -8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,emptyServicePoint,,,,,,book,,,,,,,,,,,,,,,,,2022-12-02,,Can circulate,,In transit,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683 -8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,emptyUserName,,,,,,book,,,,,,,,,,,,,,,,,2022-12-02,,Can circulate,,In transit,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683 +Item UUID,"Instance (Title, Publisher, Publication date)","Holdings (Location, Call number)",Item effective location,Effective call number,Suppress from discovery,Item HRID,Barcode,Accession number,Item identifier,Former identifier,Statistical codes,Administrative note,Material type,Copy number,Shelving order,Item level call number type,Item level call number prefix,Item level call number,Item level call number suffix,Number of pieces,Description of pieces,Enumeration,Chronology,Volume,"Year, caption",Number of missing pieces,Missing pieces,Missing pieces date,Item damaged status,Item damaged status date,Notes,Permanent loan type,Temporary loan type,Status,Check in note,Check out note,Item permanent location,Item temporary location,Electronic access,Is bound with,Bound with titles,Tags,Holdings UUID,Tenant +8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,emptyCallNumberType,,,,,,book,,,,,,,,,,,,,,,,,,,Can circulate,,Available,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683,diku +8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,emptyDamagedStatus,,,,,,book,,,,,,,,,,,,,,,,,2022-12-02,,Can circulate,,Available,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683,diku +8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,emptyNoteType,,,,,,book,,,,,,,,,,,,,,,,,2022-12-02,;Note;false;diku;null,Can circulate,,Available,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683,diku +8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,emptyServicePoint,,,,,,book,,,,,,,,,,,,,,,,,2022-12-02,,Can circulate,,In transit,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683,diku +8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,emptyUserName,,,,,,book,,,,,,,,,,,,,,,,,2022-12-02,,Can circulate,,In transit,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683,diku diff --git a/src/test/resources/output/bulk_edit_items_reference_not_found.csv b/src/test/resources/output/bulk_edit_items_reference_not_found.csv index 0323ba005..f6c34ad37 100644 --- a/src/test/resources/output/bulk_edit_items_reference_not_found.csv +++ b/src/test/resources/output/bulk_edit_items_reference_not_found.csv @@ -1,7 +1,7 @@ -Item UUID,"Instance (Title, Publisher, Publication date)","Holdings (Location, Call number)",Item effective location,Effective call number,Suppress from discovery,Item HRID,Barcode,Accession number,Item identifier,Former identifier,Statistical codes,Administrative note,Material type,Copy number,Shelving order,Item level call number type,Item level call number prefix,Item level call number,Item level call number suffix,Number of pieces,Description of pieces,Enumeration,Chronology,Volume,"Year, caption",Number of missing pieces,Missing pieces,Missing pieces date,Item damaged status,Item damaged status date,Notes,Permanent loan type,Temporary loan type,Status,Check in note,Check out note,Item permanent location,Item temporary location,Electronic access,Is bound with,Bound with titles,Tags,Holdings UUID -8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,badCallNumberType,,,,,,book,,,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,,,,,,,,,,,,,,,Can circulate,,Available,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683 -8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,badDamagedStatus,,,,,,book,,,,,,,,,,,,,,,,a6236035-b88f-4d0c-a1ca-48af9bdde3c8,2022-12-02,,Can circulate,,Available,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683 -8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,badNoteType,,,,,,book,,,,,,,,,,,,,,,,,2022-12-02,43286219-6c16-4884-b967-65b419e48f8d;Note;false,Can circulate,,Available,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683 -8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,badServicePoint,,,,,,book,,,,,,,,,,,,,,,,,2022-12-02,,Can circulate,,In transit,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683 -8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,badStatisticalCode,,,,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,book,,,,,,,,,,,,,,,,,2022-12-02,,Can circulate,,Available,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683 -8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,badUserName,,,,,,book,,,,,,,,,,,,,,,,,2022-12-02,,Can circulate,,In transit,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683 +Item UUID,"Instance (Title, Publisher, Publication date)","Holdings (Location, Call number)",Item effective location,Effective call number,Suppress from discovery,Item HRID,Barcode,Accession number,Item identifier,Former identifier,Statistical codes,Administrative note,Material type,Copy number,Shelving order,Item level call number type,Item level call number prefix,Item level call number,Item level call number suffix,Number of pieces,Description of pieces,Enumeration,Chronology,Volume,"Year, caption",Number of missing pieces,Missing pieces,Missing pieces date,Item damaged status,Item damaged status date,Notes,Permanent loan type,Temporary loan type,Status,Check in note,Check out note,Item permanent location,Item temporary location,Electronic access,Is bound with,Bound with titles,Tags,Holdings UUID,Tenant +8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,badCallNumberType,,,,,,book,,,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,,,,,,,,,,,,,,,Can circulate,,Available,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683,diku +8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,badDamagedStatus,,,,,,book,,,,,,,,,,,,,,,,a6236035-b88f-4d0c-a1ca-48af9bdde3c8,2022-12-02,,Can circulate,,Available,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683,diku +8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,badNoteType,,,,,,book,,,,,,,,,,,,,,,,,2022-12-02,43286219-6c16-4884-b967-65b419e48f8d;Note;false;diku;43286219-6c16-4884-b967-65b419e48f8d,Can circulate,,Available,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683,diku +8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,badServicePoint,,,,,,book,,,,,,,,,,,,,,,,,2022-12-02,,Can circulate,,In transit,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683,diku +8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,badStatisticalCode,,,,1d51c4da-0d5f-4303-a922-3e9d7a9b22f3,,book,,,,,,,,,,,,,,,,,2022-12-02,,Can circulate,,Available,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683,diku +8feb55ab-a2d1-4358-abb0-6ca462e7d847,title,Main Library > TK5105.88815 . A58 2004 FT MEADE,Main Library,,,it00000000001,badUserName,,,,,,book,,,,,,,,,,,,,,,,,2022-12-02,,Can circulate,,In transit,,,,,,false,,,9f5e359f-3e23-4936-942b-3827f7d5e683,diku