Skip to content

Commit

Permalink
mod-data-export-worker 4.0.0 upgrades
Browse files Browse the repository at this point in the history
* Refine fetching to allow filtering on: fee type - closes #16

* compared UUIDs instead of strings

* Update submodule

* update submodule (again)

* Add location ID filter (closes #19)

* Add location ID filter (closes #19) (#55)

* Fetch all ancillary data (close #48)

* Create filter and job config context architecture

* Add location ID filter (closes #19)

* Remove unnecessary import and code

* Add support for conditional constant token

* Clean up unnecessary imports

* Add support for constant conditional token and clean up imports

* Add service point ID filter

* add supoort for item metadata data token - closes #33

* allow filtering on fine amount - closes #14

* Update submodule

* Fix typo

* Changes to aggregate token

* Update submodule

* Update submodule

* update submodule

* Fix aggregate amounts

* Update submodule

* Add pass filter

* Assorted token formatter improvements

* Update submodule

* Create new class to store aggregated data by user

* Properly handle optional user and item data (closes #33, closes #36)

* Fix submodule reversion

* Properly handle optional user and item data (closes #33, closes #36) (#69)

* Fetch all ancillary data (close #48)

* Create filter and job config context architecture

* Add location ID filter (closes #19)

* Properly handle optional user and item data (closes #33, closes #36)

* Fix submodule reversion

* get filtered accounts in TransferFeesFinesTasklet - closes #68

* Fix typo

* Prevent empty user/item queries

* Add support for FeeDate data token

* Updated submodule folio-export-common

* Update submodule

* Add aggregating accounts implemetation in reader

* update folio-export-common

* implement transfer fee/fines functionality - closes #8

* Create new reader for the aggregated accounts and clean up old reader

* Create new filter evaluator for aggregate accounts

* update to FEC

* update to fec

* check size of accountsToBeTransferred List for each filter condition before creating transfer request

* Updated submodule folio-export-common

* Clean up imports

* Implement aggregated account filterer

* Delete to reset commit

* Create new filter evaluator for aggregate accounts

* Add log error to invalid case condition

* Use int instead for comparing values

* Add support for formatting aggregated accounts by users

* Add findTotalAmount() method to AggregatedAccounts class

* Change the user formatting methods' parameter to reuse it for aggregate

* Change to use findTotalAmount() method instead

* Delete comment

* Fix bugs for aggregate reader, formatter, and filter evaluator

* Implement conditional flow for aggregate by patron job

* Add check for groupByPatron for conditional flow

* Update src/main/java/org/folio/dew/batch/bursarfeesfines/BursarExportJobConfig.java

Co-authored-by: Noah Overcash <ncovercash@ua.edu>

* Clean up code

* Remove redundant call to .user()

* Add transfer aggregated accounts

* closes #81

* pull from folio-export-common

* update folio-export-common

* Add support for new data token - Conditional - #81 (#83)

* closes #81

* pull from folio-export-common

* update folio-export-common

* added support for BursarExportFilterFeeFineOwner

* closes #86

* closes #86

* fixed formatting

* removed support for CORNELL_EXTERNAL_SYSTEM_ID option in UserData data token

* closes #101

* Add jacoco

* Use hamcrest

* resolve issues

* Changes to help with testing

* Closes #100

* Closes #104

* resolve issues

* Fix bad jacoco configuration

* Fix tests

* resolve issues

* Cleanup bursar export service

* More coverage for BursarFilterAggregateEvaluatorTest

* Add testing aggregateEvaluate with inapplicable filter

* Unit test for BursarExportUtils

* Modify and write unit test for AggregatedAccountReader

* added barebones for integration testing

* closes #105

* Update sonar

* Correct java version

* Correct java distribution

* Correct sonar key

* Test without actions?

* Test inconsequential sonar

* Revert "Test without actions?"

This reverts commit 869c2ad.

* Update submodule

* Fix merge error

* skip bad tests

* Add sonar URL

* Try only one test for speed

* bruh

* Restore all sonar tests

* Update checkout depth for sonar recency

* no clue how i did that...

* Move bursar integration tests into separate folder (#126)

* Update submodule

* Move no fees fines test

* Remove temporary test code in AccountReader

* resolve issues with original pull request :#122

---------

Co-authored-by: GoZaddy <faruqyusuff437@gmail.com>

* Modify and write unit test for BursarTokenFormatter

* Clean up unnecessary logs

* Remove log import

* Write unit test for BursarTokenFormatter - closes #102 (#127)

* Modify and write unit test for BursarTokenFormatter

* Clean up unnecessary logs

* Remove log import

* Create an integration test for: one fee/fine criteria - #107 (#128)

* Update submodule

* Move no fees fines test

* Remove temporary test code in AccountReader

* resolve issues with original pull request :#122

* closes #107

* fixed issues in PR

* fixed formatting

* check that filesInStorage is not null

* updates

---------

Co-authored-by: Noah Overcash <novercash@ebsco.com>

* Clean up code smells

* Change log errors to be more specific

* Make number 7 into a constant

* Clean up import

* Clean up code smells (#129)

* Modify and write unit test for BursarTokenFormatter

* Clean up unnecessary logs

* Remove log import

* Clean up code smells

* Change log errors to be more specific

* Make number 7 into a constant

* Clean up import

* Create integration test for no fee fine matching criteria

* Add expected output

* Create integration test for no fee fine matching criteria (#131)

* Modify and write unit test for BursarTokenFormatter

* Clean up unnecessary logs

* Remove log import

* Clean up code smells

* Change log errors to be more specific

* Make number 7 into a constant

* Clean up import

* Create integration test for no fee fine matching criteria

* Add expected output

* Create integration test for multiple matching fees

* Create integration test for multiple fees matching criteria (#132)

* Remove non-ISO week year date format options

* Add checks for null/optional account dates

* Create integration test that uses most output tokens

* Clean up unit tests

* Create integration test that uses most output tokens (#133)

* Modify and write unit test for BursarTokenFormatter

* Clean up unnecessary logs

* Remove log import

* Clean up code smells

* Change log errors to be more specific

* Make number 7 into a constant

* Clean up import

* Create integration test for no fee fine matching criteria

* Add expected output

* Create integration test for multiple matching fees

* Create integration test that uses most output tokens

* Clean up unit tests

* Clean up code

* Create integration test for fees/fines matching initial criteria but not aggregate criteria

* Clean up imports

* Correct mocked request content

* Create integration test for fee/fines that match initial criteria but not aggregate criteria (#135)

* Modify and write unit test for BursarTokenFormatter

* Clean up unnecessary logs

* Remove log import

* Clean up code smells

* Change log errors to be more specific

* Make number 7 into a constant

* Clean up import

* Create integration test for no fee fine matching criteria

* Add expected output

* Create integration test for multiple matching fees

* Create integration test that uses most output tokens

* Clean up unit tests

* Clean up code

* Create integration test for fees/fines matching initial criteria but not aggregate criteria

* Clean up imports

* Correct mocked request content

* Closes #112, #114

* Remove temporary GA

* Fix mismatch file

* Fix: add new line to data token of MultipleFeeFinesAcrossPatronsAggregateTest

* Update submodule

* Add draft of new test

* Create test for BursarExportService

* Fix modifier

* Create test for BursarExportService (#137)

* Add draft of new test

* Create test for BursarExportService

* Fix modifier

* Fix assorted unsafe null checks to Item

* Modify and complete BursarExportService test

* Clean up and add necessary imports

* Cleanup

* Add test for getAllAccounts() and refactor other tests

* Finish integration test for BursarExportService (#138)

* Modify and complete BursarExportService test

* Clean up and add necessary imports

* Cleanup

* Add test for getAllAccounts() and refactor other tests

---------

Co-authored-by: Noah Overcash <novercash@ebsco.com>

* Add descriptive failure error messages (closes #139) (#140)

* Add descriptive failure error messages (closes #139)

* Change exception handling

* Change assertion method

* Update module descriptor

* Update version in pom.xml

* Update module descriptor

* Revert changes

* Update module descriptor (#141)

* Modify and complete BursarExportService test

* Clean up and add necessary imports

* Cleanup

* Add test for getAllAccounts() and refactor other tests

* Update module descriptor

* Update version in pom.xml

* Update module descriptor

* Revert changes

---------

Co-authored-by: Noah Overcash <novercash@ebsco.com>

* Updated submodule folio-export-common

* Updated submodule folio-export-common

* Updated submodule folio-export-common

* Updated submodule folio-export-common

* Correct UserData types and add date options

* Added tests for processDateToken new options (#145)

* Remove test log

* Fix code smell

* Add coverage to modified code

* Add date formatting options (and correct userData types) (#144)

* Modify and complete BursarExportService test

* Clean up and add necessary imports

* Cleanup

* Add test for getAllAccounts() and refactor other tests

* Update module descriptor

* Update version in pom.xml

* Update module descriptor

* Revert changes

* Updated submodule folio-export-common

* Updated submodule folio-export-common

* Correct UserData types and add date options

* Added tests for processDateToken new options (#145)

* Remove test log

* Fix code smell

* Add coverage to modified code

---------

Co-authored-by: Noah Overcash <novercash@ebsco.com>
Co-authored-by: Dane Tsao <danetsaoo@gmail.com>

* Updated submodule folio-export-common

* Updated submodule folio-export-common

* Add conditions to filter age  (#147)

* Add condition for age filter and additional tests

* Remove unnecessary log

* fix bug: issue #148 (#149)

* fix bug: issue #148

* fix bug: issue #148

---------

Co-authored-by: Noah Overcash <ncovercash@ua.edu>
Co-authored-by: GoZaddy <faruqyusuff437@gmail.com>
Co-authored-by: Noah Overcash <novercash@ebsco.com>
Co-authored-by: Dane Tsao <danetsaoo@gmail.com>
  • Loading branch information
5 people authored Sep 26, 2023
1 parent 6bcc1a4 commit 3891ce7
Show file tree
Hide file tree
Showing 62 changed files with 7,649 additions and 1,306 deletions.
3 changes: 2 additions & 1 deletion .gitmodules
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
[submodule "folio-export-common"]
path = folio-export-common
url = https://github.com/folio-org/folio-export-common.git
url = https://github.com/ualibweb/folio-export-common.git
branch = master
184 changes: 95 additions & 89 deletions descriptors/ModuleDescriptor-template.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@
"id": "users",
"version": "15.4 16.0"
},
{
"id": "inventory",
"version": "13.0"
},
{
"id": "configuration",
"version": "2.0"
Expand Down Expand Up @@ -129,6 +133,10 @@
{
"id": "statistical-codes",
"version": "1.0"
},
{
"id": "service-points",
"version": "3.3"
}
],
"provides": [
Expand All @@ -137,9 +145,9 @@
"version": "3.0",
"handlers": [
{
"methods": [ "POST" ],
"methods": ["POST"],
"pathPattern": "/bulk-edit/{id}/upload",
"permissionsRequired": [ "bulk-edit.item.post" ],
"permissionsRequired": ["bulk-edit.item.post"],
"modulePermissions": [
"addresstypes.collection.get",
"addresstypes.item.get",
Expand Down Expand Up @@ -188,9 +196,11 @@
]
},
{
"methods": [ "POST" ],
"methods": ["POST"],
"pathPattern": "/bulk-edit/{id}/item-content-update/upload",
"permissionsRequired": [ "bulk-edit.items-content-update.collection.post" ],
"permissionsRequired": [
"bulk-edit.items-content-update.collection.post"
],
"modulePermissions": [
"inventory-storage.call-number-types.item.get",
"inventory-storage.call-number-types.collection.get",
Expand Down Expand Up @@ -218,9 +228,11 @@
]
},
{
"methods": [ "POST" ],
"methods": ["POST"],
"pathPattern": "/bulk-edit/{id}/user-content-update/upload",
"permissionsRequired": [ "bulk-edit.users-content-update.collection.post" ],
"permissionsRequired": [
"bulk-edit.users-content-update.collection.post"
],
"modulePermissions": [
"users.collection.get",
"users.item.get",
Expand All @@ -234,9 +246,11 @@
]
},
{
"methods": [ "POST" ],
"methods": ["POST"],
"pathPattern": "/bulk-edit/{id}/holdings-content-update/upload",
"permissionsRequired": [ "bulk-edit.holdings-content-update.collection.post" ],
"permissionsRequired": [
"bulk-edit.holdings-content-update.collection.post"
],
"modulePermissions": [
"inventory-storage.instances.item.get",
"inventory-storage.holdings-types.collection.get",
Expand All @@ -257,9 +271,9 @@
]
},
{
"methods": [ "POST" ],
"methods": ["POST"],
"pathPattern": "/bulk-edit/{id}/start",
"permissionsRequired": [ "bulk-edit.start.item.post" ],
"permissionsRequired": ["bulk-edit.start.item.post"],
"modulePermissions": [
"users.collection.get",
"users.item.get",
Expand Down Expand Up @@ -309,13 +323,9 @@
]
},
{
"methods": [
"POST"
],
"methods": ["POST"],
"pathPattern": "/bulk-edit/{jobId}/roll-back",
"permissionsRequired": [
"bulk-edit.roll-back.item.post"
],
"permissionsRequired": ["bulk-edit.roll-back.item.post"],
"modulePermissions": [
"users.collection.get",
"users.item.get",
Expand All @@ -330,19 +340,19 @@
]
},
{
"methods": [ "GET" ],
"methods": ["GET"],
"pathPattern": "/bulk-edit/{jobId}/preview/users",
"permissionsRequired": [ "bulk-edit.preview-users.collection.get" ],
"permissionsRequired": ["bulk-edit.preview-users.collection.get"],
"modulePermissions": [
"users.collection.get",
"addresstypes.collection.get",
"usergroups.collection.get"
]
},
{
"methods": [ "GET" ],
"methods": ["GET"],
"pathPattern": "/bulk-edit/{jobId}/preview/items",
"permissionsRequired": [ "bulk-edit.preview-items.collection.get" ],
"permissionsRequired": ["bulk-edit.preview-items.collection.get"],
"modulePermissions": [
"inventory.items.collection.get",
"inventory-storage.material-types.collection.get",
Expand All @@ -361,9 +371,9 @@
]
},
{
"methods": [ "GET" ],
"methods": ["GET"],
"pathPattern": "/bulk-edit/{jobId}/preview/holdings",
"permissionsRequired": [ "bulk-edit.preview-holdings.collection.get" ],
"permissionsRequired": ["bulk-edit.preview-holdings.collection.get"],
"modulePermissions": [
"inventory-storage.holdings.collection.get",
"inventory-storage.holdings-types.collection.get",
Expand All @@ -383,33 +393,39 @@
]
},
{
"methods": [ "GET" ],
"methods": ["GET"],
"pathPattern": "/bulk-edit/{jobId}/errors",
"permissionsRequired": [ "bulk-edit.errors.collection.get" ],
"permissionsRequired": ["bulk-edit.errors.collection.get"],
"modulePermissions": []
},
{
"methods": [ "GET" ],
"methods": ["GET"],
"pathPattern": "/bulk-edit/{jobId}/preview/updated-items/download",
"permissionsRequired": [ "bulk-edit.preview.updated-items.collection.get" ],
"permissionsRequired": [
"bulk-edit.preview.updated-items.collection.get"
],
"modulePermissions": []
},
{
"methods": [ "GET" ],
"methods": ["GET"],
"pathPattern": "/bulk-edit/{jobId}/preview/updated-users/download",
"permissionsRequired": [ "bulk-edit.preview.updated-users.collection.get" ],
"permissionsRequired": [
"bulk-edit.preview.updated-users.collection.get"
],
"modulePermissions": []
},
{
"methods": [ "GET" ],
"methods": ["GET"],
"pathPattern": "/bulk-edit/{jobId}/preview/updated-holdings/download",
"permissionsRequired": [ "bulk-edit.preview.updated-holdings.collection.get" ],
"permissionsRequired": [
"bulk-edit.preview.updated-holdings.collection.get"
],
"modulePermissions": []
},
{
"methods": [ "GET" ],
"methods": ["GET"],
"pathPattern": "/refresh-presigned-url",
"permissionsRequired": [ "refresh-presigned-url.get" ],
"permissionsRequired": ["refresh-presigned-url.get"],
"modulePermissions": []
}
]
Expand All @@ -420,9 +436,7 @@
"interfaceType": "system",
"handlers": [
{
"methods": [
"POST"
],
"methods": ["POST"],
"pathPattern": "/_/tenant",
"modulePermissions": [
"circulation-logs.collection.get",
Expand All @@ -438,10 +452,7 @@
]
},
{
"methods": [
"GET",
"DELETE"
],
"methods": ["GET", "DELETE"],
"pathPattern": "/_/tenant/{id}"
}
]
Expand All @@ -452,49 +463,47 @@
"interfaceType": "multiple",
"handlers": [
{
"methods": [
"GET"
],
"methods": ["GET"],
"pathPattern": "/_/jsonSchemas"
}
]
}
],
"permissionSets": [
{
"permissionName" : "bulk-edit.item.post",
"displayName" : "upload identifiers list",
"description" : "Upload list of item identifiers"
"permissionName": "bulk-edit.item.post",
"displayName": "upload identifiers list",
"description": "Upload list of item identifiers"
},
{
"permissionName" : "bulk-edit.items-content-update.collection.post",
"displayName" : "upload items content update collection",
"description" : "Upload collection of item content updates"
"permissionName": "bulk-edit.items-content-update.collection.post",
"displayName": "upload items content update collection",
"description": "Upload collection of item content updates"
},
{
"permissionName" : "bulk-edit.users-content-update.collection.post",
"displayName" : "upload users content update collection",
"description" : "Upload collection of user content updates"
"permissionName": "bulk-edit.users-content-update.collection.post",
"displayName": "upload users content update collection",
"description": "Upload collection of user content updates"
},
{
"permissionName" : "bulk-edit.holdings-content-update.collection.post",
"displayName" : "upload holdings records content update collection",
"description" : "Upload collection of holdings records content updates"
"permissionName": "bulk-edit.holdings-content-update.collection.post",
"displayName": "upload holdings records content update collection",
"description": "Upload collection of holdings records content updates"
},
{
"permissionName" : "bulk-edit.preview.updated-items.collection.get",
"displayName" : "download updated items preview csv",
"description" : "Download csv of preview updated items"
"permissionName": "bulk-edit.preview.updated-items.collection.get",
"displayName": "download updated items preview csv",
"description": "Download csv of preview updated items"
},
{
"permissionName" : "bulk-edit.preview.updated-users.collection.get",
"displayName" : "download updated users preview csv",
"description" : "Download csv with preview of updated users"
"permissionName": "bulk-edit.preview.updated-users.collection.get",
"displayName": "download updated users preview csv",
"description": "Download csv with preview of updated users"
},
{
"permissionName" : "bulk-edit.preview.updated-holdings.collection.get",
"displayName" : "download updated holdings records preview csv",
"description" : "Download csv with preview of updated holdings records"
"permissionName": "bulk-edit.preview.updated-holdings.collection.get",
"displayName": "download updated holdings records preview csv",
"description": "Download csv with preview of updated holdings records"
},
{
"permissionName": "bulk-edit.start.item.post",
Expand All @@ -507,36 +516,35 @@
"description": "Roll back list of uploaded items"
},
{
"permissionName" : "bulk-edit.preview-users.collection.get",
"displayName" : "get N users for preview",
"description" : "Get N users for preview"
"permissionName": "bulk-edit.preview-users.collection.get",
"displayName": "get N users for preview",
"description": "Get N users for preview"
},
{
"permissionName" : "bulk-edit.preview-items.collection.get",
"displayName" : "get N items for preview",
"description" : "Get N items for preview"
"permissionName": "bulk-edit.preview-items.collection.get",
"displayName": "get N items for preview",
"description": "Get N items for preview"
},
{
"permissionName" : "bulk-edit.preview-holdings.collection.get",
"displayName" : "get N holdings for preview",
"description" : "Get N holdings for preview"
"permissionName": "bulk-edit.preview-holdings.collection.get",
"displayName": "get N holdings for preview",
"description": "Get N holdings for preview"
},
{
"permissionName" : "bulk-edit.errors.collection.get",
"displayName" : "get errors for preview",
"description" : "Get errors for preview"

"permissionName": "bulk-edit.errors.collection.get",
"displayName": "get errors for preview",
"description": "Get errors for preview"
},
{
"permissionName" : "refresh-presigned-url.get",
"displayName" : "Get refreshed presigned url for export file",
"description" : "Get refreshed presigned url for export file"
"permissionName": "refresh-presigned-url.get",
"displayName": "Get refreshed presigned url for export file",
"description": "Get refreshed presigned url for export file"
},
{
"permissionName" : "bulk-edit.all",
"displayName" : "bulk-edit all",
"description" : "All permissions for bulk-edit module",
"subPermissions" : [
"permissionName": "bulk-edit.all",
"displayName": "bulk-edit all",
"description": "All permissions for bulk-edit module",
"subPermissions": [
"bulk-edit.item.post",
"bulk-edit.items-content-update.collection.post",
"bulk-edit.users-content-update.collection.post",
Expand All @@ -553,12 +561,10 @@
]
},
{
"permissionName" : "refresh-presigned-url.all",
"displayName" : "refresh-presigned-url all",
"description" : "All permissions for refresh presigned url",
"subPermissions" : [
"refresh-presigned-url.get"
]
"permissionName": "refresh-presigned-url.all",
"displayName": "refresh-presigned-url all",
"description": "All permissions for refresh presigned url",
"subPermissions": ["refresh-presigned-url.get"]
}
],
"launchDescriptor": {
Expand Down
2 changes: 1 addition & 1 deletion folio-export-common
Submodule folio-export-common updated 37 files
+45 −0 schemas/bursar-export/bursarExportJob.json
+50 −0 schemas/bursar-export/filter/bursarExportFilter.json
+26 −0 schemas/bursar-export/filter/bursarExportFilterAge.json
+28 −0 schemas/bursar-export/filter/bursarExportFilterAggregate.json
+24 −0 schemas/bursar-export/filter/bursarExportFilterAmount.json
+22 −0 schemas/bursar-export/filter/bursarExportFilterCondition.json
+16 −0 schemas/bursar-export/filter/bursarExportFilterFeeFineOwner.json
+16 −0 schemas/bursar-export/filter/bursarExportFilterFeeType.json
+16 −0 schemas/bursar-export/filter/bursarExportFilterLocation.json
+15 −0 schemas/bursar-export/filter/bursarExportFilterNegation.json
+12 −0 schemas/bursar-export/filter/bursarExportFilterPass.json
+16 −0 schemas/bursar-export/filter/bursarExportFilterPatronGroup.json
+16 −0 schemas/bursar-export/filter/bursarExportFilterServicePoint.json
+3 −3 schemas/bursar-export/legacy/legacyBursarFeeFines.json
+3 −3 schemas/bursar-export/legacy/legacyBursarFeeFinesTypeMapping.json
+26 −0 schemas/bursar-export/legacy/legacyExportTypeSpecificParameters.json
+101 −0 schemas/bursar-export/legacy/legacyJob.json
+23 −0 schemas/bursar-export/legacy/legacyJobCollection.json
+50 −0 schemas/bursar-export/tokens/bursarExportDataToken.json
+16 −0 schemas/bursar-export/tokens/bursarExportHeaderFooter.json
+23 −0 schemas/bursar-export/tokens/bursarExportTokenAggregate.json
+31 −0 schemas/bursar-export/tokens/bursarExportTokenConditional.json
+15 −0 schemas/bursar-export/tokens/bursarExportTokenConstant.json
+21 −0 schemas/bursar-export/tokens/bursarExportTokenCurrentDate.json
+21 −0 schemas/bursar-export/tokens/bursarExportTokenDateType.json
+18 −0 schemas/bursar-export/tokens/bursarExportTokenFeeAmount.json
+28 −0 schemas/bursar-export/tokens/bursarExportTokenFeeDate.json
+19 −0 schemas/bursar-export/tokens/bursarExportTokenFeeMetadata.json
+30 −0 schemas/bursar-export/tokens/bursarExportTokenItemData.json
+30 −0 schemas/bursar-export/tokens/bursarExportTokenLengthControl.json
+19 −0 schemas/bursar-export/tokens/bursarExportTokenUserData.json
+22 −0 schemas/bursar-export/tokens/bursarExportTokenUserDataOptional.json
+37 −0 schemas/bursar-export/transfer/bursarExportTransferCriteria.json
+1 −1 schemas/exportTypeSpecificParameters.json
+5 −2 schemas/feesfines/feefineaction.json
+59 −0 schemas/feesfines/owner.json
+25 −0 schemas/feesfines/ownerDataCollection.json
Loading

0 comments on commit 3891ce7

Please sign in to comment.