Bulkrax OAI importer issues

[ShanaLMoore]

I created two Bulkrax OAI importers on demo.adventist-knapsack. Both importers pulled 10 records from the adl:image OAI set. The importers both have status of "Complete (with failures)." The error code is ArgumentError. All 20 records came into the repo, but no files came in.

1. First importer, selected to skip thumbnails
2. Second importer, selected to include thumbnails

The importers did not map metadata exactly according to the ADL and Hyku Maps document.

Work on Demo (first importer): https://demo.adventist-knapsack-staging.notch8.cloud/concern/images/20119303_untitled?locale=en
Work on ADL Prod (first importer): https://adl.b2.adventistdigitallibrary.org/concern/images/20119303_untitled
Work's OAI metadata: http://oai.adventistdigitallibrary.org/OAI-script?verb=GetRecord&metadataPrefix=oai_adl&identifier=20119303

Work on Demo (second importer): https://demo.adventist-knapsack-staging.notch8.cloud/concern/images/20119409_may_covington?locale=en
Work on ADL Prod (second importer): https://adl.b2.adventistdigitallibrary.org/concern/images/20119409_may_covington
Work's OAI metadata: http://oai.adventistdigitallibrary.org/OAI-script?verb=GetRecord&metadataPrefix=oai_adl&identifier=20119409

MAPPING ISSUES:

• < identifier> should map to the Identifier (local) field. It did not map at all.
• <aark_id> should map to AARK Identifier. It mapped there and also mapped to Identifier (local). Note that on prod it maps to both, creating a second Identifier (local) entry after the contents of < identifier > map there.
• Related URL and Thumbnail URL failed to map. No files imported. No URL shows in the related URL field. Expected behavior would bring in files and show the URL.

[ShanaLMoore]

Continued testing shows that all OAI sets import with status "Complete (with failures)." Consistent ArgumentError across all tests.

Checking metadata mapping for all OAI sets and work types that import via existing Adventist OAI shows same identifier and AARK identifier mapping errors and failure to import files. Other fields map as expected.

Work types tested:
Published work (adl:book and adl:issue)
Generic work (adl:other)
Thesis work (adl:thesis)

[ShanaLMoore]

Please note that importers I created for testing on 10/9 did not have files attached at the time, and no files have rendered in the repo. Today (10/11) I reran an adl:image importer and selected "reharvest." The files are still not showing up.

https://demo.adventist-knapsack-staging.notch8.cloud/importers/35?locale=en

[ShanaLMoore]

Triage:

• On previous adventist we're using v5.3.0
• Knapsack uses v5.4.1

I have compared the parser mappings and Knapsack adds a handler for description.abstract import. Which means we should have the same mappings.

I'll be looking into the loading sequence as well as the difference between v5.3.0 and v5.4.1 to see what might have introduced the errors.

[ShanaLMoore]

One possible bug introduced is from this PR: samvera/bulkrax#853

[ShanaLMoore]

Team, I've continued testing Bulkrax imports to Knapsack, and I want to report that a CSV with a valid URL in the Related URL field has not imported the file. The importer says it failed due to an argument error, but the metadata has imported and the work is created. The file is what's missing, and it did import correctly on SDAPI staging (see links below). The title of this ticket should perhaps more accurately read "Bulkrax importer issues on Knapsack." Both OAI and CSV imports are impacted.

Knapsack (file failed to import): https://demo.adventist-knapsack-staging.notch8.cloud/importers/80?locale=en
SDAPI Staging (file imported as expected): https://sdapi.s2.adventistdigitallibrary.org/importers/141?locale=en

This is critical and I would mark this ticket among the highest priorities to fix with whatever remains of our Knapsack/upgrade hours.

[ShanaLMoore]

An update: As of today (11-29-2023), trying to create either a CSV or OAI importer causes an error message to appear, and nothing works. The importers don't create and fail. They just don't create.

[ShanaLMoore]

Thanks for checking @KatharineV we'll take a look

[ShanaLMoore]

@KatharineV seems it was a Fedora issue, we've since restarted it and it should be working, can you check again when you get a chance?

[ShanaLMoore]

@kirkkwang The CSV importer worked beautifully this time: https://adl.adventist-knapsack-staging.notch8.cloud/importers/35?locale=en

The OAI importer looks successful, but the works it created won't open. I can see them in the catalog view, but trying to open the works generates an error message. Could it be related to the Fedora issue?

https://adl.adventist-knapsack-staging.notch8.cloud/concern/theses/20121848_problems_in_presenting_the_gospel_to_the_hindu_mind

[ShanaLMoore]

@KatharineV I believe this issue has been resolved. I clicked on the link you've provided and see the work.

[KatharineV]

@ShanaLMoore It's weirdly still not loading for me! I tried Firefox, Edge, and Chrome in case it was a browser issue. I got the "We're sorry but something went wrong" message on all three attempts.

[ShanaLMoore]

Ok! I see it happening now too. Diving in! Pulling this back to Softserv QA til we can figure it out.

[ShanaLMoore]

dupe