Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/implement ocr button #2731

Open
wants to merge 15 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/backoffice-v2/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@
"i18next-http-backend": "^2.1.1",
"leaflet": "^1.9.4",
"libphonenumber-js": "^1.10.49",
"lucide-react": "^0.239.0",
"lucide-react": "^0.445.0",
"match-sorter": "^6.3.1",
"msw": "^1.0.0",
"posthog-js": "^1.154.2",
Expand Down
4 changes: 4 additions & 0 deletions apps/backoffice-v2/public/locales/en/toast.json
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,10 @@
"pdf_certificate": {
"error": "Failed to open PDF certificate."
},
"document_ocr": {
"success": "OCR performed successfully.",
"error": "Failed to perform OCR on the document."
},
"business_report_creation": {
"success": "Merchant check created successfully.",
"error": "Error occurred while creating a merchant check.",
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
import { ctw } from '@/common/utils/ctw/ctw';
import { ComponentProps, FunctionComponent } from 'react';
import { ScanTextIcon } from 'lucide-react';

export interface IImageOCR extends ComponentProps<'div'> {
onOcrPressed?: () => void;
isOcrDisabled: boolean;
documentId: string;
}

export const ImageOCR: FunctionComponent<IImageOCR> = ({
isOcrDisabled,
onOcrPressed,
documentId,
className,
...props
}) => (
<>
<button
type={`button`}
className={ctw(
`btn btn-circle btn-ghost btn-sm bg-base-300/70 text-[0.688rem] focus:outline-primary`,
)}
onClick={() => onOcrPressed?.()}
disabled={isOcrDisabled}
>
<ScanTextIcon />
</button>
</>
);
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
import { useMutation, useQueryClient } from '@tanstack/react-query';
import { fetchWorkflowDocumentOCRResult } from '@/domains/workflows/fetchers';
import { toast } from 'sonner';
import { t } from 'i18next';
import { workflowsQueryKeys } from '@/domains/workflows/query-keys';
import { useFilterId } from '@/common/hooks/useFilterId/useFilterId';

export const useDocumentOcr = ({ workflowId }: { workflowId: string }) => {
const filterId = useFilterId();
const workflowById = workflowsQueryKeys.byId({ workflowId, filterId });
const queryClient = useQueryClient();

return useMutation({
mutationFn: ({ documentId }: { documentId: string }) => {
return fetchWorkflowDocumentOCRResult({
workflowDefinitionId: workflowId,
documentId,
});
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Ensure consistency in parameter naming

The mutation function uses workflowDefinitionId, but the hook receives workflowId as a parameter. This inconsistency could lead to confusion.

Consider one of the following options:

  1. Rename the parameter to match the usage:
-export const useDocumentOcr = ({ workflowId }: { workflowId: string }) => {
+export const useDocumentOcr = ({ workflowDefinitionId }: { workflowDefinitionId: string }) => {
  // ... (update all occurrences of workflowId to workflowDefinitionId)
  1. Or, if workflowId is the correct term, update the fetch function call:
  return fetchWorkflowDocumentOCRResult({
-   workflowDefinitionId: workflowId,
+   workflowId,
    documentId,
  });

Choose the option that best aligns with your project's terminology and the actual purpose of this ID.

Committable suggestion was skipped due to low confidence.

onSuccess: (data, variables) => {
void queryClient.invalidateQueries(workflowsQueryKeys._def);
toast.success(t('toast:document_ocr.success'));
},
onError: (_error, _variables) => {
console.error(_error);
void queryClient.invalidateQueries(workflowsQueryKeys._def);
toast.error(t('toast:document_ocr.error'));
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Refine error handling approach

While the current error handling provides user feedback and logs the error, there are a couple of points to consider:

  1. Invalidating queries on error might not be necessary and could potentially cause issues by triggering unnecessary refetches.
  2. The error details are not utilized in the user-facing error message.

Consider the following improvements:

-    onError: (_error, _variables) => {
-      console.error(_error);
-      void queryClient.invalidateQueries(workflowsQueryKeys._def);
-      toast.error(t('toast:document_ocr.error'));
+    onError: (error, variables) => {
+      console.error('OCR error:', error, 'for document:', variables.documentId);
+      toast.error(t('toast:document_ocr.error', { documentId: variables.documentId }));
     },

This change:

  1. Removes the query invalidation on error, as it's likely unnecessary.
  2. Improves error logging by including context (document ID).
  3. Potentially enriches the error message with the document ID (update your i18n file accordingly).

Also, consider adding more specific error handling if the fetchWorkflowDocumentOCRResult function provides structured error responses.

Committable suggestion was skipped due to low confidence.

});
};
18 changes: 18 additions & 0 deletions apps/backoffice-v2/src/domains/workflows/fetchers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -298,3 +298,21 @@ export const createWorkflowRequest = async ({

return handleZodError(error, workflow);
};

export const fetchWorkflowDocumentOCRResult = async ({
workflowDefinitionId,
documentId,
}: {
workflowDefinitionId: string;
documentId: string;
}) => {
const [workflow, error] = await apiClient({
method: Method.PATCH,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PATCH?

url: `${getOriginUrl(
env.VITE_API_URL,
)}/api/v1/internal/workflows/${workflowDefinitionId}/documents/${documentId}/run-ocr`,
schema: z.any(),
});

return handleZodError(error, workflow);
};
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import { CommonWorkflowEvent } from '@ballerine/common';
import { ComponentProps, FunctionComponent, useCallback, useEffect, useState } from 'react';
import { toast } from 'sonner';
import { useApproveTaskByIdMutation } from '../../../../../../domains/entities/hooks/mutations/useApproveTaskByIdMutation/useApproveTaskByIdMutation';
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,11 @@ export const MultiDocuments: FunctionComponent<IMultiDocumentsProps> = ({ value

return (
<div className={`m-2 rounded p-1`}>
<Case.Documents documents={documents} isLoading={value?.isLoading} />
<Case.Documents
documents={documents}
isLoading={value?.isLoading}
onOcrPressed={value?.onOcrPressed}
/>
</div>
);
};
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
export interface IMultiDocumentsProps {
value: {
isLoading: boolean;
onOcrPressed: () => void;
data: Array<{
imageUrl: string;
title: string;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { MotionButton } from '@/common/components/molecules/MotionButton/MotionButton';
import { checkIsBusiness } from '@/common/utils/check-is-business/check-is-business';
import { ctw } from '@/common/utils/ctw/ctw';
import { CommonWorkflowStates, StateTag, valueOrNA } from '@ballerine/common';
import { CommonWorkflowStates, isObject, StateTag, valueOrNA } from '@ballerine/common';
import { useApproveTaskByIdMutation } from '@/domains/entities/hooks/mutations/useApproveTaskByIdMutation/useApproveTaskByIdMutation';
import { useRejectTaskByIdMutation } from '@/domains/entities/hooks/mutations/useRejectTaskByIdMutation/useRejectTaskByIdMutation';
import { useRemoveDecisionTaskByIdMutation } from '@/domains/entities/hooks/mutations/useRemoveDecisionTaskByIdMutation/useRemoveDecisionTaskByIdMutation';
Expand Down Expand Up @@ -29,6 +29,7 @@ import { X } from 'lucide-react';
import * as React from 'react';
import { FunctionComponent, useCallback, useMemo } from 'react';
import { toTitleCase } from 'string-ts';
import { useDocumentOcr } from '@/domains/entities/hooks/mutations/useDocumentOcr/useDocumentOcr';

export const useDocumentBlocks = ({
workflow,
Expand Down Expand Up @@ -79,6 +80,14 @@ export const useDocumentBlocks = ({

const { mutate: mutateApproveTaskById, isLoading: isLoadingApproveTaskById } =
useApproveTaskByIdMutation(workflow?.id);
const {
mutate: mutateOCRDocument,
isLoading: isLoadingOCRDocument,
data: ocrResult,
} = useDocumentOcr({
workflowId: workflow?.id,
});

Comment on lines +83 to +90
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Inconsistent Variable Naming: Use CamelCase for mutateOCRDocument and isLoadingOCRDocument

The variables mutateOCRDocument and isLoadingOCRDocument use an uppercase OCR in the middle of their names, which is inconsistent with camelCase conventions used elsewhere in the codebase (e.g., useDocumentOcr). For consistency and readability, consider renaming these variables to mutateOcrDocument and isLoadingOcrDocument.

Apply this diff to rename the variables:

 const {
-  mutate: mutateOCRDocument,
-  isLoading: isLoadingOCRDocument,
+  mutate: mutateOcrDocument,
+  isLoading: isLoadingOcrDocument,
   data: ocrResult,
 } = useDocumentOcr({
   workflowId: workflow?.id,
 });
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const {
mutate: mutateOCRDocument,
isLoading: isLoadingOCRDocument,
data: ocrResult,
} = useDocumentOcr({
workflowId: workflow?.id,
});
const {
mutate: mutateOcrDocument,
isLoading: isLoadingOcrDocument,
data: ocrResult,
} = useDocumentOcr({
workflowId: workflow?.id,
});

const { isLoading: isLoadingRejectTaskById } = useRejectTaskByIdMutation(workflow?.id);

const { comment, onClearComment, onCommentChange } = useCommentInputLogic();
Expand Down Expand Up @@ -358,6 +367,19 @@ export const useDocumentBlocks = ({
})
.cellAt(0, 0);

const documentEntries = Object.entries(
{
...additionalProperties,
...propertiesSchema?.properties,
} ?? {},
).map(([title, formattedValue]) => {
if (isObject(formattedValue)) {
formattedValue.value ||= ocrResult?.parsedData?.[title];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In React its more important not to make mutations. Just pass the value to the right instead of formattedValue.

}

return [title, formattedValue];
});
Comment on lines +370 to +381
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Avoid Mutating formattedValue Directly

The current implementation directly mutates formattedValue by setting formattedValue.value. This could lead to unintended side effects if formattedValue is used elsewhere. Instead, consider using an immutable approach to create a new object with the updated value.

Apply this diff to prevent direct mutation:

 ).map(([title, formattedValue]) => {
   if (isObject(formattedValue)) {
-    formattedValue.value ||= ocrResult?.parsedData?.[title];
+    formattedValue = {
+      ...formattedValue,
+      value: formattedValue.value || ocrResult?.parsedData?.[title],
+    };
   }

   return [title, formattedValue];
 });

This change ensures that you are not modifying the original formattedValue object but creating a new one with the updated value.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const documentEntries = Object.entries(
{
...additionalProperties,
...propertiesSchema?.properties,
} ?? {},
).map(([title, formattedValue]) => {
if (isObject(formattedValue)) {
formattedValue.value ||= ocrResult?.parsedData?.[title];
}
return [title, formattedValue];
});
const documentEntries = Object.entries(
{
...additionalProperties,
...propertiesSchema?.properties,
} ?? {},
).map(([title, formattedValue]) => {
if (isObject(formattedValue)) {
formattedValue = {
...formattedValue,
value: formattedValue.value || ocrResult?.parsedData?.[title],
};
}
return [title, formattedValue];
});


const detailsCell = createBlocksTyped()
.addBlock()
.addCell({
Expand All @@ -370,12 +392,7 @@ export const useDocumentBlocks = ({
value: {
id,
title: `${category} - ${docType}`,
data: Object.entries(
{
...additionalProperties,
...propertiesSchema?.properties,
} ?? {},
)?.map(
data: documentEntries?.map(
([
title,
{
Expand Down Expand Up @@ -450,6 +467,7 @@ export const useDocumentBlocks = ({
type: 'multiDocuments',
value: {
isLoading: storageFilesQueryResult?.some(({ isLoading }) => isLoading),
onOcrPressed: () => mutateOCRDocument({ documentId: id }),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Inconsistent Function Name: Rename mutateOCRDocument to mutateOcrDocument

The function mutateOCRDocument is being called here. To maintain consistency with the corrected variable names and camelCase conventions, consider renaming it to mutateOcrDocument.

Apply this diff to update the function name:

 onOcrPressed: () => mutateOCRDocument({ documentId: id }),
+onOcrPressed: () => mutateOcrDocument({ documentId: id }),

Committable suggestion was skipped due to low confidence.

data:
documents?.[docIndex]?.pages?.map(
({ type, fileName, metadata, ballerineFileId }, pageIndex) => ({
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { ImageViewer } from '@/common/components/organisms/ImageViewer/ImageView
import { ctw } from '@/common/utils/ctw/ctw';
import { isPdf } from '@/common/utils/is-pdf/is-pdf';
import { useDocumentsToolbarLogic } from '@/pages/Entity/components/Case/hooks/useDocumentsToolbarLogic/useDocumentsToolbarLogic';
import { ImageOCR } from '@/common/components/molecules/ImageOCR/ImageOCR';

export const DocumentsToolbar: FunctionComponent<{
image: { id: string; imageUrl: string; fileType: string; fileName: string };
Expand All @@ -13,14 +14,18 @@ export const DocumentsToolbar: FunctionComponent<{
onRotateDocument: () => void;
onOpenDocumentInNewTab: (id: string) => void;
shouldDownload: boolean;
onOcrPressed?: () => void;
shouldOCR: boolean;
fileToDownloadBase64: string;
}> = ({
image,
isLoading,
hideOpenExternalButton,
onRotateDocument,
onOpenDocumentInNewTab,
onOcrPressed,
shouldDownload,
shouldOCR,
fileToDownloadBase64,
}) => {
const { onOpenInNewTabClick } = useDocumentsToolbarLogic({
Expand All @@ -31,6 +36,11 @@ export const DocumentsToolbar: FunctionComponent<{

return (
<div className={`absolute z-50 flex space-x-2 bottom-right-6`}>
{shouldOCR && image && (
<div className={`gap-y-50 mb-10 flex h-full flex-col items-center`}>
<ImageOCR isOcrDisabled={!shouldOCR} onOcrPressed={onOcrPressed} />
</div>
)}
{!hideOpenExternalButton && !isLoading && image?.id && (
<button
type={`button`}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { ImageViewer } from '@/common/components/organisms/ImageViewer/ImageView
import { ctw } from '@/common/utils/ctw/ctw';
import { keyFactory } from '@/common/utils/key-factory/key-factory';
import { DocumentsToolbar } from '@/pages/Entity/components/Case/Case.Documents.Toolbar';
import { useDocuments } from './hooks/useDocuments/useDocuments';
import { useDocumentsLogic } from './hooks/useDocuments/useDocumentsLogic';
import { IDocumentsProps } from './interfaces';

/**
Expand All @@ -24,6 +24,7 @@ import { IDocumentsProps } from './interfaces';
*/
export const Documents: FunctionComponent<IDocumentsProps> = ({
documents,
onOcrPressed,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Ensure 'onOcrPressed' is defined in 'IDocumentsProps'

You've added onOcrPressed to the component props. Please ensure that onOcrPressed is defined in the IDocumentsProps interface to maintain type safety and prevent potential TypeScript errors.

isLoading,
hideOpenExternalButton,
}) => {
Expand All @@ -32,7 +33,6 @@ export const Documents: FunctionComponent<IDocumentsProps> = ({
onCrop,
onCancelCrop,
isCropping,
onOcr,
selectedImageRef,
initialImage,
skeletons,
Expand All @@ -45,8 +45,9 @@ export const Documents: FunctionComponent<IDocumentsProps> = ({
onTransformed,
isRotatedOrTransformed,
shouldDownload,
shouldOCR,
fileToDownloadBase64,
} = useDocuments(documents);
} = useDocumentsLogic(documents);

return (
<ImageViewer selectedImage={selectedImage} onSelectImage={onSelectImage}>
Expand Down Expand Up @@ -88,6 +89,8 @@ export const Documents: FunctionComponent<IDocumentsProps> = ({
onOpenDocumentInNewTab={onOpenDocumentInNewTab}
// isRotatedOrTransformed={isRotatedOrTransformed}
shouldDownload={shouldDownload}
shouldOCR={shouldOCR}
onOcrPressed={onOcrPressed}
// isCropping={isCropping}
// isLoadingOCR={isLoadingOCR}
// onCancelCrop={onCancelCrop}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
import { t } from 'i18next';
import { toast } from 'sonner';
import { ComponentProps, useCallback, useRef, useState } from 'react';

import { IDocumentsProps } from '../../interfaces';
Expand All @@ -11,59 +9,17 @@ import { useFilterId } from '@/common/hooks/useFilterId/useFilterId';
import { useTesseract } from '@/common/hooks/useTesseract/useTesseract';
import { createArrayOfNumbers } from '@/common/utils/create-array-of-numbers/create-array-of-numbers';
import { useStorageFileByIdQuery } from '@/domains/storage/hooks/queries/useStorageFileByIdQuery/useStorageFileByIdQuery';
import { copyToClipboard } from '@/common/utils/copy-to-clipboard/copy-to-clipboard';
import { useCustomerQuery } from '@/domains/customer/hook/queries/useCustomerQuery/useCustomerQuery';

export const useDocuments = (documents: IDocumentsProps['documents']) => {
export const useDocumentsLogic = (documents: IDocumentsProps['documents']) => {
const initialImage = documents?.[0];
const {
crop,
isCropping,
onCrop,
cropImage,
toggleOnIsCropping,
toggleOffIsCropping,
onCancelCrop,
} = useCrop();
const { data: customer } = useCustomerQuery();
const { crop, isCropping, onCrop, onCancelCrop } = useCrop();
const [isLoadingOCR, , toggleOnIsLoadingOCR, toggleOffIsLoadingOCR] = useToggle(false);
const selectedImageRef = useRef<HTMLImageElement>();
const recognize = useTesseract();
const filterId = useFilterId();
const onOcr = useCallback(async () => {
if (!isCropping) {
toggleOnIsCropping();

return;
}

toggleOnIsLoadingOCR();

try {
const croppedBase64 = await cropImage(selectedImageRef.current);
const result = await recognize(croppedBase64);
const text = result?.data?.text;

if (!text) {
throw new Error('No document OCR text found');
}

await copyToClipboard(text)();
} catch (err) {
console.error(err);

toast.error(t('toast:ocr_document_error'));
}

toggleOffIsLoadingOCR();
toggleOffIsCropping();
}, [
isCropping,
toggleOnIsLoadingOCR,
toggleOffIsLoadingOCR,
toggleOffIsCropping,
toggleOnIsCropping,
cropImage,
recognize,
]);
const skeletons = createArrayOfNumbers(4);
const [selectedImage, setSelectedImage] = useState<{
imageUrl: string;
Expand Down Expand Up @@ -114,7 +70,7 @@ export const useDocuments = (documents: IDocumentsProps['documents']) => {
onCrop,
onCancelCrop,
isCropping,
onOcr,
shouldOCR: true,
selectedImageRef,
initialImage,
skeletons,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ export interface IDocumentsProps {
fileName: string;
title: string;
}>;
onOcrPressed: (documentId: string) => void;
isLoading?: boolean;
hideOpenExternalButton?: boolean;
}
Expand Down
Loading
Loading