-
Notifications
You must be signed in to change notification settings - Fork 22
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* TLDR-405 remove is_one_column_document_list (#332) * TLDR-405 remove is_one_column_document_list * TLDR-405 fix tests * TLDR-405 review fix * TLDR-448-Fix draw coordinates bug (#330) * Fix draw coordinates bug * Fix draw coordinates conversion * TLDR-451 tutorial new doc type (#331) * docs added * add code testing * some fixes * some fixes * add tabula and some fixes * add python-djvulibre * delete python-djvulibre and add djvulibre-bin * add poppler-utils * add tesseract * some fixes * flake8 stylefix * fix docs after flake8 * update last part of adding_new_doc_type_tutorial * rewrite dedoc_add_new_doc_type_tutorial * minor fixes * minor fixes * minor fixes * some fixes * add more code examples * some fixes --------- Co-authored-by: Nikita Shevtsov <shevtsov@ispras.ru> Co-authored-by: Nasty <bogatenkova.anastasiya@mail.ru> * updated txt layer correctness classifier (#334) Co-authored-by: Alexander Golodkov <golodkov@ispras.ru> * Esl 137 added boxes into table (#333) * ESL-137 added box extraction skeleton into scan table extraction * ESL-138 ESL-137 a lot of table changes - added CellWithMeta - change output table structure, remove CellProperies in output - change logic bbox extraction from image tables after debugging - change output in CSV, HTML, TABBY, PDF, SCAN readers - change all tests with tables - fixed styles * ESL-137 chnaged draw table script * ESL-148 added script of table word boxes drawing * TLDR-471 added angle rotation from PdfImageReader and Tables * ESL-137 fixed unit-tests * ESL-137 fixed after review; removing some unused functions - fixed after review - removing some unused functions * ESL-137 update docs * ESL-137 after review * Updated columns orientation classifier (#335) * updated txt columns orientation classifier * deleted "no_lines" parameter --------- Co-authored-by: Alexander Golodkov <golodkov@ispras.ru> * fix pdf reader (#337) Co-authored-by: Nikita Shevtsov <shevtsov@ispras.ru> * TLDR-472 add flake8-fill-one-line and flake8-multiline-containers and fix lint (#336) * add flake8-fill-one-line and flake8-multiline-containers and fix lint * update precommit hook * TLDR-475 fix table documentation (#338) * TLDR-475 fix table documentation * Small fixes * TLDR-474 remove insert_table parameter (#339) * TLDR-474 remove insert_table parameter * TLDR-474 remove is_inserted attribute * ESL-470 fixed rotation operation of table word boxes (#341) rotates a table image and saving image.shape during rotation. It is important for word bounding box extraction * TLDR-478 docx table refactoring (#342) * TLDR-478 docx table refactoring * Small fixes * TLDR-483 fixed box extraction from cropped cells (#343) * TLDR-473 add dedoc utils (#340) * use dedoc utils BBox class * use AdaptiveBinarizer from dedoc-utils * use SkewCorrector from dedoc-utils * fix style * fix rotated angle error * delete BBox from docs * fix angles * delete print * fix dedocutils * dedocutils set ver. 0.3.5 * fix mistakes and names --------- Co-authored-by: Nikita Shevtsov <shevtsov@ispras.ru> * TLDR-481 html refactoring (#344) * delete unused files * Delete unused files, refactor html * Refactor query parameters * Fix tests * Refactor train dataset api * Fix style * Change python version in tests * Review fixes * TLDR-490 changed uuid1 on uuid4; fixed bug in tabby's table uuid (#345) * TLDR-490 changed uuid1 on uuid4; fixed bug in tabby's table uuid * TLDR-490 fixes after review * Added running API examples instruction (#346) * added linewithmeta comparison operator (#347) Co-authored-by: Alexander Golodkov <golodkov@ispras.ru> * ESL-156 fix pdfminer boxes output (#348) * ESL-156 fix pdfminer boxes output * ESL-156 after review * ESL-159 fixed extract boxes from pdfminer reader (#350) * new version 1.0 (#351) --------- Co-authored-by: Andrey Mikhailov <mikhailov@icc.ru> Co-authored-by: Nikita Shevtsov <61932814+Travvy88@users.noreply.github.com> Co-authored-by: Nikita Shevtsov <shevtsov@ispras.ru> Co-authored-by: Alexander Golodkov <55749660+alexander1999-hub@users.noreply.github.com> Co-authored-by: Alexander Golodkov <golodkov@ispras.ru> Co-authored-by: Oksana Belyaeva <belyaeva@ispras.ru> Co-authored-by: Andrew Perminov <perminov@ispras.ru>
- Loading branch information
8 people
authored
Oct 10, 2023
1 parent
79f4cb5
commit ff26829
Showing
264 changed files
with
5,260 additions
and
5,122 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,7 +16,6 @@ exclude = | |
| resources, | ||
| dedoc/scripts, | ||
| examples, | ||
| docs, | ||
| venv, | ||
| build, | ||
| dedoc.egg-info | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 0.11.2 | ||
| 1.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,103 +1,54 @@ | ||
| from typing import Any, Optional | ||
|
|
||
| from fastapi import Body | ||
| from pydantic import BaseModel | ||
|
|
||
|
|
||
| class QueryParameters(BaseModel): | ||
| document_type: Optional[str] | ||
| structure_type: Optional[str] | ||
| return_format: Optional[str] | ||
|
|
||
| with_attachments: Optional[str] | ||
| need_content_analysis: Optional[str] | ||
| recursion_deep_attachments: Optional[str] | ||
| return_base64: Optional[str] | ||
| attachments_dir: Optional[str] | ||
|
|
||
| insert_table: Optional[str] | ||
| need_pdf_table_analysis: Optional[str] | ||
| table_type: Optional[str] | ||
| orient_analysis_cells: Optional[str] | ||
| orient_cell_angle: Optional[str] | ||
|
|
||
| pdf_with_text_layer: Optional[str] | ||
| language: Optional[str] | ||
| pages: Optional[str] | ||
| is_one_column_document: Optional[str] | ||
| document_orientation: Optional[str] | ||
| need_header_footer_analysis: Optional[str] | ||
| need_binarization: Optional[str] | ||
|
|
||
| delimiter: Optional[str] | ||
| encoding: Optional[str] | ||
| html_fields: Optional[str] | ||
| handle_invisible_table: Optional[str] | ||
|
|
||
| def __init__(self, | ||
| # type of document structure parsing | ||
| document_type: Optional[str] = Body(description="a document type. Default: ''", enum=["", "law", "tz", "diploma"], default=None), # noqa | ||
| structure_type: Optional[str] = Body(description="output structure type (linear or tree). Default: 'tree'", enum=["linear", "tree"], default=None), # noqa | ||
| return_format: Optional[str] = Body(description="an option for returning a response in html form, json, pretty_json or tree. Assume that one should use json in all cases, all other formats are used for debug porpoises only. Default: 'json'", default=None), # noqa | ||
|
|
||
| # attachments handling | ||
| with_attachments: Optional[str] = Body(description="an option to enable the analysis of attached files. Default: 'false'", default=None), # noqa | ||
| need_content_analysis: Optional[str] = Body(description="turn on if you need parse the contents of the document attachments. Default: 'false'", default=None), # noqa | ||
| recursion_deep_attachments: Optional[str] = Body(description="the depth on which nested attachments will be parsed if need_content_analysis=true. Default: '10'", default=None), # noqa | ||
| return_base64: Optional[str] = Body(description="returns images in base64 format. Default: 'false'", default=None), # noqa | ||
| attachments_dir: Optional[str] = Body(description="path to the directory where to save files' attachments", default=None), # noqa | ||
|
|
||
| # tables handling | ||
| insert_table: Optional[str] = Body(description="Insert table into the result tree's content or not. Default: 'false'", default=None), # noqa | ||
| need_pdf_table_analysis: Optional[str] = Body(description="include a table analysis into pdfs. Default: 'true'", default=None), # noqa | ||
| table_type: Optional[str] = Body(description="a pipeline mode for a table recognition. Default: ''", default=None), # noqa | ||
| orient_analysis_cells: Optional[str] = Body(description="a table recognition option enables analysis of rotated cells in table headers. Default: 'false'", default=None), # noqa | ||
| orient_cell_angle: Optional[str] = Body(description="an option to set orientation of cells in table headers. \"270\" - cells are rotated 90 degrees clockwise, \"90\" - cells are rotated 90 degrees counterclockwise (or 270 clockwise)", default=None), # noqa | ||
|
|
||
| # pdf handling | ||
| pdf_with_text_layer: Optional[str] = Body(description="an option to extract text from a text layer to PDF or using OCR methods for image-documents. Default: 'auto_tabby'", enum=["true", "false", "auto", "auto_tabby", "tabby"], default=None), # noqa | ||
| language: Optional[str] = Body(description="a recognition language. Default: 'rus+eng'", enum=["rus+eng", "rus", "eng"], default=None), # noqa | ||
| pages: Optional[str] = Body(description="an option to limit page numbers in pdf, archives with images. left:right, read pages from left to right. Default: ':'", default=None), # noqa | ||
| is_one_column_document: Optional[str] = Body(description="an option to set one or multiple column document. \"auto\" - system predict number of columns in document pages, \"true\" - is one column documents, \"false\" - is multiple column documents. Default: 'auto'", default=None), # noqa | ||
| document_orientation: Optional[str] = Body(description="an option to set vertical orientation of the document without using an orientation classifier \"auto\" - system predict angle (0, 90, 180, 270) and rotate document, \"no_change\" - do not predict orientation. Default: 'auto'", enum=["auto", "no_change"], default=None), # noqa | ||
| need_header_footer_analysis: Optional[str] = Body(description="include header-footer analysis into pdf with text layer. Default: 'false'", default=None), # noqa | ||
| need_binarization: Optional[str] = Body(description="include an adaptive binarization into pdf without a text layer. Default: 'false'", default=None), # noqa | ||
|
|
||
| # other formats handling | ||
| delimiter: Optional[str] = Body(description="a column separator for csv-files", default=None), # noqa | ||
| encoding: Optional[str] = Body(description="a document encoding", default=None), # noqa | ||
| html_fields: Optional[str] = Body(description="a list of fields for JSON documents to be parsed as HTML documents. It is written as a json string of a list, where each list item is a list of keys to get the field. Default: ''", default=None), # noqa | ||
| handle_invisible_table: Optional[str] = Body(description="handle table without visible borders as tables in html. Default: 'false'", default=None), # noqa | ||
|
|
||
|
|
||
| **data: Any) -> None: # noqa | ||
|
|
||
| super().__init__(**data) | ||
| self.document_type: str = document_type or "" | ||
| self.structure_type: str = structure_type or "tree" | ||
| self.return_format: str = return_format or "json" | ||
|
|
||
| self.with_attachments: str = with_attachments or "false" | ||
| self.need_content_analysis: str = need_content_analysis or "false" | ||
| self.recursion_deep_attachments: str = recursion_deep_attachments or "10" | ||
| self.return_base64: str = return_base64 or "false" | ||
| self.attachments_dir: str = attachments_dir | ||
|
|
||
| self.insert_table: str = insert_table or "false" | ||
| self.need_pdf_table_analysis: str = need_pdf_table_analysis or "true" | ||
| self.table_type: str = table_type or "" | ||
| self.orient_analysis_cells: str = orient_analysis_cells or "false" | ||
| self.orient_cell_angle: str = orient_cell_angle or "90" | ||
|
|
||
| self.pdf_with_text_layer: str = pdf_with_text_layer or "auto_tabby" | ||
| self.language: str = language or "rus+eng" | ||
| self.pages: str = pages or ":" | ||
| self.is_one_column_document: str = is_one_column_document or "auto" | ||
| self.document_orientation: str = document_orientation or "auto" | ||
| self.need_header_footer_analysis: str = need_header_footer_analysis or "false" | ||
| self.need_binarization: str = need_binarization or "false" | ||
|
|
||
| self.delimiter: str = delimiter | ||
| self.encoding: str = encoding | ||
| self.html_fields: str = html_fields or "" | ||
| self.handle_invisible_table: str = handle_invisible_table or "false" | ||
| from dataclasses import asdict, dataclass | ||
| from typing import Optional | ||
|
|
||
| from fastapi import Form | ||
|
|
||
|
|
||
| @dataclass | ||
| class QueryParameters: | ||
| # type of document structure parsing | ||
| document_type: str = Form("", enum=["", "law", "tz", "diploma"], description="Document domain") | ||
| structure_type: str = Form("tree", enum=["linear", "tree"], description="Output structure type") | ||
| return_format: str = Form("json", enum=["json", "html", "plain_text", "tree", "collapsed_tree", "ujson", "pretty_json"], | ||
| description="Response representation, most types (except json) are used for debug purposes only") | ||
|
|
||
| # attachments handling | ||
| with_attachments: str = Form("false", enum=["true", "false"], description="Enable attached files extraction") | ||
| need_content_analysis: str = Form("false", enum=["true", "false"], description="Enable parsing contents of the attached files") | ||
| recursion_deep_attachments: str = Form("10", description="Depth on which nested attachments will be parsed if need_content_analysis=true") | ||
| return_base64: str = Form("false", enum=["true", "false"], description="Save attached images to the document metadata in base64 format") | ||
| attachments_dir: Optional[str] = Form(None, description="Path to the directory where to save files' attachments") | ||
|
|
||
| # tables handling | ||
| need_pdf_table_analysis: str = Form("true", enum=["true", "false"], description="Enable table recognition for pdf") | ||
| table_type: str = Form("", description="Pipeline mode for table recognition") | ||
| orient_analysis_cells: str = Form("false", enum=["true", "false"], description="Enable analysis of rotated cells in table headers") | ||
| orient_cell_angle: str = Form("90", enum=["90", "270"], | ||
| description='Set cells orientation in table headers, "90" means 90 degrees counterclockwise cells rotation') | ||
|
|
||
| # pdf handling | ||
| pdf_with_text_layer: str = Form("auto_tabby", enum=["true", "false", "auto", "auto_tabby", "tabby"], | ||
| description="Extract text from a text layer of PDF or using OCR methods for image-like documents") | ||
| language: str = Form("rus+eng", enum=["rus+eng", "rus", "eng"], description="Recognition language") | ||
| pages: str = Form(":", description='Page numbers range for reading PDF or images, "left:right" means read pages from left to right') | ||
| is_one_column_document: str = Form("auto", enum=["auto", "true", "false"], | ||
| description='One or multiple column document, "auto" - predict number of page columns automatically') | ||
| document_orientation: str = Form("auto", enum=["auto", "no_change"], | ||
| description='Orientation of the document pages, "auto" - predict orientation (0, 90, 180, 270 degrees), ' | ||
| '"no_change" - set vertical orientation of the document without using an orientation classifier') | ||
| need_header_footer_analysis: str = Form("false", enum=["true", "false"], description="Exclude headers and footers from PDF parsing result") | ||
| need_binarization: str = Form("false", enum=["true", "false"], description="Binarize document pages (for images or PDF without a textual layer)") | ||
|
|
||
| # other formats handling | ||
| delimiter: Optional[str] = Form(None, description="Column separator for CSV files") | ||
| encoding: Optional[str] = Form(None, description="Document encoding") | ||
| html_fields: str = Form("", description="List of fields for JSON documents to be parsed as HTML documents") | ||
| handle_invisible_table: str = Form("false", enum=["true", "false"], description="Handle tables without visible borders as tables in HTML") | ||
|
|
||
| def to_dict(self) -> dict: | ||
| parameters = {} | ||
|
|
||
| for parameter_name, parameter_value in asdict(self).items(): | ||
| parameters[parameter_name] = getattr(parameter_value, "default", parameter_value) | ||
|
|
||
| return parameters |
Oops, something went wrong.