-
Notifications
You must be signed in to change notification settings - Fork 52
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* move span qa predictor * HUGE reorganization of predictors into different files; fix recipes and tests * increment pyproject; add sys.path debug to github workflow * print sys path after * move base predictor up * making import relative * basepredictors * commit predictors * remove circular imports
- Loading branch information
Showing
34 changed files
with
265 additions
and
90 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,30 @@ | ||
# Predictors | ||
|
||
Some rules about Predictors: | ||
|
||
* Each Predictor is named after the type of Entities it produces. For example, `sentences` come from the `SentencePredictor`. | ||
|
||
* Organization looks like: | ||
|
||
``` | ||
predictors/ | ||
|-- base_predictors/ | ||
|-- hf_predictors.py | ||
|-- lp_predictors.py | ||
|-- api_predictors.py | ||
|-- spacy_predictors.py | ||
|-- sklearn_predictors.py | ||
|-- block_predictors.py | ||
|-- paragraph_predictors.py | ||
|-- sentence_predictors.py | ||
|-- word_predictors.py | ||
|-- token_predictors.py | ||
``` | ||
* Note that `base_predictors/` contains reusable implementations; users never have to import them, but developers may want to reuse these. Users of the library instead import the desired predictor for a given emitted entity type. | ||
|
||
|
||
* We try to name our predictors `[Framework][Model][Dataset][Entity]`. | ||
|
||
## `SpanQAPredictor` (Using GPT3 as a Predictor) | ||
|
||
The `span_qa.predictor.py` file includes an example of using the `decontext` library to use GPT3 as a predictor. The example involves span-based classification: for example, a a user can highlight a span of text in a paper and ask a question about it. (The span is a field, and the question is metadata on the field.) The predictor runs retrieval over specified the specified units and feeds the question, context, and highlighted span to GPT3 to answer the question. See `tests/test_predictors/test_span_qa_predictor.py` for examples of how this predictor is used. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,21 +1,21 @@ | ||
from papermage.predictors.api_predictors.span_qa_predictor import APISpanQAPredictor | ||
from papermage.predictors.hf_predictors.bio_tagger_predictor import HFBIOTaggerPredictor | ||
from papermage.predictors.hf_predictors.vila_predictor import ( | ||
IVILATokenClassificationPredictor, | ||
) | ||
from papermage.predictors.hf_predictors.whitespace_predictor import WhitespacePredictor | ||
from papermage.predictors.lp_predictors.block_predictor import LPBlockPredictor | ||
from papermage.predictors.sklearn_predictors.word_predictor import SVMWordPredictor | ||
from papermage.predictors.spacy_predictors.sentence_predictor import ( | ||
PysbdSentencePredictor, | ||
) | ||
from papermage.predictors.base_predictors.base_predictor import BasePredictor | ||
from papermage.predictors.base_predictors.hf_predictors import HFBIOTaggerPredictor | ||
from papermage.predictors.block_predictors import LPEffDetPubLayNetBlockPredictor | ||
from papermage.predictors.formula_predictors import LPEffDetFormulaPredictor | ||
from papermage.predictors.sentence_predictors import PysbdSentencePredictor | ||
from papermage.predictors.span_qa_predictors import APISpanQAPredictor | ||
from papermage.predictors.token_predictors import HFWhitspaceTokenPredictor | ||
from papermage.predictors.vila_predictors import IVILATokenClassificationPredictor | ||
from papermage.predictors.word_predictors import SVMWordPredictor | ||
|
||
__all__ = [ | ||
"HFBIOTaggerPredictor", | ||
"APISpanQAPredictor", | ||
"LPBlockPredictor", | ||
"IVILATokenClassificationPredictor", | ||
"WhitespacePredictor", | ||
"HFWhitspaceTokenPredictor", | ||
"SVMWordPredictor", | ||
"PysbdSentencePredictor", | ||
"LPEffDetPubLayNetBlockPredictor", | ||
"LPEffDetFormulaPredictor", | ||
"APISpanQAPredictor", | ||
"BasePredictor", | ||
] |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
""" | ||
@kylel | ||
""" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
""" | ||
@kylel | ||
""" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
""" | ||
@kylel | ||
""" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
""" | ||
Predict visual blocks in a page. | ||
@kylel | ||
""" | ||
|
||
from typing import Dict, Optional | ||
|
||
from papermage.predictors.base_predictors.lp_predictors import LPPredictor | ||
|
||
|
||
class LPEffDetPubLayNetBlockPredictor(LPPredictor): | ||
@classmethod | ||
def from_pretrained( | ||
cls, | ||
device: Optional[str] = None, | ||
): | ||
return super().from_pretrained( | ||
config_path="lp://efficientdet/PubLayNet", | ||
device=device, | ||
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
""" | ||
Predict formula blocks in a page. | ||
@kylel | ||
""" | ||
|
||
from typing import Dict, Optional | ||
|
||
from papermage.predictors.base_predictors.lp_predictors import LPPredictor | ||
|
||
|
||
class LPEffDetFormulaPredictor(LPPredictor): | ||
@classmethod | ||
def from_pretrained( | ||
cls, | ||
device: Optional[str] = None, | ||
): | ||
return super().from_pretrained( | ||
config_path="lp://efficientdet/MFD", | ||
device=device, | ||
) |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Empty file.
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.