A web API for SAM implemented with FastAPI.
This is a part of the following paper. Please cite it when you use this project. You will also cite the original SAM paper and the MobileSAM paper.
- Sugawara, K. Training deep learning models for cell image segmentation with sparse annotations. bioRxiv 2023. doi:10.1101/2023.06.13.544786
Create a conda environment.
conda create -n samapi -y python=3.10
conda activate samapiIf you're using a computer with CUDA-compatible GPU, install cudatoolkit.
conda install -c conda-forge -y cudatoolkit=11.8If you're using a computer with CUDA-compatible GPU on Windows, install torch with GPU-support with the following command.
# Windows with CUDA-compatible GPU only
python -m pip install "torch>=2.3.1,<2.4" torchvision --index-url https://download.pytorch.org/whl/cu118Install samapi and its dependencies.
python -m pip install git+https://github.com/ksugar/samapi.gitIf you are using WSL2, LD_LIBRARY_PATH will need to be updated as follows.
export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATHIf you want to update the samapi server, run the following command in the conda environment.
python -m pip install -U git+https://github.com/ksugar/samapi.gitSince v0.4.0, it is important to launch the server with --workers 2 (or more) to enable cancellation of a download of a weights file.
export PYTORCH_ENABLE_MPS_FALLBACK=1 # Required for running on Apple silicon
uvicorn samapi.main:app --workers 2The command above will launch a server at http://localhost:8000.
INFO: Started server process [21258]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
Note
If you want to access remotely, you may need to launch with --host 0.0.0.0.
uvicorn samapi.main:app --workers 2 --host 0.0.0.0For more information, see uvicorn documentation.
If you try to process a large image and receive the following error, you may need to increase the PIL.Image.MAX_IMAGE_PIXELS value (default: 89478485), or completely disable it (i.e. set the variable to the empty valie).
PIL.Image.DecompressionBombError: Image size (xxxxxxxxx pixels) exceeds limit of 178956970 pixels, could be decompression bomb DOS attack.In Linux and MacOS, you can set the environment variable as follows.
export PIL_MAX_IMAGE_PIXELS="" # or specific value (integer)In Windows, you can set the environment variable as follows.
set PIL_MAX_IMAGE_PIXELS="" # or specific value (integer)class SAMBody(BaseModel):
type: Optional[ModelType] = ModelType.vit_h
bbox: Tuple[int, int, int, int] = Field(example=(0, 0, 0, 0))
b64img: str| key | value |
|---|---|
| type | One of vit_h, vit_l, vit_b or vit_t |
| bbox | Coordinate of a bbox (x1, y1, x2, y2) |
| b64img | Base64-encoded image data |
class SAMAutoMaskBody(BaseModel):
type: Optional[ModelType] = ModelType.vit_h
b64img: str
points_per_side: Optional[int] = 32
points_per_batch: int = 64
pred_iou_thresh: float = 0.88
stability_score_thresh: float = 0.95
stability_score_offset: float = 1.0
box_nms_thresh: float = 0.7
crop_n_layers: int = 0
crop_nms_thresh: float = 0.7
crop_overlap_ratio: float = 512 / 1500
crop_n_points_downscale_factor: int = 1
min_mask_region_area: int = 0
output_type: str = "Single Mask"
include_image_edge: bool = False| key | value |
|---|---|
| type | One of vit_h, vit_l, or vit_b. |
| b64img | Base64-encoded image data. |
| points_per_side | The number of points to be sampled along one side of the image. The total number of points is points_per_side**2. |
| points_per_batch | Sets the number of points run simultaneously by the model. Higher numbers may be faster but use more GPU memory. |
| pred_iou_thresh | A filtering threshold in [0,1], using the model's predicted mask quality. |
| stability_score_thresh | A filtering threshold in [0,1], using the stability of the mask under changes to the cutoff used to binarize the model's mask predictions. |
| stability_score_offset | The amount to shift the cutoff when calculated the stability score. |
| box_nms_thresh | The box IoU cutoff used by non-maximal suppression to filter duplicate masks. |
| crop_n_layers | If >0, mask prediction will be run again on crops of the image. Sets the number of layers to run, where each layer has 2**i_layer number of image crops. |
| crop_nms_thresh | The box IoU cutoff used by non-maximal suppression to filter duplicate masks between different crops. |
| crop_overlap_ratio | Sets the degree to which crops overlap. In the first crop layer, crops will overlap by this fraction of the image length. Later layers with more crops scale down this overlap. |
| crop_n_points_downscale_factor | The number of points-per-side sampled in layer n is scaled down by crop_n_points_downscale_factor**n. |
| min_mask_region_area | If >0, postprocessing will be applied to remove disconnected regions and holes in masks with area smaller than min_mask_region_area. Requires opencv. |
| output_type | If 'Single Mask' is selected, the model will return single masks per prompt. If 'Multi-mask' is selected, the model will return three masks per prompt. 'Multi-mask (all)' keeps all three masks. One of the three masks is kept if the option 'Multi-mask (largest)', 'Multi-mask (smallest)', or 'Multi-mask (best quality)' is selected. |
| include_image_edge | If True, include a crop area at the edge of the original image. |
- point_grids is not supported.
The response body contains a list of GeoJSON Feature objects.
Supporting other formats is a future work.
Returns the version of the SAM API.
The version of the SAM API.
0.4.1
Returns a list of the available weights.
| key | value |
|---|---|
| type (Optional) | One of vit_h, vit_l, vit_b or vit_t. |
A list of the available weights.
| key | value |
|---|---|
| type | One of vit_h, vit_l, vit_b or vit_t. |
| name | The name of the registered SAM weights. |
| URL | The URL of the registered SAM weights. |
Registers SAM weights.
class SAMWeightsBody(BaseModel):
type: ModelType
name: str
url: str| key | value |
|---|---|
| type | One of vit_h, vit_l, vit_b or vit_t. |
| name | The name of the SAM weights to register. |
| URL | The URL to the SAM weights file to register. |
A message indicating whether the registration is successful.
name https://path/to/weights/file.pth is registered.
Cancel the download of the SAM weights.
A message indicating that the cancel signal is sent.
Cancel signal sent
Returns the progress.
The progress.
| key | value |
|---|---|
| message | A message indicating the progress. |
| percent | Integer value in [0, 100]. |
- Support SAM2 models.
- Update dependencies. Related to: ksugar/qupath-extension-sam#16 by @halqadasi and ksugar/samapi#18 by @ArezooGhodsifard.
- cuda-toolkit from
11.7to11.8. - gdown from
^4.7.1to^5.1.0. - torch from
^1.13.1to^2.2.2. - torchvision from
^0.14.1to^0.17.2.
- cuda-toolkit from
- Support for registering SAM weights from URL. ksugar/qupath-extension-sam#8 ksugar/samapi#11 by @constantinpape
-
Support points and multi-mask output by @petebankhead
-
Support SamAutomaticMaskGenerator
-
Support MobileSAM
-
Add opencv-python to dependencies
- Support for MPS backend (MacOS) by @petebankhead
Please cite my paper on bioRxiv.
@article {Sugawara2023.06.13.544786,
author = {Ko Sugawara},
title = {Training deep learning models for cell image segmentation with sparse annotations},
elocation-id = {2023.06.13.544786},
year = {2023},
doi = {10.1101/2023.06.13.544786},
publisher = {Cold Spring Harbor Laboratory},
abstract = {Deep learning is becoming more prominent in cell image analysis. However, collecting the annotated data required to train efficient deep-learning models remains a major obstacle. I demonstrate that functional performance can be achieved even with sparsely annotated data. Furthermore, I show that the selection of sparse cell annotations significantly impacts performance. I modified Cellpose and StarDist to enable training with sparsely annotated data and evaluated them in conjunction with ELEPHANT, a cell tracking algorithm that internally uses U-Net based cell segmentation. These results illustrate that sparse annotation is a generally effective strategy in deep learning-based cell image segmentation. Finally, I demonstrate that with the help of the Segment Anything Model (SAM), it is feasible to build an effective deep learning model of cell image segmentation from scratch just in a few minutes.Competing Interest StatementKS is employed part-time by LPIXEL Inc.},
URL = {https://www.biorxiv.org/content/early/2023/06/13/2023.06.13.544786},
eprint = {https://www.biorxiv.org/content/early/2023/06/13/2023.06.13.544786.full.pdf},
journal = {bioRxiv}
}