update setup and requirements (#241)

* update requirements

* update setup

* add sentencepiece back in

* black formatter

* change 'dev' to 'eval', add scipy back in

* move scipy

* separate out requirements, update environment.yml

* fix pycocotools

* update README

* moved tqdm to eval, wandb only in trainnig

* update version

* black format

---------

Co-authored-by: Alexander Loftus <alex@creyonbio.com>

README.md

@@ -36,6 +36,20 @@ or to create a conda environment for running OpenFlamingo, run
 conda env create -f environment.yml
 ```
 
+To install training or eval dependencies, run one of the first two commands. To install everything, run the third command.
+```
+pip install open-flamingo[training]
+pip install open-flamingo[eval]
+pip install open-flamingo[all]
+```
+
+There are three `requirements.txt` files: 
+- `requirements.txt` 
+- `requirements-training.txt`
+- `requirements-eval.txt`
+
+Depending on your use case, you can install any of these with `pip install -r <requirements-file.txt>`. The base file contains only the dependencies needed for running the model.
+
 # Approach
 OpenFlamingo is a multimodal language model that can be used for a variety of tasks. It is trained on a large multimodal dataset (e.g. Multimodal C4) and can be used to generate text conditioned on interleaved images/text. For example, OpenFlamingo can be used to generate a caption for an image, or to generate a question given an image and a text passage. The benefit of this approach is that we are able to rapidly adapt to new tasks using in-context learning.
 

environment.yml

@@ -7,4 +7,6 @@ dependencies:
  - pip
  - pip:
  - -r requirements.txt
+ - -r requirements-training.txt
+ - -r requirements-eval.txt
  - -e .

requirements-eval.txt

@@ -0,0 +1,13 @@
+scipy
+torchvision
+nltk
+inflection
+pycocoevalcap
+pycocotools
+tqdm
+
+black
+mypy
+pylint
+pytest
+requests

requirements-training.txt

@@ -0,0 +1,5 @@
+torchvision
+braceexpand
+webdataset
+tqdm
+wandb

requirements.txt

@@ -2,17 +2,6 @@ einops
 einops-exts
 transformers>=4.28.1
 torch==2.0.1
-torchvision
 pillow
-more-itertools
-datasets
-braceexpand
-webdataset
-wandb
-nltk
-scipy
-inflection
-sentencepiece==0.1.98
-pycocoevalcap
-pycocotools
-open_clip_torch>=2.16.0
+open_clip_torch>=2.16.0 
+sentencepiece==0.1.98 

setup.py

@@ -6,47 +6,51 @@
  with Path(Path(__file__).parent, "README.md").open(encoding="utf-8") as file:
  long_description = file.read()
 
- # TODO: This is a hack to get around the fact that we can't read the requirements.txt file, we should fix this.
- # def _read_reqs(relpath):
- # fullpath = os.path.join(Path(__file__).parent, relpath)
- # with open(fullpath) as f:
- # return [
- # s.strip()
- # for s in f.readlines()
- # if (s.strip() and not s.startswith("#"))
- # ]
-
  REQUIREMENTS = [
  "einops",
  "einops-exts",
  "transformers>=4.28.1",
  "torch==2.0.1",
- "torchvision",
  "pillow",
- "more-itertools",
- "datasets",
- "braceexpand",
- "webdataset",
- "wandb",
- "nltk",
+ "open_clip_torch>=2.16.0",
+ "sentencepiece==0.1.98",
+ ]
+
+ EVAL = [
  "scipy",
+ "torchvision",
+ "nltk",
  "inflection",
- "sentencepiece==0.1.98",
- "open_clip_torch>=2.16.0",
+ "pycocoevalcap",
+ "pycocotools",
+ "tqdm",
+ ]
+
+ TRAINING = [
+ "wandb",
+ "torchvision",
+ "braceexpand",
+ "webdataset",
+ "tqdm",
  ]
 
  setup(
  name="open_flamingo",
  packages=find_packages(),
  include_package_data=True,
- version="2.0.0",
+ version="2.0.1",
  license="MIT",
  description="An open-source framework for training large multimodal models",
  long_description=long_description,
  long_description_content_type="text/markdown",
  data_files=[(".", ["README.md"])],
  keywords=["machine learning"],
  install_requires=REQUIREMENTS,
+ extras_require={
+ "eval": EVAL,
+ "training": TRAINING,
+ "all": list(set(REQUIREMENTS + EVAL + TRAINING)),
+ },
  classifiers=[
  "Development Status :: 4 - Beta",
  "Intended Audience :: Developers",