Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Requesting for environment.yaml or requirements.txt? #8

Open
TikaToka opened this issue Aug 17, 2023 · 0 comments
Open

Requesting for environment.yaml or requirements.txt? #8

TikaToka opened this issue Aug 17, 2023 · 0 comments

Comments

@TikaToka
Copy link

TikaToka commented Aug 17, 2023 •
edited
Loading

Hello,
first of all, thank you for sharing your work!

I am trying to reproduce your result, but i think i am suffering from enviroment issues shown as below (Looks like torchtext problem)

/home/jckim/mambaforge/envs/tan/lib/python3.11/site-packages/torch/distributed/launch.py:181: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use-env is set by default in torchrun.
If your script expects `--local-rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  warnings.warn(
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
Traceback (most recent call last):
  File "/home/jckim/MMN/train_net.py", line 10, in <module>
    from mmn.data import make_data_loader
  File "/home/jckim/MMN/mmn/data/__init__.py", line 5, in <module>
    from . import datasets as D
  File "/home/jckim/MMN/mmn/data/datasets/__init__.py", line 2, in <module>
    from .tacos import TACoSDataset
  File "/home/jckim/MMN/mmn/data/datasets/tacos.py", line 4, in <module>
    from .utils import moment_to_iou2d, bert_embedding, get_vid_feat
  File "/home/jckim/MMN/mmn/data/datasets/utils.py", line 7, in <module>
    import torchtext
  File "/home/jckim/mambaforge/envs/tan/lib/python3.11/site-packages/torchtext/__init__.py", line 6, in <module>
    from torchtext import _extension  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jckim/mambaforge/envs/tan/lib/python3.11/site-packages/torchtext/_extension.py", line 7, in <module>
    _LIB_DIR = Path(os.environ["SP_DIR"]) / "torch" / "lib"
                    ~~~~~~~~~~^^^^^^^^^^
  File "<frozen os>", line 679, in __getitem__
KeyError: 'SP_DIR'
Traceback (most recent call last):
  File "/home/jckim/MMN/train_net.py", line 10, in <module>
    from mmn.data import make_data_loader
  File "/home/jckim/MMN/mmn/data/__init__.py", line 5, in <module>
    from . import datasets as D
  File "/home/jckim/MMN/mmn/data/datasets/__init__.py", line 2, in <module>
    from .tacos import TACoSDataset
  File "/home/jckim/MMN/mmn/data/datasets/tacos.py", line 4, in <module>
    from .utils import moment_to_iou2d, bert_embedding, get_vid_feat
  File "/home/jckim/MMN/mmn/data/datasets/utils.py", line 7, in <module>
    import torchtext
  File "/home/jckim/mambaforge/envs/tan/lib/python3.11/site-packages/torchtext/__init__.py", line 6, in <module>
    from torchtext import _extension  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jckim/mambaforge/envs/tan/lib/python3.11/site-packages/torchtext/_extension.py", line 7, in <module>
    _LIB_DIR = Path(os.environ["SP_DIR"]) / "torch" / "lib"
                    ~~~~~~~~~~^^^^^^^^^^
  File "<frozen os>", line 679, in __getitem__
KeyError: 'SP_DIR'
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2494446) of binary: /home/jckim/mambaforge/envs/tan/bin/python
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/jckim/mambaforge/envs/tan/lib/python3.11/site-packages/torch/distributed/launch.py", line 196, in <module>
    main()
  File "/home/jckim/mambaforge/envs/tan/lib/python3.11/site-packages/torch/distributed/launch.py", line 192, in main
    launch(args)
  File "/home/jckim/mambaforge/envs/tan/lib/python3.11/site-packages/torch/distributed/launch.py", line 177, in launch
    run(args)
  File "/home/jckim/mambaforge/envs/tan/lib/python3.11/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/jckim/mambaforge/envs/tan/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jckim/mambaforge/envs/tan/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
train_net.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2023-08-17_19:55:32
  host      : vtt-server
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 2494447)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-08-17_19:55:32
  host      : vtt-server
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 2494446)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

so, can i have specific environment setting?

I tried to build env by following the repository you referenced as baseline, but it didn't helped.
(Errors like AttributeError: module 'torch' has no attribute '_six' occurs)

my currrent enviroment is this

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
aiohttp                   3.8.5           py311h459d7ec_0    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
annotated-types           0.5.0              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.3              pyhd8ed1ab_0    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
aws-c-auth                0.7.3                h28f7589_1    conda-forge
aws-c-cal                 0.6.1                hc309b26_1    conda-forge
aws-c-common              0.9.0                hd590300_0    conda-forge
aws-c-compression         0.2.17               h4d4d85c_2    conda-forge
aws-c-event-stream        0.3.1                h2e3709c_4    conda-forge
aws-c-http                0.7.11               h00aa349_4    conda-forge
aws-c-io                  0.13.32              h4a1a131_0    conda-forge
aws-c-mqtt                0.9.3                hb447be9_1    conda-forge
aws-c-s3                  0.3.14               hf3aad02_1    conda-forge
aws-c-sdkutils            0.1.12               h4d4d85c_1    conda-forge
aws-checksums             0.1.17               h4d4d85c_1    conda-forge
aws-crt-cpp               0.21.0               hb942446_5    conda-forge
aws-sdk-cpp               1.10.57             h85b1a90_19    conda-forge
brotli-python             1.0.9           py311ha362b79_9    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
catalogue                 2.0.9           py311h38be061_0    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
click                     8.1.6           unix_pyh707e725_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
confection                0.1.1           py311h781c19f_0    conda-forge
cuda-version              11.8                 h70ddcb2_2    conda-forge
cudatoolkit               11.8.0              h4ba93d1_12    conda-forge
cudnn                     8.8.0.121            h0800d71_1    conda-forge
cymem                     2.0.7           py311ha362b79_1    conda-forge
cython-blis               0.7.10          py311h1f0f07a_1    conda-forge
dataclasses               0.8                pyhc8e2a94_3    conda-forge
datasets                  2.14.4             pyhd8ed1ab_0    conda-forge
dill                      0.3.7              pyhd8ed1ab_0    conda-forge
double-conversion         3.3.0                h59595ed_0    conda-forge
filelock                  3.12.2             pyhd8ed1ab_0    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
frozenlist                1.4.0           py311h459d7ec_0    conda-forge
fsspec                    2023.6.0           pyh1a96a4e_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
glog                      0.6.0                h6f12383_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gmpy2                     2.1.2           py311h6a5fa03_1    conda-forge
h5py                      3.9.0           nompi_py311he78b9b8_101    conda-forge
hdf5                      1.14.1          nompi_h4f84152_100    conda-forge
huggingface_hub           0.16.4             pyhd8ed1ab_0    conda-forge
icu                       72.1                 hcb278e6_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
importlib_metadata        6.8.0                hd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
langcodes                 3.3.0              pyhd8ed1ab_0    conda-forge
lcms2                     2.15                 haa2dc70_1    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20230125.3      cxx17_h59595ed_0    conda-forge
libaec                    1.0.6                hcb278e6_1    conda-forge
libarrow                  12.0.1           hb87d912_8_cpu    conda-forge
libblas                   3.9.0           17_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_9    conda-forge
libbrotlidec              1.0.9                h166bdaf_9    conda-forge
libbrotlienc              1.0.9                h166bdaf_9    conda-forge
libcblas                  3.9.0           17_linux64_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcurl                   8.2.1                hca28451_0    conda-forge
libdeflate                1.18                 h0b41bf4_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgfortran-ng            13.1.0               h69a702a_0    conda-forge
libgfortran5              13.1.0               h15d22d2_0    conda-forge
libgoogle-cloud           2.12.0               hac9eb74_1    conda-forge
libgrpc                   1.54.3               hb20ce57_0    conda-forge
libhwloc                  2.9.2           nocuda_h7313eea_1008    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libjpeg-turbo             2.1.5.1              h0b41bf4_0    conda-forge
liblapack                 3.9.0           17_linux64_openblas    conda-forge
libmagma                  2.7.1                hc72dce7_3    conda-forge
libmagma_sparse           2.7.1                hc72dce7_4    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libnuma                   2.0.16               h0b41bf4_1    conda-forge
libopenblas               0.3.23          pthreads_h80387f5_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libprotobuf               3.21.12              h3eb15da_0    conda-forge
libsentencepiece          0.1.99               h180e1df_1    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libthrift                 0.18.1               h8fd135c_2    conda-forge
libtiff                   4.5.1                h8b53f26_0    conda-forge
libutf8proc               2.8.0                h166bdaf_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.1                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxml2                   2.11.5               h0d562d8_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
llvm-openmp               16.0.6               h4dfa4b3_0    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
magma                     2.7.1                ha770c72_4    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.3           py311h459d7ec_0    conda-forge
mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
mkl                       2022.2.1         h84fe81f_16997    conda-forge
mpc                       1.3.1                hfe3b2da_0    conda-forge
mpfr                      4.2.0                hb012696_0    conda-forge
mpmath                    1.3.0              pyhd8ed1ab_0    conda-forge
multidict                 6.0.4           py311h2582759_0    conda-forge
multiprocess              0.70.15         py311h459d7ec_0    conda-forge
murmurhash                1.0.9           py311ha362b79_1    conda-forge
nccl                      2.18.3.1             h12f7317_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
networkx                  3.1                pyhd8ed1ab_0    conda-forge
nltk                      3.8.1              pyhd8ed1ab_0    conda-forge
numpy                     1.25.2          py311h64a7726_0    conda-forge
openjpeg                  2.5.0                hfec8fc6_2    conda-forge
openssl                   3.1.2                hd590300_0    conda-forge
orc                       1.9.0                h2f23424_1    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pandas                    2.0.3           py311h320fe9a_1    conda-forge
pathy                     0.10.2             pyhd8ed1ab_0    conda-forge
pillow                    10.0.0          py311h0b84326_0    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
preshed                   3.0.8           py311ha362b79_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pyarrow                   12.0.1          py311h39c9aba_8_cpu    conda-forge
pydantic                  2.1.1              pyhd8ed1ab_0    conda-forge
pydantic-core             2.4.0           py311h46250e7_0    conda-forge
pygments                  2.16.1             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.4          hab00c5b_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python-xxhash             3.3.0           py311h459d7ec_0    conda-forge
python_abi                3.11                    3_cp311    conda-forge
pytorch                   2.0.0           cuda112py311h13fee9e_200    conda-forge
pytz                      2023.3             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0             py311hd4cff14_5    conda-forge
rdma-core                 28.9                 h59595ed_1    conda-forge
re2                       2023.03.02           h8c504da_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
regex                     2023.8.8        py311h459d7ec_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
revtok                    0.0.3.1            pyhd8ed1ab_0    conda-forge
rich                      13.5.1             pyhd8ed1ab_0    conda-forge
rocm-smi                  5.6.0                h59595ed_1    conda-forge
s2n                       1.3.48               h06160fa_0    conda-forge
sacremoses                0.0.53             pyhd8ed1ab_0    conda-forge
safetensors               0.3.2           py311h46250e7_0    conda-forge
setuptools                68.0.0             pyhd8ed1ab_0    conda-forge
shellingham               1.5.2              pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sleef                     3.5.1                h9b69904_2    conda-forge
smart_open                5.2.1              pyhd8ed1ab_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
spacy                     3.6.1           py311h92ebd52_0    conda-forge
spacy-legacy              3.0.12             pyhd8ed1ab_0    conda-forge
spacy-loggers             1.0.4              pyhd8ed1ab_0    conda-forge
srsly                     2.4.7           py311hb755f60_0    conda-forge
sympy                     1.12            pypyh9d50eac_103    conda-forge
tbb                       2021.10.0            h00ab1b0_0    conda-forge
terminaltables            3.1.10             pyhd8ed1ab_0    conda-forge
thinc                     8.1.12          py311h92ebd52_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tokenizers                0.13.3          py311h1b04a43_0    conda-forge
torchtext                 0.15.2          py311hab83ad5_2    conda-forge
torchvision               0.15.2          cuda112py311hc057aef_1    conda-forge
tqdm                      4.66.1             pyhd8ed1ab_0    conda-forge
transformers              4.31.0             pyhd8ed1ab_0    conda-forge
typer                     0.9.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.7.1                hd8ed1ab_0    conda-forge
typing_extensions         4.7.1              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
ucx                       1.14.1               h4a2ce2d_3    conda-forge
urllib3                   2.0.4              pyhd8ed1ab_0    conda-forge
wasabi                    1.1.2           py311h38be061_0    conda-forge
wheel                     0.41.1             pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xxhash                    0.8.2                hd590300_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yacs                      0.1.8              pyhd8ed1ab_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yarl                      1.9.2           py311h459d7ec_0    conda-forge
zipp                      3.16.2             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.2                hfc55251_7    conda-forge

Thank you in advance!
@TikaToka TikaToka changed the title Looking for environment.yaml or requirements.txt? Requesting for environment.yaml or requirements.txt? Aug 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@TikaToka