Wave app with Jupyter proxy #2068

bijucyborg · 2023-07-10T12:06:13Z

bijucyborg
Jul 10, 2023

I wish to run a wave app (H2O LLM Studio) via a Jupyter hub interface using Jupyter server proxy and did get it to work but only after a few hacks. The benefit is that this provides an environment where users login and get access to select tools.

Jupyter proxy allows access to web apps running inside the environment via a URL which is
http://jupyter-url/proxy//
Since the wave app is listening on port 10101 this url will be http://jupyter-url/proxy/10101/

So I tried several combinations of environment variables H2O_WAVE_ADDRESS, H2O_WAVE_APP_ADDRESS but nothing worked.
But then I changed two things

Changed the index.html in the web server root.

In the JS file wave-static/index-558cdd08.js and
Changed the const e=RC.getAttribute("data-wave-socket-url");return e?e.startsWith("/")?e.substring(1):e:"proxy/10101/_s/"}
to create web socket in a different way.
Changed the app.py serving function to offer a different end point.
@app("/proxy/10101/", on_startup=on_startup)

Question is can all these be achieved by setting parameters? since I do not wish to hack the setup so much.

Thanks in advance.

mturoci · 2023-07-10T12:35:11Z

mturoci
Jul 10, 2023
Maintainer

Hi @bijucyborg.

It seems like H2O_WAVE_BASE_URL is what you are looking for.

the base URL (path prefix) to be used for resolving relative URLs (e.g. /foo/ or /foo/bar/, without the host (default "/")

See configuration docs for more options.

0 replies

bijucyborg · 2023-07-10T13:36:06Z

bijucyborg
Jul 10, 2023
Author

Hi @mturoci thanks for responding so quickly. I set the env variable H2O_WAVE_BASE_URL="/proxy/10101/"

:~/h2o-llmstudio$ make wave
H2O_WAVE_BASE_URL="/proxy/10101/"
H2O_WAVE_MAX_REQUEST_SIZE=25MB
H2O_WAVE_NO_LOG=true
H2O_WAVE_PRIVATE_DIR="/download/@/home/jovyan/h2o-llmstudio/output/download"
python3.10 -m pipenv run wave run app
2023/07/10 13:32:13 #
2023/07/10 13:32:13 # ┌────────────────┐ H2O Wave
2023/07/10 13:32:13 # │ ┐┌┐┐┌─┐┌ ┌┌─┐ │ 0.26.0 20230626122504
2023/07/10 13:32:13 # │ └┘└┘└─└└─┘└── │ © 2021 H2O.ai, Inc.
2023/07/10 13:32:13 # └────────────────┘
2023/07/10 13:32:13 # ┌──────────────────────────────────────────────────┐
2023/07/10 13:32:13 # │ Running at http://localhost:10101/proxy/10101/ │
2023/07/10 13:32:13 # └──────────────────────────────────────────────────┘
2023/07/10 13:32:13 # {"address":"download/","source":"/home/jovyan/h2o-llmstudio/output/download","t":"private_dir"}
2023/07/10 13:32:13 # {"address":":10101","base-url":"/proxy/10101/","t":"listen","web-dir":"/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/www"}
`

http://127.0.0.1:8848/proxy/10101/ Request Method:GETStatus Code:200 OK --> So the index file loads fine

but the JS and CSS does not load

http://127.0.0.1:8848/wave-static/index-558cdd08.js Request Method:GETStatus Code:404 Not Found

http://127.0.0.1:8848/wave-static/index-46c254f1.css Request Method:GETStatus Code:404 Not Found

because they are now available at

http://127.0.0.1:8848/proxy/10101/proxy/10101/wave-static/index-558cdd08.js Request Method:GETStatus Code:200 OK

http://127.0.0.1:8848/proxy/10101/proxy/10101/wave-static/index-46c254f1.css Request Method:GETStatus Code:200 OK

I wish the index.html be served from the default address but the CSS and JS paths be appended with /proxy/10101/, is that possible?

1 reply

mturoci Jul 10, 2023
Maintainer

I wish the index.html be served from the default address but the CSS and JS paths be appended with /proxy/10101/, is that possible?

That's not possible. It's the base URL for all the assets. However, wouldn't setting your @app("/proxy/10101/", on_startup=on_startup) to @app("/", on_startup=on_startup) solve your problem?

http://127.0.0.1:8848/proxy/10101/proxy/10101/wave-static/index-558cdd08.js

This is odd, it should be just http://127.0.0.1:8848/proxy/10101/wave-static/index-558cdd08.js, maybe a typo?

Also, make sure you got rid of all your hacks. I tried the following simple example and it seems to work as expected. You can give it a try and tell me what changes you would need. It loads the default index.html at http://localhost:10101/proxy/10101/ and all the CSS/JS assets properly.

from h2o_wave import Q, ui, main, app


@app('/')
async def serve(q: Q):
    q.page['wizard'] = ui.form_card(box='1 1 2 4', items=[ui.text('hello')])

    await q.page.save()

H2O_WAVE_BASE_URL="/proxy/10101/" wave run app

bijucyborg · 2023-07-11T11:47:41Z

bijucyborg
Jul 11, 2023
Author

I need to do some more testing and will respond in a couple of days. Yesterday I tried what you mentioned and it did not work because of how Jupyter proxy is designed but I have feeling that the browser cache could have played a role as well.

0 replies

bijucyborg · 2023-07-11T20:14:51Z

bijucyborg
Jul 11, 2023
Author

I'm going to give up on this but thanks for such prompt response. Its not per se an issue with wave but how jupyter proxy works.

Just for the record, here is what I was trying to do. I was trying to create a jupyter proxy extension for H2O LLM Studio.
If successful, it would mean users could launch h2o llm studio from within their jupyter lab/hub environment.

bijucyborg/jupyter_h2o-llm-studio_proxy@a8d1b21

I believe this could also serve as an example for someone who wishes to launch a wave dashboard from jupyter hub. There is a jupyter proxy streamlit example.

So I set the parameters.

        os.environ["H2O_WAVE_MAX_REQUEST_SIZE"] = "25MB"
        os.environ["H2O_WAVE_NO_LOG"] = "true"
        os.environ["H2O_WAVE_PRIVATE_DIR"] = "/download/@output/download"
        os.environ["H2O_WAVE_LISTEN"] = f":{port}"
        os.environ["H2O_WAVE_ADDRESS"] = f"http://127.0.0.1:{port}"
        os.environ["H2O_WAVE_BASE_URL"] = "/h2ollmstudio/"
        return ["make", "wave"]

Jupyter server proxy automatically assigns a port to the waved server and opens the URL http://127.0.0.1:8848/h2ollmstudio/

2023/07/11 19:34:52 # 
2023/07/11 19:34:52 # ┌────────────────┐ H2O Wave 
2023/07/11 19:34:52 # │  ┐┌┐┐┌─┐┌ ┌┌─┐ │ 0.26.1 20230703100207
2023/07/11 19:34:52 # │  └┘└┘└─└└─┘└── │ © 2021 H2O.ai, Inc.
2023/07/11 19:34:52 # └────────────────┘
2023/07/11 19:34:52 # ┌───────────────────────────────────────────────────┐
2023/07/11 19:34:52 # │  Running at http://localhost:47353/h2ollmstudio/  │
2023/07/11 19:34:52 # └───────────────────────────────────────────────────┘
2023/07/11 19:34:52 # {"address":"download/","source":"/home/jovyan/h2o-llmstudio/output/download","t":"private_dir"}
2023/07/11 19:34:52 # {"address":":47353","base-url":"/h2ollmstudio/","t":"listen","web-dir":"/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/www"}
[W 2023-07-11 19:34:52.610 ServerApp] 404 GET /h2ollmstudio/ (172.17.0.1) 1286.82ms referer=http://127.0.0.1:8848/lab
INFO:     Will watch for changes in these directories: ['/home/jovyan/h2o-llmstudio']

So in this scenario, it does not find the index.html

Request URL:
http://127.0.0.1:8848/h2ollmstudio/
Request Method:
GET
Status Code:
404 Not Found
Remote Address:
127.0.0.1:8848

The index can be resolved at http://127.0.0.1:8848/h2ollmstudio/h2ollmstudio/

And if I don't set the Base URL then it does not find the JS and CSS.

Thanks again for creating wave, keep up the good work.

7 replies

bijucyborg Jul 12, 2023
Author

Wow that actually helped me progress, I tried that yesterday but did not work. Today I combined the "absolute_url": True, with os.environ["H2O_WAVE_BASE_URL"] = "/h2ollmstudio/" and now we have an issue with socket connection.

Docker logs

2023/07/12 15:14:58 # {"client":"127.0.0.1:50908","err":"websocket: close 1005 (no status)","t":"socket_read"}
2023/07/12 15:14:58 # {"addr":"127.0.0.1:50908","t":"ui_drop"}
[I 2023-07-12 15:14:58.576 ServerApp] Trying to establish websocket connection to ws://localhost:33033/h2ollmstudio/_s/
[I 2023-07-12 15:14:58.578 ServerApp] Websocket connection established to ws://localhost:33033/h2ollmstudio/_s/
2023/07/12 15:14:58 # {"addr":"127.0.0.1:47570","route":"/","t":"ui_add"}

The code is here
https://github.com/bijucyborg/jupyter_h2o-llm-studio_proxy/blob/23468d59da9b7ce328e46e381a69df02ffef2808/jupyter_h2o_llm_studio_proxy/__init__.py

mturoci Jul 13, 2023
Maintainer

Good progress @bijucyborg! Seems like the last obstacle before you get it working.

Can you show me the full wave server logs? Starting with

2023/07/13 09:10:30 # 
2023/07/13 09:10:30 # ┌────────────────┐ H2O Wave 
2023/07/13 09:10:30 # │  ┐┌┐┐┌─┐┌ ┌┌─┐ │ (version) (build)
2023/07/13 09:10:30 # │  └┘└┘└─└└─┘└── │ © 2021 H2O.ai, Inc.
2023/07/13 09:10:30 # └────────────────┘
2023/07/13 09:10:30 # ┌──────────────────────────────────────┐
2023/07/13 09:10:30 # │  Running at http://localhost:10101/  │
2023/07/13 09:10:30 # └──────────────────────────────────────┘
2023/07/13 09:10:30

Also these seem suspicious:

[I 2023-07-12 15:14:58.576 ServerApp] Trying to establish websocket connection to ws://localhost:33033/h2ollmstudio/_s/
[I 2023-07-12 15:14:58.578 ServerApp] Websocket connection established to ws://localhost:33033/h2ollmstudio/_s/

Why 33033 if your app listens on 8848?

bijucyborg Jul 13, 2023
Author

First of all děkuji for your continued support.
The way Jupyter extensions work is that you need to allow Jupyter server proxy to assign ports randomly so thats why you see 33033 etc. 8848 is the port on which jupyter is running.

The port cannot be pre-decided. Thats why in my extension code, I set the environment variables.

        os.environ["H2O_WAVE_LISTEN"] = f":{port}"
        os.environ["H2O_WAVE_ADDRESS"] = f"http://127.0.0.1:{port}/"
        os.environ["H2O_WAVE_BASE_URL"] = "/h2ollmstudio/"

And here are the logs

H2O_WAVE_MAX_REQUEST_SIZE=25MB \
H2O_WAVE_NO_LOG=true \
H2O_WAVE_PRIVATE_DIR="/download/@/home/jovyan/h2o-llmstudio/output/download" \
python3.10 -m pipenv run wave run app
2023/07/13 08:28:46 # 
2023/07/13 08:28:46 # ┌────────────────┐ H2O Wave 
2023/07/13 08:28:46 # │  ┐┌┐┐┌─┐┌ ┌┌─┐ │ 0.26.1 20230703100207
2023/07/13 08:28:46 # │  └┘└┘└─└└─┘└── │ © 2021 H2O.ai, Inc.
2023/07/13 08:28:46 # └────────────────┘
2023/07/13 08:28:46 # ┌───────────────────────────────────────────────────┐
2023/07/13 08:28:46 # │  Running at http://localhost:54693/h2ollmstudio/  │
2023/07/13 08:28:46 # └───────────────────────────────────────────────────┘
2023/07/13 08:28:46 # {"address":"download/","source":"/home/jovyan/h2o-llmstudio/output/download","t":"private_dir"}
2023/07/13 08:28:46 # {"address":":54693","base-url":"/h2ollmstudio/","t":"listen","web-dir":"/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/www"}
[I 2023-07-13 08:28:46.850 ServerApp] Trying to establish websocket connection to ws://localhost:54693/h2ollmstudio/_s/
[I 2023-07-13 08:28:46.852 ServerApp] Websocket connection established to ws://localhost:54693/h2ollmstudio/_s/
2023/07/13 08:28:46 # {"addr":"127.0.0.1:52124","route":"/","t":"ui_add"}
INFO:     Will watch for changes in these directories: ['/home/jovyan/h2o-llmstudio']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [154] using StatReload

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/h2ollmstudio')}
  warn(msg)
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/download/@/home/jovyan/h2o-llmstudio/output/download')}
  warn(msg)
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/conda/bin/jupyter lab')}
  warn(msg)
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('54693'), PosixPath('http'), PosixPath('//127.0.0.1')}
  warn(msg)
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('http'), PosixPath('//127.0.0.1'), PosixPath('8000')}
  warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
ERROR: /home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/bin/python: undefined symbol: cudaRuntimeGetVersion
CUDA SETUP: libcudart.so path is None
CUDA SETUP: Is seems that your cuda installation is not in your path. See https://github.com/TimDettmers/bitsandbytes/issues/85 for more information.
CUDA SETUP: CUDA version lower than 11 are currently not supported for LLM.int8(). You will be only to use 8-bit optimizers and quantization routines!!
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
  warn(msg)
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 00
CUDA SETUP: Loading binary /home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
INFO:     Started server process [162]
INFO:     Waiting for application startup.
2023-07-13 08:28:50,377 - INFO: STARTING APP
INFO:     Application startup complete.

Please ignore errors about bitsandbytes libraries not found, these can be fixed.

mturoci Jul 13, 2023
Maintainer

The way Jupyter extensions work is that you need to allow Jupyter server proxy to assign ports randomly so thats why you see 33033 etc. 8848 is the port on which jupyter is running.

I see.

The reason why you are seeing blank screen is the fact that no app was registered yet. You would need a line like 2023/07/13 10:40:55 # {"host":"http://127.0.0.1:8000","route":"/","t":"app_add"}. It's odd that you get Application startup complete., but no app is registered. I assume http://localhost:54693 is accessible from within Wave app.

I will look into how to set python logging level to debug and let you know.

wave/py/h2o_wave/h2o_wave/server.py

Line 280 in 01cfeae

async def _register(self):

mturoci Jul 13, 2023
Maintainer

You can try to set the logging to DEBUG so that we can learn more. Add the following to the main app.py file (the one with @on decorator).

import logging
logging.basicConfig(level=logging.DEBUG)

Once done, send the logs again, we may learn more.

bijucyborg · 2023-07-13T13:31:29Z

bijucyborg
Jul 13, 2023
Author

I tried two things simultaneously but no luck with any additional messages.

I set the logging level to debug in app.py

import logging
import os

from app_utils.sections.chat import show_chat_is_running_dialog
from llm_studio.src.utils.logging_utils import initialize_logging

os.environ["MKL_THREADING_LAYER"] = "GNU"

from h2o_wave import Q, app, copy_expando, main, ui  # noqa: F401

from app_utils.handlers import handle
from app_utils.initializers import initialize_app, initialize_client
from app_utils.sections.common import interface

logger = logging.getLogger(__name__)

def initialize_logging():
    logging.basicConfig(level=logging.DEBUG) ### <----- LOGGING LEVEL SET TO DEBUG
    # Additional logging configuration if needed

def on_startup():
    initialize_logging()
    logger.info("STARTING APP")


@app("/", on_startup=on_startup)
async def serve(q: Q):
    """Serving function."""

    # Chat is still being streamed but user clicks on another button.
    # Wait until streaming has been completed,
    # as currently there is no stop streaming functionality implemented.
    if q.client["experiment/display/chat/finished"] is False:
        await show_chat_is_running_dialog(q)
        return

    if not q.app.initialized:
        await initialize_app(q)

    copy_expando(q.args, q.client)

    await initialize_client(q)
    await handle(q)

    if not q.args["experiment/display/chat/chatbot"]:
        await interface(q)

    await q.page.save()

I set the H2O_WAVE_DEBUG to true and H2O_WAVE_NO_LOG to false.

H2O_WAVE_MAX_REQUEST_SIZE=25MB \
H2O_WAVE_NO_LOG=false \
H2O_WAVE_PRIVATE_DIR="/download/@/home/jovyan/h2o-llmstudio/output/download" \
H2O_WAVE_DEBUG=true
python3.10 -m pipenv run wave run app
2023/07/13 13:24:43 # 
2023/07/13 13:24:43 # ┌────────────────┐ H2O Wave 
2023/07/13 13:24:43 # │  ┐┌┐┐┌─┐┌ ┌┌─┐ │ 0.26.1 20230703100207
2023/07/13 13:24:43 # │  └┘└┘└─└└─┘└── │ © 2021 H2O.ai, Inc.
2023/07/13 13:24:43 # └────────────────┘
2023/07/13 13:24:43 # ┌───────────────────────────────────────────────────┐
2023/07/13 13:24:43 # │  Running at http://localhost:42295/h2ollmstudio/  │
2023/07/13 13:24:43 # └───────────────────────────────────────────────────┘
2023/07/13 13:24:43 # {"address":"download/","source":"output/download","t":"private_dir"}
2023/07/13 13:24:43 # {"address":":42295","base-url":"/h2ollmstudio/","t":"listen","web-dir":"/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/www"}
[I 2023-07-13 13:24:44.334 ServerApp] Trying to establish websocket connection to ws://localhost:42295/h2ollmstudio/_s/
[I 2023-07-13 13:24:44.336 ServerApp] Websocket connection established to ws://localhost:42295/h2ollmstudio/_s/
2023/07/13 13:24:44 # {"addr":"127.0.0.1:44484","route":"/","t":"ui_add"}
INFO:     Will watch for changes in these directories: ['/home/jovyan/h2o-llmstudio']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [147] using StatReload

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/conda/bin/jupyter lab')}
  warn(msg)
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('http'), PosixPath('//127.0.0.1'), PosixPath('42295')}
  warn(msg)
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/h2ollmstudio')}
  warn(msg)
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/download/@output/download')}
  warn(msg)
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('8000'), PosixPath('http'), PosixPath('//127.0.0.1')}
  warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
ERROR: /home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/bin/python: undefined symbol: cudaRuntimeGetVersion
CUDA SETUP: libcudart.so path is None
CUDA SETUP: Is seems that your cuda installation is not in your path. See https://github.com/TimDettmers/bitsandbytes/issues/85 for more information.
CUDA SETUP: CUDA version lower than 11 are currently not supported for LLM.int8(). You will be only to use 8-bit optimizers and quantization routines!!
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
  warn(msg)
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 00
CUDA SETUP: Loading binary /home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
INFO:     Started server process [162]
INFO:     Waiting for application startup.
INFO:app:STARTING APP
INFO:     Application startup complete.

Attached is the HAR file in case that helps.

127.0.0.1.har.zip

6 replies

mturoci Jul 13, 2023
Maintainer

I am probably too tired for today lol. It needs to be configured at the file scope, rather than within on_startup because we are interested in the startup logs. So try:

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.DEBUG) ### <----- LOGGING LEVEL SET TO DEBUG

def on_startup():
    logger.info("STARTING APP")


@app("/", on_startup=on_startup)
async def serve(q: Q):

bijucyborg Jul 13, 2023
Author

Hurray....! that helped fix the issue.

The issue was that I had put a / at the end of the environment variable H2O_WAVE_ADDRESS

os.environ["H2O_WAVE_ADDRESS"] = f"http://127.0.0.1:{port}/" and as you can see in the debug logs below, this was causing the problem.

So now the env variable is set to
os.environ["H2O_WAVE_ADDRESS"] = f"http://127.0.0.1:{port}" <== Without the slash

and voila we can launch H2O_LLM_Studio from Jupyter.

I will make my git repo prettier, get some of the other issues sorted out and create a post on show and tell.

When I post on Linkedin, I will make sure I tag and credit you @mturoci
Thanks so so much.

H2O_WAVE_MAX_REQUEST_SIZE=25MB \
H2O_WAVE_NO_LOG=false \
H2O_WAVE_PRIVATE_DIR="/download/@/home/jovyan/h2o-llmstudio/output/download" \
H2O_WAVE_DEBUG=true
python3.10 -m pipenv run wave run app
2023/07/13 14:42:27 # 
2023/07/13 14:42:27 # ┌────────────────┐ H2O Wave 
2023/07/13 14:42:27 # │  ┐┌┐┐┌─┐┌ ┌┌─┐ │ 0.26.1 20230703100207
2023/07/13 14:42:27 # │  └┘└┘└─└└─┘└── │ © 2021 H2O.ai, Inc.
2023/07/13 14:42:27 # └────────────────┘
2023/07/13 14:42:27 # ┌───────────────────────────────────────────────────┐
2023/07/13 14:42:27 # │  Running at http://localhost:49787/h2ollmstudio/  │
2023/07/13 14:42:27 # └───────────────────────────────────────────────────┘
2023/07/13 14:42:27 # {"address":"download/","source":"output/download","t":"private_dir"}
2023/07/13 14:42:27 # {"address":":49787","base-url":"/h2ollmstudio/","t":"listen","web-dir":"/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/www"}
[I 2023-07-13 14:42:28.681 ServerApp] Trying to establish websocket connection to ws://localhost:49787/h2ollmstudio/_s/
[I 2023-07-13 14:42:28.683 ServerApp] Websocket connection established to ws://localhost:49787/h2ollmstudio/_s/
INFO:     Will watch for changes in these directories: ['/home/jovyan/h2o-llmstudio']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [162] using StatReload
2023/07/13 14:42:28 # {"addr":"127.0.0.1:39574","route":"/","t":"ui_add"}

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/conda/bin/jupyter lab')}
  warn(msg)
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('http'), PosixPath('49787'), PosixPath('//127.0.0.1')}
  warn(msg)
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/h2ollmstudio')}
  warn(msg)
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/download/@output/download')}
  warn(msg)
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('8000'), PosixPath('http'), PosixPath('//127.0.0.1')}
  warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
ERROR: /home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/bin/python: undefined symbol: cudaRuntimeGetVersion
CUDA SETUP: libcudart.so path is None
CUDA SETUP: Is seems that your cuda installation is not in your path. See https://github.com/TimDettmers/bitsandbytes/issues/85 for more information.
CUDA SETUP: CUDA version lower than 11 are currently not supported for LLM.int8(). You will be only to use 8-bit optimizers and quantization routines!!
/home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
  warn(msg)
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 00
CUDA SETUP: Loading binary /home/jovyan/.local/share/virtualenvs/h2o-llmstudio-iVwEjDid/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
DEBUG:httpx:load_ssl_context verify=False cert=None trust_env=True http2=False
DEBUG:httpx:load_ssl_context verify=False cert=None trust_env=True http2=False
INFO:h2o_wave.server:Server Mode: None
INFO:h2o_wave.server:Server Route: /
INFO:h2o_wave.server:App Address: http://127.0.0.1:8000
INFO:h2o_wave.server:Hub Address: http://127.0.0.1:49787//h2ollmstudio/
DEBUG:h2o_wave.server:Hub Access Key ID: access_key_id
DEBUG:h2o_wave.server:Hub Access Key Secret: access_key_secret
INFO:     Started server process [170]
INFO:     Waiting for application startup.
DEBUG:h2o_wave.server:Registering app at http://127.0.0.1:8000 ...
DEBUG:httpcore.connection:connect_tcp.started host='127.0.0.1' port=49787 local_address=None timeout=5.0 socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x7f929af8ffd0>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'POST']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'POST']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'POST']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 301, b'Moved Permanently', [(b'Location', b'/h2ollmstudio/'), (b'Date', b'Thu, 13 Jul 2023 14:42:32 GMT'), (b'Content-Length', b'0')])
INFO:httpx:HTTP Request: POST http://127.0.0.1:49787//h2ollmstudio/ "HTTP/1.1 301 Moved Permanently"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'POST']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:h2o_wave.server:Register: success!
INFO:app:STARTING APP
2023-07-13 14:42:32,035 - INFO: STARTING APP
INFO:     Application startup complete.

mturoci Jul 14, 2023
Maintainer

Perfect! :) Thanks for taking the time to integrate this.

mtanco Jul 14, 2023
Collaborator

@bijucyborg Amazing work, please do share a link to the post here when it's up, would love to see it and share with others!

pascal-pfeiffer Jul 15, 2023

Amazing, @bijucyborg ! That's a huge improvement over using reverse proxies.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wave app with Jupyter proxy #2068

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 14 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Wave app with Jupyter proxy #2068

bijucyborg Jul 10, 2023

Replies: 5 comments · 14 replies

mturoci Jul 10, 2023 Maintainer

bijucyborg Jul 10, 2023 Author

mturoci Jul 10, 2023 Maintainer

bijucyborg Jul 11, 2023 Author

bijucyborg Jul 11, 2023 Author

bijucyborg Jul 12, 2023 Author

mturoci Jul 13, 2023 Maintainer

bijucyborg Jul 13, 2023 Author

mturoci Jul 13, 2023 Maintainer

mturoci Jul 13, 2023 Maintainer

bijucyborg Jul 13, 2023 Author

mturoci Jul 13, 2023 Maintainer

bijucyborg Jul 13, 2023 Author

mturoci Jul 14, 2023 Maintainer

mtanco Jul 14, 2023 Collaborator

pascal-pfeiffer Jul 15, 2023

bijucyborg
Jul 10, 2023

Replies: 5 comments 14 replies

mturoci
Jul 10, 2023
Maintainer

bijucyborg
Jul 10, 2023
Author

mturoci Jul 10, 2023
Maintainer

bijucyborg
Jul 11, 2023
Author

bijucyborg
Jul 11, 2023
Author

bijucyborg Jul 12, 2023
Author

mturoci Jul 13, 2023
Maintainer

bijucyborg Jul 13, 2023
Author

mturoci Jul 13, 2023
Maintainer

mturoci Jul 13, 2023
Maintainer

bijucyborg
Jul 13, 2023
Author

mturoci Jul 13, 2023
Maintainer

bijucyborg Jul 13, 2023
Author

mturoci Jul 14, 2023
Maintainer

mtanco Jul 14, 2023
Collaborator