Vispy example crashing

[astrofrog]

This is a very nice package, thanks! 👍

The basic examples in the notebooks work for me, but when I try and run the vispy one I get the following warnings:

WARNING: could not connect to display 
WARNING: Could not load the Qt platform plugin "xcb" in "" even though it was found.

then the kernel dies. I am using the latest developer version of both jupyter_rfb and vispy.

[astrofrog]

If I install glfw, I no longer get those warnings or the kernel crashing but instead I get:

/home/tom/python/dev/lib/python3.11/site-packages/glfw/__init__.py:914: GLFWError: (65544) b'X11: The DISPLAY environment variable is missing'
  warnings.warn(message, GLFWError)
/home/tom/python/dev/lib/python3.11/site-packages/glfw/__init__.py:914: GLFWError: (65537) b'The GLFW library is not initialized'
  warnings.warn(message, GLFWError)
WARNING: OpenGL version could not be determined, which might be a sign that OpenGL is not loaded correctly.
WARNING: Error drawing visual <vispy.visuals.mesh.MeshVisual object at 0x7a6a66c62f10>
WARNING: Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/tom/python/dev/lib/python3.11/site-packages/ipykernel_launcher.py", line 17, in <module>
    app.launch_new_instance()
  File "/home/tom/python/dev/lib/python3.11/site-packages/traitlets/config/application.py", line 1043, in launch_instance
    app.start()
  File "/home/tom/python/dev/lib/python3.11/site-packages/ipykernel/kernelapp.py", line 736, in start
    self.io_loop.start()
  File "/home/tom/python/dev/lib/python3.11/site-packages/tornado/platform/asyncio.py", line 195, in start
    self.asyncio_loop.run_forever()
  File "/usr/lib/python3.11/asyncio/base_events.py", line 608, in run_forever
    self._run_once()
  File "/usr/lib/python3.11/asyncio/base_events.py", line 1936, in _run_once
    handle._run()
  File "/usr/lib/python3.11/asyncio/events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
  File "/home/tom/python/dev/lib/python3.11/site-packages/ipykernel/kernelbase.py", line 516, in dispatch_queue
    await self.process_one()
  File "/home/tom/python/dev/lib/python3.11/site-packages/ipykernel/kernelbase.py", line 505, in process_one
    await dispatch(*args)
  File "/home/tom/python/dev/lib/python3.11/site-packages/ipykernel/kernelbase.py", line 412, in dispatch_shell
    await result
  File "/home/tom/python/dev/lib/python3.11/site-packages/ipykernel/kernelbase.py", line 740, in execute_request
    reply_content = await reply_content
  File "/home/tom/python/dev/lib/python3.11/site-packages/ipykernel/ipkernel.py", line 422, in do_execute
    res = shell.run_cell(
  File "/home/tom/python/dev/lib/python3.11/site-packages/ipykernel/zmqshell.py", line 546, in run_cell
    return super().run_cell(*args, **kwargs)
  File "/home/tom/python/dev/lib/python3.11/site-packages/IPython/core/interactiveshell.py", line 3024, in run_cell
    result = self._run_cell(
  File "/home/tom/python/dev/lib/python3.11/site-packages/IPython/core/interactiveshell.py", line 3079, in _run_cell
    result = runner(coro)
  File "/home/tom/python/dev/lib/python3.11/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
    coro.send(None)
  File "/home/tom/python/dev/lib/python3.11/site-packages/IPython/core/interactiveshell.py", line 3284, in run_cell_async
    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
  File "/home/tom/python/dev/lib/python3.11/site-packages/IPython/core/interactiveshell.py", line 3466, in run_ast_nodes
    if await self.run_code(code, result, async_=asy):
  File "/home/tom/python/dev/lib/python3.11/site-packages/IPython/core/interactiveshell.py", line 3526, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "/tmp/ipykernel_8769/1751093664.py", line 27, in <module>
    canvas
  File "/home/tom/python/dev/lib/python3.11/site-packages/IPython/core/displayhook.py", line 268, in __call__
    format_dict, md_dict = self.compute_format_data(result)
  File "/home/tom/python/dev/lib/python3.11/site-packages/IPython/core/displayhook.py", line 157, in compute_format_data
    return self.shell.display_formatter.format(result)
  File "/home/tom/python/dev/lib/python3.11/site-packages/IPython/core/formatters.py", line 149, in format
    format_dict, md_dict = self.mimebundle_formatter(obj, include=include, exclude=exclude)
  File "/home/tom/python/dev/lib/python3.11/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/home/tom/python/dev/lib/python3.11/site-packages/IPython/core/formatters.py", line 223, in catch_format_error
    r = method(self, *args, **kwargs)
  File "/home/tom/python/dev/lib/python3.11/site-packages/IPython/core/formatters.py", line 974, in __call__
    return method(include=include, exclude=exclude)
  File "/home/tom/Code/vispy/vispy/app/canvas.py", line 520, in _repr_mimebundle_
    return f(*args, **kwargs)
  File "/home/tom/python/dev/lib/python3.11/site-packages/jupyter_rfb/widget.py", line 129, in _repr_mimebundle_
    data["text/html"] = self.snapshot()._repr_html_()
  File "/home/tom/python/dev/lib/python3.11/site-packages/jupyter_rfb/widget.py", line 208, in snapshot
    array = self.get_frame()
  File "/home/tom/Code/vispy/vispy/app/backends/_jupyter_rfb.py", line 187, in get_frame
    self._vispy_canvas.events.draw(region=None)
  File "/home/tom/Code/vispy/vispy/util/event.py", line 453, in __call__
    self._invoke_callback(cb, event)
  File "/home/tom/Code/vispy/vispy/util/event.py", line 471, in _invoke_callback
    _handle_exception(self.ignore_callback_errors,
  << caught exception here: >>
  File "/home/tom/Code/vispy/vispy/util/event.py", line 469, in _invoke_callback
    cb(event)
  File "/home/tom/Code/vispy/vispy/scene/canvas.py", line 219, in on_draw
    self._draw_scene()
  File "/home/tom/Code/vispy/vispy/scene/canvas.py", line 278, in _draw_scene
    self.draw_visual(self.scene)
  File "/home/tom/Code/vispy/vispy/scene/canvas.py", line 316, in draw_visual
    node.draw()
  File "/home/tom/Code/vispy/vispy/scene/visuals.py", line 106, in draw
    self._visual_superclass.draw(self)
  File "/home/tom/Code/vispy/vispy/visuals/visual.py", line 668, in draw
    v.draw()
  File "/home/tom/Code/vispy/vispy/visuals/visual.py", line 514, in draw
    self._program.draw(self._vshare.draw_mode,
  File "/home/tom/Code/vispy/vispy/visuals/shaders/program.py", line 102, in draw
    Program.draw(self, *args, **kwargs)
  File "/home/tom/Code/vispy/vispy/gloo/program.py", line 543, in draw
    canvas.context.flush_commands()
  File "/home/tom/Code/vispy/vispy/gloo/context.py", line 172, in flush_commands
    self.glir.flush(self.shared.parser)
  File "/home/tom/Code/vispy/vispy/gloo/glir.py", line 584, in flush
    self._shared.flush(parser)
  File "/home/tom/Code/vispy/vispy/gloo/glir.py", line 506, in flush
    parser.parse(self._filter(self.clear(), parser))
  File "/home/tom/Code/vispy/vispy/gloo/glir.py", line 824, in parse
    self._parse(command)
  File "/home/tom/Code/vispy/vispy/gloo/glir.py", line 798, in _parse
    ob.attach(*args)
  File "/home/tom/Code/vispy/vispy/gloo/glir.py", line 1803, in attach
    self.deactivate()
  File "/home/tom/Code/vispy/vispy/gloo/glir.py", line 1774, in deactivate
    gl.glBindFramebuffer(gl.GL_FRAMEBUFFER, stack[-1])
                                            ~~~~~^^^^
IndexError: list index out of range
ERROR: Invoking <bound method SceneCanvas.on_draw of <SceneCanvas (jupyter_rfb) at 0x7a6aba2c2e90>> for DrawEvent

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
File ~/python/dev/lib/python3.11/site-packages/IPython/core/formatters.py:974, in MimeBundleFormatter.__call__(self, obj, include, exclude)
    971     method = get_real_method(obj, self.print_method)
    973     if method is not None:
--> 974         return method(include=include, exclude=exclude)
    975     return None
    976 else:

File ~/Code/vispy/vispy/app/canvas.py:520, in Canvas._repr_mimebundle_(self, *args, **kwargs)
    518 f = getattr(self._backend, "_repr_mimebundle_", None)
    519 if f is not None:
--> 520     return f(*args, **kwargs)
    521 else:
    522     # Let Jupyter know this failed - otherwise the standard repr is not shown
    523     raise NotImplementedError()

File ~/python/dev/lib/python3.11/site-packages/jupyter_rfb/widget.py:129, in RemoteFrameBuffer._repr_mimebundle_(self, **kwargs)
    127 # Add initial snapshot.
    128 if self._view_name is not None:
--> 129     data["text/html"] = self.snapshot()._repr_html_()
    131 return data

File ~/python/dev/lib/python3.11/site-packages/jupyter_rfb/widget.py:208, in RemoteFrameBuffer.snapshot(self, pixel_ratio, _initial)
    206     self.handle_event(evt)
    207 # Render a frame
--> 208 array = self.get_frame()
    209 # Reset pixel ratio
    210 if new_pixel_ratio and ref_resize_event:

File ~/Code/vispy/vispy/app/backends/_jupyter_rfb.py:192, in CanvasBackend.get_frame(self)
    188     array = self._helper.get_frame()
    190 # Flush commands here to clean up - otherwise we get errors related to
    191 # framebuffers not existin.
--> 192 self._vispy_canvas.context.flush_commands()
    194 return array

File ~/Code/vispy/vispy/gloo/context.py:172, in GLContext.flush_commands(self, event)
    170         fbo = 0
    171     self.shared.parser.parse([('CURRENT', 0, fbo)])
--> 172 self.glir.flush(self.shared.parser)

File ~/Code/vispy/vispy/gloo/glir.py:584, in GlirQueue.flush(self, parser)
    582 def flush(self, parser):
    583     """Flush all current commands to the GLIR interpreter."""
--> 584     self._shared.flush(parser)

File ~/Code/vispy/vispy/gloo/glir.py:506, in _GlirQueueShare.flush(self, parser)
    504     show = self._verbose if isinstance(self._verbose, str) else None
    505     self.show(show)
--> 506 parser.parse(self._filter(self.clear(), parser))

File ~/Code/vispy/vispy/gloo/glir.py:824, in GlirParser.parse(self, commands)
    821     self._objects.pop(id_)
    823 for command in commands:
--> 824     self._parse(command)

File ~/Code/vispy/vispy/gloo/glir.py:800, in GlirParser._parse(self, command)
    798     ob.attach(*args)
    799 elif cmd == 'FRAMEBUFFER':  # FrameBuffer
--> 800     ob.set_framebuffer(*args)
    801 # elif cmd == 'SHADERS':  # Program
    802 #     ob.set_shaders(*args)
    803 elif cmd == 'LINK':  # Program

File ~/Code/vispy/vispy/gloo/glir.py:1760, in GlirFrameBuffer.set_framebuffer(self, yes)
   1758         self._validate()
   1759 else:
-> 1760     self.deactivate()

File ~/Code/vispy/vispy/gloo/glir.py:1774, in GlirFrameBuffer.deactivate(self)
   1772 while self._handle in stack:
   1773     stack.remove(self._handle)
-> 1774 gl.glBindFramebuffer(gl.GL_FRAMEBUFFER, stack[-1])

IndexError: list index out of range

[astrofrog]

Setting:

import os
os.environ['DISPLAY'] = ':0'

at the start of the notebook fixes things. Should the notebook be updated to include this, and also mention that glfw should be installed?

[kushalkolar]

I don't use vispy but my guess is that it's trying to use glfw or Qt before finally using jupyter_rfb? (Almar, like the wgpu.gui selection before the recent changes)

[astrofrog]

The above was on my Linux Desktop. On my laptop, the vispy example notebook works out of the box without any issues.

[almarklein]

I think the best place to solve this is in vispy's offscreen render util (https://github.com/vispy/vispy/blob/main/vispy/app/backends/_offscreen_util.py). It actually needs a real GUI to provide a GL context. It prefers glfw for that, but falls back to whatever gui backends is available otherwise.