Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

micropy stubs create fails after verifying integrity #495

Open
romankulev opened this issue May 31, 2023 · 7 comments
Open

micropy stubs create fails after verifying integrity #495

romankulev opened this issue May 31, 2023 · 7 comments
Labels
bug Something isn't working

Comments

@romankulev
Copy link

romankulev commented May 31, 2023

Describe the bug
When I try to create stub from my esp32 i get this error (saved all output in error.txt):
KeyError: 'sysname'
error.txt

Steps to Reproduce

1.Plug esp32 to computer, flash micropython binary
2.run micropy stubs create /dev/ttyUSB0

Logs

attached micropy.log file
micropy.log

Context (Environment)

  • OS: Endeavour OS
  • Micropy Version: 4.2.1
  • Python Version: 3.11.3
@romankulev romankulev added the bug Something isn't working label May 31, 2023
@skipper85
Copy link

Hello @romankulev , did you manage to solve this? I experience the same issue.

@BradenM
Copy link
Owner

BradenM commented Sep 2, 2023

Hi @skipper85, @romankulev

Sorry for the late reply.

Have either of you tried using the rshell backend as an alternative to the upysh backend?

See micropy stubs create --help for details

@skipper85
Copy link

Hi @BradenM ,

Thank you for your response.
I tried today with the rshell backend, but then I face a different error.
i use micropy in a virtual environment.

Python version : 3.10.12
rshell : 0.0.31
micropy : 0.6.2
micropython bin : ESP32_GENERIC_S2-20230426-v1.20.0.bin

micropy stubs create /dev/ttyUSB0 --backend rshell

MicroPy  Connecting to Pyboard @ /dev/ttyUSB0
MicroPy  ✔ Connected!
2023-09-08 10:30:12.881 | DEBUG    | stubber.minify:minify_script:253 - Original length : 26905
2023-09-08 10:30:12.881 | INFO     | stubber.minify:minify_script:254 - Minified length : 11631
2023-09-08 10:30:12.881 | INFO     | stubber.minify:minify_script:255 - Reduced by      : 15274 
2023-09-08 10:30:13.040 | DEBUG    | stubber.minify:cross_compile:328 - mpy-cross compiled to    : mpy_cross_mxm05fa9.mpy
MicroPy  Executing stubber on pyboard...
MicroPy  Failed to execute script: 'utf-8' codec can't decode byte 0x83 in position 4: invalid start byte
MicroPy  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 4: invalid start byte

─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/gerard/Documents/test_esp32s2/env/lib/python3.10/site-packages/micropy/app/stubs.py:179 in │
│ stubs_create                                                                                     │
│                                                                                                  │
│   176 │   dev_path = DevicePath("createstubs.mpy") if compile else DevicePath("createstubs.py"   │
│   177 │   log.info("Executing stubber on pyboard...")                                            │
│   178 │   try:                                                                                   │
│ ❱ 179 │   │   pyb.run_script(create_stubs, DevicePath(dev_path))                                 │
│   180 │   except Exception as e:                                                                 │
│   181 │   │   # TODO: Handle more usage cases                                                    │
│   182 │   │   log.error(f"Failed to execute script: {str(e)}", exception=e)                      │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │        _get_desc = <function stubs_create.<locals>._get_desc at 0x7ffbb65c2a70>              │ │
│ │          backend = <CreateBackend.rshell: 'rshell'>                                          │ │
│ │          compile = True                                                                      │ │
│ │     create_stubs = <_io.BytesIO object at 0x7ffbb51186d0>                                    │ │
│ │              ctx = <click.core.Context object at 0x7ffbb65d38e0>                             │ │
│ │         dev_path = 'createstubs.mpy'                                                         │ │
│ │          exclude = []                                                                        │ │
│ │ exclude_defaults = True                                                                      │ │
│ │              log = <micropy.logger.ServiceLog object at 0x7ffbb65d0040>                      │ │
│ │  message_handler = MessageHandlers(                                                          │ │
│ │                    │   on_message=<function stubs_create.<locals>.<lambda> at                │ │
│ │                    0x7ffbb65c3250>,                                                          │ │
│ │                    )                                                                         │ │
│ │           module = []                                                                        │ │
│ │  module_defaults = True                                                                      │ │
│ │               mp = <micropy.main.MicroPy object at 0x7ffbb6547ee0>                           │ │
│ │             port = '/dev/ttyUSB0'                                                            │ │
│ │              pyb = <micropy.pyd.pydevice.PyDevice object at 0x7ffbb65d3700>                  │ │
│ │          pyb_log = <micropy.logger.ServiceLog object at 0x7ffbb65d3d30>                      │ │
│ │          variant = <CreateStubsVariant.BASE: 'base'>                                         │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /home/gerard/Documents/test_esp32s2/env/lib/python3.10/site-packages/micropy/pyd/pydevice.py:94  │
│ in run_script                                                                                    │
│                                                                                                  │
│   91 │   │   │   )                                                                               │
│   92 │   │   │   else content.read()                                                             │
│   93 │   │   )                                                                                   │
│ ❱ 94 │   │   return self.pydevice.eval_script(_content, target_path, consumer=self.consumer)     │
│   95 │                                                                                           │
│   96 │   def run(self, content: str) -> str | None:                                              │
│   97 │   │   return self.pydevice.eval(content, consumer=self.consumer)                          │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │    _content = b'M\x06\x00\x1f\x83\x19u\x1ccreatestubs.py\x00\x0f\x06pyb\x00\n{}/{}\x00\x0el… │ │
│ │     content = <_io.BytesIO object at 0x7ffbb51186d0>                                         │ │
│ │        self = <micropy.pyd.pydevice.PyDevice object at 0x7ffbb65d3700>                       │ │
│ │ target_path = 'createstubs.mpy'                                                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /home/gerard/Documents/test_esp32s2/env/lib/python3.10/site-packages/micropy/pyd/backend_rshell. │
│ py:225 in eval_script                                                                            │
│                                                                                                  │
│   222 │   ):                                                                                     │
│   223 │   │   _contents: str | bytes = contents                                                  │
│   224 │   │   if isinstance(_contents, bytes):                                                   │
│ ❱ 225 │   │   │   _contents = _contents.decode()                                                 │
│   226 │   │   with self.repl():                                                                  │
│   227 │   │   │   try:                                                                           │
│   228 │   │   │   │   out_bytes = self.eval(_contents, consumer=consumer)                        │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │   _contents = b'M\x06\x00\x1f\x83\x19u\x1ccreatestubs.py\x00\x0f\x06pyb\x00\n{}/{}\x00\x0el… │ │
│ │    consumer = <micropy.pyd.consumers.ConsumerDelegate object at 0x7ffbb65d37c0>              │ │
│ │    contents = b'M\x06\x00\x1f\x83\x19u\x1ccreatestubs.py\x00\x0f\x06pyb\x00\n{}/{}\x00\x0el… │ │
│ │        self = <micropy.pyd.backend_rshell.RShellPyDeviceBackend object at 0x7ffbb65d3b20>    │ │
│ │ target_path = 'createstubs.mpy'                                                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 4: invalid start byte

@VirtualWolf
Copy link

Same issue on both counts for me... without --backend rshell I get a whole ton of reboots (which might be some other issue) before it eventually fails with the KeyError: 'sysname' error: micropy.txt

And with --backend rshell I get the same invalid start byte error as above:

$ micropy stubs create /dev/tty.SLAB_USBtoUART --backend rshell

MicroPy  Connecting to Pyboard @ /dev/tty.SLAB_USBtoUART
MicroPy  ✔ Connected!
2024-04-02 21:46:34.866 | DEBUG    | stubber.minify:minify_script:253 - Original length : 26905
2024-04-02 21:46:34.867 | INFO     | stubber.minify:minify_script:254 - Minified length : 11631
2024-04-02 21:46:34.867 | INFO     | stubber.minify:minify_script:255 - Reduced by      : 15274 
2024-04-02 21:46:35.300 | DEBUG    | stubber.minify:cross_compile:328 - mpy-cross compiled to    : mpy_cross_a5xj9gos.mpy
MicroPy  Executing stubber on pyboard...
MicroPy  Failed to execute script: 'utf-8' codec can't decode byte 0x83 in position 4: invalid start byte
MicroPy  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 4: invalid start byte
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/virtualwolf/Downloads/stubbing/.venv/lib/python3.10/site-packages/micropy/app/stubs.py:17 │
│ 9 in stubs_create                                                                                │
│                                                                                                  │
│   176 │   dev_path = DevicePath("createstubs.mpy") if compile else DevicePath("createstubs.py"   │
│   177 │   log.info("Executing stubber on pyboard...")                                            │
│   178 │   try:                                                                                   │
│ ❱ 179 │   │   pyb.run_script(create_stubs, DevicePath(dev_path))                                 │
│   180 │   except Exception as e:                                                                 │
│   181 │   │   # TODO: Handle more usage cases                                                    │
│   182 │   │   log.error(f"Failed to execute script: {str(e)}", exception=e)                      │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │        _get_desc = <function stubs_create.<locals>._get_desc at 0x1185b7eb0>                 │ │
│ │          backend = <CreateBackend.rshell: 'rshell'>                                          │ │
│ │          compile = True                                                                      │ │
│ │     create_stubs = <_io.BytesIO object at 0x118ac4860>                                       │ │
│ │              ctx = <click.core.Context object at 0x11862b6a0>                                │ │
│ │         dev_path = 'createstubs.mpy'                                                         │ │
│ │          exclude = []                                                                        │ │
│ │ exclude_defaults = True                                                                      │ │
│ │              log = <micropy.logger.ServiceLog object at 0x11847b880>                         │ │
│ │  message_handler = MessageHandlers(                                                          │ │
│ │                    │   on_message=<function stubs_create.<locals>.<lambda> at 0x1185fce50>,  │ │
│ │                    )                                                                         │ │
│ │           module = []                                                                        │ │
│ │  module_defaults = True                                                                      │ │
│ │               mp = <micropy.main.MicroPy object at 0x11847b1f0>                              │ │
│ │             port = '/dev/tty.SLAB_USBtoUART'                                                 │ │
│ │              pyb = <micropy.pyd.pydevice.PyDevice object at 0x11862ac20>                     │ │
│ │          pyb_log = <micropy.logger.ServiceLog object at 0x11862abf0>                         │ │
│ │          variant = <CreateStubsVariant.BASE: 'base'>                                         │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /Users/virtualwolf/Downloads/stubbing/.venv/lib/python3.10/site-packages/micropy/pyd/pydevice.py │
│ :94 in run_script                                                                                │
│                                                                                                  │
│   91 │   │   │   )                                                                               │
│   92 │   │   │   else content.read()                                                             │
│   93 │   │   )                                                                                   │
│ ❱ 94 │   │   return self.pydevice.eval_script(_content, target_path, consumer=self.consumer)     │
│   95 │                                                                                           │
│   96 │   def run(self, content: str) -> str | None:                                              │
│   97 │   │   return self.pydevice.eval(content, consumer=self.consumer)                          │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │    _content = b'M\x06\x00\x1f\x83\x19u\x1ccreatestubs.py\x00\x0f\x06pyb\x00\n{}/{}\x00\x0el… │ │
│ │     content = <_io.BytesIO object at 0x118ac4860>                                            │ │
│ │        self = <micropy.pyd.pydevice.PyDevice object at 0x11862ac20>                          │ │
│ │ target_path = 'createstubs.mpy'                                                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /Users/virtualwolf/Downloads/stubbing/.venv/lib/python3.10/site-packages/micropy/pyd/backend_rsh │
│ ell.py:225 in eval_script                                                                        │
│                                                                                                  │
│   222 │   ):                                                                                     │
│   223 │   │   _contents: str | bytes = contents                                                  │
│   224 │   │   if isinstance(_contents, bytes):                                                   │
│ ❱ 225 │   │   │   _contents = _contents.decode()                                                 │
│   226 │   │   with self.repl():                                                                  │
│   227 │   │   │   try:                                                                           │
│   228 │   │   │   │   out_bytes = self.eval(_contents, consumer=consumer)                        │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │   _contents = b'M\x06\x00\x1f\x83\x19u\x1ccreatestubs.py\x00\x0f\x06pyb\x00\n{}/{}\x00\x0el… │ │
│ │    consumer = <micropy.pyd.consumers.ConsumerDelegate object at 0x11862ac80>                 │ │
│ │    contents = b'M\x06\x00\x1f\x83\x19u\x1ccreatestubs.py\x00\x0f\x06pyb\x00\n{}/{}\x00\x0el… │ │
│ │        self = <micropy.pyd.backend_rshell.RShellPyDeviceBackend object at 0x11862b7c0>       │ │
│ │ target_path = 'createstubs.mpy'                                                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 4: invalid start byte

Environment

  • Board — Adafruit HUZZAH32
  • MicroPython version — 1.22.2, ESP32_GENERIC firmware
  • OS — macOS 14.4.1 on Apple Silicon
  • micropy-cli version — 4.2.2
  • Python version — 3.10.13 in a virtualenv

@kheraud
Copy link

kheraud commented Aug 28, 2024

I have the same issue with a KeyError: 'sysname' . Looking at the stacktrace, it seems that the chip discovery outputs a dict with missing "sysname" key. Is it just required to auto-name the stub ?

~/.asdf/installs/python/3.11.8/lib/python3.11/site-packages/micropy/stubs/stubs.py:3 │
│ 60 in from_stubber                                                                               │
│                                                                                                  │
│   357 │   │   mod_data = json.load(mod_file.open())                                              │
│   358 │   │   dev_fware = mod_data["firmware"]                                                   │
│   359 │   │   fname = dev_fware.get("name", None)                                                │
│ ❱ 360 │   │   out_name = f"{dev_fware['sysname']}"                                               │
│   361 │   │   # TODO: Attempt to Autoresolve Firmware name and add it to info.json               │
│   362 │   │   if fname:                                                                          │
│   363 │   │   │   out_name = f"{out_name}-{fname}"                                               │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │     _path = PosixPath('/tmp/tmpmkabmyj4/stubs')                                              │ │
│ │      dest = PosixPath('/tmp/tmpmkabmyj4')                                                    │ │
│ │ dev_fware = {                                                                                │ │
│ │             │   'version': '1.23.0.',                                                        │ │
│ │             │   'mpy': 'v6.3',                                                               │ │
│ │             │   'port': 'esp32',                                                             │ │
│ │             │   'board': 'Generic_ESP32_module_with_ESP32',                                  │ │
│ │             │   'family': 'micropython',                                                     │ │
│ │             │   'build': '',                                                                 │ │
│ │             │   'arch': 'xtensawin',                                                         │ │
│ │             │   'ver': 'v1.23.0.',                                                           │ │
│ │             │   'cpu': 'ESP32'                                                               │ │
│ │             }                                                                                │ │
│ │     fname = None                                                                             │ │
│ │  mod_data = {                                                                                │ │
│ │             │   'firmware': {                                                                │ │
│ │             │   │   'version': '1.23.0.',                                                    │ │
│ │             │   │   'mpy': 'v6.3',                                                           │ │
│ │             │   │   'port': 'esp32',                                                         │ │
│ │             │   │   'board': 'Generic_ESP32_module_with_ESP32',                              │ │
│ │             │   │   'family': 'micropython',                                                 │ │
│ │             │   │   'build': '',                                                             │ │
│ │             │   │   'arch': 'xtensawin',                                                     │ │
│ │             │   │   'ver': 'v1.23.0.',                                                       │ │
│ │             │   │   'cpu': 'ESP32'                                                           │ │
│ │             │   },                                                                           │ │
│ │             │   'stubber': {'version': 'v1.13.4'},                                           │ │
│ │             │   'stubtype': 'firmware',                                                      │ │
│ │             │   'modules': [                                                                 │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': '_onewire',                                                │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/_onewire.py'  │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': '_thread',                                                 │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/_thread.py'   │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'apa106',                                                  │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/apa106.py'    │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'array',                                                   │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/array.py'     │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'binascii',                                                │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/binascii.py'  │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'bluetooth',                                               │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/bluetooth.py' │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'btree',                                                   │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/btree.py'     │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'cmath',                                                   │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/cmath.py'     │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'collections',                                             │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/collections.… │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'cryptolib',                                               │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/cryptolib.py' │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   ... +63                                                                  │ │
│ │             │   ]                                                                            │ │
│ │             }                                                                                │ │
│ │  mod_file = PosixPath('/tmp/tmpmkabmyj4/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_modu… │ │
│ │      path = PosixPath('/tmp/tmpmkabmyj4/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_modu… │ │
│ │      self = <micropy.stubs.stubs.StubManager object at 0x7cd704f46b50>                       │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'sysname'

@BradenM
Copy link
Owner

BradenM commented Aug 29, 2024

Looks like the format changed in one of the recent stubber updates and somehow was missed in the unit tests.

That piece of code specifically (adapting the module format from stubber) is far from ideal and needs some improving. I will try and look into this soon.

@BradenM
Copy link
Owner

BradenM commented Aug 29, 2024

Not ideal of course, but a temporary workaround would be to modify the resulting "modules.json" file on the device by renaming "firmware.family" -> "firmware.name" and "firmware.board" -> "firmware.sysname" (or "firmware.port", either would work)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants