-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ESP32: uart.write() doesn't send long chunk via CONSOLE_USB_SERIAL_JTAG #3657
Comments
We can't use a uart callback when the usb-serial-jtag or usb-cdc console is chosen as they don't support this, so we have to use base64|hex encoding to upload/download files. Unfortunatly, the current nodemcu firmware does not support sending long chunks via usb-serial-jtag. See nodemcu/nodemcu-firmware#3657 Therefore, some features as 'file download' and 'send chunk of code from editor window' do not work. Also, if there are many files on spiffs, the extension hangs at the stage of reading from ESP.
I found a workaround to the issue.
void platform_uart_send_multi( unsigned id, const char *data, size_t len )
{
size_t i;
// if (id == CONFIG_ESP_CONSOLE_UART_NUM) {
for( i = 0; i < len; i ++ ) {
putchar (data[ i ]);
}
// } else {
// uart_write_bytes(id, data, len);
// }
}
void platform_uart_send( unsigned id, uint8_t data )
{
// if (id == CONFIG_ESP_CONSOLE_UART_NUM)
putchar (data);
// else
// uart_write_bytes(id, (const char *)&data, 1);
}
void platform_uart_flush( unsigned id )
{
// if (id == CONFIG_ESP_CONSOLE_UART_NUM)
fflush (stdout);
// else
// uart_tx_flush(id);
} I don't know why it works, but it does. |
The underlying issue here is that the USB-SERIAL-JTAG isn't an actual UART in the first place. In the name of backwards compatibility I've tried to hide that fact with increasing amounts of special casing in the platform code. As the IDF evolves further towards POSIXness, and the introduction of CDC-ACM consoles as well, this approach has, frankly, gotten rotten. Not only has it produced inconsistent behaviour, odd edge cases (like this), but also clearly a bunch of confusion :( To resolve all these issues I think we have to bite the bullet and get around to moving the console device access into its own module. We've avoided it because it's a backwards compatibility breaking change, but it really looks to be the lesser evil by now. Let me see if I can cook up a PR for evaluation today. |
I have already rewritten the code of both utilities so that they work with the esp jtag console (on idf5.3.1) without using However, I was unable to implement sending binary data from esp to the host, in particular file dowload. |
I've almost finished the necessary refactoring, but I'm out of time for today. I'll try to have something tomorrow. |
I've only just started the testing, but if you want to have a play early the branch is here. Effectively, the console interaction is now via the |
Thanks, I'll test that. |
I tested the new firmware on esp32. Using new function console.mode(0) to disable passing data to Lua VM, I cannot get back to Interactive console. |
I've added an example on how to cleanly transfer potentially binary data using the |
In regard to the original issue, I've also tested writing a 300 byte long string using |
Thank you for such a detailed example! Now I figured it out. Minor fixes in uart.md examples: -- when 4 chars is received. I have been testing the console module on esp32 -s3, -c3, -c6, (jtag) -s2 (cdc) EDIT: -s3 (cdc) The issue is common for both idf5.3.1 and dev-console branches. Very strange behavior. If the connection is made with esp reboot ( |
Ah, thank for, I'll tidy up those uart examples! There is no inherent limit to the string passed to Edit: The IDF/C library claims to have written everything, even when it's truncated. It looks like the loss is within the IDF somewhere. I'll see if I can find a workaround... |
Okay, I have a workaround that seems to do the trick reliably here on my c3. See if it works for your setup too @serg3295 ? |
I am observing long chunks of ~4097 bytes from the ESP. Moreover, the UART console transmits data losslessly, and the JTAG console loses data randomly starting with the second long data packet. |
@serg3295 You're still seeing losses even after my most recent workaround?! It seems reliable now across UART, USB-Serial-JTAG and CDC-ACM here for me. |
I apologize, I didn't pull the last changes. :-( I have tested downloading large files from esp-s3. It's working now! |
I've just pushed Plus I updated both the I think this is ready for a PR? The one other thing that would be lovely is if we could add something to our |
I tested uploading and downloading files using Also I reworked nodemcu-tools VSCode extension and NodeMCU-Tool CLI utility to support the new firmware. |
@serg3295 fabulous! I've raised the PR, let's see what the verdict is. |
I have one more question.
|
It looks like the |
// sdkconfig
|
However, no uart module in firmware
|
What the... 🤯 Let me check here... |
Ugh, my bad 🤦♀️. Fixed now. |
Hooray, it worked! |
I tested simultaneous connection via usb-jtag-console interface and via UART0 connected to FTDI adapter. flowchart LR
host[["make monitor"]]
esp["ESP32-C3"]
ftdi[FTDI adapter]
terminal[[HTerm 0.8.5]]
host-- usb-jtag-console <-->esp<-- UART0 -->ftdi<-->terminal
Data is transferred in both directions from console to UART and vice versa. uart.setup(0, 115200, 8, uart.PARITY_NONE, uart.STOPBITS_1, {tx=21,rx=20})
uart.start(0)
uart.on(0,"data", "\n", function(data) print(data)end)
uart.write(0, "test string\n") |
I'm trying to figure out the CDC console. I am unable to send a string larger than 89 bytes from the CLI utility. When sending a command from the utility's CLI, short commands are executed correctly, a command exceeding 89 bytes suspends the exchange.
Having tested the example file:
the file was successfully written
file: |
I increased size of USB CDC RX buffer to 512 bytes and both the utilities and
But I'm not sure that this solution was right. |
esp32s2, CDC console.A console utility (both nodemcu-tool and upload-file.py ) can only be performed once. Steps to reproduce: file a.txt
upload-file.pyWe are trying to upload a file using upload-file.py
Perform hardware reset |
Can confirm weirdness on my S2 CDC-ACM. I have started looking into it, and I keep finding more weirdness 😩 |
esp32-s3; jtag console; make monitor Is there a limit on the maximum length of a Lua string? 255 bytes + \n
+1 byte causes error 😞 |
Regardless of the configured RX buffer size, the CDC-ACM is at risk of being overrun it seems. I've pushed an updated |
And to answer your other question, yes, there is a max line length ( |
I also use file upload/download in 256 byte chunks with acknowledgement. So there is no buffer overflow. |
Lua string length of 256 bytes is quite enough. |
@serg3295 It looks like the only way to change the default RX buffer is to also automatically default to using the CDC-ACM instead of the UART. I'm not sure whether that's a sensible change in defaults? |
You are right, there is no need to change the default value, since the USB CDC RX buffer size depends on the specific utility or IDE. In particular, for It might make sense to mention the setting |
Good idea, I've added extra notes to the module docs now. 👍 |
I've found a workaround to solve the issue of once connection of the utilities to USB CDC console on esp32s2. I changed EDIT: I noticed that |
esp32s3; jtag console crash log
esp32s2 breaks USB connection when trying to write LFS image. However, in both cases the image is eventually written correctly and the program is executed from LFS.
EDIT: crash log for esp32s2
|
I am not (currently) able to reproduce the "once only" issue on my s2 here. I can upload multiple files just fine, using the default log settings (Info). And yes, when the CDC console is active, WiFi initialisation interferes with USB enumeration somehow, as far as I can tell. Deferring wifi initialisation works around that issue, but results in the wifi log messages appearing after the Lua startup messages and prompt. Can you reproduce the LFS issue on the main dev-esp32 branch too? |
Unfortunately I still have the "once only" issue. I tested CDC console on esp32-s2 Mini and esp32-s3 Zero using the latest upload-file.py. I'm on Linux Mint 22 x86_64 Linux Lenovo 6.8.0-48-generic
I even tried running upload-file.py on Windows. Upload failed.
I'll try to run upload-file.py on another computer later to rule out a hardware (software?) problem with the USB port. What value is |
We can not upload LFS image to ESP via usb-serial-jtag on the dev-esp32 branch, so there is no issue with node.reload() 😄 On esp32, esp32-c6 connected via LFS reload log for esp32c6 uart console
|
I uploaded lfs.img to esp32c6; (esp_console: usb_serial_jtag; dev-esp32 branch ), then LFS reload log for esp32c6 usb-serial-jtag console
|
I connected the ESP32-S2 (using usb-cdc ) to a USB port on another machine and both |
Are you getting the LFS reload errors even after you got the utilities to work? If so, maybe we should open a new ticket for that? Unless we can demonstrate that it's related to the transmission via the console, I mean. |
Yes, LFS reload error is still present. I have opened a new issue #3667 . |
Expected behavior
Currently, to send a data fragment larger than 256 bytes via CONSOLE_USB_SERIAL_JTAG, a forced insertion of
\n
between 256-byte fragments is required.I would like the behavior to match the behavior when sending a long sequence via UART_DEFAULT. That is, data should be sent continuously without adding
\n
In the nodemcu-tool and nodemcu-tools utilities, uniterrupted sending is used to download a file from the ESP32.
Forcing
\n
to each piece will break the exchange protocol.Moreover, the size of the response to a request from the utility can exceed 256 bytes. And in this case, I can't even add
\n
Actual behavior
When sending a chunk longer than 256 bytes, only the first 256 bytes are sent.
The remaining bytes will be sent only if you add
\n
at the end of the chunk.Test code
This code works
This code does not work
NodeMCU startup banner
Startup banner
ESP-ROM:esp32c3-api1-20210207
Build:Feb 7 2021
rst:0x3 (RTC_SW_SYS_RST),boot:0xd (SPI_FAST_FLASH_BOOT)
Saved PC:0x40048b82
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fcd5820,len:0x1704
load:0x403cc710,len:0x938
load:0x403ce710,len:0x2ee8
entry 0x403cc710
I (24) boot: ESP-IDF v5.1.3 2nd stage bootloader
I (24) boot: compile time Jul 29 2024 20:30:34
I (24) boot: chip revision: v0.4
I (27) boot.esp32c3: SPI Speed : 80MHz
I (32) boot.esp32c3: SPI Mode : DIO
I (36) boot.esp32c3: SPI Flash Size : 4MB
I (41) boot: Enabling RNG early entropy source...
I (46) boot: Partition Table:
I (50) boot: ## Label Usage Type ST Offset Length
I (57) boot: 0 nvs WiFi data 01 02 00009000 00006000
I (65) boot: 1 phy_init RF data 01 01 0000f000 00001000
I (72) boot: 2 factory factory app 00 00 00010000 00180000
I (80) boot: 3 lfs unknown c2 01 00190000 00010000
I (87) boot: 4 storage Unknown data 01 82 001a0000 00070000
I (95) boot: End of partition table
I (99) esp_image: segment 0: paddr=00010020 vaddr=3c0e0020 size=2d748h (186184) map
I (137) esp_image: segment 1: paddr=0003d770 vaddr=3fc92400 size=028a8h ( 10408) load
I (139) esp_image: segment 2: paddr=00040020 vaddr=42000020 size=d19b8h (858552) map
I (279) esp_image: segment 3: paddr=001119e0 vaddr=3fc94ca8 size=001ach ( 428) load
I (280) esp_image: segment 4: paddr=00111b94 vaddr=40380000 size=1237ch ( 74620) load
I (299) esp_image: segment 5: paddr=00123f18 vaddr=50000000 size=00018h ( 24) load
I (305) boot: Loaded app from partition at offset 0x10000
I (306) boot: Disabling RNG early entropy source...
I (308) cpu_start: Unicore app
I (312) cpu_start: Pro cpu up.
I (327) cpu_start: Pro cpu start user code
I (327) cpu_start: cpu freq: 160000000 Hz
I (327) cpu_start: Application information:
I (330) cpu_start: Project name: nodemcu
I (335) cpu_start: App version: tmr-libmain-binpatch150-888-gca
I (342) cpu_start: Compile time: Jul 29 2024 20:32:43
I (348) cpu_start: ELF file SHA256: b3501ddbd7970a59...
I (354) cpu_start: ESP-IDF: v5.1.3
I (359) cpu_start: Min chip rev: v0.3
I (364) cpu_start: Max chip rev: v1.99
I (369) cpu_start: Chip rev: v0.4
I (373) heap_init: Initializing. RAM available for dynamic allocation:
I (381) heap_init: At 3FC997B0 len 00026850 (154 KiB): DRAM
I (387) heap_init: At 3FCC0000 len 0001C710 (113 KiB): DRAM/RETENTION
I (394) heap_init: At 3FCDC710 len 00002950 (10 KiB): DRAM/RETENTION/STACK
I (402) heap_init: At 50000018 len 00001FD0 (7 KiB): RTCRAM
I (409) spi_flash: detected chip: generic
I (412) spi_flash: flash io: dio
W (417) rmt(legacy): legacy driver is deprecated, please migrate to
driver/rmt_tx.h
and/ordriver/rmt_rx.h
W (427) ADC: legacy driver is deprecated, please migrate to
esp_adc/adc_oneshot.h
I (436) sleep: Configure to isolate all GPIO pins in sleep state
I (442) sleep: Enable automatic switching of GPIO sleep configuration
I (449) app_start: Starting scheduler on CPU0
I (454) main_task: Started on CPU0
I (454) main_task: Calling app_main()
E (464) mmap: esp_mmu_paddr_to_vaddr(752): paddr isn't mapped
LFS image loaded
I (474) pp: pp rom version: 9387209
I (474) net80211: net80211 rom version: 9387209
I (484) wifi:wifi driver task: 3fca87a4, prio:23, stack:6656, core=0
I (484) wifi:wifi firmware version: 0016c4d
I (484) wifi:wifi certification version: v7.0
I (484) wifi:config NVS flash: enabled
I (484) wifi:config nano formating: disabled
I (484) wifi:Init data frame dynamic rx buffer num: 32
I (484) wifi:Init static rx mgmt buffer num: 5
I (484) wifi:Init management short buffer num: 32
I (484) wifi:Init dynamic tx buffer num: 32
I (484) wifi:Init static tx FG buffer num: 2
I (484) wifi:Init static rx buffer size: 1600
I (484) wifi:Init static rx buffer num: 10
I (484) wifi:Init dynamic rx buffer num: 32
I (484) wifi_init: rx ba win: 6
I (494) wifi_init: tcpip mbox: 32
I (494) wifi_init: udp mbox: 6
I (494) wifi_init: tcp mbox: 6
I (494) wifi_init: tcp tx win: 5760
I (494) wifi_init: tcp rx win: 5760
I (494) wifi_init: tcp mss: 1440
I (494) wifi_init: WiFi IRAM OP enabled
I (494) wifi_init: WiFi RX IRAM OP enabled
NodeMCU ESP32 build 2024-07-29 20:32 powered by Lua 5.3.5 [5.3-int32-singlefp] on IDF v5.1.3
cannot open init.lua: No such file or directory
Hardware
esp32c3 and esp32c6 via CONSOLE_USB_SERIAL_JTAG
The text was updated successfully, but these errors were encountered: