esp32-s3 does not reset after upload

[prplz]

Board

esp32s3

Device Description

esp32s3 devkitc n8r8

Hardware Configuration

Native USB port plugged into computer, serial convert USB port not plugged in.

Version

v2.0.3

IDE Name

Arduino IDE, Platformio

Operating System

Windows 10, MacOS

Flash frequency

80MHz

PSRAM enabled

no

Upload speed

921600

Description

Uploading a sketch over usb otg should reset the chip after upload, but it does not.

Other Steps to Reproduce

• Select USB mode = USB-OTG, USB CDC on boot = enabled, Upload mode = USB OTG CDC.
• Upload sketch
• Board correctly resets into download mode using 1200 baud and uploads the sketch
• Board does not reset and run the sketch as it should, instead staying in download mode. Here you have to press the reset button manually. On S2 the board resets and runs the sketch after upload with no manual intervention.

Tested on s3 devkitc and all UnexpectedMaker s3 boards.

Notes:

• No esptool commands reset the board, on the s2 it does reset
• Esptool doesn't detect when the board was put into bootloader using IO0 as it does on s2 (WARNING: ESP32-S2FNR2 chip was placed into download mode using GPIO0)
• USB mode and upload mode set to hardware CDC resets as expected

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[prplz]

cc @UnexpectedMaker

[UnexpectedMaker]

@me-no-dev I assume this is not the intended behaviour? And the S3 boards should reset after flashing them like the S2 boards do right?

[me-no-dev]

@UnexpectedMaker not if you are using TinyUSB while booted. It works if the integrated CDC+JTAG is used, but the other USB no.

[UnexpectedMaker]

@UnexpectedMaker not if you are using TinyUSB while booted. It works if the integrated CDC+JTAG is used, but the other USB no.

Hmm, but it works correctly with the S2, so why not with the S3? It's really confusing people - they think their boards didn't flash, as no other ESP32 has this issue (original, S2, C3 etc) - so people are complaining about HW faults.

If we use CDC+JTAG, then the board doesn't enumerate with the custom device name, which also confuses people, so it seems like the S3 has the worst of both situations, rather than any improvement in either.

How do we get this behaviour changed?

[sukesh-ak]

Yes this has become a painful issue. Have tested from Arduino and ESP-IDF with same behaviour.

This also means you have to reconnect serial monitor after every firmware upload for proper update.

[VojtechBartoska]

This seems worth improving and documenting as well? cc @me-no-dev & @pedrominatel

[me-no-dev]

How do we get this behaviour changed?

Surely if we could we would have :) I went through ROM code and everything, but could not figure out what is preventing the board/download mode to reboot the firmware. When booting into download from TinyUSB, we first stop TinyUSB, then init CDC+JTAG and then reboot into download, so in reality it should have been the same as rebooting from CDC+JTAG, but it is not. We might find a way in the future, but this has taken too much resources already as it is.

[frankcohen]

This is pain, I'm having to use the Boot pin after every upload. And that has the side effect of closing Arduino IDE Serial Monitor randomly.

I'm glad to volunteer my hours with direction. Just tell me what to do.

Background: I'm working on the open-source Reflections wrist watch project, including an ESP32-S3-Mini based main board. I'm building this using the Adafruit Feather ESP32-S3 as a reference and using their Arduino IDE 12.3.1 board definition "Adafruit Feather ESP32-S3 No PSRAM". I'm using ESP32 version 2.0.3, installed through the Boards Manager. Repository for my project is at https://github.com/frankcohen/ReflectionsOS/tree/main/Devices/Hoober

[sukesh-ak]

@frankcohen
As a side note you should check out UM boards for S3.
https://unexpectedmaker.com/shop

[frankcohen]

Thanks, the UM boards look great. I'd be glad for UM to carry my board too. I will open-source the board. -Frank

[boberace]

Not sure if this is helpful but ... My ESP32-S3-DevKitC-1-N8R2 does reset after upload if I use the UART USB jack but it will not reset after upload when using the OTG USB jack UNLESS I upload right after I erase flash.

My UM Feather S2 does reset after upload. My UM TinyS3 does not reset after upload.

[UnexpectedMaker]

@me-no-dev Is this issue caused by security advisory AR2022-004 ?

[me-no-dev]

@UnexpectedMaker yes and no. You can think of this as two separate issues:

1. We can not use the USB-OTG in download mode (so we switch to the integrated CDC+JTAG)
2. When we switch to CDC+JTAG and reboot into Download Mode, chip is not reset after the new firmware is flashed (what we are discussing in this thread)

AR2022-004 talks about issue 1

[UnexpectedMaker]

@me-no-dev So if issue 1 is resolved, then you could use USB-OTG and not have to switch to CDC, bypassing issue 2? So that would also fix this problem on newer silicon?

[me-no-dev]

@UnexpectedMaker exactly :)

[egnor]

My understanding [EDITED based on subsequent investigations]

Background

There are multiple ways to connect from a a host (Mac/PC) USB port to an ESP32-S3 for bootloading (download) and serial terminal purposes:

1. Use a physical USB-to-serial converter chip/module, connected to the ESP32-S3's serial TX and RX lines
2. Use the ESP32-S3's onboard USB controller, wire the USB port directly to the ESP32-S3.

If using option (1) there is no problem. (For example this is the UART USB jack on the ESP32-S3 DevKit board.) However, that requires an extra part and it's otherwise nice to use the integrated USB controller. So, let's assume option (2). This requires configuring the ESP32-S3 to act as a serial port device. There are again multiple ways to do that:

1. Use the ESP32-S3's internal USB-OTG (On The Go) peripheral, which supports acting as a general purpose USB host or device (the bidirectionality is what makes it "On The Go"); in software, set the peripheral up to act as a USB device, and implement the USB CDC ("Communications Device Class", aka serial port) protocol, and thus the device shows up as a serial port to the host.
2. Use the ESP32-S3's internal USB Serial/JTAG peripheral, which is a separate special purpose USB controller that only supports USB CDC (serial port) and JTAG (debugging port, not important here) protocols.

Using the dedicated USB Serial/JTAG controller is simpler for software, since the hardware takes care of most of the details of USB. However, the USB-OTG controller is more general, allowing other device classes (such as mass storage) in addition to being a serial port.

Importantly, the bootloader (used to upload code) and the application (which wants to Serial.print to a console) might make different choices about which controller to use. The Arduino platform, for example, tends to use TinyUSB, a software USB stack that expects a fully general USB controller, which would imply the USB-OTG controller.

Issues

So far this is all lovely but now there are some wrinkles.

As @me-no-dev noted above, the bootloader cannot use the USB-OTG controller. This is related to a spot of confusion over one-time-programmable EFUSE bits as described in security advisory AR2022. Basically, for many ESP32-S3 parts, bootloader USB-OTG access is permanently disabled.

So, the bootloader uses the USB Serial/JTAG controller instead (if allowed). This works just fine and allows the flash to be programmed in the normal way. HOWEVER, it is observed that sometimes the bootloader is unable to exit into the application, and a manual reset is required to start the app.

This appears to happen if the bootloader (download mode) is entered (by reboot-to-bootloader typically triggered by DTR/RTS wiggles) FROM an application using the USB-OTG controller (not hardware USB Serial/JTAG), which is the Arduino runtime's default.

Per @me-no-dev's comment above, the reasons for this are not entirely understood, but it's thought to be something about switching between the two controllers?

Workarounds

There are two known ways to work around this:

1. After loading code, use a full hardware reset (probably a button) to launch the app. This fully resets the board and skips the USB Serial/JTAG controller, avoiding issues with switching. However it's inconvenient, impossible without physical access, and disruptive to port connectivity.
2. In the application, use the USB Serial/JTAG controller instead of the USB-OTG controller. This limits flexibility (can't be a mass storage device at the same time) and makes the port show up as a generic "Espressif USB JTAG/serial debug unit".

To switch the Arduino runtime to the USB Serial/JTAG controller (workaround (ii)):

• In the Arduino IDE, select Tools > USB Mode > Hardware CDC and JTAG (pictured above).
• With arduino-cli, use --build-property=build.usb_mode=1 on the command line (untested).
• With platformio, use build_flags = -DARDUINO_USB_MODE=1 in platformio.ini.
  NOTE, you may need to also add -DARDUINO_USB_CDC_ON_BOOT=0 OR -DARDUINO_USB_CDC_ON_BOOT=1 depending on platform version; see discussion in comments below.
• If using ESP-IDF and not Arduino, select ESP_CONSOLE_USB_SERIAL_JTAG in menuconfig.

Both workarounds are somewhat unfortunate, which is why this bug exists and people are grumpy. As a third approach, some people have had luck with patches to esptool, see comments below for discussion and links, but this is not yet a confirmed fix.

Investigation notes

As noted above, success/failure seems to be triggered not by what app we're loading but by what app was running before resetting to the bootloader:

• run OTG-using app, reset to run, flash JTAG/Serial-using app => loads but does NOT run
• reset to start JTAG/Serial-using app, flash the same app again => loads and DOES run
• flash OTG-using app => loads and DOES run right away!
• flash OTG-using app again => loads and does NOT run
• reset to run OTG-using app, flash OTG-using app again => loads and does NOT run
• reset to run OTG-using app, flash JTAG/Serial-using app => loads but does NOT run
• reset to start JTAG/Serial-using app, flash OTG-using app => loads and DOES run

So somehow, when resetting back to the bootloader, a running OTG-using app leaves state in the system that prevents the bootloader from jumping to start ANY program it loads. Maybe the reboot-to-bootloader code in the OTG-based driver is somehow doing things slightly wrong?

For reference, the USB Serial/JTAG controller can be distinguished from the USB-OTG controller externally; it shows up as "Espressif USB JTAG/serial debug unit" with VID:PID 303A:1001.

Rebooting from app to bootloader and bootloader to app (the path that fails) is generally managed by wiggling DTR and RTS serial control lines -- this is the "Hard resetting via RTS pin..." message from in esptool.py (with the default --after=hard_reset). Of course these "pins" are purely virtual in this case, contained within the emulated serial port.

The USB Serial/JTAG controller manages DTR/RTS wiggle detection directly. This is described in section 30.3.2 in the technical reference:

Here is the further discussion in section 30.4.2:

The reset-to-app code in esptool.py does this, assuming DTR=0 and RTS=0 on entry:

    def hard_reset(self):
        if self.uses_usb():
            self._check_if_can_reset()

        print("Hard resetting via RTS pin...")
        self._setRTS(True)  # EN->LOW
        if self.uses_usb():
            # Give the chip some time to come out of reset,
            # to be able to handle further DTR/RTS transitions
            time.sleep(0.2)
            self._setRTS(False)
            time.sleep(0.2)
        else:
            self._setRTS(False)

(Tangent: self.uses_usb() is currently broken; see bug espressif/esptool#756 and PR espressif/esptool#757. Fixing it doesn't help this issue, though.)

DTR and RTS can be wiggled manually for testing with miniterm, with ^T^D and ^T^R respectively. After reproducing this problem (successfully flashing an app, but failing to reboot-to-app), we can see this:

% python -m serial.tools.miniterm /dev/ttyACM0 115200
--- Miniterm on /dev/ttyACM0  115200,8,N,1 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
--- RTS inactive ---  [[ note: ^T^R was pressed here ]]
--- DTR inactive ---  [[ ^T^D was pressed here ]]
--- RTS active ---  [[ ^T^R ]]
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0x0 (DOWNLOAD(USB/UART0))
Saved PC:0x40041a76
waiting for download

So it does reboot on wiggle, but the waiting for download indicates it ended up back in the bootloader. It's as if the "download flag" was not successfully reset during the RTS=0 DTR=0 point. If I go through the other sequence to explicitly set the "download flag", I get identical output:

--- RTS inactive ---
--- DTR active ---
--- RTS active ---
--- DTR inactive ---
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0x0 (DOWNLOAD(USB/UART0))
Saved PC:0x40041a76
waiting for download

When reboot-to-app does work (after running the esp-idf/examples/system/console/basic sample app built with ESP_CONSOLE_USB_SERIAL_JTAG), we can see both of the USB/Serial controller's DTR/RTS wiggle sequences work correctly. Going from bootloader to app:

waiting for download
--- DTR inactive ---
--- RTS inactive ---
--- RTS active ---
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x400428a0
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3810,len:0x113c
load:0x403c9700,len:0xa4c
load:0x403cc700,len:0x2af0
entry 0x403c9898
 ... blah blah blah ...
Type 'help' to get the list of commands.
Use UP/DOWN arrows to navigate through command history.
Press TAB when typing command name to auto-complete.
Your terminal application does not support escape sequences.

Line editing and history features are disabled.
On Windows, try using Putty instead.
esp32s3>

Going from app back to bootloader:

esp32s3> --- RTS inactive ---
--- DTR active ---
--- RTS active ---
--- DTR inactive ---
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0x0 (DOWNLOAD(USB/UART0))
Saved PC:0x4037c14a
waiting for download

Note the boot:0x8 (SPI_FAST_FLASH_BOOT) instead of boot:0x0 (DOWNLOAD(USB/UART0)).

As a final note, fixing uses_usb() detection creates a problem with false errors:

WARNING: ESP32-S3 chip was placed into download mode using GPIO0.
esptool.py can not exit the download mode over USB. To run the app, reset the chip manually.
To suppress this note, set --after option to 'no_reset'.
*** [upload] Error 1

Apparently GPIO_STRAPPING gets zeroed out by internal resets, including resets triggered by DTR/RTS wiggles detected by the USB Serial/JTAG peripheral. That triggers the false locked-into-bootloader alert from esptool. (In actual fact if the error is bypassed, esptool can reboot into app just fine in that scenario.)

Esptool detects another method of software reboot-into-bootloader (RTC_CNTL_FORCE_DOWNLOAD_BOOT) to avoid some false alarms, but the USB Serial/JTAG peripheral doesn't use that mechanism. USB Serial/JTAG triggered reboot does have a reason code in RTC_CNTL_RTC_RESET_STATE_REG, but esptool.py's initial reset always leaves that reason code in place, even when the system is locked into the bootloader. So that's not a useful discriminator. With uses_usb() broken (as it currently is at head), at least that check is disabled and doesn't get in the way...

The main relevance of the above is that there seems to be a bunch of subtle/hidden chip state controlling whether or not the system starts in the bootloader or jumps to the app. The logic is hard to observe, hidden in the obscurities of the closed-source ROM, and thus hard to debug from outside.

[frankcohen]

Just a note of appreciation to Egnor, thank you. And a question, have you seen JTAG with platform IO and esp32s3 working?

[egnor]

@frankcohen wrote:

Just a note of appreciation to Egnor, thank you. And a question, have you seen JTAG with platform IO and esp32s3 working?

I have not tried! I am curious about it but figured I'd open one can of worms at a time. If you do try, or have tried, I'd like to know how it goes.

[frankcohen]

I'll let you know. The bigger problem to solve first is getting Platform IO to recognize the Adafruit Feather ESP32-S3 board definition file. That's next.

…

On Wed, Jul 13, 2022, 5:55 PM Daniel Egnor ***@***.***> wrote: @frankcohen <https://github.com/frankcohen> wrote: Just a note of appreciation to Egnor, thank you. And a question, have you seen JTAG with platform IO and esp32s3 working? I have not tried! I am curious about it but figured I'd open one can of worms at a time. If you do try, or have tried, I'd like to know how it goes. — Reply to this email directly, view it on GitHub <#6762 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABUEWMKREROROZRVCZXFXPTVT5QP5ANCNFSM5WDKDUNA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[UnexpectedMaker]

@egnor This is great investigative work Daniel! Thanks very much for the awesome effort.

[wuast94]

@Jason2866 testet it local and now it compiles but fail with resetting:

Wrote 935424 bytes (617112 compressed) at 0x00010000 in 8.2 seconds (effective 911.8 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting with RTC WDT...

A serial exception error occurred: Cannot configure port, something went wrong. Original message: OSError(22, 'The I/O operation has been aborted because of either a thread exit or an application request.', None, 995)
Note: This error originates from pySerial. It is likely not a problem with esptool, but with the hardware connection or drivers.
For troubleshooting steps visit: https://docs.espressif.com/projects/esptool/en/latest/troubleshooting.html

i will create the pr but think it wouldnt make much sense if we have this problem?

the flash was successfull and i can see in terminal that it works, its just the reset not working, im using the develop branch

[Jason2866]

@wuast94 I think you have a general problem with the port. Anyways the PR for the boards.json is not related to this.

[Parsaabasi]

Hello,

Due to the overwhelming volume of issues currently being addressed, we have decided to close the previously received tickets. If you still require assistance or if the issue persists, please don't hesitate to reopen the ticket.

Thanks.

[daniloc]

Hello. Definitely still an issue. Could we reopen?

[AndrePinheir0]

I have the same issue, trying to build the lvgl demo example with ESP32-S3
My COM11 port disappears and the flash doesnt work. I only can see the UI, if I displug and plug again the ESP32s3 to USB.

Edit: Waveshare ESP32-S3-Touch-LCD-4.3 controller - I have this model, that has a touch LCD 4.3 and my connection is to USB of my PC. Could it be that there is not enough power to power the esp32 and the lcd?

--- Warning: GDB cannot open serial ports accessed as COMx
--- Using \\.\COM11 instead...
--- esp-idf-monitor 1.5.0 on \\.\COM11 115200
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0x0 (DOWNLOAD(USB/UART0))
Saved PC:0x40041a79
--- 0x40041a79: ets_delay_us in ROM

waiting for download````

[daniloc]

esp32-s2 doesn't have this problem at all. Is there anything we can learn from its implementation?

[me-no-dev]

• ESP32-S2 has only USB-OTG and can be flashed through the OTG port, so flashing and rebooting after is not an issue
• ESP32-S3 has both USB-OTG and CDC/JTAG interfaces, but you can not flash over the OTG port. To overcome that issue and allow flashing over USB, when the OTG port is used in the firmware, we switch to the CDC/JTAG interface before rebooting into Download and that combination does not automatically reboot after flashing. Using just the CDC/JTAG works fine.
• I think that the latest ESPTOOL should allow for this to work on S3 in particular.

[SaintSampo]

• I think that the latest ESPTOOL should allow for this to work on S3 in particular.

So do you mean that in arduino-esp32 version 3.1.0 and up, which uses esptool v4.8.1, this issue should be fixed? I have been watching this thread for awhile, I'll be interested to test this tonight.

[FunDeckHermit]

For anyone still struggling, watchdog_reset has been added.
Documentation

When using Platformio add:
board_upload.after_reset = watchdog_reset

Not sure how to inject the "watchdog_reset" command into Arduino IDE.

[ppescher]

@FunDeckHermit Does a "watchdog_reset" clear the RTC_CNTL_FORCE_DOWNLOAD_BOOT_MASK in the register RTC_CNTL_OPTION1_REG?

If it doesn't, then you will keep entering bootloader mode after reset, if you used this method to enter boot, which is what the ESP32-S3 Arduino framework does via the native USB serial interface when you perform the "upload" command.

That register bit mask is reset by the "hard_reset" command in recent esptool releases, exactly to address this issue.

I am curious, then what is "watchdog_reset" for?

[Jason2866]

@ppescher WDT resets works for S3 when TinyUSB is used.

[ppescher]

@ppescher WDT resets works for S3 when TinyUSB is used.

So it works for USB OTG, not CDC/JTAG? I thought that to enter bootmode you had to still perform the switch to USB CDC and use RTC_CNTL_FORCE_DOWNLOAD_BOOT_MASK... is that true? So how does this work? I'm confused now...

[radimkarnis]

WDT reset:

1. enables resetting in USB-OTG mode (S2 and S3), which wasn't possible before. For this reason, watchdog_reset is default if USB-OTG is detected.
2. fixes some issues in USB-Serial/JTAG mode (e.g., getting stuck in download mode if it was entered manually with the boot button), but can lead to other issues (the controller stopping completely - requiring a power cycle). For this reason, it is available as a choice (--after watchdog_reset), but not by default. Also, it cannot be performed on the ESP32-H2.

Not sure about the TinyUSB specifics in Arduino.

[daniloc]

@FunDeckHermit that works PERFECTLY, thank you so much