ports/esp32: Print support information on panic. #14086
Conversation
e563f09 to
c541f6d
Compare
|
This does look truly helpful, thanks. Do you know a convenient way to trigger a panic at the moment to test it? |
|
One I know of is to make use of the fact that the PPP implementation can call into MicroPython code in its own task without first setting up a proper thread state or holding the GIL (see my notes in #12729 (comment)). So if we give it a zero-sized BytesIO, then make it write to it, that will trigger an reallocation of the BytesIO buffer which will fail and crash: from network import PPP
from io import BytesIO
ppp = PPP(BytesIO(0))
ppp.active(True)
ppp.connect() |
|
I tested this on my esp32c6 and it worked as intended. I would suggest though rather than the message saying to raise an issue, it might be better to print a link to a (new) wiki page about these crashes, that could then describe how to raise an issue (if needed) and could even include the details on how to decode the dump. |
|
Thanks, this looks great! But I guess it doesn't work well on esp32-s2/s3, which have native USB. Because the USB disappears when there's a crash.
Yes, I agree. Let's create a wiki page to get a URL and use that in the message. |
That might indeed be a limitation depending on how the chip is connected, but that's not necessarily always the case for these chips. For example: I've tested this on a ESP32-S3 based board which uses the native USB serial/JTAG console (
Agreed, that does indeed seem like a better solution. We can replace that second part of the message with something like this? |
c541f6d to
79b4b3a
Compare
|
I've just written a basic wiki page on debugging crashes on the ESP32, and updated the panic message in this MR accordingly. https://github.com/micropython/micropython/wiki/ESP32-debugging |
79b4b3a to
f8373de
Compare
I tested this code on my esp32c3 board (idf5.0.4) with this pr code, it's crashed, but there's no extra message showed, I wonder why? |
Did the default ESP-IDF panic handler show up, or did it just reboot? The default panic handler output usually starts with If that didn't show, then you're likely losing some output, perhaps as mentioned above due to the USB connection going away before the error output has been received. That is not something this specific PR aims to change. |
When a fatal error occurs it's important to know which precise version it occurred on in order to be able to decode the crash dump information such as the backtrace. By wrapping around the built-in IDF panic handler we can print some extra information whenever a fatal error occurs. The message links to a new wiki page which contains additional information on how to debug ESP32 issues, and links to the bug reporting issue template. Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
f8373de to
c10a74b
Compare
|
Thanks for updating. Now merged. |
Yes, just like a normally hard reset. I tested another board that connected to PC by using USB CDC mode, and this time the extra message and |
|
Hi @DvdGiessen did you by any chance see a full fatal error output on any of the new risc-v chips? I ran into one yesterday on my C6 and got your new header show upv perfectly, so it was the perfect opportunity to follow the link to the wiki instructions and follow them. I even thought then, hey it'd be handy to add a No backtrace line. Seems the fatal error output for these is slightly different? Or maybe IDF 5.2.2 is a bit different? The docs for C3 do look a bit different in this section: https://docs.espressif.com/projects/esp-idf/en/v5.2.2/esp32c3/api-guides/fatal-errors.html#register-dump-and-backtrace Where I expected to see a backtrace in the printout there was a big block of stack trace which presumably has the same information but it was formatted differently so not sure if/how it can be get into the address decoder tool. I ended up running TLDR I think the wiki page might need updating for risc-v chips but not sure exactly how yet. |
|
@andrewleech Indeed, I wrote the wiki page mostly based on the output of the Xtensa-based chips. For the RISC-V baseed chips there isn't a straight up backtrace printed, and I think instead To be more precise, the monitor doesn't seem to know what it is decoding. I think it looks for things that look like addresses in the output and then prints a line for those, without actually knowing something is a backtrace, register dump, etc. The package implementing it is So for RISC-V based chips, the wiki should probably indeed explain to look at the EDIT: For reference, the output on a random C3 board I had around: So we have memory and registers, but not a backtrace. I think for now just documenting that you can either look at the MEPC or attach a debugger to get a more complete backtrace is best, and then we can expand upon that later if we figure out additional things that might help. |
|
Thanks for all your work on this, @DvdGiessen! Even though I wrote the first version of idf_monitor many years ago, I even learned something new reading your wiki page.
Actually idf.py monitor can do a bit more than this. Here's the backtrace from ESP32-C3 if I insert an The monitor actually recognises the block of stack memory, reprocesses it, and dumps it back out as a gdb backtrace. I think the relevant code is here: I assume this is the same on C6, but I don't actually have a C6 at hand yet to test (on its way). I'm not sure what the best MicroPython-friendly solution here is. The easiest one is to run I think the best would be to build a little command line decoder tool that someone can paste a stack trace into, that calls the idf_panic_handler functions internally. |
When a fatal error occurs it's important to know which precise version it occurred on in order to be able to decode the crash dump information such as the backtrace.
I noticed it today in #14005 (comment), and quickly prototyped a small improvement.
By wrapping around the built-in IDF panic handler we can print some extra information whenever a fatal error occurs. The message link to a new wiki page which contains additional information on how to debug ESP32 issues links to the bug reporting issue template.
Example of current output:
(Another suggestion is we could perhaps mention the SHA256 hash of each ESP32 image ELF file on the website. The IDF prints the "ELF file SHA256" at the end of each fatal error message, making it easy to look up the corresponding ELF file.)