Memory leak in a Wi-Fi connection and disconnection process (IDFGH-6820)

[TakuyaAtMono]

Hello,

I have created a Wi-Fi connection and disconnection process using the sample code "example_connect".

I used the heap_trace_init_standalone function to check for memory leaks before and after the connection, and found that the part shown in the attached file was leaking.

For example, there are some processes in the library that are allocated in esp-idf/components/esp_phy/src/phy_init.c:204 and then not released.

Can you please change these memory allocations so that they are released normally when Wi-Fi is stopped? Or please let me know how to handle this.

heapTrace.txt

[MaxwellAlan]

hi @TakuyaAtMono ,
Thanks for reporting this issue, we will fix this.

[AxelLin]

@MaxwellAlan
Which esp-idf versions have such memory leak?

[MaxwellAlan]

@AxelLin
Check in master branch (f02169b), s_phy_digital_regs_mem not free.

[jack0c]

@AxelLin @MaxwellAlan @TakuyaAtMono This is somehow in purpose, We didn't allocate the memory by use to reduce the allocate time when chip is go to sleep. The memory is shared by Wi-Fi and BLE, If we want to fix this bug, we need use Wi-Fi deinit and BT_deinit to free the memory. If you do think free the memory is necessary, we will consider add it.

[TakuyaAtMono]

@jack0c -san,
I would really appreciate it if you could add a memory release function to the deinit.

The RAM space cannot be effectively utilized due to the fact that memory is continually being allocated, and this is hindering the creation of our applications.

Furthermore, it is difficult for not library engneer to identify the memory locations that are not being released one by one.
Therefore, it would be very hepl if you could add a memory deallocation function.

Specifically
I would like to release the memory that allocated by the Wi-Fi process from executing esp_wifi_init() and esp_wifi_connect() to esp_wifi_disconnect() and esp_wifi_deinit().

(FYI, our development will not use BLE.)

[KaeLL]

We didn't allocate the memory by use to reduce the allocated time when chip is go to sleep.

So if sleep functionality is disabled/unused, the allocation doesn't happen, right?

[jack0c]

OK, let's make sure two things:

1. if sleep and wake not happen, not allocate the memory
2. if both Wi-Fi and BLE deinit, free the memory

[KaeLL]

@TakuyaAtMono are/were you using sleep functionality?

[TakuyaAtMono]

I believe the problem is that the memory allocated at esp_wifi_init() and esp_wifi_connect() and that is not released at esp_wifi_disconnect() and esp_wifi_deinit().
I'm assuming that sleep is not relevant to the discussion, is it?

As @jack0c wrote, "if both Wi-Fi and BLE deinit, free the memory" that is what I want.

I hope the fix will be applied also to the ESP-IDF v4.4.1 release version not only the master verion of the IDF.

[TakuyaAtMono]

I pressed the close button by mistake. Please ignore the status change.
We continue to request a resolution to this matter.

[KaeLL]

From what I understood from @jack0c's comment, the memory isn't deallocated so as to reduce the work performed at sleep/wake-up. That would make sleep relevant, were it not for the fact that you're not using it, I assume, based on your dismissal?! If that's the case, you're onto something and there is an issue since

if both Wi-Fi and BLE deinit, free the memory

isn't actually the case.

[TakuyaAtMono]

Hi @KaeLL , @jack0c

Yes, I think

if both Wi-Fi and BLE deinit, free the memory

is not working in current revision.

[jack0c]

@TakuyaAtMono This is merged into master(after 7a8b638), do you think it is OK? If it works for you, I can backport to IDFv4.4. You can cherry-pick it to have a try.

[TakuyaAtMono]

@jack0c
Our current project is based on ESP-IDF v4.4.1 release version.
Therefore, I have tryed the ESP-IDF version to master but out project can not compile without error.

So, I copied only the changes made in 7a8b638 to ver4.1.1 and tried it, but the memory is not released.
I think MaxwellAlan mentioned "s_phy_digital_regs_mem" was not free, does not appear to have been changed.

How can we try that the changes will be effective?

[jack0c]

@TakuyaAtMono Sorry, I didn't realize s_phy_digital_regs_mem is different from s_mac_bb_pd_mem, will add a new fix, and backport both to IDFv4.4.

[TakuyaAtMono]

@jack0c
Thank you.

And let me share the information what I tryed.

I verified by using "Standalone Mode" of Heap memory debugging to connect to wifi (esp_wifi_init) and getting the leak status after disconnecting (esp_wifi_deinit).
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/heap_debug.html

As I attached files "1372 bytes 'leaked' in trace (23 allocations)" is unchanged before and after applying "commit 7a8b638", so I think that the memory is not released.

before.txt
after.txt

[jack0c]

@TakuyaAtMono A mistake happen in s3. see cb0dd5c . I have merged the new one into master, and I also backport both of them to IDFv4.4, you can find it in b7d5e81. Please have a try.

[TakuyaAtMono]

@jack0c I searched but could not find commit b7d5e81. Could you please provide a link?

[Alvin1Zhang]

@TakuyaAtMono Thanks for reporting and sharing the updates. Would you please help check b7d5e81? Thanks.

[bitmandu]

I'm seeing a memory leak from esp_wifi_init() and esp_wifi_deinit() in ESP-IDF v5.1-dev-2186-g454aeb3a48. Is this commit for v4.4 suppose to resolve this? (If so, it doesn't seem to.)

A test application I made to demonstrate this shows:

664 bytes 'leaked' in trace (13 allocations)
total allocations 64 total frees 68

[aircable]

My example is a bit more complex but I can confirm that there is a memory leak in v5.1-dev-72.

esp_wifi_disconnect();
esp_wifi_stop();
esp_wifi_deinit(); 
esp_event_handler_unregister(...); // this is necessary otherwise we get duplicate events!!!
- wait
esp_wifi_init(..)

After a number of loops through this the call will fail with ESP_ERR_NO_MEM

[Espressif-liuuuu]

Thanks for your feedback and sorry for late reply @aircable @bitmandu
Hi @bitmandu , I tested your case and let me clarify that, one of the leaked memory comes from vTaskDelay and it took 64Bytes, while the others belongs to Wi-Fi static buffers those only allocated at the first time Wi-Fi initialization. In other words there is still 664 bytes 'leaked' in trace if you run reinitialization for more times. So it would not lead to serious result.

And it seems the different case for @dbahrdt. Could you please help provide your example so that we could reproduce your issue.

[dbahrdt]

Hi @Espressif-liuuuu ,

I guess you meant to mention someone else.

[Alvin1Zhang]

Thanks for reporting, will close due to short of feedback, feel free to reopen with more updates. Thanks.

[lptr]

I am experiencing this problem with ESP-IDF 5.1.4 and even with ESP-IDF 5.3.1 with an ESP32-S3. My device connects and disconnects from WiFi once every minute; this way it eats up ~180 kB of free memory in a day or less, causing the device to restart.

What would it take to get this problem fixed? I'm happy to work on a reproducer, though it seems there is one already mentioned above. Is there some other way I can help?

[Espressif-liuuuu]

Hi @lptr , you can enable CONFIG_HEAP_TRACING_STANDALONE and use heap trace debug as the application below test application. 2 cases you can try

1. run one time connect-disconnect, and you may see some leakage in trace. No worry since some allocations are lazy free and wont allocate them in second time. Just record the one-time-leakage.
2. run many time connect-disconnect, and record the leakage to see if any leakage delta during each connect-disconnect

[lptr]

@Espressif-liuuuu Yep, I ran that, and I see the leakage. What should I do next?

This is with an ESP32-S3:

I (5535) wifi:Deinit lldesc rx mblock:10
====== Heap Trace: 13 records (100 capacity) ======
    84 bytes (@ 0x3fca94d8, Internal) allocated CPU 0 ccount 0x042ac374 caller 0x40376183:0x4037da9c
--- 0x40376183: heap_caps_malloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:84
0x4037da9c: pvPortMalloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/freertos/heap_idf.c:58

    84 bytes (@ 0x3fca9530, Internal) allocated CPU 0 ccount 0x042ae93c caller 0x40376183:0x4037da9c
--- 0x40376183: heap_caps_malloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:84
0x4037da9c: pvPortMalloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/freertos/heap_idf.c:58

    84 bytes (@ 0x3fca9988, Internal) allocated CPU 0 ccount 0x042dee78 caller 0x40376183:0x4037da9c
--- 0x40376183: heap_caps_malloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:84
0x4037da9c: pvPortMalloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/freertos/heap_idf.c:58

    12 bytes (@ 0x3fcaa114, Internal) allocated CPU 0 ccount 0x04371de8 caller 0x40376183:0x403761b8
--- 0x40376183: heap_caps_malloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:84
0x403761b8: heap_caps_malloc_default at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:110

    84 bytes (@ 0x3fcabcac, Internal) allocated CPU 0 ccount 0x04373084 caller 0x40376183:0x4037da9c
--- 0x40376183: heap_caps_malloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:84
0x4037da9c: pvPortMalloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/freertos/heap_idf.c:58

     4 bytes (@ 0x3fcaa124, Internal) allocated CPU 0 ccount 0x04373fe8 caller 0x40376183:0x403761b8
--- 0x40376183: heap_caps_malloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:84
0x403761b8: heap_caps_malloc_default at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:110

    12 bytes (@ 0x3fcaa134, Internal) allocated CPU 0 ccount 0x04374b3c caller 0x40376183:0x403761b8
--- 0x40376183: heap_caps_malloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:84
0x403761b8: heap_caps_malloc_default at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:110

    84 bytes (@ 0x3fcb1cfc, Internal) allocated CPU 0 ccount 0x04ae9118 caller 0x40376183:0x4037da9c
--- 0x40376183: heap_caps_malloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:84
0x4037da9c: pvPortMalloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/freertos/heap_idf.c:58

    16 bytes (@ 0x3fca8400, Internal) allocated CPU 0 ccount 0x33c99bd8 caller 0x40376183:0x403761b8
--- 0x40376183: heap_caps_malloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:84
0x403761b8: heap_caps_malloc_default at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:110

    16 bytes (@ 0x3fca8414, Internal) allocated CPU 0 ccount 0x33c9b370 caller 0x40376183:0x403761b8
--- 0x40376183: heap_caps_malloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:84
0x403761b8: heap_caps_malloc_default at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:110

    16 bytes (@ 0x3fcaa0f8, Internal) allocated CPU 0 ccount 0x33c9fe7c caller 0x40376183:0x403761b8
--- 0x40376183: heap_caps_malloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:84
0x403761b8: heap_caps_malloc_default at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:110

    84 bytes (@ 0x3fcb1e90, Internal) allocated CPU 0 ccount 0x34c208f4 caller 0x40376183:0x4037da9c
--- 0x40376183: heap_caps_malloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:84
0x4037da9c: pvPortMalloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/freertos/heap_idf.c:58

    84 bytes (@ 0x3fcb1ee8, Internal) allocated CPU 0 ccount 0x34c219b8 caller 0x40376183:0x4037da9c
--- 0x40376183: heap_caps_malloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/heap/heap_caps.c:84
0x4037da9c: pvPortMalloc at C:/tools/Espressif/frameworks/esp-idf-v5.3.1/components/freertos/heap_idf.c:58

====== Heap Trace Summary ======
Mode: Heap Trace Leaks
664 bytes 'leaked' in trace (13 allocations)
records: 13 (100 capacity, 46 high water mark)
total allocations: 88
total frees: 92
================================
I (5665) main_task: Returned from app_main()

[Espressif-liuuuu]

Hi @lptr , sorry for late reply. The 84bytes leakage, which is allocated by pvPortMalloc, is not likely used by Wi-Fi. You can
configure CONFIG_HEAP_TRACING_STACK_DEPTH to a bigger one, like 16, to see who the caller is.

[lptr]

Hi @Espressif-liuuuu, just to clarify: these logs are coming from the test application you linked to above, not my own application. The test application only does connecting and disconnecting to WiFi, so it should come from there.

I can run with a higher CONFIG_HEAP_TRACING_STACK_DEPTH. But since this code can be run by anyone who has access to an ESP32 MCU, is it actually useful that I do these experiments?

[Espressif-liuuuu]

Hi @Espressif-liuuuu, just to clarify: these logs are coming from the test application you linked to above, not my own application. The test application only does connecting and disconnecting to WiFi, so it should come from there.

I can run with a higher CONFIG_HEAP_TRACING_STACK_DEPTH. But since this code can be run by anyone who has access to an ESP32 MCU, is it actually useful that I do these experiments?

Maybe I confused you, I just hinted his code as an example to show how to use heap_trace_dump to monitor heap leakage, you need to use heap_trace API in your own codes to monitor those leak of consecutive wifi_connect and wifi_disconnect.

[lptr]

@Espressif-liuuuu thanks, I understand now what you meant! I'm still confused about how to proceed with this leak, though.

IIUC, there is a memory leak in ESP-IDF when repeatedly connecting/disconnecting WiFi, as shown by the test application. Shouldn't then this issue be reopened, and the leak in ESP-IDF fixed?

Or am I missing something? Is there perhaps that the test application does incorrectly, and some change could allow it to connect/disconnect WiFi repeatedly without leaking memory?

[Espressif-liuuuu]

@Espressif-liuuuu thanks, I understand now what you meant! I'm still confused about how to proceed with this leak, though.

IIUC, there is a memory leak in ESP-IDF when repeatedly connecting/disconnecting WiFi, as shown by the test application. Shouldn't then this issue be reopened, and the leak in ESP-IDF fixed?

Or am I missing something? Is there perhaps that the test application does incorrectly, and some change could allow it to connect/disconnect WiFi repeatedly without leaking memory?

Yes, if there is do some memory leak, we need to fix the issue.

The example I shared, is not official CI test case but a demo by another developer in this issue, and it monitors the memory leak in one-time wifi_init wifi_deinit. I also ran that case on v5.3.1, it showed that there is leak for 616bytes, but they were due to lazy-free in some places, which wont be allocated in next time init. Actually, I ran 10 times wifi_init and wifi_deinit, the memory leak was still 616 bytes, in that case, thats reasonable.

It seems that in your application, the memory are keeping decrease during multiple connect disconnect, and we want to know the leakage place by heap_trace if it does, then we can fix the leakage issue.

[lptr]

Actually, I ran 10 times wifi_init and wifi_deinit, the memory leak was still 616 bytes, in that case, thats reasonable.

Do I understand correctly, that repeated calls to wifi_init() and wifi_deinit() do NOT cause more memory to leak? In that case my leak is indeed not related to this. Thanks for the help!

[Espressif-liuuuu]

Actually, I ran 10 times wifi_init and wifi_deinit, the memory leak was still 616 bytes, in that case, thats reasonable.

Do I understand correctly, that repeated calls to wifi_init() and wifi_deinit() do NOT cause more memory to leak? In that case my leak is indeed not related to this. Thanks for the help!

Yes! That it is.

[lptr]

Hi @Espressif-liuuuu. I've finally managed to reproduce the problem using Arduino: espressif/arduino-esp32#10664. Do you think this is something on the arduino-esp32 side, or is it in esp-idf?

[lptr]

Update: I tried it with Wokwi, and it seems that ESP-IDF indeed doesn't leak any memory upon repeated reconnects: https://wokwi.com/projects/415915821999976449