High memory consumption on 2.0.0rc1 core

[ale-trevizoli]

Hi.
I'm using esp32-dev board 4Mb Flash.
I have a system developed with Arduino IDE 1.8.5 and ESP32 core 1.0.6.
This system print on loop() - ESP.getFreeHeap() each 5 seconds, and show me ~302K free.
I exchange 1.0.6 to 2.0.0rc1 and I only compiled again and got ~265K free, difference of 37Kbytes.
What's can happening?

Thanks!

[SuGlider]

I tested the sketch below and got about 343K free.

void setup() {
  Serial.begin(115200);
}

void loop() {
  Serial.println((unsigned long)ESP.getFreeHeap());
  delay(5000);
}

[avillacis]

This issue of additional memory use persists in recently-released 2.0.0 stable versus 1.0.6. In my case, the difference is enough to have 1.0.6 stashed alongside 2.0.0 in case one of our projects fails due to memory starvation.

For example, consider the following sketch:

WiFiScan-ESP32S2.txt

This sketch compiles properly under 1.0.6 and 2.0.0. However, note the difference in memory use, especially after WiFi is set up:

Under 1.0.6:

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:1216
ho 0 tail 12 room 4
load:0x40078000,len:10944
load:0x40080400,len:6388
entry 0x400806b4
MEMORY USE AT setup() before WiFi setup: 360808 total, 334340 free, 113792 max.alloc
Setup done
MEMORY USE AT setup() after WiFi setup: 359568 total, 283652 free, 113792 max.alloc
scan start
scan done
12 networks found
(redacted)

MEMORY USE AT after WiFi scan: 359484 total, 282476 free, 113792 max.alloc
scan start
scan done
17 networks found
(redacted)

MEMORY USE AT after WiFi scan: 359484 total, 282076 free, 113792 max.alloc

Under 2.0.0:

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1240
load:0x40078000,len:13012
load:0x40080400,len:3648
entry 0x400805f8
MEMORY USE AT setup() before WiFi setup: 366171 total, 308455 free, 65524 max.alloc
Setup done
MEMORY USE AT setup() after WiFi setup: 364203 total, 228051 free, 65524 max.alloc
scan start
scan done
14 networks found
(redacted)

MEMORY USE AT after WiFi scan: 364179 total, 226875 free, 65524 max.alloc
scan start
scan done
13 networks found
(redacted)

MEMORY USE AT after WiFi scan: 364179 total, 226955 free, 65524 max.alloc

By switching SDK releases and doing nothing else, the free memory has decreased from 282076 to 226955 (an unexplained additional 55121 bytes), and the maximum allocatable block has decreased from 113792 to 65524 bytes.

Why????

[SuGlider]

1.0.6 uses IDF 3.3
2.0.0 uses a RC for IDF 4.4
WiFi is completely based on IDF.

I guess this is the reason for a higher RAM consumption.

[SuGlider]

Maybe, in the near future, on the final release of IDF 4.4 and respective rebase of Arduino Core to this IDF, it may improve.

No promises....

[me-no-dev]

there are many reasons why there is a different memory usage in recent IDF. This is in NO WAY connected to Arduino. Arduino's is mostly disabling some IRAM optimizations, because users have sketches with WiFi, BT and all kinds of other things that make compilation impossible. That change though actually gives you memory. Do you actually have memory issues?

[avillacis]

there are many reasons why there is a different memory usage in recent IDF. This is in NO WAY connected to Arduino. Arduino's is mostly disabling some IRAM optimizations, because users have sketches with WiFi, BT and all kinds of other things that make compilation impossible.

Maybe some additional IRAM optimization was overlooked?

That change though actually gives you memory. Do you actually have memory issues?

Yes, I do have memory issues with some of my projects that were absent when compiled under 1.0.6. At least one of our projects (the firmware for the OpenVenti artificial respirator project) is unable to complete its OTA firmware update procedure due to insufficient RAM under 2.0.0, where the same source code worked successfully with 1.0.6. This has forced me to look for and remove RAM inefficiencies in the process, but the fact remains that 55Kb extra RAM use is currently a showstopper for updating to 2.0.0, at least for this project.

[igrr]

Here is some additional data based on IDF example, examples/wifi/getting-started/station:

IDF release	sdkconfig	Heap [1] after app_main	Heap [1] after esp_wifi_start	Heap used
IDF master (v4.4-dev-2890-ge93cee4605)	from arduino-esp32 2.0.0 [2]	326435	251415	75kB
IDF master (v4.4-dev-2890-ge93cee4605)	default	323884	284048	40kB
IDF release/v3.3 (v3.3.4-404-g7e63061fae)	from arduino-esp32 1.0.6 [3]	337720	291300	46kB
IDF release/v3.3 (v3.3.4-404-g7e63061fae)	default	327232	290864	36kB

First, note that with the default sdkconfig, IDF 4.4-dev has about 7kB less heap at app_main than IDF release/3.3. See the note [1] below, though. These numbers include "IRAM" heap; if we compare only DRAM heap (MALLOC_CAP_DEFAULT), then v4.4-dev has 2kB more heap than 3.3 (277032 bytes free in v4.4-dev, 275824 free in release/v3.3) when this example starts up — see the table below.

We can see that some RAM can be reclaimed by changing the configuration (sdkconfig) that arduino-esp32 is built with. The sdkconfig used in 2.0.0 results in 35kB higher RAM usage than the default sdkconfig (for the same IDF version). In 1.0.6 this was only 10kB higher.

For the rest of the difference between 1.0.6 and 2.0.0 heap usage we should look at the startup heap size: 334340 vs 308455 (26kB difference). This might be due to higher static memory usage (.data, .bss) and/or higher heap usage. The issue might be on IDF side or Arduino-esp32 side, it is hard to say without investigating it in detail. For the static memory usage, idf.py size-components (or idf_size.py) can provide per-library breakdown of static data — including libarduino.a. For the dynamic memory usage, we can enable heap tracing to find the allocations performed at startup (before setup is called). I haven't had time to look into this part yet, but I suspect that there is room for improvement.

---

Note 1: note that arduino-esp32 ESP.getFreeHeap() function returns the amount of internal DRAM and IRAM; the latter is not available via malloc, so the number is somewhat higher than the useful amount of heap memory. See #5346. To make comparisons easier, i used the same way of measuring the heap size in IDF examples, even though that's not an entirely accurate approach.

Here is the version of the table above, but this time showing only the DRAM heap (MALLOC_CAP_DEFAULT). Note that "Heap used" (difference) is the same as above, just the initial number in each row is less by the amount of free IRAM in the particular configuration:

IDF release	sdkconfig	Heap after app_main	Heap after esp_wifi_start	Heap used
IDF master (v4.4-dev-2890-ge93cee4605)	from arduino-esp32 2.0.0 [2]	239484	164040	75kB
IDF master (v4.4-dev-2890-ge93cee4605)	default	277032	237432	40kB
IDF release/v3.3 (v3.3.4-404-g7e63061fae)	from arduino-esp32 1.0.6 [3]	272068	225116	47kB
IDF release/v3.3 (v3.3.4-404-g7e63061fae)	default	275824	239288	36kB

Note 2: https://raw.githubusercontent.com/espressif/arduino-esp32/2.0.0/tools/sdk/esp32/sdkconfig
Note 3: https://raw.githubusercontent.com/espressif/arduino-esp32/1.0.6/tools/sdk/sdkconfig

[SuGlider]

@ale-trevizoli
I'm working on this issue and I was able to get the following results using the same sketch you have posted here as an example, with a customized version of Arduino Core 2.0.0:

a) bin size of 632405 bytes
b) Heap:

SDK: v4.4-dev-3401-gb86fe0c66c
MEMORY USE AT setup() before WiFi setup: 332796 total, 308416 free, 65524 max.alloc - max.alloc internal 65524 
Setup done
MEMORY USE AT setup() after WiFi setup: 331164 total, 275492 free, 65524 max.alloc - max.alloc internal 65524
scan start
scan done
21 networks found

MEMORY USE AT after WiFi scan: 331140 total, 273756 free, 65524 max.alloc - max.alloc internal 65524

I created a repository with a new Arduino Core 2.0.0 that can be tested.
https://github.com/espressif/arduino-esp32/tree/mem-optimized

In order to install it as a separated board for testing, please follow the instructions from
https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html
The only difference here would be to clone mem-optimized branch instead.

git clone -b mem-optimized https://github.com/espressif/arduino-esp32 esp32

In case it works fine for a number of users, we will commit it for the next ESP32 Arduino release.
Please test it with your project and let me know.

[olicooper]

Memory consumption is GREAT! ... but I have wifi issues using the mem-optimized branch. I am using the new esp-idf mqtt functionality (see this) which may be a contributing factor?
The logs don't seem like they will help much, but I can see the line beginning [ 8716][V] has a BSSID of 00:00:00:00:00:00 - I guess this is because it is no longer connected to an AP?

My code worked fine on Arduino v1.0.6 and Arduino v2.0.0 master branch

I have my own logging facility which is why there is additional log output below.

environment

• Heltec Wireless Stick
• Network is using WPA2-PSK (CCMP-128)
• WiFi library is configured with mostly default settings (dynamic IP) and it is close to the AP

additional config

// I use onEvent
WiFi.onEvent([&](arduino_event_id_t event, arduino_event_info_t info) { /* ... */ });
// I only use STA
WiFi.mode(WIFI_MODE_STA);
// I save wifi details elsewhere
WiFi.persistent(false);
// I manually reconnect in my loop
WiFi.setAutoReconnect(false);
// I make the initial connection later on by calling esp_wifi_connect();
WiFi.begin(ssid, password, 0 NULL, false);

logs

[   401][D][WiFiGeneric.cpp:820] _eventCallback(): Arduino Event: 0 - WIFI_READY
[   499][V][WiFiGeneric.cpp:272] _arduino_event_cb(): STA Started
[   499][D][WiFiGeneric.cpp:820] _eventCallback(): Arduino Event: 2 - STA_START
2021-10-15T18:18:44 [V][mqtt]: WiFi ssid=[MySsid]
2021-10-15T18:18:44 [D][mqtt]: WiFi connecting
[   527][V][WiFiGeneric.cpp:96] set_esp_interface_ip(): Configuring Station static IP: 0.0.0.0, MASK: 0.0.0.0, GW: 0.0.0.0
2021-10-15T18:18:44 [W][mqtt]: WiFi failed to connect
2021-10-15T18:18:44 [I][mqtt]: Service enabled
2021-10-15T18:18:44 [D][core]: Free memory: 162784 bytes
2021-10-15T18:18:44 [A][core]: System started
2021-10-15T18:18:45 [V][store]: Closed
[  6627][V][WiFiGeneric.cpp:289] _arduino_event_cb(): STA Disconnected: SSID: MySsid, BSSID: ab:cd:ef:01:02:ea, Reason: 2
[  6628][D][WiFiGeneric.cpp:820] _eventCallback(): Arduino Event: 5 - STA_DISCONNECTED
[  6635][W][WiFiGeneric.cpp:841] _eventCallback(): Reason: 2 - AUTH_EXPIRE
2021-10-15T18:18:50 [V][mqtt]: Connect timer expired
2021-10-15T18:18:50 [V][mqtt]: WiFi reconnecting
2021-10-15T18:18:50 [D][mqtt]: notifyEventSubscribers: disconnected
[  8716][V][WiFiGeneric.cpp:289] _arduino_event_cb(): STA Disconnected: SSID: MySsid, BSSID: 00:00:00:00:00:00, Reason: 205
[  8716][D][WiFiGeneric.cpp:820] _eventCallback(): Arduino Event: 5 - STA_DISCONNECTED
[  8724][W][WiFiGeneric.cpp:841] _eventCallback(): Reason: 205 - CONNECTION_FAIL
2021-10-15T18:18:52 [V][mqtt]: Connect timer expired
2021-10-15T18:18:52 [V][mqtt]: WiFi reconnecting
2021-10-15T18:18:52 [D][mqtt]: notifyEventSubscribers: disconnected
[ 14857][V][WiFiGeneric.cpp:289] _arduino_event_cb(): STA Disconnected: SSID: MySsid, BSSID: ab:cd:ef:01:02:ea, Reason: 4
[ 14858][D][WiFiGeneric.cpp:820] _eventCallback(): Arduino Event: 5 - STA_DISCONNECTED
[ 14865][W][WiFiGeneric.cpp:841] _eventCallback(): Reason: 4 - ASSOC_EXPIRE
[ 14900][V][WiFiGeneric.cpp:275] _arduino_event_cb(): STA Stopped
2021-10-15T18:18:58 [D][mqtt]: notifyEventSubscribers: disconnected
2021-10-15T18:18:58 [D][mqtt]: WiFi disconnected: r=4, WiFi disabled
[ 14933][D][WiFiGeneric.cpp:820] _eventCallback(): Arduino Event: 3 - STA_STOP

[chegewara]

Reason: 2 - AUTH_EXPIRE
This is normal when you are using smartphone as AP (maybe with routers too) during development.
When esp32 is connected and you re-flash it then smartphone keeps this connection active, even if esp32 disconnected. Then after reset esp32 is trying to connect again, but smartphone still wants to update old connection.
I dont know how it is handled in low level wifi packets exchange, just saying from experience.
There is 2 options that work for me:

1. during re-flash disable and re-enable AP,
2. or after re-flash esp32 and that error occured just restart esp32 with pwr button