AsyncAbstractResponse truncates 85 octets in the non-empty segment

[yoursunny]

Platform

ESP32

IDE / Tooling

Arduino (IDE/CLI)

Versions

esp32 v3.3.2
Async TCP v3.4.9
ESP Async WebServer v3.8.1

What happened?

My sketch uses a AsyncAbstractResponse subclass where _sendContentLength is disabled and the _fillBuffer function has the following behavior:

1. The first call returns RESPONSE_TRY_AGAIN.
2. The second call fully fills the provided buffer with letter "B" and returns buflen.
3. The third call fully fills the provided buffer with letter "C" and returns buflen.
4. The fourth call returns 0 to end the response.

With the sketch running on either ESP32 or ESP32C3, I invoke wget command to trigger the handler.
Downloaded file: 1.txt

The console log pasted below indicates that the sketch has filled 5760 "B"s and 4308 "C"s in the response.
However, the downloaded file has 5675 "B"s and 4308 "C"s.
In other words, 85 octets are truncated off the buffer filled in the second call.

Stack Trace

WiFi connected
http://192.168.5.67
=========== After Setup Start ============
INTERNAL Memory Info:
------------------------------------------
  Total Size        :   301688 B ( 294.6 KB)
  Free Bytes        :   191032 B ( 186.6 KB)
  Allocated Bytes   :   103632 B ( 101.2 KB)
  Minimum Free Bytes:   190772 B ( 186.3 KB)
  Largest Free Block:   114676 B ( 112.0 KB)
------------------------------------------
GPIO Info:
------------------------------------------
  GPIO : BUS_TYPE[bus/unit][chan]
  --------------------------------------  
    20 : UART_RX[0]
    21 : UART_TX[0]
============ After Setup End =============
_fillBuffer ch=A buflen=5675
_fillBuffer ch=B buflen=5760
_fillBuffer ch=C buflen=4308
_fillBuffer ch=D buflen=4324

Wireshark packet trace: 1.pcapng.gz

It may or may not be a coincidence, but I noticed:

• The HTTP status line and response headers have exactly 85 octets.
• The buflen passed to the first _fillBuffer call is 85 larger than the buflen passed to the second _fillBuffer call.

Minimal Reproductible Example (MRE)

#include <ESPAsyncWebServer.h>
#include <WiFi.h>

static const char* WIFI_SSID = "my-ssid";
static const char* WIFI_PASS = "my-pass";

AsyncWebServer server(80);

class MyResponse : public AsyncAbstractResponse {
public:
  explicit MyResponse() {
    _code = 200;
    _contentType = "text/plain";
    _sendContentLength = false;
  }

  bool _sourceValid() const override {
    return true;
  }

  size_t _fillBuffer(uint8_t* buf, size_t buflen) override {
    Serial.printf("_fillBuffer ch=%c buflen=%zu\n", m_ch, buflen);

    if (m_ch == 'A') {
      ++m_ch;
      return RESPONSE_TRY_AGAIN;
    }
    if (m_ch == 'D') {
      return 0;
    }

    std::fill_n(buf, buflen, static_cast<uint8_t>(m_ch));
    ++m_ch;
    return buflen;
  }

private:
  char m_ch = 'A';
};

void
setup() {
  Serial.begin(115200);
  Serial.println();
  delay(1000);

  WiFi.persistent(false);
  WiFi.mode(WIFI_STA);
  WiFi.begin(WIFI_SSID, WIFI_PASS);
  if (WiFi.waitForConnectResult() != WL_CONNECTED) {
    Serial.printf("WiFi failure %d\n", WiFi.status());
    delay(5000);
    ESP.restart();
  }
  Serial.println("WiFi connected");
  delay(1000);

  Serial.print("http://");
  Serial.println(WiFi.localIP());

  server.on("/1.txt", HTTP_GET, [](AsyncWebServerRequest* req) {
    req->send(new MyResponse());
  });
  server.begin();
}

void
loop() {
  delay(1);
}

I confirm that:

• I have read the documentation.
• I have searched for similar discussions.
• I have searched for similar issues.
• I have looked at the examples.
• I have upgraded to the lasted version of ESPAsyncWebServer (and AsyncTCP for ESP32).

[vortigont]

It's a bug indeed, I'll take it.

[mathieucarbou]

@vortigont I agree...

Line 448 of WebResponse.cpp maybe ?

[vortigont]

@yoursunny thanks for the report, nice catch! Pls check the fix, it works for me for you cam project.

NOTE:
you should not do this The first call returns RESPONSE_TRY_AGAIN, it would delay you data transfer for 1 RTT at least.

P.S. unfortunately this AbstractResponse is not the best fit for sending large chunks of data due to intermediate buffering. If you are interested in a way more efficient approach to use AsyncServer for cam streaming you can check for my recent experimental websocks in #312. There is an example specifically made for esp32 cam, it works much better with less memory footprint even without a dedicated RTOS task to run a cam.

[mathieucarbou]

@yoursunny @vortigont : FYI I added an example in the project to show how to send large responses:

#317

Sadly I am not able to reproduce the problem through these examples and with the code in main.

Is one of you capable of that (using main branch) ? If yes, could you please share your MRE ?

[vortigont]

yes, it reproduces easily with esp32-cam, and works fine with my fix.
Will try to play with your minimal example, @mathieucarbou, must be some corner case