ESP32-S3 SCTP Crash With Data_channel Disabled

Experiencing crashes in your ESP32-S3 project when data channels are disabled? This article dives deep into a specific crash occurring in the peer_default SCTP implementation, focusing on the sctp_add_ref function and the underlying __atomic_fetch_add_1 operation. We'll explore the root cause, provide steps to reproduce the issue, and offer potential solutions to get your project back on track.

Understanding the Crash

When working with the ESP32-S3 and the esp-webrtc-solution library, you might encounter a perplexing crash within the peer_default SCTP implementation. This crash manifests itself during the execution of sctp_add_ref, specifically within the __atomic_fetch_add_1 function. The error typically arises when data channels are explicitly disabled (enable_data_channel = false) in your configuration. This crash often presents as a Guru Meditation Error with a LoadProhibited exception, pinpointing an invalid memory address.

The root cause of this issue appears to be related to an invalid pointer being used for reference counting within the SCTP context. The EXCVADDR in the error logs often points to a low memory address (e.g., 0x000005ac), strongly suggesting that the SCTP context or refcount pointer is corrupted or uninitialized. This leads to a crash when __atomic_fetch_add_1 attempts to increment the reference count at this invalid address.

Key Symptoms:

• Crashes occur specifically within sctp_add_ref during __atomic_fetch_add_1.
• The EXCVADDR points to a low, invalid memory address.
• The crash only happens when enable_data_channel is set to false.
• Disabling data channels is intended to reduce overhead when they are not needed, but this crash makes that impossible.

Reproducing the Crash: Step-by-Step

To effectively troubleshoot this issue, reproducing it consistently is crucial. Here's a detailed breakdown of the steps to replicate the crash:

1. Environment Setup: Begin by setting up your development environment with the ESP-IDF v5.4.3 and target the ESP32-S3.

2. esp-webrtc-solution: Integrate the esp-webrtc-solution library into your project. Ensure you're using a version that includes the peer_default implementation.

3. Configuration: This is the most critical step. Configure your esp_webrtc_cfg_t structure as follows:

   • Set esp_webrtc_cfg_t.peer_cfg.enable_data_channel = false. This is the key trigger for the crash.
   • Use the default peer implementation: peer_impl = esp_peer_get_default_impl().
   • Employ a signaling mechanism that negotiates SCTP/data channels on the remote side. This can be WHIP (using esp_signaling_get_whip_impl()) or a custom signaling implementation like intercom WebRTC signaling (esp_signaling_get_intercom_session_impl()).

4. Server Interaction: Connect your ESP32-S3 device to a server that attempts to establish data channels. This is important because the crash occurs during the SCTP negotiation and data flow.

5. Observe the Crash: Once the connection is established and media/data exchange begins, the ESP32-S3 device should crash with the LoadProhibited exception described earlier.

Simplified Configuration Example:

The following code snippet demonstrates a simplified configuration that triggers the crash:

esp_webrtc_cfg_t cfg = {
    .peer_cfg = {
        .audio_info = {
            .codec       = ESP_PEER_AUDIO_CODEC_OPUS,
            .sample_rate = 16000,
            .channel     = 1,
        },
        .video_info = {
            .codec  = ESP_PEER_VIDEO_CODEC_NONE,
            .width  = 0,
            .height = 0,
            .fps    = 0,
        },
        .audio_dir          = ESP_PEER_MEDIA_DIR_SEND_RECV,
        .video_dir          = ESP_PEER_MEDIA_DIR_NONE,
        .enable_data_channel = false,  // <-- CRITICAL: Triggers the crash
        .no_auto_reconnect   = true,
    },
    .signaling_cfg = {
        .signal_url = base_url,
        .extra_cfg  = &session_cfg,
        .extra_size = sizeof(session_cfg),
    },
    .peer_impl      = esp_peer_get_default_impl(),
    .signaling_impl = esp_signaling_get_intercom_session_impl(),
};

Important Observation:

A critical observation is that simply changing .enable_data_channel = true (while keeping all other configurations the same) makes the crash disappear. This strongly indicates that the issue is directly related to the handling of SCTP contexts when data channels are disabled.

Analyzing the Error Logs and Backtrace

The error logs provide valuable clues to pinpoint the location and nature of the crash. Here's a breakdown of the key information to look for:

• Guru Meditation Error: This is the general indicator of a fatal error in the ESP32-S3 firmware.
• Core 1 panic'ed (LoadProhibited): This specifies the type of exception, indicating that the code attempted to load data from an invalid memory address.
• PC (Program Counter): The PC value (e.g., 0x4038ae58) points to the instruction being executed when the crash occurred. In this case, it's within __atomic_s32c1i_fetch_add_1.
• EXCCAUSE: 0x0000001c: This is the exception cause, further confirming the LoadProhibited error.
• EXCVADDR: 0x000005ac: This is the crucial piece of information. The EXCVADDR represents the invalid memory address that the code tried to access. A low address like 0x000005ac strongly suggests a null or dangling pointer.
• Backtrace: The backtrace provides a call stack, showing the sequence of function calls that led to the crash. This is invaluable for tracing the execution flow and identifying the source of the error.

Example Backtrace Analysis:

Backtrace:
 0x4038ae55: __atomic_s32c1i_fetch_add_1 (stdatomic_s32c1i.c:77)
 0x4038aa65: __atomic_fetch_add_1        (stdatomic.c:31)
 0x4201a70b: sctp_add_ref                (/home/tempo/test/esp-webrtc/components/esp_webrtc/impl/peer_default/sctp/sctp.c:43)
 0x4201c6c0: sctp_incoming_data          (.../sctp/sctp.c:580)
 0x420153d0: peer_recv_streams           (.../peer_default/core/peer_default.c:1132)
 0x420154fe: peer_main_loop              (.../peer_default/core/peer_default.c:1223)
 0x4216d5c3: esp_peer_main_loop          (components/esp_peer/src/esp_peer.c:140)
 0x42011c46: pc_task                     (components/esp_webrtc/src/esp_webrtc.c:286)
 0x403823e5: vPortTaskWrapper            (.../freertos/portable/xtensa/port.c:139)

In this backtrace, we can see that the crash originates in __atomic_s32c1i_fetch_add_1, which is called by __atomic_fetch_add_1. This function is then called by sctp_add_ref, confirming that the crash occurs during the reference counting operation within the SCTP implementation. The subsequent functions in the backtrace (sctp_incoming_data, peer_recv_streams, etc.) indicate the flow of data and processing leading up to the crash.

Key Takeaway:

The backtrace clearly shows that the crash occurs within the sctp_add_ref function when attempting to increment a reference count using atomic operations. The invalid EXCVADDR points to a likely cause: a corrupted or uninitialized SCTP context pointer.

Potential Causes and Solutions

Based on the error analysis and reproduction steps, several potential causes for this crash emerge:

1. Uninitialized SCTP Context: When data channels are disabled, the SCTP context might not be properly initialized or allocated. This could lead to a null or dangling pointer being used in sctp_add_ref.

   • Solution: Ensure that the SCTP context is correctly initialized even when data channels are disabled. Review the code paths that handle SCTP context creation and initialization, paying close attention to the conditions when enable_data_channel is false.

2. Premature Context Release: The SCTP context might be released prematurely when data channels are disabled. This could leave a dangling pointer that is later accessed by sctp_add_ref.

   • Solution: Carefully examine the SCTP context lifecycle. Ensure that the context is not released before it's no longer needed. Pay particular attention to any cleanup or release operations that are conditionally executed based on the enable_data_channel setting.

3. Race Condition: A race condition could occur if multiple threads or tasks are accessing the SCTP context concurrently, especially during initialization or release. This could lead to corruption of the context pointer.

   • Solution: Implement proper synchronization mechanisms (e.g., mutexes, semaphores) to protect the SCTP context from concurrent access. Identify the critical sections of code that access the context and ensure that they are properly synchronized.

4. Prebuilt Library Issue: The issue might reside within the prebuilt libpeer_default.a library itself. There could be a bug in the SCTP implementation that is triggered when data channels are disabled.

   • Solution: If you suspect an issue within the prebuilt library, consider building the peer_default implementation from source. This will allow you to debug the code directly and identify any potential bugs. You can also try using a different version of the esp-webrtc-solution library or the ESP-IDF to see if the issue has been resolved in a newer version.

5. Compiler Optimization: Aggressive compiler optimizations might sometimes lead to unexpected behavior, especially when dealing with pointers and memory management.

   • Solution: Try disabling compiler optimizations or adjusting the optimization level to see if it resolves the issue. This can help identify if the compiler is inadvertently introducing a bug.

Debugging Strategies

To effectively debug this crash, several strategies can be employed:

1. Logging: Add extensive logging statements throughout the SCTP-related code, especially in sctp_add_ref and the functions that manage the SCTP context lifecycle. Log the value of the context pointer, the reference count, and any relevant state information. This will help you track the flow of execution and identify when the context pointer becomes invalid.

2. GDB Debugging: Use GDB to step through the code and examine the values of variables and memory locations. Set breakpoints in sctp_add_ref and the surrounding code to inspect the SCTP context and the reference count. This allows for a detailed analysis of the crash.

3. Memory Corruption Detection: Employ tools like memory sanitizers (e.g., AddressSanitizer) to detect memory corruption issues. These tools can help identify memory leaks, buffer overflows, and other memory-related errors that might be contributing to the crash.

4. Code Review: Conduct a thorough code review of the SCTP-related code, paying close attention to memory management, pointer usage, and synchronization. Look for potential errors or vulnerabilities that could be causing the crash.

Conclusion

The crash in peer_default SCTP when data channels are disabled is a complex issue that requires careful analysis and debugging. By understanding the error logs, reproducing the crash, and systematically investigating potential causes, you can effectively troubleshoot and resolve this problem. Remember to focus on SCTP context initialization, lifecycle management, and thread synchronization. If you suspect a bug in the prebuilt library, consider building from source or trying a different version. By applying these strategies, you can ensure the stability and reliability of your ESP32-S3 WebRTC applications.

For further information on WebRTC and related topics, you can visit the WebRTC official website.