logging_backends — Log Output Backend Implementations

logging_backends — Log Output Backend Implementations

This module contains the five app-dependent log output backends for the esp3d_log subsystem: SD card, secondary UART, Telnet, WebSocket, and USB CDC (new in a14). For the full system overview (macros, level gating, format, hooks, build configuration), see esp3d_log.md.

---

Context

The esp3d_log component (components/esp3d_log/) is kept dependency-free so it can be included in any build. Backends that depend on app-level services (filesystem, network, UART driver) cannot live inside that component. They live here — one subdirectory per backend under main/modules/log/ — and are conditionally compiled in by main/CMakeLists.txt when the corresponding ESP3D_LOG_BACKEND value is selected.

main/modules/log/
├── sd/
│   └── esp3d_log_sd_backend.cpp        # Backend 1: SD card
├── uart2/
│   └── esp3d_log_uart2_backend.cpp     # Backend 2: secondary UART
├── telnet/
│   └── esp3d_log_telnet_backend.cpp    # Backend 3: reuses esp3dSocketServer
├── websocket/
│   └── esp3d_log_websocket_backend.cpp # Backend 4: reuses esp3dWsServerDataService
└── usb_cdc/
    └── esp3d_log_usb_cdc_backend.cpp   # Backend 5: native USB CDC #1

The dependency-free serial backend (value 0) lives inside the component at components/esp3d_log/backends/esp3d_log_serial.c and is always available.

---

Shared Interface

Every backend implements the same four-function struct defined in components/esp3d_log/esp3d_log_backend.h:

typedef struct {
    void (*init)(void);                       // called once from esp3d_log_init()
    void (*write)(const char *data, int len); // called per formatted line
    void (*flush)(void);                      // called after each write()
    void (*deinit)(void);                     // release resources
} esp3d_log_backend_t;

// Each backend source file provides exactly one of these:
const esp3d_log_backend_t *esp3d_log_get_backend(void);

The core dispatcher (esp3d_log_output() in esp3d_log.c) calls backend->write() for every formatted line, then backend->flush() immediately after. The data pointer passed to write() is a static buffer valid only for the duration of that call — backends must consume it synchronously or copy it.

Compile Guard

All five backends are gated by the same condition used by the core:

#if ESP3D_LOG || ESP3D_X_BENCHMARK
// ... implementation ...
#endif

Backends therefore participate in benchmark builds (ESP3D_X_BENCHMARK) even when general logging is disabled.

---

Shared Design Principles

All five backends follow the same constraints:

Principle	Rationale
Never block the calling task	write() may be called from the LVGL task on Core 1. Any block there delays UI rendering.
Best-effort / drop-on-busy	Retrying or waiting in write() would turn a log call into a flow-control event that can stall real-time tasks. The SD and USB CDC backends buffer lines in a static RAM stream buffer drained by their own low-priority task, and drop (and count) lines when it is full.
No heap allocation	Static initializers, static buffers and tasks, stack-local state only; no malloc() inside write().
Reentrancy guard where needed	Some backends route through services that internally call esp3d_log_*, creating a re-entry path through esp3d_log_output(). A volatile bool s_in_write prevents infinite recursion.
No-op flush() except serial	SD and USB CDC are asynchronous (their writer task flushes). UART2 and WebSocket would have to block on hardware drain. Only the serial backend flushes (fflush(stdout)).

---

Backend 1 — SD Card

File: main/modules/log/sd/esp3d_log_sd_backend.cpp Requires: SD_CARD_SERVICE=ON (enforced by cmake/sanity_check.cmake) Dependency: ESP3DSd via filesystem/esp3d_sd.h, FreeRTOS static stream buffer and task

Rewritten in a14 around one rule from esp3d_log_guide.md: a log client is an observer, it must never disturb the observed system.

Behaviour

• write() never blocks and never touches the SD. It copies the whole line into a static 4 KB RAM stream buffer (SD_LOG_BUF_SIZE), or drops it and counts it when the buffer is full (whole line or nothing: a partial line would corrupt the file). The observer loses, never the system.
• A dedicated writer task drains the buffer (sd_log, 4 KB static stack, priority tskIDLE_PRIORITY + 1, pinned to core 0, off the LVGL core). It wakes every 200 ms (SD_LOG_FLUSH_MS) and writes in chunks of 512 bytes: one open / append / close per flush, not per line, bounded to 8 extra chunks (SD_LOG_MAX_CHUNKS_PER_FLUSH) to keep the SD hold short for the other SD users. The tasks that log (serial RX, gcode host, LVGL) never wait on SD I/O.
• Boot lines are kept. Lines logged before the SD is mounted stay in the buffer (up to its size) and are written as soon as the SD is available.
• Boot marker. The file accumulates boots (append mode): the first flush writes ===== boot (reset reason: <reason>) ===== (esp3d_hal::resetReason()). After a rotation, the new file starts with ===== log continued (boot reset reason: <reason>) =====.
• Dropped lines are reported as [log] N line(s) dropped (log buffer full), only when the data written so far ends a line, so the notice never lands in the middle of a line.
• Rotation instead of truncation. Past SD_LOG_MAX_SIZE (1 MB), /esp3dlog.txt is renamed /esp3dlog.old.txt (the previous old file is replaced) and a new file starts.
• Shared SD. The writer takes the SD only for the few ms of a flush, and simply retries later when accessFS() reports it busy (job running, shared-SD MCU).
• flush() is a no-op (asynchronous by design). init() creates the stream buffer and the task, everything statically allocated (debug-only backend). If the buffer cannot be created, every write() is dropped.
• Single writer / single reader, as FreeRTOS stream buffers require: write() is always called under the esp3d_log_output() mutex, and the writer task is the only reader.

Consequence

During an active job or while the SD is owned elsewhere, lines wait in the buffer instead of being lost, as long as they fit in 4 KB. See shared_sd_mechanism_V2.0.md for the SD arbitration model.

---

Backend 2 — Secondary UART

File: main/modules/log/uart2/esp3d_log_uart2_backend.cpp
Requires: Board board_config.h must define three macros (see below). No board in this repo currently defines them.
Dependency: ESP-IDF driver/uart.h, driver/gpio.h

Required Board Defines

// Must be a UART port genuinely separate from UART_PORT_IDX (the CNC serial link).
#define ESP3D_LOG_UART_PORT_IDX   UART_NUM_1    // e.g. UART_NUM_1 or UART_NUM_2
#define ESP3D_LOG_UART_TX_PIN     GPIO_NUM_xx
#define ESP3D_LOG_UART_RX_PIN     GPIO_NUM_xx
// Optional — defaults to 115200 if omitted:
#define ESP3D_LOG_UART_BAUD_RATE_BPS  115200

If any of the first three are absent, a #error fires at compile time with an explanatory message. There is no silent fallback. Silently reusing the primary UART’s pins would corrupt CNC communication.

Behaviour

• init() installs the UART driver with:
  • TX ring buffer: 512 bytes (ISR-drained — write() copies data in and returns immediately)
  • RX ring buffer: 256 bytes (required by the ESP-IDF driver API; never read — this is a one-way log channel)
  • If any ESP-IDF init call fails, s_initialized stays false and all subsequent write() calls are no-ops.
• write() calls uart_write_bytes(). Safe from any task including LVGL.
• flush() is a no-op — blocking on uart_wait_tx_done() would stall the calling task.
• deinit() calls uart_driver_delete() to release the port.

Testing

Since no board defines the required pins today, verifying the backend compiles requires temporarily adding the three defines to a board’s board_config.h with arbitrary GPIO_NUM_* values, building, then reverting.

---

Backend 3 — Telnet

File: main/modules/log/telnet/esp3d_log_telnet_backend.cpp
Requires: SOCKET_SERVER_SERVICE=ON (enforced by cmake/sanity_check.cmake)
Dependency: ESP3DSocketServer via socket_server/esp3d_socket_server.h, ESP3DClient via esp3d_client.h

Behaviour

• Reuses esp3dSocketServer rather than opening a second TCP server. One log line becomes one ESP3DMessage routed via ESP3DClientType::socket_server, the same path any other socket-bound text takes.
• process() only enqueues. The actual socket write happens later on the socket server’s own task (its direct-send path is reserved to command answers, never these system-origin lines). write() is therefore safe to call from any task, including LVGL.
• Broadcast id (a14 fix). ESP3DClient::newMsg() stamps request_id with a timestamp, and the socket server only sends a message whose id is 0 or matches a client socket, so the log lines were dropped. The backend now sets msg->request_id.id = 0 (broadcast to every attached client).
• No early-boot dependency. The isConnected() guard returns false until a telnet client actually connects. The backend does nothing before that.
• Reentrancy guard (s_in_write). ESP3DSocketServer::process() calls esp3d_log_e() when its send queue is full. Without the guard that error log would re-enter telnet_log_write() through esp3d_log_output(), recursing indefinitely on a persistently full queue.

Build Constraint

SOCKET_SERVER_SERVICE and SOCKET_CLIENT_SERVICE (CNC-over-WiFi TCP) are mutually exclusive. The Telnet backend cannot be used with CNC-over-WiFi variants. See feature_resource_matrix.md for the compatibility table.

---

Backend 4 — WebSocket

File: main/modules/log/websocket/esp3d_log_websocket_backend.cpp
Requires: WIFI_SERVICE=ON, WEB_SERVICES=ON, and WS_SERVER_SERVICE=ON together (enforced by cmake/sanity_check.cmake)
Dependency: ESP3DWsDataService via websocket_server/esp3d_ws_data_service.h, ESP3DClient via esp3d_client.h

Behaviour

• Reuses esp3dWsServerDataService the same way the Telnet backend reuses esp3dSocketServer. One log line becomes one ESP3DMessage routed via ESP3DClientType::websocket_server.
• Call path: process() → BroadcastTxt() → pushMsgTxt() → httpd_ws_send_frame_async(). The final call is the ESP-IDF httpd API designed to be invoked safely from any task — it queues the frame on the httpd server’s own worker. write() is safe from any task, including LVGL.
• No early-boot dependency. The isConnected() guard returns false until a WS client is attached.
• Broadcast id (a14 fix). As for Telnet, msg->request_id.id = 0: a non-zero id made process() push to that (nonexistent) socket. With authentication, broadcast data only goes to authorized /wsdata sockets (see websocket_server.md); process() never calls the authentication service, so the logging task cannot deadlock on it.
• Reentrancy guard (s_in_write), stricter than Telnet’s. process() calls esp3d_log() unconditionally on every single invocation (not just on failure). Without the guard, the very first backend call would immediately re-enter ws_log_write() through esp3d_log_output() and recurse without end.

Build Constraint

No PiBot variant ships WS_SERVER_SERVICE=ON by default — all WiFi variants use SOCKET_CLIENT_SERVICE, which is mutually exclusive with WS_SERVER_SERVICE. Testing this backend requires a temporary custom variant. When building one, also enable WEBUI_SERVER=ON to avoid an unrelated compile gap in esp3d_http_service.cpp that occurs with WS_SERVER_SERVICE=ON + WEBUI_SERVER=OFF.

---

Backend 5 — USB CDC #1 (a14)

File: main/modules/log/usb_cdc/esp3d_log_usb_cdc_backend.cpp Requires: USB_CDC_DEVICE_SERVICE=ON (enforced by cmake/sanity_check.cmake; ESP32-S2 / S3 board with native USB) Dependency: usb_cdc_device driver (usb_cdc_device_connected(), USB_CDC_DEVICE_PORT_LOG), FreeRTOS static stream buffer and task

The native USB device exposes two CDC ports: CDC #0 carries commands (the USB CDC command client), CDC #1 carries the logs. Their sessions are independent. Same design as the SD backend:

• write() never blocks and never touches USB. It copies the line into a static 4 KB stream buffer, or drops and counts it when the buffer is full.
• A low-priority writer task (usb_log, 3 KB static stack, tskIDLE_PRIORITY + 1, core 0) drains the buffer to CDC #1 in 256-byte chunks, waking every 50 ms, only while a terminal holds that port open (DTR). A host that does not read is waited for at most 100 ms per write.
• Boot lines are kept. Lines logged before the port is opened (boot, TinyUSB not installed yet) stay in the buffer up to its size; newer lines are dropped and counted until a terminal reads them. When a terminal opens the port it gets a ===== log (reset reason: <reason>) ===== marker; dropped lines are reported as [log] N line(s) dropped (log buffer full).
• This backend never installs TinyUSB. The global driver is installed by the USB CDC command client, which is why USB_CDC_DEVICE_SERVICE is required. ROM / bootloader output and panic messages are out of reach.
• No recursion: neither write() nor the USB write path calls esp3d_log, so there is no reentrancy guard.

See Hardware_Peripheral_Drivers.md, “Native USB Device”.

---

Reentrancy Guards Compared

Backend	Has guard	Trigger for re-entry
Serial (0)	No	esp_log_write() does not call back into esp3d_log
SD card (1)	No	SD filesystem operations do not call back into esp3d_log
Secondary UART (2)	No	uart_write_bytes() does not call back into esp3d_log
Telnet (3)	Yes (s_in_write)	esp3dSocketServer::process() calls esp3d_log_e() on a full queue
WebSocket (4)	Yes (s_in_write)	esp3dWsServerDataService::process() calls esp3d_log() on every invocation
USB CDC (5)	No	Neither write() nor the USB write path calls esp3d_log

---

The -u esp3d_log_get_backend Linker Flag

components/esp3d_log/CMakeLists.txt adds:

target_link_libraries(${COMPONENT_LIB} INTERFACE "-u esp3d_log_get_backend")

whenever ESP3D_LOG_BACKEND != 0. Do not remove this. Without it, backends 1–5 fail to link with an “undefined reference” error even though the backend .cpp compiles cleanly. Root cause: ESP-IDF scans libmain.a (where these backends reside) before this component’s library, so by the time the reference in esp3d_log.c.obj is seen, the linker has already passed libmain.a and will not revisit it. The -u flag forces the symbol to be treated as “wanted” from the start of the link, pulling the backend object out of libmain.a regardless of scan order — the same idiom used by ESP-IDF’s own components.

Always use a fresh build directory when switching ESP3D_LOG_BACKEND. An incremental build may silently retain the previously compiled backend and report success with no error.

---

Adding a New Backend

1. Create a subdirectory under main/modules/log/ (e.g., main/modules/log/mybackend/).

2. Implement the four functions and the factory, following the pattern of an existing backend:

#include "esp3d_log_backend.h"

#if ESP3D_LOG || ESP3D_X_BENCHMARK

namespace {

void mybackend_init(void)   { /* open resources — or no-op */ }
void mybackend_flush(void)  { /* flush if buffered — or no-op */ }
void mybackend_deinit(void) { /* close/release — or no-op */ }

void mybackend_write(const char *data, int len) {
    if (!data || len <= 0) { return; }
    // Must never block. Drop on any busy condition.
    // Add a reentrancy guard (volatile bool s_in_write) if the
    // destination service internally calls any esp3d_log_* macro.
}

const esp3d_log_backend_t my_backend = {
    .init   = mybackend_init,
    .write  = mybackend_write,
    .flush  = mybackend_flush,
    .deinit = mybackend_deinit,
};

}  // namespace

extern "C" const esp3d_log_backend_t *esp3d_log_get_backend(void) {
    return &my_backend;
}

#endif

3. Assign a numeric value (e.g., 6) and add a prerequisite check to cmake/sanity_check.cmake for any required CMake service flags.

4. Register it in main/CMakeLists.txt under the ESP3D_LOG_BACKEND conditional block so exactly one backend subdirectory is ever added to the build.

5. Document it in cmake/dev_tools.cmake (the comment listing backends) and in esp3d_log.md (the backend table in §4).

---

Integration Points

Module	Relationship
esp3d_log.md	Parent system: macros, format pipeline, mutex, hook mechanism that call into these backends
components/esp3d_log/esp3d_log_backend.h	Defines esp3d_log_backend_t — the interface all backends implement
components/esp3d_log/esp3d_log.c	Calls esp3d_log_get_backend() at init; calls write() and flush() on every log event
main/modules/filesystem/esp3d_sd.h	SD backend (1) uses ESP3DSd::accessFS() / releaseFS() — see shared_sd_mechanism_V2.0.md
socket_server.md	Telnet backend (3) depends on esp3dSocketServer
websocket_server.md	WebSocket backend (4) depends on esp3dWsServerDataService
cmake/dev_tools.cmake	Sets ESP3D_LOG_BACKEND (default: 0); comment enumerates all valid values
cmake/sanity_check.cmake	Enforces service prerequisites for backends 1, 3, 4, and 5
usb_cdc_device driver	USB CDC backend (5) writes to CDC #1; see Hardware_Peripheral_Drivers.md
main/CMakeLists.txt	Adds exactly one backend subdirectory based on ESP3D_LOG_BACKEND