Build & Development Tools

Build & Development Tools

Purpose

The Build_&_Development_Tools module is the host-side toolchain supporting the full firmware development lifecycle for the Pibot CNC Pendant. It covers multi-board/multi-variant ESP-IDF build orchestration, UI resource generation and customization, PC-side transport debugging clients, firmware simulators for testing without real CNC hardware, asset conversion utilities (fonts, images, language packs), and an embedded Lua scripting engine component. All scripts run on the developer’s workstation — nothing in this module is flashed to the ESP32.

---

Architecture

---

Sub-modules

1. Build Management (tools/build_scripts/)

The core build orchestration layer. build_mgr.py discovers all board/variant pairs from boards/*/build_scripts/variants.py, delegates to per-board build_one.py scripts, collects binary size metrics, and produces cross-variant summaries. validate_build_scripts.py enforces consistency between board build scripts and the root CMakeLists.txt before every build.

Script	Role
build_mgr.py	Top-level orchestrator; interactive and CLI modes
validate_build_scripts.py	Cross-checks CMake option declarations
size_report.py	Parses idf.py size --format json per variant
aggregate_size_summary.py	Produces aligned DRAM/IRAM/Flash table

2. UI Resources Pipeline (tools/build_scripts/)

Generates the ui_resources flash partition containing LVGL-format images, serialized font blobs, theme palette data, and language packs. resources_config.py is the single source of truth for all assets; generate_resources.py runs at firmware build time and emits the C headers consumed by the firmware (esp3d_ui_offsets.h, esp3d_theme_palettes.h, esp3d_lang_packs.h, and since a14 esp3d_sound_defaults.h for the sound module). Since a14 the partition binary also ends with a 96-byte build trailer (variant + firmware version) checked by the SD-card update; see tools_build_scripts_ui_resources.md.

Script	Role
resources_config.py	Declares every ImageConfig and FontConfig entry
generate_resources.py	Converts PNGs and font .c files; assembles partition binary + C headers
generate_user_resources.py	Rebuilds ui_resources.bin with user overrides for SD-card flashing
generate_single_resource.py	Produces single-asset SD-card patch blobs (.bin / .fnt)
build_ui_resources_from_manifest.py	Repo-independent standalone builder using only a manifest JSON
package_user_resources_kit.py	Assembles a self-contained distributable ui_resources_kit/

3. Utility Scripts (tools/build_scripts/)

Script	Role
gen_dlc32_split_sdkconfig.py	Transforms shared DLC32 sdkconfig into transport-specific variants
gen_ota_initial.py	Generates the 8 KB ota_data_initial.bin for first-boot partition selection
png_to_lvgl_c.py	Converts PNGs to LVGL v9 .c image sources for the statically-compiled icon set

3b. Flash Manager (tools/flash_scripts/)

flash_mgr.py generates the flash map JSON of a variant (--generate, called at the end of each board build) and flashes a variant from installer/ with esptool.

Mode	Command
Interactive	python flash_mgr.py
List variants	python flash_mgr.py --list
Flash	python flash_mgr.py --variant NAME --port COM17 [--baud 921600] [--erase-flash]
Generate the flash map	python flash_mgr.py --variant NAME --generate [--flash-params PATH] or --variant-dir DIR --generate

• Flash offsets come from the variant’s real partition table (partitions_<N>mb.bin parsed next to the other files); the hardcoded PiBot-layout fallbacks are only used when no table is found.
• The bootloader offset depends on the chip (0x1000 on ESP32 / S2, 0x0 on S3 / C3 / C6 / H2, 0x2000 on C5 / H4 / P4).
• Interactive wizard (a14): variant, port, baud and erase are asked in turn, with the last saved choices as defaults (.flash_mgr_prefs.json). Every step after the first has a B) Back entry. A summary is always shown last: Enter starts, a number changes that step only, B goes back to the last step, Q quits.
• The .bat helpers (eraseflash.bat, flash4mb.bat, flash8mb.bat, dump.bat, …) are local shortcuts; eraseflash.bat now uses --chip auto.

4. UI Studio (tools/ui_studio/)

A local Flask web application providing a browser-based GUI for customizing the three UI resource types: theme colors (26 semantic tokens), images (per-icon PNG overrides), and fonts (TTF conversion or .c replacement). Produces esp3dtheme.ini, individual .bin/.fnt SD patches, or a full ui_resources.bin rebuild by shelling out to the bundled kit scripts.

5. Firmware Simulator (tools/fw_simulator/)

PC-side serial simulator that impersonates CNC and 3D-printer firmware over a physical or virtual UART, enabling end-to-end testing of the pendant’s GCode dispatch and UI flows without real CNC hardware. Supports FluidNC (6-axis, Grbl real-time status), Marlin, Repetier, and Smoothieware with temperature physics simulation.

6. Communication Clients (tools/)

Standalone debug terminals mirroring every pendant transport channel:

Tool	Transport	Target firmware module
serial_bridge.py	UART (two-port relay)	ESP3DSerialClient
fluidnc_serial.py	UART (single port)	ESP3DSerialClient
fluidnc_ble.py	BLE GATT 0xFFF0	ESP3DBTBleClient
fluidnc_bt.py / grblhal_bt.py	BT Classic SPP	ESP3DBTSerialClient
grblhal_telnet.py	TCP/Telnet	ESP3DSocketClient
ws_transfer_test.py	WebSocket V1 /wsdata	ESP3DWebsocketClient

7. Resource Generation (tools/)

Offline conversion and auditing tools for fonts, images, and language packs:

• tools/fonts/ — ttf_to_fnt.py (pure-Python TTF→.fnt via FreeType + fontTools) and font_c_to_fnt.py (lv_font_conv .c → .fnt cross-check)
• tools/images/ — image_c_to_bin.py (LVGL image .c → .bin reference converter)
• tools/images_converter/ — raw2png.py / raws2pngs.py (ESP32 screen snapshot → PNG)
• tools/language_packs/ — build_template.py (generate template.lng from *_defs.inc macros) and audit_translations.py (detect unused/undefined ESP3DLabel:: references across the codebase)

8. Lua Engine (components/EspLuaEngine/)

An embedded Lua 5.4.7 runtime component with a C++ wrapper (EspLuaEngine) for scripting support on the ESP32. Includes a cleanLua.py host-side maintenance script. This is a firmware component; the host tooling aspect is limited to the maintenance script and build integration.

---

9. CMake Configuration (CMakeLists.txt, cmake/) — a14 changes

File	Change
CMakeLists.txt	New board options ESP32_S3_TOUCH_LCD_1_47, ESP32_S3_LCD_1_47B (16 MB + 8 MB PSRAM, small status displays), BOARD_DISPLAY_TEST (LCD / touch bring-up without LVGL), service option USB_CDC_DEVICE_SERVICE (native USB CDC device command client)
cmake/build_config_name.cmake (new)	Computes BUILD_CONFIG_NAME (for example ESP32_PIBOT_CNC_PENDANT_V1_8MB_wifi_fluidnc, plus _LOG_<LEVEL> when logs are on), used for installer/<name>/ and compiled in as ESP3D_BUILD_CONFIG_NAME (reported by [ESP420]). Included from features.cmake before project(). It writes <build_dir>/build_config_name.txt, read by the board build scripts (installer_config_name())
cmake/features.cmake	UI_TIER from the board config: SMALL → ESP3D_UI_SMALL (self-contained dashboard display/small_ui, built-in LVGL fonts, no ui_resources partition, and no ESP3D_DISPLAY_FEATURE: the core treats the board as headless); unset → the pendant UI. Defines for USB_CDC_DEVICE_SERVICE and the log backend value
cmake/dev_tools.cmake	Log backends documented 0–5 (5 = USB CDC #1). ESP3D_LOG_BACKEND can be overridden with -DESP3D_LOG_BACKEND=N. Backend 3 (telnet) is an observer build: it forces the telnet server ON and the WiFi CNC socket client and discovery services (mDNS, SSDP) OFF before the options become defines, so the WiFi CNC output disappears; to debug a WiFi CNC use 1 (SD) or 0 (serial)
cmake/postbuild.cmake	Writes one empty installer/<variant>/features/<OPTION>.tag per root OPTION() that is ON in this build (list read from CMakeLists.txt, folder recreated every build), read by package_user_resources_kit.py. The app target depends on the bootloader sub-build, so the bootloader.bin copy can no longer run first on a clean build
cmake/sanity_check.cmake	USB_CDC_DEVICE_SERVICE: ESP32-S2 / S3 only, board opt-in (BOARD_USB_CDC_DEVICE_SUPPORTED), exclusive with USB_SERIAL_SERVICE; ESP3D_LOG_BACKEND=5 requires USB_CDC_DEVICE_SERVICE

---

Key Data Flows

---

References

Core Component Documentation

• Build system overview: CLAUDE.md § Build System, cmake/features.cmake, cmake/sanity_check.cmake
• UI resources binary format and partition layout: docs/ui_resources/development.md
• Theme palette token reference: docs/ui_resources/theme_palette.md
• UI style guide (LVGL): docs/ui_resources/ui_style_guide.md
• UI resources customization (end-user): docs/user documentation/ui_resources_customization.md
• Tools directory guide: docs/guides/tools.md
• Board build guidelines: docs/guides/board_build_guidelines.md
• Feature resource matrix (transport constraints): docs/features/feature_resource_matrix.md
• WebSocket V1 binary protocol: docs/architecture/websockets_protocol.md
• Logging guide: docs/guides/esp3d_log_guide.md

Runtime consumers of generated artefacts

• main/display/esp3d_resources.cpp — loads ui_resources partition at runtime (IdToFont, IdToImg, load_font)
• main/display/esp3d_ui.cpp — applies theme palette tokens (ThemeColors, s_parseThemeIniEntry)
• main/modules/update/esp3d_update_service.cpp — applies SD-card ui_resources.bin patches
• main/modules/translations/esp3d_translation_service.cpp — loads .lng language packs

Design documents (repository)

• tools.md

Related modules

• lua_engine