Configuration Architecture

Why Multiple Configuration Layers?

MyStation's configuration system spans multiple layers of the application stack, each serving a specific purpose. Understanding why this complexity exists is crucial for maintaining and extending the system.

The Five-Layer Architecture

RTC Storage Layer refers to RTC_DATA_ATTR variables for fast access after deep sleep. RTC_DATA_ATTR variables are preverved across deep sleep cycles but lost on full power loss or pushing reset button.

NVS Storage Layer refers to ESP32's Non-Volatile Storage for persistent configuration. After Power loss/pushing reset button, data in NVS remains intact.

Layer 1: HTML/UI Layer (User Interface)

Purpose: Present configuration options to the user

Technology: HTML form elements with IDs

Example:

<input type="number" id="weather-interval" value="{{WEATHER_INTERVAL}}">
<input type="text" id="city-display" value="{{CITY_NAME}}">

Why This Layer?

User-Friendly: Provides human-readable labels and input controls
Validation: Browser-native input validation (min/max, required, etc.)
Accessibility: Standard HTML forms work with screen readers and assistive devices
Template System: Mustache-style {{VARIABLES}} allow dynamic content injection

Naming Convention: kebab-case (e.g., weather-interval, city-display)

Layer 2: JavaScript/JSON Layer (Data Transport)

Purpose: Collect form data and transmit to server

Technology: JavaScript objects and JSON serialization

Example:

var weatherInterval = document.getElementById('weather-interval').value;
var config = {
    weatherInterval: parseInt(weatherInterval),
    cityName: document.getElementById('city-display').value
};

fetch('/save-config', {
    method: 'POST',
    body: JSON.stringify(config)
});

Why This Layer?

Type Conversion: Convert string inputs to proper types (int, float, bool)
Validation: Client-side validation before sending to server
Structured Data: JSON provides clear, type-safe data structure
Debugging: Easy to inspect network traffic and payloads
Standard Protocol: HTTP POST with JSON is universal and well-understood

Naming Convention: camelCase (e.g., weatherInterval, cityName)

Layer 3: C++ Application Layer (Business Logic)

Purpose: Process configuration, apply defaults, validate ranges

Technology: C++ struct with member variables

Example:

struct StationConfig {
    int weatherInterval = 60;      // Default: 60 minutes
    char cityName[32] = "Frankfurt";  // Default city
    float latitude = 50.1109;
    float longitude = 8.6821;

    // ... other fields
};

Why This Layer?

Type Safety: Strong typing prevents errors
Default Values: Compile-time defaults ensure system always has valid config
Memory Management: Fixed-size arrays prevent overflow
Performance: Direct memory access, no parsing overhead
Validation: Range checks and business logic validation
Cross-Module Access: Other C++ modules can directly access configuration

Naming Convention: camelCase (e.g., weatherInterval, cityName)

Layer 4: RTC Storage Layer (Persistence)

Purpose: Store configuration RTC Memory for fast access after deep sleep

Technology: ESP32 RTC Memory

Why This Layer?

Persistence: Survives deep sleep
Key-Value Store: Simple, efficient storage mechanism

Naming Convention: Abbreviated (≤15 chars) due to NVS limitations

Examples: weatherInt, city, stopId, transStart

Why Abbreviated?

NVS Limitation: Keys limited to 15 characters maximum
Flash Efficiency: Shorter keys = less flash usage
Performance: Faster key lookups with shorter strings

Layer 5: Flash Memory (Hardware)

Purpose: Physical storage of configuration data

Technology: SPI Flash with wear leveling

Why This Layer?

Non-Volatile: Data persists when power is removed
Wear Leveling: NVS library manages flash cell wear automatically
Partition Isolation: Configuration stored in dedicated NVS partition
Backup: Can implement backup/restore mechanisms

Data Flow: User to Flash

Save Configuration Flow

Load Configuration Flow

Key Naming Mapping Strategy

The Transformation Path

Naming Rules

Layer	Convention	Reason	Example
HTML (Layer 1)	kebab-case	HTML/CSS standard	`weather-interval`
JavaScript (L2)	camelCase	JavaScript convention	`weatherInterval`
JSON (L2)	camelCase	JSON/REST API standard	`weatherInterval`
C++ (Layer 3)	camelCase	C++ convention (members)	`weatherInterval`
RTC (Layer 4)	camelCase	C++ convention	`weatherInterval`
NVS (Layer 5)	Abbreviated	15-char limit	`weatherInt`

Consistency Benefits

Predictable: Developers can guess key names
Maintainable: Easy to trace through layers
Debuggable: Can search for variations of same name
Documented: Clear mapping tables exist

Special Cases

RTC Storage Layer (Layer 4): Fast Access After Deep Sleep

Purpose: Provide ultra-fast access to configuration data after deep sleep wake, avoiding slow NVS reads

Technology: RTC_DATA_ATTR variables stored in RTC RAM

Example:

// Configuration cached in RTC RAM
RTC_DATA_ATTR int rtc_weatherInterval = 60;
RTC_DATA_ATTR char rtc_cityName[32] = "";
RTC_DATA_ATTR bool rtc_configValid = false;

// Load strategy
void ConfigManager::loadConfig() {
    if (rtc_configValid) {
        // Fast path: Use RTC cached values after deep sleep wake
        g_stationConfig.weatherInterval = rtc_weatherInterval;
        strcpy(g_stationConfig.cityName, rtc_cityName);
    } else {
        // Slow path: Load from NVS after power loss/reset
        g_stationConfig.weatherInterval = preferences.getInt("weatherInt", 60);
        preferences.getString("city", g_stationConfig.cityName, sizeof(g_stationConfig.cityName));

        // Cache in RTC for next wake
        rtc_weatherInterval = g_stationConfig.weatherInterval;
        strcpy(rtc_cityName, g_stationConfig.cityName);
        rtc_configValid = true;
    }
}

Why This Layer?

Performance: RTC RAM access is ~100x faster than NVS flash reads
Battery Life: Reduces flash access, saving power on each wake cycle
Deep Sleep Support: Data persists across deep sleep cycles
No Flash Wear: Avoids unnecessary flash write cycles

When RTC Data is Valid:

✅ After deep sleep wake (most common case)
✅ During normal operation between sleeps

When RTC Data is Invalid:

❌ After power loss (battery removed/dead)
❌ After pressing reset button
❌ After firmware upload
❌ First boot ever

Trade-offs:

Limited Lifetime: Lost on full power loss (unlike NVS/Flash)
Small Size: RTC RAM is limited (8KB on ESP32)
No Persistence: Not suitable for long-term storage

Best Practices:

Always check validity flag before using RTC data
Always have NVS as fallback
Cache frequently-accessed config in RTC
Don't store sensitive data in RTC (not encrypted)

Performance Considerations

Access Speed Comparison

Storage Type	Read Speed	Survives Deep Sleep	Survives Power Loss
C++ RAM	Fastest	❌ No	❌ No
RTC RAM (L4)	Very Fast	✅ Yes	❌ No
NVS Flash (L5)	Slow	✅ Yes	✅ Yes

NVS Access Best Practices

Fast Operations:

✅ Reading all config at boot (once)
✅ Bulk save operation (all fields at once)
✅ Using RTC cache for frequent reads

Slow Operations:

❌ Reading NVS every wake cycle (use RTC instead)
❌ Saving single field repeatedly
❌ Loading config multiple times per wake

Typical Performance Numbers

Operation	Time	Note
Load from RTC (deep sleep)	~1 ms	Fast path, most common
Load from NVS (power loss)	~50 ms	Slow path, infrequent
Save to NVS + RTC	~100 ms	One-time cost on config save
Deep sleep wake (with RTC)	~200 ms	Includes WiFi reconnect
Cold boot (without RTC)	~250 ms	Includes NVS read + WiFi

NVS Config Version (OTA Migration Safety)

Problem

When new firmware adds, renames, or changes NVS keys, devices updating via OTA may have stale or incompatible configuration. While NVS key-value storage is inherently forward-compatible (new keys simply get defaults), structural changes can still cause issues.

Solution

A CURRENT_CONFIG_VERSION integer is stored in NVS under the key cfgVersion. On every boot, loadFromNVS() compares the saved version against the compiled-in version:

static constexpr int CURRENT_CONFIG_VERSION = 1;

// In loadFromNVS():
int savedVersion = preferences.getInt("cfgVersion", 0);
if (savedVersion < CURRENT_CONFIG_VERSION) {
    preferences.clear();  // Reset all NVS config
    preferences.putInt("cfgVersion", CURRENT_CONFIG_VERSION);
}

When to Increment

Change Type	Action	Example
Add new NVS key	No increment needed	Adding `weatherMdl` — defaults to `""`
Rename existing key	Increment + clear	`weatherInt` → `wInterval`
Change value semantics	Increment + clear	Interval unit minutes → seconds
Remove critical key	Increment + clear	Removing a required field

Why is adding a new NVS key safe but changing the RTC struct is not?

NVS is a key-value store — each key is stored independently. Adding weatherMdl between sleepEnd and weekendMode in the code doesn't change how NVS stores them. Old devices simply don't have that key yet, so preferences.getString("weatherMdl", "") returns the default. No corruption.

RTC memory (RTC_DATA_ATTR) is a raw binary struct. Field order determines the memory layout. Adding a 32-byte field shifts everything after it. However, this is safe for OTA because OTA triggers a full reboot (not a deep sleep wake). On full reboot, wakeupCount == 1, so loadFromNVS() always runs and repopulates the struct from NVS — the stale RTC layout is never read.

RTC struct changes would only be dangerous if a device could somehow deep-sleep-wake into new firmware without rebooting, which is not possible with OTA.

What Happens After Version Bump

Device updates firmware via OTA and reboots
loadFromNVS() detects cfgVersion mismatch
NVS is cleared (all settings lost)
Device enters configuration mode (Phase 1 or Phase 2)
User reconfigures via web interface

Version History

Version	Firmware	Change
0	< v0.7.0	No versioning (legacy devices)
1	v0.7.0+	Initial versioned config, added `weatherMdl`

Best Practices

Only increment for truly breaking NVS changes
Adding a new key with a default does not require an increment
Document every version bump in the table above
saveToNVS() always writes cfgVersion to ensure it's set after configuration

Testing Config Version Migration

To verify the NVS clear works correctly without creating a real OTA release:

Flash current firmware (debug build) and configure the device normally:
```
pio run -e esp32-s3-e1001-debug -t upload
```
Configure the device — set stop, weather model, intervals, etc.

Bump the version in src/config/config_manager.cpp:

static constexpr int CURRENT_CONFIG_VERSION = 2; // was 1

Flash again and monitor serial output:

pio run -e esp32-s3-e1001-debug -t upload && pio device monitor

Expected serial output:

[WARN][CONFIG_MANAGER] NVS config version 1 < 2 — clearing NVS for clean migration
[INFO][MAIN] Configuration Phase: 2 (App Setup)

Verify:
- WiFi still connects (credentials stored in separate namespace) ✅
- Device enters Phase 2 configuration mode (app settings cleared) ✅
- All app settings reset to defaults ✅

Revert the version change when done:

# Reset to original version
sed -i '' 's/CURRENT_CONFIG_VERSION = 2/CURRENT_CONFIG_VERSION = 1/' src/config/config_manager.cpp

Tip: For debug builds, increase delay() in initSerialConnector() to 3000ms if you need more time for the serial monitor to connect before boot logs appear.

Summary

Why This Complexity?

The five-layer architecture exists because:

Separation of Concerns: Each layer has one job
Type Safety: Strong typing prevents errors
Validation: Multiple validation points
Persistence: Survives power loss
Performance: Optimized for embedded systems with RTC caching
Maintainability: Clear boundaries, easy to debug
Extensibility: Easy to add new fields
Migration: Can evolve configuration format
Battery Efficiency: RTC layer minimizes flash access

Key Principle

Data flows through transformations, each layer adding value

Layer 1 - HTML: User interaction and presentation
Layer 2 - JavaScript/JSON: Client-side logic and transport format
Layer 3 - C++ Application: Business logic, validation, and runtime access
Layer 4 - RTC Storage: Fast access cache for deep sleep scenarios
Layer 5 - NVS Storage: Persistent storage surviving power loss

The Smart Loading Strategy

Power-On/Reset:  NVS (slow) → C++ → RTC → Display
                  └─────────────────────┘
                  Cache for future wakes

Deep Sleep Wake: RTC (fast) → C++ → Display
                  └─────────────────┘
                  Skip NVS entirely

This architecture has served MyStation well and should be maintained as the project evolves.

Why Multiple Configuration Layers?​

The Five-Layer Architecture​

Layer 1: HTML/UI Layer (User Interface)​

Layer 2: JavaScript/JSON Layer (Data Transport)​

Layer 3: C++ Application Layer (Business Logic)​

Layer 4: RTC Storage Layer (Persistence)​

Layer 5: Flash Memory (Hardware)​

Data Flow: User to Flash​

Save Configuration Flow​

Load Configuration Flow​

Key Naming Mapping Strategy​

The Transformation Path​

Naming Rules​

Consistency Benefits​

Special Cases​

RTC Storage Layer (Layer 4): Fast Access After Deep Sleep​

Performance Considerations​

Access Speed Comparison​

NVS Access Best Practices​

Typical Performance Numbers​

NVS Config Version (OTA Migration Safety)​

Problem​

Solution​

When to Increment​

What Happens After Version Bump​

Version History​

Best Practices​

Testing Config Version Migration​

Summary​

Why This Complexity?​

Key Principle​

The Smart Loading Strategy​

Why Multiple Configuration Layers?

The Five-Layer Architecture

Layer 1: HTML/UI Layer (User Interface)

Layer 2: JavaScript/JSON Layer (Data Transport)

Layer 3: C++ Application Layer (Business Logic)

Layer 4: RTC Storage Layer (Persistence)

Layer 5: Flash Memory (Hardware)

Data Flow: User to Flash

Save Configuration Flow

Load Configuration Flow

Key Naming Mapping Strategy

The Transformation Path

Naming Rules

Consistency Benefits

Special Cases

RTC Storage Layer (Layer 4): Fast Access After Deep Sleep

Performance Considerations

Access Speed Comparison

NVS Access Best Practices

Typical Performance Numbers

NVS Config Version (OTA Migration Safety)

Problem

Solution

When to Increment

What Happens After Version Bump

Version History

Best Practices

Testing Config Version Migration

Summary

Why This Complexity?

Key Principle

The Smart Loading Strategy