[Bug]: MQTT config does not persist reliably on RAK4631 RAK13300 Wifi and Ethernet gateway devices running 2.7.19; proxy/UI/API behavior is also unstable

### Category

WiFi

### Hardware

Rak4631

### Is this bug report about any UI component firmware like InkHUD or Meshtatic UI (MUI)?

- [ ] Meshtastic UI aka MUI colorTFT
- [ ] InkHUD ePaper
- [ ] OLED slide UI on any display

### Firmware Version

2.7.19.bb3d6d5  Rak4631_eth_gw

### Description

````markdown id="hqoict"
# MQTT configuration does not persist on RAK gateway devices running Meshtastic 2.7.19, with repeated API/UI instability and inconsistent proxy behavior

> I fed ChatGPT a mountain of serial logs and terminal history because, after hours of wrestling with this, my original notes had evolved from “technical observations” into an enormous pile of notes full of explicit adjectives and the thought of sanitizing the documentation by hand was more than I could handle today. The result below is the cleaned-up version of the same evidence, minus the sleep deprivation and creative adjectives.

## Summary

MQTT behavior on Meshtastic `2.7.19.bb3d6d5` appears broken on multiple new RAK ethernet and wifi gateway devices.  My documented testing over the last 12 hours was on the RAK4631 with the RAK13800 ethernet module, but I experienced the same exact behavior over the last two weeks with the RAK WisMesh WiFi Gateways.

Across four new devices total:
- 2 × RAK WisMesh WiFi Gateway devices
- 2 × RAK4631 devices with RAK13800 Ethernet modules

...a basic MQTT gateway configuration resulted in unstable or non-persistent behavior. The detailed evidence below is from a RAK4631 Ethernet gateway device running:

- `firmware-rak4631_eth_gw-2.7.19.bb3d6d5.uf2`
- `pioEnv: rak4631_eth_gw`

In addition to the Ethernet gateway testing, I also tested a Wi-Fi gateway-class device running the same Meshtastic release branch on:

- `pioEnv: rak11200`
- firmware version `2.7.19.bb3d6d5`

The most critical defect is that `mqtt.enabled` does not persist. The CLI accepts the write, reports success, and says it wrote the MQTT configuration to the device, but read-back immediately returns `False`, including after reboot.

At the same time, the device also exhibited:
- inconsistent `mqtt.proxy_to_client_enabled` persistence
- repeated API session churn and forced TCP disconnects
- repeated `Unknown module config type 14` errors during config transfer
- unreliable settings/UI behavior
- a general “gateway is alive, but config state is broken” condition

This does **not** look like simple setup error. Ethernet networking, Wi-Fi gateway boot, mesh participation, and general radio operation continued to work. The failure appears isolated to configuration persistence / MQTT / API handling.

---

## Affected hardware / firmware

### Confirmed from attached Ethernet gateway evidence
- Hardware: RAK4631 with Ethernet gateway path (`rak4631_eth_gw`)
- Firmware: `2.7.19.bb3d6d5`
- Firmware file used: `firmware-rak4631_eth_gw-2.7.19.bb3d6d5.uf2`
- Metadata observed:
  - `hwModel: RAK4631`
  - `pioEnv: rak4631_eth_gw`
  - `hasEthernet: true`
  - `hasBluetooth: true`
  - `hasWifi: false`
- Transport used for configuration: Meshtastic CLI over TCP host connection
- Network mode: Ethernet enabled, DHCP

### Additional Wi-Fi gateway-class device also tested
- Hardware/firmware path observed in boot log: `rak11200`
- Firmware: `2.7.19.bb3d6d5`
- Boot log confirms same release branch was also tested on Wi-Fi hardware:
  - `S:B:13,2.7.19.bb3d6d5,rak11200,meshtastic/firmware`

### User-reported scope
The same general MQTT failure pattern was observed across:
- 2 × RAK WisMesh WiFi Gateway devices
- 2 × RAK4631 + RAK13800 Ethernet gateway devices

---

## Expected behavior

When setting MQTT options on a gateway device, the device should:

1. accept the write
2. persist the new value
3. return the new value on immediate read-back
4. retain the value across reboot
5. keep the MQTT module in a consistent state

For example:

```bash
meshtastic --host [REDACTED_HOST] --set mqtt.enabled true
meshtastic --host [REDACTED_HOST] --get mqtt.enabled
````

Expected result:

```text id="ninvij"
mqtt.enabled: True
```

The same applies to `mqtt.proxy_to_client_enabled`, which should only change when explicitly set.

---

## Actual behavior

Observed behavior:

* `mqtt.enabled` does not persist
* CLI reports success, but immediate read-back returns `False`
* reboot does not make the setting stick
* `mqtt.proxy_to_client_enabled` was inconsistent and sometimes appeared to re-enable or refuse to disable
* API/config sessions were repeatedly dropped/force-closed
* `Unknown module config type 14` was repeatedly logged during config transfer
* Web/UI settings access became unstable
* the device remained otherwise functional on Ethernet and LoRa, suggesting config/MQTT/API-specific breakage rather than total device failure
* a separate Wi-Fi gateway-class device on `rak11200` / `2.7.19.bb3d6d5` was also under test, confirming this was not limited to one hardware path

---

## Reproduction steps

### Repro case A: `mqtt.enabled` does not persist

1. Connect to the gateway over TCP:

```bash id="3w6ky4"
meshtastic --host [REDACTED_HOST] --set mqtt.enabled true
```

2. CLI reports success:

```text id="w8h4ca"
Set mqtt.enabled to true
Writing modified preferences to device
Writing mqtt configuration to device
```

3. Read it back:

```bash id="zaxwgd"
meshtastic --host [REDACTED_HOST] --get mqtt.enabled
```

4. Actual result:

```text id="k3e97q"
mqtt.enabled: False
```

5. Reboot:

```bash id="3y9and"
meshtastic --host [REDACTED_HOST] --reboot
```

6. Read it again:

```bash id="gxiq07"
meshtastic --host [REDACTED_HOST] --get mqtt.enabled
```

7. Actual result remains:

```text id="nhwzji"
mqtt.enabled: False
```

This was reproduced multiple times in a row.

---

### Repro case B: `mqtt.proxy_to_client_enabled` showed inconsistent persistence

Attempted:

```bash id="l1v7jm"
meshtastic --host [REDACTED_HOST] --set mqtt.proxy_to_client_enabled false
meshtastic --host [REDACTED_HOST] --get mqtt.proxy_to_client_enabled
```

Observed failure case:

```text id="cu0dzu"
Set mqtt.proxy_to_client_enabled to false
Writing modified preferences to device
Writing mqtt configuration to device
...
mqtt.proxy_to_client_enabled: True
```

Later, after disabling MQTT first and rewriting MQTT fields individually, the proxy flag finally read back as `False`, but `mqtt.enabled` still would not persist.

---

### Repro case C: order-dependent behavior

The only partially workable sequence found was:

1. disable MQTT
2. save / let device reboot
3. modify MQTT settings individually while MQTT remains disabled
4. save / let device reboot
5. enable MQTT
6. save / let device reboot

Even then, `mqtt.enabled` still repeatedly reverted to `False`.

---

## Relevant command history

### `mqtt.enabled` write succeeds but does not persist

```text id="tcyy32"
$ meshtastic --host [REDACTED_HOST] --set mqtt.enabled true
Connected to radio
Set mqtt.enabled to true
Writing modified preferences to device
Writing mqtt configuration to device

$ meshtastic --host [REDACTED_HOST] --get mqtt.enabled
Connected to radio
mqtt.enabled: False
Completed getting preferences
```

### Reproduced again after reboot

```text id="lh9ojw"
$ meshtastic --host [REDACTED_HOST] --set mqtt.enabled true
Connected to radio
Set mqtt.enabled to true
Writing modified preferences to device
Writing mqtt configuration to device

$ meshtastic --host [REDACTED_HOST] --reboot
Connected to radio
INFO ... Telling node to reboot in 10 seconds

$ meshtastic --host [REDACTED_HOST] --get mqtt.enabled
Connected to radio
mqtt.enabled: False
Completed getting preferences
```

### Proxy write initially did not stick

```text id="jlwm63"
$ meshtastic --host [REDACTED_HOST] --set mqtt.proxy_to_client_enabled false
$ meshtastic --host [REDACTED_HOST] --get mqtt.proxy_to_client_enabled
Connected to radio
Set mqtt.proxy_to_client_enabled to false
Writing modified preferences to device
Writing mqtt configuration to device
Connected to radio
mqtt.proxy_to_client_enabled: True
Completed getting preferences
```

### Later sequence where proxy finally read back as false

```text id="7sylw0"
$ meshtastic --host [REDACTED_HOST] --set mqtt.enabled false
Connected to radio
Set mqtt.enabled to false
Writing modified preferences to device
Writing mqtt configuration to device

$ meshtastic --host [REDACTED_HOST] --set mqtt.address "[REDACTED_BROKER]"
$ meshtastic --host [REDACTED_HOST] --set mqtt.username "[REDACTED_USERNAME]"
$ meshtastic --host [REDACTED_HOST] --set mqtt.password "[REDACTED_PASSWORD]"
$ meshtastic --host [REDACTED_HOST] --set mqtt.tls_enabled false
$ meshtastic --host [REDACTED_HOST] --set mqtt.json_enabled false
$ meshtastic --host [REDACTED_HOST] --set mqtt.proxy_to_client_enabled false
$ meshtastic --host [REDACTED_HOST] --set mqtt.root "[REDACTED_TOPIC_ROOT]"

$ meshtastic --host [REDACTED_HOST] --get mqtt.proxy_to_client_enabled
Connected to radio
mqtt.proxy_to_client_enabled: False
Completed getting preferences
```

### Even then, enabling MQTT still reverted to false

```text id="p6jlr4"
$ meshtastic --host [REDACTED_HOST] --set mqtt.enabled true
Connected to radio
Set mqtt.enabled to true
Writing modified preferences to device
Writing mqtt configuration to device

$ meshtastic --host [REDACTED_HOST] --get mqtt.enabled
Connected to radio
mqtt.enabled: False
Completed getting preferences
```

---

## Serial console evidence

### Repeated API session churn / forced disconnects

```text id="bcjlwm"
INFO  [ApiServer] Previous TCP connection still open, try again in 200ms
INFO  [ApiServer] Previous TCP connection still open, try again in 400ms
INFO  [ApiServer] Force close previous TCP connection
DEBUG [ApiServer] PhoneAPI::close()
INFO  [ApiServer] Incoming API connection
INFO  [ApiServer] Incoming ethernet connection
INFO  [ServerAPI] Client wants config, nonce=[REDACTED]
...
INFO  [ServerAPI] Lost phone connection
DEBUG [ServerAPI] PhoneAPI::close()
INFO  [ServerAPI] Client dropped connection, suspend API service
```

This pattern repeated many times during normal CLI config access.

---

### Repeated config/schema error during config transfer

```text id="7v7v3d"
DEBUG [ServerAPI] Send module config: mqtt
DEBUG [ServerAPI] Send module config: serial
DEBUG [ServerAPI] Send module config: ext notification
DEBUG [ServerAPI] Send module config: store forward
DEBUG [ServerAPI] Send module config: range test
DEBUG [ServerAPI] Send module config: telemetry
DEBUG [ServerAPI] Send module config: canned message
DEBUG [ServerAPI] Send module config: audio
DEBUG [ServerAPI] Send module config: remote hardware
DEBUG [ServerAPI] Send module config: neighbor info
DEBUG [ServerAPI] Send module config: ambient lighting
DEBUG [ServerAPI] Send module config: detection sensor
DEBUG [ServerAPI] Send module config: paxcounter
ERROR [ServerAPI] Unknown module config type 14
```

This also repeated throughout the session.

---

### Network stack and basic gateway services were functioning

```text id="v3u2f8"
INFO  Start Ethernet DHCP
INFO  Local IP [REDACTED_IP]
INFO  Subnet Mask [REDACTED]
INFO  Gateway IP [REDACTED]
INFO  DNS Server IP [REDACTED]
INFO  [ethConnect] Start Ethernet network services
INFO  [ethConnect] Start NTP time client
INFO  [ethConnect] API server listening on TCP port 4403
DEBUG [ethConnect] UDP Listening on IP: 224.0.0.69:4403
```

This is important because the device was not generally dead or offline.

---

### Repeated NTP failure on boot

```text id="zuphv7"
INFO  [ethConnect] Update NTP time from
ERROR [ethConnect] NTP Update failed
```

This may be separate, but it was consistently present.

---

### Device continued participating in mesh traffic

```text id="tz4v7u"
INFO  [Router] Received DeviceTelemetry from=0x72ef5f6b ...
INFO  [Router] Received EnvironmentTelemetry from=0x72ef5f6b ...
INFO  [Router] Rebroadcast received message coming from 6b
DEBUG [Router] Broadcasting packet over UDP ...
DEBUG [Router] Forwarding to phone ...
```

This supports that radio and routing were alive while config persistence was broken.

---

## Additional Wi-Fi gateway-class boot evidence (`rak11200`)

In addition to the Ethernet gateway logs above, the following boot excerpt shows that a Wi-Fi gateway-class device on the same release branch (`2.7.19.bb3d6d5`) was also tested.

```text id="j7nrbf"
INFO  Booted, wake cause 0 (boot count 1), reset_reason=reset
DEBUG Filesystem files (20480/1048576 Bytes):
DEBUG  /prefs/channels.proto (327 Bytes)
DEBUG  /prefs/config.proto (212 Bytes)
DEBUG  /prefs/device.proto (118 Bytes)
DEBUG  /prefs/module.proto (102 Bytes)
DEBUG  /prefs/nodes.proto (867 Bytes)
INFO  Starting Bus with (SDA) 4 and (SCL) 5:
INFO  No I2C devices found
INFO  S:B:13,2.7.19.bb3d6d5,rak11200,meshtastic/firmware
INFO  Build timestamp: 1770508800
DEBUG Total heap: 179836
DEBUG Free heap: 137600
DEBUG Total PSRAM: 4192059
DEBUG Free PSRAM: 4190279
DEBUG Number of Device Reboots: 1099
INFO  MeshtasticOTA firmware version 1.0.1
INFO  Init NodeDB
INFO  Loaded saved nodedatabase version 24, with nodes count: 7
```

### Additional Wi-Fi startup anomalies seen on that device

```text id="zv4p20"
open(): /littlefs/static/static does not exist, no permits for creation
open(): /littlefs/prefs/db.proto does not exist, no permits for creation
open(): /littlefs/prefs/uiconfig.proto does not exist, no permits for creation
open(): /littlefs/prefs/cannedConf.proto does not exist, no permits for creation
gpio_isr_handler_remove(...): GPIO isr service is not installed, call gpio_install_isr_service() first
```

```text id="0p36zv"
Arduino Event: 0 - WIFI_READY
Arduino Event: 2 - STA_START
Arduino Event: 5 - STA_DISCONNECTED
Reason: 201 - NO_AP_FOUND
WiFi Reconnect Running
```

I am including this because I tested both Ethernet and Wi-Fi gateway-class RAK hardware on the same Meshtastic release branch while chasing this MQTT problem.

---

## Config snapshots showing contradictory MQTT state

### Snapshot where MQTT appeared enabled and proxy was on

```json id="b2rjvc"
{
  "mqtt": {
    "enabled": true,
    "address": "[REDACTED_BROKER]",
    "username": "[REDACTED_USERNAME]",
    "password": "[REDACTED_PASSWORD]",
    "encryptionEnabled": true,
    "tlsEnabled": true,
    "root": "[REDACTED_TOPIC_ROOT]",
    "proxyToClientEnabled": true,
    "jsonEnabled": false,
    "mapReportingEnabled": false
  }
}
```

### Later snapshot where MQTT was disabled again

```json id="vbdtgq"
{
  "mqtt": {
    "address": "[REDACTED_BROKER]",
    "username": "[REDACTED_USERNAME]",
    "password": "[REDACTED_PASSWORD]",
    "encryptionEnabled": true,
    "root": "[REDACTED_TOPIC_ROOT]",
    "enabled": false,
    "jsonEnabled": false,
    "tlsEnabled": false,
    "proxyToClientEnabled": false,
    "mapReportingEnabled": false
  }
}
```

The contradictory state snapshots strongly suggest unstable config application or persistence.

---

## Additional evidence from `--listen`

A CLI listen/debug session captured the device reporting MQTT config and proxy traffic.

### MQTT module config seen from device

```text id="d5iqw8"
Received from radio: moduleConfig {
  mqtt {
    enabled: true
    address: "[REDACTED_BROKER]"
    username: "[REDACTED_USERNAME]"
    password: "[REDACTED_PASSWORD]"
    encryption_enabled: true
    tls_enabled: true
    root: "[REDACTED_TOPIC_ROOT]"
    proxy_to_client_enabled: true
    ...
  }
}
```

### MQTT proxy message observed

```text id="0k0sox"
Received from radio: mqttClientProxyMessage {
  topic: "[REDACTED_TOPIC]"
  data: "[REDACTED_PAYLOAD]"
}
```

This confirms the proxy path was active at least some of the time.

---

## Comparison case: Bluetooth behaved differently

Bluetooth also showed delayed persistence behavior, but unlike MQTT, it eventually stuck after reboot.

```text id="jlwmsp"
$ meshtastic --host [REDACTED_HOST] --set bluetooth.enabled true
Connected to radio
Set bluetooth.enabled to true
Writing modified preferences to device
Writing bluetooth configuration to device

$ meshtastic --host [REDACTED_HOST] --get bluetooth.enabled
Connected to radio
bluetooth.enabled: False
Completed getting preferences

$ meshtastic --host [REDACTED_HOST] --reboot
Connected to radio
INFO ... Telling node to reboot in 10 seconds

$ meshtastic --host [REDACTED_HOST] --get bluetooth.enabled
Connected to radio
bluetooth.enabled: True
Completed getting preferences
```

This suggests the write/reboot path is not universally broken, and that the MQTT issue is more specific.

---

## Why this looks like a defect and not operator error

The following all worked normally during testing:

* Ethernet DHCP
* TCP host connectivity
* API server availability
* LoRa packet RX/TX
* mesh participation
* normal telemetry traffic
* Wi-Fi gateway-class hardware booting on the same release branch
* at least some non-MQTT config persistence after reboot

The following specifically did **not** work correctly:

* MQTT enable state persistence
* consistent proxy flag behavior
* stable config/API session handling
* clean module config transfer

That points toward a firmware/config handling defect in 2.7.19 rather than an incorrect gateway setup.

---

## Impact

This issue prevents reliable use of a RAK gateway device as an MQTT gateway because:

* `mqtt.enabled` cannot be trusted to stay enabled
* proxy behavior is inconsistent
* UI/API config sessions become unstable
* the device can end up in a confusing partially-configured state that is difficult to recover from without repeated trial-and-error or reflashing

For a gateway-class build, this is a core functionality failure.

---

## Minimal high-signal evidence set

If only the shortest proof is needed, these excerpts capture the issue:

```text id="m21kbq"
$ meshtastic --host [REDACTED_HOST] --set mqtt.enabled true
Connected to radio
Set mqtt.enabled to true
Writing modified preferences to device
Writing mqtt configuration to device

$ meshtastic --host [REDACTED_HOST] --get mqtt.enabled
Connected to radio
mqtt.enabled: False
Completed getting preferences
```

```text id="lqg7p5"
$ meshtastic --host [REDACTED_HOST] --reboot
Connected to radio
INFO ... Telling node to reboot in 10 seconds

$ meshtastic --host [REDACTED_HOST] --get mqtt.enabled
Connected to radio
mqtt.enabled: False
Completed getting preferences
```

```text id="bm6b06"
INFO  [ApiServer] Previous TCP connection still open, try again in 200ms
INFO  [ApiServer] Previous TCP connection still open, try again in 400ms
INFO  [ApiServer] Force close previous TCP connection
INFO  [ServerAPI] Lost phone connection
INFO  [ServerAPI] Client dropped connection, suspend API service
```

```text id="i5llzn"
ERROR [ServerAPI] Unknown module config type 14
```

```text id="w6pknc"
$ meshtastic --host [REDACTED_HOST] --set mqtt.proxy_to_client_enabled false
Connected to radio
Set mqtt.proxy_to_client_enabled to false
Writing modified preferences to device
Writing mqtt configuration to device

$ meshtastic --host [REDACTED_HOST] --get mqtt.proxy_to_client_enabled
Connected to radio
mqtt.proxy_to_client_enabled: True
Completed getting preferences
```

```text id="yg8x7g"
INFO  S:B:13,2.7.19.bb3d6d5,rak11200,meshtastic/firmware
Arduino Event: 0 - WIFI_READY
Arduino Event: 2 - STA_START
Arduino Event: 5 - STA_DISCONNECTED
Reason: 201 - NO_AP_FOUND
WiFi Reconnect Running
```

---

## Final defect statement

On Meshtastic `2.7.19.bb3d6d5`, MQTT configuration on RAK gateway-class hardware is not persisting reliably. The most serious symptom is that `mqtt.enabled` repeatedly reverts to `false` immediately after being set to `true`, including after reboot. During the same session, `mqtt.proxy_to_client_enabled` showed inconsistent persistence, the API/config session repeatedly force-closed existing TCP connections, and the device repeatedly logged `Unknown module config type 14` during config transfer. Ethernet networking and normal mesh traffic otherwise remained functional, and a separate Wi-Fi gateway-class device on `rak11200` / `2.7.19.bb3d6d5` was also under test on the same release branch. The overall evidence points to a configuration persistence / MQTT / API handling defect rather than a general hardware or network failure.

```
```


### Relevant log output

```Shell

```

Uh oh!

Uh oh!

[Bug]: MQTT config does not persist reliably on RAK4631 RAK13300 Wifi and Ethernet gateway devices running 2.7.19; proxy/UI/API behavior is also unstable #9876

Description

Category

Hardware

Is this bug report about any UI component firmware like InkHUD or Meshtatic UI (MUI)?

Firmware Version

Description

Actual behavior

Reproduction steps

Repro case A: mqtt.enabled does not persist

Repro case B: mqtt.proxy_to_client_enabled showed inconsistent persistence

Repro case C: order-dependent behavior

Relevant command history

mqtt.enabled write succeeds but does not persist

Reproduced again after reboot

Proxy write initially did not stick

Later sequence where proxy finally read back as false

Even then, enabling MQTT still reverted to false

Serial console evidence

Repeated API session churn / forced disconnects

Repeated config/schema error during config transfer

Network stack and basic gateway services were functioning

Repeated NTP failure on boot

Device continued participating in mesh traffic

Additional Wi-Fi gateway-class boot evidence (rak11200)

Additional Wi-Fi startup anomalies seen on that device

Config snapshots showing contradictory MQTT state

Snapshot where MQTT appeared enabled and proxy was on

Later snapshot where MQTT was disabled again

Additional evidence from --listen

MQTT module config seen from device

MQTT proxy message observed

Comparison case: Bluetooth behaved differently

Why this looks like a defect and not operator error

Impact

Minimal high-signal evidence set

Final defect statement

Relevant log output

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Repro case A: `mqtt.enabled` does not persist

Repro case B: `mqtt.proxy_to_client_enabled` showed inconsistent persistence

`mqtt.enabled` write succeeds but does not persist

Additional Wi-Fi gateway-class boot evidence (`rak11200`)

Additional evidence from `--listen`