Network Bridges and Gateways

1. Overview and where bridges sit

A bridge layer sits between in-vehicle buses and sensors and the cloud broker. It reads frames from CAN (J1939, ISO-TP), NMEA 2000, NMEA 0183, Modbus RTU, LIN, BLE and one-way TPMS radio, stamps and filters the data, buffers it, and publishes it over a cellular, LoRaWAN, Wi-Fi or satellite uplink.

Source figures are tagged in the research as sourced from a URL or as an engineering estimate. Values marked estimate below are engineering estimates, not verified from a fetched source, and should be used as priors with wide jitter.

2. Gateway types and protocol translation

A Teltonika-style binary AVL record is about 40-120 B, with up to 255 records per packet (estimate).

Protocol translation steps in the research: reassemble multi-frame transport (J1939 TP, ISO-TP, N2K Fast Packet), decode raw signals with scaling and sentinel handling, filter by deadband or exception, then encode into the uplink codec (see sections 4 and 7).

3. Uplinks (cellular, LoRaWAN, Wi-Fi offload, satellite)

4. Store-and-forward, batching and MQTT delivery semantics

Store-and-forward (all values estimates)

Batching, compression and edge filtering

MQTT delivery semantics

• Resend happens only on reconnect with Clean Start=0 and a session present. A clean session discards all in-flight and queued messages.
• Packet ids are reused after completion, so DUP=1 with the same id is ambiguous.
• Keepalive 30-120 s; cellular NAT timeouts 30-300 s so use 60 s or less; reconnect backoff 1-60 s with jitter; TLS handshake 2-4 RTT and 3-6 kB (resumption about 1 kB) (estimates).
• AWS IoT Core allows 128 KB maximum message size; device buffers are often 16-64 KB.
• Cloud dedupe key: (device_id, boot_id, seq). A sequence reset on reboot is a classic source of false duplicates and gaps.

5. Timestamps, clock drift and ordering

Ordering: QoS 1 retries can be overtaken by newer messages, and replay interleaved with live data arrives out of order (sections 4 and 6). Consumers should order by the record timestamp and detect gaps by sequence, not by arrival time.

6. Bridge failure modes (with frequency/signatures)

All inject rates below are estimates from the research file.

Format-level errors from the research: JSON can carry NaN/Inf, invalid UTF-8 and truncation at the buffer limit; a protobuf schema mismatch drops unknown fields and defaults to 0; a vendor binary firmware or parser mismatch garbles IO IDs.

7. Payload formats and data volumes per vehicle type

Measured reference points: Digital Matter reports 1-3 MB/month for battery trackers, 5-10 MB/month for wired trackers, 1-1.5 kB per NB-IoT heartbeat, and firmware of 300-500 kB.

8. Multi-tenant topic, schema and policy design

Sampling policy is pushed through a retained config topic or device shadow; 1-10% of devices run an old policy for hours (config lag). Payload formats per tenant are not specified in the research. Inference: assign tenants from the format table in section 7 (for example JSON for bus, CBOR or Protobuf for ambulance, batched Protobuf for marine) as generator defaults.

9. Generator guidance for the simulator

The pipeline below is from the research. The concrete parameter choices in the blocks that follow are inference built from the cited ranges, not additional sourced facts.

10. How Sense and Queen carry this data

10.1 The subvariant is stored on the tenant

A tenant with a variant (for example bus) can carry an optional subvariant (for example city-transit); an empty string means none. The field is a lowercase string matching [a-z][a-z0-9_-]{0,39}, accepted only when a variant is given; malformed values are rejected with HTTP 400. An existing tenant without one returns an empty string, and a later create call with a different non-empty value updates it (an empty or missing value never removes it).

• Business database: column TENANT_DEF.subvariant VARCHAR(40) NOT NULL DEFAULT '', added by the migration 2026-10-12-tenant-subvariant.sql.
• Static database (pn_static_db): master table TENANT_VARIANT_MASTER with id, variant, subvariant, label, description, sort order, active flag and created date, unique on (variant, subvariant); the row with an empty subvariant is the variant itself. Created and seeded with the bus subvariants by 2026-10-12-tenant-variant-master.sql; truck and ambulance are added by 2026-10-13-tenant-variant-master-truck-ambulance.sql. Further variants need only inserted rows, no code change.
• Runtime tolerance: the platform PHP detects the column and the table at runtime, so the code can be deployed before the migrations. Without the column the subvariant is returned empty and not stored; without the master table (or with the static database unreachable) only the regular expression is checked. If the master has active subvariants for the variant, an unknown value is rejected with the allowed values listed.
• Who reads it: the tenant create, list, keys-all and links endpoints return it; get_tenant_layout.php adds it to its tenant block, which is what Sense reads for the tenant identity. Sense keeps it in the setting building.subvariant only to choose the vehicle profile of its compatibility check. The stored layout format and set_tenant_layout.php are unchanged, so the layout sent to the platform does not carry it. The Controller and the simulator desktop app read it too (the simulator's change was uncommitted when checked).

10.2 Tenant meta in the transmission

Sense adds the subvariant to the tenant meta of every transmission (readings, alarms and messages) under the name subvariant, after the building manager's custom values. A custom value that is itself named subvariant is kept and the platform's is not sent; nothing is sent when the tenant has none. Sense and Queen are independent in this: a transmission with or without the subvariant meta is valid for both. In a transmission the entity group is the path down to the controller without the network, and the network is sent in the entity's meta.network.

10.3 What Queen stores: new columns and the tenant_profile table

Before this change Queen kept only a fraction of what it parsed: all tenant meta, tags, path and the non-coordinate location parts, the fixedread node meta, all entity meta and the impact and actor names of reading entities were lost on every table. The change is additive (nothing dropped or renamed, no TTL).

• bhairava.tenant_profile (new, durable, written by Queen): the tenant's own description (variant, group, tags, path, location, meta) per tenant and content, joined to the other tables on (owner_oid, tenant_type, tenant_name). It is an AggregatingMergeTree keyed with a meta_hash (first 16 hex digits of the SHA-256 of variant, group, tags, path, location and meta, with meta keys sorted), so a change of the subvariant or any custom value becomes a new row and the history is kept with the first and last time each was seen. Queen writes a row only when the profile changed or an hour has passed, about 24 small rows per tenant per day.
• bhairava.sensor_profile.entity_meta (new column, written by the sampler): the stable part of a sensor's newest entity meta (alias, controller alias, network, device, address, object type and similar), with impact and actor names under the keys @impact and @actors. Per-packet keys (raw datagram hex, source and receive times, sequence numbers, status flags, values over 200 characters) are left out.

Still lost, by design or because the code cannot carry it: the transmission @ts (the parser never reads it, and Sense does not send it), the changerecord entity @variant, and the per-packet entity meta and fixedread node meta after the pruning below.

10.4 Pruning of readings and what lasts

The sampler (SensorSampler, every 5 minutes) drops each hourly bhairava.reading partition that ends at or before the watermark minus a 10-minute overlap, after every successful run. A reading therefore lives about 1 h 10 min to 2 h 15 min, longer only while the sampler is failing or disabled (a failed run drops nothing). Anything stored only in reading is gone after that; this is why the durable parts are copied to two small durable places.

Estimate, not measured: the new columns add about 1.4 KB (0.9-2.0 KB) per reading row, mostly the raw datagram; at 26 events per second per bus and 24 buses that is about 75 GB written per day, but because of the pruning only about 3.8-7.1 GB uncompressed is on disk at any moment. Compression was not measured (5x to 10x is an expectation). The durable additions are small: about 0.3 MB a day written for tenant_profile for 24 buses. Whether raw meta should be kept at all, or for how long, is left to the owner; no TTL was applied.

10.5 Safe rollout order

1. Done on 13 Oct 2026: run the migration migrations/2026-10-13-meta-storage.sql on ClickHouse (additive, metadata only, safe to re-run): the meta columns, sensor_profile.entity_meta and tenant_profile. The Queen that is running now names its columns in every statement, so it keeps working and the new columns stay empty. Run it right after a sampler run (they are 5 minutes apart).
2. Not done yet: deploy the new Queen.
3. Not done yet: deploy the new Sense where subvariants are wanted.

The platform migrations of 10.1 can run in any order relative to the code, because the PHP detects the column and table at runtime. Rolling Queen back afterwards is safe.

10.6 What is not verified

• Not deployed: the new Queen, the platform API code and the new Sense. The migrations are applied; the code that writes the new columns and tables is not running yet, so they are empty.
• Validated in clickhouse-local before applying: the ALTER and CREATE TABLE statements (applied to the old definition they give the same columns as a fresh install, and a second run changes nothing), the insert formats with the new columns, and two profile rows merging into one with the first and last seen times kept. Not validated: ARRAY JOIN over the parallel entity arrays, and argMax over a Map. The HTTP status ClickHouse returns for code 16 was not observed (both 4xx and 5xx are handled and tested with a fake server).
• That ADD COLUMN on reading blocks nothing while the sampler runs is reasoned, not tested. The tests (Queen's save handler and sampler checks) pass against a fake HTTP server and an in-memory store, not against ClickHouse.
• The dashboard is not changed and does not use the new meta.

11. Uncertainty and assumptions