Skip to content
GREEN FUTURETECH.
Explore

Research libraryFactories on a Weak Grid

Chapter 05 of 14 · Part II · Draft web edition

A working local network should outlast a failed backhaul

Local name resolution, switching, identity and observability need their own continuity plan when external connectivity disappears.

Sections
7
Exhibits
4
Minutes
30
Sources
20
EX 19 · MQTT v5.0 QoS 1 (TCP, OASIS): Built-in session state and ordering (0 to 100) 78, Lossy and low-MTU link fit (0 to 100) 64. Quadrant: Store-and-forward brokered session (MQTT)

The cloth is woven from this chapter's 4 exhibits: each weft is one exhibit's own data, normalised; each warp group is a section. The probe reads the exhibit's own value where it crosses. Procedural visualisation, not a measurement.

The chapter as a draft: 7 sections, 2 exhibit wefts, one shuttle.

Lead exhibit · 18LayersillustrativeLocal work can survive the loss of an external route
Layer 4: Optional upstream backhaulPrimary and secondary WAN mod…Central ERP ingestion APIRemote SIEM and identity prov…Public DNS and NTP upstreamsLayer 3: Local store and supervisionTransactional SQLite edge sto…Local MQTT or HTTP ingestionLocal operator console HMILocally hosted offline runbookLayer 2: Local network control planeAuthoritative local split DNSStatic IPs and long-lease DHCPLocal NTPv4 stratum serverPinned local CA trust bundleLayer 1: Protected switching andpowerUPS-backed floor access switc…Protected PoE injectors and c…Weighbridge and scanner termi…Line controller PLC gateways

Read-outLayer 4: Optional upstream backhaul: Primary and secondary WAN modems, Central ERP ingestion API, Remote SIEM and identity provider, Public DNS and NTP upstreams

Method and source

Illustrative plant LAN dependency stack derived from NIST SP 800-82 Rev. 3 zone segmentation principles (S03) and RFC 7228 constrained-network topology considerations (S18). Every component in Layers 1 to 3 must operate with Layer 4 physically disconnected.

Unit: illustrative architecture layers. Status: illustrative. Sources: S03, S18. Illustrative figures are teaching examples, not measured factory results.

The chapter's warpBar: share of the reading · square: an exhibit

Section 5.0 · Opening · 1 min · no exhibit of its own · one movement Go to section ↓

Overture · 1 min

A factory network on a weak grid remains useful during a wide-area outage only if every service required to address, authenticate, route and observe local traffic lives inside the plant boundary and shares a protected power path [S03][S18]. Many industrial sites protect an edge server or wide-area router with an uninterruptible power supply, yet lose internal operations when the backhaul drops or a feeder trips because floor switches, address leases, name resolvers, broker credentials or operator consoles depend on unbacked distribution boards or external cloud endpoints [S03][S05]. Designing for intermittent connectivity starts on the local area network (LAN) before it touches a wide-area carrier. When local switching, naming, time distribution, transport sessions and maintenance access function without an internet route, the plant can record production events, display local alarms and coordinate floor workstations for hours or days until upstream links return [S06][S08].

¶ 01 · 5 sources

S03S18S05S06S08

This chapter defines the controls that keep a plant network operational during external disconnection and disciplined during link recovery. Following the power boundary of Chapter 4 and preceding the storage rules of Chapter 6, it maps external and electrical dependencies on the LAN [S03][S18], replaces advertised carrier bandwidth with useful application goodput measured at durable commit [S12][S24], compares transport protocols under channel loss [S09][S12][S19][S24], evaluates multi-carrier diversity through common-cause failure paths [S05][S18], bounds retry and backlog admission during recovery [S06][S19][S20], establishes controlled offline administration under cached identity and bounded clock uncertainty [S23][S27], and specifies a fault-injection qualification protocol before production cutover [S06][S28].

5.14 min · 2 sub-steps · 1 exhibit

Local operation begins with local dependencies

Related · chapter 117 paragraphs step it

Related exhibit 43 · chapter 11ScheduleillustrativeService restoration follows dependencies rather than a common restart command
02468Confirm stable supplyCheck storage and schemaEstablish clock and identityReady broker and captureQualify inspectionReconcile local ledgerAuthorise process releaseClose remote replay

Read-outConfirm stable supply: from 0 to 1 (illustrative dependency position, not elapsed time)

Method and source

Illustrative recovery order proposed by GFT. Equal numeric positions show precedence only and carry no measured or promised durations. Remote replay may transmit after local capture is ready, but its closure remains subject to reconciliation and release evidence.

Unit: illustrative dependency position, not elapsed time. Status: illustrative. Sources: S03, S05, S21. Illustrative figures are teaching examples, not measured factory results.

Read this exhibit in its chapter ↗
5.1 · Introduction · paragraph 1 of 7

Industrial digital systems frequently fail during a backhaul outage not because the application logic requires remote compute, but because basic network plumbing relies on external reachability or unprotected floor power [S03][S18]. In a typical commercial deployment, a barcode terminal or programmable logic controller gateway resolves an edge broker hostname through a domain name system (DNS) forwarder pointing to a public resolver, obtains its dynamic host configuration protocol (DHCP) lease from an office router, validates transport certificates against an online revocation responder, and authenticates operator sessions through a remote identity provider [S03][S23]. While the wide-area link is healthy and every distribution board is energised, this chain appears local to the shop floor. When upstream fibre is cut or a brownout drops an intermediate switch cupboard, the chain breaks inside the factory even though the edge server remains powered [S03][S05].

Keeping the local network useful requires treating the plant LAN as an autonomous survivorship zone with its own power, addressing, naming, time and session authorities [S03][S11]. NIST SP 800-82 Rev. 3 emphasises that operational technology networks must be segmented and engineered so that loss of external connectivity or corporate IT services does not impair essential local monitoring and control [S03]. Similarly, RFC 7228 notes that constrained networks combine node limitations with channel loss, unpredictable bandwidth and dynamic topology changes [S18]. As shown in Exhibit 18, every layer between the capture terminal and the local edge store must remain complete inside the site boundary when wide-area carriers are dark.

5.2 Distinguish a backed-up router from a protected local area network

A common procurement mistake on industrial sites is placing an uninterruptible power supply under the wide-area router and edge server while leaving intermediate access switches on raw lighting circuits [S03][S05]. Under Dependency Inference, a path is only as available as the minimum of its series elements. If a line scanner connects through an unpowered 8-port floor switch on a pillar, or through a wireless access point whose power-over-Ethernet (PoE) injector sits on an unprotected sub-board, the edge server is unreachable during a mains sag even if its own battery holds for 60 minutes (illustrative site configuration) [S03][S05].

Mapping the physical LAN path requires tracing every active device between the data source and the local store: media converters, PoE midspans, floor switches, wireless bridges and industrial firewalls [S03]. Each intermediate device must share the protected DC or UPS circuit of the service class it supports (Chapter 3) or be bypassed by a direct home-run cable to a protected switch [S03][S05]. Where a floor switch serves both essential terminals and non-essential office loads, shedding non-essential PoE ports during mains loss prevents peripheral devices from draining the switch backup supply (illustrative operational rule) [S03].

5.3 Keep addressing, naming and session establishment strictly internal

Even when every switch remains powered, software dependencies can sever local communication after a link failure or cold reboot. Four control-plane traps must be eliminated by design [S03][S11][S23]:

  1. Forwarding-only DNS resolution: If edge clients connect to broker.plant.example and the local DNS resolver acts only as a forwarding cache, cached records expire once their time-to-live (TTL) elapses [S03]. A 300-second TTL (illustrative setting) means that five minutes after the backhaul drops, any restarting client fails with a name-resolution error even though the broker sits on the same VLAN [S03]. The plant network must run a local authoritative DNS zone for internal service names, backed by static host entries on critical gateways [S03].
  2. Short DHCP leases without local authority: If dynamic addressing is served across a wide-area virtual private network (VPN) or by an unbacked office router, clients whose leases pass the 50 percent renewal (T1) and 87.5 percent rebinding (T2) marks drop their IP addresses upon lease expiry [S03]. Critical servers, brokers, controllers and consoles must use static IP assignments in the site address register, and mobile pools must be served by a protected local DHCP instance with 24 to 72 hour leases (illustrative target) [S03][S05].
  3. Synchronous online certificate revocation checks: Under TLS 1.3 (RFC 9846) [S23], clients or brokers configured to require a live Online Certificate Status Protocol (OCSP) responder or external certificate revocation list (CRL) abort the handshake when the WAN is down [S23]. Local TLS must validate against a locally pinned certificate authority bundle without mandatory online OCSP fetches [S03][S23].
  4. Cloud-redirected operator login: A local web console that redirects the browser to an external single sign-on domain or fetches scripts from public content delivery networks fails during the very outage the operator needs to diagnose [S03][S06]. The local console must authenticate against a local credential store and serve all user-interface assets locally [S03].

5.43 min · 2 sub-steps · read with a related exhibit

Measure useful application capacity rather than advertised bandwidth

Related · chapter 079 paragraphs step it

Related exhibit 27 · chapter 07FanderivedAn outage creates a backlog that a fast link must clear
02004006008000 h4 h8 h: link…10 h

Read-out0 h: Queued payload, C02 and C03 0 (decimal MB of queued payload)

Method and source

Derived from illustrative C02 and C03 assumptions: arrival 25,000 bytes/s for 28,800 s creates 720 MB. Useful restored capacity 100,000 bytes/s less live arrivals 25,000 bytes/s gives net drain 75,000 bytes/s and clearance 9,600 s. Elapsed labels are categorical and unequally spaced; 10.67 h is rounded from 10 h 40 min. Storage overhead and later interruptions are excluded.

Unit: decimal MB of queued payload. Status: derived. Sources: S08, S18.

Read this exhibit in its chapter ↗
5.4 · Introduction · paragraph 1 of 9

Carrier quotations state nominal peak bit rates, such as a 10 megabits per second (Mbit/s) wireless link or a 50 Mbit/s shared fibre circuit (illustrative offerings) [S18]. On a weak grid or across contended last-mile infrastructure, advertised link bandwidth is a poor predictor of whether an edge store can clear its backlog before the next outage window [S08][S18]. RFC 7228 warns that constrained networks exhibit limited, unpredictable bandwidth alongside lossy channels [S18]. Engineers must separate raw channel capacity from effective application goodput: the volume of unique, schema-valid application payload durably committed and acknowledged by the receiving endpoint per unit of elapsed wall-clock time [S12][S13][S24].

Between reading queued records from a local SQLite store and receiving remote confirmation of durable acceptance, a TCP stream (RFC 9293) must complete a three-way handshake, negotiate TLS 1.3 parameters (RFC 9846) and open its congestion window [S13][S23][S24]. Packet loss or wireless fading forces TCP to retransmit segments, reduce its congestion window and stall delivery until in-order sequence continuity is restored [S24]. If the bearer drops before the receiving application commits the batch and returns an application-layer confirmation (an HTTP 2xx status under RFC 9110 or an MQTT PUBACK under OASIS MQTT v5.0), the partially transmitted payload must be sent again when the link recovers [S12][S19][S24].

5.5 Define effective upload goodput across interruption cycles

Useful upload capacity over an observation window $T_{\text{window}}$ (such as a 1-hour recovery interval or 24-hour operating day, illustrative intervals) is defined at the application commit boundary [S12][S13]:

$$G_{\text{eff}} = \frac{B_{\text{committed}}}{T_{\text{window}}}$$

Here $B_{\text{committed}}$ is the total number of unique application payload bytes (excluding duplicate replays, aborted partial transfers, TLS record headers, TCP/IP headers and link framing) for which the local sender receives an explicit, durable commit acknowledgement during $T_{\text{window}}$ (illustrative formula convention) [S12][S19][S23][S24]. This metric penalises four sources of waste hidden by interface counters [S18][S24]:

  • Handshake tax: Link flaps force new DNS lookups, TCP handshakes and TLS 1.3 negotiations (1 round-trip time under RFC 9846) before application payload moves [S23][S24].
  • Transport retransmission and head-of-line blocking: TCP guarantees a reliable, in-order byte stream [S24]; packet loss generates duplicate segments that register as interface traffic without advancing application state [S24].
  • Uncommitted partial transfers: If a 5-megabyte inspection image (illustrative payload) fails at 90 percent completion without byte-range resumption, those 4.5 megabytes contribute zero to $B_{\text{committed}}$ [S12][S24].
  • Application-layer duplicate delivery: Under at-least-once delivery contracts (MQTT QoS 1 in OASIS MQTT v5.0 or retried HTTP PUT requests in RFC 9110), lost acknowledgements trigger retransmission of already-stored records, consuming uplink capacity even when idempotent deduplication preserves data integrity [S12][S19].

5.6 Keep six measurement series distinct during site surveys

No carrier performance figures are asserted in this paper for any region or operator, because links vary by tower load, weather, power state and time of day [S01][S18]. Before freezing batch sizes or backlog clearance budgets on a target site, the engineering team must instrument a local probe across working shifts and outage windows to record six distinct time series [S18][S24]:

  1. Physical and link-layer modulation rate (bit/s): Negotiated speed between the modem and first carrier hop, exposing signal fade or base-station power curtailment [S18].
  2. Network-layer throughput (bit/s): Total IP bytes per second across the WAN interface, including framing overhead and retransmissions [S24].
  3. Round-trip latency and jitter (milliseconds): Active probing to the target ingestion endpoint, bounding throughput when in-flight windows are small [S09][S24].
  4. Packet loss and connection reset rate (percentage and resets/hour): Lost probe fraction and frequency of TCP RST or bearer drops, governing batch size between interruptions [S18][S24].
  5. Useful application goodput ($G_{\text{eff}}$, bytes/s): Net rate of durably acknowledged unique payload bytes over 1-minute, 15-minute and 60-minute rolling windows (illustrative buckets) [S12][S19].
  6. End-to-end application commit delay (seconds): Elapsed time from local event creation to confirmed remote commit at the 50th, 95th and 99th percentiles (illustrative percentiles) [S12][S32].

5.74 min · 2 sub-steps · 1 exhibit

The transport protocol must fit device and failure behaviour

Step 1 of 6CoAP NON (UDP, RFC 7252)

Exhibit 19 · sourcedProtocol selection depends on the required delivery contract

Constrained datagram telemetry (RFC 7252)Store-and-forward brokered session (MQTT)Unacknowledged local beaconsStateless transactional batches (RFC 9110)Built-in session state and ordering (0 to 100) →Lossy and low-MTU link fit (0 to 100) →CoAP NON (UDP, RFC 72…CoAP CON (UDP, RFC 72…MQTT v5.0 QoS 0 (TCP…MQTT v5.0 QoS 1 (TCP…MQTT v5.0 QoS 2 (TCP…HTTP PUT/POST batches…
Constrained datagramtelemetry (RFC 7252)Store-and-forward brokeredsession (MQTT)Unacknowledged localbeaconsStateless transactionalbatches (RFC 9110)Built-in session state and ordering (0 to 100) →Lossy and low-MTU link fit (0 to 100) →CoAP NON (UDP, RFC 72…CoAP CON (UDP, RFC 72…MQTT v5.0 QoS 0 (TCP…MQTT v5.0 QoS 1 (TCP…MQTT v5.0 QoS 2 (TCP…HTTP PUT/POST batches…

CoAP NON (UDP, RFC 7252)

CoAP NON (UDP, RFC 7252): Built-in session state and ordering (0 to 100) 18, Lossy and low-MTU link fit (0 to 100) 88. Quadrant: Constrained datagram telemetry (RFC 7252) · sourced

No single transport protocol is universally superior across both the shop floor and the wide-area backhaul .

CoAP CON (UDP, RFC 7252)

CoAP CON (UDP, RFC 7252): Built-in session state and ordering (0 to 100) 38, Lossy and low-MTU link fit (0 to 100) 82. Quadrant: Constrained datagram telemetry (RFC 7252) · sourced

MQTT v5.0 QoS 0 (TCP, OASIS)

MQTT v5.0 QoS 0 (TCP, OASIS): Built-in session state and ordering (0 to 100) 52, Lossy and low-MTU link fit (0 to 100) 35. Quadrant: Stateless transactional batches (RFC 9110) · sourced

MQTT v5.0 QoS 1 (TCP, OASIS)

MQTT v5.0 QoS 1 (TCP, OASIS): Built-in session state and ordering (0 to 100) 78, Lossy and low-MTU link fit (0 to 100) 64. Quadrant: Store-and-forward brokered session (MQTT) · sourced

MQTT v5.0 QoS 2 (TCP, OASIS)

MQTT v5.0 QoS 2 (TCP, OASIS): Built-in session state and ordering (0 to 100) 92, Lossy and low-MTU link fit (0 to 100) 42. Quadrant: Stateless transactional batches (RFC 9110) · sourced

HTTP PUT/POST batches (RFC 9110)

HTTP PUT/POST batches (RFC 9110): Built-in session state and ordering (0 to 100) 65, Lossy and low-MTU link fit (0 to 100) 48. Quadrant: Stateless transactional batches (RFC 9110) · sourced

Method and source · Exhibit 19

Source-based comparison of protocol mechanics specified in RFC 7252 (CoAP, S09), RFC 9110 (HTTP Semantics, S12), OASIS MQTT v5.0 (S19) and RFC 9293 (TCP, S24). Coordinates represent relative design placement (0 to 100); exact wire specifications and GFT outbox recommendations are tabulated in the text.

Unit: relative design score (0 to 100). Status: sourced. Sources: S09, S12, S19, S24.

5.7 · the bodyExhibit 19 above; its figures are illustrative where marked.

No single transport protocol is universally superior across both the shop floor and the wide-area backhaul [S09][S12][S19]. Each protocol embodies specific assumptions about transport reliability, framing overhead, state retention, message ordering and congestion control [S09][S12][S19][S24]. A protocol suited to 32-byte sensor readings on a local mesh can perform poorly when draining 50,000 queued production records over a flapping cellular link (illustrative workload contrast) [S09][S18][S19]. As mapped in Exhibit 19, selection depends on the constraint and loss profile of the network path and the delivery contract required between endpoints [S09][S12][S19][S24].

¶ 01 · 5 sources

A protocol suited to 32-byte sensor readings on a local mesh can perform poorly…S09, S18, S19 · illustrative

Exhibit 19Related · Exhibit 25 · chapter 06

S09S12S19S24S18

5.8 Compare wire mechanics, retry semantics and overhead from primary specifications

5.8

Engineers must separate what each standard guarantees on the wire from what the surrounding storage architecture must provide [S09][S12][S13][S19]:

¶ 02 · 4 sources

S09S12S13S19

  • Constrained Application Protocol (CoAP, RFC 7252): Designed for constrained nodes and lossy channels [S09][S18], baseline CoAP binds to UDP with a 4-byte fixed header, a 0 to 8 byte Token, options and payload [S09]. CoAP defines Confirmable (CON), Non-confirmable (NON), Acknowledgement (ACK) and Reset (RST) messages matched by a 16-bit Message ID [S09]. For CON messages, RFC 7252 specifies stop-and-wait retransmission with exponential backoff under default parameters (ACK_TIMEOUT = 2 s, ACK_RANDOM_FACTOR = 1.5, MAX_RETRANSMIT = 4, NSTART = 1, DEFAULT_LEISURE = 5 s, PROBING_RATE = 1 byte/s, MAX_TRANSMIT_SPAN = 45 s and MAX_TRANSMIT_WAIT = 93 s) and recommends fitting messages inside a single IP packet (1,280-byte IP MTU, 1,152-byte message bound and 1,024-byte payload upper bound, or 127-byte IEEE 802.15.4 frames) [S09]. CoAP fits constrained local radio links, but NSTART = 1 and the 1,024-byte payload bound limit bulk WAN replay without block-wise tuning [S09].
  • Message Queuing Telemetry Transport (MQTT v5.0, OASIS Standard): Operating over TCP/IP (RFC 9293), MQTT uses a 2-byte minimum fixed header (1 control byte plus a 1 to 4 byte Variable Byte Integer encoding Remaining Length up to 268,435,455 bytes) [S19][S24]. OASIS MQTT v5.0 defines three Quality of Service levels: QoS 0 ("At most once"), QoS 1 ("At least once", via PUBLISH and PUBACK where duplicates can occur) and QoS 2 ("Exactly once", via PUBLISH, PUBREC, PUBREL, PUBCOMP) [S19]. It also defines Clean Start and Session Expiry Interval for session state, Receive Maximum to cap concurrent unacknowledged QoS 1/2 packets, and Topic Alias to compress repeated topic strings into a 2-byte integer [S19].
  • HTTP Semantics (RFC 9110): HTTP provides stateless request-response semantics over TCP and TLS [S12][S23][S24]. RFC 9110 specifies that GET, HEAD, OPTIONS and TRACE are safe, while PUT, DELETE and all safe methods are idempotent: a client can safely repeat an unacknowledged PUT batch after a connection reset [S12]. POST is not idempotent unless keyed by an application UUID [S12][S31]. HTTP also supports compression (Content-Encoding) and byte-range resumption (Range) [S12].
ProtocolFramingOrdering and deliveryFlow controlWeak-grid strengthsMisapplication traps
CoAP [S09]UDP; 4-byte header, 0 to 8 byte Token, 1,024-byte payload bound [S09]CON/ACK stop-and-wait; no cross-datagram ordering [S09]ACK_TIMEOUT = 2 s, MAX_RETRANSMIT = 4, NSTART = 1 [S09]No TCP head-of-line blocking on lossy radio links [S09][S18]NSTART = 1 and 1,024-byte bound stall bulk replay [S09]
MQTT v5.0 [S19]TCP (RFC 9293); 2-byte min header plus 1 to 4 byte length [S19][S24]Ordered per TCP stream; QoS 0, QoS 1 (PUBACK) or QoS 2 [S19][S24]Receive Maximum, Session Expiry Interval, Topic Alias [S19]Persistent pub/sub session; low overhead; local queuing [S19][S20]PUBACK confirms broker receipt, not database commit [S13][S19]
HTTP [S12]TCP (RFC 9293) and TLS 1.3 (RFC 9846); headers and body [S12][S23][S24]Ordered per stream; explicit status codes (2xx, 4xx, 5xx) [S12][S24]Idempotent PUT/DELETE/GET; Range resumption [S12]End-to-end confirmation from target API; range tooling [S12]Verbose per-record headers; POST duplicates without UUID [S12][S31]

5.9 Protocol delivery is hop-by-hop, not end-to-end durable ingestion

5.9

Assuming that MQTT QoS 1/2 or CoAP CON guarantees end-to-end database durability is an architectural error [S09][S13][S19]. Both protocols define delivery across a single hop [S09][S19]. In MQTT, the broker returns PUBACK as soon as it accepts the packet under its local configuration [S19]. If Eclipse Mosquitto runs with persistence false in mosquitto.conf, or flushes state to mosquitto.db only on periodic autosave_interval cycles, a power cut on the broker host immediately after PUBACK loses the message after the publisher has discarded its copy [S19][S20]. Even when mosquitto.db is persisted, a bridge worker that crashes after acknowledging the broker packet but before committing to its SQLite store drops the record [S13][S19][S20].

¶ 03 · 4 sources

Assuming that MQTT QoS 1/2 or CoAP CON guarantees end-to-end database durability is an architectural errorS09, S13, S19

S09S13S19S20

GFT's reference architecture separates local collection from wide-area synchronisation (GFT architectural recommendation) [S08][S13]. Inside the plant LAN, use MQTT v5.0 QoS 1 over protected Ethernet/Wi-Fi (with Mosquitto persistence true and an explicit persistence_location) or CoAP CON for constrained sensor sub-networks [S09][S19][S20]. Across the intermittent WAN, commit records atomically to a local SQLite outbox (Chapter 6) and drain that outbox using batched, compressed, idempotent HTTP PUT requests (RFC 9110) or MQTT v5.0 QoS 1 batches paired with an explicit application-level commit receipt naming the committed event UUIDs [S12][S13][S19][S31].

¶ 04 · 7 sources

idempotent HTTP PUT requests (RFC 9110) or MQTT v5.0 QoS 1 batches paired with…S12, S13, S19, S31

S08S13S09S19S20S12S31

Interlude · the cloth half-woven

4 exhibits carry this chapter's argument.

Woven: sections 5.0 to 5.7Still to weave: 5.10 to 5.18

  1. 18Local work can survive the loss of an external routeLayers
  2. 19Protocol selection depends on the required delivery contractDecision map
  3. 20Common-cause failures defeat naive dual-carrier availability mathRegister
  4. 21Backhaul restoration is a sequence of dependenciesTimeline
Hover or focus an exhibit to light its weft; select it to jump to it.

5.103 min · 1 sub-steps · 1 exhibit

A second carrier is useful only if its failure path differs

When a single wide-area connection proves unreliable, the standard response is to buy a second subscription from another operator or insert a second cellular SIM into a dual-WAN router [S05]. Decision-makers are often shown a naive calculation: if Carrier A and Carrier B each offer 95 percent standalone availability, combining them in active-passive failover supposedly yields $1 - (0.05 \times 0.05) = 99.75\%$ availability, cutting expected downtime from 36 hours per month to 1.8 hours per month (illustrative naive calculation) [S05].

¶ 01 · 1 source

if Carrier A and Carrier B each offer 95 percent standalone availabilityS05 · illustrative

Related · Exhibit 47 · chapter 12

S05

On a weak grid, that multiplication is invalid because the two carriers are not statistically independent [S05][S18]. Reliability engineering separates independent random faults from common-cause failures where a single physical or electrical event disables both paths simultaneously [S05]. NIST SP 800-34 Rev. 1 emphasises that telecommunications contingency planning must evaluate physical routing diversity, shared central offices, local loop dependencies and backup power at intermediate facilities [S05]. Two fibres in the same street trench fail under the same excavator bucket; two mobile operators on the same roadside mast fail together once a prolonged feeder outage exhausts the mast's shared rectifier batteries [S01][S05][S18].

¶ 02 · 3 sources

NIST SP 800-34 Rev. 1 emphasises that telecommunications contingency planning must evaluate physical routing diversityS05

S05S18S01

5.11 Audit five common-cause failure domains before buying a backup link

5.11

Before contracting a secondary backhaul, engineers must require written path disclosures and conduct a physical walkthrough across five failure domains [S03][S05][S18]:

¶ 03 · 3 sources

S03S05S18

  1. On-site termination and power boundary: Terminating both links in one dual-SIM router fed by a single power brick creates a single point of failure; separate primary and secondary modems onto distinct interfaces powered from fused branches of the essential UPS bus [S03][S05].
  2. Last-mile physical exit and civil route: Trace wired conduits from the patch panel to the street chamber; cables sharing a trench, culvert or pole line are vulnerable to a single civil cut [S05].
  3. Radio base station and tower power reserve: Record the physical cell ID (PCI) or tower coordinates to verify whether operators co-locate on the same mast and how long tower batteries last during blackouts [S01][S05][S18].
  4. Regional backhaul and metropolitan aggregation: Retail service providers frequently lease wholesale metro transport from the same tier-1 fibre carrier or exchange point (IXP) [S05].
  5. Logical and contractual failure modes: Shared data caps, simultaneous quota exhaustion or reliance on the same upstream DNS resolver or VPN concentrator can silence both links together [S03][S05].
List · 4 sources

S03S05S01S18

As shown in Exhibit 20, common-cause downtime dominates combined wide-area unavailability whenever physical or power paths overlap.

Exhibit 20RegisterderivedCommon-cause failures defeat naive dual-carrier availability math
Diversity scenario
1. Dual SIMs, same mast and grid feeder
Carrier A (h/mo)
36.0 (5.0%)
Carrier B (h/mo)
36.0 (5.0%)
Shared fault (h/mo)
24.0 (3.33%)
Naive math (h/mo)
1.8 (0.25%)
Actual dual-WAN (h/mo)
24.2 (3.36%)

Read-outDiversity scenario: 1. Dual SIMs, same mast and grid feeder. Carrier A (h/mo): 36.0 (5.0%). Carrier B (h/mo): 36.0 (5.0%). Shared fault (h/mo): 24.0 (3.33%). Naive math (h/mo): 1.8 (0.25%). Actual dual-WAN (h/mo): 24.2 (3.36%)

Open the full register (4 rows)
1. Dual SIMs, same mast and grid feeder36.0 (5.0%)36.0 (5.0%)24.0 (3.33%)1.8 (0.25%)24.2 (3.36%)
2. Two retail ISPs in shared street trench21.6 (3.0%)28.8 (4.0%)14.4 (2.00%)0.9 (0.12%)14.6 (2.02%)
3. Fibre + cellular on shared metro backbone21.6 (3.0%)36.0 (5.0%)7.2 (1.00%)1.1 (0.15%)7.9 (1.09%)
4. Terrestrial fibre + UPS-backed satellite21.6 (3.0%)14.4 (2.0%)0.7 (0.10%)0.4 (0.06%)1.1 (0.15%)
Method and source

Illustrative reliability derivation applying NIST SP 800-34 Rev. 1 telecommunications diversity principles (S05) and RFC 7228 channel constraints (S18). Actual combined downtime equals shared downtime plus independent simultaneous downtime. All values are illustrative scenarios.

Unit: hours per 720-hour month (illustrative). Status: derived. Sources: S05, S18.

Note · 4 sources

S01S05S18S08

5.123 min · 2 sub-steps · 1 exhibit

Retry policy must preserve capacity for urgent work

Step 1 of 7Bearer and IP route restored

Exhibit 21 · illustrativeBackhaul restoration is a sequence of dependencies

0Bearer and IProute restored1Control-planecheck (DNS and NT…2Jittered TLS 1.3session setup3Lane P0 livealarms admitted4Link stabilitysoak probe5Lane P1 throttledoutbox drain6Lanes P2 and P3background sync
0Bearer andIP route…1Control-planecheck (DN…2JitteredTLS 1.3 s…3Lane P0live alar…4Linkstability…5Lane P1throttled…6Lanes P2and P3 ba…

Bearer and IP route restored

0: Bearer and IP route restored. WAN modem re-establishes bearer; router verifies gateway without disturbing LAN. · illustrative

The most fragile moment for a weak-grid network is not the middle of a disconnection, when local services run inside the LAN, but the first ten minutes after a degraded wide-area link returns .

Control-plane check (DNS and NTS)

1: Control-plane check (DNS and NTS). Gateway validates upstream DNS and checks NTS/NTP offset without stepping clocks. · illustrative

Jittered TLS 1.3 session setup

2: Jittered TLS 1.3 session setup. Coordinator applies random jitter and completes TCP and TLS 1.3 negotiation. · illustrative

Lane P0 live alarms admitted

3: Lane P0 live alarms admitted. Current plant state, active safety alarms and queue depth metrics transmit first. · illustrative

Link stability soak probe

4: Link stability soak probe. Coordinator observes packet loss and RTT over a 30 to 120 s soak window. · illustrative

Lane P1 throttled outbox drain

5: Lane P1 throttled outbox drain. Committed SQLite outbox batches drain using bounded in-flight windows (1 to 5). · illustrative

Lanes P2 and P3 background sync

6: Lanes P2 and P3 background sync. Historical sensor series and resumable HTTP byte-range blobs transfer last. · illustrative

Method and source · Exhibit 21

Illustrative recovery choreography synthesised from NIST SP 800-184 recovery prioritisation (S06), RFC 9293 TCP connection semantics (S24) and RFC 9846 TLS 1.3 handshake requirements (S23). Numeric positions represent dependency order (0 to 6) rather than fixed wall-clock minutes.

Unit: relative recovery step index (0 to 6). Status: illustrative. Sources: S06, S23, S24. Illustrative figures are teaching examples, not measured factory results.

5.12 · the bodyExhibit 21 above; its figures are illustrative where marked.

The most fragile moment for a weak-grid network is not the middle of a disconnection, when local services run inside the LAN, but the first ten minutes after a degraded wide-area link returns [S06][S18]. During a four-hour outage, edge gateways, floor terminals and diagnostic agents accumulate thousands of queued production events, telemetry points, journal entries and images (illustrative backlog) [S08][S22]. If every process detects link restoration simultaneously and flushes its backlog at maximum concurrency, the resulting reconnection and replay storm saturates uplink buffers, inflates round-trip latency into multiple seconds, triggers TCP retransmissions and TLS timeouts, and starves live high-priority alarms [S06][S19][S24]. NIST SP 800-184 emphasises that recovery planning must establish prioritised restoration sequences so critical operational functions recover before secondary workloads [S06]. On a factory network, that requires traffic prioritisation, bounded in-flight concurrency and jittered exponential backoff [S09][S19][S20].

¶ 01 · 8 sources

NIST SP 800-184 emphasises that recovery planning must establish prioritised restoration sequences so critical…S06

Related · Exhibit 39 · chapter 10

S06S18S08S22S19S24S09S20

5.13 Separate traffic lanes by operational urgency and payload size

5.13

When the wide-area link transitions from DOWN to DEGRADED or UP, outbound traffic from the plant edge must pass through a scheduler or broker bridge enforcing four priority lanes (illustrative GFT scheduling policy aligned with Chapter 3 service classes) [S03][S06]:

¶ 02 · 2 sources

S03S06

  • Lane P0 (Live safety, security and plant health alarms): Compact state transitions, threshold breaches and heartbeat summaries (under 1 to 2 kilobytes per message, illustrative size), prioritised ahead of all queued history so remote supervisors see current plant state within seconds [S03][S06].
  • Lane P1 (Current and queued transactional production records): Idempotent records of weighbridge tickets, batch completions and operator sign-offs, drained in watermarked batches once Lane P0 is clear [S12][S13].
  • Lane P2 (Historical environmental telemetry and diagnostic summaries): High-frequency sensor series downsampled or batched into compressed chunks [S18].
  • Lane P3 (Bulk binary blobs, camera images, full logs and firmware pulls): Multi-megabyte payloads scheduled only when P0 and P1 queues sit below low-water marks, using resumable HTTP Range requests (RFC 9110) so link flaps do not waste transferred bytes [S12][S22].
List · 6 sources

S03S06S12S13S18S22

5.14 Bound in-flight concurrency and apply jittered backoff

5.14

Pipelining hundreds of unacknowledged requests across a thin wide-area link merely moves the queue from the local edge disk (where it can be inspected and reprioritised) into carrier router buffers (where it causes head-of-line blocking) [S18][S24]. Edge configurations must enforce explicit concurrency bounds [S09][S19][S20]:

¶ 03 · 5 sources

S18S24S09S19S20

  • Bound in-flight messages on MQTT bridges: OASIS MQTT v5.0 provides the Receive Maximum property to cap concurrent unacknowledged QoS 1/2 PUBLISH packets [S19]. In Eclipse Mosquitto (mosquitto.conf), max_inflight_messages defaults to 20 (max_inflight_bytes defaults to 0, and max_queued_messages defaults to 1000) [S20]. On a lossy WAN bridge, tightening max_inflight_messages to 1 to 5 with an explicit byte cap (illustrative tuning) prevents a stalled batch from filling kernel TCP buffers [S19][S20][S24].
  • Apply exponential backoff with random jitter: RFC 7252 sets CoAP NSTART = 1 by default and chooses initial Confirmable (CON) retransmission timeouts randomly between ACK_TIMEOUT (2 seconds) and ACK_TIMEOUT * ACK_RANDOM_FACTOR (3 seconds), doubling up to MAX_RETRANSMIT (4 retries) [S09]. Custom HTTP or TCP reconnect loops must apply equivalent jittered backoff so rebooting devices do not hammer gateways in lockstep [S09][S12].
  • Gate bulk replay behind a link stability soak timer: Because modems often flap repeatedly during carrier fault clearing [S18], the edge synchroniser should verify DNS and time, send a P0 status beacon, and observe a configurable soak window (such as 30 to 120 seconds of stable reachability, illustrative policy) before opening P1/P2 backlog drain, as shown in Exhibit 21 [S06][S23][S24].

5.153 min · 2 sub-steps · read with a related exhibit

Local administration must remain controlled while offline

When a factory is cut off from corporate headquarters and cloud portals for thirty-six hours, local technicians still need to inspect queue depths, replace a failed scanner, rotate logs or restart a wedged PLC connector [S03][S06]. If the security architecture requires real-time authorisation from a cloud identity provider or remote bastion host, the site team faces an unacceptable choice during an outage: let production stall, or bypass security controls by enabling shared root passwords and opening unauthenticated management ports [S03]. NIST SP 800-82 Rev. 3 requires that operational technology maintenance and access controls account for isolated or degraded network modes without sacrificing least privilege, authentication or auditability [S03]. Offline administration must therefore be engineered as a permanent, auditable operating mode [S03][S06].

¶ 01 · 2 sources

NIST SP 800-82 Rev. 3 requires that operational technology maintenance and access controls account…S03

Related · Exhibit 42 · chapter 10

S03S06

5.16 Design an auditable local maintenance path and offline runbook

5.16

A disciplined offline administration architecture rests on four local controls [S03][S06][S22][S23]:

¶ 02 · 4 sources

S03S06S22S23

  1. Pre-provisioned, role-scoped local maintenance accounts: Provision individual named accounts or locally verifiable SSH/TLS client certificates signed by the plant's internal authority so every offline action is attributable to an individual technician [S03][S23].
  2. Locally hosted documentation and runbooks: Serve a read-only local HTML or PDF copy of the network diagram, IP address register, LED fault codes, safe-shutdown procedure and recovery playbook directly from the local appliance alongside a physical cabinet binder [S05][S06].
  3. Persistent, bounded local audit logging: Record every offline login, configuration change or queue action locally and queue audit records for upstream sync [S03][S22]. On Linux gateways using systemd-journald, journald.conf must set Storage=persistent (writing to /var/log/journal rather than volatile /run/log/journal), enforce SystemMaxUse= and SystemKeepFree= (defaulting to 10 percent and 15 percent of the file system, capped at 4 gigabytes) so logs cannot exhaust disk space, and enable Seal=yes for tamper-evident Forward Secure Sealing [S22]. Because journald flushes CRIT, ALERT and EMERG messages immediately while syncing lower priorities over SyncIntervalSec= (default 5 minutes), avoid ForwardToConsole=yes on slow serial consoles where a hung console can block journald [S22].
  4. No perimeter rule weakening during outages: Never expose management listeners (SSH, HTTPS, database or broker admin ports) to untrusted Wi-Fi or cellular interfaces when cloud tunnels fail; restrict maintenance to designated local management ports [S03].
List · 5 sources

S03S23S05S06S22

5.17 Handle certificate validity and time security under prolonged disconnection

5.17

Under TLS 1.3 (RFC 9846), certificate validation checks that wall-clock time falls between notBefore and notAfter [S23]. Similarly, Network Time Security (NTS, RFC 8915) uses TLS-based NTS Key Establishment (NTS-KE) to verify certificates and issue AEAD cookies for authenticated NTPv4 time exchange [S10][S27]. On a weak grid, two time-related failure modes must be prevented [S11][S23][S27]:

¶ 03 · 4 sources

Under TLS 1.3 (RFC 9846)S23

S23S10S27S11

  • Clock reset on cold boot without battery-backed real-time clock (RTC): Gateways lacking a monitored RTC battery may cold-boot after a blackout with system time reset to the Unix epoch (1970-01-01T00:00:00Z in RFC 3339 format), causing every valid TLS 1.3 certificate to fail as "not yet valid" (current_time < notBefore) and breaking local HTTPS consoles, MQTT-over-TLS and NTS-KE simultaneously [S23][S27][S32]. Every local time server and edge gateway must include a monitored hardware RTC (or local GNSS receiver) and persist the latest timestamp to disk as a monotonic floor across reboots [S11][S27].
  • Short-lived certificate expiration during multi-day cuts: If a site uses 7-day leaf certificates (illustrative policy) and a fibre cut isolates the plant for 10 days, local TLS handshakes fail mid-outage [S23]. Certificate lifetimes and renewal triggers on intermittent sites must exceed the maximum credible disconnection window by a wide margin (such as 90-day to 365-day validity renewed at one-third elapsed lifetime, illustrative policy) [S03][S23].
List · 5 sources

S23S27S32S11S03

Note · 5 sources

S20S23S03S10S11

5.183 min · 2 sub-steps · read with a related exhibit

Network qualification should reproduce the actual failure modes

A factory network designed for weak-grid resilience cannot be qualified by checking that ping succeeds on a sunny afternoon when mains power and fibre backhaul are healthy [S06][S28]. Just as SQLite's testing guide notes that power-loss durability requires simulated crash testing across the exact failure boundary [S28], qualifying a plant network requires deliberate fault injection that reproduces weak-grid failure modes and records evidence in both local store logs and remote ingestion audit trails [S06][S22][S28].

5.19 Seven mandatory fault-injection tests for site acceptance

5.19

The site acceptance test (SAT) checklist must pass seven fault scenarios without losing a single locally committed record or requiring an unscripted process restart [S06][S13][S28]:

¶ 02 · 3 sources

S06S13S28

  1. Hard WAN demarcation pull: Unplug all WAN cables and cellular antennas during active floor capture [S06]. Pass criteria: Local capture latency, broker delivery and console loads show zero errors while the local outbox increments cleanly [S03][S19].
  2. Cold-start LAN boot while WAN is dark: With all WAN links unplugged, cold-boot floor switches, capture terminals, local DNS/DHCP, local NTP server and edge gateway [S06][S28]. Pass criteria: Every device obtains its IP, resolves internal names, syncs to local time, completes local TLS handshakes and resumes capture within the recovery time objective (under 5 minutes, illustrative target) with zero external packets [S03][S11][S23].
  3. Blackhole routing and zombie TCP session test: Keep the local WAN interface electrically UP while dropping 100 percent of packets two hops upstream [S24]. Pass criteria: The edge synchroniser detects application-layer timeout, marks the path degraded and continues local capture without hanging threads [S12][S24].
  4. Upstream DNS and NTP silence: Block upstream DNS (port 53) and NTP/NTS-KE (ports 123 and 4460) while leaving IP routing open [S10][S27]. Pass criteria: Internal names resolve authoritatively, the local clock enters holdover without stepping, and the console raises a DNS_FAILURE diagnostic alert [S03][S11].
  5. Degraded, high-loss link simulation: Inject 15 percent packet loss, 600 milliseconds delay, 200 milliseconds jitter and a 64 kilobits per second cap on the WAN test port (illustrative degraded parameters) [S18][S24]. Pass criteria: Bounded in-flight windows prevent bufferbloat, Lane P0 alarms arrive within budget, Lane P3 bulk transfers pause, and zero duplicate records commit remotely [S12][S19][S24].
  6. Reconnection storm and backlog drain test: Accumulate a synthetic backlog of 10,000 production records and 50 diagnostic attachments (illustrative test load) offline, then restore the WAN under a 256 kilobits per second ceiling while generating live P0 alarms [S06][S28]. Pass criteria: Live P0 alarms arrive ahead of historical P1/P2 batches and the backlog drains monotonically following Exhibit 21 [S06][S19].
  7. Dual-carrier failover and failback under load: Trip the primary WAN mid-batch to force failover to the secondary carrier, then restore the primary path 10 minutes later (illustrative timing) [S05][S06]. Pass criteria: Interrupted batches retry idempotently without duplicating records, and failback respects a hold-down timer [S05][S12].

5.20 Require an explicit rollback plan for every network change

5.20

Pushing a mistyped firewall rule, VLAN tag or static route over a remote link can sever both the backhaul and the shop-floor LAN [S03][S06]. Every change to routing, DNS, DHCP, firewall rules or broker bridges must therefore use an automated commit-confirm rollback timer that reverts to the prior known-good configuration if local and upstream health checks fail to confirm within 5 minutes (illustrative safeguard) [S03][S06]. While full cross-system enterprise commissioning is addressed in wp18, no plant LAN should enter service until it passes the local autonomy and degraded-backhaul qualification tests above [S06].

¶ 01 · 2 sources

…checks fail to confirm within 5 minutes (illustrative safeguard)S03, S06 · illustrative

S03S06

In summary, a factory network on a weak grid survives external disconnection when every dependency required for local capture, addressing, naming, time, security and operator visibility resides inside the plant's protected power boundary, when transport protocols and retry policies match actual channel loss and preserve headroom for live alarms, and when dual-carrier paths and recovery sequences are qualified by physical fault injection rather than brochure math [S03][S05][S06][S18]. With the local network kept alive and disciplined, the next requirement is ensuring that data arriving over that network is committed to non-volatile storage before any sender is told the record is safe, which is the subject of Chapter 6 [S13].

¶ 02 · 5 sources

S03S05S06S18S13

Check the record

The chapter keeps its sources within reach.

Every knot is a source this chapter cites; select one to read its record, or filter the register by grade. Publication dates and inherited-source limits stay explicit. Read the paper's full evidence summary.

20 sources · 368 citation marks on this pageMost cited: S03, 45 times

Grade AGrade BGrade CKnot size: citations on this page

All 20 records shown

c05-S01 · Grade A

World Bank

Enterprise Surveys: Cameroon 2024 Country Profile

All indicators are based on the responses of firms.

2025. Register access date: 2026-10-30. Source provenance is supplied by the paper evidence register.

Open the recorded source ↗
Chapter source notes

S03 NIST. 2023.

S18 IETF / RFC Editor. 2014.

S05 NIST. 2010.

S06 NIST. 2016.

S08 IETF / RFC Editor. 2007.

S12 IETF / RFC Editor. 2022.

S24 IETF / RFC Editor. 2022.

S09 IETF / RFC Editor. 2014.

S19 OASIS. 2019-03-07.

S20 Eclipse Mosquitto. 2026-10-30.

S23 IETF / RFC Editor. 2026.

S27 IETF / RFC Editor. 2020.

S28 SQLite. 2026-10-30.

S11 IETF / RFC Editor. 2019.

S13 SQLite. 2026-10-30.

S01 World Bank. 2025.

S32 IETF / RFC Editor. 2002.

S31 IETF / RFC Editor. 2024.

S22 systemd project. 2026-10-30.

S10 IETF / RFC Editor. 2010.

Keep the thread · chapter 05 of 14

Chapter 06 picks up the thread.

Move the shuttle along the rack to preview any chapter: its sections, exhibits and reading time. Draft web edition. This research is not investment, legal or tax advice; illustrative models are not measured factory results.

The paper's rack · 14 boltsBolt length: reading time

06Part II: Keep the local record alive

The local event must survive the acknowledgement

The transaction boundary, storage path and acknowledgement semantics decide whether a production record survives abrupt loss.

  • 6.1An event envelope should carry enough context to recover
  • 6.3Write the event and outbox in one transaction
  • 6.6Consistency does not guarantee the latest commit is durable
  • 6.10Keep a local database on a supported storage path
Reading time
40 min
Exhibits
5
Sections
7
Continue to chapter 06
← Previous: chapter 04, A protected digital circuit should have a measured dutyContinue the threadChapter 06 · The local event must survive the acknowledgement

Find a thread.

Type to search.