The easiest way to support a device you cannot physically reach is to leave yourself a way in. A reverse tunnel. The device dials out to a rendezvous server, and now you have a terminal into a box behind customer NAT without opening any inbound port. It is a genuinely good pattern. An earlier version of this codebase shipped exactly that. A masked-by-default agent you could turn on for a while when something broke. It works. I took it out on purpose.
The uncomfortable question is not "can I reach the device." It is "what did I just install in a customer's plant." A resident remote-shell agent is a login surface that lives on a box inside someone else's OT network, dialing out to a server I run. Even masked by default, even with a TTL, it exists. If it can be enabled, it can be enabled by the wrong person, or left on, or become the interesting thing an attacker pivots through. The convenience is real and the surface is real. For a device whose whole job is to sit quietly next to a PLC and be trustworthy, I decided the surface was not worth the convenience.
So the rule became blunt. Gateways run no remote-login service. Not disabled by default. Not break-glass. None. That is a constraint you set once and then have to live with, which is the hard part, because now support has to work without the shell you just gave up.
The replacement is a ladder, not a door. The first rung is diagnostics verbs. A small, typed set of things the device will tell you about itself. Link health, last uplink, tag freshness, modem signal. Delivered over the command channel it already uses, not an open prompt where you can run anything. Most "what is wrong with it" questions get answered here without touching the box. The second rung is a support dongle that bridges the on-device console only while it is physically plugged in. Real access, but it needs a person standing at the device. That changes the threat model completely. There is no persistent outbound tunnel to abuse, because access does not exist unless someone is there. The last rung is a device swap. It is cattle, so a truly dead box gets replaced, not nursed back over SSH.
The command channel under all of this matters more than it looks. Commands go into a typed inbox with a TTL, and it converges offline. The device pulls its instructions, they expire if they are stale, and a box that was unreachable for a day catches up cleanly when it comes back instead of running a pile of hours-old orders. That is the part I would underline. Remote support for an edge fleet is a messaging problem, not a terminal problem. Once commands are typed, expiring, and idempotent, you stop needing a shell for the things you were using the shell for.
What I would check, designing this again: for every support scenario, can it be handled without an interactive login? If yes, add a verb. If it genuinely needs hands, does it need my hands remotely or someone's hands locally? Most of the time it is the second one, and the honest answer is a dongle and a runbook, not a tunnel.
The surprise was how much removing the shell clarified the product. When the escape hatch is gone, you have to make the normal path good. The diagnostics have to actually diagnose. The config push has to actually be authoritative. The command inbox has to actually converge. You cannot quietly SSH in and paper over the gaps. The shell was not just a risk. It was a crutch that let the real interfaces stay weak.
Notes for next time: treat every persistent remote-access mechanism as something you will have to defend in a security review, because you will. Prefer access the customer can physically revoke, like unplugging a dongle, over access only you can, like remembering to disable an agent. Build the support ladder first, diagnostics, physical bridge, swap, and you will usually find you do not need a shell. And write the threat model down, because "no remote-login surface" only survives if it is a documented decision, not a mood.