Make the remote browser die with the run
The leash that ties the Windows browser to the run never held. Three findings, each measured: - stdin is not a signal: only the first chunk a caller writes reaches the remote command, so the stdin heartbeat looked like a dead caller and killed a run at check 44. - a dropped SSH connection does not kill the remote command; Windows OpenSSH leaves it running. Six orphaned chrome.exe survived a SIGKILLed tunnel -- this is what locked up the desktop. - Get-NetTCPConnection goes through WMI, which is denied to the non-admin test account; the broken CIM session took a browser down mid-run. The launcher now watches the forwarded CDP port with .NET sockets: a connected client means the run is alive, 45 s without one ends it. A job object with KILL_ON_JOB_CLOSE and a 30-minute cap catch the rest, and the caller shoots the browser on the way out instead of waiting for the idle window. Measured: survives 80 s of an idle held connection, gone 44 s after kill -9 on node, immediately on an ordered end. Full suite 267 checks in 128 s and 24 smoke checks in 16 s against an RTX 4070, with this machine at load 0.04.
This commit is contained in:
1 parent
9bc109518e
commit
47492bcbe8
3 files changed
+125
-21
No files matched your search
@@ -420,14 +420,24 @@ all, so the app never renders.)
|
||||
**`GM_REMOTE=user@host` moves the browser to another machine** – `tools/lib/browser.mjs` starts it
|
||||
over SSH and forwards its CDP port back to `127.0.0.1`, so the harness itself is unchanged. The
|
||||
checks load the public base URL, so the remote side needs only a browser and internet: no repo, no
|
||||
node. Setup is an SSH key (`GM_REMOTE_KEY`, default `~/.ssh/gm-win.key`). `tools/remote/gm-browser.ps1`
|
||||
is copied over on every run and ties the browser to the SSH session – closing stdin kills it, however
|
||||
the run ends; without that leash a dropped connection left eight orphaned `chrome.exe` behind.
|
||||
node. Setup is an SSH key (`GM_REMOTE_KEY`, default `~/.ssh/gm-win.key`). Measured against a Windows
|
||||
desktop with an RTX 4070: **267 checks in 128 s and 24 smoke checks in 16 s, with this machine idle**
|
||||
(0.4 s of CPU for the whole smoke run, load average 0.04).
|
||||
|
||||
Local CPU during a remote run is genuinely idle, but **the last remote attempt was not usable**: a
|
||||
single CDP call cost ~5 ms at first, then the tunnel degraded until the suite crawled at ~25 s per
|
||||
check. Since the GPU flags fixed the local cost, remote is a fallback that has not been made to work
|
||||
end to end – treat it as unfinished, not as a supported path.
|
||||
`tools/remote/gm-browser.ps1` is copied over on every run, and getting the browser to die with the
|
||||
run took three tries, because two obvious leashes do not hold on Windows:
|
||||
|
||||
- **stdin is not a signal.** Only the first chunk a caller writes ever reaches the remote command,
|
||||
so a heartbeat over stdin is indistinguishable from a dead caller – it killed a run at check 44.
|
||||
- **A dropped connection does not kill the remote command.** Windows OpenSSH leaves it running;
|
||||
measured, six orphaned `chrome.exe` survived a SIGKILLed tunnel.
|
||||
|
||||
What holds is the CDP port: while a run is under way the forwarded connection is established, and it
|
||||
closes the moment the run or the tunnel dies. The launcher watches for that with .NET rather than
|
||||
`Get-NetTCPConnection` – that one goes through WMI, which is denied to a non-admin account, and its
|
||||
broken CIM session took a browser down mid-run. A job object with `KILL_ON_JOB_CLOSE` catches the
|
||||
rest, and a 30-minute cap bounds anything still unforeseen. Measured: the browser survives 80 s of an
|
||||
idle held connection and is gone 44 s after the local node process is killed with `-9`.
|
||||
|
||||
**Prettier owns the formatting** (110 columns, single quotes, `.prettierrc.json`); Markdown is
|
||||
excluded because it pads table cells to equal width. ESLint therefore only ever means "this is a
|
||||
|
||||
+26
-3
@@ -95,9 +95,32 @@ export function launchBrowser({ port, width, height }) {
|
||||
remote,
|
||||
`powershell -NoProfile -ExecutionPolicy Bypass -File gm-browser.ps1 -Port ${port} -Size ${width},${height}`,
|
||||
],
|
||||
// stdin stays open on purpose: closing it is what tells the launcher to kill the browser.
|
||||
{ stdio: ['pipe', 'ignore', 'pipe'] },
|
||||
{ stdio: ['ignore', 'ignore', 'pipe'] },
|
||||
);
|
||||
|
||||
return { proc, stop: () => proc.stdin.end(), where: `remote browser on ${remote}` };
|
||||
return {
|
||||
proc,
|
||||
/**
|
||||
* The launcher notices a finished run on its own, but only after its idle window – a browser
|
||||
* lingering a minute on someone else's desktop is rude, so end it now and let that window be
|
||||
* the net for runs that die without reaching here.
|
||||
*/
|
||||
stop: () => {
|
||||
try {
|
||||
execFileSync(
|
||||
'ssh',
|
||||
[
|
||||
...sshArgs(key),
|
||||
remote,
|
||||
'powershell -NoProfile -Command "Get-Process chrome,msedge -ErrorAction SilentlyContinue | Stop-Process -Force"',
|
||||
],
|
||||
{ stdio: 'pipe', timeout: 20_000 },
|
||||
);
|
||||
} catch {
|
||||
// Unreachable host, denied, already gone: the idle window still collects it.
|
||||
}
|
||||
proc.kill('SIGTERM');
|
||||
},
|
||||
where: `remote browser on ${remote}`,
|
||||
};
|
||||
}
|
||||
+82
-11
@@ -1,9 +1,54 @@
|
||||
# Launches a headless browser with the CDP port open and ties its life to this SSH session:
|
||||
# stdin closing (the caller went away, however violently) kills the whole process tree. Without
|
||||
# that leash the browser survives a dropped connection - measured: 8 orphaned chrome.exe.
|
||||
param([int]$Port = 9333, [string]$Size = "1280,800")
|
||||
# Launches a headless browser with the CDP port open and ties its life to whoever is using it.
|
||||
#
|
||||
# Two measured facts shape this. Windows OpenSSH does not kill its child when the connection
|
||||
# drops - waiting for the caller to die leaves the browser running forever (6 orphaned chrome.exe
|
||||
# after a SIGKILLed tunnel). And stdin is no signal either: only the first chunk a caller writes
|
||||
# ever arrives here, so a heartbeat over stdin looks identical to a dead caller.
|
||||
#
|
||||
# What is reliable is the CDP port itself: while a run is under way the forwarded connection is
|
||||
# established, and it closes the moment the tunnel or the run dies. So the browser stays for as
|
||||
# long as somebody is talking to it. The job object then makes the kill total - when this process
|
||||
# ends for any reason, KILL_ON_JOB_CLOSE takes the whole browser tree with it.
|
||||
param([int]$Port = 9333, [string]$Size = "1280,800", [int]$IdleSeconds = 45, [int]$MaxMinutes = 30)
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
Add-Type -TypeDefinition @'
|
||||
using System;
|
||||
using System.Runtime.InteropServices;
|
||||
public static class GmJob {
|
||||
[DllImport("kernel32.dll", CharSet = CharSet.Unicode)]
|
||||
static extern IntPtr CreateJobObject(IntPtr attrs, string name);
|
||||
[DllImport("kernel32.dll")]
|
||||
static extern bool SetInformationJobObject(IntPtr job, int infoClass, IntPtr info, uint len);
|
||||
[DllImport("kernel32.dll")]
|
||||
static extern bool AssignProcessToJobObject(IntPtr job, IntPtr process);
|
||||
|
||||
const int ExtendedLimitInformation = 9;
|
||||
const int LimitKillOnJobClose = 0x2000;
|
||||
// Offset of LimitFlags inside JOBOBJECT_EXTENDED_LIMIT_INFORMATION on x64, and the struct's size.
|
||||
const int LimitFlagsOffset = 16;
|
||||
const int StructSize = 144;
|
||||
|
||||
public static IntPtr Create() {
|
||||
IntPtr job = CreateJobObject(IntPtr.Zero, null);
|
||||
if (job == IntPtr.Zero) throw new Exception("CreateJobObject failed");
|
||||
IntPtr info = Marshal.AllocHGlobal(StructSize);
|
||||
try {
|
||||
for (int i = 0; i < StructSize; i++) Marshal.WriteByte(info, i, 0);
|
||||
Marshal.WriteInt32(info, LimitFlagsOffset, LimitKillOnJobClose);
|
||||
if (!SetInformationJobObject(job, ExtendedLimitInformation, info, StructSize))
|
||||
throw new Exception("SetInformationJobObject failed: " + Marshal.GetLastWin32Error());
|
||||
} finally { Marshal.FreeHGlobal(info); }
|
||||
return job;
|
||||
}
|
||||
|
||||
public static void Assign(IntPtr job, IntPtr process) {
|
||||
if (!AssignProcessToJobObject(job, process))
|
||||
throw new Exception("AssignProcessToJobObject failed: " + Marshal.GetLastWin32Error());
|
||||
}
|
||||
}
|
||||
'@
|
||||
|
||||
$candidates = @(
|
||||
"C:\Program Files\Google\Chrome\Application\chrome.exe",
|
||||
"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe",
|
||||
@@ -25,19 +70,45 @@ $flags = @(
|
||||
'about:blank'
|
||||
)
|
||||
|
||||
$job = [GmJob]::Create()
|
||||
$proc = Start-Process -FilePath $exe -ArgumentList $flags -PassThru -NoNewWindow
|
||||
[Console]::Error.WriteLine("gm-browser: $exe pid=$($proc.Id) port=$Port")
|
||||
[GmJob]::Assign($job, $proc.Handle)
|
||||
[Console]::Error.WriteLine("gm-browser: $exe pid=$($proc.Id) port=$Port job=on")
|
||||
|
||||
# Either the browser dies on its own, or our caller does. Both must end this script.
|
||||
# Either the browser dies on its own, or nobody is driving it any more. Both must end this script.
|
||||
# The window has to survive the gap before the first client connects, hence starting the clock now.
|
||||
$reason = "the browser exited on its own with code"
|
||||
$started = Get-Date
|
||||
$lastClient = $started
|
||||
while (-not $proc.HasExited) {
|
||||
if ([Console]::In.ReadLine() -eq $null) { break }
|
||||
Start-Sleep -Seconds 3
|
||||
if (((Get-Date) - $started).TotalMinutes -ge $MaxMinutes) {
|
||||
$reason = "the $MaxMinutes-minute cap"
|
||||
break
|
||||
}
|
||||
# .NET rather than Get-NetTCPConnection: that one goes through WMI, which is denied to this
|
||||
# account - its CIM session broke mid-run and took the browser down with it.
|
||||
try {
|
||||
$tcp = [System.Net.NetworkInformation.IPGlobalProperties]::GetIPGlobalProperties().GetActiveTcpConnections()
|
||||
$clients = @($tcp | Where-Object { $_.LocalEndPoint.Port -eq $Port -and $_.State -eq 'Established' })
|
||||
} catch {
|
||||
# A probe that fails says nothing about the run; the cap above is what bounds an orphan.
|
||||
[Console]::Error.WriteLine("gm-browser: connection probe failed: $($_.Exception.Message)")
|
||||
continue
|
||||
}
|
||||
if ($clients.Count -gt 0) {
|
||||
$lastClient = Get-Date
|
||||
} elseif (((Get-Date) - $lastClient).TotalSeconds -ge $IdleSeconds) {
|
||||
$reason = "no CDP client for $IdleSeconds s"
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if ($proc.HasExited) {
|
||||
[Console]::Error.WriteLine("gm-browser: browser exited on its own with code $($proc.ExitCode)")
|
||||
[Console]::Error.WriteLine("gm-browser: $reason $($proc.ExitCode)")
|
||||
} else {
|
||||
# taskkill and tasklist filters are denied to this account; Stop-Process is not. Killing by name
|
||||
# is safe here because the account exists only to run these tests.
|
||||
# taskkill and tasklist filters are denied to this account; Stop-Process is not. The job object
|
||||
# would do it anyway when this process ends - this just makes the kill immediate.
|
||||
Get-Process chrome, msedge -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue
|
||||
[Console]::Error.WriteLine("gm-browser: stdin closed, killed the browser")
|
||||
[Console]::Error.WriteLine("gm-browser: $reason, killed the browser")
|
||||
}
|
||||
Reference in new issue
Block a user