Skip to content

feat(memory): warn about workspace memory pressure before it crashes - #1209

Open
EhabY wants to merge 1 commit into
mainfrom
feat/vsc-32-memory-pressure-warnings
Open

EhabY wants to merge 1 commit into
mainfrom
feat/vsc-32-memory-pressure-warnings

Conversation

@EhabY

@EhabY EhabY commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Closes VSC-32.

Warns in the IDE before a workspace runs out of memory, with a way to act on it.

What users get

  • The "Coder Agent Metadata" status bar item gains a memory part:
    • It stays out of the text while memory is fine, unless the template has no metadata.
    • At coder.memoryMonitor.warningThreshold (default 80%, 0 turns it off), or when memory pressure slows the workspace, it shows $(warning) Memory 92% on a warning background.
    • After an OOM kill, it shows $(error) on an error background.
  • The tooltip lists Agent metadata, then Workspace usage (Memory: 6.12/16 GiB (38%), memory the kernel cannot reclaim, as free and top count it), then the warning with Top processes and Restart links.
  • Clicking the item during a warning opens Coder: Show Top Processes, one reused terminal running top sorted by memory.
  • Coder: Restart Workspace restarts the connected workspace and reconnects. The same fix applies to restarting the connected workspace from the Workspaces panel, which used to leave the window disconnected until a manual reload.
  • Background colour now means memory only, so a metadata error no longer turns the item yellow.

How it works

Two layers in src/memory/, so the data source can be swapped:

  • Data: a MemoryUsageSource returns one memory sample. Today's source, createRemoteFileUsage (remoteFiles.ts), reads cgroup and /proc files over the window's existing remote connection, in one parallel batch per poll. It needs no template changes and runs no processes. It ports coder/clistat's memory reading at a pinned commit (see the file header), except that used memory excludes all reclaimable memory: MemTotal - MemAvailable without a limit, and memory.current - active_file - inactive_file - slab_reclaimable under one. clistat subtracts only inactive_file, which counts recently read file cache as used. It also finds cgroups through /proc/self/mountinfo (so a mount rooted at /docker/abc works) and takes v1 parent limits from hierarchical_memory_limit, where clistat reads fixed paths.
  • Handling: MemoryMonitor polls every 10s. MemoryPressureTracker turns samples into states, warning on 2 of 3 samples and leaving after 10s of calm samples, so spikes don't flap and a quick re-read can't clear a warning. The status bar renders the result. None of these know where samples come from.

Restarting the connected workspace queues the stop with the start behind it (servers >= 2.36), then reloads. The reconnect waits for that start instead of posting its own.

Future

  • Agent API source: feat: watch workspace agent resource usage (clistat) over WebSocket coder#32542 (CLIENT-1114) proposes GET /api/v2/workspaceagents/{workspaceagent}/resources/watch, a WebSocket that runs clistat in the agent only while someone listens. Supporting it means adding a source, picked in remote.ts by server version, and eventually deleting remoteFiles.ts. Nothing else changes.
  • One memory definition: clistat (and so coder stat mem, the resources monitor and the API) adopts the same definition of used memory, so every client shows the same number.
  • More stats: CPU and disk can join "Workspace usage" once the API provides them.
  • Notifications: left out of this PR to keep it quiet; the status bar is the only warning.
Implementation plan

VSC-32: In-IDE memory pressure warnings before crashes

Goal (from product)

  • Warn inside the IDE when memory pressure reaches a configurable threshold,
    before the workspace dies.
  • The warning offers a clear action: restart, or reduce load.

Constraints from us: non-invasive, few false negatives, still a reliable signal
(not noisy).

What exists today

Signal Where Gaps
Server resource monitor -> Inbox "Workspace Out Of Memory" src/inbox.ts (template id a9d027b4...) Opt-in per template (coder_agent.resources_monitoring.memory) and needs Coder notifications enabled. 20 samples x 10s window, 30 min debounce (coderd/agentapi/api.go:183). Plain info toast with no action. No public API for the values.
Agent metadata status bar item src/api/agentMetadata.ts, remote.ts:1032 (id agentMetadata) Free-form strings defined by the template. RAM is present only if the template adds it (usually coder stat mem). Cannot be parsed reliably.

Proposed approach

1. Data source: kernel memory files over the existing remote connection

The extension is extensionKind: ui, but in a remote window
vscode.workspace.fs can read vscode-remote://<authority>/... URIs through
the VS Code server that is already running:

  • Cgroups: /proc/self/cgroup mapped to directories through
    /proc/self/mountinfo.
  • cgroup v2: memory.current, memory.max (walk up to the limit with the
    least room left), memory.stat (active_file, inactive_file,
    slab_reclaimable), memory.pressure (PSI), memory.events (oom_kill).
  • cgroup v1: the lower of memory.limit_in_bytes and
    hierarchical_memory_limit, memory.usage_in_bytes, memory.stat,
    memory.oom_control.
  • No limit (VM, Kata, unlimited container): /proc/meminfo (MemTotal,
    MemAvailable), /proc/pressure/memory, /proc/vmstat (oom_kill).

Used memory is what the kernel cannot reclaim: MemTotal - MemAvailable
without a limit (what free and top report), and memory.current - active_file - inactive_file - slab_reclaimable under a cgroup limit. The
limit and scope follow coder/clistat (after CORE-693, usage is read at the
limit-owning ancestor so sibling cgroups count). clistat subtracts only
inactive_file, so coder stat mem reads higher under a limit until clistat
adopts the same definition (planned).

Non-invasive: no extra processes, no SSH exec, no template changes, nothing
installed on the workspace. A few small file reads per poll over an open
channel. /proc/self is the VS Code server, so we measure the cgroup where the
editor and terminals run (including dev containers).

Spike result (2026-11-20):

  • Code path: vscode.workspace.fs.readFile -> main thread
    FileService.readFile (unbuffered, since doReadFile sets
    preferUnbuffered) -> remote DiskFileSystemProvider.readFile on the
    server -> Node readFile(fd), which reads to end of file. stat.size is
    only used for limit checks, so size-0 files are fine
    (src/vs/platform/files/common/fileService.ts:588-757,
    node/diskFileSystemProvider.ts:223).
  • Node check on this Kata workspace: every file read fully despite
    stat.size=0. cgroup v2 at 0::/init, memory.max is max (no limit),
    so /proc/meminfo is the source. memory.events and /proc/vmstat have
    oom_kill.
  • PSI is missing here (/proc/pressure/memory and memory.pressure:
    ENOENT; kernel without PSI). PSI stays an optional signal; on such kernels
    we rely on usage and the OOM counter.
  • Still to confirm end to end in a real remote window (and in Cursor and
    Windsurf) during UAT. Fallback: B under "Alternatives".

Polling: every 10s, also while the window is not focused. Skip a tick while
the previous read is pending (reads stall under pressure).

2. Memory states

Three signals, with one clear rule per state:

State Enters when Leaves when Meaning for the user
Unavailable Files missing or reads fail A read succeeds Nothing changes in the UI. Logged once.
Normal Usage under the threshold, PSI low, no new OOM kill Nothing to do.
High Usage >= threshold in 2 of the last 3 samples, or PSI some avg10 >= 20% Usage under threshold - 5 and PSI under 10% for 10s -> Normal Close to the limit or slowed down by memory. Reduce load.
Processes killed oom_kill counter went up during this session Same exit rule as High -> Normal Something was already killed. Restart or reduce load now.
  • "Processes killed" wins over High. Exits go straight to Normal, not back to
    High, because the same exit rule applies.
  • The threshold is set to 0: High and Processes killed are never entered;
    the state is Normal or Unavailable. Usage is still shown.
  • OOM kills from before this session (while disconnected) do not change the
    state, they only show a notice (see 5).

3. Status bar: one item, two independent parts

The existing item keeps its name "Coder Agent Metadata" and id
agentMetadata. It shows two kinds of values, each computed on its own:

  • Agent metadata: supplied by the server, defined by the template (as
    today).
  • Workspace usage: calculated by the extension. Memory now; CPU or
    disk could be added later in the same part.

Because the parts are labelled separately, a template RAM value and the IDE
memory value can both appear. No RAM-entry detection is needed.

Text = metadata part, ·, memory part. Either part can be absent; the
separator appears only when both are present.

Metadata Metadata part
Values present $(dashboard) <first value> (as today)
Error $(warning) Agent Status Unavailable (as today)
No values absent
Memory state Memory part
Normal absent when there is a metadata part; $(dashboard) Memory 41% when there is none
High $(warning) Memory 92%
Processes killed $(error) Memory 97%
Unavailable absent

Rule: memory stays in the tooltip while Normal and moves into the text when
it needs attention (progressive disclosure). A warning is never color only:
it always has an icon, the word "Memory" and the value, plus the background.

  • Hidden only when both parts are absent (as today when there is no metadata
    and memory cannot be read).
  • Background: reserved for memory. Warning for High, error for
    Processes killed, none otherwise. A metadata error keeps its
    $(warning) Agent Status Unavailable text but loses the warning
    background (change from today), so a colored item always means memory.

Examples:

  • $(dashboard) 0.006/4 cores (0%) (Normal, as today)
  • $(dashboard) 0.006/4 cores (0%) · $(warning) Memory 92% (warning background)
  • $(dashboard) 0.006/4 cores (0%) · $(error) Memory 97% (error background)
  • $(warning) Agent Status Unavailable (Normal, no background)
  • $(warning) Agent Status Unavailable · $(warning) Memory 92% (warning background)
  • $(dashboard) Memory 8% (template has no metadata)

Tooltip: sections in this order, each omitted when it has no content.

Agent metadata
CPU Usage: 0.006/4 cores (0%)
RAM Usage: 1.32/16 GiB (8%)                                      (or the metadata error)

Workspace usage
Memory: 14.72/16 GiB (92%)
Memory pressure: 3% (when the kernel has PSI)

Memory is running low. Processes may be killed soon.             (High / Processes killed only)
Show top processes · Restart workspace

Click: in High or Processes killed, runs Show Top Processes.
Otherwise no command (as today).

Code: a small pure function (metadata, memory) -> item view that the
existing metadata watcher and the memory monitor both feed. Unit-test it for
every combination of the two part tables.

4. Actions (no notifications)

Notifications were dropped to keep the change small and quiet; the status
bar is the only warning.

  • Clicking the item in High or Processes killed runs Show Top Processes:
    one reused terminal on the remote running top -o %MEM, falling back to
    plain top (BusyBox, old procps) or a message when top is missing.
  • The tooltip's warning has links to Show top processes and Restart
    workspace
    . Restart runs coder.workspace.restart, which goes through the
    Workspaces panel's WorkspaceActionRunner (same confirmation and builds).

5. Removed from scope

  • Notifications, Don't Show Again, cooldowns, the "killed while
    disconnected" notice and its stored counter.

6. Setting ("scope": "application")

One setting, under coder.memoryMonitor.* so settings added later (CPU,
disk) group with it:

  • coder.memoryMonitor.warningThreshold: number, default 80, 0 to 99.
    Set to 0 to turn off memory warnings; usage is still shown in the
    tooltip.

No visibility setting: memory is shown whenever it can be read. Users who do
not want the item can hide it with right-click, like any status bar item.
Read live through watchConfigurationChanges; no reload needed.

7. Inbox OOM notification

Kept separate and unchanged: different system (Coder notifications, not
always enabled), own threshold and debounce.

8. Telemetry, docs

  • memory.state_transitioned (state, trigger, percent, source, pressure) in
    EVENTS.md.
  • CHANGELOG.md entry. No README change (it does not list settings).

Code layout

  • src/memory/usage.ts: reads and parses the files (pure, takes a reader).
  • src/memory/pressure.ts: state machine with hysteresis (pure).
  • src/memory/monitor.ts: 10s polling over vscode-remote://, telemetry.
  • src/memory/topProcesses.ts: the reused top terminal.
  • src/remote/agentStatusBar.ts: renders the metadata part and the memory
    part; wired in createAgentMetadataStatusBar in src/remote/remote.ts.

Alternatives considered

  • A. Server monitor + Inbox only: zero client cost, but opt-in per
    template and needs notifications enabled. Left unchanged and separate.
  • B. coder ssh <ws> -- cat ... or coder stat mem over SSH: works
    without the remote FS, but starts processes on every poll. Fallback only.
  • C. Parse agent metadata: template-specific format. Not reliable.
  • D. Remote helper extension: installs something on the workspace. Too
    invasive.
  • Separate memory status bar item: rejected in favor of one item with
    two labelled parts.
  • Show IDE memory only when High, plus an alwaysShowUsage setting:
    needed RAM-entry detection and suffix rules to avoid duplicates, and hid
    memory when metadata failed. Rejected for always showing both parts.

Relation to coder/coder work

  • ENG-230 / Resource alerting (custom notifications) coder#15177 (Resource alerting, custom notifications):
    Backlog since 2024-10, no comments or PRs. Admin-defined notifications
    triggered through an API and delivered to IDEs. Would reach us through the
    Inbox; we would only add its template ids to src/inbox.ts. VSC-32 does not
    depend on it and stays useful after it, since it needs no admin setup.
  • CORE-857 (canceled 2026-11-18): the VS Code server is an early OOM
    victim (high oom_score_adj). Warnings must come before the kill, and reads
    can stall under pressure.
  • CORE-693 (done): clistat counts sibling cgroups. We do the same.
  • CORE-299: configurable OOM algorithm, server side, out of scope.

Threshold research

Source Value
Common Kubernetes "memory near limit" alerts (working set / limit) 80%, sometimes up to 90%
Production report: 75% gave about 3m40s before failures; 90% fired after reclaim started 75 to 80%
Coder resources_monitoring docs example 90%

Our "used" excludes reclaimable page cache (like working set). 80% is the
most common standard and gives the earliest window. The 2 of 3 rule and the
exit gap limit noise from large steady heaps.

Decisions

  • Threshold default 80%.
  • One status bar item: the existing "Coder Agent Metadata" item, name and id
    unchanged, with a server part (agent metadata) and a client part (measured
    by the IDE). Memory is in the text only when it needs attention or there
    is no metadata; always in the tooltip. Background reserved for memory, so
    color appears only when the user needs to act (metadata errors keep their
    text but lose the warning background).
  • One setting: coder.memoryMonitor.warningThreshold.
  • No notifications for now; restart goes through the Workspaces panel's
    action runner.
  • Usage shown as 6.12/16 GiB (38%), the agent metadata format.
  • top for reduce load; poll while unfocused; disk out of scope; inbox OOM
    notification stays separate.

Rollout

  1. Spike: remote FS reads of /proc and /sys in VS Code, Cursor, Windsurf.
  2. Monitor, parsing and state machine with unit tests.
  3. Status bar view, actions, setting, telemetry.
  4. UAT on dogfood: stress-ng --vm in a container with a limit and in a Kata
    VM without one; templates with and without RAM metadata.

Generated by Coder Agents on behalf of @EhabY.

@linear-code

linear-code Bot commented Nov 20, 2026

Copy link
Copy Markdown

VSC-32

@EhabY
EhabY force-pushed the feat/vsc-32-memory-pressure-warnings branch from 6dc9294 to 5e5cdbb Compare November 20, 2026 15:33
@EhabY
EhabY force-pushed the feat/vsc-32-memory-pressure-warnings branch 10 times, most recently from 0c33f6b to 07f1a1e Compare November 20, 2026 20:01
@EhabY
EhabY force-pushed the feat/vsc-32-memory-pressure-warnings branch from 07f1a1e to 4280c79 Compare November 20, 2026 21:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant