x86 Test Host OOM Analysis for dotnet/roslyn TemporaryStorageService

oom-analysis.md

x86 Test Host OOM Analysis

Problem Statement

The x86 test host for Microsoft.CodeAnalysis.CSharp.EditorFeatures.UnitTests (net472) crashes with an OutOfMemoryException during CI runs. The x86 process has a 4 GB virtual address (VA) space limit, and the test host exceeds this limit during the ~16,591 test run.

Root Cause

TemporaryStorageService uses a bump-pointer allocator that packs small items into shared 8 MB memory-mapped file (MMF) blocks. When any single handle in a block is still alive, the entire 8 MB block stays mapped. Over the course of 16K+ tests, hundreds of these blocks accumulate because transient items keep entire blocks pinned:

• 525+ blocks × 8 MB = ~4.2 GB VA consumed by MMFs alone
• This exceeds the x86 4 GB VA limit → OOM crash

The bump-pointer allocator exists as a performance optimization to avoid creating thousands of individual MMFs. However, the fragmentation trade-off makes it counterproductive for long-running test processes.

Experiments

Test Suites Used

Suite	Project	Framework	Test Count
Workspaces	Microsoft.CodeAnalysis.Workspaces.UnitTests	net472	~2,300
EditorFeatures	Microsoft.CodeAnalysis.CSharp.EditorFeatures.UnitTests	net472	~16,591

All x86 runs use -- RunConfiguration.TargetPlatform=x86 to launch testhost.net472.x86.exe without requiring a full x86 rebuild.

Experiment 1: Baseline (no changes)

The unmodified code with the bump-pointer allocator.

Results (x86):

Suite	PeakVM	Result	Duration
Workspaces	1462 MB	✅ Pass (2,297 passed)	~5 min
EditorFeatures	3714 MB	❌ CRASH (OOM)	~27 min

Observation: Workspaces completes but EditorFeatures crashes. The crash occurs because the cumulative MMF VA usage exceeds the 4 GB x86 limit.

Experiment 2: Individual MMFs (remove bump-pointer allocator)

Changed TemporaryStorageService.CreateTemporaryStorage() to create a dedicated MemoryMappedFile for each item instead of packing into shared 8 MB blocks. When a handle is GC'd, only its specific small MMF is freed — no coupling between unrelated items.

Key implementation details:

• Removed constants SingleFileThreshold (256 KB) and MultiFileBlockSize (8 MB)
• Removed fields _gate, _fileReference, _name, _fileSize, _offset
• Added Math.Max(size, 1) guard for zero-length streams (MemoryMappedFile.CreateNew requires capacity ≥ 1)
• Updated tests to use interface types (ITemporaryStorageServiceInternal, ITemporaryStorageStreamHandle)

Results (x86):

Suite	PeakVM	Result	Duration
Workspaces	1318 MB	✅ Pass (2,297 passed)	~5 min
EditorFeatures	3089 MB	✅ Completes (16,591 tests)	~27 min

pvanalyze results (net10.0, x64 Workspaces):

Metric	Baseline	Post-fix	Delta
Peak heap	2199 MB	1793 MB	-18.5%
Total GC time	2.82s	1.58s	-44%
Gen0 collections	304	227	-25%

Observation: This fix eliminates the fragmentation problem. Each MMF is independently reclaimable. PeakVM drops 625 MB for EditorFeatures, enough to avoid the x86 crash. The fix benefits all architectures (x64 also sees 18.5% heap reduction).

Experiment 3: TrivialTemporaryStorageService on x86

Changed only TemporaryStorageService.Factory.cs to return TrivialTemporaryStorageService.Instance when !Environment.Is64BitProcess. This bypasses MMFs entirely on x86 — all temporary storage is held in managed memory (byte arrays on the heap).

No changes to TemporaryStorageService.cs or tests.

Results (x86):

Suite	PeakVM	Result	Duration
Workspaces	4005 MB	✅ Pass (2,311 tests)	6 min 36s
EditorFeatures	4010 MB	❌ ABORTED (13,258 of 16,591)	38 min

Observation: This approach is worse than baseline. Instead of fragmenting VA with MMFs, it bloats the managed heap — but both consume the same 4 GB x86 VA space. Workspaces PeakVM jumps +174% (1462 → 4005 MB). EditorFeatures still crashes/aborts with 3,333 fewer tests completed than baseline. Execution time also increases ~40% due to GC pressure from large managed allocations.

Experiment 4: Hybrid — forceSingleFile on x86 only

Due to team concerns about potential performance regression in production (x64) scenarios, implemented a hybrid approach: keep the bump-pointer allocator on x64 but force individual MMFs on x86.

Changes:

• Added bool _forceSingleFile field to TemporaryStorageService, set via constructor
• Factory passes forceSingleFile: !Environment.Is64BitProcess
• CreateTemporaryStorage condition: if (_forceSingleFile || size >= SingleFileThreshold) → individual MMF path (with Math.Max(size, 1) guard for zero-length streams)
• On x64: bump-pointer allocator behavior is completely unchanged
• On x86: every item gets its own MMF (same as Experiment 2 behavior)
• No test changes required

Results (x86):

Suite	PeakVM	Result	Duration
Workspaces	1136 MB	✅ Pass (2,297 passed)	~2 min
EditorFeatures	2688 MB	✅ Completes (16,480 passed)	~21 min

Observation: Best PeakVM results of all approaches. Workspaces is 22% below baseline (1462 → 1136 MB), EditorFeatures is 28% below baseline (3714 → 2688 MB) and completes successfully. The ~97 EditorFeatures failures are pre-existing StringCopyPaste test issues, unrelated to this change. Duration is consistent with baseline (~27 min).

Comparison Summary

CSharp.EditorFeatures.UnitTests (x86, net472)

Approach	PeakVM	Tests Run	Result	Duration
Baseline (shared 8 MB MMFs)	3714 MB	~16,591	❌ CRASH	~27 min
TrivialTemporaryStorageService (x86)	4010 MB	13,258	❌ ABORTED	38 min
Individual MMFs (all archs)	3089 MB	16,591	✅ Completes	~27 min
Hybrid forceSingleFile (x86 only)	2688 MB	16,480	✅ Completes	~21 min

Workspaces.UnitTests (x86, net472)

Approach	PeakVM	Tests Run	Result	Duration
Baseline (shared 8 MB MMFs)	1462 MB	2,297	✅ Pass	~5 min
TrivialTemporaryStorageService (x86)	4005 MB	2,311	✅ Pass	6m 36s
Individual MMFs (all archs)	1318 MB	2,297	✅ Pass	~5 min
Hybrid forceSingleFile (x86 only)	1136 MB	2,297	✅ Pass	~2 min

Heap Analysis (EditorFeatures, mid-run dump at 1636 MB WS)

A dotnet-dump collect was taken during an EditorFeatures test run to understand what consumes memory beyond MMFs.

Managed Heap Breakdown

Category	Size	% of Heap
Free/fragmentation	589 MB	40%
Strings	289 MB	20%
MEF composition	~100 MB	7%
.NET Remoting infrastructure	~50 MB	3%
xUnit discovery metadata	~30 MB	2%
TestPlatform	~15 MB	1%
Roslyn workspace/MMF objects	<1 MB	<0.1%

Workspace Lifecycle

Despite 136 workspace instances being on the heap, analysis confirmed they are not leaked:

• 135 of 136 have _workQueueTokenSource.m_state = 3 (cancelled), meaning Workspace.Dispose() was called
• 1 is actively running (the test executing at dump time)
• !gcroot on disposed workspaces returns 0 unique roots — they are unreachable Gen2 garbage awaiting collection
• The active workspace is rooted through CSharpTestWorkspaceFixture._workspace → ReferenceCountedDisposable (legitimate)

.NET Remoting / xUnit v2 Memory Impact

148,463 of 149,537 strong GCHandles (99.3%) are ServerIdentity objects from .NET Remoting's MarshalByRefObject tracking. xUnit v2 types (XunitTestCase, TestMethod, ReflectionMethodInfo, etc.) all derive from MarshalByRefObject:

• 16,595 test cases × ~9 MBRO objects each ≈ 148K handles
• These strong handles are never released during process lifetime
• LongLivedMarshalByRefObject.InitializeLifetimeService() returns null (infinite lease)
• This prevents GC heap compaction → 589 MB fragmentation (40% of managed heap)

This is a fundamental property of xUnit v2 on .NET Framework 4.7.2 and cannot be fixed without migrating to xUnit v3 (drops MBRO) or .NET Core (no remoting overhead).

Current Fix

The hybrid forceSingleFile approach (Experiment 4) is the recommended fix. It adds a _forceSingleFile flag to TemporaryStorageService that bypasses the bump-pointer allocator, giving each item its own dedicated MMF. The flag is set to true on x86 processes only, preserving the existing bump-pointer behavior on x64 (no production perf regression risk).

Files changed (not yet committed, pending review):

• src/Workspaces/Core/Portable/TemporaryStorage/TemporaryStorageService.cs — added _forceSingleFile field, updated constructor and CreateTemporaryStorage() logic
• src/Workspaces/Core/Portable/TemporaryStorage/TemporaryStorageService.Factory.cs — passes forceSingleFile: !Environment.Is64BitProcess