Scheduling Research: Dynamic Session/Model Scheduling in ML Inference Runtimes

scheduling.md

NXRT Scheduling Research: Dynamic Session/Model Scheduling in ML Inference Runtimes

Date: 2026-07-15
Purpose: Survey existing implementations relevant to NXRT's scheduling/session management design.

---

1. vLLM — Session Preemption & KV Cache Swap

What It Does

vLLM implements request-level preemption with two modes for handling evicted sessions:

• SWAP mode (V0): Offloads KV cache blocks from GPU → CPU memory. When the request resumes, blocks are swapped back in without recomputation.
• RECOMPUTE mode (V1 default): Drops KV cache entirely; recomputes from prompt on resume. Lower overhead in V1's architecture.

API / Architecture

# Scheduler decides preemption based on memory pressure
# PreemptionMode.SWAP or PreemptionMode.RECOMPUTE
# V0: configurable via --preemption-mode swap
# V1: recompute only (swap deprecated), but KV connectors (LMCache) proposed for offload

• Scheduler is a central Python class (vllm.core.scheduler) that maintains three queues: waiting, running, swapped
• Preemption policy: oldest-first (FCFS priority by default)
• RFC #6077 proposes priority scheduling with forced preemption + KV cache preservation

What's Pluggable vs Hardcoded

Aspect	Status
Preemption trigger	Hardcoded (memory threshold)
Preemption policy (which to evict)	Hardcoded FCFS, priority RFC pending
Swap vs Recompute	Configurable (V0), hardcoded recompute (V1)
Cost model	Latency-only (throughput maximization)
External cost signals (thermal, power)	Not supported

Lessons for NXRT

• KV cache = session state analogy. vLLM's swap is essentially "session hibernation" for LLM inference state.
• Swap to CPU is expensive; V1 abandoned it for recompute. For NXRT: consider whether recompilation is cheaper than state serialization for small models.
• The scheduler is monolithic — no plugin interface for custom cost functions. This is a gap NXRT can fill.
• LMCache/KV connectors show the community wants modular offload backends.

---

2. TensorRT — Multiple Optimization Profiles

What It Does

TensorRT supports multiple optimization profiles per engine — each profile defines a (min, opt, max) shape range. At runtime, you can switch between profiles on an IExecutionContext to handle different input shapes optimally.

API

// Build time: define multiple profiles
IOptimizationProfile* profile1 = builder->createOptimizationProfile();
profile1->setDimensions("input", OptProfileSelector::kMIN, Dims4(1,3,224,224));
profile1->setDimensions("input", OptProfileSelector::kOPT, Dims4(8,3,224,224));
profile1->setDimensions("input", OptProfileSelector::kMAX, Dims4(32,3,224,224));
config->addOptimizationProfile(profile1);

IOptimizationProfile* profile2 = builder->createOptimizationProfile();
profile2->setDimensions("input", OptProfileSelector::kMIN, Dims4(32,3,224,224));
profile2->setDimensions("input", OptProfileSelector::kOPT, Dims4(64,3,224,224));
profile2->setDimensions("input", OptProfileSelector::kMAX, Dims4(128,3,224,224));
config->addOptimizationProfile(profile2);

// Runtime: switch profiles
context->setOptimizationProfileAsync(1, stream); // switch to profile 1

Key Behaviors

• Profile switching triggers shape recomputation and tactic resource reallocation — not free, but much cheaper than recompilation
• Each profile independently selects optimal kernel tactics at build time for the opt shape
• Single engine file contains all profiles; no separate compilation needed
• Cannot have two profiles with identical shape ranges (dedup)

What's Pluggable vs Hardcoded

Aspect	Status
Profile definitions	User-defined at build time
Which profile to activate	User-controlled at runtime
Tactic selection within profile	Hardcoded (latency-optimal auto-tuning)
Switching cost model	Not exposed — user must measure
Thermal/power-aware profile selection	Not built-in (user must implement)

Lessons for NXRT

• Pre-compiled multi-plan is proven. TensorRT validates the concept of building multiple execution strategies into one artifact.
• Profile switching has non-zero cost (shape propagation + resource recompute). NXRT should consider pre-computing transition costs.
• TensorRT doesn't help you decide when to switch — that's left to the application. NXRT's scheduler fills exactly this gap.
• The "opt shape" concept (performance sweet spot) maps to NXRT's notion of preferred operating points.

---

3. Apple Core ML — Compute Unit Selection & Thermal Behavior

What It Does

Core ML provides compute unit hints that control which hardware a model runs on:

• MLComputeUnits.all — system decides (ANE + GPU + CPU)
• MLComputeUnits.cpuAndGPU — exclude ANE
• MLComputeUnits.cpuAndNeuralEngine — exclude GPU
• MLComputeUnits.cpuOnly

The runtime can split a model across units (e.g., some layers on ANE, others on GPU/CPU) based on op support.

Thermal-Aware Behavior (Implicit)

• Core ML does NOT expose explicit thermal-aware scheduling APIs
• However, the system implicitly throttles: ANE draws ~12.7W vs GPU ~24.7W; GPU throttles 50%+ within 60s of sustained load while ANE stays stable
• ProcessInfo.thermalState (iOS/macOS API) provides .nominal, .fair, .serious, .critical — apps can react by switching compute units
• No built-in automatic switching based on thermals — must be app-implemented

API

let config = MLModelConfiguration()
config.computeUnits = .all  // or .cpuAndNeuralEngine, etc.
let model = try MLModel(contentsOf: modelURL, configuration: config)

// Thermal monitoring (manual)
NotificationCenter.default.addObserver(
    forName: ProcessInfo.thermalStateDidChangeNotification, ...) {
    if ProcessInfo.processInfo.thermalState == .serious {
        // Reload model with .cpuAndNeuralEngine (lower power)
    }
}

What's Pluggable vs Hardcoded

Aspect	Status
Compute unit selection	User-controlled (load-time only)
Model splitting across units	System-controlled, not configurable
Thermal response	Manual (app must monitor + react)
Dynamic switching at runtime	Not supported without model reload
Cost model	Opaque (Apple internal)

Lessons for NXRT

• No runtime dynamic switching — Core ML requires model reload to change compute units. NXRT can do better with pre-compiled multi-target plans.
• Thermal state is available but scheduling response is entirely app-responsibility. NXRT's scheduler should consume thermal signals automatically.
• The ANE vs GPU power/performance tradeoff is exactly the kind of multi-dimensional cost NXRT should model.
• Core ML's implicit model splitting shows value of heterogeneous execution — but lack of control is frustrating for developers.

---

4. Qualcomm QNN / SNPE — Backend Selection & Power Modes

What It Does

QNN (successor to SNPE) provides a unified API with backend-specific libraries targeting:

• Snapdragon CPU (Kryo)
• Adreno GPU (OpenCL/Vulkan)
• Hexagon HTP/NPU (dedicated neural processor)

Power/Performance Modes

SNPE/QNN expose performance profiles:

SNPE_PERFORMANCE_PROFILE_DEFAULT
SNPE_PERFORMANCE_PROFILE_SUSTAINED_HIGH_PERFORMANCE
SNPE_PERFORMANCE_PROFILE_HIGH_PERFORMANCE
SNPE_PERFORMANCE_PROFILE_POWER_SAVER
SNPE_PERFORMANCE_PROFILE_SYSTEM_SETTINGS
SNPE_PERFORMANCE_PROFILE_BALANCED

These control DVFS (clock scaling) on the selected backend — not which backend to use.

API

// SNPE
snpe = snpeBuilder.setPerformanceProfile(
    zdl::DlSystem::PerformanceProfile_t::HIGH_PERFORMANCE)
    .setRuntimeOrder({zdl::DlSystem::Runtime_t::DSP,
                      zdl::DlSystem::Runtime_t::GPU_FLOAT16,
                      zdl::DlSystem::Runtime_t::CPU})
    .build();

// QNN — backend selection at graph creation time
QnnGraph_Config_t graphConfig;
// Backend is selected by which backend library you load (libQnnHtp.so, libQnnGpu.so, etc.)

What's Pluggable vs Hardcoded

Aspect	Status
Backend selection	User-controlled (load/build time)
Performance profile	User-controlled (runtime changeable in SNPE)
Runtime backend switching	NOT supported without rebuild
Cost model for backend choice	Not provided — user benchmarks
Thermal response	System-level DVFS (transparent to app)
Custom scheduling policy	Not supported

Lessons for NXRT

• Performance profiles are a good API pattern — simple enum that maps to system-level power/perf tuning.
• Like Core ML, backend choice is a build-time decision. No dynamic switching.
• The "runtime order" (fallback chain) in SNPE is interesting — NXRT could generalize this as a priority-ordered execution plan list.
• QNN's offline graph preparation (snpe-dlc-graph-prepare) is analogous to NXRT's pre-compilation step.

---

5. TFLite / LiteRT / MediaPipe — Delegate System

What It Does

TFLite's delegate architecture allows plugging in hardware-specific backends:

• XNNPACK (optimized CPU)
• GPU delegate (OpenGL ES / OpenCL / Metal / Vulkan)
• NNAPI delegate (Android neural networks API)
• Hexagon delegate (Qualcomm DSP)
• Core ML delegate (Apple)

How Delegation Works

// Android example
Interpreter.Options options = new Interpreter.Options();
GpuDelegate gpuDelegate = new GpuDelegate(new GpuDelegate.Options()
    .setQuantizedModelsAllowed(true));
options.addDelegate(gpuDelegate);
Interpreter interpreter = new Interpreter(modelFile, options);

• Delegates claim ops they support; unsupported ops stay on CPU
• Graph is partitioned at load time — delegate subgraphs + CPU fallback subgraphs
• No runtime switching between delegates without interpreter rebuild

What's Pluggable vs Hardcoded

Aspect	Status
Delegate selection	User-controlled (build time)
Op partitioning	Automatic (delegate claims ops)
Runtime delegate switching	NOT supported
Custom delegates	Fully pluggable (implement C API)
Cost model / scheduling	None — user picks delegate
Thermal/power awareness	None at framework level

Lessons for NXRT

• TFLite's delegate plugin API is the gold standard for extensible backend support — clean C interface, any vendor can implement.
• But scheduling is completely absent — you pick one delegate and that's it.
• Graph partitioning (split across backends) happens automatically but is load-time only.
• NXRT can learn from the plugin interface but must add the scheduling layer TFLite lacks.

---

6. Academic Papers & Research

"Adyna: Accelerating Dynamic Neural Networks with Adaptive Scheduling" (HPCA 2025)

• Hardware accelerator that dynamically selects kernels per operator based on actual dynamic dimension values
• Controller infers best-fit kernel for each op at runtime
• Relevance: Validates per-op adaptive execution plan selection, but at hardware level

"LLM Inference Scheduling: A Survey" (TechRxiv, Oct 2025)

• Comprehensive survey covering latency, cost efficiency, scalability
• Identifies open direction: ML-driven adaptive scheduling that considers multiple objectives
• Relevance: Confirms the gap — no existing system does multi-objective scheduling well

"Deep Learning Workload Scheduling in GPU Datacenters: A Survey" (ACM CSUR, 2024)

• Covers cluster-level scheduling (Gandiva, Tiresias, Pollux)
• Multi-resource scheduling (GPU memory + compute + network)
• Relevance: Shows multi-dimensional cost at cluster level; NXRT applies similar thinking at device level

"Thermal-Aware Scheduling for Integrated CPUs-GPU Platforms" (ACM, 2019)

• CPU-GPU thermal coupling modeling via system identification
• Proactive DTM (Dynamic Thermal Management) policies
• DVFS + task migration based on thermal predictions
• Relevance: Directly applicable — thermal-aware scheduling for heterogeneous compute. Key insight: scheduling must be proactive (predict thermal state) not just reactive.

"Optimal Scheduling Algorithms for LLM Inference" (UT Austin, 2025)

• Theoretical analysis of optimal request scheduling
• Proves certain scheduling policies are optimal under specific assumptions
• Relevance: Foundation for proving NXRT scheduling properties

---

7. Summary: What Exists vs What NXRT Needs

Capability	vLLM	TensorRT	Core ML	QNN	TFLite	NXRT Goal
Session hibernation/swap	✅ KV swap	❌	❌	❌	❌	✅ Full state offload
Multiple pre-compiled plans	❌	✅ Profiles	❌	❌	❌	✅ Per-target plans
Runtime plan switching	❌	✅ (with cost)	❌ (reload)	❌ (rebuild)	❌ (rebuild)	✅ Zero-copy switch
Pluggable cost model	❌	❌	❌	❌	❌	✅ User-defined
Multi-dimensional cost	❌	❌	❌	Partial (perf profiles)	❌	✅ Latency+power+thermal+memory
Thermal-aware scheduling	❌	❌	Manual only	System DVFS	❌	✅ Automatic
Backend plugin interface	❌	❌	❌	❌	✅ Delegates	✅

Key Gaps NXRT Can Fill

1. No existing runtime combines pre-compiled multi-plan + automatic scheduling. TensorRT has multi-plan; none have automatic switching based on runtime conditions.
2. No pluggable cost model exists. Every system uses hardcoded latency-only optimization or simple enum-based power modes.
3. Session hibernation is LLM-specific (KV cache). No general-purpose compiled model session suspend/resume exists.
4. Thermal-aware scheduling is always app-responsibility. No runtime does it automatically.

Design Implications

• NXRT's CostModel trait (pluggable, multi-dimensional) is genuinely novel in this space
• Pre-compiling N plans + scheduler that picks based on runtime signals = unique combination
• Session state serialization (weights + activations + scratch buffers) for hibernation goes beyond what any current system offers for compiled models
• The closest analog to NXRT's full vision is a combination of: TensorRT profiles + vLLM swap + Core ML thermal monitoring + TFLite delegate extensibility