Skip to content

Instantly share code, notes, and snippets.

@apivovarov
Last active March 31, 2026 02:24
Show Gist options
  • Select an option

  • Save apivovarov/f179446dac26ea6df64bcafc5725d345 to your computer and use it in GitHub Desktop.

Select an option

Save apivovarov/f179446dac26ea6df64bcafc5725d345 to your computer and use it in GitHub Desktop.
executor->GetOrNullResource<se::DeviceInterconnectResource>() causes SIGILL in tests
[ RUN ] RaggedAllToAllTest/RaggedAllToAllTest.RaggedAllToAll_2GPUs_CommandBuffer/sync_one_shot_with_multi_gpu_barrier_with_nccl
*** SIGILL (UD1@0x7f9c00625cc9), see go/stacktraces#debugging-ubsan
received by PID 102457 (TID 104591) on cpu 145; stack trace: ***
E0330 19:21:38.081043 104590 process_state.cc:1066] RAW: Signal 4 raised at PC: 0x7f9c00625cc9 while already in FailureSignalHandler!
E0330 19:21:38.081811 104590 process_state.cc:1070] RAW: tid: 104590 raised new signal (old_tid: 104591)
PC: @ 0x7f9c00625cc9 (unknown) xla::gpu::RaggedAllToAllThunk::RunCollective()
@ 0x7f92d4008f08 1904 FailureSignalHandler()
@ 0x7f9d30337c60 (unknown) (unknown)
@ 0x7f9bf5215d8c 288 xla::gpu::CollectiveThunk::ExecuteOnStream()
@ 0x7f9bfa60d618 464 xla::gpu::ThunkExecutor::ExecuteOnStream()
@ 0x7f9bfca0909c 48 xla::gpu::SequentialThunk::ExecuteOnStream()
@ 0x7f9c150119fa 304 xla::gpu::CommandBufferThunk::ExecuteOnStream()
@ 0x7f9bfa60d618 464 xla::gpu::ThunkExecutor::ExecuteOnStream()
@ 0x7f9c16e5b914 1504 xla::gpu::(anonymous namespace)::ExecuteThunksImpl()
@ 0x7f9c16e587c6 416 xla::gpu::GpuExecutable::ExecuteThunks()
@ 0x7f9c16e57291 2016 xla::gpu::GpuExecutable::ExecuteAsyncOnStreamImpl()
@ 0x7f9c16e55f6d 80 xla::gpu::GpuExecutable::ExecuteAsyncOnStream()
@ 0x7f9980410826 1008 xla::Executable::ExecuteAsyncOnStreamWrapper()
@ 0x7f9d0da1427c 2512 xla::LocalExecutable::RunAsync()
@ 0x7f9d0da14e97 288 xla::LocalExecutable::RunAsync()
@ 0x7f9d17cbc671 3296 xla::TfrtGpuExecutable::ExecuteHelper()::$_0::operator()()
@ 0x7f9d17cbbd20 64 absl::internal_any_invocable::RemoteInvoker<>()
@ 0x7f9d17c95313 32 std::__u::__function::__policy_func<>::__call_func<>()
@ 0x7f99fb81aebb 48 absl::internal_any_invocable::RemoteInvoker<>()
@ 0x7f93312143bf 368 thread::TMWorker()
@ 0x7f9300827296 256 Thread::ThreadBody()
@ 0x7f9d3032f0bf 192 start_thread
@ 0x7f9d2f79067f (unknown) clone
E0330 19:21:38.220636 104591 coredump_hook.cc:274] RAW: Remote crash gathering disabled for unit test. Enable that with --remote_coredump_enabled_for_unit_test flag
E0330 19:21:45.685045 104591 process_state.cc:704] RAW: Raising signal 4 with default behavior
/foo/tools/test/local-runtest.sh: line 209: 102457 Illegal instruction (core dumped) "$@"
-- 2026-03-30 19:21:48 PDT Local runner: Test killed by SIGILL while running on h100
================================================================================
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment