Created
March 16, 2026 21:59
-
-
Save apivovarov/848f7bf1814522f8848e9a1e4012c7d9 to your computer and use it in GitHub Desktop.
--xla_gpu_execution_terminate_timeout=20s
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| W 2026-03-16T14:24:05.274810-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2bc4eb700 (name: py_xla_execute/5842) stack: --- | |
| stack used: 166 KiB of 2104 KiB | |
| I 2026-03-16T14:24:05.274834-07:00 0 W0316 14:24:05.274810 7452 thread.cc:1808] --- Thread 7fb2bc4eb700 (name: py_xla_execute/5842) stack: --- | |
| I 2026-03-16T14:24:05.274836-07:00 0 stack used: 166 KiB of 2104 KiB | |
| W 2026-03-16T14:24:05.489353-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e02cc178 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:05.490090-07:00 0 W0316 14:24:05.489353 7452 thread.cc:1808] @ 0x7fb4e02cc178 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:05.669908-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e5e960 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:05.669938-07:00 0 W0316 14:24:05.669908 7452 thread.cc:1808] @ 0x7fb4e0e5e960 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:05.840079-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e00ee124 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:05.840120-07:00 0 W0316 14:24:05.840079 7452 thread.cc:1808] @ 0x7fb4e00ee124 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:06.013610-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:06.013639-07:00 0 W0316 14:24:06.013610 7452 thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:06.187846-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:06.187874-07:00 0 W0316 14:24:06.187846 7452 thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:06.361640-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:06.361667-07:00 0 W0316 14:24:06.361640 7452 thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:06.533566-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:06.533595-07:00 0 W0316 14:24:06.533566 7452 thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:06.712370-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| I 2026-03-16T14:24:06.712400-07:00 0 W0316 14:24:06.712370 7452 thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| W 2026-03-16T14:24:06.890253-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| I 2026-03-16T14:24:06.890282-07:00 0 W0316 14:24:06.890253 7452 thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| W 2026-03-16T14:24:07.066448-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| I 2026-03-16T14:24:07.066476-07:00 0 W0316 14:24:07.066448 7452 thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| W 2026-03-16T14:24:07.242299-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| I 2026-03-16T14:24:07.242329-07:00 0 W0316 14:24:07.242299 7452 thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| W 2026-03-16T14:24:07.417729-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| I 2026-03-16T14:24:07.417757-07:00 0 W0316 14:24:07.417729 7452 thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| W 2026-03-16T14:24:07.592584-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| I 2026-03-16T14:24:07.592612-07:00 0 W0316 14:24:07.592584 7452 thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| W 2026-03-16T14:24:07.768806-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| I 2026-03-16T14:24:07.768835-07:00 0 W0316 14:24:07.768806 7452 thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| W 2026-03-16T14:24:07.948287-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| I 2026-03-16T14:24:07.948336-07:00 0 W0316 14:24:07.948287 7452 thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| W 2026-03-16T14:24:08.125989-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| I 2026-03-16T14:24:08.126019-07:00 0 W0316 14:24:08.125989 7452 thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| W 2026-03-16T14:24:08.300899-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:08.300932-07:00 0 W0316 14:24:08.300899 7452 thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| W 2026-03-16T14:24:08.478916-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:08.478947-07:00 0 W0316 14:24:08.478916 7452 thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| W 2026-03-16T14:24:08.656184-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:08.656199-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:08.656203-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:08.656205-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:08.656215-07:00 0 W0316 14:24:08.656184 7452 thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:08.656217-07:00 0 W0316 14:24:08.656199 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:08.656218-07:00 0 W0316 14:24:08.656203 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:08.656219-07:00 0 W0316 14:24:08.656205 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:08.656310-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:08.656334-07:00 0 W0316 14:24:08.656310 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:08.657023-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2bc81a700 (name: py_xla_callback/5843) stack: --- | |
| stack used: 158 KiB of 2104 KiB | |
| W 2026-03-16T14:24:08.657032-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| W 2026-03-16T14:24:08.657035-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| I 2026-03-16T14:24:08.657055-07:00 0 W0316 14:24:08.657023 7452 thread.cc:1808] --- Thread 7fb2bc81a700 (name: py_xla_callback/5843) stack: --- | |
| I 2026-03-16T14:24:08.657057-07:00 0 stack used: 158 KiB of 2104 KiB | |
| I 2026-03-16T14:24:08.657057-07:00 0 W0316 14:24:08.657032 7452 thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| I 2026-03-16T14:24:08.657059-07:00 0 W0316 14:24:08.657035 7452 thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:08.825719-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6850 xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:08.825738-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:08.825746-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:08.825749-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:08.825755-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c0948186f 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:08.825756-07:00 0 W0316 14:24:08.825719 7452 thread.cc:1808] @ 0x560c094d6850 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:08.825758-07:00 0 W0316 14:24:08.825738 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:08.825759-07:00 0 W0316 14:24:08.825746 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:08.825760-07:00 0 W0316 14:24:08.825749 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:08.825764-07:00 0 W0316 14:24:08.825755 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c0948186f 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:08.825886-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2bcb49700 (name: py_xla_cleanup/5844) stack: --- | |
| stack used: 158 KiB of 2104 KiB | |
| W 2026-03-16T14:24:08.825895-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| W 2026-03-16T14:24:08.825897-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| I 2026-03-16T14:24:08.825911-07:00 0 W0316 14:24:08.825886 7452 thread.cc:1808] --- Thread 7fb2bcb49700 (name: py_xla_cleanup/5844) stack: --- | |
| I 2026-03-16T14:24:08.825912-07:00 0 stack used: 158 KiB of 2104 KiB | |
| I 2026-03-16T14:24:08.825913-07:00 0 W0316 14:24:08.825895 7452 thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| I 2026-03-16T14:24:08.825914-07:00 0 W0316 14:24:08.825897 7452 thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:08.988604-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d67ee xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:08.988618-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:08.988621-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:08.988623-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:08.988626-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c094818e3 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:08.988635-07:00 0 W0316 14:24:08.988604 7452 thread.cc:1808] @ 0x560c094d67ee xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:08.988637-07:00 0 W0316 14:24:08.988618 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:08.988638-07:00 0 W0316 14:24:08.988621 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:08.988639-07:00 0 W0316 14:24:08.988623 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:08.988640-07:00 0 W0316 14:24:08.988626 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c094818e3 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:08.989244-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2bce78700 (name: py_xla_execute/5845) stack: --- | |
| stack used: 166 KiB of 2104 KiB | |
| I 2026-03-16T14:24:08.989268-07:00 0 W0316 14:24:08.989244 7452 thread.cc:1808] --- Thread 7fb2bce78700 (name: py_xla_execute/5845) stack: --- | |
| I 2026-03-16T14:24:08.989269-07:00 0 stack used: 166 KiB of 2104 KiB | |
| W 2026-03-16T14:24:09.155673-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0323ae3 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.155702-07:00 0 W0316 14:24:09.155673 7452 thread.cc:1808] @ 0x7fb4e0323ae3 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.322392-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e00e18b3 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.322422-07:00 0 W0316 14:24:09.322392 7452 thread.cc:1808] @ 0x7fb4e00e18b3 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.489969-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e012051b _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.489997-07:00 0 W0316 14:24:09.489969 7452 thread.cc:1808] @ 0x7fb4e012051b _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.657475-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e5e837 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.657508-07:00 0 W0316 14:24:09.657475 7452 thread.cc:1808] @ 0x7fb4e0e5e837 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.825705-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e5ec85 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.825723-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e00ee124 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.825726-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.825729-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.825731-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.825734-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.825736-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| W 2026-03-16T14:24:09.825740-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| W 2026-03-16T14:24:09.825743-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| W 2026-03-16T14:24:09.825746-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| W 2026-03-16T14:24:09.825749-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| W 2026-03-16T14:24:09.825752-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| W 2026-03-16T14:24:09.825755-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| I 2026-03-16T14:24:09.825756-07:00 0 W0316 14:24:09.825705 7452 thread.cc:1808] @ 0x7fb4e0e5ec85 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.825758-07:00 0 W0316 14:24:09.825723 7452 thread.cc:1808] @ 0x7fb4e00ee124 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.825758-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| I 2026-03-16T14:24:09.825759-07:00 0 W0316 14:24:09.825726 7452 thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.825759-07:00 0 W0316 14:24:09.825729 7452 thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.825760-07:00 0 W0316 14:24:09.825731 7452 thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.825761-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| I 2026-03-16T14:24:09.825761-07:00 0 W0316 14:24:09.825734 7452 thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.825762-07:00 0 W0316 14:24:09.825736 7452 thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| W 2026-03-16T14:24:09.825763-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:09.825763-07:00 0 W0316 14:24:09.825740 7452 thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| I 2026-03-16T14:24:09.825764-07:00 0 W0316 14:24:09.825743 7452 thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| I 2026-03-16T14:24:09.825765-07:00 0 W0316 14:24:09.825746 7452 thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| W 2026-03-16T14:24:09.825765-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:09.825765-07:00 0 W0316 14:24:09.825749 7452 thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| I 2026-03-16T14:24:09.825766-07:00 0 W0316 14:24:09.825752 7452 thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| W 2026-03-16T14:24:09.825768-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:09.825771-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:09.825774-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:09.825777-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:09.825781-07:00 0 W0316 14:24:09.825755 7452 thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| I 2026-03-16T14:24:09.825782-07:00 0 W0316 14:24:09.825758 7452 thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| I 2026-03-16T14:24:09.825782-07:00 0 W0316 14:24:09.825761 7452 thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| W 2026-03-16T14:24:09.825783-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:09.825783-07:00 0 W0316 14:24:09.825763 7452 thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:09.825784-07:00 0 W0316 14:24:09.825765 7452 thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:09.825785-07:00 0 W0316 14:24:09.825768 7452 thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:09.825786-07:00 0 W0316 14:24:09.825771 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:09.825787-07:00 0 W0316 14:24:09.825774 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:09.825788-07:00 0 W0316 14:24:09.825777 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:09.825791-07:00 0 W0316 14:24:09.825783 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:09.825915-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2bd1a7700 (name: py_xla_callback/5846) stack: --- | |
| stack used: 158 KiB of 2104 KiB | |
| W 2026-03-16T14:24:09.825926-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| W 2026-03-16T14:24:09.825928-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:09.825931-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6850 xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:09.825933-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:09.825935-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:09.825937-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:09.825941-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c0948186f 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:09.825946-07:00 0 W0316 14:24:09.825915 7452 thread.cc:1808] --- Thread 7fb2bd1a7700 (name: py_xla_callback/5846) stack: --- | |
| I 2026-03-16T14:24:09.825947-07:00 0 stack used: 158 KiB of 2104 KiB | |
| I 2026-03-16T14:24:09.825948-07:00 0 W0316 14:24:09.825926 7452 thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| I 2026-03-16T14:24:09.825949-07:00 0 W0316 14:24:09.825928 7452 thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| I 2026-03-16T14:24:09.825950-07:00 0 W0316 14:24:09.825931 7452 thread.cc:1808] @ 0x560c094d6850 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:09.825950-07:00 0 W0316 14:24:09.825933 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:09.825951-07:00 0 W0316 14:24:09.825935 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:09.825952-07:00 0 W0316 14:24:09.825937 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:09.825955-07:00 0 W0316 14:24:09.825941 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c0948186f 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:09.826051-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2bd4d6700 (name: py_xla_cleanup/5847) stack: --- | |
| stack used: 158 KiB of 2104 KiB | |
| W 2026-03-16T14:24:09.826061-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| W 2026-03-16T14:24:09.826063-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:09.826066-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d67ee xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:09.826068-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:09.826070-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:09.826072-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:09.826075-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c094818e3 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:09.826085-07:00 0 W0316 14:24:09.826051 7452 thread.cc:1808] --- Thread 7fb2bd4d6700 (name: py_xla_cleanup/5847) stack: --- | |
| I 2026-03-16T14:24:09.826086-07:00 0 stack used: 158 KiB of 2104 KiB | |
| I 2026-03-16T14:24:09.826087-07:00 0 W0316 14:24:09.826061 7452 thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| I 2026-03-16T14:24:09.826088-07:00 0 W0316 14:24:09.826063 7452 thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| I 2026-03-16T14:24:09.826088-07:00 0 W0316 14:24:09.826066 7452 thread.cc:1808] @ 0x560c094d67ee xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:09.826089-07:00 0 W0316 14:24:09.826068 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:09.826090-07:00 0 W0316 14:24:09.826070 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:09.826091-07:00 0 W0316 14:24:09.826072 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:09.826091-07:00 0 W0316 14:24:09.826075 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c094818e3 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:09.826204-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2bd805700 (name: py_xla_execute/5848) stack: --- | |
| stack used: 166 KiB of 2104 KiB | |
| I 2026-03-16T14:24:09.826227-07:00 0 W0316 14:24:09.826204 7452 thread.cc:1808] --- Thread 7fb2bd805700 (name: py_xla_execute/5848) stack: --- | |
| I 2026-03-16T14:24:09.826228-07:00 0 stack used: 166 KiB of 2104 KiB | |
| W 2026-03-16T14:24:09.994132-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0323b38 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.994146-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e00e18b3 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.994148-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e012051b _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.994155-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e5e837 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.994157-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e5ec85 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.994160-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e00ee124 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.994163-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.994166-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.994169-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.994172-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.994173-07:00 0 W0316 14:24:09.994132 7452 thread.cc:1808] @ 0x7fb4e0323b38 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.994174-07:00 0 W0316 14:24:09.994146 7452 thread.cc:1808] @ 0x7fb4e00e18b3 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.994175-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| I 2026-03-16T14:24:09.994175-07:00 0 W0316 14:24:09.994148 7452 thread.cc:1808] @ 0x7fb4e012051b _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.994176-07:00 0 W0316 14:24:09.994155 7452 thread.cc:1808] @ 0x7fb4e0e5e837 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.994176-07:00 0 W0316 14:24:09.994157 7452 thread.cc:1808] @ 0x7fb4e0e5ec85 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.994177-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| I 2026-03-16T14:24:09.994177-07:00 0 W0316 14:24:09.994160 7452 thread.cc:1808] @ 0x7fb4e00ee124 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.994178-07:00 0 W0316 14:24:09.994163 7452 thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.994179-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| I 2026-03-16T14:24:09.994180-07:00 0 W0316 14:24:09.994166 7452 thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.994181-07:00 0 W0316 14:24:09.994169 7452 thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:09.994181-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| W 2026-03-16T14:24:09.994184-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| W 2026-03-16T14:24:09.994186-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| W 2026-03-16T14:24:09.994189-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| I 2026-03-16T14:24:09.994192-07:00 0 W0316 14:24:09.994172 7452 thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:09.994193-07:00 0 W0316 14:24:09.994175 7452 thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| I 2026-03-16T14:24:09.994194-07:00 0 W0316 14:24:09.994177 7452 thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| I 2026-03-16T14:24:09.994195-07:00 0 W0316 14:24:09.994179 7452 thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| I 2026-03-16T14:24:09.994195-07:00 0 W0316 14:24:09.994181 7452 thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| I 2026-03-16T14:24:09.994196-07:00 0 W0316 14:24:09.994184 7452 thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| I 2026-03-16T14:24:09.994197-07:00 0 W0316 14:24:09.994186 7452 thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| W 2026-03-16T14:24:09.994216-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| W 2026-03-16T14:24:09.994219-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| W 2026-03-16T14:24:09.994221-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| W 2026-03-16T14:24:09.994228-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| W 2026-03-16T14:24:09.994230-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:09.994236-07:00 0 W0316 14:24:09.994189 7452 thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| W 2026-03-16T14:24:09.994237-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:09.994237-07:00 0 W0316 14:24:09.994216 7452 thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| I 2026-03-16T14:24:09.994238-07:00 0 W0316 14:24:09.994219 7452 thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| I 2026-03-16T14:24:09.994239-07:00 0 W0316 14:24:09.994221 7452 thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:09.994239-07:00 0 W0316 14:24:09.994228 7452 thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| W 2026-03-16T14:24:09.994240-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:09.994240-07:00 0 W0316 14:24:09.994230 7452 thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:09.994242-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:09.994245-07:00 0 W0316 14:24:09.994237 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:09.994246-07:00 0 W0316 14:24:09.994240 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:09.994247-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:09.994247-07:00 0 W0316 14:24:09.994242 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:09.994251-07:00 0 W0316 14:24:09.994247 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:09.994349-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2bdb13700 (name: py_xla_callback/5849) stack: --- | |
| stack used: 158 KiB of 2104 KiB | |
| W 2026-03-16T14:24:09.994358-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| W 2026-03-16T14:24:09.994360-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:09.994363-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6850 xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:09.994366-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:09.994368-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:09.994370-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:09.994373-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c0948186f 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:09.994382-07:00 0 W0316 14:24:09.994349 7452 thread.cc:1808] --- Thread 7fb2bdb13700 (name: py_xla_callback/5849) stack: --- | |
| I 2026-03-16T14:24:09.994383-07:00 0 stack used: 158 KiB of 2104 KiB | |
| I 2026-03-16T14:24:09.994384-07:00 0 W0316 14:24:09.994358 7452 thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| I 2026-03-16T14:24:09.994384-07:00 0 W0316 14:24:09.994360 7452 thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| I 2026-03-16T14:24:09.994385-07:00 0 W0316 14:24:09.994363 7452 thread.cc:1808] @ 0x560c094d6850 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:09.994386-07:00 0 W0316 14:24:09.994366 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:09.994387-07:00 0 W0316 14:24:09.994368 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:09.994387-07:00 0 W0316 14:24:09.994370 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:09.994388-07:00 0 W0316 14:24:09.994373 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c0948186f 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:09.994466-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2a1af1700 (name: py_xla_cleanup/5850) stack: --- | |
| stack used: 158 KiB of 2104 KiB | |
| W 2026-03-16T14:24:09.994475-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| W 2026-03-16T14:24:09.994477-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:09.994480-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d67ee xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:09.994482-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:09.994490-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:09.994493-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:09.994496-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c094818e3 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:09.994497-07:00 0 W0316 14:24:09.994466 7452 thread.cc:1808] --- Thread 7fb2a1af1700 (name: py_xla_cleanup/5850) stack: --- | |
| I 2026-03-16T14:24:09.994498-07:00 0 stack used: 158 KiB of 2104 KiB | |
| I 2026-03-16T14:24:09.994498-07:00 0 W0316 14:24:09.994475 7452 thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| I 2026-03-16T14:24:09.994499-07:00 0 W0316 14:24:09.994477 7452 thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| I 2026-03-16T14:24:09.994500-07:00 0 W0316 14:24:09.994480 7452 thread.cc:1808] @ 0x560c094d67ee xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:09.994501-07:00 0 W0316 14:24:09.994482 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:09.994502-07:00 0 W0316 14:24:09.994490 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:09.994503-07:00 0 W0316 14:24:09.994493 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:09.994505-07:00 0 W0316 14:24:09.994496 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c094818e3 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:09.994633-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2a17e3700 (name: py_xla_execute/5851) stack: --- | |
| stack used: 165 KiB of 2104 KiB | |
| I 2026-03-16T14:24:09.994658-07:00 0 W0316 14:24:09.994633 7452 thread.cc:1808] --- Thread 7fb2a17e3700 (name: py_xla_execute/5851) stack: --- | |
| I 2026-03-16T14:24:09.994659-07:00 0 stack used: 165 KiB of 2104 KiB | |
| W 2026-03-16T14:24:09.995416-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8211e72 clock_gettime | |
| I 2026-03-16T14:24:09.995444-07:00 0 W0316 14:24:09.995416 7452 thread.cc:1808] @ 0x7fb4e8211e72 clock_gettime | |
| W 2026-03-16T14:24:10.162131-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e01ddee4 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.162158-07:00 0 W0316 14:24:10.162131 7452 thread.cc:1808] @ 0x7fb4e01ddee4 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.329201-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e00ee3fc _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.329215-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.329219-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.329221-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.329223-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.329226-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| W 2026-03-16T14:24:10.329228-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| W 2026-03-16T14:24:10.329232-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| W 2026-03-16T14:24:10.329235-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| W 2026-03-16T14:24:10.329238-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| W 2026-03-16T14:24:10.329241-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| I 2026-03-16T14:24:10.329242-07:00 0 W0316 14:24:10.329201 7452 thread.cc:1808] @ 0x7fb4e00ee3fc _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.329243-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| I 2026-03-16T14:24:10.329244-07:00 0 W0316 14:24:10.329215 7452 thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.329245-07:00 0 W0316 14:24:10.329219 7452 thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.329246-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| I 2026-03-16T14:24:10.329246-07:00 0 W0316 14:24:10.329221 7452 thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.329246-07:00 0 W0316 14:24:10.329223 7452 thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.329247-07:00 0 W0316 14:24:10.329226 7452 thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| W 2026-03-16T14:24:10.329248-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| I 2026-03-16T14:24:10.329248-07:00 0 W0316 14:24:10.329228 7452 thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| I 2026-03-16T14:24:10.329249-07:00 0 W0316 14:24:10.329232 7452 thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| W 2026-03-16T14:24:10.329250-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:10.329250-07:00 0 W0316 14:24:10.329235 7452 thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| I 2026-03-16T14:24:10.329251-07:00 0 W0316 14:24:10.329238 7452 thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| W 2026-03-16T14:24:10.329252-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| W 2026-03-16T14:24:10.329254-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:10.329258-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:10.329260-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:10.329263-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.329267-07:00 0 W0316 14:24:10.329241 7452 thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| I 2026-03-16T14:24:10.329268-07:00 0 W0316 14:24:10.329243 7452 thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| W 2026-03-16T14:24:10.329268-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:10.329268-07:00 0 W0316 14:24:10.329246 7452 thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| I 2026-03-16T14:24:10.329269-07:00 0 W0316 14:24:10.329248 7452 thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| I 2026-03-16T14:24:10.329270-07:00 0 W0316 14:24:10.329250 7452 thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:10.329271-07:00 0 W0316 14:24:10.329252 7452 thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:10.329271-07:00 0 W0316 14:24:10.329254 7452 thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.329272-07:00 0 W0316 14:24:10.329258 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:10.329273-07:00 0 W0316 14:24:10.329260 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.329274-07:00 0 W0316 14:24:10.329263 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.329277-07:00 0 W0316 14:24:10.329268 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:10.329372-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2a14d5700 (name: py_xla_callback/5852) stack: --- | |
| stack used: 158 KiB of 2104 KiB | |
| W 2026-03-16T14:24:10.329382-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| W 2026-03-16T14:24:10.329385-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:10.329387-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6850 xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:10.329390-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:10.329392-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:10.329394-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:10.329397-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c0948186f 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:10.329405-07:00 0 W0316 14:24:10.329372 7452 thread.cc:1808] --- Thread 7fb2a14d5700 (name: py_xla_callback/5852) stack: --- | |
| I 2026-03-16T14:24:10.329406-07:00 0 stack used: 158 KiB of 2104 KiB | |
| I 2026-03-16T14:24:10.329407-07:00 0 W0316 14:24:10.329382 7452 thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| I 2026-03-16T14:24:10.329407-07:00 0 W0316 14:24:10.329385 7452 thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| I 2026-03-16T14:24:10.329408-07:00 0 W0316 14:24:10.329387 7452 thread.cc:1808] @ 0x560c094d6850 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.329409-07:00 0 W0316 14:24:10.329390 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:10.329410-07:00 0 W0316 14:24:10.329392 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.329410-07:00 0 W0316 14:24:10.329394 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.329411-07:00 0 W0316 14:24:10.329397 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c0948186f 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:10.329499-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2a11c7700 (name: py_xla_cleanup/5853) stack: --- | |
| stack used: 158 KiB of 2104 KiB | |
| W 2026-03-16T14:24:10.329511-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| W 2026-03-16T14:24:10.329514-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:10.329516-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d67ee xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:10.329519-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:10.329521-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:10.329523-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:10.329526-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c094818e3 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:10.329531-07:00 0 W0316 14:24:10.329499 7452 thread.cc:1808] --- Thread 7fb2a11c7700 (name: py_xla_cleanup/5853) stack: --- | |
| I 2026-03-16T14:24:10.329532-07:00 0 stack used: 158 KiB of 2104 KiB | |
| I 2026-03-16T14:24:10.329533-07:00 0 W0316 14:24:10.329511 7452 thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| I 2026-03-16T14:24:10.329534-07:00 0 W0316 14:24:10.329514 7452 thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| I 2026-03-16T14:24:10.329534-07:00 0 W0316 14:24:10.329516 7452 thread.cc:1808] @ 0x560c094d67ee xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.329535-07:00 0 W0316 14:24:10.329519 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:10.329536-07:00 0 W0316 14:24:10.329521 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.329537-07:00 0 W0316 14:24:10.329523 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.329538-07:00 0 W0316 14:24:10.329526 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c094818e3 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:10.329655-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2a0eb9700 (name: py_xla_execute/5854) stack: --- | |
| stack used: 166 KiB of 2104 KiB | |
| I 2026-03-16T14:24:10.329678-07:00 0 W0316 14:24:10.329655 7452 thread.cc:1808] --- Thread 7fb2a0eb9700 (name: py_xla_execute/5854) stack: --- | |
| I 2026-03-16T14:24:10.329679-07:00 0 stack used: 166 KiB of 2104 KiB | |
| W 2026-03-16T14:24:10.496300-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0323aff _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.496313-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e00e18b3 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.496317-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e012051b _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.496320-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e5e837 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.496323-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e5ec85 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.496325-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e00ee124 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.496328-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.496330-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.496333-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.496335-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.496338-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| W 2026-03-16T14:24:10.496341-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| I 2026-03-16T14:24:10.496342-07:00 0 W0316 14:24:10.496300 7452 thread.cc:1808] @ 0x7fb4e0323aff _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.496344-07:00 0 W0316 14:24:10.496313 7452 thread.cc:1808] @ 0x7fb4e00e18b3 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.496344-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| I 2026-03-16T14:24:10.496344-07:00 0 W0316 14:24:10.496317 7452 thread.cc:1808] @ 0x7fb4e012051b _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.496345-07:00 0 W0316 14:24:10.496320 7452 thread.cc:1808] @ 0x7fb4e0e5e837 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.496346-07:00 0 W0316 14:24:10.496323 7452 thread.cc:1808] @ 0x7fb4e0e5ec85 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.496346-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| I 2026-03-16T14:24:10.496347-07:00 0 W0316 14:24:10.496325 7452 thread.cc:1808] @ 0x7fb4e00ee124 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.496348-07:00 0 W0316 14:24:10.496328 7452 thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.496348-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| I 2026-03-16T14:24:10.496348-07:00 0 W0316 14:24:10.496330 7452 thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.496349-07:00 0 W0316 14:24:10.496333 7452 thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.496350-07:00 0 W0316 14:24:10.496335 7452 thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.496350-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| I 2026-03-16T14:24:10.496351-07:00 0 W0316 14:24:10.496338 7452 thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| W 2026-03-16T14:24:10.496352-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| W 2026-03-16T14:24:10.496355-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| W 2026-03-16T14:24:10.496358-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| W 2026-03-16T14:24:10.496361-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| W 2026-03-16T14:24:10.496363-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| W 2026-03-16T14:24:10.496366-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.496367-07:00 0 W0316 14:24:10.496341 7452 thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| I 2026-03-16T14:24:10.496367-07:00 0 W0316 14:24:10.496344 7452 thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| W 2026-03-16T14:24:10.496368-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:10.496368-07:00 0 W0316 14:24:10.496346 7452 thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| W 2026-03-16T14:24:10.496371-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.496371-07:00 0 W0316 14:24:10.496348 7452 thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| I 2026-03-16T14:24:10.496371-07:00 0 W0316 14:24:10.496350 7452 thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| I 2026-03-16T14:24:10.496372-07:00 0 W0316 14:24:10.496352 7452 thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| W 2026-03-16T14:24:10.496372-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.496373-07:00 0 W0316 14:24:10.496355 7452 thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| I 2026-03-16T14:24:10.496374-07:00 0 W0316 14:24:10.496358 7452 thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| I 2026-03-16T14:24:10.496375-07:00 0 W0316 14:24:10.496361 7452 thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| W 2026-03-16T14:24:10.496375-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:10.496375-07:00 0 W0316 14:24:10.496363 7452 thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:10.496382-07:00 0 W0316 14:24:10.496366 7452 thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.496383-07:00 0 W0316 14:24:10.496368 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:10.496383-07:00 0 W0316 14:24:10.496371 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.496384-07:00 0 W0316 14:24:10.496372 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.496385-07:00 0 W0316 14:24:10.496375 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:10.496481-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2a0bab700 (name: py_xla_callback/5855) stack: --- | |
| stack used: 158 KiB of 2104 KiB | |
| W 2026-03-16T14:24:10.496491-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| W 2026-03-16T14:24:10.496493-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:10.496496-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6850 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.496507-07:00 0 W0316 14:24:10.496481 7452 thread.cc:1808] --- Thread 7fb2a0bab700 (name: py_xla_callback/5855) stack: --- | |
| I 2026-03-16T14:24:10.496508-07:00 0 stack used: 158 KiB of 2104 KiB | |
| I 2026-03-16T14:24:10.496509-07:00 0 W0316 14:24:10.496491 7452 thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| I 2026-03-16T14:24:10.496510-07:00 0 W0316 14:24:10.496493 7452 thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:10.496518-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:10.496521-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:10.496523-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:10.496527-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c0948186f 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:10.496532-07:00 0 W0316 14:24:10.496496 7452 thread.cc:1808] @ 0x560c094d6850 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.496533-07:00 0 W0316 14:24:10.496518 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:10.496534-07:00 0 W0316 14:24:10.496521 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.496535-07:00 0 W0316 14:24:10.496523 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.496535-07:00 0 W0316 14:24:10.496527 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c0948186f 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:10.496627-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2a089d700 (name: py_xla_cleanup/5856) stack: --- | |
| stack used: 158 KiB of 2104 KiB | |
| W 2026-03-16T14:24:10.496635-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| W 2026-03-16T14:24:10.496638-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:10.496641-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d67ee xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:10.496643-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:10.496645-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:10.496647-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:10.496650-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c094818e3 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:10.496659-07:00 0 W0316 14:24:10.496627 7452 thread.cc:1808] --- Thread 7fb2a089d700 (name: py_xla_cleanup/5856) stack: --- | |
| I 2026-03-16T14:24:10.496660-07:00 0 stack used: 158 KiB of 2104 KiB | |
| I 2026-03-16T14:24:10.496661-07:00 0 W0316 14:24:10.496635 7452 thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| I 2026-03-16T14:24:10.496662-07:00 0 W0316 14:24:10.496638 7452 thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| I 2026-03-16T14:24:10.496662-07:00 0 W0316 14:24:10.496641 7452 thread.cc:1808] @ 0x560c094d67ee xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.496663-07:00 0 W0316 14:24:10.496643 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:10.496664-07:00 0 W0316 14:24:10.496645 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.496665-07:00 0 W0316 14:24:10.496647 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.496666-07:00 0 W0316 14:24:10.496650 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c094818e3 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:10.496770-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb2a054f700 (name: py_xla_execute/5857) stack: --- | |
| stack used: 166 KiB of 2104 KiB | |
| I 2026-03-16T14:24:10.496793-07:00 0 W0316 14:24:10.496770 7452 thread.cc:1808] --- Thread 7fb2a054f700 (name: py_xla_execute/5857) stack: --- | |
| I 2026-03-16T14:24:10.496794-07:00 0 stack used: 166 KiB of 2104 KiB | |
| W 2026-03-16T14:24:10.664235-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0323b0b _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.664249-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e00e18b3 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.664252-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e012051b _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.664256-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e5e837 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.664258-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e5ec85 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.664261-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e00ee124 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.664270-07:00 0 W0316 14:24:10.664235 7452 thread.cc:1808] @ 0x7fb4e0323b0b _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.664272-07:00 0 W0316 14:24:10.664249 7452 thread.cc:1808] @ 0x7fb4e00e18b3 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.664272-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.664273-07:00 0 W0316 14:24:10.664252 7452 thread.cc:1808] @ 0x7fb4e012051b _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.664274-07:00 0 W0316 14:24:10.664256 7452 thread.cc:1808] @ 0x7fb4e0e5e837 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.664275-07:00 0 W0316 14:24:10.664258 7452 thread.cc:1808] @ 0x7fb4e0e5ec85 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.664275-07:00 0 W0316 14:24:10.664261 7452 thread.cc:1808] @ 0x7fb4e00ee124 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.664275-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.664278-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.664281-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.664284-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| I 2026-03-16T14:24:10.664284-07:00 0 W0316 14:24:10.664272 7452 thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.664285-07:00 0 W0316 14:24:10.664275 7452 thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.664286-07:00 0 W0316 14:24:10.664278 7452 thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.664287-07:00 0 W0316 14:24:10.664281 7452 thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.664287-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| W 2026-03-16T14:24:10.664290-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| W 2026-03-16T14:24:10.664292-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| I 2026-03-16T14:24:10.664293-07:00 0 W0316 14:24:10.664284 7452 thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| I 2026-03-16T14:24:10.664294-07:00 0 W0316 14:24:10.664287 7452 thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| I 2026-03-16T14:24:10.664295-07:00 0 W0316 14:24:10.664290 7452 thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| W 2026-03-16T14:24:10.664301-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| W 2026-03-16T14:24:10.664303-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| W 2026-03-16T14:24:10.664307-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| W 2026-03-16T14:24:10.664310-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| I 2026-03-16T14:24:10.664310-07:00 0 W0316 14:24:10.664292 7452 thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| I 2026-03-16T14:24:10.664311-07:00 0 W0316 14:24:10.664301 7452 thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| I 2026-03-16T14:24:10.664312-07:00 0 W0316 14:24:10.664303 7452 thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| W 2026-03-16T14:24:10.664312-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| I 2026-03-16T14:24:10.664312-07:00 0 W0316 14:24:10.664307 7452 thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| W 2026-03-16T14:24:10.664318-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:10.664319-07:00 0 W0316 14:24:10.664310 7452 thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| I 2026-03-16T14:24:10.664320-07:00 0 W0316 14:24:10.664312 7452 thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| W 2026-03-16T14:24:10.664321-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| W 2026-03-16T14:24:10.664324-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.664325-07:00 0 W0316 14:24:10.664318 7452 thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:10.664326-07:00 0 W0316 14:24:10.664321 7452 thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| W 2026-03-16T14:24:10.664327-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:10.664330-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.664330-07:00 0 W0316 14:24:10.664324 7452 thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.664331-07:00 0 W0316 14:24:10.664327 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:10.664332-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.664336-07:00 0 W0316 14:24:10.664330 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:10.664336-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:10.664337-07:00 0 W0316 14:24:10.664332 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.664340-07:00 0 W0316 14:24:10.664336 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:10.664433-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb169fff700 (name: py_xla_callback/5858) stack: --- | |
| stack used: 158 KiB of 2104 KiB | |
| W 2026-03-16T14:24:10.664442-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| W 2026-03-16T14:24:10.664445-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:10.664447-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6850 xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:10.664449-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:10.664452-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:10.664454-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:10.664458-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c0948186f 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:10.664466-07:00 0 W0316 14:24:10.664433 7452 thread.cc:1808] --- Thread 7fb169fff700 (name: py_xla_callback/5858) stack: --- | |
| I 2026-03-16T14:24:10.664467-07:00 0 stack used: 158 KiB of 2104 KiB | |
| I 2026-03-16T14:24:10.664468-07:00 0 W0316 14:24:10.664442 7452 thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| I 2026-03-16T14:24:10.664469-07:00 0 W0316 14:24:10.664445 7452 thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| I 2026-03-16T14:24:10.664470-07:00 0 W0316 14:24:10.664447 7452 thread.cc:1808] @ 0x560c094d6850 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.664470-07:00 0 W0316 14:24:10.664449 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:10.664471-07:00 0 W0316 14:24:10.664452 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.664472-07:00 0 W0316 14:24:10.664454 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.664473-07:00 0 W0316 14:24:10.664458 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c0948186f 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:10.664549-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb169cf1700 (name: py_xla_cleanup/5859) stack: --- | |
| stack used: 158 KiB of 2104 KiB | |
| W 2026-03-16T14:24:10.664558-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| W 2026-03-16T14:24:10.664561-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:10.664563-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d67ee xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:10.664565-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:10.664567-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:10.664570-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:10.664572-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c094818e3 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:10.664580-07:00 0 W0316 14:24:10.664549 7452 thread.cc:1808] --- Thread 7fb169cf1700 (name: py_xla_cleanup/5859) stack: --- | |
| I 2026-03-16T14:24:10.664582-07:00 0 stack used: 158 KiB of 2104 KiB | |
| I 2026-03-16T14:24:10.664582-07:00 0 W0316 14:24:10.664558 7452 thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| I 2026-03-16T14:24:10.664583-07:00 0 W0316 14:24:10.664561 7452 thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| I 2026-03-16T14:24:10.664584-07:00 0 W0316 14:24:10.664563 7452 thread.cc:1808] @ 0x560c094d67ee xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.664585-07:00 0 W0316 14:24:10.664565 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:10.664586-07:00 0 W0316 14:24:10.664567 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.664586-07:00 0 W0316 14:24:10.664570 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.664587-07:00 0 W0316 14:24:10.664572 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c094818e3 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:10.664693-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb1699e3700 (name: py_xla_execute/5860) stack: --- | |
| stack used: 166 KiB of 2104 KiB | |
| I 2026-03-16T14:24:10.664717-07:00 0 W0316 14:24:10.664693 7452 thread.cc:1808] --- Thread 7fb1699e3700 (name: py_xla_execute/5860) stack: --- | |
| I 2026-03-16T14:24:10.664718-07:00 0 stack used: 166 KiB of 2104 KiB | |
| W 2026-03-16T14:24:10.831591-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0323b6d _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.831604-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e00e18b3 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.831606-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e012051b _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.831608-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e5e837 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.831612-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e5ec85 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.831614-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e00ee124 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.831616-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.831620-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.831622-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.831624-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.831627-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| W 2026-03-16T14:24:10.831630-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| W 2026-03-16T14:24:10.831633-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| W 2026-03-16T14:24:10.831636-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| I 2026-03-16T14:24:10.831638-07:00 0 W0316 14:24:10.831591 7452 thread.cc:1808] @ 0x7fb4e0323b6d _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.831638-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| I 2026-03-16T14:24:10.831639-07:00 0 W0316 14:24:10.831604 7452 thread.cc:1808] @ 0x7fb4e00e18b3 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.831640-07:00 0 W0316 14:24:10.831606 7452 thread.cc:1808] @ 0x7fb4e012051b _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.831640-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| I 2026-03-16T14:24:10.831641-07:00 0 W0316 14:24:10.831608 7452 thread.cc:1808] @ 0x7fb4e0e5e837 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.831641-07:00 0 W0316 14:24:10.831612 7452 thread.cc:1808] @ 0x7fb4e0e5ec85 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.831642-07:00 0 W0316 14:24:10.831614 7452 thread.cc:1808] @ 0x7fb4e00ee124 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.831643-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| I 2026-03-16T14:24:10.831643-07:00 0 W0316 14:24:10.831616 7452 thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.831644-07:00 0 W0316 14:24:10.831620 7452 thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.831644-07:00 0 W0316 14:24:10.831622 7452 thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.831644-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| I 2026-03-16T14:24:10.831645-07:00 0 W0316 14:24:10.831624 7452 thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.831647-07:00 0 W0316 14:24:10.831627 7452 thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| W 2026-03-16T14:24:10.831647-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| I 2026-03-16T14:24:10.831648-07:00 0 W0316 14:24:10.831630 7452 thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| I 2026-03-16T14:24:10.831649-07:00 0 W0316 14:24:10.831633 7452 thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| W 2026-03-16T14:24:10.831649-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| W 2026-03-16T14:24:10.831651-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| W 2026-03-16T14:24:10.831654-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:10.831656-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:10.831659-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:10.831662-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:10.831665-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:10.831666-07:00 0 W0316 14:24:10.831636 7452 thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| I 2026-03-16T14:24:10.831667-07:00 0 W0316 14:24:10.831638 7452 thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| I 2026-03-16T14:24:10.831667-07:00 0 W0316 14:24:10.831640 7452 thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| I 2026-03-16T14:24:10.831668-07:00 0 W0316 14:24:10.831643 7452 thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| I 2026-03-16T14:24:10.831669-07:00 0 W0316 14:24:10.831644 7452 thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| I 2026-03-16T14:24:10.831670-07:00 0 W0316 14:24:10.831647 7452 thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| I 2026-03-16T14:24:10.831671-07:00 0 W0316 14:24:10.831649 7452 thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:10.831671-07:00 0 W0316 14:24:10.831651 7452 thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:10.831672-07:00 0 W0316 14:24:10.831654 7452 thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.831673-07:00 0 W0316 14:24:10.831656 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:10.831674-07:00 0 W0316 14:24:10.831659 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.831674-07:00 0 W0316 14:24:10.831662 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.831678-07:00 0 W0316 14:24:10.831665 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:10.831752-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb1696d5700 (name: py_xla_callback/5861) stack: --- | |
| stack used: 158 KiB of 2104 KiB | |
| W 2026-03-16T14:24:10.831760-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| W 2026-03-16T14:24:10.831762-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:10.831764-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6850 xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:10.831767-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:10.831769-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:10.831771-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:10.831774-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c0948186f 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:10.831783-07:00 0 W0316 14:24:10.831752 7452 thread.cc:1808] --- Thread 7fb1696d5700 (name: py_xla_callback/5861) stack: --- | |
| I 2026-03-16T14:24:10.831784-07:00 0 stack used: 158 KiB of 2104 KiB | |
| I 2026-03-16T14:24:10.831785-07:00 0 W0316 14:24:10.831760 7452 thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| I 2026-03-16T14:24:10.831785-07:00 0 W0316 14:24:10.831762 7452 thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| I 2026-03-16T14:24:10.831786-07:00 0 W0316 14:24:10.831764 7452 thread.cc:1808] @ 0x560c094d6850 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.831787-07:00 0 W0316 14:24:10.831767 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:10.831787-07:00 0 W0316 14:24:10.831769 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.831788-07:00 0 W0316 14:24:10.831771 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.831789-07:00 0 W0316 14:24:10.831774 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c0948186f 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:10.831859-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb1693c7700 (name: py_xla_cleanup/5862) stack: --- | |
| stack used: 158 KiB of 2104 KiB | |
| W 2026-03-16T14:24:10.831877-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| W 2026-03-16T14:24:10.831879-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| W 2026-03-16T14:24:10.831881-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d67ee xla::WorkerThread::WorkLoop() | |
| W 2026-03-16T14:24:10.831883-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| W 2026-03-16T14:24:10.831885-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| W 2026-03-16T14:24:10.831887-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| W 2026-03-16T14:24:10.831890-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c094818e3 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:10.831899-07:00 0 W0316 14:24:10.831859 7452 thread.cc:1808] --- Thread 7fb1693c7700 (name: py_xla_cleanup/5862) stack: --- | |
| I 2026-03-16T14:24:10.831901-07:00 0 stack used: 158 KiB of 2104 KiB | |
| I 2026-03-16T14:24:10.831902-07:00 0 W0316 14:24:10.831877 7452 thread.cc:1808] @ 0x560c06f09a96 AbslInternalPerThreadSemWait | |
| I 2026-03-16T14:24:10.831902-07:00 0 W0316 14:24:10.831879 7452 thread.cc:1808] @ 0x560c072e5374 absl::Mutex::AwaitCommon() | |
| I 2026-03-16T14:24:10.831903-07:00 0 W0316 14:24:10.831881 7452 thread.cc:1808] @ 0x560c094d67ee xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.831904-07:00 0 W0316 14:24:10.831883 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:10.831905-07:00 0 W0316 14:24:10.831885 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.831905-07:00 0 W0316 14:24:10.831887 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.831922-07:00 0 W0316 14:24:10.831890 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c094818e3 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| W 2026-03-16T14:24:10.831967-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] --- Thread 7fb1690b9700 (name: py_xla_execute/5863) stack: --- | |
| stack used: 166 KiB of 2104 KiB | |
| I 2026-03-16T14:24:10.831990-07:00 0 W0316 14:24:10.831967 7452 thread.cc:1808] --- Thread 7fb1690b9700 (name: py_xla_execute/5863) stack: --- | |
| I 2026-03-16T14:24:10.831991-07:00 0 stack used: 166 KiB of 2104 KiB | |
| W 2026-03-16T14:24:10.999023-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0323ad0 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.999037-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e00e18b3 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.999039-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e012051b _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.999042-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e5e837 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.999044-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e5ec85 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.999046-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e00ee124 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.999048-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.999050-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.999052-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.999054-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.999056-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| W 2026-03-16T14:24:10.999059-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| W 2026-03-16T14:24:10.999062-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| W 2026-03-16T14:24:10.999065-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| W 2026-03-16T14:24:10.999068-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| W 2026-03-16T14:24:10.999070-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| W 2026-03-16T14:24:10.999073-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| W 2026-03-16T14:24:10.999076-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| W 2026-03-16T14:24:10.999079-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| W 2026-03-16T14:24:10.999081-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:10.999082-07:00 0 W0316 14:24:10.999023 7452 thread.cc:1808] @ 0x7fb4e0323ad0 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.999084-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:10.999084-07:00 0 W0316 14:24:10.999037 7452 thread.cc:1808] @ 0x7fb4e00e18b3 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.999085-07:00 0 W0316 14:24:10.999039 7452 thread.cc:1808] @ 0x7fb4e012051b _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.999086-07:00 0 W0316 14:24:10.999042 7452 thread.cc:1808] @ 0x7fb4e0e5e837 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.999086-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.999086-07:00 0 W0316 14:24:10.999044 7452 thread.cc:1808] @ 0x7fb4e0e5ec85 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.999087-07:00 0 W0316 14:24:10.999046 7452 thread.cc:1808] @ 0x7fb4e00ee124 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.999087-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:10.999088-07:00 0 W0316 14:24:10.999048 7452 thread.cc:1808] @ 0x7fb4e01b4403 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.999089-07:00 0 W0316 14:24:10.999050 7452 thread.cc:1808] @ 0x7fb4e0e13685 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.999089-07:00 0 W0316 14:24:10.999052 7452 thread.cc:1808] @ 0x7fb4e026a977 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| W 2026-03-16T14:24:10.999089-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.999090-07:00 0 W0316 14:24:10.999054 7452 thread.cc:1808] @ 0x7fb4e0261860 _binary_blaze_out_haswell_opt_cuda_genfiles_third_party_gpus_cuda_compat_cuda_compat_data_o_tmpdir_filewrapper_s0_start | |
| I 2026-03-16T14:24:10.999091-07:00 0 W0316 14:24:10.999056 7452 thread.cc:1808] @ 0x560c0fd8b186 stream_executor::gpu::CudaStream::BlockHostUntilDone() | |
| W 2026-03-16T14:24:10.999092-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.999092-07:00 0 W0316 14:24:10.999059 7452 thread.cc:1808] @ 0x560c0aa3641d xla::gpu::WhileThunk::ExecuteOnStream() | |
| I 2026-03-16T14:24:10.999093-07:00 0 W0316 14:24:10.999062 7452 thread.cc:1808] @ 0x560c0aa38f4f xla::gpu::ThunkExecutor::ExecuteOnStream() | |
| I 2026-03-16T14:24:10.999094-07:00 0 W0316 14:24:10.999065 7452 thread.cc:1808] @ 0x560c0a931015 xla::gpu::(anonymous namespace)::ExecuteThunksImpl() | |
| W 2026-03-16T14:24:10.999095-07:00 7452 tf_gpu-executable-hang-watchdog/7452 thread/thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a | |
| I 2026-03-16T14:24:10.999095-07:00 0 W0316 14:24:10.999068 7452 thread.cc:1808] @ 0x560c0a92d93b xla::gpu::GpuExecutable::ExecuteThunks() | |
| I 2026-03-16T14:24:10.999096-07:00 0 W0316 14:24:10.999070 7452 thread.cc:1808] @ 0x560c09422aa2 xla::StreamExecutorGpuClient::RunAsync() | |
| I 2026-03-16T14:24:10.999098-07:00 0 W0316 14:24:10.999073 7452 thread.cc:1808] @ 0x560c09444c5c xla::PjRtStreamExecutorRawLoadedExecutable::Execute()::$_2::operator()() | |
| I 2026-03-16T14:24:10.999099-07:00 0 W0316 14:24:10.999076 7452 thread.cc:1808] @ 0x560c09443a54 xla::PjRtStreamExecutorRawLoadedExecutable::Execute() | |
| I 2026-03-16T14:24:10.999100-07:00 0 W0316 14:24:10.999079 7452 thread.cc:1808] @ 0x560c087b8a76 xla::CommonPjRtLoadedExecutable::ExecuteLaunch() | |
| I 2026-03-16T14:24:10.999109-07:00 0 W0316 14:24:10.999081 7452 thread.cc:1808] @ 0x560c0947828e absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:10.999110-07:00 0 W0316 14:24:10.999084 7452 thread.cc:1808] @ 0x560c0945baf5 absl::internal_any_invocable::RemoteInvoker<>() | |
| I 2026-03-16T14:24:10.999111-07:00 0 W0316 14:24:10.999086 7452 thread.cc:1808] @ 0x560c094d6928 xla::WorkerThread::WorkLoop() | |
| I 2026-03-16T14:24:10.999111-07:00 0 W0316 14:24:10.999087 7452 thread.cc:1808] @ 0x560c07dfe96d Thread::ThreadBody() | |
| I 2026-03-16T14:24:10.999112-07:00 0 W0316 14:24:10.999089 7452 thread.cc:1808] @ 0x7fb4e82907dd start_thread | |
| I 2026-03-16T14:24:10.999113-07:00 0 W0316 14:24:10.999092 7452 thread.cc:1808] @ 0x7fb4e8204b7f clone | |
| I 2026-03-16T14:24:10.999114-07:00 0 W0316 14:24:10.999095 7452 thread.cc:1808] creator: 0x560c072ee950 0x560c147fe08f 0x560c094d6635 0x560c09481775 0x560c0941f4d6 0x560c09414ae4 0x560c093ea101 0x560c093e8067 0x560c08a66ad3 0x7fb4e8128352 0x560bf5e2867a |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment