root ➜ /workspaces/generative_ai_with_langchain (second_edition) $ cd chapter9/ray
root ➜ /workspaces/generative_ai_with_langchain/chapter9/ray (second_edition) $ python build_index.py
2025-06-19 00:26:35,100 INFO worker.py:1841 -- Started a local Ray instance.
modules.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 349/349 [00:00<00:00, 2.21MB/s]
config_sentence_transformers.json: 100%|███████████████████████████████████████████████████████████████████████████████| 116/116 [00:00<00:00, 1.11MB/s]
README.md: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 10.4k/10.4k [00:00<00:00, 2.99MB/s]
sentence_bert_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████| 53.0/53.0 [00:00<00:00, 422kB/s]
config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 571/571 [00:00<00:00, 1.51MB/s]
model.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 438M/438M [00:20<00:00, 21.2MB/s]
tokenizer_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 363/363 [00:00<00:00, 3.87MB/s]
vocab.txt: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 232k/232k [00:00<00:00, 1.19MB/s]
tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 466k/466k [00:00<00:00, 4.32MB/s]
special_tokens_map.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 239/239 [00:00<00:00, 2.62MB/s]
config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 190/190 [00:00<00:00, 2.55MB/s]
Loading documentation from https://docs.ray.io/en/master/...
🟢 Loaded 651 documents
(preprocess_documents pid=1709) Preprocessing batch of 50 documents
Waiting for preprocessing to complete...
(preprocess_documents pid=1715) 🟢 Generated 22154 chunks
Total chunks: 293040
Starting parallel embedding...
(embed_chunks pid=1715) Embedding batch of 73260 chunks...
(preprocess_documents pid=1718) Preprocessing batch of 50 documents [repeated 13x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(preprocess_documents pid=1718) 🟢 Generated 22576 chunks [repeated 13x across cluster]
(embed_chunks pid=1714) Embedding batch of 73260 chunks... [repeated 4x across cluster]
(raylet) [2025-06-19 00:33:35,069 E 1639 1639] (raylet) node_manager.cc:3219: 1 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node (ID: 3fbcaf6ee2eac7d5950c5cb6daea4714e0980be53d9d49a51e5d444c, IP: 172.17.0.2) over the last time period. To see more information about the Workers killed on this node, use `ray logs raylet.out -ip 172.17.0.2`
(raylet)
(raylet) Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero.
(embed_chunks pid=1713) Embedding batch of 73260 chunks...
(raylet)
(raylet) [2025-06-19 00:34:35,072 E 1639 1639] (raylet) node_manager.cc:3219: 1 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node (ID: 3fbcaf6ee2eac7d5950c5cb6daea4714e0980be53d9d49a51e5d444c, IP: 172.17.0.2) over the last time period. To see more information about the Workers killed on this node, use `ray logs raylet.out -ip 172.17.0.2`
(raylet) Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero.
(embed_chunks pid=1719) Embedding batch of 73260 chunks...
(raylet) [2025-06-19 00:35:35,077 E 1639 1639] (raylet) node_manager.cc:3219: 1 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node (ID: 3fbcaf6ee2eac7d5950c5cb6daea4714e0980be53d9d49a51e5d444c, IP: 172.17.0.2) over the last time period. To see more information about the Workers killed on this node, use `ray logs raylet.out -ip 172.17.0.2`
(raylet)
(raylet) Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero.
(embed_chunks pid=1716) Embedding batch of 73260 chunks...
(raylet)
(raylet) [2025-06-19 00:36:35,080 E 1639 1639] (raylet) node_manager.cc:3219: 1 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node (ID: 3fbcaf6ee2eac7d5950c5cb6daea4714e0980be53d9d49a51e5d444c, IP: 172.17.0.2) over the last time period. To see more information about the Workers killed on this node, use `ray logs raylet.out -ip 172.17.0.2`
(raylet) Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero.
(embed_chunks pid=1717) Embedding batch of 73260 chunks...
(raylet) [2025-06-19 00:37:35,449 E 1639 1639] (raylet) node_manager.cc:3219: 1 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node (ID: 3fbcaf6ee2eac7d5950c5cb6daea4714e0980be53d9d49a51e5d444c, IP: 172.17.0.2) over the last time period. To see more information about the Workers killed on this node, use `ray logs raylet.out -ip 172.17.0.2`
(raylet)
(raylet) Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero.
(raylet)
(raylet) [2025-06-19 00:38:35,452 E 1639 1639] (raylet) node_manager.cc:3219: 1 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node (ID: 3fbcaf6ee2eac7d5950c5cb6daea4714e0980be53d9d49a51e5d444c, IP: 172.17.0.2) over the last time period. To see more information about the Workers killed on this node, use `ray logs raylet.out -ip 172.17.0.2`
(raylet) Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero.
(embed_chunks pid=1710) Embedding batch of 73260 chunks...
(raylet) [2025-06-19 00:39:35,456 E 1639 1639] (raylet) node_manager.cc:3219: 1 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node (ID: 3fbcaf6ee2eac7d5950c5cb6daea4714e0980be53d9d49a51e5d444c, IP: 172.17.0.2) over the last time period. To see more information about the Workers killed on this node, use `ray logs raylet.out -ip 172.17.0.2`
(raylet)
(raylet) Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero.
(embed_chunks pid=1709) Embedding batch of 73260 chunks...
(raylet)
(raylet) [2025-06-19 00:41:35,461 E 1639 1639] (raylet) node_manager.cc:3219: 1 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node (ID: 3fbcaf6ee2eac7d5950c5cb6daea4714e0980be53d9d49a51e5d444c, IP: 172.17.0.2) over the last time period. To see more information about the Workers killed on this node, use `ray logs raylet.out -ip 172.17.0.2`
(raylet) Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero.
(embed_chunks pid=1712) Embedding batch of 73260 chunks...
(raylet) A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: 505ea7fab31c4ad40be23f71134674441be943f701000000 Worker ID: 2352259547fae1dccd550c8b0ecb2f7cfb05c3e2f0d3a9a8fd13d5cf Node ID: 3fbcaf6ee2eac7d5950c5cb6daea4714e0980be53d9d49a51e5d444c Worker IP address: 172.17.0.2 Worker port: 36261 Worker PID: 1711 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
(embed_chunks pid=7350) Embedding batch of 73260 chunks...
(raylet) [2025-06-19 00:42:35,773 E 1639 1639] (raylet) node_manager.cc:3219: 1 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node (ID: 3fbcaf6ee2eac7d5950c5cb6daea4714e0980be53d9d49a51e5d444c, IP: 172.17.0.2) over the last time period. To see more information about the Workers killed on this node, use `ray logs raylet.out -ip 172.17.0.2`
(raylet)
(raylet) Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero.
(embed_chunks pid=7351) Embedding batch of 73260 chunks...
Last active
June 21, 2025 00:43
-
-
Save nov05/19aefe4ae69cbf986ce29bde83813c24 to your computer and use it in GitHub Desktop.
20250618_generative_ai_with_langchain_chapter9_ray
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
❌ Error log: Chapter 9, Ray,
serve_index.py