Created
February 8, 2017 09:57
-
-
Save peripau/e69f0adfcff51e49b1ea1a901d07a433 to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Feb 8 02:00:45 hn1 kernel: [354215.900346] EXT4-fs (ploop21443p1): loaded balloon from 12 (4096 blocks) | |
| Feb 8 02:00:46 hn1 kernel: [354216.371370] ploop21443: p1 | |
| Feb 8 02:00:46 hn1 kernel: [354216.407861] ploop21443: p1 | |
| Feb 8 02:00:46 hn1 kernel: [354216.449340] EXT4-fs (ploop21443p1): mounted filesystem with ordered data mode. Opts: | |
| Feb 8 02:00:46 hn1 kernel: [354216.457748] PFCache: relink 182:343089 to "/vz/pfcache" +0 -0 =0 peers | |
| Feb 8 02:00:46 hn1 kernel: [354216.464738] EXT4-fs (ploop21443p1): loaded balloon from 12 (4096 blocks) | |
| Feb 8 02:00:46 hn1 kernel: [354216.739975] ploop21443: p1 | |
| Feb 8 02:00:47 hn1 kernel: [354217.076244] ploop21443: p1 | |
| Feb 8 02:00:47 hn1 kernel: [354217.763875] EXT4-fs (ploop21443p1): mounted filesystem with ordered data mode. Opts: | |
| Feb 8 02:00:47 hn1 kernel: [354217.807072] PFCache: relink 182:343089 to "/vz/pfcache" +0 -0 =0 peers | |
| Feb 8 02:00:47 hn1 kernel: [354217.839112] EXT4-fs (ploop21443p1): loaded balloon from 12 (4096 blocks) | |
| Feb 8 02:00:48 hn1 kernel: [354218.212422] ploop21443: p1 | |
| Feb 8 02:00:48 hn1 kernel: [354218.258824] ploop21443: p1 | |
| Feb 8 02:00:48 hn1 kernel: [354218.347410] EXT4-fs (ploop21443p1): mounted filesystem with ordered data mode. Opts: | |
| Feb 8 02:00:48 hn1 kernel: [354218.368629] PFCache: relink 182:343089 to "/vz/pfcache" +0 -0 =0 peers | |
| Feb 8 02:00:48 hn1 kernel: [354218.375739] EXT4-fs (ploop21443p1): loaded balloon from 12 (0 blocks) | |
| Feb 8 02:00:48 hn1 kernel: [354218.661547] ploop21443: p1 | |
| Feb 8 02:00:48 hn1 kernel: [354218.708457] ploop21443: p1 | |
| Feb 8 02:00:48 hn1 kernel: [354218.822544] EXT4-fs (ploop21443p1): mounted filesystem with ordered data mode. Opts: | |
| Feb 8 02:00:48 hn1 kernel: [354218.837743] PFCache: relink 182:343089 to "/vz/pfcache" +0 -0 =0 peers | |
| Feb 8 02:00:48 hn1 kernel: [354218.844762] EXT4-fs (ploop21443p1): loaded balloon from 12 (0 blocks) | |
| Feb 8 03:45:49 hn1 kernel: [360516.386845] INFO: task jbd2/ploop15252:11085 blocked for more than 120 seconds. | |
| Feb 8 03:45:49 hn1 kernel: [360516.394244] Tainted: P -- ------------ 2.6.32-042stab120.11 #1 | |
| Feb 8 03:45:49 hn1 kernel: [360516.401811] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
| Feb 8 03:45:49 hn1 kernel: [360516.409720] jbd2/ploop152 D ffff8807d0afc040 0 11085 2 0 0x00000000 | |
| Feb 8 03:45:49 hn1 kernel: [360516.417160] ffff8807f521bc10 0000000000000046 ffff8807f521bcb0 0000000000000002 | |
| Feb 8 03:45:49 hn1 kernel: [360516.424727] ffff8807f521bb80 ffffffff8128a3b2 ffff8807f521bbc0 00000000ad986b7a | |
| Feb 8 03:45:49 hn1 kernel: [360516.432288] ffff880816a28200 ffff880028039f20 ffff8807f5252000 ffff8807f521bcb0 | |
| Feb 8 03:45:49 hn1 kernel: [360516.439860] Call Trace: | |
| Feb 8 03:45:49 hn1 kernel: [360516.442395] [<ffffffff8128a3b2>] ? __generic_unplug_device+0x32/0x40 | |
| Feb 8 03:45:49 hn1 kernel: [360516.448926] [<ffffffff811fb0c0>] ? sync_buffer+0x0/0x50 | |
| Feb 8 03:45:49 hn1 kernel: [360516.454330] [<ffffffff81554893>] io_schedule+0x73/0xc0 | |
| Feb 8 03:45:49 hn1 kernel: [360516.459638] [<ffffffff811fb100>] sync_buffer+0x40/0x50 | |
| Feb 8 03:45:49 hn1 kernel: [360516.464947] [<ffffffff815559ff>] __wait_on_bit+0x5f/0x90 | |
| Feb 8 03:45:49 hn1 kernel: [360516.470440] [<ffffffff811fb0c0>] ? sync_buffer+0x0/0x50 | |
| Feb 8 03:45:49 hn1 kernel: [360516.475834] [<ffffffff81555aa8>] out_of_line_wait_on_bit+0x78/0x90 | |
| Feb 8 03:45:49 hn1 kernel: [360516.482186] [<ffffffff810b09f0>] ? wake_bit_function+0x0/0x50 | |
| Feb 8 03:45:49 hn1 kernel: [360516.488102] [<ffffffff811fc416>] __wait_on_buffer+0x26/0x30 | |
| Feb 8 03:45:49 hn1 kernel: [360516.493872] [<ffffffffa00c3116>] jbd2_journal_commit_transaction+0xaa6/0x14f0 [jbd2] | |
| Feb 8 03:45:49 hn1 kernel: [360516.501785] [<ffffffff8109775b>] ? try_to_del_timer_sync+0x7b/0xe0 | |
| Feb 8 03:45:49 hn1 kernel: [360516.508138] [<ffffffffa00c8ba8>] kjournald2+0xb8/0x220 [jbd2] | |
| Feb 8 03:45:49 hn1 kernel: [360516.514059] [<ffffffff810b0970>] ? autoremove_wake_function+0x0/0x40 | |
| Feb 8 03:45:49 hn1 kernel: [360516.520585] [<ffffffffa00c8af0>] ? kjournald2+0x0/0x220 [jbd2] | |
| Feb 8 03:45:49 hn1 kernel: [360516.526585] [<ffffffff810b057e>] kthread+0x9e/0xc0 | |
| Feb 8 03:45:50 hn1 kernel: [360516.531549] [<ffffffff8100c3ca>] child_rip+0xa/0x20 | |
| Feb 8 03:45:50 hn1 kernel: [360516.536605] [<ffffffff810b04e0>] ? kthread+0x0/0xc0 | |
| Feb 8 03:45:50 hn1 kernel: [360516.541654] [<ffffffff8100c3c0>] ? child_rip+0x0/0x20 | |
| Feb 8 03:45:50 hn1 kernel: [360516.546890] INFO: task logrotate:2351 blocked for more than 120 seconds. | |
| Feb 8 03:45:50 hn1 kernel: [360516.553677] Tainted: P -- ------------ 2.6.32-042stab120.11 #1 | |
| Feb 8 03:45:50 hn1 kernel: [360516.561239] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
| Feb 8 03:45:50 hn1 kernel: [360516.569145] logrotate D ffff8807f7e961c0 0 2351 2349 1172 0x00000000 | |
| Feb 8 03:45:50 hn1 kernel: [360516.576589] ffff8807f559fdb8 0000000000000082 0000000000000000 ffff8807f521be80 | |
| Feb 8 03:45:50 hn1 kernel: [360516.584145] 0000000000000000 ffff8807ea3af8d0 000147bc06c2452a 0000000000000000 | |
| Feb 8 03:45:50 hn1 kernel: [360516.591693] ffff8807f559fd38 0000000000000000 0000000115790728 0000000300000001 | |
| Feb 8 03:45:50 hn1 kernel: [360516.599270] Call Trace: | |
| Feb 8 03:45:50 hn1 kernel: [360516.601807] [<ffffffffa00c86e5>] jbd2_log_wait_commit+0xc5/0x140 [jbd2] | |
| Feb 8 03:45:50 hn1 kernel: [360516.608593] [<ffffffff810b0970>] ? autoremove_wake_function+0x0/0x40 | |
| Feb 8 03:45:50 hn1 kernel: [360516.615127] [<ffffffffa00c8a98>] jbd2_complete_transaction+0x68/0xb0 [jbd2] | |
| Feb 8 03:45:50 hn1 kernel: [360516.622259] [<ffffffffa00e4ab7>] ext4_sync_file+0x157/0x210 [ext4] | |
| Feb 8 03:45:50 hn1 kernel: [360516.628612] [<ffffffff811f729d>] vfs_fsync_range+0x10d/0x260 | |
| Feb 8 03:45:50 hn1 kernel: [360516.634440] [<ffffffff811f745d>] vfs_fsync+0x1d/0x20 | |
| Feb 8 03:45:50 hn1 kernel: [360516.639578] [<ffffffff811f74e3>] do_fsync+0x83/0xe0 | |
| Feb 8 03:45:50 hn1 kernel: [360516.644636] [<ffffffff815580f5>] ? page_fault+0x25/0x30 | |
| Feb 8 03:45:50 hn1 kernel: [360516.650030] [<ffffffff811f7570>] sys_fsync+0x10/0x20 | |
| Feb 8 03:45:50 hn1 kernel: [360516.655165] [<ffffffff8100b1a2>] system_call_fastpath+0x16/0x1b | |
| Feb 8 03:45:50 hn1 kernel: [360516.661257] INFO: task flush-182:24403:2353 blocked for more than 120 seconds. | |
| Feb 8 03:45:50 hn1 kernel: [360516.668556] Tainted: P -- ------------ 2.6.32-042stab120.11 #1 | |
| Feb 8 03:45:50 hn1 kernel: [360516.676118] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
| Feb 8 03:45:50 hn1 kernel: [360516.684026] flush-182:244 D ffff8807d5ca6700 0 2353 2 0 0x00000000 | |
| Feb 8 03:45:50 hn1 kernel: [360516.691467] ffff8807d5de7530 0000000000000046 ffffffffa060f066 ffff8807d5de7688 | |
| Feb 8 03:45:50 hn1 kernel: [360516.699036] 0000ee4700008000 ffff8807dd585d70 ffff8807dd580096 0000000000000008 | |
| Feb 8 03:45:50 hn1 kernel: [360516.706605] ffff8807f8c32300 ffffffff8100bcce ffff8807d5de7548 0000000000000000 | |
| Feb 8 03:45:50 hn1 kernel: [360516.714175] Call Trace: | |
| Feb 8 03:45:50 hn1 kernel: [360516.716711] [<ffffffffa060f066>] ? find_bucket+0x66/0x70 [openvswitch] | |
| Feb 8 03:45:50 hn1 kernel: [360516.723412] [<ffffffff8100bcce>] ? apic_timer_interrupt+0xe/0x20 | |
| Feb 8 03:45:50 hn1 kernel: [360516.729589] [<ffffffff810b0c9e>] ? prepare_to_wait+0x4e/0x80 | |
| Feb 8 03:45:50 hn1 kernel: [360516.735421] [<ffffffffa00c20ad>] do_get_write_access+0x29d/0x510 [jbd2] | |
| Feb 8 03:45:50 hn1 kernel: [360516.742208] [<ffffffff810b09f0>] ? wake_bit_function+0x0/0x50 | |
| Feb 8 03:45:50 hn1 kernel: [360516.748128] [<ffffffffa00c2471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2] | |
| Feb 8 03:45:50 hn1 kernel: [360516.755620] [<ffffffffa0116208>] __ext4_journal_get_write_access+0x38/0x80 [ext4] | |
| Feb 8 03:45:50 hn1 kernel: [360516.763281] [<ffffffffa01180f2>] ext4_mb_mark_diskspace_used+0xf2/0x300 [ext4] | |
| Feb 8 03:45:50 hn1 kernel: [360516.770671] [<ffffffff81067f06>] ? enqueue_task+0x66/0x80 | |
| Feb 8 03:45:50 hn1 kernel: [360516.776247] [<ffffffffa011f5eb>] ext4_mb_new_blocks+0x3eb/0x6f0 [ext4] | |
| Feb 8 03:45:50 hn1 kernel: [360516.782949] [<ffffffffa0110a5a>] ? ext4_ext_find_extent+0x2ba/0x340 [ext4] | |
| Feb 8 03:45:50 hn1 kernel: [360516.790001] [<ffffffffa0114fc7>] ext4_ext_get_blocks+0x547/0x14d0 [ext4] | |
| Feb 8 03:45:50 hn1 kernel: [360516.796875] [<ffffffff81159de4>] ? free_hot_cold_page_list+0x54/0xa0 | |
| Feb 8 03:45:50 hn1 kernel: [360516.803403] [<ffffffff81161c14>] ? release_pages+0x234/0x2a0 | |
| Feb 8 03:45:50 hn1 kernel: [360516.809232] [<ffffffff81161c14>] ? release_pages+0x234/0x2a0 | |
| Feb 8 03:45:50 hn1 kernel: [360516.815067] [<ffffffffa00ec005>] ext4_get_blocks+0xf5/0x2b0 [ext4] | |
| Feb 8 03:45:50 hn1 kernel: [360516.821418] [<ffffffff811614c5>] ? pagevec_lookup_tag+0x25/0x40 | |
| Feb 8 03:45:50 hn1 kernel: [360516.827514] [<ffffffffa00ee96c>] mpage_da_map_and_submit+0xac/0x3b0 [ext4] | |
| Feb 8 03:45:50 hn1 kernel: [360516.834563] [<ffffffffa00c13e5>] ? jbd2_journal_start+0xb5/0x100 [jbd2] | |
| Feb 8 03:45:50 hn1 kernel: [360516.841355] [<ffffffffa00ef4e4>] ext4_da_writepages+0x314/0x660 [ext4] | |
| Feb 8 03:45:50 hn1 kernel: [360516.848054] [<ffffffff81554c45>] ? wait_for_completion_io+0x105/0x120 | |
| Feb 8 03:45:50 hn1 kernel: [360516.854663] [<ffffffff8115f951>] do_writepages+0x21/0x40 | |
| Feb 8 03:45:50 hn1 kernel: [360516.860147] [<ffffffff811f1819>] __writeback_single_inode+0xa9/0x3d0 | |
| Feb 8 03:45:50 hn1 kernel: [360516.866669] [<ffffffff811f1bd8>] writeback_single_inode_ub+0x98/0xd0 | |
| Feb 8 03:45:50 hn1 kernel: [360516.873190] [<ffffffff811f1eba>] writeback_sb_inodes+0xda/0x1b0 | |
| Feb 8 03:45:50 hn1 kernel: [360516.879282] [<ffffffff811f20a8>] writeback_inodes_wb+0x118/0x170 | |
| Feb 8 03:45:50 hn1 kernel: [360516.885465] [<ffffffff811f243b>] wb_writeback+0x33b/0x460 | |
| Feb 8 03:45:50 hn1 kernel: [360516.891036] [<ffffffff811f2715>] wb_do_writeback+0x1b5/0x260 | |
| Feb 8 03:45:50 hn1 kernel: [360516.896873] [<ffffffff811f285c>] bdi_writeback_task+0x9c/0x1f0 | |
| Feb 8 03:45:50 hn1 kernel: [360516.902885] [<ffffffff81175500>] ? bdi_start_fn+0x0/0x110 | |
| Feb 8 03:45:50 hn1 kernel: [360516.908453] [<ffffffff81175500>] ? bdi_start_fn+0x0/0x110 | |
| Feb 8 03:45:50 hn1 kernel: [360516.914021] [<ffffffff81175595>] bdi_start_fn+0x95/0x110 | |
| Feb 8 03:45:50 hn1 kernel: [360516.919505] [<ffffffff81175500>] ? bdi_start_fn+0x0/0x110 | |
| Feb 8 03:45:50 hn1 kernel: [360516.925076] [<ffffffff810b057e>] kthread+0x9e/0xc0 | |
| Feb 8 03:45:50 hn1 kernel: [360516.930037] [<ffffffff8100c3ca>] child_rip+0xa/0x20 | |
| Feb 8 03:45:50 hn1 kernel: [360516.935086] [<ffffffff810b04e0>] ? kthread+0x0/0xc0 | |
| Feb 8 03:45:50 hn1 kernel: [360516.940135] [<ffffffff8100c3c0>] ? child_rip+0x0/0x20 | |
| Feb 8 03:46:01 hn1 kernel: [360527.912202] sd 7:0:0:0: timing out command, waited 180s | |
| Feb 8 03:46:01 hn1 kernel: [360527.917521] sd 7:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK | |
| Feb 8 03:46:01 hn1 kernel: [360527.924583] sd 7:0:0:0: [sdc] CDB: Write(10): 2a 00 12 79 9a 78 00 00 60 00 | |
| Feb 8 03:46:01 hn1 kernel: [360527.931870] end_request: I/O error, dev sdc, sector 309959288 | |
| Feb 8 03:46:01 hn1 kernel: [360527.937711] device-mapper: multipath: Failing path 8:32. | |
| Feb 8 03:46:01 hn1 multipathd: 8:32: mark as failed | |
| Feb 8 03:46:01 hn1 multipathd: volume-feef7618: remaining active paths: 1 | |
| Feb 8 03:46:03 hn1 multipathd: volume-feef7618: sdc - tur checker reports path is up | |
| Feb 8 03:46:03 hn1 multipathd: 8:32: reinstated | |
| Feb 8 03:46:03 hn1 multipathd: volume-feef7618: remaining active paths: 2 | |
| Feb 8 03:46:03 hn1 kernel: [360530.197023] sd 7:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA | |
| Feb 8 03:47:50 hn1 kernel: [360636.876733] INFO: task jbd2/ploop15252:11085 blocked for more than 120 seconds. | |
| Feb 8 03:47:50 hn1 kernel: [360636.884132] Tainted: P -- ------------ 2.6.32-042stab120.11 #1 | |
| Feb 8 03:47:50 hn1 kernel: [360636.891699] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
| Feb 8 03:47:50 hn1 kernel: [360636.899606] jbd2/ploop152 D ffff8807d0afc040 0 11085 2 0 0x00000000 | |
| Feb 8 03:47:50 hn1 kernel: [360636.907048] ffff8807f521bc10 0000000000000046 ffff8807f521bcb0 0000000000000002 | |
| Feb 8 03:47:50 hn1 kernel: [360636.914624] ffff8807f521bb80 ffffffff8128a3b2 ffff8807f521bbc0 00000000ad986b7a | |
| Feb 8 03:47:50 hn1 kernel: [360636.922187] ffff880816a28200 ffff880028039f20 ffff8807f5252000 ffff8807f521bcb0 | |
| Feb 8 03:47:50 hn1 kernel: [360636.929767] Call Trace: | |
| Feb 8 03:47:50 hn1 kernel: [360636.932310] [<ffffffff8128a3b2>] ? __generic_unplug_device+0x32/0x40 | |
| Feb 8 03:47:50 hn1 kernel: [360636.938840] [<ffffffff811fb0c0>] ? sync_buffer+0x0/0x50 | |
| Feb 8 03:47:50 hn1 kernel: [360636.944244] [<ffffffff81554893>] io_schedule+0x73/0xc0 | |
| Feb 8 03:47:50 hn1 kernel: [360636.949560] [<ffffffff811fb100>] sync_buffer+0x40/0x50 | |
| Feb 8 03:47:50 hn1 kernel: [360636.954872] [<ffffffff815559ff>] __wait_on_bit+0x5f/0x90 | |
| Feb 8 03:47:50 hn1 kernel: [360636.960361] [<ffffffff811fb0c0>] ? sync_buffer+0x0/0x50 | |
| Feb 8 03:47:50 hn1 kernel: [360636.965761] [<ffffffff81555aa8>] out_of_line_wait_on_bit+0x78/0x90 | |
| Feb 8 03:47:50 hn1 kernel: [360636.972118] [<ffffffff810b09f0>] ? wake_bit_function+0x0/0x50 | |
| Feb 8 03:47:50 hn1 kernel: [360636.978043] [<ffffffff811fc416>] __wait_on_buffer+0x26/0x30 | |
| Feb 8 03:47:50 hn1 kernel: [360636.983817] [<ffffffffa00c3116>] jbd2_journal_commit_transaction+0xaa6/0x14f0 [jbd2] | |
| Feb 8 03:47:50 hn1 kernel: [360636.991737] [<ffffffff8109775b>] ? try_to_del_timer_sync+0x7b/0xe0 | |
| Feb 8 03:47:50 hn1 kernel: [360636.998099] [<ffffffffa00c8ba8>] kjournald2+0xb8/0x220 [jbd2] | |
| Feb 8 03:47:50 hn1 kernel: [360637.004020] [<ffffffff810b0970>] ? autoremove_wake_function+0x0/0x40 | |
| Feb 8 03:47:50 hn1 kernel: [360637.010552] [<ffffffffa00c8af0>] ? kjournald2+0x0/0x220 [jbd2] | |
| Feb 8 03:47:50 hn1 kernel: [360637.016559] [<ffffffff810b057e>] kthread+0x9e/0xc0 | |
| Feb 8 03:47:50 hn1 kernel: [360637.021532] [<ffffffff8100c3ca>] child_rip+0xa/0x20 | |
| Feb 8 03:47:50 hn1 kernel: [360637.026588] [<ffffffff810b04e0>] ? kthread+0x0/0xc0 | |
| Feb 8 03:47:50 hn1 kernel: [360637.031640] [<ffffffff8100c3c0>] ? child_rip+0x0/0x20 | |
| Feb 8 03:47:50 hn1 kernel: [360637.036883] INFO: task logrotate:2351 blocked for more than 120 seconds. | |
| Feb 8 03:47:50 hn1 kernel: [360637.043671] Tainted: P -- ------------ 2.6.32-042stab120.11 #1 | |
| Feb 8 03:47:50 hn1 kernel: [360637.051239] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
| Feb 8 03:47:50 hn1 kernel: [360637.059147] logrotate D ffff8807f7e961c0 0 2351 2349 1172 0x00000000 | |
| Feb 8 03:47:50 hn1 kernel: [360637.066588] ffff8807f559fdb8 0000000000000082 0000000000000000 ffff8807f521be80 | |
| Feb 8 03:47:50 hn1 kernel: [360637.074158] 0000000000000000 ffff8807ea3af8d0 000147bc06c2452a 0000000000000000 | |
| Feb 8 03:47:50 hn1 kernel: [360637.081737] ffff8807f559fd38 0000000000000000 0000000115790728 0000000300000001 | |
| Feb 8 03:47:50 hn1 kernel: [360637.089304] Call Trace: | |
| Feb 8 03:47:50 hn1 kernel: [360637.091845] [<ffffffffa00c86e5>] jbd2_log_wait_commit+0xc5/0x140 [jbd2] | |
| Feb 8 03:47:50 hn1 kernel: [360637.098631] [<ffffffff810b0970>] ? autoremove_wake_function+0x0/0x40 | |
| Feb 8 03:47:50 hn1 kernel: [360637.105164] [<ffffffffa00c8a98>] jbd2_complete_transaction+0x68/0xb0 [jbd2] | |
| Feb 8 03:47:50 hn1 kernel: [360637.112305] [<ffffffffa00e4ab7>] ext4_sync_file+0x157/0x210 [ext4] | |
| Feb 8 03:47:50 hn1 kernel: [360637.118666] [<ffffffff811f729d>] vfs_fsync_range+0x10d/0x260 | |
| Feb 8 03:47:50 hn1 kernel: [360637.124499] [<ffffffff811f745d>] vfs_fsync+0x1d/0x20 | |
| Feb 8 03:47:50 hn1 kernel: [360637.129637] [<ffffffff811f74e3>] do_fsync+0x83/0xe0 | |
| Feb 8 03:47:50 hn1 kernel: [360637.134686] [<ffffffff815580f5>] ? page_fault+0x25/0x30 | |
| Feb 8 03:47:50 hn1 kernel: [360637.140083] [<ffffffff811f7570>] sys_fsync+0x10/0x20 | |
| Feb 8 03:47:50 hn1 kernel: [360637.145221] [<ffffffff8100b1a2>] system_call_fastpath+0x16/0x1b | |
| Feb 8 03:47:50 hn1 kernel: [360637.151308] INFO: task flush-182:24403:2353 blocked for more than 120 seconds. | |
| Feb 8 03:47:50 hn1 kernel: [360637.158609] Tainted: P -- ------------ 2.6.32-042stab120.11 #1 | |
| Feb 8 03:47:50 hn1 kernel: [360637.166172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
| Feb 8 03:47:50 hn1 kernel: [360637.174081] flush-182:244 D ffff8807d5ca6700 0 2353 2 0 0x00000000 | |
| Feb 8 03:47:50 hn1 kernel: [360637.181519] ffff8807d5de7530 0000000000000046 ffffffffa060f066 ffff8807d5de7688 | |
| Feb 8 03:47:50 hn1 kernel: [360637.189097] 0000ee4700008000 ffff8807dd585d70 ffff8807dd580096 0000000000000008 | |
| Feb 8 03:47:50 hn1 kernel: [360637.196679] ffff8807f8c32300 ffffffff8100bcce ffff8807d5de7548 0000000000000000 | |
| Feb 8 03:47:50 hn1 kernel: [360637.204247] Call Trace: | |
| Feb 8 03:47:50 hn1 kernel: [360637.206793] [<ffffffffa060f066>] ? find_bucket+0x66/0x70 [openvswitch] | |
| Feb 8 03:47:50 hn1 kernel: [360637.213491] [<ffffffff8100bcce>] ? apic_timer_interrupt+0xe/0x20 | |
| Feb 8 03:47:50 hn1 kernel: [360637.219676] [<ffffffff810b0c9e>] ? prepare_to_wait+0x4e/0x80 | |
| Feb 8 03:47:50 hn1 kernel: [360637.225510] [<ffffffffa00c20ad>] do_get_write_access+0x29d/0x510 [jbd2] | |
| Feb 8 03:47:50 hn1 kernel: [360637.232297] [<ffffffff810b09f0>] ? wake_bit_function+0x0/0x50 | |
| Feb 8 03:47:50 hn1 kernel: [360637.238227] [<ffffffffa00c2471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2] | |
| Feb 8 03:47:50 hn1 kernel: [360637.245714] [<ffffffffa0116208>] __ext4_journal_get_write_access+0x38/0x80 [ext4] | |
| Feb 8 03:47:50 hn1 kernel: [360637.253371] [<ffffffffa01180f2>] ext4_mb_mark_diskspace_used+0xf2/0x300 [ext4] | |
| Feb 8 03:47:50 hn1 kernel: [360637.260769] [<ffffffff81067f06>] ? enqueue_task+0x66/0x80 | |
| Feb 8 03:47:50 hn1 kernel: [360637.266353] [<ffffffffa011f5eb>] ext4_mb_new_blocks+0x3eb/0x6f0 [ext4] | |
| Feb 8 03:47:50 hn1 kernel: [360637.273057] [<ffffffffa0110a5a>] ? ext4_ext_find_extent+0x2ba/0x340 [ext4] | |
| Feb 8 03:47:50 hn1 kernel: [360637.280108] [<ffffffffa0114fc7>] ext4_ext_get_blocks+0x547/0x14d0 [ext4] | |
| Feb 8 03:47:50 hn1 kernel: [360637.286979] [<ffffffff81159de4>] ? free_hot_cold_page_list+0x54/0xa0 | |
| Feb 8 03:47:50 hn1 kernel: [360637.293512] [<ffffffff81161c14>] ? release_pages+0x234/0x2a0 | |
| Feb 8 03:47:50 hn1 kernel: [360637.299346] [<ffffffff81161c14>] ? release_pages+0x234/0x2a0 | |
| Feb 8 03:47:50 hn1 kernel: [360637.305182] [<ffffffffa00ec005>] ext4_get_blocks+0xf5/0x2b0 [ext4] | |
| Feb 8 03:47:50 hn1 kernel: [360637.311534] [<ffffffff811614c5>] ? pagevec_lookup_tag+0x25/0x40 | |
| Feb 8 03:47:50 hn1 kernel: [360637.317628] [<ffffffffa00ee96c>] mpage_da_map_and_submit+0xac/0x3b0 [ext4] | |
| Feb 8 03:47:50 hn1 kernel: [360637.324679] [<ffffffffa00c13e5>] ? jbd2_journal_start+0xb5/0x100 [jbd2] | |
| Feb 8 03:47:50 hn1 kernel: [360637.331471] [<ffffffffa00ef4e4>] ext4_da_writepages+0x314/0x660 [ext4] | |
| Feb 8 03:47:50 hn1 kernel: [360637.338169] [<ffffffff81554c45>] ? wait_for_completion_io+0x105/0x120 | |
| Feb 8 03:47:50 hn1 kernel: [360637.344787] [<ffffffff8115f951>] do_writepages+0x21/0x40 | |
| Feb 8 03:47:50 hn1 kernel: [360637.350280] [<ffffffff811f1819>] __writeback_single_inode+0xa9/0x3d0 | |
| Feb 8 03:47:50 hn1 kernel: [360637.356809] [<ffffffff811f1bd8>] writeback_single_inode_ub+0x98/0xd0 | |
| Feb 8 03:47:50 hn1 kernel: [360637.363339] [<ffffffff811f1eba>] writeback_sb_inodes+0xda/0x1b0 | |
| Feb 8 03:47:50 hn1 kernel: [360637.369428] [<ffffffff811f20a8>] writeback_inodes_wb+0x118/0x170 | |
| Feb 8 03:47:50 hn1 kernel: [360637.375612] [<ffffffff811f243b>] wb_writeback+0x33b/0x460 | |
| Feb 8 03:47:50 hn1 kernel: [360637.381183] [<ffffffff811f2715>] wb_do_writeback+0x1b5/0x260 | |
| Feb 8 03:47:50 hn1 kernel: [360637.387019] [<ffffffff811f285c>] bdi_writeback_task+0x9c/0x1f0 | |
| Feb 8 03:47:50 hn1 kernel: [360637.393023] [<ffffffff81175500>] ? bdi_start_fn+0x0/0x110 | |
| Feb 8 03:47:50 hn1 kernel: [360637.398602] [<ffffffff81175500>] ? bdi_start_fn+0x0/0x110 | |
| Feb 8 03:47:50 hn1 kernel: [360637.404171] [<ffffffff81175595>] bdi_start_fn+0x95/0x110 | |
| Feb 8 03:47:50 hn1 kernel: [360637.409662] [<ffffffff81175500>] ? bdi_start_fn+0x0/0x110 | |
| Feb 8 03:47:50 hn1 kernel: [360637.415231] [<ffffffff810b057e>] kthread+0x9e/0xc0 | |
| Feb 8 03:47:50 hn1 kernel: [360637.420196] [<ffffffff8100c3ca>] child_rip+0xa/0x20 | |
| Feb 8 03:47:50 hn1 kernel: [360637.425254] [<ffffffff810b04e0>] ? kthread+0x0/0xc0 | |
| Feb 8 03:47:50 hn1 kernel: [360637.430312] [<ffffffff8100c3c0>] ? child_rip+0x0/0x20 | |
| Feb 8 03:49:01 hn1 kernel: [360707.938606] sd 6:0:0:0: timing out command, waited 180s | |
| Feb 8 03:49:01 hn1 kernel: [360707.943919] sd 6:0:0:0: [sdd] Result: hostbyte=DID_OK driverbyte=DRIVER_OK | |
| Feb 8 03:49:01 hn1 kernel: [360707.950978] sd 6:0:0:0: [sdd] CDB: Write(10): 2a 00 12 79 9a 78 00 00 60 00 | |
| Feb 8 03:49:01 hn1 kernel: [360707.958433] end_request: I/O error, dev sdd, sector 309959288 | |
| Feb 8 03:49:01 hn1 kernel: [360707.964367] device-mapper: multipath: Failing path 8:48. | |
| Feb 8 03:49:01 hn1 multipathd: 8:48: mark as failed | |
| Feb 8 03:49:01 hn1 multipathd: volume-feef7618: remaining active paths: 1 | |
| Feb 8 03:49:03 hn1 multipathd: volume-feef7618: sdd - tur checker reports path is up | |
| Feb 8 03:49:03 hn1 multipathd: 8:48: reinstated | |
| Feb 8 03:49:03 hn1 multipathd: volume-feef7618: remaining active paths: 2 | |
| Feb 8 03:49:03 hn1 kernel: [360710.113624] sd 6:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA | |
| Feb 8 03:49:50 hn1 kernel: [360757.366316] INFO: task jbd2/ploop15252:11085 blocked for more than 120 seconds. | |
| Feb 8 03:49:50 hn1 kernel: [360757.373715] Tainted: P -- ------------ 2.6.32-042stab120.11 #1 | |
| Feb 8 03:49:50 hn1 kernel: [360757.381274] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
| Feb 8 03:49:51 hn1 kernel: [360757.389181] jbd2/ploop152 D ffff8807d0afc040 0 11085 2 0 0x00000000 | |
| Feb 8 03:49:51 hn1 kernel: [360757.396615] ffff8807f521bc10 0000000000000046 ffff8807f521bcb0 0000000000000002 | |
| Feb 8 03:49:51 hn1 kernel: [360757.404184] ffff8807f521bb80 ffffffff8128a3b2 ffff8807f521bbc0 00000000ad986b7a | |
| Feb 8 03:49:51 hn1 kernel: [360757.411744] ffff880816a28200 ffff880028039f20 ffff8807f5252000 ffff8807f521bcb0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.419306] Call Trace: | |
| Feb 8 03:49:51 hn1 kernel: [360757.421840] [<ffffffff8128a3b2>] ? __generic_unplug_device+0x32/0x40 | |
| Feb 8 03:49:51 hn1 kernel: [360757.428370] [<ffffffff811fb0c0>] ? sync_buffer+0x0/0x50 | |
| Feb 8 03:49:51 hn1 kernel: [360757.433769] [<ffffffff81554893>] io_schedule+0x73/0xc0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.439083] [<ffffffff811fb100>] sync_buffer+0x40/0x50 | |
| Feb 8 03:49:51 hn1 kernel: [360757.444394] [<ffffffff815559ff>] __wait_on_bit+0x5f/0x90 | |
| Feb 8 03:49:51 hn1 kernel: [360757.449875] [<ffffffff811fb0c0>] ? sync_buffer+0x0/0x50 | |
| Feb 8 03:49:51 hn1 kernel: [360757.455271] [<ffffffff81555aa8>] out_of_line_wait_on_bit+0x78/0x90 | |
| Feb 8 03:49:51 hn1 kernel: [360757.461621] [<ffffffff810b09f0>] ? wake_bit_function+0x0/0x50 | |
| Feb 8 03:49:51 hn1 kernel: [360757.467537] [<ffffffff811fc416>] __wait_on_buffer+0x26/0x30 | |
| Feb 8 03:49:51 hn1 kernel: [360757.473304] [<ffffffffa00c3116>] jbd2_journal_commit_transaction+0xaa6/0x14f0 [jbd2] | |
| Feb 8 03:49:51 hn1 kernel: [360757.481218] [<ffffffff8109775b>] ? try_to_del_timer_sync+0x7b/0xe0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.487577] [<ffffffffa00c8ba8>] kjournald2+0xb8/0x220 [jbd2] | |
| Feb 8 03:49:51 hn1 kernel: [360757.493495] [<ffffffff810b0970>] ? autoremove_wake_function+0x0/0x40 | |
| Feb 8 03:49:51 hn1 kernel: [360757.500023] [<ffffffffa00c8af0>] ? kjournald2+0x0/0x220 [jbd2] | |
| Feb 8 03:49:51 hn1 kernel: [360757.506029] [<ffffffff810b057e>] kthread+0x9e/0xc0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.510994] [<ffffffff8100c3ca>] child_rip+0xa/0x20 | |
| Feb 8 03:49:51 hn1 kernel: [360757.516041] [<ffffffff810b04e0>] ? kthread+0x0/0xc0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.521094] [<ffffffff8100c3c0>] ? child_rip+0x0/0x20 | |
| Feb 8 03:49:51 hn1 kernel: [360757.526323] INFO: task logrotate:2351 blocked for more than 120 seconds. | |
| Feb 8 03:49:51 hn1 kernel: [360757.533106] Tainted: P -- ------------ 2.6.32-042stab120.11 #1 | |
| Feb 8 03:49:51 hn1 kernel: [360757.540668] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
| Feb 8 03:49:51 hn1 kernel: [360757.548588] logrotate D ffff8807f7e961c0 0 2351 2349 1172 0x00000000 | |
| Feb 8 03:49:51 hn1 kernel: [360757.556045] ffff8807f559fdb8 0000000000000082 0000000000000000 ffff8807f521be80 | |
| Feb 8 03:49:51 hn1 kernel: [360757.563622] 0000000000000000 ffff8807ea3af8d0 000147bc06c2452a 0000000000000000 | |
| Feb 8 03:49:51 hn1 kernel: [360757.571210] ffff8807f559fd38 0000000000000000 0000000115790728 0000000300000001 | |
| Feb 8 03:49:51 hn1 kernel: [360757.578788] Call Trace: | |
| Feb 8 03:49:51 hn1 kernel: [360757.581333] [<ffffffffa00c86e5>] jbd2_log_wait_commit+0xc5/0x140 [jbd2] | |
| Feb 8 03:49:51 hn1 kernel: [360757.588121] [<ffffffff810b0970>] ? autoremove_wake_function+0x0/0x40 | |
| Feb 8 03:49:51 hn1 kernel: [360757.594645] [<ffffffffa00c8a98>] jbd2_complete_transaction+0x68/0xb0 [jbd2] | |
| Feb 8 03:49:51 hn1 kernel: [360757.601786] [<ffffffffa00e4ab7>] ext4_sync_file+0x157/0x210 [ext4] | |
| Feb 8 03:49:51 hn1 kernel: [360757.608147] [<ffffffff811f729d>] vfs_fsync_range+0x10d/0x260 | |
| Feb 8 03:49:51 hn1 kernel: [360757.613983] [<ffffffff811f745d>] vfs_fsync+0x1d/0x20 | |
| Feb 8 03:49:51 hn1 kernel: [360757.619129] [<ffffffff811f74e3>] do_fsync+0x83/0xe0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.624179] [<ffffffff815580f5>] ? page_fault+0x25/0x30 | |
| Feb 8 03:49:51 hn1 kernel: [360757.629586] [<ffffffff811f7570>] sys_fsync+0x10/0x20 | |
| Feb 8 03:49:51 hn1 kernel: [360757.634730] [<ffffffff8100b1a2>] system_call_fastpath+0x16/0x1b | |
| Feb 8 03:49:51 hn1 kernel: [360757.640826] INFO: task flush-182:24403:2353 blocked for more than 120 seconds. | |
| Feb 8 03:49:51 hn1 kernel: [360757.648137] Tainted: P -- ------------ 2.6.32-042stab120.11 #1 | |
| Feb 8 03:49:51 hn1 kernel: [360757.655707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
| Feb 8 03:49:51 hn1 kernel: [360757.663613] flush-182:244 D ffff8807d5ca6700 0 2353 2 0 0x00000000 | |
| Feb 8 03:49:51 hn1 kernel: [360757.671056] ffff8807d5de7530 0000000000000046 ffffffffa060f066 ffff8807d5de7688 | |
| Feb 8 03:49:51 hn1 kernel: [360757.678640] 0000ee4700008000 ffff8807dd585d70 ffff8807dd580096 0000000000000008 | |
| Feb 8 03:49:51 hn1 kernel: [360757.686218] ffff8807f8c32300 ffffffff8100bcce ffff8807d5de7548 0000000000000000 | |
| Feb 8 03:49:51 hn1 kernel: [360757.693798] Call Trace: | |
| Feb 8 03:49:51 hn1 kernel: [360757.696341] [<ffffffffa060f066>] ? find_bucket+0x66/0x70 [openvswitch] | |
| Feb 8 03:49:51 hn1 kernel: [360757.703044] [<ffffffff8100bcce>] ? apic_timer_interrupt+0xe/0x20 | |
| Feb 8 03:49:51 hn1 kernel: [360757.709229] [<ffffffff810b0c9e>] ? prepare_to_wait+0x4e/0x80 | |
| Feb 8 03:49:51 hn1 kernel: [360757.715068] [<ffffffffa00c20ad>] do_get_write_access+0x29d/0x510 [jbd2] | |
| Feb 8 03:49:51 hn1 kernel: [360757.721856] [<ffffffff810b09f0>] ? wake_bit_function+0x0/0x50 | |
| Feb 8 03:49:51 hn1 kernel: [360757.727786] [<ffffffffa00c2471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2] | |
| Feb 8 03:49:51 hn1 kernel: [360757.735275] [<ffffffffa0116208>] __ext4_journal_get_write_access+0x38/0x80 [ext4] | |
| Feb 8 03:49:51 hn1 kernel: [360757.742938] [<ffffffffa01180f2>] ext4_mb_mark_diskspace_used+0xf2/0x300 [ext4] | |
| Feb 8 03:49:51 hn1 kernel: [360757.750338] [<ffffffff81067f06>] ? enqueue_task+0x66/0x80 | |
| Feb 8 03:49:51 hn1 kernel: [360757.755922] [<ffffffffa011f5eb>] ext4_mb_new_blocks+0x3eb/0x6f0 [ext4] | |
| Feb 8 03:49:51 hn1 kernel: [360757.762634] [<ffffffffa0110a5a>] ? ext4_ext_find_extent+0x2ba/0x340 [ext4] | |
| Feb 8 03:49:51 hn1 kernel: [360757.769685] [<ffffffffa0114fc7>] ext4_ext_get_blocks+0x547/0x14d0 [ext4] | |
| Feb 8 03:49:51 hn1 kernel: [360757.776567] [<ffffffff81159de4>] ? free_hot_cold_page_list+0x54/0xa0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.783095] [<ffffffff81161c14>] ? release_pages+0x234/0x2a0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.788923] [<ffffffff81161c14>] ? release_pages+0x234/0x2a0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.794768] [<ffffffffa00ec005>] ext4_get_blocks+0xf5/0x2b0 [ext4] | |
| Feb 8 03:49:51 hn1 kernel: [360757.801119] [<ffffffff811614c5>] ? pagevec_lookup_tag+0x25/0x40 | |
| Feb 8 03:49:51 hn1 kernel: [360757.807222] [<ffffffffa00ee96c>] mpage_da_map_and_submit+0xac/0x3b0 [ext4] | |
| Feb 8 03:49:51 hn1 kernel: [360757.814272] [<ffffffffa00c13e5>] ? jbd2_journal_start+0xb5/0x100 [jbd2] | |
| Feb 8 03:49:51 hn1 kernel: [360757.821063] [<ffffffffa00ef4e4>] ext4_da_writepages+0x314/0x660 [ext4] | |
| Feb 8 03:49:51 hn1 kernel: [360757.827765] [<ffffffff81554c45>] ? wait_for_completion_io+0x105/0x120 | |
| Feb 8 03:49:51 hn1 kernel: [360757.834381] [<ffffffff8115f951>] do_writepages+0x21/0x40 | |
| Feb 8 03:49:51 hn1 kernel: [360757.839875] [<ffffffff811f1819>] __writeback_single_inode+0xa9/0x3d0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.846406] [<ffffffff811f1bd8>] writeback_single_inode_ub+0x98/0xd0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.852936] [<ffffffff811f1eba>] writeback_sb_inodes+0xda/0x1b0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.859024] [<ffffffff811f20a8>] writeback_inodes_wb+0x118/0x170 | |
| Feb 8 03:49:51 hn1 kernel: [360757.865207] [<ffffffff811f243b>] wb_writeback+0x33b/0x460 | |
| Feb 8 03:49:51 hn1 kernel: [360757.870787] [<ffffffff811f2715>] wb_do_writeback+0x1b5/0x260 | |
| Feb 8 03:49:51 hn1 kernel: [360757.876625] [<ffffffff811f285c>] bdi_writeback_task+0x9c/0x1f0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.882637] [<ffffffff81175500>] ? bdi_start_fn+0x0/0x110 | |
| Feb 8 03:49:51 hn1 kernel: [360757.888214] [<ffffffff81175500>] ? bdi_start_fn+0x0/0x110 | |
| Feb 8 03:49:51 hn1 kernel: [360757.893791] [<ffffffff81175595>] bdi_start_fn+0x95/0x110 | |
| Feb 8 03:49:51 hn1 kernel: [360757.899281] [<ffffffff81175500>] ? bdi_start_fn+0x0/0x110 | |
| Feb 8 03:49:51 hn1 kernel: [360757.904852] [<ffffffff810b057e>] kthread+0x9e/0xc0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.909825] [<ffffffff8100c3ca>] child_rip+0xa/0x20 | |
| Feb 8 03:49:51 hn1 kernel: [360757.914882] [<ffffffff810b04e0>] ? kthread+0x0/0xc0 | |
| Feb 8 03:49:51 hn1 kernel: [360757.919947] [<ffffffff8100c3c0>] ? child_rip+0x0/0x20 | |
| Feb 8 03:51:51 hn1 kernel: [360877.856188] INFO: task jbd2/dm-2-8:2772 blocked for more than 120 seconds. | |
| Feb 8 03:51:51 hn1 kernel: [360877.863154] Tainted: P -- ------------ 2.6.32-042stab120.11 #1 | |
| Feb 8 03:51:51 hn1 kernel: [360877.870712] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
| Feb 8 03:51:51 hn1 kernel: [360877.878620] jbd2/dm-2-8 D ffff88080b8fc180 0 2772 2 0 0x00000000 | |
| Feb 8 03:51:51 hn1 kernel: [360877.886049] ffff88080b2b7c10 0000000000000046 ffff88080b2b7b70 ffffffff81286274 | |
| Feb 8 03:51:51 hn1 kernel: [360877.893595] ffff880815bf9420 ffff880816a28300 ffff88080b2b7be0 ffffffffa0004f5f | |
| Feb 8 03:51:51 hn1 kernel: [360877.901163] ffff880028047840 ffffffff811fb0c0 ffff88080b2b7cb0 0000000000000002 | |
| Feb 8 03:51:51 hn1 kernel: [360877.908728] Call Trace: | |
| Feb 8 03:51:51 hn1 kernel: [360877.911269] [<ffffffff81286274>] ? blk_unplug+0x34/0x70 | |
| Feb 8 03:51:51 hn1 kernel: [360877.916694] [<ffffffffa0004f5f>] ? dm_table_unplug_all+0x5f/0x100 [dm_mod] | |
| Feb 8 03:51:51 hn1 kernel: [360877.923740] [<ffffffff811fb0c0>] ? sync_buffer+0x0/0x50 | |
| Feb 8 03:51:51 hn1 kernel: [360877.929146] [<ffffffff811fb0c0>] ? sync_buffer+0x0/0x50 | |
| Feb 8 03:51:51 hn1 kernel: [360877.934539] [<ffffffff811fb0c0>] ? sync_buffer+0x0/0x50 | |
| Feb 8 03:51:51 hn1 kernel: [360877.939938] [<ffffffff81554893>] io_schedule+0x73/0xc0 | |
| Feb 8 03:51:51 hn1 kernel: [360877.945245] [<ffffffff811fb100>] sync_buffer+0x40/0x50 | |
| Feb 8 03:51:51 hn1 kernel: [360877.950556] [<ffffffff815559ff>] __wait_on_bit+0x5f/0x90 | |
| Feb 8 03:51:51 hn1 kernel: [360877.956039] [<ffffffff811fb0c0>] ? sync_buffer+0x0/0x50 | |
| Feb 8 03:51:51 hn1 kernel: [360877.961434] [<ffffffff81555aa8>] out_of_line_wait_on_bit+0x78/0x90 | |
| Feb 8 03:51:51 hn1 kernel: [360877.967785] [<ffffffff810b09f0>] ? wake_bit_function+0x0/0x50 | |
| Feb 8 03:51:51 hn1 kernel: [360877.973701] [<ffffffff811fc416>] __wait_on_buffer+0x26/0x30 | |
| Feb 8 03:51:51 hn1 kernel: [360877.979451] [<ffffffffa00c3116>] jbd2_journal_commit_transaction+0xaa6/0x14f0 [jbd2] | |
| Feb 8 03:51:51 hn1 kernel: [360877.987361] [<ffffffff8109775b>] ? try_to_del_timer_sync+0x7b/0xe0 | |
| Feb 8 03:51:51 hn1 kernel: [360877.993714] [<ffffffffa00c8ba8>] kjournald2+0xb8/0x220 [jbd2] | |
| Feb 8 03:51:51 hn1 kernel: [360877.999634] [<ffffffff810b0970>] ? autoremove_wake_function+0x0/0x40 | |
| Feb 8 03:51:51 hn1 kernel: [360878.006161] [<ffffffffa00c8af0>] ? kjournald2+0x0/0x220 [jbd2] | |
| Feb 8 03:51:51 hn1 kernel: [360878.012167] [<ffffffff810b057e>] kthread+0x9e/0xc0 | |
| Feb 8 03:51:51 hn1 kernel: [360878.017132] [<ffffffff8100c3ca>] child_rip+0xa/0x20 | |
| Feb 8 03:51:51 hn1 kernel: [360878.022190] [<ffffffff810b04e0>] ? kthread+0x0/0xc0 | |
| Feb 8 03:51:51 hn1 kernel: [360878.027238] [<ffffffff8100c3c0>] ? child_rip+0x0/0x20 | |
| Feb 8 03:52:01 hn1 kernel: [360887.871431] sd 7:0:0:0: timing out command, waited 180s | |
| Feb 8 03:52:01 hn1 kernel: [360887.876743] sd 7:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK | |
| Feb 8 03:52:01 hn1 kernel: [360887.883801] sd 7:0:0:0: [sdc] CDB: Write(10): 2a 00 12 79 9a 78 00 00 60 00 | |
| Feb 8 03:52:01 hn1 kernel: [360887.891076] end_request: I/O error, dev sdc, sector 309959288 | |
| Feb 8 03:52:01 hn1 kernel: [360887.896908] device-mapper: multipath: Failing path 8:32. | |
| Feb 8 03:52:01 hn1 multipathd: 8:32: mark as failed | |
| Feb 8 03:52:01 hn1 multipathd: volume-feef7618: remaining active paths: 1 | |
| Feb 8 03:52:03 hn1 multipathd: volume-feef7618: sdc - tur checker reports path is up | |
| Feb 8 03:52:03 hn1 multipathd: 8:32: reinstated | |
| Feb 8 03:52:03 hn1 multipathd: volume-feef7618: remaining active paths: 2 | |
| Feb 8 03:52:03 hn1 kernel: [360890.024009] sd 7:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA | |
| Feb 8 03:55:01 hn1 multipathd: 8:48: mark as failed | |
| Feb 8 03:55:01 hn1 multipathd: volume-feef7618: remaining active paths: 1 | |
| Feb 8 03:55:01 hn1 kernel: [361067.800504] sd 6:0:0:0: timing out command, waited 180s | |
| Feb 8 03:55:01 hn1 kernel: [361067.805819] sd 6:0:0:0: [sdd] Result: hostbyte=DID_OK driverbyte=DRIVER_OK | |
| Feb 8 03:55:01 hn1 kernel: [361067.812875] sd 6:0:0:0: [sdd] CDB: Write(10): 2a 00 12 79 9a 78 00 00 60 00 | |
| Feb 8 03:55:01 hn1 kernel: [361067.820133] end_request: I/O error, dev sdd, sector 309959288 | |
| Feb 8 03:55:01 hn1 kernel: [361067.825970] device-mapper: multipath: Failing path 8:48. | |
| Feb 8 03:55:02 hn1 multipathd: volume-feef7618: sdd - tur checker reports path is up | |
| Feb 8 03:55:02 hn1 multipathd: 8:48: reinstated | |
| Feb 8 03:55:02 hn1 multipathd: volume-feef7618: remaining active paths: 2 | |
| Feb 8 03:55:02 hn1 kernel: [361068.938413] sd 6:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA | |
| Feb 8 03:55:33 hn1 kernel: [361099.454032] sd 6:0:0:0: timing out command, waited 180s | |
| Feb 8 03:55:33 hn1 kernel: [361099.459349] sd 6:0:0:0: [sdd] Result: hostbyte=DID_OK driverbyte=DRIVER_OK | |
| Feb 8 03:55:33 hn1 kernel: [361099.466410] sd 6:0:0:0: [sdd] CDB: Write(10): 2a 00 20 04 e0 c0 00 00 30 00 | |
| Feb 8 03:55:33 hn1 kernel: [361099.473698] end_request: I/O error, dev sdd, sector 537190592 | |
| Feb 8 03:55:33 hn1 kernel: [361099.479524] device-mapper: multipath: Failing path 8:48. | |
| Feb 8 03:55:33 hn1 multipathd: 8:48: mark as failed | |
| Feb 8 03:55:33 hn1 multipathd: volume-feef7618: remaining active paths: 1 | |
| Feb 8 03:55:35 hn1 multipathd: volume-feef7618: sdd - tur checker reports path is up | |
| Feb 8 03:55:35 hn1 multipathd: 8:48: reinstated | |
| Feb 8 03:55:35 hn1 multipathd: volume-feef7618: remaining active paths: 2 | |
| Feb 8 03:55:35 hn1 kernel: [361101.922131] sd 6:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA | |
| Feb 8 03:58:01 hn1 multipathd: 8:32: mark as failed | |
| Feb 8 03:58:01 hn1 multipathd: volume-feef7618: remaining active paths: 1 | |
| Feb 8 03:58:01 hn1 kernel: [361247.764979] sd 7:0:0:0: timing out command, waited 180s | |
| Feb 8 03:58:01 hn1 kernel: [361247.770295] sd 7:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK | |
| Feb 8 03:58:01 hn1 kernel: [361247.777351] sd 7:0:0:0: [sdc] CDB: Write(10): 2a 00 12 79 9a 78 00 00 60 00 | |
| Feb 8 03:58:01 hn1 kernel: [361247.784644] end_request: I/O error, dev sdc, sector 309959288 | |
| Feb 8 03:58:01 hn1 kernel: [361247.790474] device-mapper: multipath: Failing path 8:32. | |
| Feb 8 03:58:01 hn1 multipathd: volume-feef7618: sdc - tur checker reports path is up | |
| Feb 8 03:58:01 hn1 multipathd: 8:32: reinstated | |
| Feb 8 03:58:01 hn1 multipathd: volume-feef7618: remaining active paths: 2 | |
| Feb 8 03:58:01 hn1 kernel: [361247.853027] sd 7:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA | |
| Feb 8 03:58:33 hn1 multipathd: 8:32: mark as failed | |
| Feb 8 03:58:33 hn1 multipathd: volume-feef7618: remaining active paths: 1 | |
| Feb 8 03:58:33 hn1 kernel: [361279.387307] sd 7:0:0:0: timing out command, waited 180s | |
| Feb 8 03:58:33 hn1 kernel: [361279.392696] sd 7:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK | |
| Feb 8 03:58:33 hn1 kernel: [361279.399769] sd 7:0:0:0: [sdc] CDB: Write(10): 2a 00 20 04 e0 c0 00 00 30 00 | |
| Feb 8 03:58:33 hn1 kernel: [361279.407044] end_request: I/O error, dev sdc, sector 537190592 | |
| Feb 8 03:58:33 hn1 kernel: [361279.412880] device-mapper: multipath: Failing path 8:32. | |
| Feb 8 03:58:34 hn1 multipathd: volume-feef7618: sdc - tur checker reports path is up | |
| Feb 8 03:58:34 hn1 multipathd: 8:32: reinstated | |
| Feb 8 03:58:34 hn1 multipathd: volume-feef7618: remaining active paths: 2 | |
| Feb 8 03:58:34 hn1 kernel: [361280.836637] sd 7:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment