PeerSync Log Snippet

gistfile1.txt

2017-12-14 17:42:40.079 INFO  (updateExecutor-2-thread-88620-processing-n:host05.domain:8985_solr) [c:collection_name s:shard4_3 r:core_node86 x:collection_name_shard4_3_replica2] o.a.s.u.DefaultSolrCoreState Running recovery

2017-12-14 17:42:40.083 INFO  (recoveryExecutor-3-thread-11-processing-n:host05.domain:8985_solr x:collection_name_shard4_3_replica2 s:shard4_3 c:collection_name r:core_node86) [c:collection_name s:shard4_3 r:core_node86 x:collection_name_shard4_3_replica2] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=false

2017-12-14 17:44:20.269 INFO  (recoveryExecutor-3-thread-11-processing-n:host05.domain:8985_solr x:collection_name_shard4_3_replica2 s:shard4_3 c:collection_name r:core_node86) [c:collection_name s:shard4_3 r:core_node86 x:collection_name_shard4_3_replica2] o.a.s.u.PeerSync PeerSync: core=collection_name_shard4_3_replica2 url=https://host05.domain:8985/solr  Received 199819 versions from https://host04.domain:8983/solr/collection_name_shard4_3_replica1/ fingerprint:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=1586782229908946944, maxInHash=1586782229908946944, versionsHash=-4224661049140395596, numVersions=104904667, numDocs=104904667, maxDoc=2415668}

2017-12-14 17:44:20.434 INFO  (recoveryExecutor-3-thread-11-processing-n:host05.domain:8985_solr x:collection_name_shard4_3_replica2 s:shard4_3 c:collection_name r:core_node86) [c:collection_name s:shard4_3 r:core_node86 x:collection_name_shard4_3_replica2] o.a.s.u.PeerSync PeerSync: core=collection_name_shard4_3_replica2 url=https://host05.domain:8985/solr Requesting updates from https://host04.domain:8983/solr/collection_name_shard4_3_replica1/n=3624 versions=1586779702674063361...1586779702674063361,1586780577964490754...1586780577964490754,1586781832095989760...1586781832095989760,1586782118065733632...1586782118065733632,1586782118212534272...1586782118212534272,1586782121026912256...1586782121026912256,1586782121407545344...1586782121407545344,1586782122141548544...1586782122141548544,1586782122936369152...1586782122936369152,1586782125400522752...1586782169706004480,1586782229905801216...1586782231195549696


2017-12-14 17:44:32.042 INFO  (recoveryExecutor-3-thread-11-processing-n:host05.domain:8985_solr x:collection_name_shard4_3_replica2 s:shard4_3 c:collection_name r:core_node86) [c:collection_name s:shard4_3 r:core_node86 x:collection_name_shard4_3_replica2] o.a.s.c.SolrCore Caching fingerprint for searcher:Searcher@29dbaed7[collection_name_shard4_3_replica2] realtime{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_t(6.3.0):C1874261/152054:delGen=46996) Uninverting(_c(6.3.0):C2097146/65748:delGen=33450) Uninverting(_l(6.3.0):C2137088/80065:delGen=37478) Uninverting(_1y(6.3.0):C2043130/103736:delGen=41954) Uninverting(_19(6.3.0):C1945351/73108:delGen=35318) Uninverting(_11(6.3.0):C1962726/87276:delGen=38997) Uninverting(_1p(6.3.0):C2000339/69921:delGen=35036) Uninverting(_1h(6.3.0):C2044134/67289:delGen=33440) Uninverting(_4f(6.3.0):c2203403/82537:delGen=38458) Uninverting(_27(6.3.0):c1935603/198847:delGen=50856) Uninverting(_2f(6.3.0):c2043516/107923:delGen=42256) Uninverting(_2n(6.3.0):c1925722/162656:delGen=49291) Uninverting(_2v(6.3.0):c2010658/127420:delGen=45271) Uninverting(_3j(6.3.0):c2113496/160716:delGen=49172) Uninverting(_5d(6.3.0):c2415668/103809:delGen=42604) Uninverting(_34(6.3.0):c1903061/233977:delGen=52178) Uninverting(_3r(6.3.0):c2165161/136647:delGen=43499) Uninverting(_3b(6.3.0):c2099519/92515:delGen=40474) Uninverting(_3z(6.3.0):c2110207/159911:delGen=48320) Uninverting(_47(6.3.0):c2266005/90688:delGen=40507) Uninverting(_5n(6.3.0):c2062477/147462:delGen=48459) Uninverting(_4o(6.3.0):c2291792/69456:delGen=35841) Uninverting(_55(6.3.0):c2155877/168639:delGen=49618) Uninverting(_4x(6.3.0):c1872195/247425:delGen=53419) Uninverting(_51mx(6.3.0):c1470556/497691:delGen=34746) Uninverting(_7z(6.3.0):c2136537/121249:delGen=45295) Uninverting(_5v(6.3.0):c2230620/96160:delGen=41678) Uninverting(_63(6.3.0):c2383436/75324:delGen=37224) Uninverting(_6t(6.3.0):c2072568/219669:delGen=52271) Uninverting(_6c(6.3.0):c2408099/75595:delGen=37234) Uninverting(_6l(6.3.0):c2260060/112380:delGen=42766) Uninverting(_71(6.3.0):c2255731/80242:delGen=36796) Uninverting(_79(6.3.0):c2139034/122509:delGen=44978) Uninverting(_7i(6.3.0):c2246226/99931:delGen=39104) Uninverting(_7q(6.3.0):c2349128/71155:delGen=36152) Uninverting(_8w(6.3.0):c1738700/293780:delGen=53481) Uninverting(_8h(6.3.0):c1741387/237207:delGen=52189) Uninverting(_89(6.3.0):c1726117/303084:delGen=54860) Uninverting(_8p(6.3.0):c1749268/213725:delGen=51963) Uninverting(_a8l4(6.3.0):c1494071/290557:delGen=13157) Uninverting(_ao(6.3.0):c1532348/445342:delGen=56532) Uninverting(_9j(6.3.0):c1688424/380369:delGen=53939) Uninverting(_94(6.3.0):c1750952/337441:delGen=55492) Uninverting(_9p(6.3.0):c1680113/319430:delGen=55337) Uninverting(_9c(6.3.0):c1641169/340168:delGen=55512) Uninverting(_9w(6.3.0):c1613822/375468:delGen=56029) Uninverting(_a4(6.3.0):c1524336/440993:delGen=55881) Uninverting(_ab(6.3.0):c1537960/533566:delGen=56052) Uninverting(_ai(6.3.0):c1505952/508612:delGen=56609) Uninverting(_ax(6.3.0):c1433611/641661:delGen=56772) Uninverting(_ahcy(6.3.0):c1449714/321333:delGen=12270) Uninverting(_7lym(6.3.0):c1440763/555337:delGen=23828) Uninverting(_bf(6.3.0):c1382101/629105:delGen=57092) Uninverting(_b9(6.3.0):c1404350/591374:delGen=57122) Uninverting(_bk(6.3.0):c1357809/631182:delGen=57175) Uninverting(_8ahn(6.3.0):c1395060/504133:delGen=20835) Uninverting(_3bmn(6.3.0):c1419111/590708:delGen=41674) Uninverting(_bwnp(6.3.0):c1412174/172108:delGen=6904) Uninverting(_3oe3(6.3.0):c1421970/609895:delGen=40303) Uninverting(_4efv(6.3.0):c1438609/560252:delGen=37361) Uninverting(_5hjr(6.3.0):c1422934/476384:delGen=33241) Uninverting(_6rzy(6.3.0):c1404991/606453:delGen=27556) Uninverting(_9f2j(6.3.0):c1406369/507546:delGen=16139) Uninverting(_avgr(6.3.0):c1439944/286662:delGen=10794) Uninverting(_bk6x(6.3.0):c1401208/272038:delGen=8247) Uninverting(_ct3y(6.3.0):c1393861/128419:delGen=3346) Uninverting(_dkrh(6.3.0):c1451305/37142:delGen=624) Uninverting(_dhys(6.3.0):c27751/3021:delGen=746) Uninverting(_dips(6.3.0):c55248/6417:delGen=856) Uninverting(_dmtm(6.3.0):c106538/9908:delGen=604) Uninverting(_dr8z(6.3.0):c145469/6347:delGen=241) Uninverting(_dpz5(6.3.0):c85005/6354:delGen=361) Uninverting(_dryf(6.3.0):c28901/1516:delGen=183) Uninverting(_dq6r(6.3.0):c4691/533:delGen=183) Uninverting(_dtlr(6.3.0):c89757/976:delGen=20) Uninverting(_dspb(6.3.0):c15667/640:delGen=107) Uninverting(_dszo(6.3.0):c22597/1012:delGen=83) Uninverting(_dtgq(6.3.0):c2485/282:delGen=41) Uninverting(_dtkv(6.3.0):c2934/257:delGen=30) Uninverting(_dtuo(6.3.0):c15285/156:delGen=8) Uninverting(_dtr6(6.3.0):c2723/124:delGen=14) Uninverting(_dtxc(6.3.0):c1172/152:delGen=5) Uninverting(_dtv3(6.3.0):c876/57:delGen=6) Uninverting(_dtzv(6.3.0):c1086/166:delGen=1) Uninverting(_dtza(6.3.0):c1396/100:delGen=1) Uninverting(_du00(6.3.0):c1479/60:delGen=1) Uninverting(_dtyq(6.3.0):c796/149:delGen=2) Uninverting(_dtz3(6.3.0):c711/104:delGen=2) Uninverting(_dtzr(6.3.0):C2/1) Uninverting(_dtzn(6.3.0):C6/5) Uninverting(_dtzp(6.3.0):C4/2) Uninverting(_dtzu(6.3.0):C4/3) Uninverting(_dtzt(6.3.0):C5/4:delGen=1) Uninverting(_dtzo(6.3.0):C6/5) Uninverting(_dtzx(6.3.0):C2/1) Uninverting(_dtzy(6.3.0):C6/5) Uninverting(_dtzw(6.3.0):C8/6) Uninverting(_dtzz(6.3.0):C32/28:delGen=1) Uninverting(_du01(6.3.0):C3617/1079:delGen=1)))} leafReaderContext:LeafReaderContext(FilterLeafReader(Uninverting(_du01(6.3.0):C3617/1079:delGen=1)) docBase=122643675 ord=98) mavVersion:1586782231195549696


2017-12-14 17:44:32.042 INFO  (recoveryExecutor-3-thread-11-processing-n:host05.domain:8985_solr x:collection_name_shard4_3_replica2 s:shard4_3 c:collection_name r:core_node86) [c:collection_name s:shard4_3 r:core_node86 x:collection_name_shard4_3_replica2] o.a.s.u.PeerSync Fingerprint comparison: -1

2017-12-14 17:44:32.042 INFO  (recoveryExecutor-3-thread-11-processing-n:host05.domain:8985_solr x:collection_name_shard4_3_replica2 s:shard4_3 c:collection_name r:core_node86) [c:collection_name s:shard4_3 r:core_node86 x:collection_name_shard4_3_replica2] o.a.s.u.PeerSync Other fingerprint: {maxVersionSpecified=1586782231195549696, maxVersionEncountered=1586782232563941376, maxInHash=1586782231195549696, versionsHash=-3581674279066644281, numVersions=104904602, numDocs=104904668, maxDoc=2415668}, Our fingerprint: {maxVersionSpecified=1586782231195549696, maxVersionEncountered=1586782231195549696, maxInHash=1586782231195549696, versionsHash=8950458532433879327, numVersions=104904618, numDocs=104904618, maxDoc=2415668}


2017-12-14 17:44:32.042 INFO  (recoveryExecutor-3-thread-11-processing-n:host05.domain:8985_solr x:collection_name_shard4_3_replica2 s:shard4_3 c:collection_name r:core_node86) [c:collection_name s:shard4_3 r:core_node86 x:collection_name_shard4_3_replica2] o.a.s.u.PeerSync PeerSync: core=collection_name_shard4_3_replica2 url=https://host05.domain:8985/solr DONE. sync failed

2017-12-14 17:44:32.042 INFO  (recoveryExecutor-3-thread-11-processing-n:host05.domain:8985_solr x:collection_name_shard4_3_replica2 s:shard4_3 c:collection_name r:core_node86) [c:collection_name s:shard4_3 r:core_node86 x:collection_name_shard4_3_replica2] o.a.s.c.RecoveryStrategy PeerSync Recovery was not successful - trying replication.