anurag-arjun · May 26, 2021 19:05
diff --git a/gistfile1.txt b/gistfile1.txt
 TL;DR: There was a issue affecting a minor subset of the Polygon PoS validator and full nodes. The chain itself remained resilient and no state inconsistencies were found.

 There was an issue reported today that resulted in sync inconsistencies for a small subset of the Polygon PoS validator nodes and full nodes. 

 On height 5237861, the proposer of that Heimdall block went down for some time before gathering the prevotes. After coming back online, it again started to collect the prevotes of the previous voting round, but the other validators had already gossiped their prevotes, which lead to the "Error signing prevote" issue on the rest of the validators. However, waiting for the commit timeout fixed the issue.

 The repercussions of these were seen on the Bor nodes of the affected nodes, with block production and sync lagging behind on these nodes. 

 Heimdall validator nodes sync data at periodic intervals from the Ethereum state sync contract (e.g. token deposits, validator stake and more) and after 2/3+1 consensus, relays it to the Bor chain. In this particular case, the synced data from Ethereum was propagated earlier on the affected nodes relative to the rest of the nodes that received data late due to suspected high load on the Heimdall node RPC. The rest of the chain progressed as expected.

 A short-term hot fix (https://github.com/maticnetwork/bor/tree/arpit/hot-fix-26-05-2021) was formulated to enable these nodes to sync back with the network. A more robust fix will be released shortly to mitigate this scenario.

 Note that the chain itself remained running and deposits from Ethereum worked as expected albeit with a slight delay in some cases.
	TL;DR: There was a issue affecting a minor subset of the Polygon PoS validator and full nodes. The chain itself remained resilient and no state inconsistencies were found.

	There was an issue reported today that resulted in sync inconsistencies for a small subset of the Polygon PoS validator nodes and full nodes.

	On height 5237861, the proposer of that Heimdall block went down for some time before gathering the prevotes. After coming back online, it again started to collect the prevotes of the previous voting round, but the other validators had already gossiped their prevotes, which lead to the "Error signing prevote" issue on the rest of the validators. However, waiting for the commit timeout fixed the issue.

	The repercussions of these were seen on the Bor nodes of the affected nodes, with block production and sync lagging behind on these nodes.

	Heimdall validator nodes sync data at periodic intervals from the Ethereum state sync contract (e.g. token deposits, validator stake and more) and after 2/3+1 consensus, relays it to the Bor chain. In this particular case, the synced data from Ethereum was propagated earlier on the affected nodes relative to the rest of the nodes that received data late due to suspected high load on the Heimdall node RPC. The rest of the chain progressed as expected.

	A short-term hot fix (https://github.com/maticnetwork/bor/tree/arpit/hot-fix-26-05-2021) was formulated to enable these nodes to sync back with the network. A more robust fix will be released shortly to mitigate this scenario.

	Note that the chain itself remained running and deposits from Ethereum worked as expected albeit with a slight delay in some cases.