Statement for incident during epoch 332

What happened?

We were predicted a single block during this epoch that we missed due to blockchain fork

Why it happened?

Unfortunatelly due to the nature of Cardano blockchain it's possible that blockchain forks itself. This is commonly caused by foreign misconfigured pools that run more than 1 block producer node, resulting in forked chain on every block they mint. Or by so-called "slot battles", where different pools are predicted slots too close to each other. Unfortunatelly this is an unsolved problem in Cardano network, that all pools are currently randomly facing.

Logs from producer node confirming this issue

        Apr 15 02:58:35 cardano-pn bash[54179]: #033[34m[cardano-:cardano.node.Forge:Info:340]#033[0m [2022-04-15 00:58:35.00 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeIsLeader"),("slot",Number 5.8418024e7)])),("credentials",String "Cardano")]
        Apr 15 02:58:35 cardano-pn bash[54179]: #033[34m[cardano-:cardano.node.Forge:Info:340]#033[0m [2022-04-15 00:58:35.06 UTC] fromList [("val",Object (fromList [("block",String "1843afca1ec599ebb2d55d5510b666916fc7aae21a8509d560704815a111b779"),("blockNo",Number 7124878.0),("blockPrev",String "37bc52d732762d54be81e96c2643f2e5e51787b83801633f5eff8b28573a738d"),("kind",String "TraceForgedBlock"),("slot",Number 5.8418024e7)])),("credentials",String "Cardano")]
        Apr 15 02:58:35 cardano-pn bash[54179]: #033[35m[cardano-:cardano.node.ChainDB:Notice:330]#033[0m [2022-04-15 00:58:35.11 UTC] Chain extended, new tip: 1843afca1ec599ebb2d55d5510b666916fc7aae21a8509d560704815a111b779 at slot 58418024
        Apr 15 02:58:35 cardano-pn bash[54179]: #033[34m[cardano-:cardano.node.Forge:Info:340]#033[0m [2022-04-15 00:58:35.11 UTC] fromList [("val",Object (fromList [("blockHash",String "1843afca1ec599ebb2d55d5510b666916fc7aae21a8509d560704815a111b779"),("blockSize",Number 71433.0),("kind",String "TraceAdoptedBlock"),("slot",Number 5.8418024e7)])),("credentials",String "Cardano")]
        Apr 15 02:58:46 cardano-pn bash[54179]: #033[35m[cardano-:cardano.node.ChainDB:Notice:330]#033[0m [2022-04-15 00:58:46.16 UTC] Switched to a fork, new tip: 07b729550b0cdfab9ab0913e57cbf32ee5e6943df4f5308932b1ed920588eeef at slot 58418034