Executing term change block took too much time

[majecty]

In the Corgi network, executing term change block took too much time.

I copied the log from a Corgi node. Blocks Verifier #1's log message. "Blocks Verifier #1" thread does not print any log between two add_balance log. The gap is more than a second.

#164574181 2020-08-26 14:00:03.561690513 UTC Blocks Verifier #1 TRACE state  add_balance(0x263b…fcb1, 0): 1029374756
#164574181 2020-08-26 14:00:05.957492963 UTC Blocks Verifier #1 TRACE state  add_balance(0x6850…65e1, 2160): 1050470102

We can find this situation very frequently in Corgi. About 10 times in a day. It appears only in the term closing blocks.

The reason is that CodeChain reads all header in DB generated in a term. It is about 1000 headers.

[majecty]

Now we can regenerate the performance issue using ./codechain-debug execute-block -n 4649250 -d ./corgi_db/db.
Executing block 4649250 takes 410ms.
Executing block 4649251 takes 9ms.

#1988

[majecty]

I found the cause of the slow issue.

Between two add_balance call, CodeChain was reading headers that generated in a term. It's about 1000 headers in the Corgi network.