Hello everyone,
I would like to propose a new LIP for the roadmap objective “Update block header format.” This LIP covers all changes to the block header schema and block asset schema introduced by several other LIPs.
I’m looking forward to your feedback.
Here is the complete LIP draft:
LIP: <LIP number>
Title: Update block schema and block processing
Author: Andreas Kendziorra <andreas.kendziorra@lightcurve.io>
Type: Standards Track
Created: <YYYY-MM-DD>
Updated: <YYYY-MM-DD>
Requires: 0040,
Introduce a certificate generation mechanism
Abstract
This LIP changes the structure of a block, introducing the assets property alongside the block header and payload.
The block header schema is updated to add new properties introduced by several other LIPs.
We clarify the mechanism by which modules can include data in the block assets and specify the validation of each block header property.
Copyright
This LIP is licensed under the Creative Commons Zero 1.0 Universal.
Motivation
The first change proposed by this LIP is to introduce a new block property, the block assets, and to update the block schema accordingly.
This property is an array of objects containing data injected by the modules registered in the chain during the block creation.
This change clarifies the general procedure by which modules insert extra data in a block.
The second change is to update the block header schema.
In general, it is desirable to have one fixed block header schema that:
- does not need to be changed when modules are added or removed to a running blockchain,
- is used by every blockchain in the Lisk ecosystem regardless of the modules implemented in the individual chains.
Furthermore, we update the block header schema to include new properties introduced by the “State model and state root” LIP, the “Introduce BFT module” LIP, and the “Introduce a certificate generation mechanism” LIP.
Finally, this LIP specifies the validation of all block header properties in one place.
Rationale
New Block Header Properties
This LIP introduces the following new block header properties:
-
stateRoot
: The root of the sparse Merkle tree that is computed from the state of the blockchain.
See the State model and state root LIP to see why it needs to be included in a block header. -
assetsRoot
: The root of the Merkle tree computed from the block assets array.
See below for more details. -
generatorAddress
: The address of the block generator.
It replaces thegeneratorPublicKey
property.
See below for more details. -
aggregateCommit
: This property contains the aggregate signature for a certificate for a certain block height.
Based on this, any node can create a certificate for the corresponding height.
See the Introduce a certificate generation mechanism LIP for more details. -
maxHeightPrevoted
: This property is related to the Lisk BFT protocol and is used for the fork choice rule. -
maxHeightGenerated
: This property is related to the Lisk BFT protocol and is used to check for contradicting block headers. -
validatorsHash
: This property authenticates the set of validators active from the next block onward. It is used to generate certificates from the block header.
Change Generator Public Key to Generator Address
Before this proposal, the generatorPublicKey
property of a block header was fulfilling two purposes: 1) The validator account was deduced by deriving the address from it, and 2) the block signature could be validated without any on-chain data.
Both the generator address or a public key yielding this address fulfill the first purpose.
On the other hand, the second point is not possible anymore without on-chain data, as the generator key is now part of the validators module store and can be updated.
Hence, there is no further drawback in replacing the generatorPublicKey property by the generatorAddress property, while it has the advantages of reducing the size of the block header by a few bytes and skipping the address derivation step during block validation.
Separation Between Block Header and Block Assets
The separation between properties in the bock header and properties in the block assets is done according to the following rules:
- Properties created by the framework layer of the Lisk SDK are added to the block header.
- Properties created by individual modules are added to the block assets.
- It should be possible to follow the fork choice consensus rule just with the block header. This implies that the
maxHeightPrevoted
property is part of the block header. - Similarly, it should be possible to generate a certificate just with the block header. This implies that the
validatorsHash
property is part of the block header.
Moreover, thevalidatorsHash
property can only be obtained after the state transitions by the modules have been processed.
The reason is that the DPoS or PoA modules only set the validators for the next round after the asset of the last block of a round is processed.
Therefore, this property needs to be added to the block by the framework layer after the state transitions by the modules are processed.
As an example, blockchains created with the Lisk SDK that implement the random module, will insert the seed reveal property in the block assets, not in the block header.
The schema for the block assets allows each module to include its serialized data individually, which makes the inclusion of module data very flexible.
Each module can insert a single entry in the assets.
This entry is an object containing a moduleID
property, indicating the ID of the module handling it, and a generic data
property that can contain arbitrary serialized data.
Each entry of the block assets is then inserted in a Merkle tree, whose root is included in the block header as the assetsRoot
property.
Inserting the assets root rather than the full assets allows to bound the size of the block header while still authenticating the content of the block assets.
Specification
Notation and Constants
For the rest of this proposal we define the following constants.
Name | Type | Value | Description |
---|---|---|---|
MAX_PAYLOAD_SIZE_BYTES |
integer | TBD | The max size of a block payload in bytes. |
MAX_ASSET_DATA_SIZE_BYTES |
integer | TBD | The max size of an assets entry in bytes. |
SIGNATURE_LENGTH_BYTES |
integer | 64 | The length of a Ed25519 signature. |
Furthermore, in the following we indicate with block
be the block under consideration and with previousBlock
the previous block of the chain.
Calling a function fct
from another module module
is represented by module.fct
.
Block
JSON Schema
Blocks are serialized and deserialized accordingly to the following JSON schema.
blockSchema = {
"type": "object",
"properties": {
"header": {
"dataType": "bytes",
"fieldNumber": 1
},
"payload": {
"type": "array",
"items": {
"dataType": "bytes"
},
"fieldNumber": 2
},
"assets": {
"type": "array",
"items": {
"dataType": "bytes"
},
"fieldNumber": 3
}
},
"required": ["header", "payload", "assets"
]
}
Validation
Blocks are validated at three different stages of their lifecycle.
-
Static validation: When a new block is received, some initial static checks are done to ensure that the serialized object follows the general structure of a block.
These checks are performed immediately because they do not require access to the state store and can therefore be done very quickly. - Before block execution: Properties that require access to the state store before the block has been executed are validated in this stage.
- After block execution: In this stage we validate the properties that require access to the state store after the block has been executed, i.e. they can be validated only after the state transitions implied by the block execution have been performed.
As part of the static validation checks, we check that the total size of the serialized transactions contained in the block payload is at most MAX_PAYLOAD_SIZE_BYTES
.
Block ID
The block ID is calculated using the blockID
function.
This function returns a 32 bytes value or an error if the block header has an invalid signature format.
blockID():
# Check that the signature length is 64 bytes
if length(block.header.signature) != SIGNATURE_LENGTH_BYTES:
return error
let serializedBlockHeader be the serialization of block.header following the blockHeaderSchema
return SHA-256(serializedBlockHeader)
Block Assets
This LIP introduces a new block property, the block assets, which in addition with the header and the payload forms the complete block.
JSON Schema
The block assets contains data created by individual modules.
It is an array of bytes, where each value corresponds to an object serialized according to the following schema.
assetSchema = {
"type": "object",
"properties": {
"moduleID": {
"dataType": "uint32",
"fieldNumber": 1
},
"data": {
"dataType": "bytes",
"fieldNumber": 2
}
},
"required": ["moduleID", "data"]
}
Validation
The block assets is validated in the static validation stage as follows:
-
Static validation:
- Check that each entry in the assets array has
moduleID
set to the ID of a module registered in the chain, while thedata
property has size at most equal toMAX_ASSET_DATA_SIZE_BYTES
. - Each module can insert at most one entry in the block assets.
Hence, check that each entry must has a distinctmoduleID
property. - Check that the entries are sorted by increasing values of
moduleID
.
- Check that each entry in the assets array has
These validations are performed before the block is processed and without accessing the state.
Block Header
Block headers are serialized and deserialized accordingly to the following JSON schema.
JSON Schema
blockHeaderSchema = {
"type": "object",
"properties": {
"version": {
"dataType": "uint32",
"fieldNumber": 1
},
"timestamp": {
"dataType": "uint32",
"fieldNumber": 2
},
"height": {
"dataType": "uint32",
"fieldNumber": 3
},
"previousBlockID": {
"dataType": "bytes",
"fieldNumber": 4
},
"generatorAddress": {
"dataType": "bytes",
"fieldNumber": 5
},
"transactionRoot": {
"dataType": "bytes",
"fieldNumber": 6
},
"assetsRoot": {
"dataType": "bytes",
"fieldNumber": 7
},
"stateRoot": {
"dataType": "bytes",
"fieldNumber": 8
},
"maxHeightPrevoted": {
"dataType": "uint32",
"fieldNumber": 9
},
"maxHeightGenerated": {
"dataType": "uint32",
"fieldNumber": 10
},
"validatorsHash": {
"dataType": "bytes",
"fieldNumber": 11
},
"aggregateCommit": {
"type": "object",
"fieldNumber": 12,
"properties": {
"height": {
"dataType": "uint32",
"fieldNumber": 1
},
"aggregationBits": {
"dataType": "bytes",
"fieldNumber": 2
},
"signature": {
"dataType": "bytes",
"fieldNumber": 3
}
},
"required": [
"height",
"aggregationBits",
"signature"
]
},
"signature": {
"dataType": "bytes",
"fieldNumber": 13
}
},
"required": [
"version",
"timestamp",
"height",
"previousBlockID",
"generatorAddress",
"transactionRoot",
"assetsRoot",
"stateRoot",
"maxHeightPrevoted",
"maxHeightGenerated",
"validatorsHash",
"aggregateCommit"
]
}
Validation
In this section, we specify the validation for each property of the block header.
The block header is validated in all three stages of the block validation.
-
Static validation:
- Check that the block header follows the block header schema.
- Validate the
version
,transactionRoot
, andassetsRoot
properties.
-
Before block execution:
- Validate the
timestamp
,height
,previousBlockID
,generatorAddress
,maxHeightPrevoted
,maxHeightGenerated
,aggregateCommit
, andsignature
properties.
- Validate the
-
After block execution:
- Validate the
stateRoot
andvalidatorsHash
properties.
- Validate the
Version
With this LIP, the version value is incremented.
That means that block.header.version
must be equal the value of a block of the previous protocol plus one.
Timestamp
The timestamp is validated by calling the validateTimestamp
.
This function returns a boolean, indicating the success of the block validation.
validateTimestamp():
blockSlotNumber = validators.getSlotNumber(block.header.timestamp)
# Check that block is not from the future
let currentTimestamp be the current system time
if blockSlotNumber > validators.getSlotNumber(currentTimestamp):
return False
# Check that block slot is strictly larger than the block slot of previousBlock
previousBlockSlotNumber = validators.getSlotNumber(previousBlock.header.timestamp)
if blockSlotNumber <= previousBlockSlotNumber:
return False
return True
Height
The height is validated by calling the validateHeight
function.
This function returns a boolean, indicating the success of the block validation.
validateHeight():
return block.header.height == previousBlock.header.height + 1
Previous Block ID
The height is validated by calling the validatePreviousBlockID
function.
This function returns a boolean, indicating the success of the block validation.
validatePreviousBlockID():
return block.header.previousBlockID == blockID(previousBlock)
Here, the function blockID
calculates the ID of an input block as specified in LIP 20.
Generator Address
The generator address is validated by calling the validateGeneratorAddress
function.
This function returns a boolean, indicating the success of the block validation.
validateGeneratorAddress():
# Check that the generatorAddress has the correct length of 20 bytes
if length(block.header.generatorAddress) != 20:
return False
# Check that the block generator is eligible to generate in this block slot.
return block.header.generatorAddress == validators.getGeneratorAtTimestamp(block.header.timestamp)
Transaction Root
The transaction root is the root of the Merkle tree built from the ID of the transactions contained in the block payload.
It is validated by calling the validateTransactionRoot
function.
This function returns a boolean, indicating the success of the block validation.
validateTransactionRoot():
transactionIDs = [transactionID(trs) for trs in block.payload]
return block.header.transactionRoot == merkleRoot(transactionIDs)
Here, the function transactionID
calculates the ID of an input transaction as specified in LIP 19 and the function merkleRoot
calculates the Merkle root starting from an input array of bytes values as defined in LIP 31.
Assets Root
The assets root is the root of the Merkle tree built from the block assets array.
It is validated by calling the validateAssetsRoot
function.
This function returns a boolean, indicating the success of the block validation.
validateAssetsRoot():
assetHashes = [SHA-256(asset) for asset in block.assets]
return block.header.assetsRoot == merkleRoot(assetHashes)
State Root
The state root is the root of the sparse Merkle tree built from the state of the chain after the block has been processed.
It is validated by calling the validateStateRoot
function.
This function returns a boolean, indicating the success of the block validation.
validateStateRoot():
return block.header.stateRoot == stateRoot(block.header.height)
Here, the function stateRoot
calculates the state root of the chain at the input height as specified in LIP 40.
Max Height Prevoted and Max Height Generated
The properties maxHeightPrevoted
and maxHeightGenerated
are related to the Lisk-BFT protocol.
They are validated by calling the validateBFT
function.
This function returns a boolean, indicating the success of the block validation.
validateBFT():
if block.header.maxHeightPrevoted != bft.getMaxHeightPrevoted():
return False
return not bft.isHeaderContradictingChain(block.header)
Validators Hash
The validators hash authenticates the set of validators participating to Lisk-BFT from height block.header.height + 1
onward.
They are validated by calling the validateValidatorsHash
function.
The function returns a boolean, indicating the success of the block validation.
validateValidatorsHash():
return block.header.validatorsHash == bft.getValidatorsHash()
Aggregate Commit
The aggregate commit contains an aggregate BLS signature of a certificate corresponding to the block at the given height.
It attests that all signing validators consider the corresponding block final.
It is validated by calling the validateAggregateCommit
function, defined in LIP “Introduce a certificate generation mechanism”.
The function takes the block block
as input and returns a boolean, indicating the success of the block validation.
Signature
The signature is validated by calling the validateBlockSignature
function.
This function returns a boolean, indicating the success of the block validation.
validateBlockSignature():
generatorKey = validators.getValidatorAccount(block.header.generatorAddress).generatorKey
signature = block.header.signature
# Remove the signature from the block header
delete block.header.signature
# Serialize the block header without signature
let serializedUnsignedBlockHeader be the serialization of block.header following the blockHeaderSchema
let networkIdentifier be the network identifier of the chain
return verifyMessageSig(generatorKey, "LSK_BH_", networkIdentifier, serializedUnsignedBlockHeader, signature)
Here, the function verifyMessageSig
verifies the validity of a signature as specified in LIP 37.
Backwards Compatibility
This LIP results in a hard fork as nodes following the proposed protocol will reject blocks according to the previous protocol, and nodes following the previous protocol will reject blocks according to the proposed protocol.