Update block schema and block processing

Blocks
1

Hello everyone,

I would like to propose a new LIP for the roadmap objective “Update block header format.” This LIP covers all changes to the block header schema and block asset schema introduced by several other LIPs.

I’m looking forward to your feedback.

Here is the complete LIP draft:

LIP: <LIP number>
Title: Update block schema and block processing
Author: Andreas Kendziorra <andreas.kendziorra@lightcurve.io>
Type: Standards Track
Created: <YYYY-MM-DD>
Updated: <YYYY-MM-DD>
Requires: 0040,
          Introduce a certificate generation mechanism

Abstract

This LIP changes the structure of a block, introducing the assets property alongside the block header and payload.
The block header schema is updated to add new properties introduced by several other LIPs.
We clarify the mechanism by which modules can include data in the block assets and specify the validation of each block header property.

Copyright

This LIP is licensed under the Creative Commons Zero 1.0 Universal.

Motivation

The first change proposed by this LIP is to introduce a new block property, the block assets, and to update the block schema accordingly.
This property is an array of objects containing data injected by the modules registered in the chain during the block creation.
This change clarifies the general procedure by which modules insert extra data in a block.

The second change is to update the block header schema.
In general, it is desirable to have one fixed block header schema that:

  • does not need to be changed when modules are added or removed to a running blockchain,
  • is used by every blockchain in the Lisk ecosystem regardless of the modules implemented in the individual chains.

Furthermore, we update the block header schema to include new properties introduced by the “State model and state root” LIP, the “Introduce BFT module” LIP, and the “Introduce a certificate generation mechanism” LIP.

Finally, this LIP specifies the validation of all block header properties in one place.

Rationale

New Block Header Properties

This LIP introduces the following new block header properties:

  • stateRoot: The root of the sparse Merkle tree that is computed from the state of the blockchain.
    See the State model and state root LIP to see why it needs to be included in a block header.
  • assetsRoot: The root of the Merkle tree computed from the block assets array.
    See below for more details.
  • generatorAddress: The address of the block generator.
    It replaces the generatorPublicKey property.
    See below for more details.
  • aggregateCommit: This property contains the aggregate signature for a certificate for a certain block height.
    Based on this, any node can create a certificate for the corresponding height.
    See the Introduce a certificate generation mechanism LIP for more details.
  • maxHeightPrevoted: This property is related to the Lisk BFT protocol and is used for the fork choice rule.
  • maxHeightGenerated: This property is related to the Lisk BFT protocol and is used to check for contradicting block headers.
  • validatorsHash: This property authenticates the set of validators active from the next block onward. It is used to generate certificates from the block header.

Change Generator Public Key to Generator Address

Before this proposal, the generatorPublicKey property of a block header was fulfilling two purposes: 1) The validator account was deduced by deriving the address from it, and 2) the block signature could be validated without any on-chain data.

Both the generator address or a public key yielding this address fulfill the first purpose.
On the other hand, the second point is not possible anymore without on-chain data, as the generator key is now part of the validators module store and can be updated.
Hence, there is no further drawback in replacing the generatorPublicKey property by the generatorAddress property, while it has the advantages of reducing the size of the block header by a few bytes and skipping the address derivation step during block validation.

Separation Between Block Header and Block Assets

The separation between properties in the bock header and properties in the block assets is done according to the following rules:

  • Properties created by the framework layer of the Lisk SDK are added to the block header.
  • Properties created by individual modules are added to the block assets.
  • It should be possible to follow the fork choice consensus rule just with the block header. This implies that the maxHeightPrevoted property is part of the block header.
  • Similarly, it should be possible to generate a certificate just with the block header. This implies that the validatorsHash property is part of the block header.
    Moreover, the validatorsHash property can only be obtained after the state transitions by the modules have been processed.
    The reason is that the DPoS or PoA modules only set the validators for the next round after the asset of the last block of a round is processed.
    Therefore, this property needs to be added to the block by the framework layer after the state transitions by the modules are processed.

As an example, blockchains created with the Lisk SDK that implement the random module, will insert the seed reveal property in the block assets, not in the block header.

The schema for the block assets allows each module to include its serialized data individually, which makes the inclusion of module data very flexible.
Each module can insert a single entry in the assets.
This entry is an object containing a moduleID property, indicating the ID of the module handling it, and a generic data property that can contain arbitrary serialized data.

Each entry of the block assets is then inserted in a Merkle tree, whose root is included in the block header as the assetsRoot property.
Inserting the assets root rather than the full assets allows to bound the size of the block header while still authenticating the content of the block assets.

Specification

Notation and Constants

For the rest of this proposal we define the following constants.

Name Type Value Description
MAX_PAYLOAD_SIZE_BYTES integer TBD The max size of a block payload in bytes.
MAX_ASSET_DATA_SIZE_BYTES integer TBD The max size of an assets entry in bytes.
SIGNATURE_LENGTH_BYTES integer 64 The length of a Ed25519 signature.

Furthermore, in the following we indicate with block be the block under consideration and with previousBlock the previous block of the chain.
Calling a function fct from another module module is represented by module.fct.

Block

JSON Schema

Blocks are serialized and deserialized accordingly to the following JSON schema.

blockSchema = {
  "type": "object",
  "properties": {
    "header": {
      "dataType": "bytes",
      "fieldNumber": 1
    },
    "payload": {
      "type": "array",
      "items": {
        "dataType": "bytes"
      },
      "fieldNumber": 2
    },
    "assets": {
      "type": "array",
      "items": {
        "dataType": "bytes"
      },
      "fieldNumber": 3
    }
  },
  "required": ["header", "payload", "assets"
  ]
}

Validation

Blocks are validated at three different stages of their lifecycle.

  • Static validation: When a new block is received, some initial static checks are done to ensure that the serialized object follows the general structure of a block.
    These checks are performed immediately because they do not require access to the state store and can therefore be done very quickly.
  • Before block execution: Properties that require access to the state store before the block has been executed are validated in this stage.
  • After block execution: In this stage we validate the properties that require access to the state store after the block has been executed, i.e. they can be validated only after the state transitions implied by the block execution have been performed.

As part of the static validation checks, we check that the total size of the serialized transactions contained in the block payload is at most MAX_PAYLOAD_SIZE_BYTES.

Block ID

The block ID is calculated using the blockID function.
This function returns a 32 bytes value or an error if the block header has an invalid signature format.

blockID():
  # Check that the signature length is 64 bytes
  if length(block.header.signature) != SIGNATURE_LENGTH_BYTES:
    return error

  let serializedBlockHeader be the serialization of block.header following the blockHeaderSchema
  return SHA-256(serializedBlockHeader)

Block Assets

This LIP introduces a new block property, the block assets, which in addition with the header and the payload forms the complete block.

JSON Schema

The block assets contains data created by individual modules.
It is an array of bytes, where each value corresponds to an object serialized according to the following schema.

assetSchema = {
  "type": "object",
  "properties": {
    "moduleID": {
      "dataType": "uint32",
      "fieldNumber": 1
    },
    "data": {
      "dataType": "bytes",
      "fieldNumber": 2
    }
  },
  "required": ["moduleID", "data"]
}

Validation

The block assets is validated in the static validation stage as follows:

  • Static validation:
    • Check that each entry in the assets array has moduleID set to the ID of a module registered in the chain, while the data property has size at most equal to MAX_ASSET_DATA_SIZE_BYTES.
    • Each module can insert at most one entry in the block assets.
      Hence, check that each entry must has a distinct moduleID property.
    • Check that the entries are sorted by increasing values of moduleID.

These validations are performed before the block is processed and without accessing the state.

Block Header

Block headers are serialized and deserialized accordingly to the following JSON schema.

JSON Schema

blockHeaderSchema = {
  "type": "object",
  "properties": {
    "version": {
      "dataType": "uint32",
      "fieldNumber": 1
    },
    "timestamp": {
      "dataType": "uint32",
      "fieldNumber": 2
    },
    "height": {
      "dataType": "uint32",
      "fieldNumber": 3
    },
    "previousBlockID": {
      "dataType": "bytes",
      "fieldNumber": 4
    },
    "generatorAddress": {
      "dataType": "bytes",
      "fieldNumber": 5
    },
    "transactionRoot": {
      "dataType": "bytes",
      "fieldNumber": 6
    },
    "assetsRoot": {
      "dataType": "bytes",
      "fieldNumber": 7
    },
    "stateRoot": {
      "dataType": "bytes",
      "fieldNumber": 8
    },
    "maxHeightPrevoted": {
      "dataType": "uint32",
      "fieldNumber": 9
    },
    "maxHeightGenerated": {
      "dataType": "uint32",
      "fieldNumber": 10
    },
    "validatorsHash": {
      "dataType": "bytes",
      "fieldNumber": 11
    },
    "aggregateCommit": {
      "type": "object",
      "fieldNumber": 12,
      "properties": {
        "height": {
          "dataType": "uint32",
          "fieldNumber": 1
        },
        "aggregationBits": {
          "dataType": "bytes",
          "fieldNumber": 2
        },
        "signature": {
          "dataType": "bytes",
          "fieldNumber": 3
        }
      },
      "required": [
        "height",
        "aggregationBits",
        "signature"
      ]
    },
    "signature": {
      "dataType": "bytes",
      "fieldNumber": 13
    }
  },
  "required": [
    "version",
    "timestamp",
    "height",
    "previousBlockID",
    "generatorAddress",
    "transactionRoot",
    "assetsRoot",
    "stateRoot",
    "maxHeightPrevoted",
    "maxHeightGenerated",
    "validatorsHash",
    "aggregateCommit"
  ]
}

Validation

In this section, we specify the validation for each property of the block header.

The block header is validated in all three stages of the block validation.

  • Static validation:
    • Check that the block header follows the block header schema.
    • Validate the version, transactionRoot, and assetsRoot properties.
  • Before block execution:
    • Validate the timestamp, height, previousBlockID, generatorAddress, maxHeightPrevoted, maxHeightGenerated, aggregateCommit, and signature properties.
  • After block execution:
    • Validate the stateRoot and validatorsHash properties.
Version

With this LIP, the version value is incremented.
That means that block.header.version must be equal the value of a block of the previous protocol plus one.

Timestamp

The timestamp is validated by calling the validateTimestamp.
This function returns a boolean, indicating the success of the block validation.

validateTimestamp():
  blockSlotNumber = validators.getSlotNumber(block.header.timestamp)

  # Check that block is not from the future
  let currentTimestamp be the current system time
  if blockSlotNumber > validators.getSlotNumber(currentTimestamp):
    return False

  # Check that block slot is strictly larger than the block slot of previousBlock
  previousBlockSlotNumber = validators.getSlotNumber(previousBlock.header.timestamp)
  if blockSlotNumber <= previousBlockSlotNumber:
      return False

  return True
Height

The height is validated by calling the validateHeight function.
This function returns a boolean, indicating the success of the block validation.

validateHeight():
  return block.header.height == previousBlock.header.height + 1
Previous Block ID

The height is validated by calling the validatePreviousBlockID function.
This function returns a boolean, indicating the success of the block validation.

validatePreviousBlockID():
  return block.header.previousBlockID == blockID(previousBlock)

Here, the function blockID calculates the ID of an input block as specified in LIP 20.

Generator Address

The generator address is validated by calling the validateGeneratorAddress function.
This function returns a boolean, indicating the success of the block validation.

validateGeneratorAddress():

  # Check that the generatorAddress has the correct length of 20 bytes
  if length(block.header.generatorAddress) != 20:
    return False

  # Check that the block generator is eligible to generate in this block slot.
  return block.header.generatorAddress == validators.getGeneratorAtTimestamp(block.header.timestamp)
Transaction Root

The transaction root is the root of the Merkle tree built from the ID of the transactions contained in the block payload.
It is validated by calling the validateTransactionRoot function.
This function returns a boolean, indicating the success of the block validation.

validateTransactionRoot():
  transactionIDs = [transactionID(trs) for trs in block.payload]
  return block.header.transactionRoot == merkleRoot(transactionIDs)

Here, the function transactionID calculates the ID of an input transaction as specified in LIP 19 and the function merkleRoot calculates the Merkle root starting from an input array of bytes values as defined in LIP 31.

Assets Root

The assets root is the root of the Merkle tree built from the block assets array.
It is validated by calling the validateAssetsRoot function.
This function returns a boolean, indicating the success of the block validation.

validateAssetsRoot():
  assetHashes = [SHA-256(asset) for asset in block.assets]
  return block.header.assetsRoot == merkleRoot(assetHashes)
State Root

The state root is the root of the sparse Merkle tree built from the state of the chain after the block has been processed.
It is validated by calling the validateStateRoot function.
This function returns a boolean, indicating the success of the block validation.

validateStateRoot():
  return block.header.stateRoot == stateRoot(block.header.height)

Here, the function stateRoot calculates the state root of the chain at the input height as specified in LIP 40.

Max Height Prevoted and Max Height Generated

The properties maxHeightPrevoted and maxHeightGenerated are related to the Lisk-BFT protocol.
They are validated by calling the validateBFT function.
This function returns a boolean, indicating the success of the block validation.

validateBFT():
  if block.header.maxHeightPrevoted != bft.getMaxHeightPrevoted():
    return False

  return not bft.isHeaderContradictingChain(block.header)
Validators Hash

The validators hash authenticates the set of validators participating to Lisk-BFT from height block.header.height + 1 onward.
They are validated by calling the validateValidatorsHash function.
The function returns a boolean, indicating the success of the block validation.

validateValidatorsHash():
  return block.header.validatorsHash == bft.getValidatorsHash()
Aggregate Commit

The aggregate commit contains an aggregate BLS signature of a certificate corresponding to the block at the given height.
It attests that all signing validators consider the corresponding block final.
It is validated by calling the validateAggregateCommit function, defined in LIP “Introduce a certificate generation mechanism”.
The function takes the block block as input and returns a boolean, indicating the success of the block validation.

Signature

The signature is validated by calling the validateBlockSignature function.
This function returns a boolean, indicating the success of the block validation.

validateBlockSignature():
  generatorKey = validators.getValidatorAccount(block.header.generatorAddress).generatorKey
  signature = block.header.signature 

  # Remove the signature from the block header
  delete block.header.signature
  # Serialize the block header without signature
  let serializedUnsignedBlockHeader be the serialization of block.header following the blockHeaderSchema

  let networkIdentifier be the network identifier of the chain

  return verifyMessageSig(generatorKey, "LSK_BH_", networkIdentifier, serializedUnsignedBlockHeader, signature)

Here, the function verifyMessageSig verifies the validity of a signature as specified in LIP 37.

Backwards Compatibility

This LIP results in a hard fork as nodes following the proposed protocol will reject blocks according to the previous protocol, and nodes following the previous protocol will reject blocks according to the proposed protocol.

2 Likes
2

I updated the LIP extensively to include the general processing for a block and a new block schema.

3

I opened a PR on the LIPs repository.

4

The PR was merged as LIP 0055.