Define schema and use generic serialization for blocks

alessandro.ricottone · February 18, 2020, 11:04am

Hello everyone,

In this thread, I want to propose a new LIP for the roadmap objective “Introduce universal serialization method”. This proposal defines how the generic serialization algorithm will be applied to blocks and specify the appropriate JSON schema.

Looking forward to your feedback.

Here is the complete LIP draft:

LIP: <LIP number>
Title: Define schema and use generic serialization for blocks 
Author: Alessandro Ricottone <alessandro.ricottone@lightcurve.io>
Type: Standards Track
Created: <YYYY-MM-DD>
Updated: <YYYY-MM-DD>
Requires: 00xx: "A generic, deterministic and size efficient serialization method"

Abstract

This LIP defines how the generic serialization defined in LIP 00xx “A generic, deterministic and size efficient serialization method” is applied to blocks by specifying the appropriate JSON schema. We restructure the block header and introduce the block asset property, storing properties specific to the corresponding chain. We further specify the block-asset schema for the Lisk mainchain.

Copyright

This LIP is licensed under the Creative Commons Zero 1.0 Universal.

Motivation

Having a standard way of serializing blocks is beneficial in several parts of the Lisk protocol:

Blocks will be serialized and inserted in a Merkle tree to calculate the block root for the “Introduce decentralized re-genesis” roadmap objective.
An optimized block serialization can improve storage efficiency.
The Lisk P2P protocol can use the generic serialization for transmission to reduce bandwidth requirements.
An expandable block-asset schema can be included in the SDK and used for sidechains to make the development of custom blockchains easier.

Given these premises, we aim for a minimal, yet flexible and expandable schema for block serialization. LIP 00xx “A generic, deterministic and size efficient serialization method” defines the general serialization method and how JSON schemas are used to serialize data in the Lisk protocol. In this LIP we specify the JSON schemas to serialize a full block, a block header and a block asset in the Lisk mainchain.

Rationale

We define the block schema for the full block serialization. This schema contains two properties, the header property, whose serialization is specified by the schema blockHeader, and the payload property. The separation of block header and payload is due to the fact that typically transactions in the payload are stored separately from the block header, so that it is easy to query and access both transactions and blocks separately. These transactions are serialized according to the method defined in LIP 00xx “Use generic serialization for transactions”.

Among other properties, the blockHeader schema contains the asset property for the block asset. Similarly to the serialization process for transactions, the block asset is serialized separately from the rest of the block header, using the blockAsset schema specified by the version property in the block header. When the protocol version changes, the version property is updated, and some properties of the block asset may change as well. Using this method, we can upgrade the blockAsset schema without changing the whole block schema. In this LIP, we specify the blockAsset schema valid for version=2 on the Lisk mainchain. The same schema is used as default for sidechains developed using the Lisk SDK. Sidechain developers can also define their own custom schemas with additional properties, e.g., for custom transactions.

Some properties that are part of the block header in the current protocol will be removed:

id: The block ID. This value can be obtained from the SHA-256 hash of the serialized block header.
numberOfTransactions: The number of transactions in the payload. This value can be inferred from the payload.
totalAmount: The amount of tokens transferred with the transactions included in the block. This value can be inferred from the transactions included in the payload.
totalFee: The sum of the fees associated with the transactions included in the block. This value can be inferred from the transactions included in the payload.
payloadHash: The SHA-256 hash of the block payload. After the implementation of LIP 00xx “Replace payloadHash with Merkle tree root in block header”, this value is replaced by the transactionRoot.
payloadLength: The length in bytes of the payload. This value can be inferred from the payload.

Specification

`block` Schema

A schematic of the block-serialization structure. The block header is serialized according to the blockHeader schema. The payload is an array of transactions, serialized according to the method described in LIP00xx.

The block schema is used to serialize blocks, for example before transmitting them to peers in the P2P layer.

The block schema contains 2 properties:

header: The serialized block header. Its serialization is specified by the blockHeader schema.
payload: The block payload, containing all transactions included in the block, serialized according to LIP00xx.

Serialization

Consider a data structure blockData to be serialized, representing a valid block according to the Lisk protocol. The serialization procedure is done in 3 steps:

Each transaction trs in the block payload blockData.payload is serialized according to the method described in LIP00xx. The resulting bytes replace the original trs in the payload.
The block header blockData.header is serialized using the method described below, and the resulting bytes replace the original value in blockData.
blockData is serialized according to the block schema.

Deserialization

Consider a binary message blockMsg to be deserialized. The deserialization procedure is done in 3 steps:

blockMsg is deserialized according to the block schema.
Each transaction in the block payload block.payload is deserialized using the method defined in LIP00xx.
The block header block.header is deserialized using the the method described below.

block = {
    "type": "object",
    "properties": {
        "header": {
            "dataType": "bytes",
            "fieldNumber": 1 
        },     
        "payload": { 
            "type": "array",
            "items": {
                "dataType": "bytes"
            },
            "fieldNumber": 2
        }
    },
    "required": [
        "header",
        "payload"
    ]
}

`blockHeader` Schema

The blockHeader schema is used to serialize the block header as part of the full block serialization and to calculate the block signature and ID. Furthermore, block headers can be serialized to be stored in a database separated from the block payload. All properties of the blockHeader schema are required, with the exception of the signature.

The blockHeader schema contains the following properties:

version: An integer indicating the protocol version used by the block. It specifies the JSON schema to be used to serialize and deserialize the asset property of the block. The value of this property at the time of adoption of this LIP is version=2.
timestamp: An integer indicating the epoch timestamp of the block creation starting from the genesis block.
height: An integer indicating the block height.
previousBlockID: The ID of the previous block in the chain. A valid block ID is 32 bytes long.
transactionRoot: The Merkle root of the payload tree. This value is 32 bytes long.
generatorPublicKey: The public key of the block forger, used to sign the block header. A valid public key is 32 bytes long.
reward: An integer indicating the reward in Beddows for the block forger.
asset: The asset stores blockchain-specific properties.
signature: The signature of the block header. Notice that this property is not required (see Block Signature section below). A valid signature is 64 bytes long.

blockHeader = {
    "type": "object",
    "properties": {
        "version": {
            "dataType": "uint32",
            "fieldNumber": 1
        },
        "timestamp": {
            "dataType": "uint32",
            "fieldNumber": 2
        },
        "height": {
            "dataType": "uint32",
            "fieldNumber": 3
        },
        "previousBlockID": {
            "dataType": "bytes",
            "fieldNumber": 4
        },
        "transactionRoot": {
            "dataType": "bytes",
            "fieldNumber": 5
        },
        "generatorPublicKey": {
            "dataType": "bytes",
            "fieldNumber": 6
        },
        "reward": {
            "dataType": "uint64",
            "fieldNumber": 7
        },
        "asset": {
            "dataType": "bytes",
            "fieldNumber": 8
        },
        "signature": {
            "dataType": "bytes",
            "fieldNumber": 9
        },
    },
    "required": [
        "version",
        "timestamp",
        "height",
        "previousBlockID",
        "transactionRoot",
        "generatorPublicKey",
        "reward",
        "asset"
    ]
}

Serialization

Consider a data structure blockHeaderData to be serialized, representing a valid block header according to the Lisk protocol. The serialization procedure is done in 3 steps:

The correct blockAsset schema is selected according to the value of the blockHeaderData.version property. The block asset blockHeaderData.asset is serialized according to the blockAsset schema.
The binary value from step 1 is inserted in the blockHeaderData.asset property replacing the original value.
The blockHeaderData from step 2 is serialized according to the blockHeader schema.

Deserialization

Consider a binary message blockHeaderMsg to be deserialized. The deserialization procedure is done in 3 steps:

blockHeaderMsg is deserialized according to the blockHeader schema to obtain blockHeaderData.
The serialized block asset blockHeaderData.asset is deserialized using the blockAsset schema, chosen according to the value of the blockHeaderData.version property.
The deserialized block asset from step 2 is inserted in the blockHeaderData.asset property.

Block Signature Calculation

The blockHeader schema specifies how to serialize all the information necessary to sign a block header and generate the block ID. In the Lisk protocol, block headers are serialized and signed by the forging delegate.

Given a data structure unsignedBlockHeaderData representing a block header with no signature property, the block signature is calculated as follows:

unsignedBlockHeaderData is serialized using the method explained above. In particular, the serialized data does not contain the signature.
The block signature is calculated by signing the binary message from step1.
unsignedBlockHeaderData.signature is set to the output of step 2.

Block Signature Validation

Given a binary message signedBlockHeaderMsg representing a serialized block header with a valid signature property, the block signature is verified as follows:

signedBlockHeaderMsg is deserialized using the blockHeader schema into blockHeaderData, a data structure representing a signed block header.
The block signature is read from blockHeaderData.signature and the signature property is then removed from blockHeaderData.
blockHeaderData is re-serialized according to the method described above. In particular, the serialized message does not contain the signature.
The block signature is verified against the output of step 3.

Block ID

Given a data structure signedBlockHeaderData representing a block header with a signature property, the block ID is calculated as follows:

signedBlockHeaderData is serialized using the method explained above.
The block ID is calculated as the SHA-256 hash of the binary message from step1.

`blockAsset` Schema

The blockAsset schema for the Lisk mainchain contains properties related to the Lisk consensus algorithm. The block asset is serialized separately from the rest of the block, to be able to upgrade the blockAsset schema whenever the protocol version changes.

The blockAsset schema is used to serialize the block-header asset as part of the block header serialization. All properties of the blockAsset schema are required.

The blockAsset schema contains the following properties:

maxHeightPreviouslyForged: An integer indicating the largest height of any block previously forged by the delegate.
maxHeightPrevoted: An integer indicating the height of the last ancestor block with at least 68 prevotes. The fork-choice rule in the BFT consensus protocol specifies that delegates choose the longest chain that contains the highest maxHeightPrevoted.
seedReveal: A value revealed by each forging delegate for the Randao-based random number generation used to select stand-by delegates. This value is 16 bytes long.

blockAsset = {
    "type": "object",
    "properties": {
        "maxHeightPreviouslyForged": {
            "dataType": "uint32",
            "fieldNumber": 1
        },
        "maxHeightPrevoted": {
            "dataType": "uint32",
            "fieldNumber": 2
        },
        "seedReveal": {
            "dataType": "bytes",
            "fieldNumber": 3
        }
    },
    "required": [
        "maxHeightPreviouslyForged",
        "maxHeightPrevoted",
        "seedReveal"
    ]
}

Backwards Compatibility

This proposal introduces a hard fork in the network. After its implementation, block serialization will change, resulting in different block signatures and block IDs.

Appendix A: Block Schema in the Current Protocol

In this section, we present the block schema baseBlockSchema used in the current protocol, as a reference for comparison with the new schema defined in this LIP.

baseBlockSchema = {
    "type": "object",
    "properties": {
        "id": {
            "type": "string",
            "format": "id",
            "minLength": 1,
            "maxLength": 20,
        },
        "height": {
            "type": "integer"
        },
        "blockSignature": {
            "type": "string",
            "format": "signature"
        },
        "generatorPublicKey": {
            "type": "string",
            "format": "publicKey"
        },
        "numberOfTransactions": {
            "type": "integer"
        },
        "payloadHash": {
            "type": "string",
            "format": "hex"
        },
        "payloadLength": {
            "type": "integer"
        },
        "previousBlockId": {
            "type": "string",
            "format": "id",
            "minLength": 1,
            "maxLength": 20
        },
        "timestamp": {
            "type": "integer"
        },
        "totalAmount": {
            "type": "object",
            "format": "amount"
        },
        "totalFee": {
            "type": "object",
            "format": "amount"
        },
        "reward": {
            "type": "object",
            "format": "amount"
        },
        "transactions": {
            "type": "array",
            "uniqueItems": true
        },
        "version": {
            "type": "integer",
            "minimum": 0
        }
    },
    "required": [
        "blockSignature",
        "generatorPublicKey",
        "numberOfTransactions",
        "payloadHash",
        "payloadLength",
        "timestamp",
        "totalAmount",
        "totalFee",
        "reward",
        "transactions",
        "version"
    ]
};

Appendix B: Serialization Example

blockData = {
  "header": {
    "version": 3,
    "timestamp": 180,
    "height": 16,
    "previousBlockID": e194ce4e908c148ea4d11719cd40a016d07f393d31031ea150d7a8b7904a22d5,
    "transactionRoot": ,
    "generatorPublicKey": ed3b9fd50b188d35f5d2ea3fef05cb894363931c5ba50a5967c224ae5b16b339,
    "reward": 100000000n,
    "asset": {
      "maxHeightPreviouslyForged": 3,
      "maxHeightPrevoted": 10,
      "seedReveal": 8038ec83c421fa4844c5c65995cb2a66
    },
    "signature": bfc186b17132180057c8604640c276b85169fcaba72255bdc24f9220e620aa3e9731e6308a131f87097979e5696a7c38a25212bcb4779b099fab7df576b50207
  },
  "payload": []
}

blockMsg = { 
  0aac01: { 
      08: 03,
      10: b401,
      18: 10,
      2220: e194ce4e908c148ea4d11719cd40a016d07f393d31031ea150d7a8b7904a22d5,
      2a00: ,
      3220: ed3b9fd50b188d35f5d2ea3fef05cb894363931c5ba50a5967c224ae5b16b339,
      38: 80c2d72f,
      4216: {
        08: 03,
        10: 0a,
        1a10: 8038ec83c421fa4844c5c65995cb2a66
      },
      4a40: bfc186b17132180057c8604640c276b85169fcaba72255bdc24f9220e620aa3e9731e6308a131f87097979e5696a7c38a25212bcb4779b099fab7df576b50207
  } 
}

binaryMsg [175 bytes] = 0aac01080310b40118102220e194ce4e908c148ea4d11719cd40a016d07f393d31031ea150d7a8b7904a22d52a003220ed3b9fd50b188d35f5d2ea3fef05cb894363931c5ba50a5967c224ae5b16b3393880c2d72f42160803100a1a108038ec83c421fa4844c5c65995cb2a664a40bfc186b17132180057c8604640c276b85169fcaba72255bdc24f9220e620aa3e9731e6308a131f87097979e5696a7c38a25212bcb4779b099fab7df576b50207

blockID = e4448ec3d3366680ef65f0fbc60d49979361f3c4767f562bad2bb2841fe4702d

privateKey = 13f10fde4d5aa4298fe248707e7ec7392b854cdc1a655c2d67864e4117c4db2eed3b9fd50b188d35f5d2ea3fef05cb894363931c5ba50a5967c224ae5b16b339

alessandro.ricottone · March 19, 2020, 8:40am

The pull request has been merged and the LIP is now drafted:

github.com

LiskHQ/lips/blob/master/proposals/lip-0029.md

```
LIP: 0029
Title: Define schema and use generic serialization for blocks
Author: Alessandro Ricottone <alessandro.ricottone@lightcurve.io>
Discussions-To: https://research.lisk.com/t/define-schema-and-use-generic-serialization-for-blocks/209
Status: Active
Type: Informational
Created: 2020-02-18
Updated: 2021-09-24
Requires: 0027
```

## Abstract

This LIP defines how the generic serialization defined in [LIP 0027][lip-0027] is applied to blocks by specifying the appropriate JSON schema. We restructure the block header and introduce the block asset property, storing properties specific to the corresponding chain. We further specify the block-asset schema for the Lisk mainchain.

## Copyright

This LIP is licensed under the [Creative Commons Zero 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/).

This file has been truncated. show original

Define schema and use generic serialization for blocks

Abstract

Copyright

Motivation

Rationale

Specification

block Schema

Serialization

Deserialization

blockHeader Schema

Serialization

Deserialization

Block Signature Calculation

Block Signature Validation

Block ID

blockAsset Schema

Backwards Compatibility

Appendix A: Block Schema in the Current Protocol

Appendix B: Serialization Example

`block` Schema

`blockHeader` Schema

`blockAsset` Schema