Hello everyone,
In this thread, I want to propose a new LIP for the roadmap objective “Introduce universal serialization method”. This proposal defines how the generic serialization algorithm will be applied to transactions and specify the appropriate JSON schemas.
Looking forward to your feedback.
Here is the complete LIP draft:
LIP: <LIP number>
Title: Define schema and use generic serialization for transactions
Author: Iker Alustiza <iker@lightcurve.io>
Type: Standards Track
Created: <YYYY-MM-DD>
Updated: <YYYY-MM-DD>
Requires: LIP-00xx: 'A generic serialization method'
Abstract
This LIP defines a generic serialization for transactions by specifying the appropriate JSON schema. The proposed serialization method is applied in two steps, first to the asset property, where type-specific properties are defined, and then to the base transaction object. It will be used for signing but also for storing and transmitting transactions in the Lisk protocol. Further, this shall be the default serialization method for custom transactions created via the Lisk SDK.
Copyright
This LIP is licensed under the Creative Commons Zero 1.0 Universal.
Motivation
A standard way of serializing transactions is beneficial in several parts of the Lisk protocol:
- A generic and standard transaction serialization facilitates the task of managing and upgrading critical points of the protocol as signature and transaction ID generation or payload-hash computation.
- An optimized transaction serialization can improve storage efficiency.
- This serialization can also improve efficiency both for blocks and transactions propagation in the Lisk P2P protocol.
When a transaction object is serialized, also the asset property has to be serialized. In the current protocol, there exists a customized way to serialize the asset property for each transaction type. If a new transaction type is added, either to the Lisk mainchain or to a sidechain, then a new serialization method for the transaction specific to the new asset property has to be defined and implemented. This increases the barrier for creating sidechains with custom transactions. To simplify the development process, it is desirable to have a generic way to serialize the asset property which makes the mentioned specification and implementation unnecessary. This generic method may then be the default for custom transactions created with the Lisk SDK.
Rationale
This proposal follows the specifications in the LIP “A generic serialization method” to define a generic and deterministic way of serializing transactions in the Lisk protocol. This implies that JSON schemas have to be defined in order to perform the serialization-deserialization process.
As it is shown in the diagram below, every transaction object in the Lisk protocol, regardless of the type, has a common set of 6 properties. The data related to the specific transaction type is given in the asset
property. This architecture is also expected to be used for custom transactions created with the Lisk SDK. For simplicity and consistency, custom transactions should have these 6 properties, while all the new logic and data is defined with respect to the custom asset
property.
For this reason, this LIP proposes to divide the serialization-deserialization process. First it defines a serialization-deserialization process for the whole transaction object with the baseTransaction
JSON schema where the asset
property is assumed to be of type bytes
. Secondly, it defines the serialization-deserialization process for the asset
property with the corresponding asset
property JSON schema depending on the type of the transaction (specified in the type
property). There are 7 different asset
property schemas, one for each of the transaction types in the Lisk protocol. In order to simplify the creation of custom transactions with the SDK and to make the validation process less error prone, every property defined inside a given asset
schema is required.
Specification
Assuming that the JSON schemas for the baseTransaction
and for the asset
properties are provided according to the specifications in the LIP “A generic serialization method”, the serialization and deserialization of a transaction object are done as follows.
Serialization
Consider a data structure trsData
representing a valid (signed or unsigned) transaction object to be serialized. The serialization procedure is done in 3 steps:
- The
trsData.asset
property is serialized to bytes according to the correspondingasset
property schema specified by thetrsData.type
property (see below). - The resulting bytes are inserted into the
trsData.asset
property replacing the original value. - The transaction object from step 2 is serialized according to the
baseTransaction
schema.
Deserialization
Consider a binary message trsMsg
to be deserialized. The deserialization procedure is done in 3 steps:
- The serialized bytes of
trsMsg
are deserialized according to thebaseTransaction
schema to obtaintrsData
. - The bytes value of the
trsData.asset
property is deserialized according to the correspondingasset
property schema specified by thetrsData.type
property. - The deserialized transaction object is the object from step 1,
trsData
, where the value oftrsData.asset
is the output of the step 2.
Transaction signature calculation
Consider a data structure unsignedTrsData
representing a valid transaction object in which the signatures
array is initialized to the default value (an empty array). The signature of the object is calculated as follows:
-
unsignedTrsData
is serialized using the method explained above. In particular, the emptysignatures
array is not part of the serialized data (see below). - The transaction signature is calculated by signing the binary message from step 1.
- The signature bytes from step 2 are inserted into the corresponding element of the
unsignedTrsData.signatures
array.
Transaction signature validation
Consider a binary message trsMsg
representing a serialized transaction object. The signature of the object is validated as follows:
-
trsMsg
is deserialized to an object,trsData
, as explained above in Deserialization. - The signatures in
trsData.signatures
are removed fromtrsData
, but kept for signature verification purposes. The arraytrsData.signatures
is set to its default value. - The transaction object,
trsData
, from step 2 is serialized using the method explained above. - The signatures saved in step 2 are verified against the binary message from step 3 and the associated public keys.
Transaction ID
Given a data structure, signedTrsData
, representing a transaction object with the corresponding signatures present in the signatures
property, the transaction ID is calculated as follows:
-
signedTrsData
is serialized as explained above. - The transaction ID is calculated as the SHA-256 hash of the binary message from step 1.
baseTransaction
schema
The schema baseTransaction
contains 6 properties:
-
type
: An integer identifying the type of the transaction. It specifies the JSON schema to be used to serialize-deserialize theasset
property of the transaction. -
nonce
: An integer which is unique for each transaction from the account corresponding to thesenderPublicKey.
-
fee
: An integer that specifies the fee in Beddows to be spent by the transaction. -
senderPublicKey
: The public key of the account issuing the transaction. A valid public key is 32 bytes long. -
asset
: The serialized asset. -
signatures
: An array with the signatures of the transaction. Note that this property is an array which implies that in case ofsignatures
being initialized to its default value, it does not appear in the binary message. The elements of this array are bytes, either of length 64, or of length 0.
baseTransaction = {
"type": "object",
"properties": {
"type": {
"dataType": "uint32",
"fieldNumber": 1
},
"nonce": {
"dataType": "uint64",
"fieldNumber": 2
},
"fee": {
"dataType": "uint64",
"fieldNumber": 3
},
"senderPublicKey": {
"dataType": "bytes",
"fieldNumber": 4
},
"asset": {
"dataType": "bytes",
"fieldNumber": 5
},
"signatures": {
"type": "array",
"items": {
"dataType": "bytes",
},
"fieldNumber": 6
},
},
"required": [
"type",
"nonce",
"fee",
"senderPublicKey",
"asset"
]
}
asset
property schema
There is a unique asset
property schema per transaction type, implied by the value of the type
property. All the properties defined in any asset
schema are required.
Balance transfer
The schema balanceTransferAsset
contains 3 properties:
-
amount
: The amount in Beddows to be transferred to the account specified inrecipientAddress
. -
recipientAddress
: The address of the recipient ofamount
. This property is 20 bytes long. -
data
: A string with a maximum length of 64 bytes.
balanceTransferAsset = {
"type": "object",
"properties": {
"amount": {
"dataType": "uint64",
"fieldNumber": 1
},
"recipientAddress": {
"dataType": "bytes",
"fieldNumber": 2
},
"data": {
"dataType": "string",
"fieldNumber": 3
},
},
"required": [
"amount",
"recipientAddress",
"data"
]
}
Vote
The voteAsset
schema contains an array of objects. Each of these objects contains the following properties:
-
delegateAddress
: The address of the voted delegate. An address is 20 bytes long. -
amount
: The amount, in Beddows, voted for the delegate.
voteAsset = {
"type": "object",
"properties": {
"votes": {
"type": "array",
"items": {
"type": "object",
"properties": {
"delegateAddress": {
"dataType": "bytes",
"fieldNumber": 1
},
"amount": {
"dataType": "sint64",
"fieldNumber": 2
}
},
"required": [
"delegateAddress",
"amount"
]
},
"fieldNumber": 1
}
},
"required": [
"votes"
]
}
Delegate registration
The schema delegateRegAsset
contains the username
property. This property specifies the name of the registered delegate and is a string with a minimum length of 1 character and a maximum length of 20 characters.
delegateRegAsset = {
"type": "object",
"properties": {
"username": {
"dataType": "string",
"fieldNumber": 1
},
},
"required": [
"username"
]
}
Multisignature registration
The schema multisigRegAsset
contains 3 properties:
-
numberOfSignatures
: The number of private keys that must sign a transaction. -
mandatoryKeys
: An array of public keys. The corresponding private keys necessarily have to sign the transaction. A valid public key is 32 bytes long. -
optionalKeys
: An array of public keys. A valid public key is 32 bytes long.
multisigRegAsset = {
"type": "object",
"properties": {
"numberOfSignatures": {
"dataType": "uint32",
"fieldNumber": 1
},
"mandatoryKeys": {
"type": "array",
"items": {
"dataType" : "bytes",
},
"fieldNumber": 2
},
"optionalKeys": {
"type": "array",
"items": {
"dataType": "bytes",
},
"fieldNumber": 3
},
},
"required": [
"numberOfSignatures",
"mandatoryKeys",
"optionalKeys"
]
}
Unlock vote
The unlockVoteAsset
schema contains an array of objects. Each of these objects contains the following properties:
-
delegateAddress
: The address of the unvoted delegate. An address is 20 bytes long. -
amount
: The amount, in Beddows, unvoted in a previous vote transaction. -
unvoteHeight
: The height at which the transaction to unvote the delegate has been included in the blockchain.
unlockVoteAsset = {
"type": "object",
"properties": {
"unlockObjects": {
"type": "array",
"items": {
"type": "object",
"properties": {
"delegateAddress": {
"dataType": "bytes",
"fieldNumber": 1
},
"amount": {
"dataType": "uint64",
"fieldNumber": 2
},
"unvoteHeight": {
"dataType": "uint32",
"fieldNumber": 3
}
},
"required": [
"delegateAddress",
"amount",
"unvoteHeight"
]
}
},
"fieldNumber": 1
},
"required": [
"unlockObjects",
]
}
PoM
The pomAsset
schema contains two properties: header1
and header2
. Each of the properties contain the serialized version of a signed block header of the Lisk blockchain. The specification to generate the serialized block headers is explained in the LIP "Define schema and use generic serialization for blocks proposal.
pomAsset = {
"type": "object",
"properties:" {
"header1": {
"dataType": "bytes",
"fieldNumber": 1
},
"header2": {
"dataType": "bytes",
"fieldNumber": 2
},
},
"required": [
"header1",
"header2",
]
}
Reclaim
The schema reclaimAsset
contains the amount
property. This property accounts for the balance in Beddows to be reclaimed.
reclaimAsset = {
"type": "object",
"properties": {
"amount": {
"dataType": "uint64",
"fieldNumber": 1
},
},
"required": [
"amount",
]
}
Backwards Compatibility
This proposal introduces a hard fork in the network. Since the transaction serialization is computed in a different way, this proposal impacts the way in which transactions are signed, stored and transmitted in the Lisk protocol. This also affects how transaction IDs are computed.
Appendix: Serialization Example of Balance Transfer Transaction
Transaction object to serialize:
myTrs = {
"type": 21,
"nonce": 15701796739323706378n,
"fee": 3156364651n,
"senderPublicKey":
0x8f057d088a585d938c20d63e430a068d4cea384e588aa0b758c68fca21644dbc,
"asset": {
"amount": 9375038900n,
"recipientAddress": 0xf214d75bbc4b2ea89e433f3a45af803725416ec3,
"data": 'Odi et amo. Quare id faciam, fortasse requiris.'
},
"signatures": [ 0x204514eb1152355799ece36d17037e5feb4871472c60763bdafe67eb6a38bec632a8e2e62f84a32cf764342a4708a65fbad194e37feec03940f0ff84d3df2a05,
0x0b6730e5898ca56fe0dc1c73de9363f6fc8b335592ef10725a8463bff101a4943e60311f0b1a439a2c9e02cca1379b80a822f4ec48cf212bff1f1c757e92ec02 ]
}
Binary message (264 bytes):
0x0815108af89ab6959efff3d90118eb9a89e10b22208f057d088a585d938c20d63e430a068d4cea384e588aa0b758c68fca21644dbc2a4d08b4fbaef6221214f214d75bbc4b2ea89e433f3a45af803725416ec31a2f4f646920657420616d6f2e2051756172652069642066616369616d2c20666f7274617373652072657175697269732e3240204514eb1152355799ece36d17037e5feb4871472c60763bdafe67eb6a38bec632a8e2e62f84a32cf764342a4708a65fbad194e37feec03940f0ff84d3df2a0532400b6730e5898ca56fe0dc1c73de9363f6fc8b335592ef10725a8463bff101a4943e60311f0b1a439a2c9e02cca1379b80a822f4ec48cf212bff1f1c757e92ec02
Transaction ID: 0x4bef5fa471e8828b2fc8785a59abb222b9e1a9249dad07e5c5010f89aefb43cd
Private keys:
0x2b2d2ac6f698a9cef440003c780bbcb5fe0e0a640d4633e43723b95c59d38d69d220071195b2307db7c722b6ec32888eaae92ab8206594aad13a0ec6e65d80eb
0x5e747e595407e689fa2cf5e5b720b85b88fb5c701d789332f4fef198070180ad94bedc715354abe8f8bf6966782726fff1dbb275aa89f81cf5305a0a2a63cf26