Update Lisk SDK modular blockchain architecture

Hello everyone,

I would like to propose a new informational LIP. This LIP explains how Lisk SDK framework hooks works with the new module system.

I’m looking forward to your feedback.

LIP: <LIP number>
Title: Update Lisk SDK modular blockchain architecture
Author: Shusetsu Toda <shusetsu@lightcurve.io>
Type: Informational
Created: <YYYY-MM-DD>
Updated: <YYYY-MM-DD>

Abstract

The purpose of this LIP is to describe the updated Lisk SDK architecture including the block lifecycle and hooks with related new terminologies.

Copyright

This LIP is licensed under the Creative Commons Zero 1.0 Universal.

Rationale

Overview

The theoretical model for a blockchain is a replicated state-machine. In this model, a state-machine that transitions from one state to another based on its inputs (blocks in the context of blockchain) is replicated among all nodes of the respective blockchain network. Taking this model as a basis, in the Lisk SDK we distinguish between the following three domains:

  • Networking domain: Responsible for the communication of the peer-to-peer network.
  • Application domain: Responsible for transitioning the blockchain state with deterministic logic.
  • Consensus domain: Responsible for the replication of the same sequence of states among all nodes in the network. This is achieved by nodes in the network following a consensus protocol and utilizing the application and network domains.

In general, when executing a block B, the state-machine changes its state from a state S to a state S’. This one state transition can further be broken down into smaller intermediate state transitions, for instance, by considering that executing a block means executing all transactions in that block sequentially. In the Lisk SDK, we want to allow developers to define the whole state machine in a modular way by allowing them to easily define one part of the state and state transitions. This is achieved by implementing modules which can:

  • Define their own state as specified in the LIP 0040 - Define state model and state root.
  • Define arbitrary block asset data that is added to every block by block generators as defined in LIP 0055 - Update block schema and block processing.
  • Define specific logic that is executed with every block, before or after executing the transactions in the block (see LIP 0055). This is in particular helpful to process the module-specific block asset.
  • Define specific logic that is executed with every transaction, before or after executing the command referenced in the transaction (see LIP 0055). This is helpful to define logic that is executed for all commands in the blockchain, independent of the respective module.
  • Define commands that can be triggered by sending a transaction.

The state-machine in the Lisk SDK provides a certain interface to the consensus domain which is called during the block execution, and each module can define hooks which will be called from the state-machine during the execution.

New Terminologies

Command

A command is a group of state-transition logics triggered by a transaction identified by the module and command ID of the transaction, previously known as asset. Historically, the names “custom transaction” and “custom asset” were used. However, this was conceptually misleading because the parameters property of a transaction accepts any binary data, which is used as input to the state transition logic defined in a module, and it is not customizing the transaction nor the logic.

Cross-chain Command

With the introduction of cross-chain messages (CCMs), we introduce the concept of cross-chain commands. CCMs trigger the logic defined by the cross-chain command identified by module ID and cross-chain command ID.

Parameters

Parameters is a property of a transaction which is passed as an input to the command being triggered by the transaction, previously known as transaction asset. Because it is used as input parameters for the above command, calling it parameters is more suited than the transaction asset.

API

API is the interface for the module-to-module communication. Previously, the term reducer was used to describe this interface. However, the term reducer generally refers to functions that change the state or derive the new state from the input. For the interface, there are functions which just return a calculated value or information from the state without any mutation. Also, the concept of the interface was desired to be more generic.

Endpoint

Endpoint is the interface for a module to an external system through an RPC endpoint. Action was the term used to define the RPC endpoint handlers for the plugins or external system to call. Similar to the change from reducer to API, action was also not only mutating the state. Therefore, a more generic name was desired.

Application hooks

Application hooks are module methods which are called during the block execution. These hooks cannot mutate the block, but they can introduce state changes.

Block Generation hooks

Block generation hooks are module methods which are called only during the block generation. These hooks cannot introduce state changes, but they can add information to the block assets.

Execute

The terms apply, process, execute were used interchangeably to describe the execution of a block or a transaction. The term is unified to execute. A block is executed and hooks apply the state changes through the state machine.

Events

Events are on-chain data emitted during the processing of a block which add extra information about the execution of state transitions.
The protocol for events is defined in LIP 00XX.

Specifications

Module

Lisk SDK modules can define block generation hooks, state machine hooks and commands to add logic to the state machine. Each hook of the module will be called as described in the life cycle below (also described in LIP 0055).

Block executions

Block execution happens when the consensus receives a new block from a peer, and it follows the below life cycle.

Assets verification

The hook verifyAssets is only called before executing a block. If this stage fails, the block is considered invalid and will be rejected. In particular, the following hooks will not get executed. This hook is used for verification before any state changes. For example, at this stage, each module checks if the expected assets exist in the block.

In this hook, the state cannot be mutated and events cannot be emitted.

Before transactions execution

The hook beforeTransactionsExecute is the first hook that is triggered by the block.

In this hook, the state can be mutated and events can be emitted.

Transaction verification

The hook verifyTransaction is called for all the transactions within a block regardless of the command they trigger. This ensures that all transactions included in a block satisfy the verifications defined in this hook.

This hook is used also for transaction verification in the transaction pool to reject invalid transactions early before transmitting them to the network. For example, signature verification is done in this hook.

In this hook, the state cannot be mutated and events cannot be emitted.

Command verification

The hook Command.verify is called only for the command that is referenced by the moduleID and the commandID in the transaction. Similar to the verifyTransaction above, Command.verify will be called also in the transaction pool, and it is to ensure the verification defined in this hook is respected when the transactions are included in a block.

In this hook, the state cannot be mutated and events cannot be emitted.

Before command execution

The hook beforeCommandExecute is called for all the transactions within a block regardless of the command they trigger, similar to verifyTransaction.
If the hook fails during the execution, the transaction becomes invalid and the block containing this transaction will be invalid.

In this hook, the state can be mutated and events can be emitted.

Command execution

The hook Command.execute is triggered by a transaction identified by the moduleID and the commandID.
If the hook execution fails, the transaction that triggered this command is still valid, but the state changes applied during this hook are reverted.
Additionally, an event will be emitted that provides the information whether a command is executed successfully or failed.

In this hook, the state can be mutated and events can be emitted.

After command execution

The hook afterCommandExecute is called for all the transactions within a block regardless of the command they trigger.
If the hook fails during the execution, the transaction becomes invalid and the block containing this transaction will be invalid.

In this hook, the state can be mutated and events can be emitted.

After transaction execution

The hook afterTransactionsExecute is the last hook allowed to define state changes that are triggered by the block. Additionally, when defining the afterTransactionsExecute logic for a module, the transactions included in the block are available in that context and can be used in this logic. For example, this hook can be used to sum the fees of the transactions included in a block and transfer them to the block generator.

In this hook, the state can be mutated and events can be emitted.

Genesis block execution

Genesis block execution happens only once when starting the blockchain without any data, and all hooks for genesis block execution can mutate the state.

Genesis state initialization

The hook initGenesisState is called at the beginning of the genesis block execution. Each module must initialize their state using an associated block asset. It is recommended not to use APIs from other modules because their state might not be initialized yet depending on the order of the hook execution.

Genesis state finalization

The hook finalizeGenesisState is called at the end of genesis block execution. In this hook, it can be assumed that the state initialization via initGenesisState of every module is completed and therefore APIs from other modules can be used.

Block generation

Block generation hooks are only called during the block generation, and not during the block executions. The main purpose of the hooks is to add additional information into the block by block generators.

Header creation and result verification in the diagram below are not hooks and they are handled by the framework.

Also, state change cannot be applied in these hooks.

Assets insertion

The hook insertAssets is called at the very beginning of the block generation. The assets added during the execution of this hook can be used in all the execution hooks afterwards.

For example, the seedReveal property is added to the block asset in this hook by the Random module.

Backwards Compatibility

This LIP is informational. It does not introduce any protocol change.

Reference Implementation

TBD

2 Likes

I will spend some time on the terminology chosen here. Doing this my goal is to facilitate the onboarding of the average developer and allow them to get started as quickly as possible.

Cross-chain Command
I wonder if there is a difference in the definition (read “the code written”) of a cross-chain command ? It’s behaviour will be different of course but isn’t it just a command with parameters that will eventually perform a state transition ? I guess my question is why create a separate terminology for this kind of command ?

API
I believe using this name will induce a gigantic amount of confusion. For most developers an API is just an URI that can be called using HTTP. Most dev don’t read this word as the generic term is was initially intended.
I suggest a more lisk blockchain specific term like MMI (Module to Module Interface). It still contains the word interface which correctly defines what it is, but is more specific about what the interface is about.

Endpoint
Even though I opposed the term API I think endpoint can apply here, even using the MMI terminology. In everyone’s mind an endpoint is callable. So you could totally define your MMI and expose its methods to endpoints. It also unconsciously create this “2 layers” thinking of private and public methods which drives good code practices.

Assets verification
Since we now have commands and parameters it is confusing to see the term asset. To facilitate the adoption of the new terminology I would be more specific and refer to “block asset”.

** Genesis block execution**

Genesis block execution happens only once when starting the blockchain without any data

I think the chain state should store if a module has been initialized and not rely on the genesis block. My rationale is new modules can be introduced later in the blockchain as soft forks and still require state initialization. Lisk should support that by design and not expect developers to implement their own checks in the new module down the line. Happy to discuss the implementation details in the later stage.

Thank you for the feedback!

Cross-chain Command

Command is triggered by a transaction, while cross-chain command is triggered by Cross-chain update transaction. That’s why we want to distinguish those terminology.
Work flow is like

Cross chain update Transaction – triggers → command – triggers → cross-chain command

API

As you mentioned the generic term was intended. We wanted to avoid inventing a new terminology for this, and still convey what it does.
Some of the other options we considered was [Procedure, Subroutine, Function].
Do you have some suggestion which is generic but better conveys the concept?

Endpoint

Interface of endpoint also only receives the getter for the state, so it should be quite clear what it does.

Assets verification

After change of the terminology, asset is only used for block asset. Also, if we call asset block asset, we might need to start using block transaction since they are quite similar.

Genesis block execution

I agree that it would be nice to be able to initialize the module without genesis block, and I think it is already possible using beforeTransactionsExecute hook or block asset.
Main problem I see is how we inject the large off-chain data without using genesis block.

If we remove the genesis block handling, it will be protocol change, so that would be out of scope of this LIP. However, if you have good idea, please open a topic on this forum, and we can discuss further =)

Cross-chain Command
Is the interface the same as an inner-chain command ?

API
Consider the README file of a user made module.
It will have to mentions it API and its endpoints. Yet the endpoints are unrelated to the API. The endpoints will probably call some API methods. The endpoints, together, are another API, external this time. Do you see the problem I’m trying to point here ?

That’s why I believe we should use a more specific term to remove ambiguity about what we are talking about. MMI (Module to Module Interface) or just Module Interface still contains the generic term of interface, so the concept it describes is not new, but contains the specificity of the usage of this interface.

Endpoints
We are OK on this one

Assets verification
I understand your point. It’s ok even though I still believe that being more specific is better since its users can still shorten the name when the context is known, but the opposite is not possible without a common ground. This common ground being the documentation that references the terms defined here.

Genesis block execution
Let’s discuss this somewhere else :slight_smile:

Cross-chain Command
Is the interface the same as an inner-chain command?

The interface/input for the cross-chain command is different from the command because it needs to carry some information from Cross-chain update and cross chain message.

API
MMI (Module to Module Interface) or just Module Interface still contains the generic term of interface, so the concept it describes is not new, but contains the specificity of the usage of this interface.

I understand the problem, but also Module interface or MMI is also include duplicate information as a name like below (module is repeated).

const tokenModule = new TokenModule();
tokenModule.moduleInterface();
tokenModule.mmi();

and we also considered interface but interface is a reserved keyword in various language, so we avoided the term.
Any other good idea which doesn’t contain Module in the name?

Assets verification
I understand your point. It’s ok even though I still believe that being more specific is better since its users can still shorten the name when the context is known, but the opposite is not possible without a common ground. This common ground being the documentation that references the terms defined here.

New terminologies will be documented in the lisk-docs page along with other new/updated terminologies, so it should be all clear when it’s ready =)

Ok so we are left with the API. I’m sticking to my guns, we must find something else.

Ok, thinking out-loud here.
By your saying, it is a module-to-module communication interface.
This module object will hold different APIs, m2m, external, and maybe more in the future. So we have to namespace them. I don’t see why you want to keep the word interface. I mean what is the method name to get the endpoints ? It is an API so following your previous reasoning it should contain the word interface, correct ?

This interface is designed to be call while processing on-chain data right ? So why not on chain interface ?

const tokenModule = new TokenModule();
tokenModule.getEndpoints();
tokenModule.getOnChainInterface();
tokenModule.getOnChainActions(); // if you are willing to drop the word "interface", I like this one better

// yes, I like getters

What do you think ?

Yes, I understand API is not so good.
I don’t think we need to maintain the word interface
actions might be also fine, but some of the function just provides data (ie: not taking any action), so it might be nicer if we can encapsulate the idea as well.

Some of the idea we had was APIs / interface / method / action / intermoduleInterface / intermoduleAPI also taking inspiration from other projects

We are also brainstorming other terminologies in the SDK team now :smiley:

Proposition then : (clap your hands if you’ve got the reference)

const tokenModule = new TokenModule();
tokenModule.getOnChainService(); // mmi
tokenModule.getOffChainService(); // endpoints