Ethereum mainstream client: Geth overall architecture

This article is the first in the Geth source code series. Through this series, we will build a framework to study Geth implementation, and developers can conduct research on some of their interests based on this framework. There are six articles in this series. In the first article, the design architecture of the execution layer client Geth and the startup process of the Geth node will be studied. The Geth code is updated very quickly, and the code you see in the future may be different, but the overall design is generally consistent, and new code can be read with the same idea.

01\Ethereum Client

Before Ethereum upgraded The Merge, Ethereum had only one client, which was responsible for the execution of transactions, and was also responsible for the consensus of the blockchain to ensure that the blockchain generates new blocks in a certain order. After The Merge upgrade, the Ethereum client is divided into an execution layer and a consensus layer. The execution layer is responsible for the execution of transactions, state and data maintenance, while the consensus layer is responsible for the implementation of consensus functions. The execution layer and consensus layer communicate through the API. The execution layer and the consensus layer have their own specifications. Clients can use different languages ​​to implement them, but they must comply with the corresponding specifications. Geth is an implementation of the execution layer client. The current mainstream execution layer and consensus layer clients have the following implementations:

Execution layer

• Geth: Maintained by a team directly funded by the Ethereum Foundation, developed in the Go language, it is recognized as the most stable and proven client.
• Nethermind: Developed and maintained by the Netherlands team, developed in C# language, early funded by the Ethereum Foundation and Gitcoin community
• Besu: Originally developed by ConsenSys' PegaSys team, it is now a Hyperledger community project, developed in Java language
• Eragon: Developed and maintained by the Eragon team, and funded by the Ethereum Foundation and BNB Chain. Forked from Geth in 2017, with the goal of improving synchronization speed and disk efficiency
• Reth: developed by Paradigm, the development language is Rust, which emphasizes modularity and high performance. It is now approaching maturity and can be used in production environments.

Consensus layer

• Prysm: Maintained by Prysmatic Labs, it is one of the earliest consensus layer clients of Ethereum. It is developed in Go language, focusing on availability and security. It was funded by the Ethereum Foundation in the early stage.
• Lighthouse: Maintained by the Sigma Prime team, developed in the Rust language, focusing on high performance and enterprise-level security, suitable for high-load scenarios
• Teku: Early rise was developed by ConsenSys' PegaSys team, and later became part of the Hyperledger Besu community, developed in the Java language
• Nimbus: Developed and maintained by the Status Network team, developed in the Nim language, optimized for resource-constrained devices (such as mobile phones, IoT devices), with the goal of achieving lightweight operation in embedded systems

02\Introduction to the execution layer

The Ethereum execution layer can be regarded as a transaction-driven state machine. The most basic function of the execution layer is to update state data through EVM execution transactions. In addition to transaction execution, there are also functions such as saving and verifying block and status data, running the p2p network and maintaining transaction pools.

Transactions are generated by users (or programs) in the format defined by the Ethereum execution layer specification. Users need to sign the transaction. If the transaction is legal (Nonce continuous, the signature is correct, the gas fee is sufficient, and the business logic is correct), the transaction will eventually be executed by the EVM, thereby updating the status of the Ethereum network. The status here refers to a collection of data structures, data and databases, including external account addresses, contract addresses, address balances, and codes and data.

The execution layer is responsible for executing transactions and maintaining the status after transaction execution, and the consensus layer is responsible for selecting which transactions to execute. EVM is a state transition function in this state machine. The input of the function will come from multiple places, which may come from the latest block information provided by the consensus layer, or from blocks downloaded by the p2p network.

The consensus layer and the execution layer communicate through the Engine API, which is the only way to communicate between the execution layer and the consensus layer. If the consensus layer obtains the block output right, the execution layer will use the Engine API to make the execution layer produce new blocks. If the block output right is not obtained, the latest block will be synchronized for the execution layer to verify and execute, thus maintaining consensus with the entire Ethereum network.

The execution layer can be logically divided into 6 parts:

• EVM: Responsible for executing transactions, and transaction execution is also the only way to modify the number of states.
• Storage: Responsible for the storage of state and blocks and other data
• Transaction pool: used for transactions submitted by users, temporarily stored, and will be propagated between different nodes through the p2p network
• p2p network: used to discover nodes, synchronize transactions, download blocks and other functions
• RPC service: provides the ability to access nodes, such as users send transactions to nodes, interactions between consensus layer and execution layer
• BlockChain: Responsible for managing blockchain data on Ethereum

The following figure shows the key processes of the execution layer, and the functions of each section:

For the execution layer (only Full Node is discussed here for the time being), there are three key processes:

• If it is a new node joining Ethereum, you need to synchronize the block and status data from other nodes through the p2p network. If it is Full Sync, you will download the blocks one by one from the Genesis block, verify the blocks and rebuild the status database through EVM. If it is Snap Sync, skip the verification process of all blocks and directly download the latest checkpoint status data and future block data
• If it is a node that has been synchronized to the latest state, it will continue to obtain the currently latest output block from the consensus layer through the Engine API, and verify the block. Then, perform all transactions in the block through the EVM to update the state database and write the block to the local chain.
• If the node that has been synchronized to the latest state and the consensus layer has obtained the block output right, it will drive the execution layer to produce the latest block through the Engine API. The execution layer obtains transactions from the transaction pool and executes them, and then assembles them into blocks and passes them to the consensus layer through the Engine API. The consensus layer broadcasts the block to the consensus layer p2p network

03\Source code structure

The code structure of go-ethereum is very large, but many of the codes are auxiliary code and unit testing. When studying Geth source code, you only need to pay attention to the core implementation of the protocol. The functions of each module are as follows. We need to focus on core, eth, ethdb, node, p2p, rlp, trie & tryb and other modules:

• accounts: manage Ethereum accounts, including the generation of public and private key pairs, signature verification, address derivation, etc.
• beacon: handles the interactive logic with the Ethereum beacon chain, and supports the merge function of the Proof of Stake (PoS) consensus.
• build: build scripts and compilation configurations (such as Dockerfile, cross-platform compilation support)
• cmd: Command line tool entry, containing multiple subcommands
• common: general tool classes, such as byte processing, address format conversion, mathematical functions
• consensus: define consensus engine, including previous proof of work (Ethash), stand-alone proof of stake (Clique), Beacon engine, etc.
• console: Provides an interactive JavaScript console, allowing users to interact directly with Ethereum nodes through the command line (such as calling Web3 API, managing accounts, and querying blockchain data)
• core: blockchain core logic, life cycle management, state machine, Gas computing, etc. for processing block/transactions, etc.
• crypto: Implementation of encryption algorithm, including elliptic curve (secp256k1), hash (Keccak-256), signature verification
• docs: Documentation (such as design specifications, API descriptions)
• eth: The complete implementation of the Ethereum network protocol, including node services, block synchronization (such as fast synchronization, archive mode), transaction broadcasting, etc.
• ethclient: implements the Ethereum client library, encapsulates the JSON-RPC interface, and allows Go developers to interact with Ethereum nodes (such as querying blocks, sending transactions, and deploying contracts)
• ethdb: database abstraction layer, supports LevelDB, Pebble, in-memory database, etc., and stores blockchain data (blocks, states, transactions)
• ethstats: Collect and report the node's operating status to statistics service, used to monitor network health status
• event: Implement event subscription and publishing mechanisms, and support asynchronous communication between modules within nodes (such as new block arrival and transaction pool update)
• graphql: provides GraphQL interface and supports complex queries (replaces some JSON-RPC functions)
• internal: code that restricts external access by internal tools or code
• log: log system, supports hierarchical log output and context logging
• Mertrics: Performance Metrics Collection (Prometheus Support)
• miner: Mining related logic, generating new blocks and packaging transactions (in PoW scenario)
• node: node service management, integrating the startup and configuration of p2p, RPC, database and other modules
• p2p: Point-to-point network protocol implementation, supporting node discovery, data transmission, and encrypted communication
• params: define Ethereum network parameters (main network, test network, Genesis block configuration)
• rlp: implements Ethereum-specific data serialization protocol RLP (Recursive Length Prefix), used to encode/decode blocks, transactions and other data structures
• rpc: implements JSON-RPC and IPC interfaces for external programs to interact with nodes
• Signer: Transaction signature management (hardware wallet integration)
• tests: Integration testing and state testing, verify protocol compatibility
• trie & trialb: The implementation of Merkle Patricia Trie, used to efficiently store and manage account status and contract storage

04\Execution layer module division

There are two forms of external access to the Geth node, one is through RPC and the other is through Console. RPC is suitable for external users to use, and Console is suitable for node administrators. But whether it is through RPC or Console, it uses capabilities that have been encapsulated internally, and these capabilities are built in a layered manner.

The outermost layer is the API's capabilities for external access nodes. Engine API is used to communicate between the execution layer and the consensus layer. Eth API is used to external users or programs to send transactions and obtain block information. Net API is used to obtain the status of the p2p network, etc. For example, if a user sends a transaction through the API, the transaction will eventually be submitted to the transaction pool and managed through the transaction pool. For example, if the user needs to obtain a block data, he needs to call the database's ability to obtain the corresponding block.

At the next level of the API, the implementation of core functions, including transaction pooling, transaction packaging, output blocks, blocks and state synchronization, etc. These functions need to rely on lower-level capabilities, such as the synchronization of transaction pools, blocks and states requires the ability to rely on the p2p network, the generation of blocks and blocks synchronized from other nodes need to be verified before they can be written to the local database, which requires the ability to rely on the EVM and data storage.

Execution layer core data structure

Ethereum

The Ethereum structure in eth/backend.go is an abstraction of the entire Ethereum protocol, basically including the main components in Ethereum, but EVM is an exception. It will be instantiated every time a transaction is processed, and does not need to be initialized with the entire node. The following Ethereum refers to this structure:

type Ethereum struct { // Ethereum configuration, including chain configuration config *ethconfig.Config // Transaction pool, after the user's transaction is submitted, go to the transaction pool txPool *txpool.TxPool // Used to track and manage local transactions localTxTracker *locals.TxTracker // Blockchain structure blockchain *core.BlockChain // It is a core component of the network layer of the Ethereum node, responsible for handling all communications with other nodes, including block synchronization, transaction broadcasting and reception, and managing peer node connection handler *handler // Responsible for node discovery and node source management discmix *enode.FairMix // Responsible for persistent storage of blockchain data chainDb ethdb.Database // Responsible for publishing and subscribing to various internal events *event.TypeMux // Consensus engine consensus.Engine // Manage user accounts and keys accountManager *accounts.Manager // Manage log filters and block filters filterMaps *filtermaps.FilterMaps // Used to safely close filterMaps channels to ensure that resources are cleaned correctly when nodes are closed close closeFilterMaps chan chan struct{} // Provide backend support for RPC API APIBackend *EthAPIBackend // Under PoS, collaborate with the consensus engine to verify block miner *miner.Miner // The minimum gas price accepted by nodes gasPrice *big.Int // Network ID networkID uint64 // Provide network-related RPC services, allowing network status to be queried through RPC netRPCService *ethapi.NetAPI // Manage P2P network connections, handle node discovery and connection establishment, and provide underlying network transmission functions p2pServer *p2p.Server // Protect concurrent access of variable fields lock sync.RWMutex // Track whether the node is closed normally, help restore shutdownTracker after abnormal shutdown *shutdowncheck.ShutdownTracker }

Node

Node in node/node.go is another core data structure, which acts as a container, is responsible for managing and coordinating the operation of various services. In the following structure, you need to pay attention to the lifecycles field, which is used by Lifecycle to manage the lifecycle of internal functions. For example, the above Ethereum abstraction needs to rely on Node to start and is registered in lifecycles. This can separate specific functions from the abstraction of nodes and improve the scalability of the entire architecture. This Node needs to be distinguished from the Node in devp2p.

type Node struct { eventmux *event.TypeMux config *Config // Account manager, responsible for managing wallets and accounts accman *accounts.Manager log log.Logger keyDir string keyDirTemp bool dirLock *flock.Flock stop chan struct{} // p2p network instance server *p2p.Server startStopLock sync.Mutex // Track node lifecycle status (initialized, running, closed) state int lock sync.Mutex // All registered backend, service and auxiliary services lifecycles []Lifecycle // API currently provided List rpcAPIs []rpc.API // Different access methods provided for RPC http *httpServer ws *httpServer httpAuth *httpServer wsAuth *httpServer ipc *ipcServer inprocHandler *rpc.Server databases map[*closeTrackingDB]struct{} }

If we look at the execution layer of Ethereum from an abstract dimension, Ethereum, as a world computer, needs to include three parts, network, computing and storage, then the components corresponding to these three parts in the execution layer of Ethereum are:

• Network: devp2p
• Calculation: EVM
• Storage: ethdb

devp2p

Ethereum is essentially a distributed system, and each node is connected to other nodes through a p2p network. The implementation of the p2p network protocol in Ethereum is devp2p.

devp2p has two core functions. One is node discovery, which allows nodes to establish contact with other nodes when accessing the network; the other is data transmission service. After establishing contact with other nodes, you can want to exchange data.

The Node structure in p2p/enode/node.go represents a node in the p2p network. The enr.Record structure stores key-value pairs of node details, including identity information (signature algorithms used by node identity, public key), network information (IP address, port number), supported protocol information (such as supporting eth/68 and snap protocols) and other custom information. These information are encoded through RLP. The specific specification is defined in eip-778:

type Node struct { // Node record, containing various attributes of the node r enr.Record // Unique identifier of the node, 32 byte length id ID // hostname Track the node's DNS name hostname string // Node's IP address ip netip.Addr // UDP port udp uint16 // TCP port tcp uint16 }// enr.Recordtype Record struct { // Serial number seq uint64 // Signature signature []byte // RLP encoded record raw []byte // Sorting list of all key-value pairs []pair }

The Table structure in p2p/discover/table.go is the core data structure of the node discovery protocol in devp2p. It implements a distributed hash table similar to Kademlia, which is used to maintain and manage node information in the network.

printf("type Table struct { mutex sync.Mutex // Index known nodes by distance buckets [nBuckets]*bucket // Boot node nursery []*enode.Node rand reseedingRandom ips netutil.DistinctNetSet revalidation tableRevalidation // Database of known nodes db *enode.DB net transport cfg Config log log.Logger // Periodically handle various events in the network refreshReq chan chan struct{} revalResponseCh chan revalidationResponse addNodeCh chan addNodeOp addNodeHandled chan bool trackRequestCh chan trackRequestOp initDone chan struct{} closeReq chan struct{} closed chan struct{} // Add and remove the interface nodeAddedHook func(*bucket, *tableNode) nodeRemovedHook func(*bucket, *tableNode)} world!");

ethdb

ethdb completes the abstraction of Ethereum data storage and provides a unified storage interface. The underlying specific database can be leveldb, pebble or other databases. There can be many extensions as long as it remains unified at the interface level.

Some data (such as block data) can be read and written directly to the underlying database through the ethdb interface. Other data storage interfaces are based on the ethdb. For example, a large part of the data in the database is state data, which will be organized into an MPT structure. The corresponding implementation in Geth is trie. During the operation of the node, the trie data will generate many intermediate states. These data cannot be directly called ethdb for reading and writing. Triedb is needed to manage these data and intermediate states, and finally it is persisted through ethdb.

The interface for defining the reading and writing capabilities of the underlying database in ethdb/database.go, but does not include specific implementations. The specific implementations will be implemented by different databases themselves. For example, leveldb or pebble database. Two layers of data reading and writing interfaces are defined in Database, where the KeyValueStore interface is used to store active and frequently changing data, such as the latest blocks, status, etc. AncientStore is used to process historical block data, which rarely changes once written.

// Top-level interface of the database type Database interface { KeyValueStore AncientStore}// KV data read and write interface type KeyValueStore interface { KeyValueReader KeyValueWriter KeyValueStater KeyValueRangeDeleter Batcher Iteratee Compacter io.Closer}// Interface type AncientStore interface { AncientReader AncientWriter AncientStater io.Closer}

EVM

EVM is a state transition function of the state machine of Ethereum. All state data updates can only be done through EVM. The p2p network can accept transaction and block information. After this information is processed by EVM, it will become part of the state database. EVM blocks the differences in underlying hardware, allowing programs to be executed on EVMs on different platforms to get consistent results. This is a very mature design method, and JVM is also similar in Java language.

The implementation of EVM has three main components. The EVM structure in core/vm/evm.go defines the overall structure and dependencies of EVM, including execution context, state database dependencies, etc.; the EVMInterpreter structure in core/vm/interpreter.go defines the implementation of the interpreter and is responsible for executing the EVM bytecode; the Contract structure in core/vm/contract.go encapsulates the specific parameters of the contract call, including the caller, contract code, input, etc., and defines all current opcodes in core/vm/opcodes.go:

// EVMtype EVM struct { // Block context, containing block-related information Context BlockContext // Transaction context, containing transaction-related information TxContext // State database, used to access and modify account state StateDB StateDB // Current call depth int // Chain configuration parameters chainConfig *params.ChainConfig chainRules params.Rules // EVM configuration Config Config // Bytecode interpreter *EVMInterpreter // Flags for abort atomic.Bool callGasTemp uint64 // Precompiled contract mapping precompiles map[common.Address]PrecompiledContract jumpDests map[common.Hash]bitvec }type EVMInterpreter struct { // Point to the EVM instance to which it belongs evm *EVM // Opcode jump table table *JumpTable // Keccak256 hasher instance, sharing hasher crypto.KeccakState // Keccak256 hash result buffer hasherBuf common.Hash // Whether it is read-only mode, state modification is not allowed in read-only mode readOnly bool // The last CALL return data is used for subsequent reuse returnData []byte }type Contract struct { // Caller address caller common.Address // Contract address address common.Address jumpdests map[common.Hash]bitvec analysis bitvec // Contract bytecode []byte // Code hash CodeHash common.Hash // Call Input []byte // Whether to deploy IsDeployment bool for the contract // Whether to use system call IsSystemCall bool // Gas quantity available Gas uint64 // Call the accompanying number of ETH value *uint256.Int }

Other module implementations

The functions of the execution layer are implemented in a layered manner, and other modules and functions are built on the basis of these three core components. Here are a few core modules.

Under eth/protocols, there is the implementation of the current Ethereum p2p network subprotocol. There are eth/68 and snap subprotocols, which are built on devp2p.

eth/68 is the core protocol of Ethereum. The protocol name is eth, and 68 is its version number. On the basis of this protocol, functions such as transaction pool (TxPool), block synchronization (Downloader), and transaction synchronization (Fetcher) are implemented. The snap protocol is used to quickly synchronize blocks and status data when a new node joins the network, which can greatly reduce the time for the new node to start.

ethdb provides the reading and writing capabilities of the underlying database. Since there are many complex data structures in the Ethereum protocol, the management of these data cannot be achieved directly through ethdb, rawdb and statedb are implemented on ethdb to manage block and state data respectively.

EVM runs through all main processes, whether it is block construction or block verification, transactions need to be executed using EVM.

05\Geth node startup process

The startup of Geth will be divided into two stages. The first stage will initialize the components and resources required by the node, and the second node will officially start the node and then serve externally.

Node initialization

When starting a getth node, the following code is involved:

The initialization of each module is as follows:

• cmd/geth/main.go: geth node startup entrance

• cmd/geth/config.go (makeFullNode): load the configuration, initialize the node

• node/node.go: Initialize the core container of Ethereum nodes

1. node.rpcstack.go: Initialize the RPC module
2. accounts.manager.go: Initialize accountManager

• eth/backend.go: Initialize the Ethereum instance

1. node/node.go OpenDatabaseWithFreezer: Initialize chaindb
2. eth/ethconfig/config.go: Initialize the consensus engine instance (the consensus engine here does not really participate in the consensus, it only verifies the results of the consensus layer and processes the validator's withdrawal request)
3. core/blockchain.go: Initialize blockchain
4. core/filterMaps.go: Initialize filtermaps
5. core/txpool/blobpool/blobpool.go: Initialize the blob transaction pool
6. core/txpool/legacypool/legacypool.go: Initialize the normal transaction pool
7. cord/txpool/locals/tx_tracker.go: Local transaction tracking (local transaction tracking needs to be enabled, local transactions will be processed with higher priority)
8. eth/handler.go: Handler instance of initialization protocol
9. miner/miner.go: module that instantiates transaction packaging (original mining module)
10. eth/api_backend.go: Instantiate the RPC service
11. eth/gasprice/gasprice.go: Instantiate gas price query service
12. internal/ethapi/api.go: Instantiate the P2P network RPC API
13. node/node.go(RegisterAPIs): Register RPC API
14. node/node.go(RegisterProtocols): Register Ptotocols for p2p
15. node/node.go(RegisterLifecycle): Register the life cycle of each component

• cmd/utils/flags.go(RegisterFilterAPI): Register Filter RPC API
• cmd/utils/flags.go(RegisterGraphQLService): Register GraphQL RPC API (if configured)
• cmd/utils/flags.go(RegisterEthStatsService): Register the EthStats RPC API (if configured)
• eth/catalyst/api.go: Register Engine API

The initialization of the node will be completed in makeFullNode in cmd/geth/config.go, and the following three modules will be initialized.

In the first step, the Node structure in node/node.go is the entire node container. All functions need to be run in this container. The second step will initialize the Ethereum structure, including the implementation of various core functions of Ethereum. Ethereum also needs to be registered in Node. The third step is to register the Engine API into Node.

Node initialization is to create a Node instance, and then initialize p2p server, account management, and http to external protocol ports exposed to the outside.

The initialization of Ethereum will be much more complicated, and most core functions are initialized here. First, ethdb will be initialized, and the chain configuration will be loaded from the storage, and then a consensus engine will be created. The consensus engine here will not perform consensus operations, but will only verify the results returned by the consensus layer. If a withdrawal request occurs in the consensus layer, the actual withdrawal operation will be completed here. Then the Block Chain structure and transaction pool are initialized.

After all these are completed, the handler will be initialized. The handler is the processing portal for all p2p network requests, including transaction synchronization, block download, etc. It is a key component for Ethereum to achieve decentralized operation. After all these are completed, some sub-protocols implemented on the basis of devp2p, such as eth/68, snap, etc., will be registered in the Node container. Finally, Ethereum will be registered in the Node container as a lifecycle, and Ethereum will be initialized.

Finally, the initialization of the Engine API is relatively simple, just registering the Engine API into Node. At this point, the node initialization is all completed.

Node start

After completing the initialization of the node, you need to start the node. The process of starting the node is relatively simple. You only need to start all registered RPC services and Lifecycle, and the entire node can provide services to the outside world.

06\Summary

Before deeply understanding the implementation of the Ethereum execution layer, we need to have a holistic understanding of Ethereum. We can regard Ethereum as a transaction-driven state machine. The execution layer is responsible for the execution of transactions and state changes, while the consensus layer is responsible for driving the execution layer to run, including allowing the execution layer to produce blocks, determine the order of transactions, vote for blocks, and make the blocks final. Since this state machine is decentralized, it is necessary to communicate with other nodes through the p2p network to jointly maintain the consistency of state data.

The execution layer is not responsible for determining the order of transactions, but is only responsible for executing transactions and recording the state changes after transaction execution. There are two forms of records here. One is to record all state changes in the form of blocks, and the other is to record the current state in the database. At the same time, the execution layer is also the entrance to transactions, and transactions that have not been packaged into the block are stored through the transaction pool. If other nodes need to obtain block, state and transaction data, the execution layer will send this information through the p2p network.

For the execution layer, there are three core modules: computing, storage, and networking. Compute the implementation of the corresponding EVM, storage corresponds to the implementation of ethdb, and network complies with the implementation of devp2p. With such a holistic understanding, you can deeply understand each submodule without getting lost in specific details.

07\Ref

[1]https://ethereum.org/zh/what-is-ethereum/

[2]https://epf.wiki/#/wiki/protocol/architecture

[3]https://clientdiversity.org/#distribution

[4]https://github.com/ethereum/devp2p

[5]https://github.com/ethereum/execution-specs

[6]https://github.com/ethereum/consensus-specs

·END·

Content | Ray

Editing & Typesetting | Ring

Design | Daisy