cross-chain-indexer.md

Multi-Chain Indexing Architecture

Let's say we want to indexing multiple networks (EVM and Solana) using a unified query layer with Spice.ai

flowchart TD
    subgraph Blockchain Networks
        E[EVM Chains] 
        S[Solana]
    end

    subgraph Indexing Layer
        RI[Reth Indexer]
        SI[Solana Indexer]
    end

    subgraph Storage Layer
        PG[(PostgreSQL)]
        RDB[(RocksDB)]
    end

    subgraph Query Federation Layer
        SPICE[Spice.ai]
        subgraph Acceleration
            AR[Arrow In-Memory]
            DD[DuckDB Analytics]
            SL[SQLite Local]
        end
    end

    APP[Application Layer]

    E --> RI
    S --> SI
    RI --> PG
    SI --> RDB
    PG --> SPICE
    RDB --> SPICE
    AR --> APP
    DD --> APP
    SL --> APP

    SPICE --> AR
    SPICE --> DD
    SPICE --> SL

Loading

Component Details

1. Indexing Layer

• Reth Indexer (EVM): Direct filesystem access to EVM node data. All set up with reth node.
• Solana Indexer: High-performance write throughput. RocksDB-based storage.

2. Storage Layer

• PostgreSQL (EVM Data): Write Speed: ~10K ops/second
• RocksDB (Solana Data): Write Speed: ~100K ops/second

3. Query Federation Layer (Spice.ai)

flowchart LR
    subgraph Data Sources
        PG[(PostgreSQL)]
        RDB[(RocksDB)]
    end

    subgraph Spice.ai
        DF[DataFusion Engine]
        subgraph Acceleration
            AR[Arrow]
            DD[DuckDB]
            SL[SQLite]
        end
        QP[Query Planner]
    end

    subgraph Client
        SQL[SQL Query]
        RES[Results]
    end

    PG --> DF
    RDB --> DF
    DF --> AR & DD & SL
    SQL --> QP
    QP --> DF
    AR & DD & SL --> RES

Loading

We will get the Unified SQL interface. Can do cross chain data aggregation for different chains. We can do query like these

SELECT 
    eth.address,
    eth.balance as eth_balance,
    sol.balance as sol_balance
FROM ethereum.balances eth
JOIN solana.balances sol 
    ON eth.address = sol.mapped_address
WHERE eth.balance > 0 
    OR sol.balance > 0;

flowchart LR
    subgraph Chains[Blockchain Networks]
        direction LR
        EVM[EVM Chain]
        SOL[Solana Chain]
    end

    subgraph Indexers[Indexer Layer]
        direction LR
        RETH[Reth Indexer]
        SI[Solana Indexer]
        MED[Medic]
        PW[Parquet Writing]
        style RETH fill:#4CAF50,color:#ffffff
        style SI fill:#4CAF50,color:#ffffff
        style MED fill:#4CAF50,color:#ffffff
        style PW fill:#4CAF50,color:#ffffff
    end

    subgraph Storage[Storage Layer]
        direction TB
        PG[Real-time System<br/>Postgres DB]
        DL[Datalake<br/>Azure Blob Storage<br/>10TBs]
        style PG fill:#F44336,color:#ffffff
        style DL fill:#F44336,color:#ffffff
    end

    subgraph Query[Query Layer]
        direction LR
        DR[Dremio]
        API[API Gateway]
        style DR fill:#FFA000,color:#000000
        style API fill:#673AB7,color:#ffffff
    end

    %% Connections
    EVM --> RETH
    SOL --> SI
    
    RETH --> PG
    SI --> PG
    MED --> PG
    
    PG -- "Past 30min to 1 day" --> PW
    PW --> DL
    
    PG --> DR
    DL --> DR
    DR --> API

    %% Annotations
    classDef default font-size:12px,color:#ffffff
    classDef annotation font-size:10px,fill:none,stroke:none,color:#ffffff
    
    note1[30min - 1 day data]
    class note1 annotation
    PW --> note1
    
    note2[10 TBs]
    class note2 annotation
    DL --> note2

    %% Style all text black by default
    style Chains fill:#ffffff,color:#000000
    style Indexers fill:#ffffff,color:#000000
    style Storage fill:#ffffff,color:#000000
    style Query fill:#ffffff,color:#000000

Loading