Optimizing Smart Contract Event Indexers

Web3 applications rely heavily on indexing blockchain logs to present real-time dashboards to users. Fetching raw events directly from Ethereum or L2 JSON-RPC nodes during web requests is far too slow. This post covers building a custom indexer optimized for speed and reliability.

The Bottleneck: JSON-RPC over HTTP

Most Web3 backend systems query RPC nodes like Infura, Alchemy, or a self-hosted node. These nodes process query logs via eth_getLogs. This is computationally expensive for nodes because it involves scanning block state databases.

A naive indexing loop looks like this:

// Extremely slow approach - do not use in production!
async function naiveIndexer(rpcProvider, contractAddress, startBlock, endBlock) {
  for (let block = startBlock; block <= endBlock; block++) {
    const logs = await rpcProvider.getLogs({
      address: contractAddress,
      fromBlock: block,
      toBlock: block
    });
    await saveToDatabase(logs);
  }
}

This makes one network request per block. Over a range of 10,000 blocks, this takes minutes and risks triggering rate limit blocks from the RPC provider.

Designing a High-Throughput Indexer

To speed up indexing, we apply three core design principles:

1. Dynamic Partition Range Batching

Instead of indexing block-by-block, index in ranges (e.g., 2,000 blocks at a time). If a range fails due to payload size limits, dynamically half the block range and retry.

async function fetchLogsInRanges(rpcProvider, contract, from, to, maxRange = 2000) {
  let currentBlock = from;
  
  while (currentBlock <= to) {
    let range = Math.min(maxRange, to - currentBlock + 1);
    try {
      const logs = await rpcProvider.getLogs({
        address: contract,
        fromBlock: currentBlock,
        toBlock: currentBlock + range - 1
      });
      await processLogs(logs);
      currentBlock += range;
    } catch (error) {
      if (error.message.includes("limit") || error.code === -32005) {
        // RPC returned too many results, cut range in half and retry
        maxRange = Math.max(1, Math.floor(maxRange / 2));
        console.log(`Payload too large. Reducing range size to ${maxRange}`);
      } else {
        throw error;
      }
    }
  }
}

2. Multi-Node Load Balancing

Distribute requests across multiple RPC providers. Implement a round-robin connection pool that checks node health and latency periodically.

3. In-Memory Cache Layers

Cache block numbers and transaction hashes before querying the database, eliminating duplicate SQL lookups.

Summary

By shifting from block-by-block fetching to adaptive batch ranges and multi-provider load balancing, event indexing times can be reduced by over 90%, enabling real-time responsive UIs for Web3 decentralized applications.