ENGINEERING

AI Won't Build a DeFi Indexer for You

AI gave us a clean implementation in seconds. Five weeks later 😵‍💫, we were still implementing it...

The pitch

We wanted to improve our DeFi vaults indexer. Count how many people hold tokens in a DeFi vault ("holders"). One number.

It sounds like a weekend project: scan Transfer events, maintain a balance map, count non-zero entries. Ship it.

We spent over five weeks on it. Not because we are slow, but because every layer of the problem hides another layer underneath, and each of those layers actively fights you.

Yes, we used a lot of AI. This is the full story.

Take 1: The plan that looked obvious

The algorithm is textbook. Every ERC-20 token emits a Transfer event on every movement. Replay them all, maintain a dictionary of address balances, count entries where balance > 0. Done.

The first implementation landed on day 1 as a clean, 1,000+ line change. Finding a contract's deployment block alone required a dedicated binary-search module — a trivial step that already costs multiple RPC calls per token.

We added persistence, resume logic, and a full-rescan interval.

Already more than a weekend project, and we had not hit a single real obstacle yet.

Take 2: Memory, rebasing tokens, and the first 429

We ran the scanner against a Lido stETH lending strategy. The process was killed. Out of memory.

Popular tokens have millions of Transfer events, and our implementation accumulated every decoded log in memory before processing. The fix was architectural: an async generator that yields decoded chunks one at a time, with parallel fetching across 20 concurrent calls. Another 1,000+ lines of code.

The same day, a different surprise. stETH is a rebasing token — its balances change when Lido's oracle posts rewards, without emitting Transfer events. Our replay gave a number, but the number was wrong.

There is no way to fix this within the Transfer-log model. We disabled holders for Lido, then discovered Resolv's stUSR has the same property and disabled that too. No automated detection exists. Each rebasing token requires manual identification.

Then we launched a full single-shot scan across all enabled strategies. After 45 hours and 11 minutes — having completed 23% of the total — our blockchain provider returned 429 Too Many Requests and the run collapsed.

Take 3: Uniswap V3 and Aave — three algorithms, not one

Uniswap V3 liquidity positions are NFTs, not fungible tokens. There are no ERC-20 transfers to replay.

Instead: scan pool Mint events, fetch transaction receipts to extract NFT token IDs, check active liquidity, get the current owner. Five distinct RPC-call patterns, each with its own failure modes. The implementation required over 2,000 lines.

Aave adds a different indirection. Depositors hold aTokens, one per reserve asset. You must first resolve the aToken address, then run Transfer log replay on that. The deployment block detection returned the proxy's block, not the implementation's — causing the scanner to replay millions of irrelevant blocks. And Aave Transfer events fire on interest accrual, not just deposits, causing overcounting that took significant debugging to isolate.

Three protocols. Three algorithms. Three sets of edge cases. A single "indexer" must handle all of them.

Take 4: Fighting the RPC layer

With functional holder counting for all three protocol types, the remaining time was spent on what we did not expect to be the hardest part: making eth_getLogs work reliably at scale.

Every RPC provider enforces limits, but none agree on what they are. Our initial approach — 10,000-block chunks, halve on error, grow back — was catastrophically wasteful with parallel fetching. With 20 concurrent ranges each independently probing and failing, we were burning 20x the RPC calls needed.

The fix: shared hint propagation so all ranges start from the best-known chunk size; a binary search regime converging in O(log n) steps; a response count guard that caps growth when approaching the provider's log limit.

When one range hits HTTP 429, every other range retries simultaneously — a thundering herd. We built a shared gate that pauses all ranges with exponential backoff (2s, 4s, 8s, up to 60s).

We also hit malformed RLP data inside syntactically valid JSON responses. Not a range error. Not a rate limit. Transient provider-side corruption, requiring its own classifier.

Take 5: The details that never end

Splitting a 7,000+ line monolithic RPC module into 17 focused submodules. Per-chain chunk size configuration so BSC's aggressive providers do not penalize Ethereum's conservative ones. Removing middleware that added wasted round-trips during pure log scanning.

Each of these is a small change on its own. Together they represent the polish that separates "it works on my machine for one token" from "it reliably counts holders across 1,500+ strategies on 15+ chains."

The final tally

Here is what "just ask the AI" actually required a human engineer to build:

• 3 distinct counting algorithms: ERC-20 Transfer replay, Uniswap V3 NFT position resolution, Aave V3 aToken delegation
• ~8,000 lines of production code across 17 modules
• ~1,200 lines of tests
• 5+ weeks of engineering
• An adaptive chunk-sizing algorithm with shared hints, binary search convergence, and response count guards
• A rate-limit gate with exponential backoff, Retry-After parsing, and concurrent-range coordination
• A streaming architecture to keep memory bounded on tokens with millions of events
• Manual identification of rebasing tokens that break the fundamental model

And we are not done.

Why AI can't shortcut this

We used AI throughout this project (yes, Claude). It was genuinely useful — first drafts, boilerplate, lookup of obscure RPC error codes. It roughly doubled our effective pace. It did not build this system, and it could not have.

Most of these problems are not visible until you run the system at scale against real providers, real tokens, and real chains. An AI cannot observe a process being OOM-killed or watch 45 hours of scan work collapse on a rate limit wall.

Each failure needs a targeted fix that does not break the other working paths. You cannot just retry everything — that compounds rate limiting. You cannot just reduce chunk sizes — that blows up the RPC call count. You cannot just add concurrency — that triggers cascades. The constraints interact, and the fix for one provider often conflicts with another's.

Happy building! 😎