Pipeline / Archive indexer
Local history is the development and recovery surface.
Stargazer mirrors Stellar's Galexie ledger data onto local NAS storage, then replays it with a bounded Rust pipeline. Parsing experiments, schema changes, new protocol modules, and incident research can run repeatedly without depending on remote RPC retention.
01 / Data path
Compressed ledger to queryable evidence.
- 01
Mirror partitions Implemented Download closed 64,000-ledger partitions from the public Galexie lake.
- 02
Walk in order Implemented Hold only one partition of paths in memory and yield ledgers sequentially.
- 03
Decode concurrently Implemented Stream zstd into bounded XDR workers without materializing an uncompressed blob.
- 04
Extract facts Implemented Keep contract transactions, exact wallet edges, protocol roots, and versions.
- 05
Commit batches Implemented Write ledgers in chain order and advance only after the SQLite transaction commits.
- 06
Finalize for reads Implemented Build account-oriented aggregates beside the immutable evidence database.
02 / Local-first archive
A mirror, not a validator.
The NAS contains a local mirror of Galexie LedgerCloseMeta objects. It does not
participate in consensus and should not be described as a Stellar validator. The downloader works
one 64,000-ledger partition at a time, reuses existing files by size, and records completion only
for closed partitions.
- Raw evidence
- Compressed
.xdr.zstobjects stay immutable and independently replayable. - Retention window
- The current runner targets an approximate recent-year window using ledger cadence.
- Tip handling
- The newest partition is synchronized again later and never treated as closed evidence.
- Concurrency guard
- Filesystem locks prevent duplicate downloaders and duplicate index writers.
03 / Resource model
Bound memory before chasing throughput.
| Pressure point | Control | Current default or behavior |
|---|---|---|
| Directory discovery | Partition-scoped walker | At most one partition of paths, normally 64,000 files |
| Decompression | Streaming zstd reader | No decompressed ledger-sized buffer |
| Malformed input | Compressed, decoded, depth, and zstd-window caps | Reject before unbounded allocation |
| Parallel decode | Fixed worker pool and hard decode-ahead limit | Two workers, four in-flight ledgers in the NAS runner |
| Database writes | One ordered writer with atomic batches | 2,000 ledgers per compact NAS batch |
| Wallet ranking | 256 disk-backed shards, one aggregated at a time | Designed to stay bounded on the 8 GiB NAS |
A representative 1,000-ledger local soak measured 56.6 ledgers per second with two decoders; that is an engineering benchmark, not a production service-level promise. NAS throughput depends on ledger mix, storage, database size, and the derived facts enabled for a run.
04 / Compact retention
Store the dependency graph, keep raw detail in the archive.
A transaction enters the compact database when it contains at least one of:
- a direct or Soroban-authorized contract invocation;
- a contract event or diagnostic-event touch;
- a contract executable observation such as creation, update, restoration, or removal;
- a contract Wasm code observation.
Ordinary transactions without contract evidence are omitted. Failed transactions remain when they contain useful contract evidence, but failed state changes are never materialized as current state. The canonical transaction hash, ledger sequence, apply order, and archive path provide the durable route back to full local evidence.
05 / Continuity and resume
The checkpoint is a committed contiguous range.
- Bind the database to one network. A network passphrase mismatch stops the run.
- Verify the ledger chain. Adjacent ledger hashes must agree before state advances.
- Reject gaps and prepends. Schema v2 may resume or append the next ledger; it cannot silently fill a sparse range.
- Commit the whole batch. A decode or write failure rolls back the active SQLite transaction.
- Resume from durable state. Rerunning skips committed ledgers and retries only uncommitted work.
06 / Serving projection
Read performance is built after historical ingestion.
The evidence database is ordered for append-efficient ingestion. The wallet finalizer opens it strictly read-only, streams exact transaction-wallet edges into disk shards, aggregates one shard at a time, adds contract and verified-protocol counts, creates both ranking indexes, validates the result, and atomically renames the completed serving database into place.