Pipeline / Live ingestion
Live is a new source, not a second parser.
The live service will acquire recently closed Stellar ledgers, verify continuity, and submit them to the same versioned processing boundary used for archive replay. A gap switches the system into catch-up mode; it does not permit the live checkpoint to jump forward.
01 / Target data path
From ledger close to committed finding.
- 01
Acquire Planned Follow newly closed ledgers from RPC first and Captive Core when self-operation is justified.
- 02
Normalize In progress Convert every source into the same versioned ledger input used by indexing modules.
- 03
Verify Planned Require the next sequence and previous-ledger hash to match the committed watermark.
- 04
Index Planned Run generic facts, protocol modules, and deterministic rules in ledger order.
- 05
Commit Planned Persist facts, findings, and checkpoint in one atomic transaction.
- 06
Deliver Planned Publish only committed live findings through a durable outbox.
02 / Source policy
Choose the source by recovery need.
| Need | Preferred source | Fallback |
|---|---|---|
| Development and first live tail | RPC getLedgers | Captured fixtures |
| Production live tail | Captive Core | RPC ledger backend |
| Short gap recovery | RPC ledger backend | Archive RPC getLedgers |
| Long gap or new module backfill | Local Galexie mirror | Archive RPC |
RPC retention is finite. The live service must record each source's oldest and newest available ledger and switch to archive recovery before an outage moves outside the recoverable window.
03 / Commit protocol
Acknowledgement follows durable state.
- Fetch the next ledger. Record source type, endpoint, available range, and raw evidence reference.
- Verify the chain coordinate. The sequence must be checkpoint + 1 and the previous hash must match.
- Run every enabled module. Generic facts and versioned protocol decoders consume the same normalized input.
- Commit facts and checkpoint together. A parser or database failure leaves the previous checkpoint authoritative.
- Publish findings after commit. Delivery consumes a durable outbox; it cannot make an uncommitted alert visible.
04 / Gap recovery
Pause, fill, reconcile, resume.
If a source returns a conflicting hash for an already committed ledger, ingestion stops and raises an operator incident. Automatic recovery is appropriate for missing data, not for conflicting history.
05 / Alert boundary
Detection and delivery are different guarantees.
Implemented today
- Archive-derived executable transition records
- A narrow historical YieldBlox correlation rule
- Executed Classic controller policy and activity alerts
- Telegram identity pairing and user policy storage
Still required
- Durable event outbox after the live database commit
- Idempotent Telegram delivery worker and retry state
- Per-user alert history and channel delivery evidence
- Payment settlement only after delivery is dependable
06 / Multisig scope
On-chain execution is visible; private proposals are not.
The current governance watcher can detect changes to Classic signer sets and thresholds, and it can retain the bounded payload of successful operations sourced by tracked controller accounts. It cannot observe an unsigned or partially signed envelope being exchanged off-chain, nor can it infer which named signers approved an executed transaction.