CompactLog – Solving CT Scalability with LSM-Trees

Summary

⚠️ This is a work in progress. While the CT functionality works, this should not be yet used in production. CompactLog A Certificate Transparency (CT) log implementation. CompactLog implements the RFC 6962 Certificate Transparency API on top of SlateDB to explore how LSM-tree storage can address traditional CT log scalability challenges. Overview This implementation provides a complete Certificate Transparency log that: Accepts X.509 certificate chains and pre-certificates Issues Signed Certificate Timestamps (SCTs) Maintains a cryptographically verifiable Merkle tree Provides inclusion and consistency proofs Stores data in cloud object storage (S3, Azure Blob) or local filesystem Storage Architecture Core Design Decisions CompactLog makes three fundamental design choices that differentiate it from other CT implementations: LSM-tree storage via SlateDB instead of relational databases or custom storage engines. STH-boundary versioning - only persisting tree state at published checkpoints. Synchronous tree updates - achieving a Maximum Merge Delay (MMD) of 0 seconds. How MD is Eliminated Entirely Many CT log implementations have a Merge Delay (MMD) of minutes to hours, where submitted certificates aren't yet included in the Merkle tree. This exists because: Many implementations issue SCTs immediately, then incorporate certificates later via background processes. Some implementations have expensive tree update operations. Consistency requires coordinating distributed components. CompactLog eliminates a MD by reversing this order - certificates are incorporated before SCTs are issued: Submission 1 ─┐ Submission 2 ─┼─ Wait up to 500ms ─→ Batch tree update ─→ All SCTs returned Submission 3 ─┘ └── Certificates already incorporated The 500ms delay is submission latency, not a merge delay. Once SCTs are issued, certificates are already in the tree. The batching system: Collects submissions for up to 500ms (configurable) to form a batch Updates the Merkle tree once for the en...