Summary: Jeff Carpenter talks with Hiroyuki Yamada of Scalar to learn how his team have built Scalar DB, which provides a transaction capability on top of Apache Cassandra. To find out more, visit https://github.com/scalar-labs/scalardb Highlights: 0:00 - Jeff Carpenter welcomes Hiroyuki Yamada to the show. Hiroyuki gave a very interesting talk at ApacheCon on building transactions on top of Cassandra 1:40 - Hiroyuki shares his background in distributed systems and analytics 2:10 - Got involved with Cassandra looking for a highly scalable database 2:53 - Cassandra historically is pretty conservative with respect to providing consistency guarantees - what was the idea for building ACID transactions? 5:34 - Scalar has two products - Scalar DB provides ACID compliant transactions. Scalar BL uses Scalar DB to provide a ledger style database. 6:57 - Scalar DB is architected so that it could be implemented on top of multiple underlying databases, Cassandra is the first implementation. 8:00 - Scalar DB is a Java library which provides the an API based on the concept of records. You'll add metadata columns to your underlying Cassandra table, there is also a coordinator table used internally to track transaction status. 9:14 - The API is a Java interface with get/put semantics and simple CRUD operations. Transactions consist of a start statement, multiple put statements, then a commit statement. 10:30 - The implementation uses optimistic concurrency control. conflicts are detected at commit time. The algorithm is based on the Cherry Garcia protocol. 12:33 - The implementation uses Cassandra lightweight transactions (LWT) as a linearizability primitive, Scalar DB is the first implementation of the algorithm in the Cherry Garcia paper 14:07 - Testing has included destructive testing with Jepsen over the past two years. They have passed the Jepsen tests on a 100 node cluster and have demonstrated linear scalability 15:23 - The Scalar DB performance benchmark demonstrated 100,000 transactions/sec for a 100 node cluster, on I3 4XL machines, for roughly 1000 transactions per second per node 16:58 - The latency on a highly concurrent workload is typically around 100ms per transaction 17:30 - The transactions work across multiple data centers, but latency will be higher 18:46 - The performance tests used a payment processing scenario which involved lots of reads and writes and in-place updates. They also tested a second scenario with an append-only ledger model 20:00 - This work resulted in a recommended improvement to Cassandra improvements - improvements to commit log handling 21:36 - Next steps include improving the tooling around Scalar DB including a backup tool for Cassandra that takes a transactionally consistent backup and improvements to Scalar BL 23:05 - Wrapping up - having Jepsen may help decrease the time to maturity for new distributed database technology.
