15:15 – 16:00
We’re gonna need a bigger database: Scaling to multiple databases with Debezium and Kafka
As our application grows, at some point the database we’ve chosen might no longer be sufficient to do what we want it to do. Perhaps we need to scale more than our database can handle, or maybe new features gradually require things your database isn’t great at. Replacing a database in an existing system is a lot of work and risky, and besides there may not be any single database that does all the things we need. As an alternative, we could leave our original database where it is, use a change capture platform like Debezium. In a nutshell, a change capture platform observes a database and records all inserts, updates and deletes to an event log system, such as Apache Kafka. From this event log we can stream this data to other databases, and use those databases instead of our original database for read traffic. The best part is that these databases can be completely different kinds of databases. We can stream from a relational database to a graph database, or from a document database to an search index like Elasticsearch. We’ll get into the moving parts of this setup, and see what is good, what is bad, and what’s ugly.
Frank Lyaruu / CTO