Open source data tool · 2018–2020
A Configurable MySQL-to-Elasticsearch Sync Engine
A Python data synchronization engine I created when the available tools I evaluated were difficult to adapt to project-specific processing.
Full SQL loads, row-based binlog events, and scheduled queries entered one configurable transformation pipeline with explicit resume checkpoints.
DATA SYSTEM
INITBINLOGCRON01 / System
What did I build?
I created mysqlsmom, a Python engine for synchronizing MySQL data into Elasticsearch through three operating modes: an initial SQL load, a row-based binlog stream, or a scheduled query based on update time.
Each mode entered the same configurable filtering and transformation path before bulk destination writes, while Redis retained the progress needed by the incremental modes to resume.
Read the diagram description
INIT reads an initial dataset through SQL. BINLOG consumes row-based INSERT, UPDATE, and DELETE events, while CRON performs time-based incremental queries. All three modes converge on normalization, action and watched conditions, a row filter, and ordered transformation handlers. A handler sequence can select or rename fields, set document identifiers, split values, run project code, and turn one input row into multiple Elasticsearch documents. The destination performs bulk upsert or delete operations and can address more than one index. Redis is deliberately outside the destination path: BINLOG stores its current file and position there, while CRON stores its last query start so either mode can resume after restart. The diagram does not claim exactly-once delivery or unverified performance characteristics.
02 / Context
Why did I build it?
At the time, the synchronization tools I evaluated were difficult to adapt to the custom processing I needed between a source row and an Elasticsearch document.
I wanted a smaller Python tool that could handle initial loading, ongoing synchronization, and project-specific transformation through one understandable configuration model.
03 / Design
What was actually hard?
Connecting MySQL and Elasticsearch was not the difficult part. The challenge was giving three different inputs consistent processing semantics while preserving the meaning of inserts, updates, and deletes.
Filtering, ordered transformations, one-to-many output, destination actions, and restart progress also had to compose without becoming a collection of mode-specific branches.
The useful boundary was not database to database. It was source → pipeline → destination, with progress tracked separately.
04 / Open source
Did other developers find it useful?
The public repository had 281 stars and 66 forks in the verified 2026-08-28 snapshot. Public issues also record real installation, integration, compatibility, and extension attempts.
Those are useful signals of developer interest, but they do not establish how many companies used the project or whether any particular deployment was production-critical.
05 / Inputs
How did data enter the engine?
INIT ran a configured SQL query for the initial load. BINLOG listened to row-based insert, update, and delete events. CRON periodically queried rows using an update-time boundary.
The trigger and resume behavior differed, but downstream processing stayed consistent.
THREE INPUT CONTRACTS
Different triggers, one processing boundary
- 01
INITInitial full loadConfigured SQL query - 02
BINLOGReal-time incremental changesRow-based INSERT / UPDATE / DELETE - 03
CRONScheduled incremental changesUpdate-time query
06 / Pipeline
How did the transformation pipeline work?
An event first passed action and watched-field conditions, then a row filter and an ordered list of handlers. Handlers could select fields, rename them, set an Elasticsearch document id, split values, perform an additional query, or call project code.
A handler could return more than one output, so one source row could become multiple destination documents without changing the source reader.
TRANSFORMATION PIPELINE
Project-specific work stays between source and destination
- 01Normalize
- 02Action / watched
- 03Row filter
- 04Ordered handlers
- 05Bulk destination action
- select fields
- rename fields
- set document id
- split values
- additional SQL
- custom script
FAN-OUTOne source row can produce multiple Elasticsearch documents
One source table can feed multiple Elasticsearch indexes
07 / Recovery
How did synchronization resume after restart?
BINLOG stored its current log filename and position in Redis. CRON stored the previous query start associated with its configuration and query.
These values told each incremental mode where to continue. Redis held progress metadata—not the documents being synchronized—and the project did not claim exactly-once delivery.
RESUME STATE
Progress metadata stays outside destination documents
BINLOGLog filename + positionresumes row-event streamCRONLast query startresumes time-based query window
Redis stores where incremental work should continue. Elasticsearch remains the document destination; exactly-once delivery is not claimed.
08 / Destination
How flexible was the destination model?
A configuration could cover multiple source tables, and the same source table could feed more than one Elasticsearch index. The destination translated pipeline output into bulk upsert or delete actions.
Custom row filters and handlers carried project differences, leaving the source modes and destination boundary reusable.
09 / Reflection
What would I change today?
This is a historical project: its Python 2.7 runtime and Elasticsearch APIs are now outdated, and parts of the configuration and runtime are more tightly coupled than I would choose today.
I would add types, restart-focused integration tests, modern error handling, and clearer Source, Pipeline, Destination, and Checkpoint interfaces. I would still keep the central idea: custom data work belongs in a composable pipeline rather than in each reader.