Skip to content

Welcome to Zync

Zync in an integration framework designed to help you connect as many services as you need while writing as little code as necessary. By defining a single central model, Zync will automatically transform data from multiple sources into one model, then again automatically transform it into the format necessary for each service it updates. It will even keep track of the ids a record has with each service, so it always calls create or update appropriately and never causes duplicate records. All you have to do is declare the CRUD methods for your services, and some simple data structures representing how the services data fields pair with the model's fields.

Features

  • Many-to-many data syncing from any user defined sources
  • Simple effective method of restructuring data into central normal model
  • NoSQL database stores all ids to keep track of congruent data
  • Translation methods for modifying specific fields per item
  • Conditions for syncing data to services can be based on the data itself, the destination, or whether it will be calling create or update method
  • Data fields may be "normalized" and "denormalized" no matter how many levels of dictionaries and lists the final value lies within.

Inspiration

After you realize you need to keep data in sync between two places, and after the groundwork for connecting to these services is laid, you then need to begin modifying the data in the middle to accomodate the two interfaces of the services that you're dealing with. The APIs will have different field names, such as zipcode and PostalCode, and their JSON structures will be different as well, for example:

{
  "cost": "$99.99"
}
# VS.
{
  "cost": {
    "value": 99.99,
    "currency": "USD"
  }
}

So you have to write more code to translate the data back and forth between them. Now imagine a third service being added to the mix. You'd have to write another translation function per data object per existing service (currently two) per sync direction. A fourth service would require even more new code as you translate to even more existing services, and so on.

In the words of Raymond Hettinger, there must be a better way!

After dealing with these types of issues frequently, I soon realized how much more efficient it would be to keep a centralized normal form for each type of data you need to sync. This means that when you need to begin syncing an object to another service, you only have to write translations to and from its normal form rather than to all the other services you're already working with.

Another way to think of it is with a directional graph, where each service's form of the data is a node and each translation function is an edge. If you have N services (i.e. N nodes), and you're translating between all of them, then you'll end up with N*(N-1) translation functions, and each time you're adding an node you're adding N-1 more functions. This is an exponential growth in the amount of code you'll need! If instead you put one extra node in the middle, the Normal Model, and each other node only has edges with that one, you'll only ever have N*2 translation functions.

Classes

  • Config - Takes the list of packages in the app that contains items, and optionally configurations for connections to the task queue and nosql database.
  • Zync - The main class which takes the Config and gathers all the items and models to sync and begins the sync. It makes sure the models always reach all the destinations they should.
  • Item - Represents a type of data provided by a service. The user defines the various CRUD methods for obtaining and updating these items within this class.
  • NormalModel - A single class (inherits from pydantic BaseModel) which defines the common form of the data between all services. The lingua franca of the data.
  • Normalizer - The heavy-lifter for changing Items into their respective NormalModels. This is achieved through the user-defined transform attribute, a dictionary of field names mapping the NormalModel's fields to the appropriate location in the Item's raw data. With the help of its Translator inner class it can also format, cast, or run any other user-defined methods the user needs on any of the field values before creating the NormalModel. By default, all of this will occur automatically when calling the Item's CRUD methods.