Send Postgres changes anywhere

San Francisco, CA
Joined December 2020
Include

Only show posts containing:

Exclude

Hide posts containing:

Time range
-
Minimum likes
FUNCTIONS > INTERPOLATION We had an interesting debate here at @sequinstream. For our new routing feature, we want to let users dynamically route messages based on their contents. So, you can setup one pipeline and route a change message from Postgres to a different Kafka topic or HTTP endpoint or Typesense collection depending on the message. You might route based on the message's contents (e.g. `status=delivered`) or based on its metadata (e.g. `metadata.table_name`). The way a lot of tools do things like this is with interpolation. So, with say a Kafka topic, you might specify it like this statically: ``` topic: "my-topic" ``` But with interpolation, you can set the topic to whatever the row's `table_name` is: ``` topic: "{{message.metadata.table_name}}" ``` Interpolation is powerful. And what's nice is that it's clear that the goal of the code is to derive a value for `topic`. However, it has a lot of downsides: - In the UI, every input field now needs to become a powerful code editor. And what if you need multiple lines of code to derive `topic`? - How do you test this function? - What variables are in scope? It's not clear. Instead, we realized that just like we have functions for transforms, we can have a function for routing. Routing functions take in context about the message, and return a map. Each sink has options for what a `route` function can return. Kafka's can return a `topic`: ``` def route(record, action, changes, metadata) do %{ topic: metadata.table_name } end ``` You define all your routing logic for a pipeline in one place, which can be a full-blown IDE/playground. This means: - It's easy to test. - You're not overloading input fields with immense power everywhere. - Because it's a function that you define, it's clear what variables are in scope! Soon, Sequin pipelines will be customizable through functions of a few types: - Filter functions - Group functions - Route functions - Transform functions With these functions, you don't need to sprinkle interpolated code everywhere – behaviors are defined in discrete places. We'll be shipping route functions ✨ tomorrow ✨ – we're very excited for all the power they will unlock!
1
9
2,910