Meet Notion Workers: Here’s How Syncs Work

Hey, I'm Anthony, a software engineer here at Notion. I'm working on Notion Workers, and today we're going to be talking about building sync functions with workers. And sync functions let you pull in data from any data source, a CRM, a budgeting app, a fitness tracker, into Notion, so you can work with it with your team, or build custom agents on top of it, you name it. Hey, I'm Jimmy. I'm also a software engineer here at Notion. I don't work on workers, but I'm trying to build one, so I want to ask Anthony some questions about how it all works, how do I deploy, how do I get going? So, let's step through the different components of a sync, starting at the foundation, which is the database that you sync to. So, the first thing you do is declare the schema of the database, a Notion database, you want to create to sync items into. And this is a declarative schema, so in the future you can make updates to this, add columns, remove columns, and we'll migrate it for you every time that you deploy.

So, this is the sync declaration here, and we point the database to the task database that we established above. We can give it a schedule, so the sync is going to run every 30 minutes. And then the most important part is this uh JavaScript function callback right here. And so, the way that syncs work is you define a function where you do some small unit of work, such as fetch a single page from the API, uh and then you return to us, to Notion, a list of the changes associated with that page. So, in this example, we're just upserting a bunch of new tasks to a database, as well as if there's more to do, there's more pages to fetch, and then what the next state should be. So, it's very common when pulling data from an API, you go fetch a page, you return some data, you have a next page token, you go fetch the next page, fetch that data, bring it in, and so on and so forth, until you get to the end, where it has more equals false. And so, with this single function, where we pass in the last page token, you make a fetch, we pass in the next we're able to allow you to sync from any data source, and give you really great observability, which we'll show you later, cuz you're doing everything in this very incremental fashion, so we can tell if things are delayed, or how many objects you've synced, etc. That makes a lot of sense. So, you have your Notion database, you have your worker, you're able to pull in data. Every time I want to pull in data, do I have to pull in all my historical records, or do I have some more fine-grained control? Great question. So, the sync that we're looking at right here, indeed every 30 minutes, it'll wake up, pull in all the historical data. But for many data sources, that's super inefficient. You don't have to wake up and pull in a bunch of records you've already pulled in before. So, we have this concept of backfills and

deltas with worker syncs. So, a backfill is something you'll usually run once at the beginning of the sync, and that'll run a little longer and pull in all your historical data, and then the delta will run after that on a frequent interval to just pull in the delta, or incremental changes, since the last time it ran. So, I'll show you what that looks like right now. So, in this example here, we have defined two syncs. Mhm. So, the first sync is this PRs backfill from GitHub. So, this might go into GitHub and pull in tens of thousands, or hundreds of thousands of PRs. And uh this runs manually, so it's going to run once at the very beginning when I set up the sync, and then not run again unless I manually trigger it. And this is going to start at the very beginning of the GitHub stream and pull everything in. And then once it gets to the end, it'll stop. And we can pair this backfill sync to our PRs DB with a delta sync. So, the delta sync is also going to the PRs DB, but this one you'll notice has mode incremental, so it's meant to be run on this incremental basis. It's run every 5 minutes. And we won't get too deep into the weeds here, but this delta sync is going to take the cursor from the last time it ran, so 5 minutes ago, and use that to just grab the PRs that have changed since the last sync. So, in this way, we can kind of pair these two syncs together, the backfill to get historical data, or if we add new columns to Notion, we want to re-backfill those columns in from GitHub, cuz they weren't stored there before, and then the delta sync, which will just grab whatever PRs have changed since the last time it ran. Oh, wow, this is great. I You know, when you guys started this project, it was just about pulling in data, and now you have this fine-grained control over like how often do I bring it in, what do I bring in, and like keeping up to date. That's awesome. Yeah, the pattern we saw over and over again was people trying to implement this backfill delta pattern, and so we made it a

first-class feature. So, in addition to this backfill delta concept, we recently introduced this idea of a pacer, which is a way to control rate limit consumption, cuz that's happens with every API ingestion job. And we want to give coding agents all the tools they need to build a first-rate sync. So, a pacer allows you to declare a budget for requests over a given interval. You know, this is an example for the GitHub API. And then I can use this pacer in my code by simply calling await function on it. And so, this will make sure that if my sync is running really fast, or it's been running really repeatedly, that I pause, or the sync pauses, in order to allow us to remain below the rate limit. And this lets us also evenly spread rate limit consumption out over the course of a sync, versus consuming all up front, for example, in a backfill. Yeah, I love that. I mean, the fact you could just put it in, and it's like for 30 minutes, I only want this amount. I don't have to do the math, you know, number of times to go in and like calculate like how many times am I about to run this? I mean, That's right. — Yeah, this is easier. Just drop in, agent will find it and use it without even really thinking about it, to be honest. Cool. So, this is great. So, we have databases, we have workers, we can sync historical data, we can sync new data, and we got rate limits. How where do I put all of this? Yeah, uh so I'll show you how to deploy it next. So, first I just want to point out we um authentications always a big pain with these APIs. So, we have OAuth support built in. I've already gone through the flow, but basically you plug in your client ID and secret, you trigger it in the CLI, and we'll take you through the OAuth flow, and then Notion will manage your access tokens and refreshing them. Which is — just connected forever, basically. — That's right. You don't have to worry about coming in and re-authing or anything like that.

So, in our terminal here, I'm going to run ntn workers deploy, and it's going to upload the code on my local machine to Notion, build it, and then it's going to run it in a secure sandbox. After creating the database and doing the backfill here. So, after the deployment process, it's basically running, it's good to go. My data comes in. Is there any logs I can check, anything I can look at? — That's right. So, it's uh it's been deployed, so the sync should be running. We can go check on it right now. So, here's the database. We've got our pull requests flowing into it. Uh if I hop back over to the terminal, you can take a look at some of the cool CLI tooling we built, which has been a lot of fun to work on. So, we can do um we have the whole kind of sync name space for managing syncs, and we can run sync status, and that will show us how our two uh sync jobs are going. The next thing I want to show you on the CLI is that you can trigger a sync, too, which is great if, for example, you want to re-backfill, or you're doing some sort

of debugging, you just deployed, whatever it is. So, I'm going to just trigger the backfill, or sorry, the delta sync here. And so, that will cause the delta sync to run right away. And if I look at the status, we'll see that it's currently running right now. That's great if you are testing things, or just deployed something and want to see if it works. So, the last thing I want to show you is our terminal user interface. We couldn't help ourselves, it's pretty fun. If you open this up, you can see list of workers on the left-hand side, and then my syncs, so you can see my two syncs are healthy. I can drill into this. I can look at the delta sync, how it's doing. I can trigger it from here, which is pretty cool. You can also see a list of runs, and then for any one of the runs, you can go look at the logs. So, really helpful tool for debugging. Everything you do in the TUI, you can do elsewhere, too, so that it's super agent-friendly, and agents can easily close the loop, do the deploy, look at the logs, fix errors, push them. So, although we went into the technical details today, our goal for worker syncs is that you actually don't really have to know any of this, because we are telling agents, through all the skills that we ship with workers, exactly what all this is and how to make it all work together. Yeah, that's great. So, I I just opened up my Claude code in my Code X, and I'm like, "Make me this worker. Don't make any mistakes, but just sync my data. " And it's pretty much all deployed from there on out. — That's right. So, I'm going to fire up Claude right now, and you can just type /sync, and we have a sync debug skill available. — too, yeah. We have a skill for it. And with /sync, we've tried to distill as much knowledge around how to write a really robust sync with backfills, deltas, pacers, into this. It also gives it a nice little tutorial on, or you know, how to on all of our CLI tooling. So, the agent can drive the entire thing from top to bottom. And Ian, our camera guy, who Have you deployed anything before, Ian? Uh no. was able to sink in multiple health sources in an ocean. We need like a scene that goes to Ian, you know? Yeah.

So, is it launched? It's out. So, just install the n8n CLI with npm, type in n8n workers new, open up your coding agent, get to work.