Most Medallion Architectures in Microsoft Fabric Are Overbuilt

Yooo, what is up everyone? Over the last few projects I've been involved in, I've noticed a pattern. Bronze, silver, gold, three schemas, three layers every time. It doesn't really matter what the dataset looks like, we just default to it. Now, before everybody starts attacking me, I'm not telling you medallion is wrong. It is not wrong, but just applying it automatically. I think that's where we're overbuilding, where you are over architecting that solution. Architecture isn't about just adding layers, diamond, silver, platinum, gold, whatever it is, right? It's not about that. It's about deciding why those layers exist. Why do we need those layers? And in a lot of projects. We just skipped that part. So in this video, I'm gonna break it down. Do I need a medallion or can I use something more simplified? So you know what we like to do? Enough of all this talking. Let's do what, let's head over to my laptop. I have a really simple retail data set, so I'm going to expand this out. And so in my landing schema, you can see I just have dates, products, sales, and stores. It's just this, a simple data set, structured data, no streaming feed. No messy semi-structured files. Nothing volatile, just stable inputs. And I've implemented a very common pattern in fabric. And so I've created some notebooks and they materialize the data into my bronze, my silver, and my gold. And that's totally normal. So let's take a look at the bronze notebook. And it's basically raw data. I'm just taking the data from my curated tables. And writing them directly into my bronze layer. And then you have a silver tables, just some usual cleanup. Add some columns, do some type casting, trimming, some data just like guardrails. I'm putting some guardrails around this ingestion so I can ensure my data looks a specific way. And then I have my gold, which creates my aggregates. And so if you take a look at this. You can see it's just simple python where I'm doing some joining and I'm doing some aggregates by date and year, month, I have a monthly aggregate. I have a region aggregate. It looks disciplined and it feels mature. But let me just pause for one second and ask you a very serious question that I asked someone recently who did this medallion architecture. I'm like, what problem did we just solve by creating those layers? Whew. If you think about it, the landing feeds the bronze. The bronze feeds the silver and the silver. It feeds the gold. This is where I got really stressed out. Each step in the notebook materializes a physical table. I'm not telling you this is wrong. All I'm saying is this is a very common practice, but every time we materialize a step, we create a boundary. And someone has to own that boundary. We just layer bronze, silver, gold, and like I said in a previous video, sometimes I see diamond platinum. Before you automatically build three schemas, I want you to ask a couple of questions. The data in the source, the schema in the source, how often does that change? Are multiple teams modifying the structure? Is this shared infrastructure across multiple domains? If the answer is no, why are we physically separating it three times? 1, 2, 3. What I'm about to show you, I'm not changing the outcome. We still want the curated tables. We probably, we might, it depends still want the aggregates. What I'm changing is how many physical handoffs we create to get the exact same result. So what we have now is I still have my landing zone, but now I have a curated schema and my analytic schema. The landing handles the ingestion, like before the curated produces the scar schema, the facts and the dimensions. And the analytics materializes aggregates only where usage earns it. If I don't need those aggregates, if I'm using the semantic model like Power BI and Power BI can handle those aggregates without me pre aggregating it and it's still performing, then I don't need to create those aggregates. Maybe there's some other reporting tool that can't connect to my semantic model, and we need those aggregates for it. It justifies it, and we create that layer. In my curated layer, I can define data quality constraints directly in the materialized views. So the boundary isn't just cleaner, it's explicit, and it's enforced. Let me show you my materialized views. If you take a look at my materialized views, you can see I have these constraints that ensure that product and category is not null. Same thing, region and store is not null. Let's go to one of my facts, so I'm doing some checking to make sure quantity is greater than zero. Sales amount And then check this out. If I go over to, I'm just gonna do this, so you'll see how to get there. If I'm in my lake house and I've created these materialized views, if I go to manage materialized views, look at the lineage here. Remember before the lineage was landing bronze, silver. Goal now I have landing and curated. Curated is my star schema. Curated is where I'm doing like my data validation and my data quality checks. But I don't need that analytics layer.

So I'm going from four layers down to two, and the analytics layer is only there. If we deem it something viable, something that we must do, and you can see, right, it feeds my analytics layer and it's a much simpler and cleaner schema. So what this means is my layers. They should represent responsibility, boundaries, not habits. If responsibility isn't clear, layering becomes a ceremony. Ooh. Medallion is powerful when you have scale, when sources change frequently, when ownership is distributed, but stability doesn't depend on all these layers and layers. If we default to layers before we define the structure, we're not architecting, we're insulating. And insulation can be wait for it. Expensive. All right. Do you think you're over architecting? Or maybe you're like, Patrick, we always need this medallion architecture. I love to know. You know what to do. Post it in the comments below. If you wanna learn more about fabric, Power BI, AI, architecture, responsible architecture, it's probably a video flying above my head, and as always from Marthe, Adam, this guy. Thanks for watching. We'll see you in the next video.