Compiler-Driven Development in Rust

Hi friends, my name is Tris and this is no boilerplate focusing on fast technical videos Today I'm going to explain the alien magic of compiler driven development by analogy to test driven development. Demo 2 examples in Rust and make some recommendations When I begin a new Rust program, I don't start with functions or methods or any runtime code. I start with the model of my application expressed with Rust's rich type system. Interacting with the real world on disk or through the network is too slow for me at first. I need to iterate faster than that to sketch out my ideas unconstrained by the outside world. This compile checked whiteboarding is called compiler driven development. You may be familiar with repl driven development or the fast feedback of testing out your code in the browser with hot code reloading the compiler is faster than all these methods, but to be fair and balanced, it's because it's doing less pure model code. The heart of the plumbing of your app doesn't have to deal with rendering a ui, making network connections or setting up databases. None of that unimportant boilerplate. And nor should you. At first. Start by building the logic of your app in linked structs. Leave methods and functions to later don't run your code, just compile it and iterate with a compile watcher such as my favorite here Bacon running Cargo clippy with its enormous suite of lints turned on. Compiler driven development is like compiler checked whiteboarding whatever language you code in JavaScript, Python, Java, Go, even HTML and CSS to a certain extent, we developers all live at compile time. The time when our code is actually executing on the CPU or in browser plays a very small part of our days. Most of our time is not spent writing or running code, but reading it and often trying to figure out strange runtime behaviour. This is exactly what Rust's more complex syntax and comprehensive type system can help you with too, and all without writing unit tests. Tests and types are quite similar in a few ways. Tests don't get shipped, but they improve our code with feedback. Types and syntax feedback too. You don't deploy your test suite onto production, nor bundle it onto the user's device, and compilers by definition do the same. They compile out all the advanced types when building the executable. Neither tests nor types remain in the code that you ship to the user. You don't have to write tests. I'm sure you know a few cowboys and girls who proudly say they don't test their code, but in statically typed languages like Rust, you have to adhere to the models that you have expressed in the type system tests drive quality, as does modeling your application using a rich type system. If your language has an advanced compiler like Rust, you can sleep easy at night. You no longer need to wonder if the contractor you've hired has skipped running your test suite, nor worry if a junior developer has made what they think is a tiny Change directly on GitHub without ever running the code. Unstructured programming requires you to constantly execute your code to verify you've done the right thing. TDD just requires running the tests to be sure, but CDD only requires compiling the code to know that the model remains. Rust's superpowers are the direct result of being a language where the compiler can reason about as much of the code as you can. Programming inside this rich compile time environment means you might not execute your actual program for most of your workday, but when you do, you find if it compiles, it works. This magical way of coding is called compiler driven development. My video scripts are dedicated to the public domain. Everything you see here, script links and images are part of a markdown document available freely on GitHub at the above address. I will explain the conversation with the compiler that is core to compiler driven development in relation to my understanding of test driven development. I love TDD, and if you've not tried it before, you might too. It's probably not what you think it is. TDD is not really about testing, nor about writing all of your tests ahead of time. It's not even really about correctness. It's about not fooling yourself, because as Feynman said, you are the easiest person to fool. TDD for me answers the question I care most about when writing a Am I done yet? I'm not coding indefinitely for fun past 5pm I'm not even being paid to code, and I'm certainly not here for my health. Quite the opposite, actually. TDD cuts to the heart of this. I want to know when I'm done at the earliest possible moment so I can stop and get on with other things. The core method of TDD is to write a simple test before writing any other code, then watch it fail to go red, thus proving your test suite works. Then you write the minimum application code to make the tests pass, making it green again. You refactor if needed, and then improve the test, making it stricter or more precise than before. Watch the test fail again, and make it pass by improving the code. This constant tick tocking between red and green states for your test keeps you honest, focused and engaged with the question we're all trying to when am I done? Here's a tiny TDD example in Rust first we write a new test covering the functionality we're about to write a simple arithmetic dividing function that takes a numerator, denominator, and returns the result of dividing the two. Writing the test first makes us think about the problem we are solving first, rather than the plumbing of how to solve it. An important top down approach that can get lost.

In the weeds in typical programming. This same top down approach is used in CDD as we will see in a moment. Let's run this test. Of course it fails. It actually fails before even being run. Because it's calling a function, the compiler can't find our as yet unwritten function mydivider. We have successfully broken our test suite, thus proving to ourselves that the testing is doing what we want. Let's make it pass. We write the simplest code that makes our new test pass. Hard coding, as I'm doing here, is not only acceptable, but encouraged. Your tests should not be fooled by hard coded answers, should they? If you can't write a test comprehensive enough to fail a hardcoded solution, maybe you've discovered that hard coding is the correct answer this time. LLVM might actually do this for you behind the scenes. In some cases, hard coding has taken us back to green tests passing. I will finish this example here. You can already imagine how we would carry on. We'd watch the new tests pass, improve the test so hard coding doesn't cut it and watch it fail, then improve the code and make it pass, and so on. The method is simple. If the tests are failing, improve the code. passing, improve the tests. Now we've revised TDD. We can get into CDD after a word from today's sponsor, Quadratic, who I'm delighted to say are hiring Rust developers. Quadratic is a modern spreadsheet designed for data scientists, engineers and analysts. Built in Rust, WebAssembly and WebGL. Quadratic combines the functional data visualization of a spreadsheet with the power of full programming languages, starting with Python with SQL coming soon, standard Python data science libraries are built in. In fact, because Quadratic are using Pyodide inside WebAssembly, any pure Python dependency can be installed like this example of the Faker library. Loads of essential native libraries like NumPy, SciPy and Pandas have been specially ported by the Pyodide team too. Because all of Python is running locally inside WebAssembly, complex work such as here pulling data from an API is possible. This is all running at 60fps on the GPU using WebGL, all inside your browser. Quadratic built their infinite canvas on WebGL, allowing for smooth scrolling and pinch to zoom. Multiplayer support with live mouse tracking works so well. I'm mad that VS Code doesn't do this. They also have GPT integration giving you a copilot or pair programmer while you're writing a fantastic product made by some nice people and they are hiring. Quadratic are looking for great Rust engineers who have experience at a startup and leading software architecture and implementation and front end and AI devs too. Apply today or head to quadratichq. com to try it out. My thanks to Quadratic for their support of this channel. Now that we've revised TDD, we can talk about CDD. CDD operates on the same red green refactor pattern as TDD, except you can entirely skip writing tests. You're simply writing code that the compiler is not satisfied with, and the tick tocking between red and green is the conversation between you and in CDD, you don't check that your code works in the way you expect. By calling your functions or modules and examining their output or behavior in a unit test, you trust the compiler to prove the model of your system across all possible paths through your code. Because the rich type system models so much of your logic that would otherwise require runtime testing in other languages, you can skip writing tests for these cases and more. No invalid syntax, even if the bad lines are never called. In Rust coverage is always 100%. No insecure memory usage. Everything is safe to pass between concurrent processors. If it's not, the compiler won't let you do it. And my the Rust type system allows encoding much of your application's logic inside the types themselves, far more than simple objects or classes do in other popular languages. Let's look at two CDD examples to illustrate this. CDD works better the more the compiler knows about your code modelled in the type system, which extends out to much of the language. Here's a simple Rust enum modelling three different kinds of network IP addresses of both flavours and a MAC address. Here is how you'd use this enum and below a function that might use them to dispatch certain behaviour, depending on what kind of ID we have been given. Wonderful, but perhaps the customer adds a requirement for our application to handle more IDs than this short list. No problem. We can fearlessly improve our model with this new information by adding a few more variants. We've now added radio frequency coordinates and a UUID. We've improved the model of our code, adding the new ID variants, and our compiler goes red. This is good. If this change silently broke our app, that would be terrifying. But this is Rust and we're modelling correctly. The compiler is keeping us safe. We're back to red compile error. We've improved the model and so we must now improve the code. Unlike in TDD, we don't just get an error output, the compiler tells us exactly what it wants. And you know what you have to do when the compiler tells you what it wants. This friendly compiler error tells us that we're not handling all cases right. That makes sense. We have added new kinds of IDs, so we must update the code to take this into account. The match expression has kept us safe. This is why in Rust I prefer using match to if wherever it makes sense to do so. Handling the three new variants in our match expression brings us back to green, the compiler happy. And if the compiler is happy, I am happy. When you're writing Rust, you must strive to get to a compiling state as soon as you can. Until you do, all bets are off. LSP or your editor may act strangely, Clippy can't suggest advanced features to use, and CDD can't work, let alone your team's runtime test suite. Breaking the Code. Getting to a red state is an important transition. Don't stay there for very long. Break up big new features into atomic compilable chunks and iterate upwards from compiling states. Here's the flowchart that you should tattoo inside your eyelids. Remember, just as in TDD where we'd get to green by improving the co.

Code and we'd get to red by improving the test in CDD. We get to green by improving the code still. But to get to red we must improve our model using the type system. The previous simple example modelled our code with simple types. When you use CDD in Rust, you use the alien magic of the type system to improve your model so much you have no choice but to improve your code to satisfy the compiler. However, let's look again at this previous function. This doesn't quite spark joy for me. That Match arm, though kept safe by the model of our code, does not encode as much of our business logic as I'd like. We can do much better with a little Rust magic, and I've saved the best till last. The type state pattern encodes information about the runtime state of an object in its compile time type. This is a simple idea that has far reaching implications. I first became aware of this pattern after seeing some interesting syntax in the Rust standard library that I didn't know how to interpret, falling down a rabbit hole and ending up at Cliff Biffle's fantastic 2019 article the Typestate Pattern is annoying to implement in other popular languages. You may have never seen it used for this reason, but it's extremely ergonomic in Rust here. Here I'm modeling a light switch with two states, on and off and transition methods to move between them, including the new method pattern on the off state. You can't turn on a light that is already on, and I want the compiler to not allow such a transition at compile time. The type state pattern allows us to model functions that are only available when the struct is in certain states, and a way of encoding these states at the type level so that attempts to use functions in the wrong state fail to compile, enforcing that functions in previous states are no longer possible. In short, this is an extremely powerful way to make invalid states unrepresentable. Here is the usage of our light struct. The builder pattern lets us abstract away all of the intermediate states into a simple compile guaranteed interface. Now let's try to turn on a light that's already on. Not only is it a compile error, but it tells us what states we should be in if we want to use the turn on transition. Only lights in the off state are valid for this glorious you can of course write methods that work in any state, either simply by specifying the behavior in different states, like with the flip method here, or by using Rust's generics. There's more detail here, such as restricting behaviour to groups of states using traits. Do read Cliff Biffle's article for all the flexible options here. However, you model your application logic with the Rust type system, structs, enums, the type state pattern, or dozens more advanced methods, you can enrich the conversation between you and the compiler using compiler driven development. Just as in TDD, we can have enormous confidence that we've done the right thing with CDD by getting the compiler to pull its weight and do some work for us. The goal when modeling in Rust and languages like it is to move runtime behavior to compile time guarantees. I have a few announcements. Firstly, thank you to everyone who suggested setting up a Ko-Fi. I have now done so with the same tiers and benefits as Patreon. Additionally, I'll start crediting producer and sponsor tier patrons from all platforms on the end screen of all future videos as you can see here. Thank you all so much. If you would like to support my channel, get early ad free and tracking free videos, your name in the credits or one to one mentoring. Head to my Patreon or Ko-Fi. If you're interested in transhumanism and hopepunk, please check out my weekly sci-fi audio fiction podcast, Lost Terminal. Or if Urban Fantasy is more your bag, do listen to a strange and beautiful podcast I produce every full moon called Modem Prometheus. Transcripts and compile checked markdown source code are available on GitHub. Links in the description and corrections are in the pinned errata. Comment. Thank you so much for watching. Talk to you on Discord.