Advanced set extensions: replacing buildings with invisible VFX

Pay attention to the following short film sequence and try to spot the visual effects. — Hey, it's Mike. Leave a voicemail and I'll get back to you shortly. — Hey Mike, sorry to wake you if you're currently sleeping. I forgot the time difference. Listen, we uncovered the data and we think he's at the farm where it all started. — Do not go there alone.

Did you catch it? For this sequence, we weren't actually filming at a working dockyard. This warehouse had to be composited over a residential apartment building. This sequence, the crime scene location, did not exist. It had to be added into the shot. My favorite form of visual effects, isn't superhero CGI. It's the invisible F effects that you hardly notice. Let's have a look at

compositing buildings into shots. When it comes to building replacements, set extensions, or while just incorporating an entirely new element into your shot, you can often be at a disadvantage when using 2D compositing software over native 3D software such as Cinema 4D. And in fact, if you ever watch a visual effects breakdown of when they're placing in new buildings into a shot, it's usually a full-on CG rig opposed to a simple 2D composite that they've just placed their actors into. However, in

this tutorial, I'm going to run over a few advanced techniques in After Effects and one in Fusion and how you can incorporate a building into your shot or replace a building entirely with something different and make it look 3D. Now, I will say a requirement for these techniques is to ensure that you do have a flat 2D plane somewhere in the background that is going to act for the building replacement. But that doesn't mean you need to have a boring static shot. You can have a tripod pan. Your camera can be handheld. And in fact, if you have your character move in front of the composite, it's going to look even better because there's a lot of interaction between the subject and the background, which makes it look more real. I sometimes find that if you just have your subject off to the left and we do a composite over to the right, like the classic video co-pilot set extension tutorial, it looks very obvious that the element in the background isn't real. So, with that being said, let's head into After Effects for stage one. To start, we're going to bring our footage inside of After Effects. We need to decide on how we're going to track our footage so that we can composite a set extension into the background. This is initially where you may gravitate towards the camera tracking tool and hope to find some track points that you can latch on to. However, that's not always the right tool. For example, if I apply track camera to the sequence and even if we open the advanced settings and tell After Effects that this is a tripod pan and we're also going to enable the detailed analysis upon analyzing the shot, it doesn't give us any data towards the building in the background. And this is because it's out of focus. There's motion blur and the exposure and the contrast of the objects in the background are fairly similar, making it difficult for the camera tracker to distinguish what exactly is going on. So, no luck in this instance.

What we can use instead is a planar tracker. This is a motion tracking tool that follows the movement, rotation, scaling, and perspective changes of a flat 2D surface within a plane of a video. And because in this specific sequence, we're only seeing the building from the front with absolutely no three-dimensional aspects of the apartment building in the shot, a planer tracking tool is going to be perfect here. So, what I'm going to do is select my footage layer, duplicate and trim the layer to the appropriate length, as we don't need to track it all, and then go to animation and select Boris effects Mocha AE. and then select the Mocha button within the effects controls window to launch the Mocha interface. Upon loading the software for the first time, you may see a few prompts about registering and some tutorials, but for the time being, we can just close these. So, we're in a new interface within After Effects here. And if you're only uh a hobbyist user of After Effects, you might not even know that this is included within the native software. So, there are a few elements here to learn, and I won't beat around the bush. It can be quite an intimidating tool to first look at. Even Mocha's introductory tutorial, which just outlines the core basics, is about half an hour long, which by the way, I recommend you go and take a look at because it's an incredibly powerful tool that you should learn. But what I will do is outline some of the core basics as I run through the motions so that you can follow the steps when you're facing a simple 2D plane that needs to be replaced and the camera tracker is no good. So what I'm first going to do is create a rectangular XPlind layer and place it around my object, which is the building. You do that by coming up to these tools, which are going to be your primary action tools when it comes to telling Mocha what we want or do not want tracked. As expected, the pen, square, and circle tool all react in the same way as creating a shape in After Effects. And if you hold down on these tools, more options will appear, similar to Photoshop tools. I think it would be important here to state that while we are in a third party tool, the language between software typically stays the same. Okay, let's create the mask. Now, with my square, I'm going to start placing it around the building, but you don't necessarily want it to be dead on the subject. Why? Well, we're not creating a mask. We're telling Mocha what we want tracked, which is this building. By leaving some visual information to the sides, it gives Mocha a better understanding of what it is we want captured in the track. Now, I could hit track here, and I'm sure I would get good results. But in this particular instance, we do have an issue. And that is our character walks directly in front of the 2D plane that we're trying to track. And while it doesn't always cause an issue, it can. So in this instance, we're going to tell Mocha to essentially avoid this. And we can do this by creating a separate layer that exists as a trackmat. I'm going to take my pen tool and draw around our subject. And it doesn't necessarily have to be neat. But once that's done, I want to draw your attention here to the layers panel. And we can see that it sits on top of the background layer. And in the hierarchy of Mocha, any layers that are above will cut out the layers below. Now, with these two elements done, let's have a look at our track motion options. So, we've got transform, scale, rotate, skew, and perspective. And if I just scroll through the video in this particular instance, we don't necessarily have any form of perspective change. There's a slight bit of skewing and there's also not necessarily any rotation or scale. But the way in which these properties work is that if you turn off say scale but one perspective, all of the options before it turn on anyway. So here I'm going to deselect perspective. And it should also be noted that you really want to ensure that you are being vigilant with the selections that you're choosing. You don't want to select all of them for the sake of hoping to get a better track. In this instance, we do want to only select what is necessary. With that done, I'm going to select track forward. Then I'll return to the starting position where we created our mask and press track backwards. And if I click play, we can now see that it stays fairly consistent for this specific track. It does go fairly wonky towards the end once the tracking mask is out of frame. So, what I'm going to do is go to this option here, select it to tell Mocha that it no longer needs to be tracked past this frame. So, the next issue that we need to cross off our list is how do we ensure all of this data stays on the building itself? So, what

we need to do is with layer one selected, press show plan our surface. And we can now take these corners to the very corner of the building, fitting them nice and tightly. And as we move forward, you should now see this remain locked into place. Perfect for when we take this information into After Effects and replace it with our composite. And we can also get a visual aid by going into layer properties, insert clip, click grid, and we can now see it stays locked into place as we preview forward and backwards. If everything is looking good at this point, we can press save and then go to exit. Next, we're going to prep our composite image to be placed over the top of the existing building. We're going to follow the same steps as we did in the previous tutorial in that we're going to go to window generative fill and then click reference frame. We're not going to use generative fill itself, but I just find this a much simpler process to get the frame directly over to Photoshop without having to export the still and then open Photoshop. It's all done automatically. Now, there are a wide variety of things that we could do in order to add a building over the top of this one. We could, if necessary, use generative film or we could head over to story blocks. Look through the image library here to find something that fit. And it's worth noting that there are a number of images within the library which have been taken directly from video clips that exist within the storylocks video library. And I sometimes find that these work a lot better than your standard image because a standard image has a higher fidelity. The clarity is better. And sometimes that is what can make your composite looks so out of place is because of the composited element is of such higher quality. Whereas when you're taking a still from a video, it kind of bears imperfections in regards to quality that a video still would have in comparison to a photograph. Once you found what you need and brought it into Photoshop, you can start chipping away at the image so it fits correctly. You essentially just need to mask this building out and if necessary uh use any skew tools to slightly adjust the building to fit within uh the original shape. Then there are two things that we can do to match the image to our baseline foundation which is we can either press harmonize and Photoshop will use AI tools to give it a similar color hue, color temperature and exposure or we can do this manually in After Effects. Personally, I prefer to do it in After Effects because I feel like I have more control over the global parameters there. Now, once you have your image all set up and ready, we're going to do something a little bit different than last time. If you remember, I pressed saved and it was brought back into the reference file into After Effects. Uh, this time, what I'm going to do is actually copy my image and open up a new Photoshop document and paste it in at the uh, composition dimensions that were predetermined by the image itself. I'll make sure that my background is off. I will save it as a PNG file and then bring that into After Effects. Here, I'm going to pre-ompose it into a new composition and then change the composition size to the size of the original composition. And then I'm going to go to layer, transform, and fit to comp. It's now going to look very stretched out in our primary composition, but there is a reason for that, and it's to do with the Mocha tracking. So, let's return to the layer which has our Mocha data on. And if we go up to the effects control panel, press Mocha, and then we're going to select create track data. We're going to choose our selected layer with the necessary tracking data. In this case, it's layer one, which was the initial mask around the building. Then we're going to go to export options, corner pin, support motion blur, and set the layer export to the new pre-MP that we just made, and select apply export. Upon doing so, we now have our tracked 2D plane correctly in place and moving with the necessary motion of the camera. This looks fantastic. If you do run into any issues here after you've clicked export, again, I do recommend in having a look at the Mocha guides as they are incredibly in-depth, very lengthy, but they will get you right up to scratch.

Now, while I think it looks good, we of course do have the primary issue of our lead character being cut off by the comp itself. Here, we're going to have to do some rotoscoping. Now, there is a method within Mocha itself to track something like this and mask out. However, as we have a gap between the model's hair, her shoulder along with the poles and various other elements, we need to go for a more classical approach of rotoscoping. So, we of course have the roto brush. It replaces the very tedious process of having to manually rotoscope objects. You simply double click your layer, bring it into the layer viewer, select roto brush, and place a broad paint stroke over your subject. There may be moments where you need to add an additional paint stroke to tell After Effects that it's missed a section. Or you may need to remove a paint stroke if After Effects has included something you don't want, which you do so by holding Alt. In most instances, given that it's now in its third iteration, you won't have to do much manipulation of the tool. It's pretty fantastic as is. And it's also worth noting that in the beta version of After Effects, this tool is currently being phased out and being replaced with the object mat selection tool, which is even more accurate and requires even less manipulation. In my particular instance here, given that the subject is moving, the motion blur helps a lot with the edges. So on the roto brush refinement panel, I'm only going to need to adjust the feather, reduce the chatter, and ensure that I've selected use motion blur. Of course, all of these parameters will likely change depending on what type of shot you have. And if you would like a more in-depth rundown of the roto brush tool, we have a dedicated rotoscoping after effects tutorial on the channel. I now need to place our model over our building replacement, and it's looking pretty decent. We now need to add the final touches to our building to ensure that it blends in nicely with the background.

So, I'm going to add some Gaussian blur, adjust the brightness as well as the exposure in order to ensure it adequately matches these other buildings within that area. And then add some directional blur so it blends in nicely with the camera blur already present within the shot. And this is the final result. Now, along with visual effects, I'm also a freelance filmmaker. And when it comes to my personal projects, I've kind of had a an on andoff relationship with stock footage because I've kind of felt like it's cheating. It's not me who's done that. And, you know, it's another cinematographers's touch. But at the same time, when you look at say the Story Blocks library and the vast amount of footage they have with the wide variety of different shot types, locations, weather, and so forth, it does really allow me to obtain footage and insert shots that I just physically would not have been able to do. However

when I composite elements into these shots, such as adding an additional building, I can then make it my own thing. So with that said, let's head into Da Vinci Resolve into the Fusion page to have a look at how we can add a building into this shot from the Story Blocks library. So first on the edit page, select the clip that you will be compositing onto and also ensure that your element is in your media pool. Then

head on over to the Fusion tab. With the media in node selected, hit shift plus spacebar and search for camera tracker. You can also find this in the tool sidebar, but you know the shortcut saves time. Now, the reason why we're using the camera tracker specifically is that it will generate a virtual camera in 3D space. If we use the normal tracker, which you can also find in the color page, we'll only be tracking a 2D point, which doesn't really work well for compositing. Next, select the tracker and open the inspector in the top right corner. The 3D camera tracker is robust enough that we can leave most of these settings as they are. However, it's worth turning on preview autotrack location so you can see the tracking points and check for any issues. And you can also lower the minimum feature separation, which will give you more tracking points to work with. Then hit autotrack and let Resolve handle the rest. Once that's done, go to the solve tab and hit solve. Now, ideally, we want the average solve error to be under one. And you can maybe get away if it's at 1. 5. Uh, but if it's anywhere beyond that, we would need to remove or fix some tracking points. Now, to do this, we want to set the solve weight to zero. Click set and then adjust these three parameters to start isolating points that you want to remove and delete. That said, don't go too heavy on this. Uh, you just want to start by messing with the sliders and then try solving again until we can bring that average solve error down. And bear in mind that the quality of your footage will dictate how well Fusion solves the scene. Overtly shaky footage with motion blur won't give you that many trackable points to begin with. Now, once you're happy, go to the export tab within the inspector and hit export. Resolve will now add a bunch of extra nodes to the node graph, but we don't need them all and we do need to rearrange some nodes. So, let's get rid of what is no longer needed. So, we'll start by disconnecting and removing the ground plane node. And then we'll delete the initial camera tracker node. You want to disconnect the media in to media out. And then connect the camera tracker to media out. It should look something like this on your screen. Now, we need to add our element to the scene. So, add an image plane 3D node by clicking this button and then connect it to the merge 3D node. Next, bring in your image into the fusion node graph from the media poolool. You just simply drag it from the media pool into the node graph, and it will create a new media node, which we will then connect to the image plane 3D node. The building should now appear in Resolve, but probably not as you want it. So, if you need to adjust the scale or the position, select the image plane 3D node, go to transform within the inspector, and make your changes there. Finally, if you have depth of field in your shot, you may need to add some blur to the object. So to do this, you want to select your media in two node, hit shift plus spacebar, and search for blur and adjust accordingly. And this also goes for any color or exposure adjustments you want to make to uh make this composite more blended. All right, guys. I've been Lewis here today with Story Blocks. And we've covered a good few uh advanced compositing techniques in order to bring your footage to life or I guess give it a new sense of life with some new buildings in the foreground, in the background, and so forth. If you found this helpful, be sure to leave us a comment and say what you're going to do with this information. Or if you're a silent watcher, a thumbs up will do just as well. Until next time.