New Super Resolution AI - Enhance ~10x Faster!

I did not think this would ever be physically possible look at that this is a new super resolution technique for videos and video games so in goes a really coarse pixelated piece of footage and out comes this absolute Beauty and the input is look 270p that is a really low resolution a potato can compute that so whenever you see LR that is the low resolution input ours marks the new technique and GT is the ground truth that is rendering the True Result in a high resolution without any help which of course takes much longer but is necessary for comparisons the Eastern Village scene offers some of the best super resolution results that I have ever seen goodness wow it is really close to the ground truth too and so far you have seen nothing yet look at the medieval dark scene here hm that is some kind of joke right I mean just look at the vegetation here A bunch of pixels would you be able to make a drawing pixel by pixel and guess what this exactly should look like if it were in a higher resolution for me not a chance but this new technique wow that is stunning yes it still has trouble with thin structures like this that is the two-minute paper special but to think that it can go from this to this sounds like something straight out of a science fiction movie and I am really hoping that this can one day be the future of gaming could we just buy a cheapo video card render the game at a really low resolution let this AI do its magic and off we go also this is all well and good but how fast or slow is it if you take an hour to do this per image that is not very practical is it so how long does all this take let's have a look together dear fellow Scholars this is 2minute papers with Dr car well for a previous method from just 3 years ago took more than 100 milliseconds per frame which is 10 frames per second just for super resolution and we haven't even rendered the game so that is clearly not practical in fact it wouldn't fit easily on this chart because the bars would be so long w wow that is almost 10x faster than the technique from 3 years ago that is stunning but there are other techniques out there so what about those well if you are looking for a mild 2x doubling the resolution the new one blows the doors off it can do this 82 times per second easily twice almost three times as fast as some of its competitors from last year super good now let's bump it up to 3x oh yes think things are getting closer now but it still comes out ahead and if we are really demanding we want to quadruple the input resolution it takes a tiny bit longer than fuse Sr a fantastic technique from just a year ago so in that case I wonder if it is worth waiting this much for let's have a look together yes the new one is certainly better here and is worth the longer execution time while other cases are a bit more subjective which one do you feel is better let me know in the comments below and now let's try to resolve this somehow together so far we eyeballed the results now let's look at the numbers meaning let's put a ton of these cases together and calculate a score mathematically what does mathematics think about subjective cases like this who wins well did we get an answer yes and no but mostly yes let me try to explain if we just measure the distance between the pixels compared to the True Image the new technique is clearly better however this metric does not understand the image it just measures distances if you take an image and you darken it a bit or brighten it up it does not know that it is the same image thus it gives it a poor score so they also checked structural similarity that is a bit smarter as it can take into consideration that a darker version of the image is still the same image if we look at the overall scores the new technique does not win everywhere all the time but it wins most of the cases I am comfortable handing it the win also technically this new technique is chefus why well get this it uses the fuer transform a mathematical technique for breaking down a signal for instance a song like taking the Basse guitar out of a song and it mixes that with deep neural networks and hence this is called

Deep fuer based super resolution okay so what are the limitations is it perfect clearly not thin structures are not great there are also cases which are highly subjective this one is a very typical case of oversharpening versus over blurring H pick your poison temporal splotches are also present I believe a bit less than for the previous technique from a year ago but it's still there that is what we are looking for to make this more practical and finally a disadvantage that is relatively easy to isolate that is fog and particle systems in those cases you flat out cannot use this super resolution not because it theoretically can't math because it looks at geometric data from something called the G buffer and there is no data for partical systems in there that is a bit of a bummer as those can get really costly to simulate but it is theoretically possible to pull it off so you know just imagine what we will be capable of two more papers down the line that is the first law of papers for instance I can imagine that we won't even need the high resolution assets like geometry or textures for a game that we are creating because we will be able to use super resolution so reliably that we simply won't need them anymore perhaps this will be like compression but taken to the extreme what a time to be alive and to think that we have amazing papers like this appearing day by day and scientists giving the description of the technique away for all of us for free a fantastic scientific contribution and a great gift to humanity thank you so what do you fellow scholar think let me know in the comments below we need new tools for the era of llms and weights and biases now has weave a lightweight toolkit to confidently iterate on llm applications use traces to debug how data flows through each step of your app and use evaluations to measure your progress it is the best try it out now at wb. me/ papers llm or click the link in the description below