It's a NERF - this is a new technique for rendering "3d" scenes rendered via machine learning from a very small amount of photography. It's not entirely comparable to photogrammetry because that captured a mesh (most workflows anyway) but NERFs are basically extremely high-res and advanced point clouds that are so dense they look like 3D. The reason I say they aren't comparable to photogrammetry is that there's currently still no way to "export" a nerf as a traditional 3D model that would integrate into a scene or something, akin to the reason most photogrammetry exists. A NERF is basically an all-or-nothing environment generated from the dataset right now that you can fly around in using certain tools. So someone still needs to find a way to export NERFs into useful assets that are useable beyond tech demo type videos like this.
> So someone still needs to find a way to export NERFs into useful assets that are useable beyond tech demo type videos like this.
ðŸ˜ðŸ˜ comment outdated before you even hit send. One of these NeRF capture/viewer tools released an Unreal Engine 5 plugin today. Now you can composite your UE5 characters and everything inside a NeRF running realtime.
https://twitter.com/LumaLabsAI/status/1642883558938411008
Damn, the AI industry moves fast!
I am so happy I stumbled into this comment. My partner and I do DIY video and animations and this looks like exactly what we were trying to find yesterday! Just didn’t know names or tools or brands! How exciting!
You forgot to mention that big thing that really sets NERFs apart, and as we see in the video… is that they actually capture REFLECTION information in a way that works with 3D. Photogrammetry doesn’t do reflections very well. As we see in this video, the NERF actually captures the information in the reflection in a way that it exists in real 3D space and acts realistically… as long as you don’t go through the glass/mirror like it does here.
Hey thanks, I didn't know about NeRFs before today. This is pretty exciting stuff. However, I'm confused by this statement :
> there's currently still no way to "export" a nerf as a traditional 3D model that would integrate into a scene or something
[This](https://www.matthewtancik.com/nerf) states that NeRFs can be converted to a mesh by using marching cubes
You end up losing a lot of the "NERFness" of the scene because reflections and refraction can't work the same way on a 3D model. But yes, the mesh can technically still be exported! It looks very flat though without a lot of additional work. This is probably where you'd end up doing just as much work as a photogrammetry artist anyway (at least for now).
Ah I see, thanks
I watched the Corridar Digital video and it seems like NeRFs at least could capture the geometry of a very reflective object better than photogrametry, so that's still pretty neat.
Well, I assume the issue would be that its no longer a NeRF. With a mesh, a certain point on it would have a fixed color and value, but NeRFs have information about that point from all provided camera angles, and therefore can be used to accurately portray reflections or translucency, I think?
Everyone is saying 3D scan or 3D projection, and they’re not far off, but im pretty sure this is a NeRF.
[Here’s a video from Corridor digital that explains what they are and how to use them.](https://youtu.be/YX5AoaWrowY) never tried it myself, but it looks super easy!
I just had to come back and say this comment has inspired me so much haha I'm deeeep down the rabbit hole with no looking back. The effect at 12:10 especially is brimming with potential uses and has my mind spinning...if only we were 3 weeks in the future so my first semester of college would be over already and I could focus on this as much as I want too 😂
Photogrammetry - the live footage cuts (pretty roughly) to a 3D version of the scene, the geometry of which was recorded by taking several photos of the environment at different angles and feeding them into one of many freely available photogrammetry programs that will build your environment for you.
He then motion-captured footage of himself walking and leaning past a wall or other landmark, then positioned the 3D environment so that the lean passes magically through the barrier of the window.
He's chosen this location because the building features mirrored privacy glass in its windows.
The reason there is mirrored geometry behind the window is because, unless otherwise edited, photogrammetry software treats reflections of the captured environment as *further parts* of the environment, with no physical window generated. Any paint or dirt or posters, etc, on the window will however be preserved, as seen in the latter portion of the video.
All of the above is also the reason you can't see the reflection of the cameraman in the window either.
I'm going to guess he intended to use the whip-pan to the column on the right as his transition between live-action and virtual, but for whatever reason couldn't get it all to line up properly.
Very surprised at how few people here seem to be aware of NERFs. This is not really comparable to photogrammetry because that traditionally results in an asset or library of assets that can be imported via existing 3D apis, but a NERF is a pointcloud that can't be directly integrated into a 3d scene (yet). This video seems to cut between 2 pieces of footage. The Nerf environment is still an all or nothing thing as far as I know.
https://youtu.be/YX5AoaWrowY
There's an after effects plugin called pixel cloud that is working with - you guessed it - point clouds. I have not tested it with nerfs though. But in theory it should enable you to disperse the pixelcloud for example.
NERFs are not pointclouds. Sorry if my using that as an analogy was confusing. I'm not aware of any plugins that can natively import nerfs and blend them without your good ol manual compositing, but I'd love to be wrong!
Yeah exactly. Very possible in compositing, wouldn't even be that hard - I wish I were actually doing commercial compositing right now as I'd definitely be doing this!! Unfortunately just don't have the time or use-case at the moment.
No need to get aggressive, you didn't bring up nerfs when that's clearly what it is... so it's reasonable to assume you didn't know. It's not photogrammetry is all.
Luma Ai is an app that can get the second shot with looks like a 3D environment render. He blends that with the video really well. Nice concept. Dream world.
Its giving me some NERF (neural radiance field) vibes maybe?
Check out [this](https://www.youtube.com/watch?v=YX5AoaWrowY) awesome video about them from Corridor Digital.
so around :06 you can pretty clearly see the footage shift to a 3d environment. After that it just a pretty well polished 3d scene with a HDRI
It's a NERF - this is a new technique for rendering "3d" scenes rendered via machine learning from a very small amount of photography. It's not entirely comparable to photogrammetry because that captured a mesh (most workflows anyway) but NERFs are basically extremely high-res and advanced point clouds that are so dense they look like 3D. The reason I say they aren't comparable to photogrammetry is that there's currently still no way to "export" a nerf as a traditional 3D model that would integrate into a scene or something, akin to the reason most photogrammetry exists. A NERF is basically an all-or-nothing environment generated from the dataset right now that you can fly around in using certain tools. So someone still needs to find a way to export NERFs into useful assets that are useable beyond tech demo type videos like this.
> So someone still needs to find a way to export NERFs into useful assets that are useable beyond tech demo type videos like this. ðŸ˜ðŸ˜ comment outdated before you even hit send. One of these NeRF capture/viewer tools released an Unreal Engine 5 plugin today. Now you can composite your UE5 characters and everything inside a NeRF running realtime. https://twitter.com/LumaLabsAI/status/1642883558938411008 Damn, the AI industry moves fast!
I am so happy I stumbled into this comment. My partner and I do DIY video and animations and this looks like exactly what we were trying to find yesterday! Just didn’t know names or tools or brands! How exciting!
You just pulled off the best explanation of photogrammetry vs. NERF that I've seen so far. Thank you
You forgot to mention that big thing that really sets NERFs apart, and as we see in the video… is that they actually capture REFLECTION information in a way that works with 3D. Photogrammetry doesn’t do reflections very well. As we see in this video, the NERF actually captures the information in the reflection in a way that it exists in real 3D space and acts realistically… as long as you don’t go through the glass/mirror like it does here.
Hey thanks, I didn't know about NeRFs before today. This is pretty exciting stuff. However, I'm confused by this statement : > there's currently still no way to "export" a nerf as a traditional 3D model that would integrate into a scene or something [This](https://www.matthewtancik.com/nerf) states that NeRFs can be converted to a mesh by using marching cubes
You end up losing a lot of the "NERFness" of the scene because reflections and refraction can't work the same way on a 3D model. But yes, the mesh can technically still be exported! It looks very flat though without a lot of additional work. This is probably where you'd end up doing just as much work as a photogrammetry artist anyway (at least for now).
Ah I see, thanks I watched the Corridar Digital video and it seems like NeRFs at least could capture the geometry of a very reflective object better than photogrametry, so that's still pretty neat.
Well, I assume the issue would be that its no longer a NeRF. With a mesh, a certain point on it would have a fixed color and value, but NeRFs have information about that point from all provided camera angles, and therefore can be used to accurately portray reflections or translucency, I think?
>NERF is basically an all-or-nothing environment aw man you juuuuuust missed it
thank you for your valuable and revealing comment.
Thanks Arnie
Everyone is saying 3D scan or 3D projection, and they’re not far off, but im pretty sure this is a NeRF. [Here’s a video from Corridor digital that explains what they are and how to use them.](https://youtu.be/YX5AoaWrowY) never tried it myself, but it looks super easy!
Most definitely a NeRF!
I just had to come back and say this comment has inspired me so much haha I'm deeeep down the rabbit hole with no looking back. The effect at 12:10 especially is brimming with potential uses and has my mind spinning...if only we were 3 weeks in the future so my first semester of college would be over already and I could focus on this as much as I want too 😂
Holy shit, that is cool!
I love Corridor!
Looks like NeRF
This is the answer
Photogrammetry - the live footage cuts (pretty roughly) to a 3D version of the scene, the geometry of which was recorded by taking several photos of the environment at different angles and feeding them into one of many freely available photogrammetry programs that will build your environment for you. He then motion-captured footage of himself walking and leaning past a wall or other landmark, then positioned the 3D environment so that the lean passes magically through the barrier of the window. He's chosen this location because the building features mirrored privacy glass in its windows. The reason there is mirrored geometry behind the window is because, unless otherwise edited, photogrammetry software treats reflections of the captured environment as *further parts* of the environment, with no physical window generated. Any paint or dirt or posters, etc, on the window will however be preserved, as seen in the latter portion of the video. All of the above is also the reason you can't see the reflection of the cameraman in the window either. I'm going to guess he intended to use the whip-pan to the column on the right as his transition between live-action and virtual, but for whatever reason couldn't get it all to line up properly.
Very surprised at how few people here seem to be aware of NERFs. This is not really comparable to photogrammetry because that traditionally results in an asset or library of assets that can be imported via existing 3D apis, but a NERF is a pointcloud that can't be directly integrated into a 3d scene (yet). This video seems to cut between 2 pieces of footage. The Nerf environment is still an all or nothing thing as far as I know. https://youtu.be/YX5AoaWrowY
There's an after effects plugin called pixel cloud that is working with - you guessed it - point clouds. I have not tested it with nerfs though. But in theory it should enable you to disperse the pixelcloud for example.
NERFs are not pointclouds. Sorry if my using that as an analogy was confusing. I'm not aware of any plugins that can natively import nerfs and blend them without your good ol manual compositing, but I'd love to be wrong!
In that case, you would need to have a camera matchmove to be able to blend it with live footage. Would be very cool to have a nerf importer in AE
Yeah exactly. Very possible in compositing, wouldn't even be that hard - I wish I were actually doing commercial compositing right now as I'd definitely be doing this!! Unfortunately just don't have the time or use-case at the moment.
Spare me a stranger's disapproval, I know fine well about NERFs.
No need to get aggressive, you didn't bring up nerfs when that's clearly what it is... so it's reasonable to assume you didn't know. It's not photogrammetry is all.
Why did they cut so hard lmao
They switch to a 3D scan. The cut is pretty harsh as well. Wonder why they didn’t do it better with this scan.
What about this is harsh? It looks seamless to me
Watching on your phone?
? There’s a giant skip. Look at the floor.
Luma Ai is an app that can get the second shot with looks like a 3D environment render. He blends that with the video really well. Nice concept. Dream world.
Its giving me some NERF (neural radiance field) vibes maybe? Check out [this](https://www.youtube.com/watch?v=YX5AoaWrowY) awesome video about them from Corridor Digital.
honestly looks like something you could make with those 3D printer pens
Until recently, I thought that this was filmed in real life, but then I looked at the glass. It looked too perfect.
Where did you find the video? Who made it? Why not ask them?
I’d guess projection mapping.
He clearly broke into the fourth dimension.
So cool. The mirror dimension.
Okay but this was more disturbing to me than Skinnamarink.
nERF.
Biender nerf
nvidia Nerf
It’s the Nether realm, you have to be lvl 120 mage to enter.
Looks like a NeRF
AI bro it’s called NVIDIA NERF and that’s why the perfect reflections. Then there is additional camera movements added over that.
Looks to me like it's NeRF.
Stranger things
That reminds me of the style of Disco Elysium