This looks pretty interesting, for a few reasons: it can fit into existing ray tracing rendering pipelines, and you get most of the ray tracing benefits (reflection, shadows from geometry, refraction, depth of field, camera geometry) along with it. These are both pretty big.
Render quality is high / equivalent to MipNeRF (or however it’s capitalized). PSNR is equivalent or better, and the rendered output can be denoised with, say OptiX.
Some downsides/caveats — it works best if you retrain a little, so you won’t get the best quality if you’re pulling over mipnerf trained Gaussians, it’s slower to render than a straight rasterizer, like 50% slower, and of course these splats still don’t have geometry to them, as is much discussed elsewhere.
They spent a lot of work optimizing doing this for Nvidia’s RTX series, and the raytracing task is a little different than the typical one, which is to say it’s rare in ‘normal’ raytracing that you’re adding up the colors of 100s of transmissive, semi-transparent colors/radiances to get a single pixel; usually the bulk of the color from a raytraced scene comes from a smaller number of rays. If this method becomes popular, then NVIDIA could no doubt optimize the raytracing architectures further in the future and you’d get back some of that speed.
All this to say, I hope this gets rolled into existing engines, it’s practical engineering that would add a lot of options to workflows, and pretty neat!
I hope with this we can stop using "splats" to refer to the primitives that are being trained and rendered. The primitive are the 3D gaussians, "splat" was always an implementation detail of the rasterization approach.
Render quality is high / equivalent to MipNeRF (or however it’s capitalized). PSNR is equivalent or better, and the rendered output can be denoised with, say OptiX.
Some downsides/caveats — it works best if you retrain a little, so you won’t get the best quality if you’re pulling over mipnerf trained Gaussians, it’s slower to render than a straight rasterizer, like 50% slower, and of course these splats still don’t have geometry to them, as is much discussed elsewhere.
They spent a lot of work optimizing doing this for Nvidia’s RTX series, and the raytracing task is a little different than the typical one, which is to say it’s rare in ‘normal’ raytracing that you’re adding up the colors of 100s of transmissive, semi-transparent colors/radiances to get a single pixel; usually the bulk of the color from a raytraced scene comes from a smaller number of rays. If this method becomes popular, then NVIDIA could no doubt optimize the raytracing architectures further in the future and you’d get back some of that speed.
All this to say, I hope this gets rolled into existing engines, it’s practical engineering that would add a lot of options to workflows, and pretty neat!