Home

GraftRay: Physically Based Mesh Insertion into Gaussian Splatting Scenes via Path Tracing

Technical University of Munich
GraftRay teaser: objects inserted into Gaussian Splatting scenes, before and after

GraftRay inserts mesh objects into 3D Gaussian Splatting scenes with physically consistent shading and shadows. Left: objects with albedo, roughness, metallic and normal maps are inserted into a Gaussian scene (ellipsoids | render) whose materials, normals and emitters come from 2D priors and LDR images. Middle: the composited result; framed close-ups and the large view on the right come from the marked cameras (matching colours). Bottom: insertions in real and generated scenes; insets: the input scene without objects.

Abstract

Inserting virtual objects into 2D/3D Gaussian Splatting (2DGS/3DGS) scenes matters for content creation, AR and synthetic data; a realistic insertion must be lit by the scene and cast shadows onto it. Existing methods harmonize appearance in image space or approximate the illumination with an environment map, which cannot represent the near-field windows and lamps that dominate indoor lighting. We present GraftRay, a unified method for physically consistent insertion of mesh objects into 2DGS and 3DGS indoor scenes. We lift materials and normals from diffusion-based 2D priors onto the Gaussians, model windows and lamps as analytic area emitters, and recover their absolute radiance from ordinary LDR captures by treating saturated pixels as censored observations. A path tracer over Gaussians and meshes then produces view-consistent shading, interreflections and soft shadows, composited onto the original reconstruction. On FIPT and NFINITE, GraftRay outperforms ten recent baselines including IRGS and EAG-PT in fidelity and shadow accuracy at competitive cost, and users prefer its lighting and shadows over those of every baseline.

Method

(1) Material and normal priors from RGB↔X or DiffusionRenderer are back-projected onto the Gaussians; (2) windows and lamps marked once with SAM become rectangular area emitters whose radiance is fitted against the captured views, treating clipped pixels as censored observations; (3) a path tracer over the Gaussians and the inserted meshes renders the scene with and without the objects, and differential compositing transfers the change onto the original reconstruction.

GraftRay pipeline

Material and normal estimation. Virtual views of the 3DGS scene are decomposed with RGB↔X and back-projected onto the Gaussians with a transmittance-based visibility test, giving per-Gaussian albedo, roughness, metallic and normals.

Path tracing on Gaussian scenes

Path tracing on Gaussian scenes. Camera rays and bounce rays intersect the 3D Gaussians and the inserted meshes; at each vertex, next-event estimation sends a shadow ray to a sampled rectangular emitter and the path continues in a BSDF-sampled direction. Rendering each emitter at unit radiance yields the per-emitter basis images used to fit the emitter radiances; the differential composite transfers the objects' shadows and bounce light onto the original 3DGS rendering.

Results

Our results on all 18 benchmark scenes: camera sweeps with the inserted objects. Drag the line to reveal the empty input scene on the left.

Video

Interactive Comparisons

Camera sweeps through 18 benchmark scenes. Drag the line to compare ours (right of the line) with the empty scene or a baseline (left).

Ablation Results

Point lights
Point lights
Env. map (PT)
Env. map (PT)
Env. map (3DGS)
Env. map (3DGS)
1 bounce
1 bounce
2 bounces
2 bounces
4 bounces
4 bounces
Shadow only
Shadow only
One-way
One-way
Emissive 3DGS
Emissive 3DGS
Gray albedo
Gray albedo
Gray albedo (PT)
Gray albedo (PT)
w/o diff. comp.
w/o diff. comp.
Empty room
Empty room
Ours
Ours
GT
GT

Ablations, qualitative (bedroom, the view of the compositing example); composites unless marked PT. Point lights over-brighten, an environment map darkens; direct light alone is far too dark; a gray albedo distorts the room (PT), compositing hides it. ΔPSNR (Tab. 4 of the paper): point lights 26.5, env. map 27.6/27.4, 1/2/4 bounces 25.7/30.8/34.0, shadow only 31.3, one-way 27.5, emissive 27.5, gray albedo 23.5, w/o compositing 24.8, ours 35.4 dB; emitter-geometry perturbations cost at most 1.3 dB.

BibTeX

@article{ai2026graftray,
  title   = {GraftRay: Physically Based Mesh Insertion into
             Gaussian Splatting Scenes via Path Tracing},
  author  = {Ai, Jinxin and Nie{\ss}ner, Matthias},
  journal = {arXiv preprint arXiv:2610.37036},
  year    = {2026}
}

This website is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

The source code of the website is borrowed from here.