Two years ago, if you wanted a photorealistic 3D representation of a real-world space, you had two reasonable options: hire someone with a LiDAR scanner and photogrammetry software, or accept that the result would look like a video game from 2015.

3D Gaussian splatting (3DGS) has changed that. Walk around a space with your iPhone for five minutes, process the footage through one of a growing number of tools, and you end up with a photorealistic 3D scene you can view in a headset or browser. Not perfect — the technology has specific failure modes — but the quality-to-effort ratio is genuinely remarkable.

Here’s what it is, how it works at a level that’s actually useful for practitioners, and where it fits into the current XR content pipeline.

What Gaussian splatting actually does

Traditional 3D capture from images (photogrammetry) works by finding matching points across multiple photos, reconstructing a 3D point cloud, then building a mesh and projecting textures onto it. The mesh step is where most quality loss happens — the software has to make decisions about surface topology that are often wrong for complex organic geometry, glass, fur, foliage, or anything with complex reflectance properties.

Neural Radiance Fields (NeRF) took a different approach: instead of explicit geometry, train a neural network to represent the scene as a continuous volumetric function that predicts the colour and density of any point in space. The results were photorealistic but slow — rendering a NeRF in real-time wasn’t feasible on consumer hardware.

Gaussian splatting replaces the neural network with an explicit representation: the scene is described as millions of small 3D Gaussian “splats” — think translucent ellipsoids — each with a position, orientation, scale, colour, and opacity. Rendering is a matter of projecting these splats onto the screen and blending them. It’s fast enough to run in real-time at 60fps on a modern GPU, and the quality is comparable to NeRF while being much more practical for interactive use.

The training process still starts from photos or video. You shoot footage (or capture stills) from many angles, run Structure from Motion (SfM) to get camera positions and a sparse point cloud, then optimise the Gaussian splats to match what the cameras saw. The output is a .ply file or similar format containing all the splat data.

How the quality actually holds up

The honest answer: excellent in the right conditions, patchy in others.

Where Gaussian splatting excels:

  • Indoor scenes with good lighting and textured surfaces
  • Architecture and real estate walkthroughs
  • Objects photographed in controlled conditions
  • Scenes with complex geometry that would trip up mesh-based photogrammetry (foliage, fabric)

Where it struggles:

  • Specular reflections and mirrors (the optimiser can’t handle view-dependent effects well)
  • Moving objects during capture (people, plants in a breeze)
  • Uniform or low-texture surfaces (plain white walls become muddy)
  • Very large outdoor scenes (works but requires careful capture methodology and significant compute)

The edges of the captured area often look bad — you need to explicitly discard splats outside the scene boundary, otherwise you get “floaters” (misplaced splats that look like artefacts when you move through the scene).

Practical capture workflow

You don’t need specialised hardware. A modern smartphone is sufficient for most interior spaces.

Capture:

  • Shoot at the highest resolution your device supports, ideally 4K 30fps
  • Move slowly and steadily — fast camera movement causes motion blur that breaks the SfM step
  • Cover the space from multiple heights and angles; don’t just walk around at eye level
  • Plan for 100-300 frames per major viewpoint; overlap is important
  • 200-400 frames total is typical for a medium room; more for large spaces
  • Avoid changing lighting conditions during capture

Processing: Several software options are available, ranging from local GPU-intensive processing to cloud pipelines:

  • Luma AI: Upload video, process in the cloud, download the splat. The easiest entry point. Free tier is limited but sufficient for experimentation.
  • Polycam (iOS/Android): Capture in-app, process on-device or in the cloud. Good for smaller spaces.
  • Postshot: Local processing, good quality, requires a decent GPU (RTX 3080 or better for reasonable training times).
  • NVIDIA Instant3D / gsplat: Research-quality tools for more control over the training process.

For production quality, local training with control over training parameters produces better results than cloud pipelines, but requires hardware investment. For experimentation or one-off captures, cloud tools are the right starting point.

Getting splats into XR

This is where it gets practical.

WebXR / Web: Three.js has Gaussian splat rendering support via the @mkkellogg/gaussian-splats-3d library. You can embed an interactive splat viewer in a web page that works in browser-based WebXR. Niantic’s 8th Wall platform supports Gaussian splatting natively.

Unity: The UnityGaussianSplatting package (open source, available on GitHub) renders splats in Unity. Performance in VR is dependent on splat count and the target hardware — Quest 3 can handle around 1-2 million splats at acceptable frame rates, with quality/performance tradeoffs at higher counts.

Unreal Engine: Community plugins exist, and Unreal’s Lumen rendering pipeline has had experimental splat support since late 2025. Stability for production use is improving.

Apple Vision Pro: Spatial video captures (captured in 3D using the Vision Pro’s dual cameras) are a different technology, but there’s active development on splat viewers for visionOS. Polycam exports to a format viewable on Vision Pro.

Meta Quest: The Meta Presence Platform doesn’t have first-party splat support, but Unity renders splats on Quest 3 via the community package. Frame rate depends heavily on splat density.

Where this fits in the XR content pipeline

For the kind of use cases where you want a photorealistic recreation of a real-world space — real estate walkthroughs, cultural heritage preservation, retail environments, industrial training scenarios, virtual location scouting — Gaussian splatting has become the practical standard. The alternatives either require expensive hardware (LiDAR + professional photogrammetry) or produce lower-quality results.

It’s not a complete replacement for purpose-built 3D assets. If you need an interactive environment where users can pick up objects, walk through walls, or modify the space, you still need traditional 3D geometry. Splats are static representations — you can move through them but not interact with the scene geometry in a physical sense.

The combination that’s emerging in production workflows is using splat captures for environmental reference and background fidelity, combined with traditional 3D assets for interactive elements in the foreground. The splat provides a convincing sense of place; the geometry provides the interactivity.

Getting started today

The lowest-friction path: install Polycam on your iPhone, capture a room, and view the result in the Polycam viewer. That gives you a feel for the quality achievable with a basic capture. If it’s compelling, the next step is experimenting with Luma AI for slightly better quality, and then local training tools once you have a specific use case that justifies the GPU investment.

The technology is moving fast. What took hours of training on a research workstation in 2024 now runs in minutes on a consumer GPU. Viewer support across XR platforms is improving every quarter. If you’ve been watching Gaussian splatting from the sidelines, the tooling is now mature enough to experiment practically.