LifeSim Interiors: From One Photo to a 3D Room You Can Take Apart

1 Introduction

A 3D scene is more than a surface. It is a room you could rearrange, a sofa you could move, a shelf you could replace. A system that understands space has to recover those parts, not just a single frozen mesh.

LifeSim Interiors generates an environment plus every object as its own mesh. It builds on recent work in object-to-scene distillation and on Microsoft's TRELLIS.2 [1]. We built the full stack around it in two days on Google Cloud. Our contributions are:

• A world factory: text becomes images, images become 3D assets, and assets become physically valid rooms, at scale (Section 2).
• A scene generator: a 3D object model extended to generate an environment and many objects jointly, in one shared frame (Section 3).
• Object refinement: each object is regenerated in its own frame at higher detail, conditioned on the rest of the scene (Section 3.3).
• Measurement: a public benchmark protocol, plus our own method for placing generated scenes in a real camera's frame (Section 4).

2 Manufacturing worlds

There is no large dataset of 3D rooms broken into separate parts, so we manufacture one. A pretrained object generator makes the parts, and our pipeline assembles them into rooms.

2.1 From words to parts

Every object and every room starts as a short description. An image model draws it, and TRELLIS.2 lifts the drawing into a textured 3D asset (Figure 1).

2.2 The hardest asset is an empty room

An empty room is surprisingly hard to generate. Standard background removal treats pale walls and floors as background, so our first “empty living room” came back as a window and two curtains. Masking environments by colour keeps the whole room intact (Figure 2).

2.3 Rooms that obey physics

Each object is dropped onto a real support surface. Anything that floats or passes through a wall is rejected, and collisions are resolved. Cameras aim at the furnished region, with the field of view fitted to every object, and only views in which every object is visible are kept (Figure 3).

2.4 At scale

In under an hour, the factory produced 5,500 assets across 16 room types and 89 object categories. From these it composed 10,216 training scenes, with about six objects each (Figure 4).

Scale exposed a problem the pilot had hidden: image models often ignore “no furniture” and draw furnished rooms anyway, which teaches the opposite of decomposition. Our geometric filter looks down on each room and measures how much of the floor is free. It kept 537 of 1,493 environments.

3 Generating scenes, not objects

3.1 One sequence for the whole room

All components of a scene, one environment and up to thirty objects, share a single token sequence over a frozen 3D generator. Every object can attend to every other object and to the room around it.

3.2 Starting from what the model already knows

Learnable type and slot embeddings tell the environment apart from objects and objects apart from each other. They start at zero, so on the first training step the model behaves exactly like the pretrained one. Only lightweight adapters are trained.

3.3 Refinement in the object's own frame

Objects occupy only a few voxels of a room-sized grid. Refinement moves each object into its own normalized frame, regenerates it while attending to the whole scene, and places it back exactly where it was.

3.4 Architecture

• Base model. TRELLIS.2-4B [1], a sparse-voxel image-to-3D generator with flow-matching transformers [3] for structure, shape and texture.
• Adapters. LoRA [2] at rank 32 on every attention and MLP layer: 37M trainable parameters on a frozen backbone.
• Factorized attention. Dense attention over a 30-component scene costs 43 seconds per training step. In our design, five of thirty layers attend across the scene and the rest attend within each component, the regime the base model was pretrained in. This is about 3× cheaper.

4 Results

4.1 Alignment to the camera

Real-world evaluation needs predictions in the camera's metric frame. We register the generated scene to the visible depth and score each candidate pose by depth-map consistency. Without this, a 5% scale error alone costs 26 points of F-score.

4.2 Model scaling

LifeSim Interiors improves with training compute. On held-out synthetic scenes, the F-score at 10 cm (higher is better) climbs at every checkpoint, gaining about nine points each time training doubles (Table 1).

The pipeline itself is sound: decoding ground-truth latents and scoring them as predictions gives an F-score of 100%. The model also scores about the same on scenes it trained on as on new ones, so it is still undertrained, not overfit. The model has seen a small fraction of the training compute this kind of method is usually given.

5 What comes next

The model is learning; we stopped early. Three steps come next: train longer, compose more realistic rooms, and generate truly empty environments.

We already have a head start on the second step. Our text-to-interior lab writes room layouts with an LLM and checks every asset with a vision model. Feeding those rooms into the world factory is our next experiment.

The model at the center is compact. Most of the real work lives around it: manufacturing believable worlds, rendering them efficiently, and measuring results honestly against the real one. We'll share more as LifeSim Interiors matures.