DenseField: NeRF Volume Renderer for Unreal Engine

A high-performance Neural Radiance Field (NeRF) volume renderer plugin for Unreal Engine 5, featuring cross-platform CPU and GPU (Metal) rendering backends with real-time view-dependent lighting and depth occlusion.

Overview

DenseField implements the volrend algorithm (PlenOctrees) to render pre-trained NeRF models directly within Unreal Engine 5 scenes. It supports:

• ✅ Real-time rendering of NeRF models composited over live UE5 scenes
• ✅ Spherical Harmonics (SH) view-dependent lighting (degree 0–4)
• ✅ Depth occlusion — NeRF objects correctly occlude and are occluded by UE5 geometry
• ✅ Cross-platform — CPU and Metal GPU rendering backends
• ✅ UE5.7+ compatibility (tested on Mac, extensible to Windows/Linux)

Repository Structure

DenseField/
├── Plugins/DenseField/          # UE5 Plugin (drop into any UE5 project)
│   ├── Source/
│   │   ├── Public/              # Plugin headers and component classes
│   │   ├── Private/             # Plugin implementation
│   │   └── ThirdParty/          # External dependencies (e.g., cnpy)
│   ├── Shaders/                 # HLSL shader sources
│   └── NeRFVolRendPlugin.uplugin # Plugin manifest
├── native/                       # Standalone native (C++) implementations
│   ├── src/                     # Core math, octree, rendering
│   ├── CMakeLists.txt           # CMake build configuration
│   └── shaders/                 # GLSL shader reference (CPU path-tracing)
└── README.md                    # This file

Quick Start

For UE5 Integration

1. Copy the plugin to your UE5 project:

   cp -r Plugins/DenseField <your-ue5-project>/Plugins/

2. Regenerate the project:

   cd <your-ue5-project>
   /Volumes/UEDRIVE/Epic\ Games/UE_5.7/Engine/Build/BatchFiles/Mac/Build.sh \
     <ProjectName>Editor Mac Development <path-to-uproject>

3. Enable the plugin in UE5:

   • Open your project
   • Edit → Plugins → Search for "DenseField"
   • Enable it and restart the editor

4. Add a NeRF component to your scene:

   • Create an Actor
   • Add a "NeRF Render Component"
   • Set properties: NeRF origin, scale, data file path
   • Add a Post Process Volume with the M_NeRFComposite material

See SETUP.md for detailed step-by-step instructions.

For Standalone Native Development

Build the CPU/Metal renderer:

cd native
cmake -S . -B build && make -C build

Outputs test renders to output.ppm / output.png.

Key Features

NeRF Model Loader

• Loads .npz files (NumPy archive format)
• Automatically converts fp16 → fp32 using cnpy library
• Tested with lego.npz, hotdog.npz (PlenOctree format)

Coordinate System

• UE5: x-forward, y-right, z-up (cm)
• NeRF: x-right, y-forward, z-up (NeRF units)
• Automatic conversion via NeRFOriginInWorld and CmPerNeRFUnit parameters

Rendering Pipeline

1. CPU/Metal trace rays through octree
2. Spherical Harmonics evaluate view-dependent color
3. Depth tracking for occlusion handling
4. Composite over UE5 scene in post-process material

Depth Occlusion Strategy (Metal/Mac)

On Metal, reading SceneDepth in plugin shaders causes issues. Instead:

• Render NeRF depth to a separate render target (RT_NeRFDepth)
• Compare depths in post-process material (M_NeRFComposite)
• Correct occlusion without runtime shader hacks

See Plugins/DenseField/DEPTH_OCCLUSION_WRITEUP.md for technical details.

Requirements

For UE5 Plugin

• Unreal Engine 5.7+ (Mac, tested on Apple Silicon/Intel)
• Xcode 14+ with Metal toolchain
• HLSL Compiler (part of UE5 distribution)

For Native CPU/Metal Builds

• CMake 3.15+
• C++17 compiler (clang/MSVC)
• Metal SDK (macOS only)

Data Files

• NeRF model in .npz format (e.g., lego.npz from PlenOctree dataset)
• Place in project root or specify path in component properties

Architecture

Plugin Components

NeRFVolRend Module:

• FNeRFRenderComponent — Actor component exposing NeRF rendering
• FNeRFViewExtension — Engine hook for frame-time rendering (RDG integration)
• FNeRFShaders — HLSL shader definitions and bindings

Shaders (HLSL):

• NeRFRender.usf — Ray tracing and volumetric rendering
• NeRFComposite.hlsl — Post-process material for compositing + depth testing

Native Code

Core Modules:

• NeRFMath.h — Vector/matrix math, fp16↔fp32 conversion
• N3TreeLoader.h — NPZ parsing, octree loading
• OctreeQuery.h — Volumetric octree traversal (CPU port of volrend)
• VolumeRenderer.h — Ray tracing and SH evaluation
• SHEval.h — Spherical Harmonics basis functions (degrees 0–4)
• MetalRenderBackend.mm — GPU-accelerated rendering (Metal)

Usage Example

In C++

// Create component
ANeRFActor* NeRFActor = GetWorld()->SpawnActor<ANeRFActor>();
FNeRFRenderComponent* NeRFComponent = NewObject<FNeRFRenderComponent>(NeRFActor);
NeRFComponent->NeRFOriginInWorld = FVector(245.f, 245.f, 200.f);  // cm
NeRFComponent->CmPerNeRFUnit = 100.f;
NeRFComponent->RegisterComponent();

In Editor

1. Select your Actor
2. Add Component → "NeRF Render Component"
3. Set NeRF Origin and Scale in Details panel
4. Load .npz file via the component property
5. Position camera and play

Coordinate System Reference

Axis	UE5	NeRF	Mapping
X	Forward	Right	NeRF_X = UE5_Y
Y	Right	Forward	NeRF_Y = UE5_X
Z	Up	Up	NeRF_Z = UE5_Z

Scaling: 1 NeRF unit = CmPerNeRFUnit cm (default 100 = 1 meter)

Performance Notes

• CPU path: Single-threaded ray tracing, suitable for offline preview
• Metal GPU path: 100–200M rays/sec (M1/M2), real-time at 1080p
• Optimization: Adaptive step size, early-exit on low density
• Memory: ~2–4GB for typical NeRF octrees

Troubleshooting

"Plugin failed to compile"

• Ensure UE5.7+ is installed and Build.sh path is correct
• Check Xcode version (14+) and Metal toolchain
• See SETUP.md for detailed compilation steps

"NeRF not visible / black"

• Verify .npz file path in component properties
• Check NeRF Origin position (should be near where NeRF sits in world)
• Ensure Post Process Material M_NeRFComposite is applied
• Camera must be within reasonable distance (typically 100–500 cm from origin)

"Incorrect occlusion"

• Metal-specific: See depth occlusion writeup
• Check render target format (must be RGBA16f or RGBA32f)
• Ensure post-process material is in correct Post Process Volume

License

This project is licensed under the MIT License — see the LICENSE file for full details.

In summary: You are free to use, modify, and distribute DenseField in commercial and private projects, as long as you include a copy of the license and copyright notice.

Attribution

If you use DenseField in your project, we appreciate (but don't require) attribution:

DenseField by Miles Labrador
https://github.com/MilesLabrador/DenseField

References

• volrend: PlenOctrees for Real-time Rendering
• NeRF: Neural Radiance Fields
• UE5 Plugin Docs: Unreal Engine Plugin Development

About This Project

DenseField is a personal project exploring real-time NeRF rendering within Unreal Engine.

Authors

• Miles Labrador — Core implementation, UE5 integration

---

Last Updated: May 2026
Status: Stable (POC Phase)