Virtual Production at Notre Dame Studios
Overview
At Notre Dame Studios, I built real-time virtual production capabilities using Unreal Engine 5, stYpe tracking, custom C++ integrations, real-time depth and pose approximation, and digital-twin-based workflows. The work required translating leadership goals into technical milestones, testing hardware and software options, building proof points for production and budget decisions, documenting repeatable workflows, and helping train the next team to operate the capability.
This includes real-time gaussian splats in UE5, digital-twin workflows, depth-based scene understanding, relighting experiments, production-ready documentation, and reusable systems designed to help future operators reproduce and extend the capability.
Updates
Updates here are shown in reverse chronological order
Update 9
Trained new team members on the existing pipelines and workflows to ensure a smooth transition.
Update 8
Experimented with using Gaussian Splats in UE5 for real-time rendering of real locations.
A complete pipeline from capturing raw video to a complete splat in UE5 was built and documented.
Raw footage is aligned using RealityScan, then the colmap dataset is color corrected using Davinci Resolve so the splat's lighting matches the studio lighting, then the colmap dataset is imported into LichtFeld and the gaussian splat .ply is exported to UE5.
Update 7
Key assets and workflows have been shipped in a UE5 plugin for easy reuse and management across projects.
The plugin is version controlled using git and is stored on github.
Update 6
Documentation based in Obsidian with a simplified version exported to pdf as well for easy viewing. Including both Obsidian and pdf versions from the same base allow documentation combining top-level information with the ability to dive into details necessary for future development.
Obsidian integration allowed for better hyperlinks, integrated flow-charts to visually represent key functionality and hardware setups, and a more dynamic format that can be easily updated as the project evolves.
Update 5
Built a offline rendering pipeline for compositing using depth maps and re-lighting techniques. Depth maps generated by modified video-depth-anything pipelines to output raw 32-bit .EXR image sequences while preserving the uniform depth across the entire sequence.
Geometry was generated using geometry nodes in Blender then exported to UE5 as a .usd animation. Source lighting was baked from an HDRI using Cycles in Blender then imported into UE5 as an image sequence. The baked lighting was divided out of the original footage to achieve a "de-lit" equivalent footage in a shader inside UE5.
The de-lit footage was re-lit by the UE5 scene including reflections and shadows.
The original depth map was generated using Video Depth Anything modified to export 32-bit raw depth estimations as .exr image sequences.
After the depth maps have been generated, they were converted into a digital twin using Geometry Nodes in Blender. Each pixel was scaled away from the camera origin such that from the camera's perspective the pixel does not change location. The geometry was then exported as a .USD geometry cache and sent to UE5.
OpenUSD was chosen for its support of complex animated geometry and seamless integration into UE5 using geometry caches with UE5's Sequencer.
In Blender, a HDRI of the studio was used to bake the original lighting on the geometry so it could be removed once the the geometry was imported into UE5.
From the camera's perspective in UE5, the footage looks like the original footage, but since it is actual geometry it reflects the lighting on the scene.
The geometry with a light moving in front of it
Finally, the geometry gets placed in a UE5 scene. Since it is actual geometry, it supports lighting, reflection, and placing different sections of the footage at different depths in the scene.
In this example, James is behind the desk, and the desk includes the his reflection on the surface.
Update 4
Built a news studio set. Implemented custom blueprints for lighting using rect lights and splines to better use the strengths of megalights for scenes with many light sources.
Also built custom master materials for the Maya to UE5 pipeline to allow artists to easily import materials from Maya with the correct settings and parameters for UE5.
Update 3
Built a pipeline for converting gaussian splats into usable models with human-usable UV maps and continuous meshes. Also wrote documentation on the process so it could be replicated by future team members.
The process is automated in Blender using a custom python script.
Update 2
Explored UE5's experimental Neural Network Engine (NNE) and built three proof of concept workflows with a custom blueprint library in c++ for handling common operations.
These workflows were shelved due to performance limitations of NNE and the desire to utilize CUDA.
Update 1
Initial work on the project primarily focused on creating interactive elements that can be used in real-time by the talent without requiring input from someone else. This gives the talent more control over what they are doing and lowers the staffing requirements per scene, allowing smaller teams to produce interactive content.
All interactive elements utilize polymorphic blueprints for easy reuse and interchangeability.