OmniScan AR — Spatial knowledge graph for industrial maintenance

In short

OmniScan AR is what I'm building now: point a smartphone or a Meta Quest 3 at an industrial machine, and it recognises the machine, anchors the documentation and an interactive 3D twin around it, and answers spoken questions about it.

It's a solo capstone, planned as three two-month phases, each broken into sprints and tracked on a public GitHub project board. This page describes what exists today and what's planned — and keeps the two apart.

The problem

Maintenance technicians lose time on the shop floor searching paper manuals and PDFs for the page about the part in front of them. The information exists; getting from this machine, this component to that paragraph is the slow part.

That makes it a lookup shaped like a graph — machine, component, document, 3D asset — with a camera as the query. And it has to run on two very different devices: a phone, which every technician already carries, and a headset, which frees both hands.

Architecture

Built so far (phase 1):

A Neo4j knowledge graph — machines, components, documents, 3D assets and vision labels as nodes, a uniqueness constraint on each, and seed data to develop against.
A FastAPI backend with a repository layer over the graph and typed schemas, returning a machine's whole neighbourhood in one call: GET /api/v1/machines/{id}/subgraph.
A YOLOv8 vision service — its own FastAPI microservice that classifies a camera frame, with the infrastructure in place to fine-tune the model on the target machines.
The Unity client's foundations: AR Foundation initialised, a typed API client for both services with explicit result types, and the interaction interface the phone and headset versions will share.
Docker Compose runs the backend stack.

Planned: documentation anchored in AR on the phone and in mixed reality on the Quest 3 with hand tracking; the interactive digital twin; a voice assistant over a retrieval pipeline grounded in the graph; and a Next.js back office to manage the data for every device.

Decisions & trade-offs

One Unity codebase for phone and headset

The central bet: the same client compiled for iOS and Android and for Quest OS, aiming for more than 85% shared code. The phone is the product a maintenance team can adopt tomorrow; the headset is the premium tier. Two apps would mean every feature built twice and drifting apart. The interaction layer is an interface from the start, so touch and hand tracking are two implementations of one contract rather than two codebases.

A graph, because the questions are relationships

"Where's the manual for the pump on this line?" is three hops: machine, component, document. In a relational schema that's a join that grows with every new kind of link; in a graph it's the natural query. Returning a machine's subgraph in one request is exactly what the AR view needs to lay everything out around the machine at once — and it will ground the assistant, so answers are about this machine rather than machines in general.

Vision as its own service

The recognition model will be retrained and swapped more often than anything else in the system. Behind its own HTTP endpoint, fine-tuning YOLOv8 on new machines never touches the client or the graph — the Unity app depends on a response shape, not on a model.

What's hard so far

Scope. The full vision — two platforms, computer vision, a graph, generative AI, a back office — is a year of work for a team, and I'm one person with two-month phases. Delivering a working slice at the end of each phase, rather than every layer half-built, is the discipline the whole project depends on.

The other is that it lives in physical space. A detection model is only as good as the photographs it was fine-tuned on, and an anchor that drifts by ten centimetres puts the manual on the wrong part.

Where it stands

Phase 1 of 3. The backend stack, the knowledge graph, the vision service and the client's networking layer are merged, through reviewed pull requests against planned issues. The typed client already calls both services; wiring recognition to the subgraph and anchoring the result in AR on a phone is the loop that turns these parts into a product.

What I'd do next

That first loop on a phone. Then the Quest 3 build on the same code, with passthrough and hand tracking; the interactive digital twin; and the voice assistant grounded in the graph.