Responsive STEP Handling Architecture

Problem Statement

Large vendor STEP files, such as camera bodies and imaging lens barrels, can make Open 3D interaction lag. The visible symptom is slow hover feedback: moving the mouse over a CAD body waits a long time before an edge or face is highlighted, and every small mouse move can restart the same lag cycle.

The same vendor files feel responsive in FreeCAD and normal CAD tools because those tools keep the STEP model as CAD topology first. They tessellate for display, but they do not treat the display mesh as the only source of truth for selection, face identity, and assembly state.

Current Kraken Bottleneck

Kraken currently uses a mesh-first path for imported CAD in several places:

  • cad_import_service.convert_step_to_stl converts STEP/IGES through Gmsh into STL. This is useful for the existing optical-solid kernel, but it loses STEP topology, assembly labels, stable face identity, and exact edge curves.

  • Imported STEP overlay picking can synthesize a temporary optical-solid row plan via _step_overlay_optical_solid_row_plan just to resolve a face. That can involve transformed mesh generation, STL save/read paths, mesh diagnostics, scene placement metadata, and face metadata normalization.

  • Hover/right-click picking can read STL triangles from disk, ray-test face triangles, rebuild planar outlines, and run PyVista/VTK feature-edge extraction during the interaction path.

  • _picked_feature_info scans the actor’s cells to find a coplanar connected component for a picked triangle. This is roughly proportional to mesh cell count for a new hovered cell, so dense camera/lens STEP meshes are painful.

  • Triangle/cell IDs from generated STL are used as face-selection stand-ins. Those IDs are display artifacts; they are not stable STEP topology IDs.

This explains why the UI can lag even when the renderer itself can draw the model. The renderer is not the only cost. The selection path is recomputing geometry and topology-like information after the mouse moves.

What CadQuery Shows

The local CadQuery source is useful as an architecture reference. It does not solve the problem by magic; it shows the right layering.

  • Simple STEP import uses OpenCascade STEPControl_Reader and returns TopoDS shapes wrapped by CadQuery Shape objects. The STEP shape is retained as CAD topology, not immediately collapsed to STL.

  • Assembly/color/subshape workflows use XCAF concepts such as STEPCAFControl_Reader, XCAFDoc_DocumentTool, document explorers, and subshape labels. This is the path that can preserve vendor assembly names, colors, and subshape metadata.

  • Shape.mesh checks whether OpenCascade triangulation already exists with BRepTools.Triangulation_s before calling BRepMesh_IncrementalMesh. The triangulation is cached on the shape instead of recreated per hover.

  • Shape.tessellate iterates CAD faces and pulls each face’s existing triangulation through BRep_Tool.Triangulation_s. That pattern naturally creates a face-id to triangle-range map.

  • Shape.toVtkPolyData uses OpenCascade’s IVtk bridge (IVtkOCC_Shape, IVtkOCC_ShapeMesher, IVtkVTK_ShapeData) and then splits display edges from display faces once for rendering.

  • CadQuery assembly export keeps a compound cache to avoid remeshing the same repeated object instance.

The Kraken development environment currently has pythonocc-core available as OCC. It does not currently have CadQuery or cadquery-ocp/OCP installed. That means the first implementation should use the same OpenCascade architecture with the available OCC backend, while keeping a CadQuery/OCP adapter as an optional study path.

Target Architecture

The fundamental change should be a persistent CAD scene cache. Imported STEP should be loaded once into a document/shape cache and then exposed through separate display, picking, and optical-physics views.

Proposed service boundary:

CadDocumentCache

Owns the source file key, file timestamp/size/hash, unit, backend, and loaded OpenCascade document/shapes. This is the authoritative STEP state.

CadTopologyIndex

Owns stable solids, shells, faces, edges, bounding boxes, centroids, normals, areas, surface type hints, assembly names, colors, and source labels. It should qualify face IDs by component/solid, not only F001.

CadDisplayMeshCache

Owns coarse/idle display tessellations, VTK/PyVista polydata, edge polydata, actor references, and cell-data arrays mapping each display cell back to component_id and face_id.

CadPickCache

Owns lightweight pick proxies: bounding boxes, face polygons, internal planes, and prebuilt highlight outlines. Hover should resolve actor/cell to face ID from cached data, not reread STL or rebuild face outlines.

CadOpticalAdapter

Owns the mesh/triangle view used by current optical tracing. Short term it can still expose triangles, but with stable face IDs and no repeated disk reads. Longer term it can add exact OpenCascade ray/surface intersection or a BVH over cached triangles.

Interaction Rules

The Open 3D interaction loop should become cheap:

  1. Import STEP in a worker thread/process. Show a bounding box placeholder and progress/status text while topology and display caches are built.

  2. Create body and edge actors once from cached polydata. During drag/rotate, update actor transforms only; do not remesh and do not rebuild face metadata.

  3. On hover, use VTK picking to get actor/cell ID, then lookup face ID through cached cell data. If the actor/cell/face is unchanged, do nothing.

  4. Highlight using prebuilt face outline actors or cached face-boundary polydata. Do not run extract_feature_edges on mouse move.

  5. For transparent solids and internal beam-splitter planes, use pick proxies or a controlled pick list. Prefer internal pickable faces only when the active command needs them.

  6. Right click should assign physics to the cached face ID. Promotion to a row must reuse the same face identity instead of generating a separate temporary mesh with different face labels.

  7. Use level of detail: coarse mesh while interacting, refined mesh when idle. Optical tracing must use the validated optical mesh/cache, not the transient display LOD.

Expected Improvement

The first major win is not exact NURBS tracing. It is removing expensive work from the hover path.

With the cache architecture, hover should be reduced to:

  • VTK pick.

  • actor/cell/face lookup.

  • optional highlight actor visibility or transform update.

  • render.

That is the same basic model used by CAD tools: topology and tessellation are prepared ahead of interaction, while hover only switches selection state.

Implementation Plan

Phase 1: Profile And Freeze The Current Baseline

Add a validator/diagnostic that imports a large camera or imaging-lens STEP and reports import time, triangle count, actor count, hover-pick latency, and highlight latency. The initial target should be p95 hover latency under 50 ms, then under 16 ms for ordinary display-only hover. The branch now includes python -m KrakenOS.UI.diagnose_open3d_hover_latency as the headless cache/contract diagnostic; GUI picker timing can be added to that command when a stable display backend is available in CI.

Phase 2: Cache The Existing Mesh Path

Before changing import backends, remove repeated work:

  • cache transformed STEP overlay meshes by source path plus transform;

  • cache STL triangle arrays in memory;

  • cache face metadata for imported overlays;

  • cache face-id to outline polydata;

  • remove temporary row-plan generation from hover/right-click face lookup.

This phase should improve responsiveness without changing the optical tracing contract.

Current branch progress: the mesh-path responsiveness foundation is complete. Open 3D imported STEP hover and transparent face-pick paths now reuse cached triangle arrays, cached face triangle slices, and cached face-outline artifacts for the existing mesh path. Ordinary passive hover is intentionally lightweight: it does not pick dense CAD body actors. It only uses a rotation-handle actor pick list, then defers transparent face-ray testing, feature scanning, and outline generation to explicit click, right-click assignment, or axis/face-pick commands.

Phase 3: Add An OpenCascade Topology Adapter

Build a new adapter using the installed pythonocc-core/OCC backend:

  • import STEP using STEPControl_Reader for simple shapes;

  • study STEPCAFControl_Reader for vendor assembly names, colors, and subshape labels;

  • traverse solids/faces/edges with OpenCascade topology tools;

  • mesh once with BRepMesh_IncrementalMesh;

  • write VTK cell arrays for component_id and face_id.

Phase 4: Compare CadQuery/OCP

Install CadQuery/OCP only in an isolated optional environment and compare:

  • STEP import fidelity;

  • XCAF assembly/subshape metadata access;

  • IVtk meshing speed and display quality;

  • face/edge identity stability;

  • package size and deployment cost.

If CadQuery/OCP is better, add it behind the same adapter interface. Do not make it a required runtime dependency until this is proven.

Phase 5: Integrate With Optical Physics

Keep the North Star rule: display and UI convenience must not hide or corrupt native optical state. The topology cache must preserve face function, coating, material, detector/absorber roles, ray event metadata, and CSV/export diagnostics. If exact OpenCascade surface intersection is added later, it should be validated against the current triangle-based non-sequential trace before becoming default.