6D pose localization and detection6D 位姿定位与检测

Localization and detection share estimate_frame_many_categories_many_instances: predict 2D regions, estimate their poses in batches, then select the outputs. Localization supplies object identities and instance counts through obj_ids and inst_count; detection leaves instance counts unconstrained. Both use RGB-D, camera intrinsics and CAD meshes, and share the score thresholds and output format below.

定位与检测共用 estimate_frame_many_categories_many_instances：预测 2D 区域，批量估计位姿，再筛选输出。定位通过 obj_ids 和 inst_count 提供物体类别及实例数量；检测不预设实例数量。两者均使用 RGB-D、相机内参与 CAD 网格，并采用下文相同的分数阈值和输出格式。

Specify categories and instance counts指定类别与实例数量

The table describes API input combinations. Known-count localization requires both obj_ids and inst_count. Passing inst_count=None leaves the number of instances unconstrained; with obj_ids, it restricts the candidate categories rather than specifying the known-count localization task.

下表列出接口支持的输入组合。已知数量的定位需要同时给出 obj_ids 与 inst_count。若设置 inst_count=None，实例数量便不受约束；此时 obj_ids 仅限定候选类别，不构成已知数量的定位任务。

A single category with a single instance is a localization case too, when its region must first be found. Category count and instance count are constraints on the same frame pipeline. If regions are already supplied, use the pose-only entries instead; whether there is a 2D detector is independent of whether you have one or many instances.

需要先寻找区域时，单类单实例也属于定位用法。类别数与实例数是同一整帧流程上的约束。区域已经给定时，直接使用位姿估计入口；是否需要 2D 检测，与处理单个还是多个实例是两个独立维度。

from wapr.frame import estimate_frame_many_categories_many_instances

# Search for driller regions, then retain at most one estimated pose.
# 先搜索电钻区域，再最多保留一个估计位姿。
poses, timing, detections = estimate_frame_many_categories_many_instances(
    estimator, detector, rgb, depth_m, K, meshes,
    obj_ids=[8], inst_count={8: 1},
)

For frame inference, pass a mesh map made from the prepared snapshots. timing["pose_hot_seconds"] excludes setup and warmup; mesh_setup_seconds, pose_warmup_seconds and detector_warmup_seconds record those separately. hot_frame_seconds adds the measured detection/filter segment and pose segment, excluding the intervening warmup gap and final output assembly. It is not a continuous camera latency. A new pose workload receives three discarded warmup calls before its reported pose time.

整帧推理传入由已准备快照构成的网格字典。timing["pose_hot_seconds"] 不包含准备与预热；mesh_setup_seconds、pose_warmup_seconds 和 detector_warmup_seconds 分别记录这些阶段。hot_frame_seconds 相加实测检测/过滤阶段和位姿阶段，排除中间的预热间隔与最终记录整理，不是相机连续延迟。新位姿负载在报告耗时前丢弃三次预热调用。

The count cap is applied after candidate pose estimation, ranked by score_6d. A cap of one neither guarantees one detection nor limits computation to one candidate. The numbers above illustrate configuration, not the contents of every scene.

数量限制在候选位姿估计之后按 score_6d 排序应用。限制为一，不保证一定检测到一个，也不代表只计算一个候选。表中数字是配置示例，不表示每个场景都含这些实例。

examples/04_6d_localization.py is the localization call on this page. The categories and the count of each category are known before the image is read. The frame is LM-O^[2] scene 2, image 307. test_targets_bop19.json names eight categories, ids 1, 5, 6, 8, 9, 10, 11, and 12, and each count is 1. The script writes those two values as obj_ids and inst_count, then calls estimate_frame_many_categories_many_instances. A category that is not in obj_ids is not detected and is not estimated. inst_count is the most that call keeps of that category, higher score_6d first. It does not require the detector to find that many. This call is not estimate_one_category_one_instance. The detection configuration below leaves inst_count unset.

examples/04_6d_localization.py 就是这一节的定位。类别，以及每一类的数量，在读图之前就已经知道。这一帧是 LM-O^[2] 场景 2 第 307 帧。test_targets_bop19.json 写了八个类别，编号 1、5、6、8、9、10、11、12，每个数量是 1。脚本把这两个值写成 obj_ids 和 inst_count，再调用 estimate_frame_many_categories_many_instances。不在 obj_ids 里的类别不检测，也不估计。inst_count 是这一类这次最多留下几条，score_6d 高的在前。检测器不一定找得到这么多。这次调用不是 estimate_one_category_one_instance。下文的检测配置不传 inst_count。

Build the arguments first, then pass them in. The copy-this form is a custom folder plus one mask file. load_scene in wapr/scene_files.py returns the image, the depth, the camera, and the meshes of every category in that folder. It does not return a mask, and it does not return a pose. examples/06_custom_scene.py calls that same function, then passes rgb, depth_m, K, and meshes to estimate_frame_many_categories_many_instances. That frame call supports one or several categories and instances. For a supplied single region, take one mesh from the dict and call estimate_one_category_one_instance; to find that region first, use the frame entry with one category and a count cap of one.

先把参数造出来，再传进去。可照抄的写法是一个自定义目录加一张 mask。wapr/scene_files.py 的 load_scene 返回图像、深度、相机，以及这个目录里每个类别的网格。它不返回 mask，也不返回位姿。examples/06_custom_scene.py 调用的是同一个函数，然后把 rgb、depth_m、K、meshes 传给 estimate_frame_many_categories_many_instances。整帧调用支持单个或多个类别与实例。已有单个区域时，从字典中取出一份网格，调用 estimate_one_category_one_instance；需要先寻找区域时，使用整帧入口，并指定一个类别及数量上限一。

import cv2
from wapr import WAPREstimator
from wapr.scene_files import load_scene

rgb, depth_m, K, meshes, names = load_scene(scene_dir)
mesh, diameter_m = meshes[obj_id]
mask = cv2.imread(mask_path, cv2.IMREAD_GRAYSCALE)
n_view = 4
n_inplane = 3

est = WAPREstimator(device="cuda:0")
pose_mesh = est.prepare_meshes([mesh])[0]
mesh_setup_seconds = est.last_mesh_setup_seconds
instances = [{"mesh": pose_mesh, "diameter_m": diameter_m, "mask": mask}]
warmup_seconds = est.warmup_pose(rgb, depth_m, K, instances,
                                n_view=n_view, n_inplane=n_inplane)
out = est.estimate_one_category_one_instance(
    rgb, depth_m, K, pose_mesh, diameter_m,
    bbox_xywh=None, mask=mask,
    n_view=n_view, n_inplane=n_inplane,
)

Prepare CAD geometry once with prepare_meshes, then reuse the returned snapshots in every frame. Centering, normals, mesh packing and GL upload belong to setup. Treat snapshot geometry as immutable: edit the source mesh and prepare a new snapshot when the geometry changes. warmup_pose discards three complete calls on the actual input workload; its returned seconds are separate from hot inference. For a pose timer, synchronize CUDA immediately before and after the estimation call. Model setup, mesh setup, warmup, file reading and drawing stay outside that timer. One-instance estimation uses the same GPU batch path as many-instance estimation.

通过 prepare_meshes 一次性准备 CAD，后续各帧复用返回的网格快照。居中、法线计算、网格打包和 GL 上传属于准备阶段。快照几何不可原地修改：形状变化时编辑源网格，再生成新快照。warmup_pose 用实际输入负载完整预热三次并丢弃输出，单独返回预热秒数。测量位姿耗时时，在估计调用前后立即同步 CUDA；模型准备、网格准备、预热、读盘和绘图均在计时外。单实例估计也走与多实例相同的 GPU 批量计算路径。

estimate_frame_many_categories_many_instances starts from 2D detection and accepts any of the four category/instance combinations above. The estimate_one_category_one_instance call above is one category and one instance, and it does not take a class list. obj_ids is applied first. 2D matching scores only those classes, so a class outside the list is never chosen. 6D estimates only the boxes that remain. One id in obj_ids narrows this frame call to one category, and it can still return many instances of that category. That narrowed call is still not estimate_one_category_one_instance. An id that is not in meshes raises KeyError. An id that is not in the template bank raises ValueError. Omit obj_ids, or pass None, and the list is the mesh ids that are also in the bank. The user guideline does this for IC-BIN^[4] scene 1, image 26: object 1 only, even though object 2 is in the full IC-BIN set. That example is one category and many instances, not one instance.

estimate_frame_many_categories_many_instances 是多类别、多实例，从 2D 检测来。上面的 estimate_one_category_one_instance 是单类别、单实例，它不接收类别名单。obj_ids 最先用上。2D 匹配只给这些类别打分，名单以外的类别不会被选中。6D 只估计留下来的框。obj_ids 里只放一个 id，会把这次整帧调用限定为单类别，并且仍可以返回这一类的多个实例。收窄之后仍然不是 estimate_one_category_one_instance。不在 meshes 里的 id 会抛出 KeyError。不在模板库里的 id 会抛出 ValueError。不传 obj_ids，或传 None，名单就是既在网格里、也在模板库里的 id。使用指南里 IC-BIN^[4] 场景 1 第 26 帧就是这样：只要物体 1，尽管物体 2 也在完整的 IC-BIN 里。那个例子是单类别、多实例，不是单实例。

The two score gates are arguments, and they run after the class list. score_2d_min is the 2D gate. A box stays when its best allowed class is >= that value. Omit it, or pass None, and the gate is score_threshold in wapr/det2d.py, which is 0.1. Pass another float to use a different gate on this call. score_6d_min is the WBPS gate on score_6d. A pose stays when that score is >= the gate. Omit it, or pass None, and no pose is dropped by this score. The LM-O result uses a 0.50 2D input gate and silhouette NMS, without a WBPS score gate. Its extra driller predictions have WBPS scores around 0.25. A WBPS cutoff is not a general confidence probability or correctness threshold: a saved ROBI^[3] chrome-screw pose below meets that case's equivalent-pose ADD criterion with a WBPS score of 0.262. Choose a threshold using validation data for the intended setting.

两个分数阈值都是参数，在类别名单之后起作用。score_2d_min 是 2D 检测阈值：允许类别中的最高分不低于该值，候选框才保留。省略或传入 None 时，采用 wapr/det2d.py 的 score_threshold = 0.1；也可为本次调用传入其他数值。score_6d_min 作用于 WBPS 的 score_6d；省略或传入 None 时，不按位姿分数剔除。LM-O 展示结果采用 0.50 的 2D 输入阈值与轮廓 NMS，没有 WBPS 分数阈值；额外的 driller 预测，其 WBPS 分数约为 0.25。WBPS 截断值并非普适的置信概率或正确性阈值：下文ROBI^[3] 铬螺丝案例中，一条满足该案例等价姿态 ADD 判据的位姿，其 WBPS 分数仅为 0.262。实际阈值应根据目标场景的验证数据选择。

The per-class count is the last cut, after both score gates. inst_count maps an obj_id to how many poses of that class to keep. Each class is ordered by score_6d, high first, and only that many rows stay. A class missing from the dict is not capped. None does not cap any class. Same-class masks still pass IoU 0.5 inside the detector, before this count. In the configuration below, 14 is an explicit per-class upper limit; it is not a detected count or a guarantee that fourteen poses pass the score gates.

每个类别的数量是最后一道，在两个分数阈值之后。inst_count 把 obj_id 映射到这一类留下几条。每一类按 score_6d 从高到低排，只留这个数量。字典里没有的类别不截断。None 则每一类都不截断。同一类的 mask 仍先在检测器里按 IoU 0.5 抑制，这一步在数量之前。下面配置中的 14 是显式的每类数量上限，不是检出的数量，也不保证十四条位姿都能通过分数阈值。

from wapr.frame import estimate_frame_many_categories_many_instances

obj_ids = [1]
inst_count = {1: 14}
score_2d_min = 0.1
score_6d_min = 0.5

poses, timing, detections = estimate_frame_many_categories_many_instances(
    estimator, detector, rgb, depth_m, K, meshes,
    obj_ids=obj_ids,
    inst_count=inst_count,
    score_2d_min=score_2d_min,
    score_6d_min=score_6d_min,
    n_view=4, n_inplane=3,
)

n_view and n_inplane are one hypothesis setting for the whole call, not a setting per box. Every bounding box in this call is estimated with that same pair. The default pair is 4 tetrahedron views × 3 in-plane spins, 12 hypotheses. within_group picks one of those 12. Pass other integers to change the pair for every box together. Omit them, or pass None, to read wapr/recipe.py.

n_view 和 n_inplane 是本次调用共用的候选采样设置，所有包围盒都使用这两个参数。默认是 4 个正四面体视角 × 3 次面内旋转，一共 12 个候选姿态。within_group 在这 12 个里选一个。要换次数，就改这一次的这两个整数，整批框一起换。不传，或传 None，就读 wapr/recipe.py。

rgb, depth_m, K, and mask are read with numpy.asarray. Each one may be a numpy.ndarray, a rectangular nested list, or a torch.Tensor on CPU. A dict is not accepted. A CUDA tensor is not accepted: asarray raises TypeError. bbox_xywh and diameter_m go through Python float, so those two also accept a CUDA tensor. mesh is not an array.

rgb、depth_m、K、mask 用 numpy.asarray 读入。每一个可以是 numpy.ndarray、整齐的嵌套 list，或 CPU 上的 torch.Tensor。不接受 dict。不接受 CUDA 上的 Tensor：asarray 抛出 TypeError。bbox_xywh 和 diameter_m 走 Python float，这两个也可以是 CUDA 上的 Tensor。mesh 不是数组。

The supplied-region estimate_one_category_one_instance call returns a dict. pose_4x4 is a NumPy^[1] float32 array of shape (4, 4), object to camera. R is float32 (3, 3). t_m is float32 (3,), meters. score_6d is a Python float, (100 - max(between_group)) / 200. within_group picks the hypothesis inside the group. A larger between_group makes the returned score smaller.

给定区域的 estimate_one_category_one_instance 调用返回一个 dict。pose_4x4 是 NumPy^[1] float32，形状 (4, 4)，物体到相机。R 是 float32 的 (3, 3)。t_m 是 float32 的 (3,)，米。score_6d 是 Python float，等于 (100 - max(between_group)) / 200。within_group 在这一组里选候选姿态。between_group 越大，返回的分数越小。

Detection without known instance counts实例数量未知时的检测

examples/05_bop_6d_detection.py applies full 6D detection to one BOP frame without a known instance count. estimate_frame_many_categories_many_instances first runs the 2D detector, then passes masks containing finite, positive depth to one estimate_many_categories_many_instances pose batch. A detection without usable depth remains in the 2D return but cannot yield a pose. The visible mask defines the pose region; the detector box is retained as metadata. This configuration leaves inst_count unset, so it does not cap a category's instance count.

examples/05_bop_6d_detection.py 在不预先知道实例数量的情况下，对一帧 BOP 数据执行完整 6D 检测。estimate_frame_many_categories_many_instances 先运行 2D 检测器，再把 mask 内具有有限正深度的实例送入一次 estimate_many_categories_many_instances 位姿批处理。没有可用深度的检测仍保留在 2D 返回值中，但不能产生位姿。可见 mask 定义位姿区域，检测框只作为元数据保留。此配置不设置 inst_count，因此不限制每类的实例数量。

from wapr.frame import estimate_frame_many_categories_many_instances

poses, timing, detections = estimate_frame_many_categories_many_instances(
    estimator, detector, rgb, depth_m, K, meshes,
    obj_ids=None,
    inst_count=None,
    score_2d_min=0.1,
    score_6d_min=None,
    n_view=4, n_inplane=3,
)

meshes maps obj_id to (mesh, diameter_m). Each item in poses has obj_id, score_2d, bbox_xywh, pose_4x4, R, t_m, score_6d, and time_s. score_2d is the detector ranking score. It is not score_6d. time_s is that pose call divided by the number of instances that entered it, in seconds. Rows dropped by score_6d_min or inst_count are absent, so the kept rows sum to the batch only when nothing was dropped after the batch. Repeated obj_id values stay separate instances. name is included when the mesh metadata has one.

meshes 把 obj_id 映射到 (mesh, diameter_m)。poses 的每一项含 obj_id、score_2d、bbox_xywh、pose_4x4、R、t_m、score_6d、time_s。score_2d 是检测器的排序分，不是 score_6d。time_s 是这次位姿调用总耗时除以进入调用的实例数，单位秒。被 score_6d_min 或 inst_count 删掉的行不在列表里，所以只有 batch 之后没有再删行时，留下的行才加回这次调用。重复的 obj_id 仍是各自的实例。网格 metadata 里有 name 时，位姿里也有它。

The third return, detections, contains every row that passed the 2D gate, including masks without usable depth. Each row has obj_id, score_2d, bbox, and mask. The detector visualization draws blue boxes, tinted masks, and labeled 2D scores; examples 04 and 05 show only their 12 highest-scoring rows to keep the image legible, while the full rows still enter the depth check. The 6D visualization draws red boxes, rendered contours, and labeled WBPS score_6d. Example 05 also limits this illustration to the 12 highest WBPS scores; inference itself is unchanged. recipe.visualize controls whether the 6D image is saved.

第三个返回值 detections 包含通过 2D 阈值的全部检测行，其中也可能有 mask 内无有效深度的行。每行有 obj_id、score_2d、bbox 和 mask。2D 可视化绘制蓝色框、带颜色的 mask，并标注 2D 分数；示例 04、05 的插图只展示分数最高的 12 条，避免标签遮挡，但完整检测结果仍进入深度检查。6D 可视化绘制红色框、渲染轮廓，并标注 WBPS 的 score_6d。示例 05 也只在插图中展示 WBPS 分数最高的 12 条，推理本身不变。recipe.visualize 控制是否保存 6D 图。