Lightweight and Resource-Efficient Perception for Robotic Guide Dogs

Lightweight and Resource-Efficient Perception for Robotic Guide Dogs Jinse Kwon1 , Yoojin Lim1 , Choonghan Lee1 , Yongseung Yu1 , Yongin Kwon2 , and Jemin Lee3,† Electronics and Telecommunications Research Institute (ETRI), Korea {kwonse, yoojin.lim, choonghan.lee, ys.yu}@etri.re.kr 2 School of Computer Science and Engineering, Pusan National University, Korea yongin@pusan.ac.kr 3 Division of Electronics and Information Engineering, Jeonbuk National University, Korea jemin.lee@jbnu.ac.kr

arXiv:2610.03187v1 [cs.DC] 2 Oct 2026

1

Abstract. Robotic guide dogs should understand their surroundings, objects, and potential risks. Prior research has focused on raw sensor data from cameras and 2D or 3D LiDAR, which precisely measure distance points rather than provide a semantic understanding of the scene. While these physical measurements are effective for robot-centric collision avoidance and robot safety, they are not suitable for human-centric guidance. The system should recognize the type and relevance of obstacles and explain them, clearly and actionably, in terms of their spatial relation to the user. We present complete on-device perception modules that fuse a 360 camera and a 2D LiDAR for reliable collision avoidance, with moving-object detection and tracking for human-centric guidance. Finally, in walking-impossible situations, a vision–language model delivers pathway explanations as a safety mechanism to reduce user anxiety. In experiments, verification of fused 360 camera–LiDAR depth shows reliable near-range perception but inherent mid-range bias, while the system as a whole sustained real-time performance under 55 W. On the realworld egocentric GuideDogQA benchmark, our system achieved 83.8% accuracy, compared with 67.1% for GPT-4o. These results demonstrate that practical human-centric guidance with real-time on-device inference is feasible even on quadrupeds. Keywords: Robotic guide dog · Human-centric perception · Camera– LiDAR fusion · Assistive mobility · On-device AI

1

Introduction

Mobility for blind and low-vision people hinges on reliable perception and safe, efficient locomotion. Guide dogs give people with visual impairments greater † ⋆

Corresponding author Code available at: https://github.com/JinseKwon/guidedog-perception

2

J. Kwon et al. §3-4. Pathway Explanation via VLM

Human-Centric Guidance

§3-3. Object Description ID 01 ID 01

“Car on the front side, about 7 meters away moving.”

Omni-directional Perception Light-weight : -Sensors, AI models Resource-efficient - Low Power NPUs, Pipeline

“On the left, there is a person. On the right, there is a Caution barrier. The pathway is blocked by a box…”

“Person on the right side, about 4 meters away present.” ID 02

2D LiDAR 360 Camera Robot-Centric Guidance

§3-1. Omni-directional Perception

(a)

§3-2. Collision Avoidance

(b)

Fig. 1: (a) A 360 camera and a 2D LiDAR fused with on-device AI and multi-NPU acceleration, moving from robot-centric toward human-centric guidance. (b) Overview of the proposed system, integrating perception, obstacle understanding, and VLMbased guidance.

independence and mobility, but access is severely limited. Training requires years of effort and tens of thousands of dollars, while guide dogs can only work for a short portion of their lives. Because of these constraints, only about 2–3% of those who could benefit are ever matched with a guide dog [9, 13, 16]. These limitations have motivated efforts to develop robotic guide dogs as an alternative. Unlike biological dogs, robotic guides should not only ensure their own locomotion safety but also detect hazards relevant to human users [16]. Potential dangers include overhanging obstacles above the head, surface irregularities or damages near the ground, as well as dynamic obstacles approaching from the sides or even the rear. A robotic guide therefore requires omnidirectional sensing, accurate distance measurement, and stable performance, all under strict limits on size, weight, power, and cost. We develop omnidirectional perception modules for compact quadruped guide dogs and detail their algorithmic design and on-device performance. Fig. 1 summarizes the proposed platform and overall pipeline. The handler expresses high-level intent, while the robot performs local path following and collision avoidance. Our system provides the omnidirectional geometry and human-centric scene understanding required for this division of labor. Our contributions are: • On-device omnidirectional metric depth: We propose a cube-map based omnidirectional depth pipeline that anchors per-face monocular depth with LiDAR-guided affine correction. • Human-centric perception and guidance: We integrate moving-object perception and event-driven VLM–TTS explanation for human-centric guidance, achieving 83.8% accuracy on GuideDogQA relative-depth comparison compared with 67.1% for GPT-4o. • Real-time deployable system: We deploy the complete stack on a Jetson AGX Orin with multiple Hailo-8 NPUs, sustaining real-time depth estima-

Lightweight and Resource-Efficient Perception for Robotic Guide Dogs

3

tion at 25.3 FPS per cube-map face and object detection at 11.4 FPS per cube-map face within a 55 W power budget.

2

Related Work

Robotic guide dogs. Prior work spans safety formalization for quadruped guides [16], leash-guided physical human–robot interaction [26], studies of how handlers and guide dogs cooperate and co-construct perception [10, 13], and the functional and aesthetic preferences of blind and low-vision (BLV) users [9]. On the perception side, Hwang et al. [14] built a camera-only system whose inference ran on a tethered laptop GPU. Even low-resolution segmentation was difficult to sustain on device. Aoki et al. [5] combined a 3D LiDAR with a stereo depth camera, achieving accurate 3D geometry. However, the system is heavy, power hungry, and costs nearly ten thousand dollars. Moreover, it still provides only distance measurements rather than the semantic descriptions required for human-centric guidance of BLV users. We instead pair a consumer 360 camera with a low-cost 2D LiDAR within a small quadruped’s size–weight–power–cost budget, keeping every perception module on-device. Lightweight depth estimation and 360 fusion. Monocular depth estimation has evolved toward diverse accuracy–efficiency trade-offs. Representative methods include FastDepth [25] for embedded real-time applications, SC-DepthV3 [24] for self-supervised depth estimation in dynamic scenes, Depth-Anything-V2 (DAV2) [28] for large-scale generalization, and the recently proposed Depth-AnythingV3 [19], which further improves zero-shot depth estimation. However, these methods estimate only relative or scale-ambiguous depth, making their predictions insufficient for direct use in real-world navigation. We adopt SC-DepthV3 for its native Hailo-8 [2] acceleration and recover metric scale using a per-face affine fit against the corresponding 2D LiDAR scan. For 360 imagery, OmniFusion [18] removes projection distortion using tangent-image patches. However, it requires many overlapping inferences per frame. Instead, we retain the more efficient four-face cube-map decomposition, whose faces naturally align with the LiDAR’s angular sectors. Residual boundary distortion is mitigated through overlapping views and seam-aware blending. VLMs for BLV spatial assistance. A consistent body of evidence shows that current VLMs have limited spatial reasoning capabilities. This limitation has been reported across embodied question answering [20], 3D language interfaces [12,27], spatially fine-tuned VLMs [8], and even frontier models [7]. We observed similar failures when prompting VLMs with structured spatial metadata. Therefore, our moving-object descriptions are generated geometrically, while language models are reserved for event-driven pathway explanation following the space-aware instruction tuning of SA-VLM [11]. GuideDog [17] is the closest work to our evaluation. It introduces a large egocentric BLV guidance dataset together with the GuideDogQA benchmark, on which we compare our deployed on-device perception modules against substantially larger VLMs.

J. Kwon et al.

Depth Perception Pipeline

4

(2) Light-weight Pipeline via ZeroMQ

(1) Conversion via gstreamer

(3) Multi-NPU Pipeline

Front

Right

Back Right

Right Back

Equirectangular Panorama

(5) Point Cloud Generation

Front

Front

Left

(4) LiDAR-Depth Online Correction Left

Left

Cube Map Decomposition

(a) ERP-Cube Map Decomposition

Sub-view ZMQ Streaming

Left

Front

Right

Back

Back

Depth Estimation (Rough Depth)

Column-wise Online Depth Correction via 2D LiDAR

Calibrated Depth Points

(b) Synthesizing 3D Depth with AI-based Estimation

Fig. 2: Omnidirectional depth pipeline: (1) gstreamer-based cube-map decomposition, (2) ZMQ streaming to parallel NPUs, (3) SC-DepthV3 depth estimation, (4) LiDARbased affine correction, and (5) 3D point generation for collision avoidance.

3

Omnidirectional Perception Modules

No single affordable sensor meets our requirements: a monocular camera is rich in semantics but weak in depth and field of view, a 360 camera needs distortion handling, and a 2D LiDAR ranges accurately at low power but only in one plane. We therefore combine a 360 camera with a 2D LiDAR and perform online fusion. 3.1

Equirectangular to Cube map decomposition

We capture an equirectangular panorama (ERP) and forward it to a gstreamer pipeline, then decompose each frame into four perspective faces, namely front, left, back, and right, that together cover the entire horizontal view, as illustrated in Fig. 2. We explicitly discard the top and bottom faces and restrict each face to a vertical field of view (VFOV) of 90◦ , which is sufficient for human-level clearance checks on a guide-dog platform and saves compute. To accelerate processing, we precompute and cache mapping grids per resolution, ensuring linear runtime complexity in the number of pixels. When integrated into a gstreamer pipeline, the remapping further benefits from hardware decoder support, resulting in much faster performance on embedded hardware. The faces are then published over ZeroMQ (ZMQ) [3] as independent streams, front, back, left, and right, so that downstream modules such as detection, depth, and tracking can subscribe selectively and run in parallel on separate NPUs. 3.2

Synthesizing 3D Depth with AI-based Estimation

The cube map decomposition gives the robot omnidirectional vision, but RGB alone is insufficient for reliable clearance. We therefore attach monocular depth estimation to each face to obtain a dense depth field for near-field guidance. Among DA-V2, SC-DepthV3 [24], and FastDepth [25], we adopt SC-DepthV3 as the default estimator, since it offers a practical balance of accuracy, efficiency, and deployability with native acceleration on Hailo-8 [2]. Fig. 2 shows the complete pipeline for generating 3D point information by combining a 360 camera with a 2D LiDAR, which we describe step-by-step below.

Lightweight and Resource-Efficient Perception for Robotic Guide Dogs

5

(1) Per-face affine depth correction. For face columns u = 0, . . . , W − 1 we know the azimuth by construction, \theta _u \;=\; \theta _{\text {start}} + \frac {u}{W-1}\,(\theta _{\text {end}}-\theta _{\text {start}}), \label {eq:theta}

(1)

and we subscribe to the matching LiDAR sector, where the 2D LiDAR measurements are also broadcast as four directional ZMQ streams by our module. Let xu be the monocular depth on the mid-row, i.e., the horizontal center line of the estimated depth map, and r̃u be the LiDAR range interpolated at θu . A tiny, angle-dependent affine fit stabilizes scale, \tilde r_u \;\approx \; a(\theta _u)\,x_u + b(\theta _u), \label {eq:affine}

(2)

with a(·), b(·) represented by a few sinusoidal bases (constant, sin θ, cos θ). Only six coefficients are solved per face to align the monocular depth with the 2D LiDAR measurements, which keeps the computation lightweight enough for realtime processing. We specify Eq. (2) concretely by parameterizing the angle-dependent gains with just three bases per term: \begin {split} a(\theta )=\beta _0+\beta _1\sin \theta +\beta _2\cos \theta ,\\ b(\theta )=\beta _3+\beta _4\sin \theta +\beta _5\cos \theta . \end {split} \label {eq:abparam}

(3)

Substituting Eq. (3) into Eq. (2) turns each valid mid-row column u into one lin⊤ ear observation r̃u ≈ ϕ⊤ u β with ϕu = [xu , xu sin θu , xu cos θu , 1, sin θu , cos θu ] , where θu comes from Eq. (1) and xu is the monocular depth at the mid row. Given M valid columns, we estimate the coefficients by ridge regression, \hat {\boldsymbol \beta } = \arg \min _{\boldsymbol \beta \in \mathbb {R}^6} \sum _{u=1}^M \big (\boldsymbol \phi _u^\top \boldsymbol \beta - \tilde r_u\big )^2 + \lambda \|\boldsymbol \beta \|^2. \label {eq:ridge}

(4)

Rather than materializing the M × 6 design P matrix, we accumulate the suffiP 6×6 cient statistics G = u ϕu ϕ⊤ ∈ R and h = u u ϕu r̃u , and solve the resulting tiny normal equations (G + λI) β̂ = h, which takes only microseconds per face and constant memory. To stabilize frame-to-frame behavior, we smooth coefficients with an exponential moving average and gate updates when data are insufficient or residuals exceed a threshold. Precomputing {sin θu , cos θu } per face and unwrapping angles ensures robust interpolation. This compact six-parameter controller grounds monocular depth to a real-world scale and applies the affine correction across the entire face, rescaling each pixel as a(θu )xu,v + b(θu ). In this way all rows share a consistent distance scale, yielding depth maps that are globally aligned yet lightweight enough for on-device use and accurate enough for guide-dog safety. (2) Overlapping faces and seam smoothing. Although the per-face correction stabilizes the scale within each face, the depth values near face boundaries remain noticeably unstable. To mitigate this effect, we construct four additional

6

J. Kwon et al.

Moving Object Description

Raw Image Streaming

2D LiDAR Streaming

Object Detection on NPU

Left

TTS

Front

Left

Car /4m Person /6m Bicycle /7m

Left Right

Front

Front Back

Right

Right Back Back

Receiving RGB + Depth via ZMQ

Adding Object Information

Moving Object Tracking(MOT)

Image Pipe

Depth Pipe

“Car on the back side, about 4 meters away moving.”

Moving Object Description

Vocoder

Acoustic Model (Transformer)

Linguistic Analyzer

Description to Speech

Description Pipe

Fig. 3: Moving-object description: four faces stream RGB–depth into NPU detection (YOLOv8m) and lightweight tracking for object kinematics; per-face tracks are aggregated with safety-based prioritization into one spoken description per cycle.

overlapping views, each formed by concatenating two adjacent faces with 50% horizontal overlap, and apply the same regression procedure on these overlaps so that the affine correction is solved with shared support across neighboring faces. The final coefficients are obtained by blending the per-face and overlapping estimates, yielding smoother transitions across seams. After distance locking, we further smooth the seams by interpolating across a boundary of width w pixels. Here we parameterize the horizontal position within this seam by a normalized coordinate t ∈ [0, 1], rather than by angle, and apply a cosine ramp \rho (t)=\tfrac {1}{2}\big (1-\cos (\pi t)\big ). \label {eq:cosine} (5) The weight ρ(t) smoothly increases from 0 to 1, and the fused depth strip is D_{\text {fused}}=\rho (t)\,D_{\text {ov}}+(1-\rho (t))\,D_{\text {base}}, where Dbase is the per-face estimate and Dov is the overlapping estimate from the adjacent face. The left seam uses the overlap with the previous face, the right seam uses the next face, while the center keeps the base result to avoid unnecessary computation. 3.3

Moving Object Description

Given four synchronized faces each provides an RGB frame I ∈ R480×480×3 and a depth map D ∈ R256×320 where every pixel carries an estimated distance in millimeters. We attach a lightweight detection and tracking head that transforms raw pixels into human-meaningful descriptions of the scene, as illustrated in Fig. 3. It identifies the class of each object, estimates its position in terms of azimuth, elevation, and range, and characterizes its motion through radial speed and angular rates. The module executes independently of the depth fusion stage and then aggregates the results across all four faces to generate short spoken summaries, which are delivered to the visually impaired user. (1) Inputs and calibration. We crop a centered ROI in the RGB frame to match the depth aspect ratio (height 384 in our implementation), then map

Lightweight and Resource-Efficient Perception for Robotic Guide Dogs

7

256 RGB pixel coordinates (x, y) to depth indices with fixed scales sx = 320 480 , sy = 384 and a vertical offset y0 , i.e., (x̂, ŷ) = (round(sx x), round(sy (y − y0 ))). Azimuth and elevation are then computed from the horizontal and vertical fields of view (HFOV, VFOV):

\begin {split} \theta _h(x) &= \Big (\tfrac {x+\tfrac 12}{W}-\tfrac 12\Big )\,\mathrm {HFOV}, \\ \theta _v(y) &= -\Big (\tfrac {(y-y_0)+\tfrac 12}{H_{\mathrm {roi}}}-\tfrac 12\Big )\,\mathrm {VFOV}. \end {split}

(6)

where W = 480 and Hroi = 384. (The negative sign follows the standard imagedownwards y axis.) Although HFOV and VFOV are in principle free parameters, we did not retrain the depth estimation backbone and simply used the pretrained model at 320 × 256, which fixes HFOV at 90◦ and limits the VFOV of the depth ROI to 72◦ . (2) Per-face detection. Each face executes a compact YOLO-family detector compiled for Hailo-8 [2]. While any YOLO variant can in principle be supported, we adopt YOLOv8m [15] as a practical trade-off between accuracy and efficiency on our hardware. To avoid wasting resources on indiscriminate tracking, we restrict the set of classes to those relevant for mobility (person, bicycle, car, bus, truck, dog, cat, etc.), assuming typical pedestrian environments and discarding very small or highly specific classes that do not obstruct walking. For the same reason, we apply a confidence threshold of 0.8 to suppress marginal detections. For a detection k with center (xk , yk ) we then read the aligned depth dk = D(x̂k , ŷk ). (3) Lightweight tracking and kinematics. We maintain per-class tracks with greedy nearest-neighbor association (pixel distance < 60 px, class must match) and a small miss counter to drop stale tracks. For each track i updated at time t with depth di (t) and bearings (θh (t), θv (t)), we estimate radial velocity vd,i and angular velocities ωh,i , ωv,i using finite differences followed by exponential smoothing with factor γ ∈ [0, 1) to reduce noise. A simple rule flags objects as “moving” if |vd,i | ≥ τmove (default 0.10 m/s). (4) From four faces to a single description. Every Tsum seconds (default 1 s) we aggregate the most recent tracks from all faces, following the pipeline shown in Fig. 3. In our current implementation, we adopt a simple priority scheme that emphasizes the most critical cues for mobility: we describe only one sentence per cycle, giving precedence to the closest object, objects in the front direction, and vehicle-related classes such as cars and buses. This prioritization ensures that the spoken output remains concise and focused on safety-critical elements. We initially evaluated LLM and VLM based free-form descriptions of moving objects. However, they did not reliably identify moving or high-risk objects, even when provided with structured JSON spatial metadata. We therefore retain the deterministic priority scheme described above.

8

J. Kwon et al.

VLM Decoder FFN

Vocoder

Acoustic Model (Transformer)

Linguistic Analyzer

“On the left, there is a person. On the right, there is a Caution barrier. The pathway is blocked by a box…”

Cross-Attention

Self-Attention

Prompt

Vision Encoder

VLM-TTS based Scene Explanation

Description to Speech

Image Pipe

WalkingImpossible Situation

VLM-TTS based Explanation

TTS

Description Pipe

Fig. 4: Event-driven pathway explanation: on a blocked path, a VLM turns the front view into a space-aware description of obstacles and options, streamed sentence-bysentence to TTS to explain why walking stopped. 360 Camera (Ricoh Theta Z1) 2D LiDAR (SLAMTEC C1) Speaker

PCIe Switch Ex-Card (m.2 x4 / PLX8725)

PCIe Switch Ex-Card (m.2 x4 / PLX8725)

NPU (Hailo-8 x 4)

NPU (Hailo-8 x 1)

Computing Board (Jetson AGX ORIN)

Fig. 5: Hardware setup: Ricoh THETA Z1 360 camera, SLAMTEC C1 2D LiDAR, Jetson AGX Orin, and five Hailo-8 NPUs.

3.4

VLM-TTS Based Explanation

When the collision avoidance stack declares walking_impossible, the robot first stops and generates a spoken pathway explanation. Following the spaceaware guidance design [11], the VLM describes relevant obstacles, their locations, and immediate options, explaining why the robot stopped and whether further walking is possible. As illustrated in Fig. 4, the VLM analyzes only the front image and generates a spatially aware description of the forward scene, including left, right, and pathway context. The output is immediately delivered through Text-to-Speech (TTS). Perception, VLM, and TTS are decoupled over a lightweight ZMQ message bus. Generated sentences are streamed to TTS to reduce first-utterance latency, while omnidirectional depth perception continues monitoring the surroundings. Once the path becomes clear, the explanation mode terminates and normal guidance resumes.

Lightweight and Resource-Efficient Perception for Robotic Guide Dogs

9

Table 1: Throughput and per-rail (CPU/GPU/SYS/NPU) peak power and temperature as modules are progressively enabled on the Jetson AGX Orin. AI Model (AI Task)

Module

IDLE — PCIe-m.2 — ext-card x2 ERP-Dec

—

3D-Depth MOD TTS Path Desc Total *

SC-DepthV3 (Depth Est) YOLOv8m (Object Det) MeloTTS (TTS) Qwen2.5-VL-7B (VLM)

Resolution Thr./face (W×H×B) (FPS)

Thr.

—

—

—

—

1920×960×1 → 480×480×4 480×480×8 → 320×256×8 480×480×4 → 640×640×4

Power (Peak W) CPU

GPU

—

0.4 W

—

—

30.0

120.0 FPS

25.3

*

Thermal (◦ C)

SYS

NPU

CPU GPU SYS NPU

3.2 W

6.1 W

—

47.5

44.5

44.9

—

18.8 W

—

—

—

43.8

—

0.8 W

—

0.4 W

—

48.5

—

46.5

—

101.3 FPS 2.1 W

—

—

8.2 W

53.1

—

—

38.7

36.0

11.4

45.7 FPS

0.3 W

—

—

1.2 W

53.5

—

—

40.5

—

—

0.3 W

1.2 W

—

—

54.1

49.5

—

—

640×480×1

—

8.18 TPS

0.3 W 10.1 W

—

—

56.8

53.5

—

—

—

—

—

—

—

—

—

—

—

4.2 W 14.5 W 25.3 W 9.4 W

Throughput is reported for the four effective cube-map faces. The remaining four views are overlapping regions used for seam-aware blending.

4

Experiments

4.1

On-device Platform and Configuration

Our hardware configuration is illustrated in Fig. 5: an NVIDIA Jetson AGX Orin [21], a Ricoh THETA Z1 360 camera [22] (USB UVC), and a SLAMTEC C1 2D LiDAR [23]. The camera outputs equirectangular frames, which we split into four cube map faces: front, left, back, and right. Each face is cropped to a 90◦ vertical field of view, producing 480×480 BGR tiles for detection, along with 320×256 fused depth maps in millimeters. The LiDAR operates at ≥10 Hz. Its scan is divided into four directional sectors that are aligned with the corresponding camera faces to enforce distance-scale alignment. Because the Orin exposes a single PCIe Gen4 root port, a PCIe Gen3 switchbased expansion branches it into multiple M.2 endpoints. This configuration connects five Hailo-8 NPUs together with an NVMe SSD without requiring an external host. Four NPUs run SC-DepthV3, one for each cube-map face, while the fifth runs YOLO for object detection. The VLM runs locally in a lightweight configuration and is triggered only by walking_impossible events to generate a six-sentence route explanation. All modules communicate over a lightweight ZMQ publish–subscribe bus that supports multiple message types, enabling flexible and asynchronous data exchange. As a result, downstream stages always process fresh data, and a slow module cannot stall the pipeline. The perception core operates independently of ROS. A ROS bridge exposes ROS topics to the host robot without affecting the internal execution timing. 4.2

Runtime, Power, and Thermal Behavior

Table 1 summarizes the end-to-end throughput, power consumption, and temperature as modules are progressively enabled on a single Jetson AGX Orin. CPU, GPU, and system temperatures are obtained from the internal sensors.

J. Kwon et al.

Measured mean depth (m)

10

12 10 8 6 4

BG = 3 m BG = 4 m BG = 5 m

2 2

4

6

True distance (m)

8

BG = 6 m BG = 7 m BG = 10 m

10

12

Fig. 6: Measured mean depth vs. true distance for a person at 1–10 m under different background-wall (BG) positions (error bars: std. dev.; dotted: y = x). Near-field accuracy is high (MAE 0.34 m at 1–3 m), but mid-range estimates drift toward the wall distance.

NPU and PCIe switch temperatures are measured using a FLIR One [1] under an ambient temperature of approximately 24 \pm 1^{\circ }\mathrm {C}. Three observations can be made. First, equirectangular to cube map decoding introduces only about 1.2 W of additional power while operating at 30 FPS per face. This efficiency is achieved using NVDEC, nvvidconv, and zero copy GStreamer pipelines. Second, the Hailo-8 NPUs execute the computationally intensive workloads with low power consumption. Depth estimation requires about 8 W. Four cube map faces are processed in parallel, allowing each Hailo-8 to sustain more than 50 FPS. YOLOv8m increases power consumption by only about 1 W, leaving the CPU responsible mainly for data transfer and lightweight post processing. The event driven VLM (Qwen2.5-VL-7B [6]) operates at approximately 8 tokens/s and increases GPU power by about 10 W. TTS adds approximately 1 W. Since both modules are activated only when needed, the average GPU power remains low and the response latency is short. Third, the PCIe infrastructure contributes a fixed system power overhead. Two PCIe Gen3 M.2 switches consume about 19 W regardless of workload. Even at maximum load, the total board power remains about 53 W without thermal throttling. Takeaway: The proposed system simultaneously sustains 25.3 FPS per face for depth estimation and 11.4 FPS per face for object detection while maintaining total board power below 55 W and stable operating temperatures. 4.3

Validation of Corrected Depth Estimation

This experiment evaluates the reliability of the fused 360 monocular–LiDAR depth under repeatable and controlled conditions. The background wall was placed at maximum distances of 3, 4, 5, 6, 7, and 10 meters, and a person was positioned at 1-meter intervals from the camera. For each combination, multiple

Lightweight and Resource-Efficient Perception for Robotic Guide Dogs

11

(a) Relative-depth comparison : The two reference objects are outlined by thin blue A/B boxes. Estimated distance are shown above each object (Green = nearer, red = farther)

Person (GT 1.9m) vs Person (GT 17.4m)

Car (GT 2.1m) vs Sign (GT 15.6m)

Barrier Stump (GT 0.9m) vs Barrier Post (GT 6.2m)

Barrier Post (GT 3.6m) vs Car (GT 24.7m)

Bench (GT 3.0m) vs Person (GT 19.1m)

Pole (GT 1.4m) vs Car (GT 8.6m)

Stop sign (GT 10.5m) vs Lamp Post (GT 61.7m)

Raised Entryway (GT 1.1m) vs Car (GT 5.6m)

Chair (GT 3.0m) vs Person (GT 14.2m)

Motorcycle (GT 4.7m) vs Person (GT 21.8m)

(b) Object recognition : The green box marks the queried object, the green text indicates the correct answer, and the check mark shows the prediction.

Fig. 7: Qualitative GuideDogQA examples from the proposed on-device system. (a) Relative-depth comparison. Estimated distances are shown above the two referenced objects (green: nearer, red: farther). Ground truth distances are shown in the labels. (b) Object recognition. The green box marks the queried object, the green text indicates the correct answer, and the check mark denotes the predicted result.

measurements were collected, and the results were summarized in terms of the mean and standard deviation. Two consistent trends were observed. First, the performance varies distinctly by distance range. As shown in Fig. 6, the overall MAE and RMSE are 2.16 m and 2.73 m, respectively. Accuracy is highest in the safety-critical near range of 1–3 m (MAE 0.34 m, RMSE 0.47 m), while errors increase at 4–7 m (MAE 3.41 m, RMSE 3.95 m) and decrease again at 8–10 m (MAE 1.69 m, RMSE 1.79 m). This mid-range degradation indicates systematic scale drift, primarily affecting early warning rather than near-range collision avoidance. Second, the mid-range bias is primarily caused by background dominance. When a distant wall occupies most azimuth columns, the mid-row affine fit is driven more strongly by the background than by a narrower foreground object, producing a characteristic “pull-to-background” effect. For example, predictions remain nearly unbiased with a 3 m wall but drift toward the wall distance as the background moves farther away. This behavior follows from applying six angle-conditioned coefficients, estimated from mid-row LiDAR anchors, across the entire face. If the foreground occupies few columns or is offset from the single LiDAR scan plane, its influence on the fit is further reduced. The resulting bias is therefore systematic and reflects the scale limitations of the monocular depth estimate under backgrounddominant scenes.

12

J. Kwon et al.

Table 2: GuideDogQA accuracy (%), with a random chance level of 25.0. VLM baselines are reproduced from GuideDog [17]. Our method (SC-DepthV3 and YOLOv8m) is evaluated as a deployed NPU implementation. Bold and underlined values indicate the best and second-best results, respectively. Depth Comparison

Object Recognition

Random Chance

25.0

25.0

Cambrian-1 Molmo LLaVA-1.6 LLaVA-OneVision Qwen2.5-VL ,→ w/ finetuning Gemini 2.0 Flash GPT-4o

24.3 28.5 30.0 32.4 22.2 41.5 53.0 67.1

82.3 34.0 80.9 87.4 85.7 83.9 65.7 74.7

Ours

83.8

63.0*

Model

*

The COCO-80 vocabulary covers 240 of the 435 object recognition questions. Our system achieves 95.4% accuracy on this subset.

4.4

Perception Accuracy on the GuideDogQA Benchmark

Metric depth accuracy alone does not indicate how the deployed perception modules perform in cluttered real-world environments. We therefore evaluate the deployed monocular depth estimator (SC-DepthV3 [24]) and object detector (YOLOv8m) on GuideDogQA [17], a real-world egocentric benchmark for blind and low-vision (BLV) guidance. We use the two subsets corresponding to our perception modules: relative distance comparison (383 questions) and object recognition (435 questions). Random chance is 25% for both tasks. All experiments are performed using the deployed models running directly on the Hailo-8 NPU. (a) Relative depth comparison. Each question provides two ground truth bounding boxes and asks which object is closer. A prediction is counted as correct only if both the “closer” and “farther” questions are answered correctly, resulting in a random chance of 25%. We determine the answer by comparing the median predicted depth within each bounding box. Using the ground truth boxes isolates depth estimation from object detection errors, and the resulting pairwise ordering is directly comparable with VLM-based methods. (b) Object recognition. Each question presents four candidate object names. We map each candidate to a COCO-80 class using a synonym table and select the class with the highest detection confidence. Since not every answer belongs to the COCO-80 vocabulary, we report both the overall accuracy and the accuracy on the COCO-coverable subset. Results. Table 2 summarizes the quantitative results, and Fig. 7 presents representative outputs from the deployed NPU. The depth estimator achieves 83.8% accuracy on the relative distance comparison task, exceeding the strongest evaluated VLM, GPT-4o, which achieves 67.1%. This result shows that a lightweight geometric estimator is more suitable for relative depth ordering in obstacle avoid-

+2.9

+1.0

81.2 83.0

+1.3 -1.0

81.5

79.1

80.2

77.3 78.6

8× 22 4 0× 25 6

0× 38

0× 51

48

64

32

28

0

4× 19 2 22

19 2× 16

8

0× 76 8 80 ×7 36

SC-DepthV3 (Baseline) SC-DepthV3 + Proposed Correction

random chance 25.0

0× 12

13

SC-DepthV3 input resolution (W×H)

12

74.2

+0.8

+1.0

96

72.8

+3.1

2

+2.1

4

90 80 70 60 50 40 30 20 16

relative-depth ordering accuracy (%)

Lightweight and Resource-Efficient Perception for Robotic Guide Dogs

Fig. 8: Effect of input resolution on relative depth ordering accuracy. The proposed center-line correction improves ordering accuracy at most resolutions and achieves high accuracy at the operational resolution of 320 × 256.

ance. The detector achieves 63.0% overall accuracy. Among the 435 object recognition questions, 240 are covered by the COCO-80 vocabulary, on which our detector achieves 95.4% accuracy. The remaining errors are primarily caused by objects outside the COCO vocabulary, such as buildings, lamp posts, and sidewalks, rather than failures of the detector itself. 4.5

Depth Resolution and LiDAR-Grounded Evaluation

Fig. 8 evaluates the effect of input resolution on depth ordering accuracy. SCDepthV3 is evaluated at input resolutions ranging from 160×128 to 1280×736. The highest uncorrected accuracy is achieved at 320×256, which corresponds to the model’s training resolution. Accuracy decreases at both lower (72.8%) and higher (77.3–78.6%) resolutions because these inputs fall outside the training distribution. To evaluate the proposed center line correction across different input qualities, we apply it at every input resolution. Since the GuideDogQA benchmark does not provide LiDAR measurements, DA-V2 is used as a high-quality reference to emulate the LiDAR center line. We also tested Depth-Anything V3 as an alternative reference, but it resulted in lower ordering accuracy and weaker correction performance in our setting. The proposed correction improves ordering accuracy at most resolutions, with the largest gain of 3.1 percentage points at 192×160. Although the corrected result at 480×384 is slightly higher than that at the deployed setting, this resolution exceeds the real-time processing budget of the on-device system. We therefore deploy the depth estimator at 320×256, which provides the best practical balance between ordering accuracy and realtime performance. Independent LiDAR-grounded evaluation. We additionally evaluate 100 randomly sampled AI-Hub [4] adverse-weather images against projected 3D LiDAR ground truth (565 bounding boxes and 783 relative-depth pairs). We

14

J. Kwon et al.

Table 3: Relative-depth ordering accuracy and absolute depth error on 100 randomly sampled AI-Hub adverse-weather images using projected 3D LiDAR as ground truth. Bold/underlined values denote the best/second-best results. Method GPT-4o SC-DepthV3 raw Depth-Anything-V2 Ours (DA-V2 anchor) Ours (LiDAR anchor)

Acc. (%)

MAE (m)

59.8 74.3 88.8 72.8 74.7

n/a 35.76 38.49 37.29 31.93

use unseen AI-Hub pairs instead of KITTI, which was used in DA-V2 training. In Table 3, DA-V2 reaches 88.8% ordering accuracy, supporting its use as a relative-depth proxy. Changing the anchor from DA-V2 to physical LiDAR improves ordering accuracy from 72.8% to 74.7% and reduces MAE from 37.29 m to 31.93 m. The remaining absolute error should be distinguished from the controlled near-range results in Sec. 4.3.

5

Conclusions

We presented a lightweight and fully on-device perception system for robotic guide dogs. The proposed system combines a 360 camera with a 2D LiDAR using an angle conditioned affine model and seam aware blending. Controlled experiments verify accurate metric depth estimation in the near range. On the GuideDogQA benchmark, the deployed INT8 perception modules achieve 83.8% accuracy on relative depth comparison, exceeding the strongest evaluated VLM (GPT-4o, 67.1%). The object detector reaches 95.4% accuracy on the COCO coverable subset. The deployed INT8 implementation also matches the accuracy of the full precision model, demonstrating that real time on-device inference can be achieved without sacrificing perception accuracy. Several limitations remain. The proposed system inherits the mid range depth bias of monocular estimation, is constrained by the closed COCO vocabulary for static street infrastructure, and has not yet been validated through user studies with people who are blind or have low vision. Future work will focus on open vocabulary object detection within the NPU resource budget and real world field evaluations with BLV users. Acknowledgement. This work was supported by IITP grants funded by the Korea government (MSIT) (No. RS-2024-00459797, RS-2025-02217404 and IITP2026-RS-2026-25615995 under the AI Star Fellowship Support Program), a NIPA grant funded by the Korea government (MSIT) (No. RS-2026-25621588), the Technology Innovation Program funded by MOTIE, Korea (No. RS-2026-25620133), and an ETRI grant funded by the Korean government (No. 26CS1100).

Lightweight and Resource-Efficient Perception for Robotic Guide Dogs

15

References 1. FLIR ONE: Thermal imaging camera for smartphones. https://www.flir.com/ products/flir-one-gen-3/, [Online] 2. Hailo-8 AI accelerator. https://hailo.ai/products/ai-accelerators/hailo8-ai-accelerator/, [Online] 3. ZeroMQ: An open-source universal messaging library. https://zeromq.org/, [Online] 4. AI-Hub: Commercial autonomous vehicle adverse-weather dataset. https://www. aihub.or.kr/aihubdata/data/view.do?dataSetSn=71575 (2022), National Information Society Agency (NIA), Republic of Korea, accessed October 1, 2026 5. Aoki, T., Zhang, B., Lim, H.O.: 3d spatial information reflected 2d mapping for a guide dog robot. In: International Conference on 6GN for Future Wireless Networks. pp. 3–9 (2023) 6. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report (2025), https://arxiv.org/abs/2502.13923 7. Cai, Z., et al.: Has gpt-5 achieved spatial intelligence? an empirical study. arXiv preprint arXiv:2508.13142 (2025) 8. Chen, B., et al.: Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14455–14465 (2024) 9. Cohav, A.L., Gong, A.X., Kim, J.T., Zeagler, C., Ha, S., Walker, B.N.: Do looks matter? exploring functional and aesthetic design preferences for a robotic guide dog. In: 2025 IEEE International Conference on Robotics and Automation (ICRA). pp. 1951–1957 (2025) 10. Due, B.L.: Guide dog versus robot dog: Assembling visually impaired people with non-human agents and achieving assisted mobility through distributed coconstructed perception. Mobilities 18(1), 148–166 (2023) 11. Han, B., Yun, W.h., Seo, B.S., Kim, J.: Space-aware instruction tuning: Dataset and benchmark for guide dog robots assisting the visually impaired. In: 2025 IEEE International Conference on Robotics and Automation (ICRA). pp. 12430–12436 (2025) 12. Hong, Y., et al.: 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems 36, 20482–20494 (2023) 13. Hwang, H., Jung, H.T., Giudice, N.A., Biswas, J., Lee, S.I., Kim, D.: Towards robotic companions: Understanding handler-guide dog interactions for informed guide dog robot design. In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. pp. 1–20 (2024) 14. Hwang, H., Xia, T., Keita, I., Suzuki, K., Biswas, J., Lee, S.I., Kim, D.: System configuration and navigation of a guide dog robot: Toward animal guide dog-level guiding work. In: 2023 IEEE International Conference on Robotics and Automation (ICRA). pp. 9778–9784 (2023) 15. Jocher, G., Chaurasia, A., Qiu, J.: Ultralytics yolov8 (2023), https://github. com/ultralytics/ultralytics 16. Kim, J.T., Yu, W., Kothari, Y., Walker, B., Tan, J., Turk, G., Ha, S.: Transforming a quadruped into a guide robot for the visually impaired: Formalizing wayfinding, interaction modeling, and safety mechanism. In: Proceedings of The 7th Conference on Robot Learning. pp. 2288–2303 (2023)

16

J. Kwon et al.

17. Kim, J., Park, J., Park, J., Lee, S., Chung, J., Kim, J., Joung, J.H., Yu, Y.: GuideDog: A real-world egocentric multimodal dataset for blind and low-vision accessibility-aware guidance. In: Proc. Annual Meeting of the Association for Computational Linguistics (ACL) (2026) 18. Li, Y., Guo, Y., Yan, Z., Huang, X., Duan, Y., Ren, L.: Omnifusion: 360 monocular depth estimation via geometry-aware fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2801–2810 (2022) 19. Lin, H., Chen, S., Liew, J.H., Chen, D.Y., Li, Z., Shi, G., Feng, J., Kang, B.: Depth anything 3: recovering the visual space from any views. arXiv preprint arXiv:2511.10647 (2025) 20. Majumdar, A., et al.: Openeqa: Embodied question answering in the era of foundation models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16488–16498 (2024) 21. NVIDIA: Jetson AGX Orin Developer Kit. https://www.nvidia.com/en- us/ autonomous-machines/embedded-systems/jetson-orin/ (2024), accessed: 202607-03 22. Ricoh: Ricoh theta z1. https://thetaz1.com/en/ 23. SLAMTEC: RPLIDAR C1: High definition fusion dtof lidar. https://www. slamtec.com/en/c1 24. Sun, L., Bian, J.W., Zhan, H., Yin, W., Reid, I., Shen, C.: Sc-depthv3: Robust selfsupervised monocular depth estimation for dynamic scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence 46(1), 497–508 (2023) 25. Wofk, D., Ma, F., Yang, T.J., Karaman, S., Sze, V.: Fastdepth: Fast monocular depth estimation on embedded systems. In: 2019 International Conference on Robotics and Automation (ICRA). pp. 6101–6108 (2019) 26. Xiao, A., Tong, W., Yang, L., Zeng, J., Li, Z., Sreenath, K.: Robotic guide dog: Leading a human with leash-guided hybrid physical interaction. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). pp. 11470–11476 (2021) 27. Xu, R., Wang, X., Wang, T., Chen, Y., Pang, J., Lin, D.: Pointllm: Empowering large language models to understand point clouds. In: European Conference on Computer Vision. pp. 131–147 (2024) 28. Yang, L., et al.: Depth anything v2. Advances in Neural Information Processing Systems 37, 21875–21911 (2024)

Related documents

• Lightweight and Resource-Efficient Perception for Robotic Guide Dogs#1187865
• Rethinking Streaming-Perception Evaluation on Heterogeneous Edge Platforms#1201591
• Rethinking Streaming-Perception Evaluation on Heterogeneous Edge Platforms#1194394
• S4VY: Segment Anything in Feed-Forward 4D Visual Geometry#1132076
• Scalable, modular whole-slide imaging pipelines for low-resource settings: A practical framework and open notebook implementation.#457384