Hardwired Distributed Processing Units at the Sensor: Truly Parallel Vision Pre-Processing that Frees the Host CPU

Hardwired Distributed Processing Units at the Sensor: Truly Parallel Vision Pre-Processing that Frees the Host CPU Raj Mali

arXiv:2610.38752v1 [cs.DC] 28 Oct 2026

Independent Researcher raj.mali.research@gmail.com ORCID: 0009-0001-6351-0062 Abstract—A vision system that sends every pixel to its host processor makes that processor wait for each complete frame and then spend part of every frame period on routine preprocessing. We propose instead to distribute this work into hardwired distributed processing units (HDPUs): small fixed datapaths attached directly to the sensor, each dedicated to one always-on task, all operating truly in parallel on the same pixel stream and passing compact results to the host. From a single library of 3 × 3 operators with a common two-clock latency, we build three HDPUs (Canny edges, Harris corners and a line tracker that emits a steering decision) and run them side by side on one camera stream in a low-cost Artix-7 FPGA, using 2,568 LUTs and an estimated 0.318 W. Every stage of every HDPU is bitexact with its software model. In a full-system simulation with a datasheet camera model, the HDPUs deliver each decision within 11.03 camera lines of its pixel, 22–67× sooner than the earliest moment a frame-based program on the host could, with a latency that varies by at most 17 ns, whereas the host’s own run times reached 12.5× their median. Offloading the three tasks returns to the host 18–30 % of a core in every frame period at 640 × 480, and 63 % of the frame period, using all its threads, at 1080p. HDPU cost grows with frame width only through line buffers, and in a closed-loop model of a line-following vehicle the HDPU raises the highest speed at which the line is held from 4.75 to 10.00 m/s. Index Terms—hardwired distributed processing units, nearsensor computing, FPGA, hardwired datapath, parallelism, offload, latency, determinism, Canny, Harris corners

I. I NTRODUCTION In most vision systems, the camera sends every pixel to a central processor, which stores the frame and then runs a program over it. Modern processors do this well: with SIMD units, multiple cores and a library such as OpenCV [1], a laptop processes a VGA frame through several detectors in a few milliseconds [2]. Yet this organisation makes the processor the bottleneck for two reasons. It cannot decide anything until the whole frame has arrived, which takes tens of milliseconds; and the routine pre-processing (edges, corners, a tracking signal) consumes part of every frame period, every frame, leaving less time for the work that actually needs a processor. We argue for a distributed organisation in which such routine, time-critical tasks run in hardwired distributed processing units (HDPUs) placed at the sensor, in the spirit of near-sensor computing [3]. An HDPU is a fixed datapath for one task, with no instruction stream, operating system or frame buffer. Pixels

flow from the sensor pins through its logic, and every result leaves a fixed number of clock cycles and image lines after its input. Several HDPUs observe the same pixel stream at the same time, each with its own logic, so they are parallel in the literal sense: no unit waits for, shares a bus with, or is scheduled against another. The host remains the master core: it receives compact results (an edge map, a list of corners, a steering code) instead of raw frames, and spends its time on the flexible work that only it can do. We call the always-on task that an HDPU performs a reflex. A companion paper [4] presented one such unit, a Canny edge detector [5] built from a library of 3 × 3 operators with a common two-clock latency. This paper asks whether the approach generalises to a set of cooperating units, and what the system gains from them. Our contributions are: 1) Three HDPUs from one operator library, the Canny unit and two new ones, a Harris corner detector [6] and a dark-line tracker that turns morphological filtering [7] into a steering decision, running in parallel on one camera stream with one shared Gaussian stage in a single low-cost FPGA (Section III). 2) A verification chain in which every HDPU is bit-exact, stage by stage, across a Python model, RTL simulation, a full-system simulation with a camera model, and a C++ port (Section IV). 3) A quantified case for offloading: decision latency from the sensor, its variation, the share of the host’s frame period returned to it, and the data that must be held before a decision, measured against carefully controlled host software (Section V). 4) A resolution-scaling study from 320 × 240 to 1920 × 1080, and a closed-loop model that shows what the HDPU’s latency means for a vehicle (Sections V-F and V-G). II. BACKGROUND Computing near the sensor. Moving processing towards the sensor reduces the data that must be transferred and the time to a result [3]. Horowitz showed that, in a general-purpose processor, the energy of an arithmetic operation is small compared with that of fetching instructions and moving data [8], and Kuon and Rose quantified the cost of programmable logic

relative to custom circuits [9]. An FPGA datapath sits between the two: its clock is slower than a processor’s, but it fetches no instructions and its structure follows the algorithm, so its effort goes into the computation itself. Streaming vision on FPGAs. Local image operators map naturally onto line-buffered FPGA pipelines [10]. Stencil compilers such as Darkroom [11] and vendor libraries such as Vitis Vision [12] generate such pipelines, and multi-resolution architectures combine edge and corner detection on one device [13]. Most of this work reports throughput and resources; we focus instead on what a set of such units gives the system around them: the time from sensor to decision, its variation, and the load removed from the host. Specialisation and offload. Hameed et al. found a generalpurpose chip multiprocessor 500× less energy efficient than an ASIC on H.264 encoding, with about 90 % of its energy spent on overhead rather than arithmetic [14]. Conservation cores place small specialised datapaths beside a processor to cut the energy of frequently run code [15], and datacenters offload routine work from host CPUs to FPGAs, for search ranking [16] and for networking [17]. LogCA shows that such offload pays only when the cost of invoking the accelerator and moving data to it is small relative to the work [18]. HDPUs follow the same reasoning but sit on the sensor stream itself: the host never invokes them or copies data to them; it only receives their results. Fair baselines. Lee et al. showed that many reported accelerator speed-ups vanish once the processor’s code is optimised [2]. We therefore measure the host running OpenCV, single- and multi-threaded, and a bit-exact C++ port of the HDPUs’ algorithms, under controlled conditions. Delay in feedback loops. In sampled-data control, a delay between measurement and actuation erodes the phase margin and limits achievable performance [19]. Our closed-loop experiment uses a pure-pursuit path tracker [20] to make this effect concrete. III. T HREE PARALLEL HDPU S FROM O NE O PERATOR L IBRARY A. Operator contract Every operator takes a 3 × 3 window and produces its result exactly two clock cycles later; validity, coordinates and the image-border flag travel alongside in matched delay registers outside the operator. A streaming window holds two image lines in one block RAM and presents a new window for every input sample, so a chain of S windows decides a pixel about S image lines after it arrives. Each stage forces its onepixel frame border to zero, exactly as the software models do. Because all operators share one timing contract, a new HDPU is a new chain of windows and operators rather than a new control design. B. Parallel by construction The parallelism of the HDPUs is structural rather than scheduled. Every operator in every unit accepts a new pixel in every clock cycle, so in any given cycle each operator of all

three units is working on a different pixel. The units share only the Gaussian output; they have no common bus, arbiter, cache or scheduler, and none of them can delay another. A unit’s latency is therefore set by its own chain of operators alone, and adding a unit costs logic but no time. The measurements confirm this: in the companion paper, the Canny unit alone decides an edge 720.63 µs after its pixel [4]; with the Harris and line units running beside it, the same decision follows its pixel by 720.6 µs (Section V), identical to the precision reported. This differs from parallelism on a processor, where threads compete for cores, caches and memory bandwidth, and where the operating system decides when each task runs. C. The three HDPUs Fig. 1 shows the system, and Table I lists the operators each unit uses. Canny edges is the fast path of [4]: Gaussian smoothing, Sobel magnitude and quantised direction, non-maximum suppression, double thresholding and K = 8 local hysteresis stages. It never reports an edge that full hysteresis would reject. Harris corners [6] reuses the Gaussian output. A new operator forms the gradients and the three products gx2 , gy2 and gx gy (scaled to 12, 12 and 13 bits); the next sums them 1 (tr M )2 over the window and computes R = det M − 16 with three DSP multipliers, saturated to an 8-bit score; a third retains local maxima above a threshold. Ties between equal neighbours are resolved in favour of the first pixel in raster order, so a flat maximum yields exactly one corner. Line tracking finds a dark line on a light floor, as a linefollowing robot does. The library’s threshold operator, applied to the inverted pixel, marks dark pixels; erosion followed by dilation (an opening [7]) removes specks narrower than three pixels. A reducer then accumulates, for every row, the number of line pixels and the sum of their x coordinates. At the end of a chosen look-ahead row, it compares the sum with the count multiplied by W/2 ± 16, which requires no divider, and outputs a steering code: none, left, straight or right. Together, the three units turn a 16-bit RGB565 pixel stream into a one-bit edge map, a sparse set of corners and one steering code per frame: pre-processed results that the host can use directly. TABLE I O PERATORS USED BY EACH HDPU ( ALL WITH A TWO - CLOCK LATENCY ) Operator

Function

Canny

Harris

Line

gauss sobel nms thresh hyst gradprod† harris† maxsup† morph†

3 × 3 binomial smoothing gradient magnitude, direction non-maximum suppression double threshold one hysteresis step gx2 , gy2 , gx gy window sums, response R 3 × 3 maximum, threshold erosion / dilation

✓ ✓ ✓ ✓ 8×

shared

shared

†

New in this paper. The others are from [4].

✓ ✓ ✓ ✓ 2×

win sobel OV7670 camera

capture, FIFO, gray

win gauss

win nms

thresh

K× (win hyst)

edge map

win gradprod

win harris

win maxsup

corners

win thresh(p̄)

win erode

win dilate

row sums, steer

steering

Fig. 1. Three HDPUs working in parallel on one camera stream: Canny edges (top), Harris corners (middle) and line tracking (bottom). Every box marked “win” is a line-buffered 3 × 3 window; every operator produces its result exactly two clock cycles after its window. Blue: operators of the companion paper [4]; green: operators added here. Only the Gaussian stage is shared; the Canny unit uses K = 8 local hysteresis stages.

D. The host The host is an Intel Core i5-12500H with 12 cores, 16 threads and a maximum frequency of 4.5 GHz [21]. To quantify what the HDPUs take off it, we run the same three tasks on the host in two forms. The first is a C++ port that is bit-exact with the HDPUs. The second uses OpenCV 4.5.4: Gaussian blur followed by Canny; the Harris response with dilation-based maximum suppression; and inverse thresholding, morphological opening and row reduction. OpenCV is the code a practitioner would actually write; it is not bit-exact with the HDPUs, nor does it need to be. IV. V ERIFICATION AND M ETHODOLOGY Bit-exact chain. Each HDPU has a Python model that mirrors the hardware operation for operation, including shifts, saturation, border zeroing and tie-breaking. Table II lists every comparison. The RTL cores are simulated with Icarus Verilog on photographs, an RGB565 frame delivered with camera-like gaps, synthetic line-follower frames and noise, and every intermediate stream is compared with the model. The complete system (capture, clock-domain-crossing FIFO, frame controller, three HDPUs and frame buffers) is simulated in Vivado’s simulator against a behavioural OV7670 model that follows the datasheet’s VGA timing [22], and the cores are additionally simulated at four frame widths. The complete system also runs on the board with a live OV7670 camera and VGA output. In total, 14.8 million output values are compared, and not one differs.

corners at six positions across the frame and for the steering decision from the last pixel of the look-ahead row. All designs are placed and routed with Vivado 2024.2 for the XC7A35T-1 at 50 MHz; single units and the scaling study are implemented out of context. Controlled host measurements. In preliminary, uncontrolled runs on the same laptop, medians shifted by up to 55 % between two sessions, so we control the conditions. The benchmark refuses to start while other heavy tools are running or on battery power; it raises the minimum processor state of the power plan from 5 % to 100 % for the run and restores it afterwards; it pins single-thread programs to one performance core (logical CPU 2) and records the core on which each run ends; and it repeats every program in 5 independent processes of 1,000 timed runs after warm-up, shuffling the order of programs in every repetition. Programs run under the Windows Subsystem for Linux and are compiled with g++ -O3 -march=native. Timing covers the processing call only, so camera transfer and operating-system wake-up are excluded; every host latency reported below is therefore a lower bound, and every host load an underestimate. Derived quantities. The host’s sensor-to-decision latency is derived as the time from a pixel’s arrival to the end of the active frame, computed from the camera timing (65.33 µs per line, 30.0 frames per second), plus the measured median processing time. The share of the frame period returned to the host is the measured median processing time divided by the frame period (33.3 ms). Such quantities are labelled derived. V. R ESULTS

TABLE II

A. Cost of the HDPUs

V ERIFICATION AGAINST THE BIT- EXACT MODELS Comparison

Compared

Canny core (RTL) vs. model Harris and line cores (RTL) vs. models Cores at four frame widths (RTL) vs. models Three-HDPU system with camera model vs. models C++ port vs. models

Runs

Differ

every stage, K = 1, 2, 4, 8 every stage, row sums, steering edges, corners, line mask

20

0

7

0

4

0

edges, corners, line mask, steering all outputs, four resolutions

2

0

11

0

HDPU measurements. Latency is measured in the system simulation, from the camera edge that delivers a pixel to the clock edge at which its decision leaves the unit, for edges and

Table III gives the post-route cost. The complete system, comprising camera capture, three HDPUs, two frame buffers and VGA output, uses 2,568 LUTs (12 % of the device), 41.5 block RAMs (83 %) and 7 DSPs, and closes timing at 50 MHz with 6.316 ns of slack. Adding the Harris and line units to the Canny-only system costs 1,113 LUTs and, by Vivado’s estimate, 0.018 W. Most of the block RAM holds the two display frame buffers, not the HDPUs. B. The load the HDPUs remove from the host Table IV gives the host’s time for the three tasks under controlled conditions. On a stored 640 × 480 frame, OpenCV needs 2.84 ms with 16 threads and 5.98 ms on a single core, and the bit-exact C++ port 9.87 ms. The host is fast; a pipeline

Design

LUT

FF

BRAMa

DSP

WNS (ns)

Powerb (W)

Canny core, K = 8 Harris core Line-tracker core

848 720 375

1,768 844 525

6.0 3.5 1.5

2 3 2

8.047 5.634 6.923

0.080 0.091 0.076

Complete systemc Device XC7A35T

2,568 20,800

3,472 41,600

41.5 50

7 90

6.316

0.318

All at 50 MHz, post-route. a 36-kbit blocks. b Vivado estimate, default activity; cores out of context, without I/O. c Camera capture, FIFO, the three HDPUs, two frame buffers and VGA output.

that accepts one pixel per clock at 50 MHz needs 6.1 ms merely to read a stored frame (derived), so for images already in memory the host remains the right tool. At the camera’s 30.0 frames per second, however, this time is spent in every frame period: 18 % of one core with OpenCV and 30 % with the C++ port (derived). At 1920 × 1080, OpenCV takes 20.97 ms using all 16 threads, 63 % of the frame period (derived). With the HDPUs in place, this time returns to the host, which receives finished results instead. The host’s run times are also uneven: under controlled conditions, the medians of the 5 repetitions agree to within 13.7 %, and pinned runs ended on their assigned core in all but 0 of 40,000 cases, yet in every program the slowest run took 1.6–12.5× the median. TABLE IV H OST TIME PER STORED FRAME UNDER CONTROLLED CONDITIONS : OVERALL MEDIAN , RANGE OF THE PER - REPETITION MEDIANS , 99.9 TH PERCENTILE , AND WORST RUN RELATIVE TO THE MEDIAN

Program, 640 × 480 (ms)

median

C++ port (bit-exact), 1 thread, pinned Canny fast path, K = 8 6.58 Harris corners 4.21 Line tracker 0.85 All three 9.87 Exact Canny, in-place 5.81 Exact Canny, two-buffer 13.86 OpenCV, 1 thread, pinned Canny 3.59 Harris corners 2.19 Line tracker 1.78 All three 5.98 OpenCV, 16 threads Canny 0.52 Harris corners 1.84 Line tracker 1.82 All three 2.84

per rep.

p99.9

max/med

6.52–6.63 4.16–4.31 0.84–0.86 9.76–10.01 5.80–5.83 13.73–13.98

9.85 7.29 1.75 16.22 9.86 21.12

2.1 2.4 12.5 3.1 2.0 2.0

3.52–3.69 1.93–2.23 1.76–1.80 5.87–6.07

5.67 4.02 3.50 10.18

2.5 2.4 5.1 2.0

0.51–0.54 1.84–1.87 1.81–1.84 2.75–3.13

0.86 3.37 3.55 4.35

1.7 2.6 3.0 1.6

C. Decisions while the frame is still arriving Fig. 2 shows when each decision becomes available during a frame. The HDPUs track the camera at a fixed distance: an edge decision follows its pixel by 720.6 µs (11.03 lines), a corner decision by 262.1 µs, and the steering decision arrives 274.1 µs after the last pixel of the look-ahead row. A framebased program on the host can produce any decision only after the last active line has arrived and the frame has been processed. Table V compares the two: the HDPUs decide

22–67× sooner. For the steering decision of row 240, the host must still wait for the remaining 239 rows, 17.4 ms in total even with OpenCV’s fastest time. This comparison is with frame-based software; a CPU or microcontroller program written to process the image line by line could also start early, and what the HDPUs add is a latency fixed by their structure and a host that is free of the work. These host figures assume that the frame reaches the processor the instant the camera completes it and that the process is already running; both assumptions favour the host.

Time of decision (ms)

TABLE III H ARDWARE COST ON THE XC7A35T-1 (V IVADO 2024.2)

pixel arrives (camera)

hardwired: steering

hardwired: Canny edge

CPU: all three (OpenCV, 16 thr.)

hardwired: Harris corner

CPU: steering

30

20

10

0 0

100

200 300 Image row

400

Fig. 2. Time at which decisions are available, from the start of the active frame. HDPUs (hardwired): system simulation with the camera model. Host (CPU): derived from the camera timing and the measured median processing time (lower bound).

TABLE V S ENSOR - TO - DECISION LATENCY. H ARDWIRED HDPU S : SYSTEM SIMULATION ( MEDIAN OVER POSITIONS AND FRAMES ). ∗ H OST CPU: DERIVED LOWER BOUND WITH O PEN CV, 16 THREADS . Hardwired HDPU and decision Canny: edge pixel Harris: corner pixel Line: steering, row 240

CPU*

Hardwired

µs

lines

ms

sooner

720.6 262.1 274.1

11.03 4.01 4.20

16.2 17.5 17.4

22× 67× 64×

D. Determinism Fig. 3 shows the distribution of the host’s run times relative to each program’s median. Every program has a tail, and for a system with a deadline it is the tail, not the median, that sets the margin. The HDPU latencies, measured at 26 points (two frames, six positions, three units), vary by at most 17 ns, less than one 20-ns clock cycle; this residue stems from the asynchronous crossing from the camera clock domain. Nothing in an HDPU depends on the image content, on other tasks or on history, so its latency is a property of the design, not of the workload. Fig. 4 shows the outputs of all three units on an aerial river scene and on a line-follower view; in the latter,

the line unit steers left for the curving line, in agreement with the model. C++ port, all three

OpenCV 16 thr., all three

C++ port, exact Canny

OpenCV 16 thr., Canny

OpenCV 1 thr., all three

hardwired, all reflexes

Fraction of runs slower

100

10−1

10−2

10−3

10−4 1

1.5 2 Run time / median

Table VI and Fig. 5 repeat the measurements from 320×240 to 1920 × 1080. On the host, time grows with the number of pixels: the C++ port takes 26.4× longer for a frame with 27× the pixels. In the HDPUs, the logic barely changes: from the smallest to the largest frame, the LUT count grows by a factor of only 1.03, because only the line buffers and the coordinate counters depend on the width; block RAM grows from 9.5 to 20.5. The latency in image lines is constant (11.02–11.14 lines for an edge at every width), and the fraction of the frame that must be held shrinks as frames grow. At 1920 × 1080, the design also closes timing at 74.3 MHz (slack 0.245 ns), the pixel clock of 1080p at 30 frames per second, so one pixel per clock suffices even at that resolution. The larger the frame, the more host time the HDPUs save and the less, relatively, they cost.

3

Fig. 3. Fraction of host runs slower than a given multiple of the median (controlled benchmark, all repetitions pooled). Dashed vertical line: largest relative deviation of any HDPU latency.

edges, corners, line mask

TABLE VI S CALING WITH FRAME SIZE : HDPU S ( OUT- OF - CONTEXT IMPLEMENTATION ; RTL SIMULATION FOR LATENCY ) AND HOST ( CONTROLLED BENCHMARK )

Fig. 4. Outputs of the three HDPUs (models, bit-exact with the RTL) for an aerial photograph of a river and a synthetic line-follower view: Canny edges (white), Harris corners (red), line mask after opening (green), look-ahead row 240 (yellow) and the line centroid on it. In the photograph, the line unit isolates the dark river channel.

E. Data held before a decision A frame-based program needs the whole frame, 4915.2 kbit in RGB565, before its first decision. The three HDPUs together hold only their line buffers: 213.8 kbit, or 4.3 % of a frame (derived from the window widths). This is the structural reason for the latency gap: an HDPU never holds more of the image than its 3 × 3 operators require.

320×240

Frame

640×480

1280×720

Hardwired, all three HDPUs (Vivado; RTL simulation) LUT 1,889 1,943 BRAM (36k) 9.5 11.0 WNS at 50 MHz (ns) 6.161 5.978 Edge latency (lines) 11.14 11.07 State held (% frame) 8.7 4.3 CPU, all three tasks, median ms per frame C++ port, 1 thread 2.46 9.84 OpenCV, 1 thread 1.47 6.03 OpenCV, 16 threads 0.92 2.75

CPU time per frame (ms)

camera frame (gray)

F. Scaling with resolution

(a) CPU

1920×1080

1,961 20.5 6.347 11.04 2.9

1,949 20.5 6.162 11.02 1.9

29.39 17.73 9.05

64.85 39.49 20.97

(b) hardwired

20 15 10

1

10 C++ port OpenCV 1 thr. OpenCV 16 thr.

100

LUT (thousands) BRAM (36k blocks)

5 0

10−1

100

Megapixels per frame

320

640

1280

1920

Frame width (pixels)

Fig. 5. (a) Host time per frame for all three tasks grows with the pixel count. (b) HDPU resources for all three units grow only through line buffers.

G. Does latency matter in a loop? To assess what the latency means for a machine, we model a line-following vehicle (model; details in the repository). A unicycle drives at constant speed along a sinusoidal line (amplitude 150 mm, wavelength 2.0 m). Once per frame, the camera’s look-ahead row 240 observes the line 0.30 m ahead, and a pure-pursuit rule [20] sets the curvature from that observation. All cases see the same rows of the same frames;

Peak lateral error (mm)

only the time at which the decision is applied differs. For the HDPU, this is the measured steering latency. For the host, it is the remainder of the frame, 15.6 ms, plus a processing time drawn from that program’s measured distribution, which again omits transfer and wake-up. The line counts as lost when the lateral error exceeds 100 mm, and each speed is run 8 times for 30 s. Fig. 6 and Table VII show the result. With the HDPU, the vehicle holds the line up to 10.00 m/s, practically the same as with zero delay (9.75 m/s; the two differ by one step of the speed sweep, which we do not interpret as an advantage of the delay). With the same decision computed on the host from the same rows, the limit falls to 3.50–4.75 m/s, depending on the program. The exact figures depend on the vehicle and controller, but the mechanism is general [19]: the delay of frame-based processing is a large fraction of the frame period, and it is incurred in every control cycle. Zero delay (reference)

CPU, OpenCV 1 thr., line

Hardwired line reflex

CPU, C++ port, all three

CPU, C++ port, line only

CPU, OpenCV 16 thr., all three

103

line lost

102

101

0

2

4 6 8 Vehicle speed (m/s)

10

12

Fig. 6. Peak lateral error of the line-following vehicle versus speed, for different sources of the steering decision (model, with measured latencies).

TABLE VII H IGHEST SPEED AT WHICH THE LINE IS HELD IN EVERY RUN ( MODEL ). ∗ R EST OF THE FRAME PLUS MEDIAN PROCESSING TIME ; THE MODEL DRAWS PROCESSING TIMES FROM THE MEASURED DISTRIBUTION . Steering decision from Zero delay (reference) Hardwired line reflex CPU, C++ port, line only CPU, OpenCV 1 thr., line CPU, C++ port, all three CPU, OpenCV 16 thr., all three

delay (ms)

max. speed (m/s)

0.00 0.27 16.46∗ 17.39∗ 25.48∗ 18.46∗

9.75 10.00 4.75 4.50 3.50 4.50

H. Power Vivado estimates the complete three-HDPU system at 0.318 W, of which the Harris and line units add 0.018 W. The host’s documented base power is 45 W, with a turbo limit of 95 W [21]. The two figures are not a like-for-like comparison (a tool estimate for the HDPUs, a datasheet value for the host),

so we draw no ratio from them; the difference is nevertheless consistent with the instruction and data-movement overheads described by Horowitz [8], which an HDPU does not incur. A host relieved of the pre-processing can also spend more of its time in low-power states. VI. D ISCUSSION Why distribute the work. The measurements show what each part of the system does best. The host is fast and flexible on data in memory. The HDPUs decide while the frame is still arriving, with a latency that does not vary, while holding a few lines rather than a frame and drawing, by Vivado’s estimate, a fraction of a watt. Their advantage comes not from clock speed (50 MHz against 4.5 GHz) but from organisation: each unit is dedicated to one task, works on the pixel stream as it arrives, and runs in parallel with every other unit without competing for shared resources. HDPUs and a host. The two are complementary. A set of hardwired HDPUs next to the sensor handles the always-on, time-critical pre-processing and passes compact results (edge maps, corner lists, a steering code) to the host, which performs the deliberate and flexible work with the time the HDPUs free. Because every HDPU is built on the same operator contract and verified against a model, adding one is a matter of composing operators, as the Harris and line units demonstrate, and it adds no latency to the units already present. Limitations. The hardware results derive from RTL and system simulation with a datasheet camera model and from Vivado implementation; the complete system runs on the board with a live camera, but its latency was measured in simulation, and measuring latency and power on the board remains to be reported. The host figures cover one laptop processor under the Windows Subsystem for Linux; its latencies are derived lower bounds and its freed load is derived from processing times. The closed-loop experiment is a model with an idealised vehicle and a simple controller; it demonstrates the effect of the measured latencies, not a validated vehicle. VII. C ONCLUSION A single library of 3 × 3 operators with a common twoclock latency yields three hardwired distributed processing units (Canny edges, Harris corners and a steering line tracker) that run truly in parallel on one camera stream in a low-cost FPGA and are bit-exact with their software models. Placed at the sensor, the HDPUs decide 22–67× sooner than a framebased program on the host could, with a latency that varies by at most 17 ns, while holding only 4.3 % of a frame and drawing an estimated 0.318 W. They return to the host 18– 30 % of a core in every frame period at VGA resolution, and their cost grows with resolution only through line buffers. In a closed-loop model, they raise the speed at which a vehicle can follow a line from 4.75 to 10.00 m/s. These results support distributing routine, time-critical vision into hardwired units at the sensor, and reserving the host processor for the work that needs its flexibility.

R EFERENCES [1] G. Bradski, “The OpenCV library,” Dr. Dobb’s Journal of Software Tools, 2000. [2] V. W. Lee, C. Kim, J. Chhugani, M. Deisher, D. Kim, A. D. Nguyen, N. Satish, M. Smelyanskiy, S. Chennupaty, P. Hammarlund, R. Singhal, and P. Dubey, “Debunking the 100X GPU vs. CPU myth: An evaluation of throughput computing on CPU and GPU,” in Proc. International Symposium on Computer Architecture (ISCA), 2010, pp. 451–460. [3] F. Zhou and Y. Chai, “Near-sensor and in-sensor computing,” Nature Electronics, vol. 3, no. 11, pp. 664–671, 2020. [4] R. Mali, “Fast path, exact path: One library of hardwired operators for streaming and exact Canny edge detection on a low-cost FPGA,” arXiv:2610.33107 [cs.AR], 2026. [Online]. Available: https: //arxiv.org/abs/2610.33107 [5] J. Canny, “A computational approach to edge detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-8, no. 6, pp. 679–698, 1986. [6] C. Harris and M. Stephens, “A combined corner and edge detector,” in Proc. Alvey Vision Conference, 1988, pp. 23.1–23.6. [7] R. M. Haralick, S. R. Sternberg, and X. Zhuang, “Image analysis using mathematical morphology,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-9, no. 4, pp. 532–550, 1987. [8] M. Horowitz, “Computing’s energy problem (and what we can do about it),” in Proc. IEEE International Solid-State Circuits Conference (ISSCC), 2014, pp. 10–14. [9] I. Kuon and J. Rose, “Measuring the gap between FPGAs and ASICs,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 26, no. 2, pp. 203–215, 2007. [10] H. S. Neoh and A. Hazanchuk, “Adaptive edge detection for real-time video processing using FPGAs,” in Global Signal Processing Expo (GSPx), 2004. [11] J. Hegarty, J. Brunhaver, Z. DeVito, J. Ragan-Kelley, N. Cohen, S. Bell, A. Vasilyev, M. Horowitz, and P. Hanrahan, “Darkroom: Compiling high-level image processing code into hardware pipelines,” ACM Transactions on Graphics, vol. 33, no. 4, 2014. [12] AMD, “Vitis vision library,” https://docs.amd.com/r/en-US/Vitis_ Libraries/vision/index.html, version 2026.1, accessed Oct. 2026. [13] P. R. Possa, S. A. Mahmoudi, N. Harb, C. Valderrama, and P. Manneback, “A multi-resolution FPGA-based architecture for real-time edge and corner detection,” IEEE Transactions on Computers, vol. 63, no. 10, pp. 2376–2388, 2014. [14] R. Hameed, W. Qadeer, M. Wachs, O. Azizi, A. Solomatnikov, B. C. Lee, S. Richardson, C. Kozyrakis, and M. Horowitz, “Understanding sources of inefficiency in general-purpose chips,” in Proc. International Symposium on Computer Architecture (ISCA), 2010. [15] G. Venkatesh, J. Sampson, N. Goulding, S. Garcia, V. Bryksin, J. LugoMartinez, S. Swanson, and M. B. Taylor, “Conservation cores: Reducing the energy of mature computations,” in Proc. International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2010. [16] A. Putnam, A. M. Caulfield et al., “A reconfigurable fabric for accelerating large-scale datacenter services,” in Proc. International Symposium on Computer Architecture (ISCA), 2014. [17] D. Firestone et al., “Azure accelerated networking: SmartNICs in the public cloud,” in Proc. USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2018. [18] M. S. B. Altaf and D. A. Wood, “LogCA: A high-level performance model for hardware accelerators,” in Proc. International Symposium on Computer Architecture (ISCA), 2017. [19] K. J. Åström and B. Wittenmark, Computer-Controlled Systems: Theory and Design, 3rd ed. Prentice Hall, 1997. [20] R. C. Coulter, “Implementation of the pure pursuit path tracking algorithm,” Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, Tech. Rep. CMU-RI-TR-92-01, Jan. 1992. [21] Intel, “Intel Core i5-12500H processor product specifications,” https://www.intel.com/content/www/us/en/products/sku/96141/ intel-core-i512500h-processor-18m-cache-up-to-4-50-ghz/ specifications.html, accessed Oct. 2026. [22] OmniVision, “OV7670/OV7171 CMOS VGA (640x480) CameraChip sensor,” Preliminary datasheet, version 1.4, 2006.

Related documents

• Hardwired Distributed Processing Units at the Sensor: Truly Parallel Vision Pre-Processing that Frees the Host CPU#1489822
• MVP: A Motion-Predictive Speculative Vision Pipeline with Non-Blocking Drift Correction#1064250
• RAFI -- A Ray/Work Forwarding Infrastructure for Data Parallel Multi-Node/Multi-GPU Computing#238560
• Dispatching Odyssey: Exploring Performance in Computing Clusters under Real-world Workloads#163119
• WarpDRF: The Unwritten Contract of GPU Warp Programming#1393835