← Back to Project Overview  |  GitHub Repository

1. Target Hardware — OMRON FH-L551

The check_avx.cpp utility uses CPUID to confirm: SSE4.2: YES | AVX: NO | AVX2: NO. This single diagnostic permanently ruled out three optimization paths.

2. Adapter Architecture

File Structure

Inference Pipeline

Image → Letterbox Resize → fillBlobCHW (fused BGR→RGB + normalize + HWC→CHW)
      → inferFixed() (zero-copy IoBinding)
      → Decode (PP-OCR DBNet / YOLOX anchor-free / Generic fallback)
      → [Optional: REC pipeline for text recognition]
      → CSV benchmark log + M4 EWMA compliance check

3. Build History & Compilation

All builds use MSVC cl.exe from a Visual Studio Developer Command Prompt. Dependencies are installed via setup_deps.ps1 (ONNX Runtime 1.17.1 + OpenCV 4.9.0).

Build Commands

V0 Baseline:

cl /O2 /EHsc /Fe:generic_infer.exe ^
   main.cpp ONNXInference.cpp ^
   /I "deps\onnxruntime\include" /I "deps\opencv\build\include" ^
   /link /LIBPATH:"deps\onnxruntime\lib" /LIBPATH:"deps\opencv\build\x64\vc16\lib" ^
   onnxruntime.lib opencv_world490.lib

V1 Optimized (build_opt.bat):

cl /O2 /EHsc /Fe:optimized_generic_infer.exe ^
   main_opt.cpp ONNXInference_opt.cpp ^
   /I "deps\onnxruntime\include" /I "deps\opencv\build\include" ^
   /link /LIBPATH:"deps\onnxruntime\lib" /LIBPATH:"deps\opencv\build\x64\vc16\lib" ^
   onnxruntime.lib opencv_world490.lib

V2 Final (build_final_generic_infer.bat):

cl /O2 /EHsc /Fe:final_generic_infer.exe ^
   main_opt_yoloxopt.cpp ONNXInference_opt_yoloxopt.cpp ^
   /I "deps\onnxruntime\include" /I "deps\opencv\build\include" ^
   /link /LIBPATH:"deps\onnxruntime\lib" /LIBPATH:"deps\opencv\build\x64\vc16\lib" ^
   onnxruntime.lib opencv_world490.lib

10 Build Iterations

4. Optimization Details

4.1 Zero-Copy IoBinding (V1)

Before (V0): infer() copies all input data into ONNX Runtime’s internal buffers, then copies output data back out.

After (V1): inferFixed() binds raw float* pointers via Ort::IoBinding. ONNX Runtime reads/writes directly to caller’s memory.

// V2 binding cache: skip re-bind when pointers unchanged
const bool binding_unchanged =
    binding_cache_valid_ &&
    last_input_ptrs_ == inputPtrs &&
    last_output_ptrs_ == outputPtrs;

if (!binding_unchanged) {
    binding_->ClearBoundInputs();
    binding_->ClearBoundOutputs();
    // ... rebind inputs and outputs ...
    binding_cache_valid_ = true;
}
session_->Run(Ort::RunOptions{nullptr}, *binding_);

4.2 Fused Preprocessing (V1)

Before (V0) — makeBlobCHW(): 3 separate passes through the image:

1. cv::cvtColor (BGR→RGB) — full image scan #1
2. convertTo (normalize) — full image scan #2
3. cv::split + memcpy (HWC→CHW) — full image scan #3

After (V1) — fillBlobCHW(): Single fused loop. For each pixel: read BGR, swap to RGB, normalize, write directly to CHW plane pointers.

// Single pass: read BGR, write normalized RGB in CHW layout
const uint8_t* ptr = bgr.ptr<uint8_t>(0);
for (int i = 0; i < N; ++i) {
    float b = ptr[3*i + 0] * scale;
    float g = ptr[3*i + 1] * scale;
    float r = ptr[3*i + 2] * scale;
    if (do_norm) { r = (r - m0)/s0; g = (g - m1)/s1; b = (b - m2)/s2; }
    pR[i] = r; pG[i] = g; pB[i] = b;
}

4.3 Hoisted Buffer Allocation (V1)

Before: std::vector<float> detBlob; declared inside loop → heap alloc/free every frame.

After: Declared before loop with reserve(). Inside loop: clear() resets size without freeing memory.

5. OS-Level Tuning

5.1 ONNX Runtime Session Configuration

5.2 Memory Architecture

• Arena Allocator: ONNX Runtime pre-reserves a memory pool at startup. Disabling it caused 2.2% latency increase from repeated OS-level malloc/free calls.
• Hoisted Allocation: Reduced per-frame heap allocations from ~14 to 0.
• Zero-Copy IoBinding: Eliminates internal memcpy — critical when working set exceeds L2 cache.

5.3 OS Scheduler Interaction

SetThreadAffinityMask(GetCurrentThread(), 1) was tested and rejected. On a 2-core system, forcing inference to core 0 competes with Windows kernel threads, causing 7.8× worse jitter (260 ms vs 33 ms).

6. Benchmark Results

All experiments benchmarked across COCO val2017 (5,000 images) with the built-in CSV logger.

The benchmark harness records: det_pre_ms, det_run_ms, det_decode_ms, rec_total_ms, total_ms, CER, WER per frame.

7. API Reference

// Construction
ONNXInference engine("model.onnx");                // Default options
ONNXInference engine("model.onnx", customOpts);     // Custom config

// Legacy inference (copy-based, V0)
std::vector<Tensor> outputs = engine.infer({inputBlob});

// Zero-copy inference (production, V1+)
engine.prepareBinding();
engine.inferFixed(inputPtrs, inputShapes, outputPtrs, outputShapes);

// Shape management
engine.fixDynamicHW(640, 640);
engine.fixExactInputShape(0, {1, 3, 48, 320});

// Runtime tuning (triggers session rebuild)
engine.setIntraOpThreads(2);
engine.setGraphOptimization(ORT_ENABLE_ALL);
engine.enableArena(true);
engine.rebuild();

8. Deployment

# Install dependencies
.\setup_deps.ps1

# Build production binary
.\build_final_generic_infer.bat

# Package for FH-L551
.\package_deployment.ps1
# Creates OMRON_FHL551_Deployment.zip

# Run inference
final_generic_infer.exe det.onnx rec.onnx --images list.txt --runs 50 --csv results.csv
final_generic_infer.exe yolox_s.onnx --images coco.txt --opt-level 99 --watchdog-ms 5000 --m4-log

• Back to Project
• GitHub

Feel free to reach out for any opportunities or inquiries.

• stafford_ho@rocketmail.com
• (+65) 8506 0453
• 10 Dover Drive
  Singapore 138683