face-rknn/README.md

# Face Recognition on Rockchip RK3588

Native C++ face detection and face recognition pipeline for Rockchip RK3588 devices using RKNN Runtime.

Validated on an Orange Pi 5 Plus with Rockchip RK3588.

## Pipeline

The pipeline combines:

- SCRFD 500M KPS face detector at 640x640
- 5-point facial landmarks
- SFace 2021 face recognition model
- RKNN Runtime 2.3.2
- Native C++ inference
- stb_image for image loading
- Cosine similarity for face comparison

The complete pipeline is:

```text
Image
  |
  v
stb_image
  |
  v
SCRFD 500M KPS 640
  |
  +--> face bounding boxes
  |
  +--> 5 facial landmarks
          |
          v
     SFace alignment
        112x112
          |
          v
     SFace recognition
          |
          v
     128-D embedding
          |
          v
     L2 normalization
          |
          v
     cosine similarity
```

## Repository Layout

```text
face-rknn-repo/
├── src/
│   └── face_recognition.cc
├── include/
│   ├── rknn_api.h
│   └── stb_image.h
├── models/
│   ├── onnx/
│   │   ├── SCRFD_500M_KPS_640.onnx
│   │   └── face_recognition_sface_2021dec.onnx
│   ├── rknn/
│   │   ├── SCRFD_500M_KPS_640.rknn
│   │   └── face_recognition_sface_2021dec.rknn
│   └── SHA256SUMS
├── tools/
│   └── conversion/
│       ├── convert_scrfd_rknn.py
│       └── convert_legacy.py
├── scripts/
│   ├── build.sh
│   └── test.sh
├── test/
│   └── test3f.jpg
├── runtime/
├── .gitignore
└── README.md
```

The runtime directory is intentionally empty in Git. The Rockchip vendor runtime library is installed separately on the target device.

## Tested Environment

### Target Device

- Orange Pi 5 Plus
- Rockchip RK3588
- ARM64 / aarch64
- Linux
- RKNN Runtime 2.3.2

Runtime version:

```text
librknnrt version: 2.3.2
```

Validated runtime library:

- version: 2.3.2
- size: 7,726,232 bytes
- MD5: a37ee1d5d664c79836bf6e35b7ef6289

The runtime library is not committed to this repository.

## Conversion Environment

Model conversion was performed on a Debian x86 system using:

```text
Python 3.11.2
RKNN Toolkit2 2.3.2
RKNN Toolkit2 commit: bd980be9
```

The Python virtual environment used for conversion was:

```text
/home/fabio/photo-ai/rknn-env
```

The virtual environment is not included in the repository.

# Models

## SCRFD 500M KPS

SCRFD is used for face detection and extraction of five facial landmarks.

ONNX model:

```text
models/onnx/SCRFD_500M_KPS_640.onnx
```

RKNN model:

```text
models/rknn/SCRFD_500M_KPS_640.rknn
```

Target platform:

```text
rk3588
```

Quantization:

```text
disabled
```

Conversion script:

```text
tools/conversion/convert_scrfd_rknn.py
```

## SFace

SFace is used to generate a 128-dimensional face embedding.

ONNX model:

```text
models/onnx/face_recognition_sface_2021dec.onnx
```

RKNN model:

```text
models/rknn/face_recognition_sface_2021dec.rknn
```

The RKNN model stored in this repository is the verified model used by the C++ application.

The exact historical conversion recipe for the SFace RKNN model was not completely preserved, therefore this repository does not claim that the SFace conversion is fully reproducible byte-for-byte.

# SCRFD Configuration

The SCRFD input is:

- 640x640
- RGB
- FP16
- NHWC

Image preprocessing:

- top-left letterbox
- aspect ratio preserved
- padding added to reach 640x640
- RGB channel order

Normalization:

```text
(pixel - 127.5) / 128
```

Detector configuration:

- strides: 8, 16, 32
- anchors per location: 2
- detection threshold: 0.50
- NMS IoU threshold: 0.45

The detector produces:

- bounding boxes
- confidence scores
- five facial landmarks

# SFace Configuration

The five canonical SFace landmarks are:

```text
(38.2946, 51.6963)
(73.5318, 51.5014)
(56.0252, 71.7366)
(41.5493, 92.3655)
(70.7299, 92.2041)
```

The detected face is aligned using these landmarks and warped to:

```text
112x112
```

SFace input:

- RGB
- uint8 image values represented as FP16 NHWC
- range 0..255
- pass_through=1

The output embedding has 128 dimensions.

The embedding is L2-normalized before comparison.

# C++ Application

The main application is:

```text
src/face_recognition.cc
```

The executable is:

```text
bin/face_recognition
```

The application expects two image paths:

```text
./bin/face_recognition image1.jpg image2.jpg
```

It detects faces in both images, extracts the corresponding embeddings and computes cosine similarity.

The current test application compares the first detected face in each image.

# Face Comparison

Face similarity is computed using cosine similarity between the two L2-normalized 128-dimensional embeddings.

The comparison threshold used by the current application is:

```text
0.363
```

A similarity above this threshold is considered a match by the current test application.

This threshold is part of the validated application configuration and should not be interpreted as a universal SFace threshold for every deployment or dataset.

## Self-Comparison Test

Comparing an image with itself produces:

```text
cosine similarity = 1.0
```

The reference test image:

```text
test/test3f.jpg
```

contains exactly three detected faces in the validated test.

# Build

The application is intended to be compiled on the ARM64/RK3588 target.

Build script:

```text
scripts/build.sh
```

Run:

```bash
./scripts/build.sh
```

The resulting executable is:

```text
bin/face_recognition
```

The executable is linked against the runtime library located in:

```text
runtime/librknnrt.so
```

The build uses an rpath relative to the executable:

```text
$ORIGIN/../runtime
```

This allows the application to use a repository-local runtime without requiring a system-wide installation.

# Test

The test script is:

```text
scripts/test.sh
```

Run the default self-comparison:

```bash
./scripts/test.sh
```

This uses:

```text
test/test3f.jpg
```

for both inputs.

Two explicit images can also be supplied:

```bash
./scripts/test.sh image1.jpg image2.jpg
```

# Model Integrity

SHA256 checksums for all committed models are stored in:

```text
models/SHA256SUMS
```

Verify the models with:

```bash
cd models
sha256sum -c SHA256SUMS
```

Expected result:

```text
face_recognition_sface_2021dec.onnx: OK
SCRFD_500M_KPS_640.onnx: OK
face_recognition_sface_2021dec.rknn: OK
SCRFD_500M_KPS_640.rknn: OK
```

Current SHA256 values:

```text
face_recognition_sface_2021dec.onnx
0ba9fbfa01b5270c96627c4ef784da859931e02f04419c829e83484087c34e79

SCRFD_500M_KPS_640.onnx
857efab2e0a5184ec86ffa7d0bf33ac94da92591e7650d5353622ce367218faf

face_recognition_sface_2021dec.rknn
5f36840c6fea8a4772a45fe5eb3456b7bd2031e2986947154fcbd165f918f2ec

SCRFD_500M_KPS_640.rknn
7d74abdedebc5fe25c98195db75cc915df74dd58d941be60c1c260186ac764e2
```

# Conversion

Model conversion was performed separately from the target runtime.

The repository contains the conversion scripts used for the validated SCRFD conversion and the historical generic conversion tooling.

## SCRFD Conversion

The SCRFD conversion script is:

```text
tools/conversion/convert_scrfd_rknn.py
```

Its essential configuration is:

```python
from rknn.api import RKNN

ONNX_MODEL = "SCRFD_500M_KPS_640.onnx"
RKNN_MODEL = "SCRFD_500M_KPS_640.rknn"

rknn = RKNN(verbose=True)
rknn.config(target_platform="rk3588")
rknn.load_onnx(model=ONNX_MODEL)
rknn.build(do_quantization=False)
rknn.export_rknn(RKNN_MODEL)
rknn.release()
```

The script expects the ONNX model in the current working directory.

## SFace Conversion

The repository includes:

```text
tools/conversion/convert_legacy.py
```

This is historical generic ONNX-to-RKNN conversion tooling.

The validated SFace RKNN model is committed to the repository, but the complete original conversion procedure, including all intermediate optimization steps and exact conversion inputs, was not fully preserved.

Therefore:

- the committed SFace RKNN model is reproducible as an artifact
- its SHA256 checksum is verified
- the exact original byte-for-byte conversion process is not claimed to be reproducible

# Reproducibility

The repository is intended to preserve the working state of the validated pipeline.

The following are versioned:

- C++ source
- RKNN API header
- stb_image header
- ONNX models
- RKNN models
- conversion scripts
- build script
- test script
- test image
- SHA256 checksums
- documentation

The following are intentionally not versioned:

- Python virtual environments
- build artifacts
- compiled executables
- shared libraries
- vendor runtime binaries
- temporary conversion files
- editor configuration

The target application can therefore be rebuilt on an ARM64/RK3588 system while keeping the validated model artifacts and source code under version control.

# Runtime Library

The Rockchip RKNN runtime is a vendor-provided binary.

The validated version is:

```text
2.3.2
```

The repository deliberately does not commit:

```text
librknnrt.so
```

The target device must provide a compatible RKNN Runtime installation or the runtime library must be placed locally in:

```text
runtime/librknnrt.so
```

The build system uses that local library when compiling.

# Third-Party Components

This project uses third-party components including:

- Rockchip RKNN Runtime
- Rockchip RKNN Toolkit2
- SCRFD
- SFace
- stb_image

Their respective licenses and redistribution terms remain applicable.

This repository does not claim ownership of those third-party components.

# License

The application source in this repository should be considered project-specific code.

Third-party components, models, headers and runtime libraries remain subject to their original licenses and terms.

Before redistributing the complete repository or its models, verify the applicable licenses and redistribution permissions for each third-party component.

# Status

Current validated status:

- SCRFD RKNN inference: working
- SFace RKNN inference: working
- Face landmark extraction: working
- Face alignment: working
- 128-D embedding generation: working
- L2 normalization: working
- Cosine similarity: working
- Self-comparison: cosine similarity 1.0
- Three-face test image: validated
- ARM64/RK3588 native C++ application: working
- Local RKNN runtime loading: working
- Model SHA256 verification: working

The repository represents the validated working baseline of the RK3588 face-recognition pipeline.