Fabio/face-rknn
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
face-rknn/README.md

10 KiB
Raw Blame History

Face Recognition on Rockchip RK3588

Native C++ face detection and face recognition pipeline for Rockchip RK3588 devices using RKNN Runtime.

Validated on an Orange Pi 5 Plus with Rockchip RK3588.

Pipeline

The pipeline combines:

  • SCRFD 500M KPS face detector at 640x640
  • 5-point facial landmarks
  • SFace 2021 face recognition model
  • RKNN Runtime 2.3.2
  • Native C++ inference
  • stb_image for image loading
  • Cosine similarity for face comparison

The complete pipeline is:

Image
  |
  v
stb_image
  |
  v
SCRFD 500M KPS 640
  |
  +--> face bounding boxes
  |
  +--> 5 facial landmarks
          |
          v
     SFace alignment
        112x112
          |
          v
     SFace recognition
          |
          v
     128-D embedding
          |
          v
     L2 normalization
          |
          v
     cosine similarity

Repository Layout

face-rknn-repo/
├── src/
│   └── face_recognition.cc
├── include/
│   ├── rknn_api.h
│   └── stb_image.h
├── models/
│   ├── onnx/
│   │   ├── SCRFD_500M_KPS_640.onnx
│   │   └── face_recognition_sface_2021dec.onnx
│   ├── rknn/
│   │   ├── SCRFD_500M_KPS_640.rknn
│   │   └── face_recognition_sface_2021dec.rknn
│   └── SHA256SUMS
├── tools/
│   └── conversion/
│       ├── convert_scrfd_rknn.py
│       └── convert_legacy.py
├── scripts/
│   ├── build.sh
│   └── test.sh
├── test/
│   └── test3f.jpg
├── runtime/
├── .gitignore
└── README.md

The runtime directory is intentionally empty in Git. The Rockchip vendor runtime library is installed separately on the target device.

Tested Environment

Target Device

  • Orange Pi 5 Plus
  • Rockchip RK3588
  • ARM64 / aarch64
  • Linux
  • RKNN Runtime 2.3.2

Runtime version:

librknnrt version: 2.3.2

Validated runtime library:

  • version: 2.3.2
  • size: 7,726,232 bytes
  • MD5: a37ee1d5d664c79836bf6e35b7ef6289

The runtime library is not committed to this repository.

Conversion Environment

Model conversion was performed on a Debian x86 system using:

Python 3.11.2
RKNN Toolkit2 2.3.2
RKNN Toolkit2 commit: bd980be9

The Python virtual environment used for conversion was:

/home/fabio/photo-ai/rknn-env

The virtual environment is not included in the repository.

Models

SCRFD 500M KPS

SCRFD is used for face detection and extraction of five facial landmarks.

ONNX model:

models/onnx/SCRFD_500M_KPS_640.onnx

RKNN model:

models/rknn/SCRFD_500M_KPS_640.rknn

Target platform:

rk3588

Quantization:

disabled

Conversion script:

tools/conversion/convert_scrfd_rknn.py

SFace

SFace is used to generate a 128-dimensional face embedding.

ONNX model:

models/onnx/face_recognition_sface_2021dec.onnx

RKNN model:

models/rknn/face_recognition_sface_2021dec.rknn

The RKNN model stored in this repository is the verified model used by the C++ application.

The exact historical conversion recipe for the SFace RKNN model was not completely preserved, therefore this repository does not claim that the SFace conversion is fully reproducible byte-for-byte.

SCRFD Configuration

The SCRFD input is:

  • 640x640
  • RGB
  • FP16
  • NHWC

Image preprocessing:

  • top-left letterbox
  • aspect ratio preserved
  • padding added to reach 640x640
  • RGB channel order

Normalization:

(pixel - 127.5) / 128

Detector configuration:

  • strides: 8, 16, 32
  • anchors per location: 2
  • detection threshold: 0.50
  • NMS IoU threshold: 0.45

The detector produces:

  • bounding boxes
  • confidence scores
  • five facial landmarks

SFace Configuration

The five canonical SFace landmarks are:

(38.2946, 51.6963)
(73.5318, 51.5014)
(56.0252, 71.7366)
(41.5493, 92.3655)
(70.7299, 92.2041)

The detected face is aligned using these landmarks and warped to:

112x112

SFace input:

  • RGB
  • uint8 image values represented as FP16 NHWC
  • range 0..255
  • pass_through=1

The output embedding has 128 dimensions.

The embedding is L2-normalized before comparison.

C++ Application

The main application is:

src/face_recognition.cc

The executable is:

bin/face_recognition

The application expects two image paths:

./bin/face_recognition image1.jpg image2.jpg

It detects faces in both images, extracts the corresponding embeddings and computes cosine similarity.

The current test application compares the first detected face in each image.

Face Comparison

Face similarity is computed using cosine similarity between the two L2-normalized 128-dimensional embeddings.

The comparison threshold used by the current application is:

0.363

A similarity above this threshold is considered a match by the current test application.

This threshold is part of the validated application configuration and should not be interpreted as a universal SFace threshold for every deployment or dataset.

Self-Comparison Test

Comparing an image with itself produces:

cosine similarity = 1.0

The reference test image:

test/test3f.jpg

contains exactly three detected faces in the validated test.

Build

The application is intended to be compiled on the ARM64/RK3588 target.

Build script:

scripts/build.sh

Run:

./scripts/build.sh

The resulting executable is:

bin/face_recognition

The executable is linked against the runtime library located in:

runtime/librknnrt.so

The build uses an rpath relative to the executable:

$ORIGIN/../runtime

This allows the application to use a repository-local runtime without requiring a system-wide installation.

Test

The test script is:

scripts/test.sh

Run the default self-comparison:

./scripts/test.sh

This uses:

test/test3f.jpg

for both inputs.

Two explicit images can also be supplied:

./scripts/test.sh image1.jpg image2.jpg

Model Integrity

SHA256 checksums for all committed models are stored in:

models/SHA256SUMS

Verify the models with:

cd models
sha256sum -c SHA256SUMS

Expected result:

face_recognition_sface_2021dec.onnx: OK
SCRFD_500M_KPS_640.onnx: OK
face_recognition_sface_2021dec.rknn: OK
SCRFD_500M_KPS_640.rknn: OK

Current SHA256 values:

face_recognition_sface_2021dec.onnx
0ba9fbfa01b5270c96627c4ef784da859931e02f04419c829e83484087c34e79

SCRFD_500M_KPS_640.onnx
857efab2e0a5184ec86ffa7d0bf33ac94da92591e7650d5353622ce367218faf

face_recognition_sface_2021dec.rknn
5f36840c6fea8a4772a45fe5eb3456b7bd2031e2986947154fcbd165f918f2ec

SCRFD_500M_KPS_640.rknn
7d74abdedebc5fe25c98195db75cc915df74dd58d941be60c1c260186ac764e2

Conversion

Model conversion was performed separately from the target runtime.

The repository contains the conversion scripts used for the validated SCRFD conversion and the historical generic conversion tooling.

SCRFD Conversion

The SCRFD conversion script is:

tools/conversion/convert_scrfd_rknn.py

Its essential configuration is:

from rknn.api import RKNN

ONNX_MODEL = "SCRFD_500M_KPS_640.onnx"
RKNN_MODEL = "SCRFD_500M_KPS_640.rknn"

rknn = RKNN(verbose=True)
rknn.config(target_platform="rk3588")
rknn.load_onnx(model=ONNX_MODEL)
rknn.build(do_quantization=False)
rknn.export_rknn(RKNN_MODEL)
rknn.release()

The script expects the ONNX model in the current working directory.

SFace Conversion

The repository includes:

tools/conversion/convert_legacy.py

This is historical generic ONNX-to-RKNN conversion tooling.

The validated SFace RKNN model is committed to the repository, but the complete original conversion procedure, including all intermediate optimization steps and exact conversion inputs, was not fully preserved.

Therefore:

  • the committed SFace RKNN model is reproducible as an artifact
  • its SHA256 checksum is verified
  • the exact original byte-for-byte conversion process is not claimed to be reproducible

Reproducibility

The repository is intended to preserve the working state of the validated pipeline.

The following are versioned:

  • C++ source
  • RKNN API header
  • stb_image header
  • ONNX models
  • RKNN models
  • conversion scripts
  • build script
  • test script
  • test image
  • SHA256 checksums
  • documentation

The following are intentionally not versioned:

  • Python virtual environments
  • build artifacts
  • compiled executables
  • shared libraries
  • vendor runtime binaries
  • temporary conversion files
  • editor configuration

The target application can therefore be rebuilt on an ARM64/RK3588 system while keeping the validated model artifacts and source code under version control.

Runtime Library

The Rockchip RKNN runtime is a vendor-provided binary.

The validated version is:

2.3.2

The repository deliberately does not commit:

librknnrt.so

The target device must provide a compatible RKNN Runtime installation or the runtime library must be placed locally in:

runtime/librknnrt.so

The build system uses that local library when compiling.

Third-Party Components

This project uses third-party components including:

  • Rockchip RKNN Runtime
  • Rockchip RKNN Toolkit2
  • SCRFD
  • SFace
  • stb_image

Their respective licenses and redistribution terms remain applicable.

This repository does not claim ownership of those third-party components.

License

The application source in this repository should be considered project-specific code.

Third-party components, models, headers and runtime libraries remain subject to their original licenses and terms.

Before redistributing the complete repository or its models, verify the applicable licenses and redistribution permissions for each third-party component.

Status

Current validated status:

  • SCRFD RKNN inference: working
  • SFace RKNN inference: working
  • Face landmark extraction: working
  • Face alignment: working
  • 128-D embedding generation: working
  • L2 normalization: working
  • Cosine similarity: working
  • Self-comparison: cosine similarity 1.0
  • Three-face test image: validated
  • ARM64/RK3588 native C++ application: working
  • Local RKNN runtime loading: working
  • Model SHA256 verification: working

The repository represents the validated working baseline of the RK3588 face-recognition pipeline.