Fabio/face-clustering-rk3588.
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add reusable RKNN face detection and embedding toolkit

Browse source
This commit is contained in:
Fabio 2026-09-23 14:12:52 +02:00
commit 647824fd26
20 changed files with 10927 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

38
.gitignore vendored Normal file
View file

@ -0,0 +1,38 @@
# Build output
/build/
/bin/*
# Mantiene la directory bin nel repository
!/bin/.gitkeep
# RKNN runtime libraries
/lib/*.so
/lib/*.so.*
# RKNN models
/models/*.rknn
# Local test images
/test/*.jpg
/test/*.jpeg
/test/*.png
/test/*.webp
# Generated data
*.json
*.log
*.out
# Temporary and backup files
*.tmp
*.bak
*.swp
*~
# Editors
.vscode/
.idea/
# Operating system
.DS_Store
Thumbs.db

99
Makefile Normal file
View file

@ -0,0 +1,99 @@
CXX ?= g++
ROOT := $(abspath .)
CXXFLAGS := \
-std=c++17 \
-O2 \
-Wall \
-Wextra \
-Wpedantic \
-Iinclude \
-DFACE_CLUSTERING_ROOT=\"$(ROOT)\"
LDFLAGS := \
-L$(ROOT)/lib \
-Wl,-rpath,$(ROOT)/lib
LDLIBS := \
-lrknnrt \
-ldl \
-lpthread
BIN_DIR := bin
BUILD_DIR := build
ENGINE_SOURCE := src/face_engine.cc
ENGINE_OBJECT := $(BUILD_DIR)/face_engine.o
FACE_RECOGNITION_SOURCE := src/face_recognition.cc
FACE_RECOGNITION_OBJECT := $(BUILD_DIR)/face_recognition.o
FACE_RECOGNITION_BINARY := $(BIN_DIR)/face_recognition
FACE_SCAN_SOURCE := src/face_scan.cc
FACE_SCAN_OBJECT := $(BUILD_DIR)/face_scan.o
FACE_SCAN_BINARY := $(BIN_DIR)/face_scan
.PHONY: all clean test info
all: \
$(FACE_RECOGNITION_BINARY) \
$(FACE_SCAN_BINARY)
$(BUILD_DIR):
mkdir -p $(BUILD_DIR)
$(BIN_DIR):
mkdir -p $(BIN_DIR)
$(ENGINE_OBJECT): $(ENGINE_SOURCE) include/face_engine.h | $(BUILD_DIR)
$(CXX) $(CXXFLAGS) -c $< -o $@
$(FACE_RECOGNITION_OBJECT): \
$(FACE_RECOGNITION_SOURCE) \
include/face_engine.h \
include/face_config.h \
| $(BUILD_DIR)
$(CXX) $(CXXFLAGS) -c $< -o $@
$(FACE_SCAN_OBJECT): \
$(FACE_SCAN_SOURCE) \
include/face_engine.h \
include/face_config.h \
| $(BUILD_DIR)
$(CXX) $(CXXFLAGS) -c $< -o $@
$(FACE_RECOGNITION_BINARY): \
$(ENGINE_OBJECT) \
$(FACE_RECOGNITION_OBJECT) \
| $(BIN_DIR)
$(CXX) $^ $(LDFLAGS) $(LDLIBS) -o $@
$(FACE_SCAN_BINARY): \
$(ENGINE_OBJECT) \
$(FACE_SCAN_OBJECT) \
| $(BIN_DIR)
$(CXX) $^ $(LDFLAGS) $(LDLIBS) -o $@
test: all
./bin/face_recognition \
test/test.jpg \
test/test3f.jpg
./bin/face_scan \
test/test3f.jpg \
> /tmp/face_scan_test.json
python3 -m json.tool \
/tmp/face_scan_test.json \
> /dev/null
@echo "Tests completed successfully"
info:
@echo "ROOT=$(ROOT)"
@echo "CXX=$(CXX)"
@echo "CXXFLAGS=$(CXXFLAGS)"
@echo "LDFLAGS=$(LDFLAGS)"
clean:
rm -rf $(BUILD_DIR)
rm -f $(FACE_RECOGNITION_BINARY)
rm -f $(FACE_SCAN_BINARY)

558
README.md Normal file
View file

@ -0,0 +1,558 @@
# Face Clustering RK3588
Toolkit C++ riutilizzabile per il rilevamento dei volti, la generazione di
embedding facciali e il futuro clustering tramite HDBSCAN su dispositivi
Rockchip RK3588.
Il progetto usa RKNN per eseguire sulla NPU:
- SCRFD per il rilevamento dei volti;
- SFace per la generazione degli embedding;
- normalizzazione L2 degli embedding;
- confronto tramite cosine similarity.
Il motore non dipende da database, server o applicazioni specifiche e può
essere integrato in altri programmi C++ oppure utilizzato tramite il
programma `face_scan` e il relativo output JSON.
## Stato del progetto
Funzionalità attualmente disponibili:
- caricamento delle immagini RGB;
- rilevamento facciale tramite SCRFD;
- confidence score per ogni volto;
- bounding box;
- cinque landmark facciali;
- non-maximum suppression;
- allineamento del volto a 112 x 112 pixel;
- generazione di embedding SFace a 128 dimensioni;
- conversione dell'output in float32;
- normalizzazione L2;
- confronto tramite cosine similarity;
- output JSON per applicazioni esterne;
- compilazione tramite Makefile.
Funzionalità pianificate:
- ricerca kNN tramite OpenCL;
- TOPK uguale a 32;
- core distance;
- mutual reachability distance;
- minimum spanning tree;
- condensed tree;
- clustering HDBSCAN;
- assegnazione di un cluster a ogni embedding facciale.
## Pipeline facciale
```text
Immagine
|
v
Caricamento RGB
|
v
Resize proporzionale
|
v
Input SCRFD 640 x 640
|
v
SCRFD su RKNN/NPU
|
+-- confidence score
+-- bounding box
+-- cinque landmark
|
v
Non-maximum suppression
|
v
Similarity transform
|
v
Volto allineato 112 x 112
|
v
SFace su RKNN/NPU
|
v
Embedding float32[128]
|
v
Normalizzazione L2
```
Poiché gli embedding sono normalizzati, la cosine similarity tra due
embedding corrisponde al loro prodotto scalare.
## Programmi disponibili
### face_recognition
`face_recognition` è il programma dimostrativo con output leggibile
da una persona.
Può analizzare una singola immagine oppure confrontare tutti i volti trovati
in due immagini.
Scansione di una singola immagine:
```bash
./bin/face_recognition test/image.jpg
```
Confronto tra due immagini:
```bash
./bin/face_recognition \
test/reference.jpg \
test/query.jpg
```
Per ogni volto il programma mostra:
- confidence score;
- bounding box;
- cinque landmark.
Nel confronto fra due immagini calcola la cosine similarity tra ogni volto
della prima immagine e ogni volto della seconda.
La soglia dimostrativa attuale è:
```text
0.363
```
Questa soglia deve essere validata sul dataset reale dell'applicazione.
### face_scan
`face_scan` è destinato all'integrazione con altri programmi.
Accetta il percorso di una singola immagine:
```bash
./bin/face_scan test/image.jpg
```
Scrive:
- il risultato JSON su standard output;
- errori e diagnostica su standard error;
- exit code 0 in caso di successo;
- exit code diverso da 0 in caso di errore.
Per salvare il risultato:
```bash
./bin/face_scan \
test/image.jpg \
> result.json
```
Per verificare e formattare il JSON:
```bash
python3 -m json.tool result.json
```
Il JSON contiene:
- percorso dell'immagine;
- larghezza e altezza originali;
- elenco dei volti;
- indice sequenziale del volto;
- confidence score;
- bounding box;
- cinque landmark;
- embedding di 128 valori.
Il formato completo è documentato in
`docs/json-format.md`.
## Requisiti
- Linux AArch64;
- Rockchip RK3588 o piattaforma RKNN compatibile;
- compilatore con supporto C++17;
- GNU Make;
- runtime RKNN;
- modello SCRFD in formato RKNN;
- modello SFace in formato RKNN;
- Python 3 per la validazione dei test JSON.
## File locali richiesti
### Modelli RKNN
Copiare i modelli nella directory `models`:
```text
models/SCRFD_500M_KPS_640.rknn
models/face_recognition_sface_2021dec.rknn
```
### Runtime RKNN
Copiare la libreria runtime in:
```text
lib/librknnrt.so
```
I modelli RKNN e la libreria runtime sono esclusi dal repository tramite
`.gitignore`.
Prima di distribuire questi file, verificare le rispettive condizioni di
licenza e redistribuzione.
## Compilazione
Dalla directory principale del repository:
```bash
make
```
Compilazione parallela:
```bash
make -j"$(nproc)"
```
I programmi vengono creati in:
```text
bin/face_recognition
bin/face_scan
```
Per effettuare una compilazione pulita:
```bash
make clean
make -j"$(nproc)"
```
## Test
Le immagini di test locali devono essere copiate nella directory
`test`.
Per eseguire i test automatici:
```bash
make test
```
Il test:
1. esegue `face_recognition`;
2. esegue `face_scan`;
3. salva temporaneamente il JSON;
4. valida la sintassi JSON tramite Python.
Le immagini sono escluse dal repository tramite `.gitignore`.
## Utilizzo da un altro programma C++
Includere l'header pubblico:
```cpp
#include "face_engine.h"
```
Inizializzare il motore:
```cpp
FaceEngine engine;
if (!engine.init(
"models/SCRFD_500M_KPS_640.rknn",
"models/face_recognition_sface_2021dec.rknn")) {
return 1;
}
```
Elaborare un'immagine:
```cpp
std::vector<Face> faces;
int image_width = 0;
int image_height = 0;
if (!engine.process_image(
"image.jpg",
faces,
image_width,
image_height)) {
return 1;
}
```
Ogni elemento `Face` contiene:
- `score`, confidence del detector;
- `x1`, `y1`, `x2` e `y2`;
- cinque elementi `kps`;
- un embedding L2-normalizzato di 128 elementi.
Esempio di accesso ai risultati:
```cpp
for (const Face& face : faces) {
std::cout
<< "score=" << face.score
<< " bbox="
<< face.x1 << ","
<< face.y1 << ","
<< face.x2 << ","
<< face.y2
<< "\n";
if (face.embedding.size() == 128) {
std::cout << "Embedding valido\n";
}
}
```
## Utilizzo da altri linguaggi
Programmi Node.js, Python, Java, Dart o scritti in altri linguaggi possono
eseguire `bin/face_scan` come processo figlio.
Il contratto di integrazione è:
```text
Input percorso di una singola immagine
stdout oggetto JSON completo
stderr diagnostica ed errori
exit 0 scansione completata
exit != 0 errore
```
Un'immagine valida senza volti produce comunque exit code 0 e:
```json
{
"image": {
"path": "image.jpg",
"width": 1920,
"height": 1080
},
"faces": []
}
```
## Formato dell'embedding
Ogni embedding contiene:
```text
Dimensione logica 128
Tipo nel motore float32
Norma L2 circa 1.0
Valori tutti finiti
```
Se salvato come sequenza binaria float32:
```text
128 x 4 byte = 512 byte
```
Questo formato sarà utilizzabile direttamente dai futuri moduli OpenCL.
## Struttura del repository
```text
face-clustering-rk3588/
|-- Makefile
|-- README.md
|-- include/
| |-- face_config.h
| |-- face_engine.h
| |-- rknn_api.h
| `-- stb_image.h
|-- src/
| |-- face_engine.cc
| |-- face_recognition.cc
| `-- face_scan.cc
|-- docs/
| |-- architecture.md
| `-- json-format.md
|-- models/
| `-- README.md
|-- lib/
| `-- README.md
|-- test/
| `-- README.md
|-- modules/
| |-- knn-opencl/
| |-- mutual-reachability/
| |-- mst/
| `-- hdbscan/
|-- build/
`-- bin/
```
## Architettura del motore
Il codice è diviso in tre componenti principali.
### face_engine
Implementa:
- caricamento immagine;
- resize bilineare;
- conversione float32 e float16;
- inferenza SCRFD;
- decodifica dei risultati;
- non-maximum suppression;
- allineamento facciale;
- inferenza SFace;
- normalizzazione dell'embedding.
### face_recognition
Fornisce un programma dimostrativo e permette di verificare visivamente:
- volti rilevati;
- bounding box;
- landmark;
- similarità fra embedding.
### face_scan
Fornisce un'interfaccia machine-readable tramite JSON.
Il programma non scrive log su standard output, così l'output può essere
letto direttamente con un parser JSON.
## Pipeline futura di clustering
Il repository verrà esteso con questa pipeline:
```text
Embedding SFace float32[N][128]
|
v
OpenCL brute-force kNN
|
v
TOPK=32
|
+-- indices[N][32]
`-- scores[N][32]
|
v
Core distance
|
v
Mutual reachability graph
|
v
Minimum spanning tree
|
v
Single linkage hierarchy
|
v
Condensed tree
|
v
HDBSCAN cluster assignment
```
## Moduli futuri
### knn-opencl
Calcolerà i 32 vicini più prossimi per ogni embedding usando il prodotto
scalare tra vettori L2-normalizzati.
### mutual-reachability
Calcolerà:
```text
mrd(i,j) =
max(
core_distance(i),
core_distance(j),
distance(i,j)
)
```
### mst
Costruirà il minimum spanning tree del grafo di mutual reachability.
### hdbscan
Produrrà:
- gerarchia single linkage;
- condensed tree;
- stabilità dei cluster;
- assegnazione finale dei cluster;
- identificazione del rumore.
## Roadmap
```text
Face detection e embedding completato
Confronto cosine completato
Output JSON completato
Makefile completato
Documentazione completato
OpenCL kNN TOPK=32 pianificato
Core distance pianificato
Mutual reachability pianificato
Minimum spanning tree pianificato
HDBSCAN condensed tree pianificato
Cluster assignment pianificato
```
## Licenze e componenti esterni
Il repository contiene o utilizza componenti esterni, inclusi:
- stb_image;
- header API RKNN;
- runtime RKNN;
- modelli SCRFD;
- modello SFace.
Prima della pubblicazione o redistribuzione verificare separatamente le
condizioni di licenza di ogni componente.
I modelli e il runtime non vengono inseriti automaticamente nel repository.
## Obiettivo finale
L'obiettivo è fornire una pipeline indipendente e riutilizzabile:
```text
Immagini
|
v
Volti e embedding
|
v
Grafo kNN
|
v
HDBSCAN
|
v
Cluster di identità facciali
```
Il server o l'applicazione che utilizza il toolkit rimane responsabile della
persistenza, dell'interfaccia utente e dell'eventuale associazione dei
cluster a persone reali.

0
bin/.gitkeep Normal file
View file

144
docs/architecture.md Normal file
View file

@ -0,0 +1,144 @@
# Architettura
Il progetto separa il motore di elaborazione facciale dai programmi che lo
utilizzano.
## Componenti principali
### face_engine.h
Definisce l'interfaccia pubblica C++:
- `Point`;
- `Face`;
- `FaceEngine`;
- inizializzazione dei modelli;
- elaborazione di una singola immagine.
### face_engine.cc
Implementa:
- caricamento delle immagini tramite stb_image;
- resize bilineare;
- conversione float32 e float16;
- inferenza SCRFD tramite RKNN;
- decodifica di score, bounding box e landmark;
- non-maximum suppression;
- trasformazione similarity;
- allineamento facciale;
- inferenza SFace tramite RKNN;
- conversione dell'output;
- normalizzazione L2 dell'embedding.
### face_recognition.cc
Programma dimostrativo con output leggibile.
Può:
- elaborare una singola immagine;
- elaborare due immagini;
- confrontare tutti i volti rilevati tramite cosine similarity.
### face_scan.cc
Programma destinato all'integrazione con altri software.
Caratteristiche:
- elabora una singola immagine;
- scrive esclusivamente JSON su standard output;
- scrive diagnostica ed errori su standard error;
- valida la dimensione degli embedding;
- rifiuta valori NaN o infiniti;
- restituisce exit code documentati.
## Pipeline runtime
```text
File immagine
|
v
Caricamento RGB
|
v
Resize proporzionale
|
v
Input SCRFD 640 x 640 FLOAT16 NHWC
|
v
SCRFD su NPU
|
v
Decodifica livelli stride 8, 16 e 32
|
v
Score threshold
|
v
Non-maximum suppression
|
v
Bounding box e cinque landmark
|
v
Similarity transform
|
v
Volto allineato 112 x 112
|
v
SFace su NPU
|
v
Embedding 128D
|
v
Normalizzazione L2
```
## Riutilizzo
Le applicazioni C++ possono utilizzare direttamente `FaceEngine`.
Applicazioni scritte in altri linguaggi possono eseguire
`face_scan` come processo figlio e analizzare il JSON prodotto.
Il motore non dipende:
- dal server della galleria;
- dal database;
- dall'interfaccia utente;
- da uno specifico framework applicativo.
## Estensione futura
Gli embedding prodotti da `FaceEngine` saranno utilizzati dalla
pipeline di clustering:
```text
FaceEngine
|
v
Embedding float32[N][128]
|
v
kNN OpenCL TOPK=32
|
v
Core distance
|
v
Mutual reachability
|
v
Minimum spanning tree
|
v
HDBSCAN
|
v
Cluster ID per embedding
```

113
docs/json-format.md Normal file
View file

@ -0,0 +1,113 @@
# Formato JSON di face_scan
`face_scan` scrive un singolo oggetto JSON su standard output.
Gli errori e i messaggi diagnostici vengono scritti su standard error.
## Invocazione
```bash
./bin/face_scan /percorso/immagine.jpg
```
## Esempio
```json
{
"image": {
"path": "/percorso/immagine.jpg",
"width": 1500,
"height": 2000
},
"faces": [
{
"face_index": 0,
"score": 0.8671875,
"bbox": {
"x": 1079.3457,
"y": 489.648438,
"width": 131.591797,
"height": 182.275391
},
"landmarks": [
[1107.8125, 558.325195],
[1170.99609, 559.423828],
[1134.17969, 594.500732],
[1106.22559, 615.83252],
[1168.34717, 616.967773]
],
"embedding": [
-0.0426222458,
0.0678814054
]
}
]
}
```
L'array `embedding` reale contiene sempre 128 valori.
## Oggetto image
Campi disponibili:
- `path`: percorso ricevuto dal programma;
- `width`: larghezza originale dell'immagine;
- `height`: altezza originale dell'immagine.
## Oggetto face
Campi disponibili:
- `face_index`: indice sequenziale del volto;
- `score`: confidence SCRFD;
- `bbox.x`: coordinata sinistra;
- `bbox.y`: coordinata superiore;
- `bbox.width`: larghezza;
- `bbox.height`: altezza;
- `landmarks`: cinque coppie di coordinate;
- `embedding`: vettore L2-normalizzato di 128 elementi.
## Immagini senza volti
Un'immagine elaborata correttamente ma senza volti produce:
```json
{
"image": {
"path": "image.jpg",
"width": 1920,
"height": 1080
},
"faces": []
}
```
L'exit code resta `0`.
## Exit code
```text
0 scansione completata
2 argomenti non validi
3 inizializzazione FaceEngine fallita
4 caricamento o elaborazione immagine fallita
5 risultato facciale o embedding non valido
```
## Requisiti dell'embedding
Per ogni volto:
```text
Dimensione 128
Tipo logico float32
Norma L2 circa 1.0
Valori tutti finiti
```
Quando viene salvato in formato binario float32:
```text
128 x 4 byte = 512 byte
```

5
include/face_config.h Normal file
View file

@ -0,0 +1,5 @@
#pragma once
#ifndef FACE_CLUSTERING_ROOT
#define FACE_CLUSTERING_ROOT "."
#endif

46
include/face_engine.h Normal file
View file

@ -0,0 +1,46 @@
#pragma once
#include <string>
#include <vector>
struct Point {
float x = 0.0f;
float y = 0.0f;
};
struct Face {
float score = 0.0f;
float x1 = 0.0f;
float y1 = 0.0f;
float x2 = 0.0f;
float y2 = 0.0f;
Point kps[5]{};
// SFace: embedding L2-normalizzato da 128 float32.
std::vector<float> embedding;
};
class FaceEngine {
public:
FaceEngine();
~FaceEngine();
FaceEngine(const FaceEngine&) = delete;
FaceEngine& operator=(const FaceEngine&) = delete;
bool init(
const std::string& scrfd_model,
const std::string& sface_model);
bool process_image(
const std::string& image_path,
std::vector<Face>& faces,
int& image_width,
int& image_height);
private:
struct Impl;
Impl* impl_;
};

804
include/rknn_api.h Normal file
View file

@ -0,0 +1,804 @@
/****************************************************************************
*
* Copyright (c) 2017 - 2022 by Rockchip Corp. All rights reserved.
*
* The material in this file is confidential and contains trade secrets
* of Rockchip Corporation. This is proprietary information owned by
* Rockchip Corporation. No part of this work may be disclosed,
* reproduced, copied, transmitted, or used in any way for any purpose,
* without the express written permission of Rockchip Corporation.
*
*****************************************************************************/
#ifndef _RKNN_API_H
#define _RKNN_API_H
#ifdef __cplusplus
extern "C" {
#endif
#include <stdint.h>
/*
Definition of extended flag for rknn_init.
*/
/* set high priority context. */
#define RKNN_FLAG_PRIOR_HIGH 0x00000000
/* set medium priority context */
#define RKNN_FLAG_PRIOR_MEDIUM 0x00000001
/* set low priority context. */
#define RKNN_FLAG_PRIOR_LOW 0x00000002
/* asynchronous mode.
when enable, rknn_outputs_get will not block for too long because it directly retrieves the result of
the previous frame which can increase the frame rate on single-threaded mode, but at the cost of
rknn_outputs_get not retrieves the result of the current frame.
in multi-threaded mode you do not need to turn this mode on. */
#define RKNN_FLAG_ASYNC_MASK 0x00000004
/* collect performance mode.
when enable, you can get detailed performance reports via rknn_query(ctx, RKNN_QUERY_PERF_DETAIL, ...),
but it will reduce the frame rate. */
#define RKNN_FLAG_COLLECT_PERF_MASK 0x00000008
/* allocate all memory in outside, includes weight/internal/inputs/outputs */
#define RKNN_FLAG_MEM_ALLOC_OUTSIDE 0x00000010
/* weight sharing with the same network structure */
#define RKNN_FLAG_SHARE_WEIGHT_MEM 0x00000020
/* send fence fd from outside */
#define RKNN_FLAG_FENCE_IN_OUTSIDE 0x00000040
/* get fence fd from inside */
#define RKNN_FLAG_FENCE_OUT_OUTSIDE 0x00000080
/* dummy init flag: could only get total_weight_size and total_internal_size by rknn_query*/
#define RKNN_FLAG_COLLECT_MODEL_INFO_ONLY 0x00000100
/* allocate internal memory in outside */
#define RKNN_FLAG_INTERNAL_ALLOC_OUTSIDE 0x00000200
/* set GPU as the preferred execution backend When the operator is not supported by the NPU */
#define RKNN_FLAG_EXECUTE_FALLBACK_PRIOR_DEVICE_GPU 0x00000400
/* enable allocate sram type buffers */
#define RKNN_FLAG_ENABLE_SRAM 0x00000800
/* sram type buffers are shared among different contexts */
#define RKNN_FLAG_SHARE_SRAM 0x00001000
/* default nice -19, this flag can disable default priority */
#define RKNN_FLAG_DISABLE_PROC_HIGH_PRIORITY 0x00002000
/* don't flush input buffer cache, the user must ensure that the input tensor has flushed the cache before calling rknn_run.
!!! Don't use this flags when you call rknn_inputs_set() to set input data. */
#define RKNN_FLAG_DISABLE_FLUSH_INPUT_MEM_CACHE 0x00004000
/* Don't invalid output buffer cache.
Users cannot directly access output_mem->virt_addr,
which will cause cache consistency problems.
If you want to use output_mem->virt_addr,
you must use rknn_mem_sync (ctx, mem, RKNN_MEMORY_SYNC_FROM_DEVICE) to flush the cache.
This flags is generally used when the output data of the NPU is not accessed by the CPU,
but is accessed by the GPU or RGA to reduce the time required to flush the cache.
!!! Don't use this flags when you call rknn_outputs_get() to get output data.*/
#define RKNN_FLAG_DISABLE_FLUSH_OUTPUT_MEM_CACHE 0x00008000
/* This flag is used when the model data buffer is allocated by NPU, and can be accessed by NPU directly. */
#define RKNN_FLAG_MODEL_BUFFER_ZERO_COPY 0x00010000
/* This flag is a memory allocation flag, which is used in rknn_create_mem2() when no context is available. */
#define RKNN_MEM_FLAG_ALLOC_NO_CONTEXT 0x00020000
/*
Error code returned by the RKNN API.
*/
#define RKNN_SUCC 0 /* execute succeed. */
#define RKNN_ERR_FAIL -1 /* execute failed. */
#define RKNN_ERR_TIMEOUT -2 /* execute timeout. */
#define RKNN_ERR_DEVICE_UNAVAILABLE -3 /* device is unavailable. */
#define RKNN_ERR_MALLOC_FAIL -4 /* memory malloc fail. */
#define RKNN_ERR_PARAM_INVALID -5 /* parameter is invalid. */
#define RKNN_ERR_MODEL_INVALID -6 /* model is invalid. */
#define RKNN_ERR_CTX_INVALID -7 /* context is invalid. */
#define RKNN_ERR_INPUT_INVALID -8 /* input is invalid. */
#define RKNN_ERR_OUTPUT_INVALID -9 /* output is invalid. */
#define RKNN_ERR_DEVICE_UNMATCH -10 /* the device is unmatch, please update rknn sdk
and npu driver/firmware. */
#define RKNN_ERR_INCOMPATILE_PRE_COMPILE_MODEL -11 /* This RKNN model use pre_compile mode, but not compatible with current driver. */
#define RKNN_ERR_INCOMPATILE_OPTIMIZATION_LEVEL_VERSION -12 /* This RKNN model set optimization level, but not compatible with current driver. */
#define RKNN_ERR_TARGET_PLATFORM_UNMATCH -13 /* This RKNN model set target platform, but not compatible with current platform. */
/*
Definition for tensor
*/
#define RKNN_MAX_DIMS 16 /* maximum dimension of tensor. */
#define RKNN_MAX_NUM_CHANNEL 15 /* maximum channel number of input tensor. */
#define RKNN_MAX_NAME_LEN 256 /* maximum name lenth of tensor. */
#define RKNN_MAX_DYNAMIC_SHAPE_NUM 512 /* maximum number of dynamic shape for each input. */
#ifdef __arm__
typedef uint32_t rknn_context;
#else
typedef uint64_t rknn_context;
#endif
/*
The query command for rknn_query
*/
typedef enum _rknn_query_cmd {
RKNN_QUERY_IN_OUT_NUM = 0, /* query the number of input & output tensor. */
RKNN_QUERY_INPUT_ATTR = 1, /* query the attribute of input tensor. */
RKNN_QUERY_OUTPUT_ATTR = 2, /* query the attribute of output tensor. */
RKNN_QUERY_PERF_DETAIL = 3, /* query the detail performance, need set
RKNN_FLAG_COLLECT_PERF_MASK when call rknn_init,
this query needs to be valid after rknn_outputs_get. */
RKNN_QUERY_PERF_RUN = 4, /* query the time of run,
this query needs to be valid after rknn_outputs_get. */
RKNN_QUERY_SDK_VERSION = 5, /* query the sdk & driver version */
RKNN_QUERY_MEM_SIZE = 6, /* query the weight & internal memory size */
RKNN_QUERY_CUSTOM_STRING = 7, /* query the custom string */
RKNN_QUERY_NATIVE_INPUT_ATTR = 8, /* query the attribute of native input tensor. */
RKNN_QUERY_NATIVE_OUTPUT_ATTR = 9, /* query the attribute of native output tensor. */
RKNN_QUERY_NATIVE_NC1HWC2_INPUT_ATTR = 8, /* query the attribute of native input tensor. */
RKNN_QUERY_NATIVE_NC1HWC2_OUTPUT_ATTR = 9, /* query the attribute of native output tensor. */
RKNN_QUERY_NATIVE_NHWC_INPUT_ATTR = 10, /* query the attribute of native input tensor. */
RKNN_QUERY_NATIVE_NHWC_OUTPUT_ATTR = 11, /* query the attribute of native output tensor. */
RKNN_QUERY_DEVICE_MEM_INFO = 12, /* query the attribute of rknn memory information. */
RKNN_QUERY_INPUT_DYNAMIC_RANGE = 13, /* query the dynamic shape range of rknn input tensor. */
RKNN_QUERY_CURRENT_INPUT_ATTR = 14, /* query the current shape of rknn input tensor, only valid for dynamic rknn model*/
RKNN_QUERY_CURRENT_OUTPUT_ATTR = 15, /* query the current shape of rknn output tensor, only valid for dynamic rknn model*/
RKNN_QUERY_CURRENT_NATIVE_INPUT_ATTR = 16, /* query the current native shape of rknn input tensor, only valid for dynamic rknn model*/
RKNN_QUERY_CURRENT_NATIVE_OUTPUT_ATTR = 17, /* query the current native shape of rknn output tensor, only valid for dynamic rknn model*/
RKNN_QUERY_CMD_MAX
} rknn_query_cmd;
/*
the tensor data type.
*/
typedef enum _rknn_tensor_type {
RKNN_TENSOR_FLOAT32 = 0, /* data type is float32. */
RKNN_TENSOR_FLOAT16, /* data type is float16. */
RKNN_TENSOR_INT8, /* data type is int8. */
RKNN_TENSOR_UINT8, /* data type is uint8. */
RKNN_TENSOR_INT16, /* data type is int16. */
RKNN_TENSOR_UINT16, /* data type is uint16. */
RKNN_TENSOR_INT32, /* data type is int32. */
RKNN_TENSOR_UINT32, /* data type is uint32. */
RKNN_TENSOR_INT64, /* data type is int64. */
RKNN_TENSOR_BOOL,
RKNN_TENSOR_INT4,
RKNN_TENSOR_BFLOAT16,
RKNN_TENSOR_TYPE_MAX
} rknn_tensor_type;
inline static const char* get_type_string(rknn_tensor_type type)
{
switch(type) {
case RKNN_TENSOR_FLOAT32: return "FP32";
case RKNN_TENSOR_FLOAT16: return "FP16";
case RKNN_TENSOR_INT8: return "INT8";
case RKNN_TENSOR_UINT8: return "UINT8";
case RKNN_TENSOR_INT16: return "INT16";
case RKNN_TENSOR_UINT16: return "UINT16";
case RKNN_TENSOR_INT32: return "INT32";
case RKNN_TENSOR_UINT32: return "UINT32";
case RKNN_TENSOR_INT64: return "INT64";
case RKNN_TENSOR_BOOL: return "BOOL";
case RKNN_TENSOR_INT4: return "INT4";
case RKNN_TENSOR_BFLOAT16: return "BF16";
default: return "UNKNOW";
}
}
/*
the quantitative type.
*/
typedef enum _rknn_tensor_qnt_type {
RKNN_TENSOR_QNT_NONE = 0, /* none. */
RKNN_TENSOR_QNT_DFP, /* dynamic fixed point. */
RKNN_TENSOR_QNT_AFFINE_ASYMMETRIC, /* asymmetric affine. */
RKNN_TENSOR_QNT_MAX
} rknn_tensor_qnt_type;
inline static const char* get_qnt_type_string(rknn_tensor_qnt_type type)
{
switch(type) {
case RKNN_TENSOR_QNT_NONE: return "NONE";
case RKNN_TENSOR_QNT_DFP: return "DFP";
case RKNN_TENSOR_QNT_AFFINE_ASYMMETRIC: return "AFFINE";
default: return "UNKNOW";
}
}
/*
the tensor data format.
*/
typedef enum _rknn_tensor_format {
RKNN_TENSOR_NCHW = 0, /* data format is NCHW. */
RKNN_TENSOR_NHWC, /* data format is NHWC. */
RKNN_TENSOR_NC1HWC2, /* data format is NC1HWC2. */
RKNN_TENSOR_UNDEFINED,
RKNN_TENSOR_FORMAT_MAX
} rknn_tensor_format;
/*
the mode of running on target NPU core.
*/
typedef enum _rknn_core_mask {
RKNN_NPU_CORE_AUTO = 0, /* default, run on NPU core randomly. */
RKNN_NPU_CORE_0 = 1, /* run on NPU core 0. */
RKNN_NPU_CORE_1 = 2, /* run on NPU core 1. */
RKNN_NPU_CORE_2 = 4, /* run on NPU core 2. */
RKNN_NPU_CORE_0_1 = RKNN_NPU_CORE_0 | RKNN_NPU_CORE_1, /* run on NPU core 0 and core 1. */
RKNN_NPU_CORE_0_1_2 = RKNN_NPU_CORE_0_1 | RKNN_NPU_CORE_2, /* run on NPU core 0 and core 1 and core 2. */
RKNN_NPU_CORE_ALL = 0xffff, /* auto choice, run on NPU cores depending on platform */
RKNN_NPU_CORE_UNDEFINED,
} rknn_core_mask;
inline static const char* get_format_string(rknn_tensor_format fmt)
{
switch(fmt) {
case RKNN_TENSOR_NCHW: return "NCHW";
case RKNN_TENSOR_NHWC: return "NHWC";
case RKNN_TENSOR_NC1HWC2: return "NC1HWC2";
case RKNN_TENSOR_UNDEFINED: return "UNDEFINED";
default: return "UNKNOW";
}
}
/*
the information for RKNN_QUERY_IN_OUT_NUM.
*/
typedef struct _rknn_input_output_num {
uint32_t n_input; /* the number of input. */
uint32_t n_output; /* the number of output. */
} rknn_input_output_num;
/*
the information for RKNN_QUERY_INPUT_ATTR / RKNN_QUERY_OUTPUT_ATTR.
*/
typedef struct _rknn_tensor_attr {
uint32_t index; /* input parameter, the index of input/output tensor,
need set before call rknn_query. */
uint32_t n_dims; /* the number of dimensions. */
uint32_t dims[RKNN_MAX_DIMS]; /* the dimensions array. */
char name[RKNN_MAX_NAME_LEN]; /* the name of tensor. */
uint32_t n_elems; /* the number of elements. */
uint32_t size; /* the bytes size of tensor. */
rknn_tensor_format fmt; /* the data format of tensor. */
rknn_tensor_type type; /* the data type of tensor. */
rknn_tensor_qnt_type qnt_type; /* the quantitative type of tensor. */
int8_t fl; /* fractional length for RKNN_TENSOR_QNT_DFP. */
int32_t zp; /* zero point for RKNN_TENSOR_QNT_AFFINE_ASYMMETRIC. */
float scale; /* scale for RKNN_TENSOR_QNT_AFFINE_ASYMMETRIC. */
uint32_t w_stride; /* the stride of tensor along the width dimention of input,
Note: it is read-only, 0 means equal to width. */
uint32_t size_with_stride; /* the bytes size of tensor with stride. */
uint8_t pass_through; /* pass through mode, for rknn_set_io_mem interface.
if TRUE, the buf data is passed directly to the input node of the rknn model
without any conversion. the following variables do not need to be set.
if FALSE, the buf data is converted into an input consistent with the model
according to the following type and fmt. so the following variables
need to be set.*/
uint32_t h_stride; /* the stride along the height dimention of input,
Note: it is write-only, if it was set to 0, h_stride = height. */
} rknn_tensor_attr;
typedef struct _rknn_input_range {
uint32_t index; /* input parameter, the index of input/output tensor,
need set before call rknn_query. */
uint32_t shape_number; /* the number of shape. */
rknn_tensor_format fmt; /* the data format of tensor. */
char name[RKNN_MAX_NAME_LEN]; /* the name of tensor. */
uint32_t dyn_range[RKNN_MAX_DYNAMIC_SHAPE_NUM][RKNN_MAX_DIMS]; /* the dynamic input dimensions range. */
uint32_t n_dims; /* the number of dimensions. */
} rknn_input_range;
/*
the information for RKNN_QUERY_PERF_DETAIL.
*/
typedef struct _rknn_perf_detail {
char* perf_data; /* the string pointer of perf detail. don't need free it by user. */
uint64_t data_len; /* the string length. */
} rknn_perf_detail;
/*
the information for RKNN_QUERY_PERF_RUN.
*/
typedef struct _rknn_perf_run {
int64_t run_duration; /* real inference time (us) */
} rknn_perf_run;
/*
the information for RKNN_QUERY_SDK_VERSION.
*/
typedef struct _rknn_sdk_version {
char api_version[256]; /* the version of rknn api. */
char drv_version[256]; /* the version of rknn driver. */
} rknn_sdk_version;
/*
the information for RKNN_QUERY_MEM_SIZE.
*/
typedef struct _rknn_mem_size {
uint32_t total_weight_size; /* the weight memory size */
uint32_t total_internal_size; /* the internal memory size, exclude inputs/outputs */
uint64_t total_dma_allocated_size; /* total dma memory allocated size */
uint32_t total_sram_size; /* total system sram size reserved for rknn */
uint32_t free_sram_size; /* free system sram size reserved for rknn */
uint32_t reserved[10]; /* reserved */
} rknn_mem_size;
/*
the information for RKNN_QUERY_CUSTOM_STRING.
*/
typedef struct _rknn_custom_string {
char string[1024]; /* the string of custom, lengths max to 1024 bytes */
} rknn_custom_string;
/*
The flags of rknn_tensor_mem.
*/
typedef enum _rknn_tensor_mem_flags {
RKNN_TENSOR_MEMORY_FLAGS_ALLOC_INSIDE = 1, /*Used to mark in rknn_destroy_mem() whether it is necessary to release the "mem" pointer itself.
If the flag RKNN_TENSOR_MEMORY_FLAGS_ALLOC_INSIDE is set, rknn_destroy_mem() will call free(mem).*/
RKNN_TENSOR_MEMORY_FLAGS_FROM_FD = 2, /*Used to mark in rknn_create_mem_from_fd() whether it is necessary to release the "mem" pointer itself.
If the flag RKNN_TENSOR_MEMORY_FLAGS_FROM_FD is set, rknn_destroy_mem() will call free(mem).*/
RKNN_TENSOR_MEMORY_FLAGS_FROM_PHYS = 3, /*Used to mark in rknn_create_mem_from_phys() whether it is necessary to release the "mem" pointer itself.
If the flag RKNN_TENSOR_MEMORY_FLAGS_FROM_PHYS is set, rknn_destroy_mem() will call free(mem).*/
RKNN_TENSOR_MEMORY_FLAGS_UNKNOWN
} rknn_tensor_mem_flags;
/*
The mode to allocate rknn memory.
*/
typedef enum _rknn_mem_alloc_flags {
RKNN_FLAG_MEMORY_FLAGS_DEFAULT = 0 << 0, /* Same with RKNN_FLAG_MEMORY_CACHEABLE */
RKNN_FLAG_MEMORY_CACHEABLE = 1 << 0, /* Create Cacheable memory. */
RKNN_FLAG_MEMORY_NON_CACHEABLE = 1 << 1, /* Create NON-Cacheable memory. */
RKNN_FLAG_MEMORY_TRY_ALLOC_SRAM = 1 << 2, /* Try to allocate memory in SRAM if possible. if SRAM is not enough, allocate rest memory in DRAM. */
} rknn_mem_alloc_flags;
/*
The mode to sync cacheable rknn memory.
*/
typedef enum _rknn_mem_sync_mode {
RKNN_MEMORY_SYNC_TO_DEVICE = 0x1, /* the mode used for consistency of device access after CPU accesses data. */
RKNN_MEMORY_SYNC_FROM_DEVICE = 0x2, /* the mode used for consistency of CPU access after device accesses data. */
RKNN_MEMORY_SYNC_BIDIRECTIONAL = RKNN_MEMORY_SYNC_TO_DEVICE | RKNN_MEMORY_SYNC_FROM_DEVICE, /* the mode used for consistency of data access
between device and CPU in both directions. */
} rknn_mem_sync_mode;
/*
the memory information of tensor.
*/
typedef struct _rknn_tensor_memory {
void* virt_addr; /* the virtual address of tensor buffer. */
uint64_t phys_addr; /* the physical address of tensor buffer. */
int32_t fd; /* the fd of tensor buffer. */
int32_t offset; /* indicates the offset of the memory. */
uint32_t size; /* the size of tensor buffer. */
uint32_t flags; /* the flags of tensor buffer, reserved */
void * priv_data; /* the private data of tensor buffer. */
} rknn_tensor_mem;
/*
the input information for rknn_input_set.
*/
typedef struct _rknn_input {
uint32_t index; /* the input index. */
void* buf; /* the input buf for index. */
uint32_t size; /* the size of input buf. */
uint8_t pass_through; /* pass through mode.
if TRUE, the buf data is passed directly to the input node of the rknn model
without any conversion. the following variables do not need to be set.
if FALSE, the buf data is converted into an input consistent with the model
according to the following type and fmt. so the following variables
need to be set.*/
rknn_tensor_type type; /* the data type of input buf. */
rknn_tensor_format fmt; /* the data format of input buf.
currently the internal input format of NPU is NCHW by default.
so entering NCHW data can avoid the format conversion in the driver. */
} rknn_input;
/*
the output information for rknn_outputs_get.
*/
typedef struct _rknn_output {
uint8_t want_float; /* want transfer output data to float */
uint8_t is_prealloc; /* whether buf is pre-allocated.
if TRUE, the following variables need to be set.
if FALSE, the following variables do not need to be set. */
uint32_t index; /* the output index. */
void* buf; /* the output buf for index.
when is_prealloc = FALSE and rknn_outputs_release called,
this buf pointer will be free and don't use it anymore. */
uint32_t size; /* the size of output buf. */
} rknn_output;
/*
the extend information for rknn_init.
*/
typedef struct _rknn_init_extend {
rknn_context ctx; /* rknn context */
int32_t real_model_offset; /* real rknn model file offset, only valid when init context with rknn file path and zero-copy model model */
uint32_t real_model_size; /* real rknn model file size, only valid when init context with rknn file path and zero-copy model model */
int32_t model_buffer_fd; /* the fd of model buffer. */
uint32_t model_buffer_flags; /* the flags of model_buffer */
uint8_t reserved[112]; /* reserved */
} rknn_init_extend;
/*
the extend information for rknn_run.
*/
typedef struct _rknn_run_extend {
uint64_t frame_id; /* output parameter, indicate current frame id of run. */
int32_t non_block; /* block flag of run, 0 is block else 1 is non block */
int32_t timeout_ms; /* timeout for block mode, in milliseconds */
int32_t fence_fd; /* fence fd from other unit */
} rknn_run_extend;
/*
the extend information for rknn_outputs_get.
*/
typedef struct _rknn_output_extend {
uint64_t frame_id; /* output parameter, indicate the frame id of outputs, corresponds to
struct rknn_run_extend.frame_id.*/
} rknn_output_extend;
/* rknn_init
initial the context and load the rknn model.
input:
rknn_context* context the pointer of context handle.
void* model if size > 0, pointer to the rknn model, if size = 0, filepath to the rknn model.
uint32_t size the size of rknn model.
uint32_t flag extend flag, see the define of RKNN_FLAG_XXX_XXX.
rknn_init_extend* extend the extend information of init.
return:
int error code.
*/
int rknn_init(rknn_context* context, void* model, uint32_t size, uint32_t flag, rknn_init_extend* extend);
/* rknn_dup_context
initial the context and load the rknn model.
input:
rknn_context* context_in the pointer of context in handle.
rknn_context* context_out the pointer of context out handle.
return:
int error code.
*/
int rknn_dup_context(rknn_context* context_in, rknn_context* context_out);
/* rknn_destroy
unload the rknn model and destroy the context.
input:
rknn_context context the handle of context.
return:
int error code.
*/
int rknn_destroy(rknn_context context);
/* rknn_query
query the information about model or others. see rknn_query_cmd.
input:
rknn_context context the handle of context.
rknn_query_cmd cmd the command of query.
void* info the buffer point of information.
uint32_t size the size of information.
return:
int error code.
*/
int rknn_query(rknn_context context, rknn_query_cmd cmd, void* info, uint32_t size);
/* rknn_inputs_set
set inputs information by input index of rknn model.
inputs information see rknn_input.
input:
rknn_context context the handle of context.
uint32_t n_inputs the number of inputs.
rknn_input inputs[] the arrays of inputs information, see rknn_input.
return:
int error code
*/
int rknn_inputs_set(rknn_context context, uint32_t n_inputs, rknn_input inputs[]);
/*
rknn_set_batch_core_num
set rknn batch core_num.
input:
rknn_context context the handle of context.
int core_num the core number.
return:
int error code.
*/
int rknn_set_batch_core_num(rknn_context context, int core_num);
/* rknn_set_core_mask
set the core mask for the model.(only supported on multi-core NPU platform)
RKNN_NPU_CORE_AUTO: auto mode, default value
RKNN_NPU_CORE_0: core 0 mode
RKNN_NPU_CORE_1: core 1 mode
RKNN_NPU_CORE_2: core 2 mode
RKNN_NPU_CORE_0_1: combine core 0/1 mode
RKNN_NPU_CORE_0_1_2: combine core 0/1/2 mode
RKNN_NPU_CORE_ALL: auto mode, select multiple npu cores to run depending on platform
input:
rknn_context context the handle of context.
rknn_core_mask core_mask the core mask.
return:
int error code.
*/
int rknn_set_core_mask(rknn_context context, rknn_core_mask core_mask);
/* rknn_run
run the model to execute inference.
input:
rknn_context context the handle of context.
rknn_run_extend* extend the extend information of run.
return:
int error code.
*/
int rknn_run(rknn_context context, rknn_run_extend* extend);
/* rknn_wait
wait the model after execute inference.
input:
rknn_context context the handle of context.
rknn_run_extend* extend the extend information of run.
return:
int error code.
*/
int rknn_wait(rknn_context context, rknn_run_extend* extend);
/* rknn_outputs_get
wait the inference to finish and get the outputs.
this function will block until inference finish.
the results will set to outputs[].
input:
rknn_context context the handle of context.
uint32_t n_outputs the number of outputs.
rknn_output outputs[] the arrays of output, see rknn_output.
rknn_output_extend* the extend information of output.
return:
int error code.
*/
int rknn_outputs_get(rknn_context context, uint32_t n_outputs, rknn_output outputs[], rknn_output_extend* extend);
/* rknn_outputs_release
release the outputs that get by rknn_outputs_get.
after called, the rknn_output[x].buf get from rknn_outputs_get will
also be free when rknn_output[x].is_prealloc = FALSE.
input:
rknn_context context the handle of context.
uint32_t n_ouputs the number of outputs.
rknn_output outputs[] the arrays of output.
return:
int error code
*/
int rknn_outputs_release(rknn_context context, uint32_t n_ouputs, rknn_output outputs[]);
/* new api for zero copy */
/* rknn_create_mem_from_phys (memory allocated outside)
initialize tensor memory from physical address.
input:
rknn_context ctx the handle of context.
uint64_t phys_addr physical address.
void *virt_addr virtual address.
uint32_t size the size of tensor buffer.
return:
rknn_tensor_mem the pointer of tensor memory information.
*/
rknn_tensor_mem* rknn_create_mem_from_phys(rknn_context ctx, uint64_t phys_addr, void *virt_addr, uint32_t size);
/* rknn_create_mem_from_fd (memory allocated outside)
initialize tensor memory from file description.
input:
rknn_context ctx the handle of context.
int32_t fd file description.
void *virt_addr virtual address.
uint32_t size the size of tensor buffer.
int32_t offset indicates the offset of the memory (virt_addr without offset).
return:
rknn_tensor_mem the pointer of tensor memory information.
*/
rknn_tensor_mem* rknn_create_mem_from_fd(rknn_context ctx, int32_t fd, void *virt_addr, uint32_t size, int32_t offset);
/* rknn_create_mem_from_mb_blk (memory allocated outside)
create tensor memory from mb_blk.
input:
rknn_context ctx the handle of context.
void *mb_blk mb_blk allocate from system api.
int32_t offset indicates the offset of the memory.
return:
rknn_tensor_mem the pointer of tensor memory information.
*/
rknn_tensor_mem* rknn_create_mem_from_mb_blk(rknn_context ctx, void *mb_blk, int32_t offset);
/* rknn_create_mem (memory allocated inside)
create tensor memory.
input:
rknn_context ctx the handle of context.
uint32_t size the size of tensor buffer.
return:
rknn_tensor_mem the pointer of tensor memory information.
*/
rknn_tensor_mem* rknn_create_mem(rknn_context ctx, uint32_t size);
/* rknn_create_mem2 (memory allocated inside)
create tensor memory.
input:
rknn_context ctx the handle of context.
uint64_t size the size of tensor buffer.
uint64_t alloc_flags memory allocation flags.
return:
rknn_tensor_mem the pointer of tensor memory information.
*/
rknn_tensor_mem* rknn_create_mem2(rknn_context ctx, uint64_t size, uint64_t alloc_flags);
/* rknn_destroy_mem (support allocate inside and outside)
destroy tensor memory.
input:
rknn_context ctx the handle of context.
rknn_tensor_mem *mem the pointer of tensor memory information.
return:
int error code
*/
int rknn_destroy_mem(rknn_context ctx, rknn_tensor_mem *mem);
/* rknn_set_weight_mem
set the weight memory.
input:
rknn_context ctx the handle of context.
rknn_tensor_mem *mem the array of tensor memory information
return:
int error code.
*/
int rknn_set_weight_mem(rknn_context ctx, rknn_tensor_mem *mem);
/* rknn_set_internal_mem
set the internal memory.
input:
rknn_context ctx the handle of context.
rknn_tensor_mem *mem the array of tensor memory information
return:
int error code.
*/
int rknn_set_internal_mem(rknn_context ctx, rknn_tensor_mem *mem);
/* rknn_set_io_mem
set the input and output tensors buffer.
input:
rknn_context ctx the handle of context.
rknn_tensor_mem *mem the array of tensor memory information.
rknn_tensor_attr *attr the attribute of input or output tensor buffer.
return:
int error code.
*/
int rknn_set_io_mem(rknn_context ctx, rknn_tensor_mem *mem, rknn_tensor_attr *attr);
/* rknn_set_input_shape(deprecated)
set the input tensor shape (only valid for dynamic shape rknn model).
input:
rknn_context ctx the handle of context.
rknn_tensor_attr *attr the attribute of input or output tensor buffer.
return:
int error code.
*/
int rknn_set_input_shape(rknn_context ctx, rknn_tensor_attr* attr);
/* rknn_set_input_shapes
set all the input tensor shapes. graph will run under current set of input shapes after rknn_set_input_shapes.(only valid for dynamic shape rknn model).
input:
rknn_context ctx the handle of context.
uint32_t n_inputs the number of inputs.
rknn_tensor_attr attr[] the attribute array of all input tensors.
return:
int error code.
*/
int rknn_set_input_shapes(rknn_context ctx, uint32_t n_inputs, rknn_tensor_attr attr[]);
/* rknn_mem_sync
sync cacheable rknn memory when both cpu and device access data.
input:
rknn_context context the handle of context.
rknn_tensor_mem *mem the pointer of tensor memory information.
rknn_mem_sync_mode mode the mode of sync cache.
return:
int error code.
*/
int rknn_mem_sync(rknn_context context, rknn_tensor_mem* mem, rknn_mem_sync_mode mode);
#ifdef __cplusplus
} //extern "C"
#endif
#endif //_RKNN_API_H

7762
include/stb_image.h Normal file
View file

File diff suppressed because it is too large Load diff

17
lib/README.md Normal file
View file

@ -0,0 +1,17 @@
# Runtime RKNN
Copiare in questa directory la libreria runtime:
```text
lib/librknnrt.so
```
La libreria deve essere compatibile con:
- il dispositivo Rockchip utilizzato;
- la versione del driver RKNN;
- i modelli RKNN.
Il runtime è escluso dal repository tramite `.gitignore`.
Prima della distribuzione verificare le condizioni di licenza.

25
models/README.md Normal file
View file

@ -0,0 +1,25 @@
# Modelli RKNN
Copiare in questa directory i modelli richiesti:
```text
models/SCRFD_500M_KPS_640.rknn
models/face_recognition_sface_2021dec.rknn
```
## SCRFD
Il modello SCRFD rileva i volti e produce:
- confidence score;
- bounding box;
- cinque landmark.
## SFace
Il modello SFace riceve un volto allineato a 112 x 112 pixel e produce un
embedding di 128 elementi.
I modelli sono esclusi dal repository tramite `.gitignore`.
Prima della distribuzione verificare le rispettive licenze.

48
modules/hdbscan/README.md Normal file
View file

@ -0,0 +1,48 @@
# HDBSCAN
Modulo futuro per completare la pipeline di clustering degli embedding
facciali.
## Pipeline prevista
```text
kNN graph
|
v
core distance
|
v
mutual reachability
|
v
minimum spanning tree
|
v
single linkage hierarchy
|
v
condensed tree
|
v
cluster stability
|
v
cluster assignment
```
## Output previsto
Per ogni embedding:
```text
face_index
cluster_id
membership_strength
noise_flag
```
Il risultato del clustering resterà distinto dall'identità semantica
assegnata dall'utente.
Un cluster automatico non rappresenta necessariamente una persona
definitiva e più cluster potranno essere associati alla stessa persona.

43
modules/knn-opencl/README.md Normal file
View file

@ -0,0 +1,43 @@
# kNN OpenCL
Modulo futuro per costruire il grafo dei vicini degli embedding SFace
tramite OpenCL.
## Input previsto
```text
embedding float32[N][128]
```
Gli embedding saranno già L2-normalizzati.
Di conseguenza:
```text
cosine_similarity(a,b) = dot(a,b)
```
## Output previsto
```text
indices[N][32]
scores[N][32]
```
## Obiettivo iniziale
```text
embedding float32[N][128]
|
v
brute-force dot product
|
v
TOPK=32
|
+-- neighbor index
`-- similarity score
```
La prima implementazione privilegerà la correttezza e la validazione
CPU/GPU.

29
modules/mst/README.md Normal file
View file

@ -0,0 +1,29 @@
# Minimum Spanning Tree
Modulo futuro per costruire il minimum spanning tree del grafo di mutual
reachability.
## Input previsto
Un grafo pesato:
```text
source
destination
mutual_reachability_weight
```
## Output previsto
Per un dataset con `N` vertici connessi, l'MST conterrà:
```text
N - 1 archi
```
La prima implementazione privilegerà:
- correttezza;
- determinismo;
- confronto con una versione CPU di riferimento;
- verificabilità dei risultati.

37
modules/mutual-reachability/README.md Normal file
View file

@ -0,0 +1,37 @@
# Mutual Reachability
Modulo futuro per convertire il grafo kNN in un grafo pesato utilizzabile
da HDBSCAN.
## Core distance
Con `min_samples=32`, la core distance di un punto corrisponde alla
distanza dal suo trentaduesimo vicino.
## Mutual reachability distance
```text
mrd(i,j) =
max(
core_distance(i),
core_distance(j),
distance(i,j)
)
```
## Input previsto
```text
indices[N][32]
scores[N][32]
```
## Output previsto
Un insieme di archi candidati:
```text
source
destination
mutual_reachability_weight
```

737
src/face_engine.cc Normal file
View file

@ -0,0 +1,737 @@
#include <algorithm>
#include <cmath>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <iostream>
#include <string>
#include <vector>
#define STB_IMAGE_IMPLEMENTATION
#include "stb_image.h"
#include "rknn_api.h"
#include "face_engine.h"
static constexpr int SCRFD_SIZE = 640;
static constexpr int SFACE_SIZE = 112;
static constexpr int EMBED_DIM = 128;
static uint16_t float_to_fp16(float value)
{
uint32_t bits;
std::memcpy(&bits, &value, sizeof(bits));
uint32_t sign = (bits >> 31) & 1;
int exp = ((bits >> 23) & 0xff) - 127;
uint32_t mant = bits & 0x7fffff;
if (exp == 128) {
return (sign << 15) | 0x7c00;
}
if (exp > 15) {
return (sign << 15) | 0x7c00;
}
if (exp < -14) {
if (exp < -24)
return sign << 15;
mant |= 0x800000;
int shift = -exp - 14;
uint16_t m = mant >> (shift + 13);
return (sign << 15) | m;
}
uint16_t h_exp = (uint16_t)(exp + 15);
uint16_t h_mant = (uint16_t)(mant >> 13);
return (sign << 15) | (h_exp << 10) | h_mant;
}
static float fp16_to_float(uint16_t h)
{
uint32_t sign = (h >> 15) & 1;
uint32_t exp = (h >> 10) & 0x1f;
uint32_t mant = h & 0x3ff;
uint32_t bits;
if (exp == 0) {
if (mant == 0) {
bits = sign << 31;
} else {
float v = mant / 1024.0f;
v = std::ldexp(v, -14);
return sign ? -v : v;
}
} else if (exp == 31) {
bits = (sign << 31) | 0x7f800000 | (mant << 13);
} else {
uint32_t fexp = exp - 15 + 127;
bits = (sign << 31) | (fexp << 23) | (mant << 13);
}
float v;
std::memcpy(&v, &bits, sizeof(v));
return v;
}
static float iou(const Face& a, const Face& b)
{
float xx1 = std::max(a.x1, b.x1);
float yy1 = std::max(a.y1, b.y1);
float xx2 = std::min(a.x2, b.x2);
float yy2 = std::min(a.y2, b.y2);
float w = std::max(0.0f, xx2 - xx1);
float h = std::max(0.0f, yy2 - yy1);
float inter = w * h;
float area_a = std::max(0.0f, a.x2-a.x1) *
std::max(0.0f, a.y2-a.y1);
float area_b = std::max(0.0f, b.x2-b.x1) *
std::max(0.0f, b.y2-b.y1);
return inter / (area_a + area_b - inter + 1e-6f);
}
static void nms(std::vector<Face>& faces, float threshold)
{
std::sort(faces.begin(), faces.end(),
[](const Face& a, const Face& b) {
return a.score > b.score;
});
std::vector<Face> result;
for (const auto& f : faces) {
bool keep = true;
for (const auto& r : result) {
if (iou(f, r) > threshold) {
keep = false;
break;
}
}
if (keep)
result.push_back(f);
}
faces.swap(result);
}
struct Image {
int w = 0;
int h = 0;
std::vector<uint8_t> rgb;
};
static bool load_image(const std::string& path, Image& img)
{
int c = 0;
unsigned char* p =
stbi_load(path.c_str(), &img.w, &img.h, &c, 3);
if (!p) {
std::cerr << "Errore caricamento: " << path << "\n";
return false;
}
img.rgb.assign(p, p + img.w * img.h * 3);
stbi_image_free(p);
return true;
}
static void resize_bilinear_rgb(
const Image& src,
std::vector<uint8_t>& dst,
int dw,
int dh)
{
dst.resize(dw * dh * 3);
float sx = (float)src.w / dw;
float sy = (float)src.h / dh;
for (int y = 0; y < dh; ++y) {
float fy = (y + 0.5f) * sy - 0.5f;
int y0 = (int)std::floor(fy);
float wy = fy - y0;
if (y0 < 0) {
y0 = 0;
wy = 0;
}
int y1 = std::min(y0 + 1, src.h - 1);
for (int x = 0; x < dw; ++x) {
float fx = (x + 0.5f) * sx - 0.5f;
int x0 = (int)std::floor(fx);
float wx = fx - x0;
if (x0 < 0) {
x0 = 0;
wx = 0;
}
int x1 = std::min(x0 + 1, src.w - 1);
for (int c = 0; c < 3; ++c) {
float p00 = src.rgb[(y0*src.w+x0)*3+c];
float p01 = src.rgb[(y0*src.w+x1)*3+c];
float p10 = src.rgb[(y1*src.w+x0)*3+c];
float p11 = src.rgb[(y1*src.w+x1)*3+c];
float v =
p00 * (1-wx) * (1-wy) +
p01 * wx * (1-wy) +
p10 * (1-wx) * wy +
p11 * wx * wy;
dst[(y*dw+x)*3+c] =
(uint8_t)std::clamp((int)std::round(v), 0, 255);
}
}
}
}
class Scrfd {
public:
rknn_context ctx = 0;
bool init(const char* model)
{
FILE* fp = fopen(model, "rb");
if (!fp) {
perror(model);
return false;
}
fseek(fp, 0, SEEK_END);
size_t size = ftell(fp);
fseek(fp, 0, SEEK_SET);
std::vector<uint8_t> data(size);
fread(data.data(), 1, size, fp);
fclose(fp);
int ret = rknn_init(&ctx, data.data(), size, 0, nullptr);
if (ret != 0) {
std::cerr << "rknn_init SCRFD failed: " << ret << "\n";
return false;
}
return true;
}
std::vector<Face> detect(const Image& image)
{
float scale =
std::min(640.0f / image.w, 640.0f / image.h);
int rw = (int)std::round(image.w * scale);
int rh = (int)std::round(image.h * scale);
std::vector<uint8_t> resized;
resize_bilinear_rgb(image, resized, rw, rh);
std::vector<uint16_t> input(
SCRFD_SIZE * SCRFD_SIZE * 3);
for (int y = 0; y < rh; ++y) {
for (int x = 0; x < rw; ++x) {
int dst = (y * SCRFD_SIZE + x) * 3;
int src = (y * rw + x) * 3;
for (int c = 0; c < 3; ++c) {
float v =
((float)resized[src+c] - 127.5f) / 128.0f;
input[dst+c] = float_to_fp16(v);
}
}
}
rknn_input in{};
in.index = 0;
in.type = RKNN_TENSOR_FLOAT16;
in.fmt = RKNN_TENSOR_NHWC;
in.size = input.size() * sizeof(uint16_t);
in.buf = input.data();
in.pass_through = 1;
int ret = rknn_inputs_set(ctx, 1, &in);
if (ret != 0) {
std::cerr << "SCRFD inputs_set failed\n";
return {};
}
ret = rknn_run(ctx, nullptr);
if (ret != 0) {
std::cerr << "SCRFD run failed\n";
return {};
}
rknn_output outputs[9]{};
for (int i = 0; i < 9; ++i) {
outputs[i].want_float = 0;
}
ret = rknn_outputs_get(ctx, 9, outputs, nullptr);
if (ret != 0) {
std::cerr << "SCRFD outputs_get failed\n";
return {};
}
const int strides[3] = {8, 16, 32};
const int counts[3] = {12800, 3200, 800};
std::vector<Face> faces;
const float score_threshold = 0.50f;
for (int level = 0; level < 3; ++level) {
const int stride = strides[level];
const int count = counts[level];
const uint16_t* scores =
(const uint16_t*)outputs[level].buf;
const uint16_t* bbox =
(const uint16_t*)outputs[3 + level].buf;
const uint16_t* kps =
(const uint16_t*)outputs[6 + level].buf;
int feat_w = SCRFD_SIZE / stride;
int feat_h = SCRFD_SIZE / stride;
for (int i = 0; i < count; ++i) {
float score = fp16_to_float(scores[i]);
if (score < score_threshold)
continue;
int anchor = i % 2;
int p = i / 2;
int gx = p % feat_w;
int gy = p / feat_w;
float cx = gx * stride;
float cy = gy * stride;
/*
* AnchorGenerator con due anchor:
* centers duplicati sullo stesso punto.
*/
(void)anchor;
float l = fp16_to_float(bbox[i*4+0]) * stride;
float t = fp16_to_float(bbox[i*4+1]) * stride;
float r = fp16_to_float(bbox[i*4+2]) * stride;
float b = fp16_to_float(bbox[i*4+3]) * stride;
Face f;
f.score = score;
f.x1 = (cx - l) / scale;
f.y1 = (cy - t) / scale;
f.x2 = (cx + r) / scale;
f.y2 = (cy + b) / scale;
for (int j = 0; j < 5; ++j) {
float x =
cx + fp16_to_float(kps[i*10+j*2])
* stride;
float y =
cy + fp16_to_float(kps[i*10+j*2+1])
* stride;
f.kps[j].x = x / scale;
f.kps[j].y = y / scale;
}
f.x1 = std::clamp(f.x1, 0.0f, (float)image.w);
f.y1 = std::clamp(f.y1, 0.0f, (float)image.h);
f.x2 = std::clamp(f.x2, 0.0f, (float)image.w);
f.y2 = std::clamp(f.y2, 0.0f, (float)image.h);
faces.push_back(f);
}
}
rknn_outputs_release(ctx, 9, outputs);
nms(faces, 0.45f);
return faces;
}
~Scrfd()
{
if (ctx)
rknn_destroy(ctx);
}
};
static const Point sface_template[5] = {
{38.2946f, 51.6963f},
{73.5318f, 51.5014f},
{56.0252f, 71.7366f},
{41.5493f, 92.3655f},
{70.7299f, 92.2041f}
};
/*
* Trova la trasformazione similarity:
*
* x' = a*x - b*y + tx
* y' = b*x + a*y + ty
*/
static void similarity_transform(
const Point src[5],
float& a,
float& b,
float& tx,
float& ty)
{
double sx = 0, sy = 0;
double dx = 0, dy = 0;
for (int i = 0; i < 5; ++i) {
sx += src[i].x;
sy += src[i].y;
dx += sface_template[i].x;
dy += sface_template[i].y;
}
sx /= 5;
sy /= 5;
dx /= 5;
dy /= 5;
double num_a = 0;
double num_b = 0;
double den = 0;
for (int i = 0; i < 5; ++i) {
double x = src[i].x - sx;
double y = src[i].y - sy;
double X = sface_template[i].x - dx;
double Y = sface_template[i].y - dy;
num_a += x*X + y*Y;
num_b += x*Y - y*X;
den += x*x + y*y;
}
a = (float)(num_a / den);
b = (float)(num_b / den);
tx = (float)(dx - a*sx + b*sy);
ty = (float)(dy - b*sx - a*sy);
}
static std::vector<uint16_t> align_face(
const Image& image,
const Face& face)
{
float a, b, tx, ty;
similarity_transform(face.kps, a, b, tx, ty);
/*
* Inversa della similarity transform.
*
* src = A^-1 * (dst - t)
*/
float denom = a*a + b*b;
std::vector<uint16_t> output(
SFACE_SIZE * SFACE_SIZE * 3);
for (int y = 0; y < SFACE_SIZE; ++y) {
for (int x = 0; x < SFACE_SIZE; ++x) {
float X = x - tx;
float Y = y - ty;
float sx =
(a*X + b*Y) / denom;
float sy =
(-b*X + a*Y) / denom;
int x0 = (int)std::floor(sx);
int y0 = (int)std::floor(sy);
float wx = sx - x0;
float wy = sy - y0;
uint8_t rgb[3] = {0,0,0};
if (x0 >= 0 &&
y0 >= 0 &&
x0 + 1 < image.w &&
y0 + 1 < image.h) {
int x1 = x0 + 1;
int y1 = y0 + 1;
for (int c = 0; c < 3; ++c) {
float p00 =
image.rgb[(y0*image.w+x0)*3+c];
float p01 =
image.rgb[(y0*image.w+x1)*3+c];
float p10 =
image.rgb[(y1*image.w+x0)*3+c];
float p11 =
image.rgb[(y1*image.w+x1)*3+c];
float v =
p00*(1-wx)*(1-wy) +
p01*wx*(1-wy) +
p10*(1-wx)*wy +
p11*wx*wy;
rgb[c] =
(uint8_t)std::clamp(
(int)std::round(v), 0, 255);
}
}
int idx = (y*SFACE_SIZE+x)*3;
/*
* SFace OpenCV:
* blobFromImage(..., swapRB=true, scalefactor=1)
*
* Noi partiamo già da RGB, quindi non facciamo
* nessun ulteriore swap e nessuna normalizzazione.
*/
output[idx+0] = float_to_fp16(rgb[0]);
output[idx+1] = float_to_fp16(rgb[1]);
output[idx+2] = float_to_fp16(rgb[2]);
}
}
return output;
}
class SFace {
public:
rknn_context ctx = 0;
bool init(const char* model)
{
FILE* fp = fopen(model, "rb");
if (!fp) {
perror(model);
return false;
}
fseek(fp, 0, SEEK_END);
size_t size = ftell(fp);
fseek(fp, 0, SEEK_SET);
std::vector<uint8_t> data(size);
fread(data.data(), 1, size, fp);
fclose(fp);
int ret = rknn_init(&ctx, data.data(), size, 0, nullptr);
if (ret != 0) {
std::cerr << "rknn_init SFace failed: "
<< ret << "\n";
return false;
}
return true;
}
std::vector<float> feature(
const std::vector<uint16_t>& input)
{
rknn_input in{};
in.index = 0;
in.type = RKNN_TENSOR_FLOAT16;
in.fmt = RKNN_TENSOR_NHWC;
in.size = input.size() * sizeof(uint16_t);
in.buf = (void*)input.data();
in.pass_through = 1;
int ret = rknn_inputs_set(ctx, 1, &in);
if (ret != 0) {
std::cerr << "SFace inputs_set failed\n";
return {};
}
ret = rknn_run(ctx, nullptr);
if (ret != 0) {
std::cerr << "SFace run failed\n";
return {};
}
rknn_output output{};
output.want_float = 0;
ret = rknn_outputs_get(
ctx, 1, &output, nullptr);
if (ret != 0) {
std::cerr << "SFace outputs_get failed\n";
return {};
}
const uint16_t* p =
(const uint16_t*)output.buf;
std::vector<float> emb(EMBED_DIM);
for (int i = 0; i < EMBED_DIM; ++i)
emb[i] = fp16_to_float(p[i]);
rknn_outputs_release(ctx, 1, &output);
/*
* L2 normalization.
*/
double norm = 0;
for (float v : emb)
norm += (double)v * v;
norm = std::sqrt(norm);
if (norm > 0) {
for (float& v : emb)
v /= (float)norm;
}
return emb;
}
~SFace()
{
if (ctx)
rknn_destroy(ctx);
}
};
static std::vector<Face> process_image(
const Image& image,
Scrfd& detector,
SFace& recognizer)
{
auto faces = detector.detect(image);
for (auto& face : faces) {
auto aligned = align_face(image, face);
face.embedding =
recognizer.feature(aligned);
}
return faces;
}
struct FaceEngine::Impl {
Scrfd detector;
SFace recognizer;
bool initialized = false;
};
FaceEngine::FaceEngine()
: impl_(new Impl())
{
}
FaceEngine::~FaceEngine()
{
delete impl_;
}
bool FaceEngine::init(
const std::string& scrfd_model_path,
const std::string& sface_model_path)
{
if (impl_->initialized)
return true;
if (!impl_->detector.init(scrfd_model_path.c_str()))
return false;
if (!impl_->recognizer.init(sface_model_path.c_str()))
return false;
impl_->initialized = true;
return true;
}
bool FaceEngine::process_image(
const std::string& image_path,
std::vector<Face>& faces,
int& image_width,
int& image_height)
{
faces.clear();
image_width = 0;
image_height = 0;
if (!impl_->initialized) {
std::cerr << "FaceEngine non inizializzato\n";
return false;
}
Image image;
if (!load_image(image_path, image))
return false;
image_width = image.w;
image_height = image.h;
faces = ::process_image(
image,
impl_->detector,
impl_->recognizer);
for (const Face& face : faces) {
if (face.embedding.size() != EMBED_DIM) {
std::cerr
<< "Embedding non valido per "
<< image_path
<< ": attesi "
<< EMBED_DIM
<< " valori, trovati "
<< face.embedding.size()
<< "\n";
faces.clear();
return false;
}
}
return true;
}

144
src/face_recognition.cc Normal file
View file

@ -0,0 +1,144 @@
#include <cstddef>
#include <iostream>
#include <string>
#include <vector>
#include "face_engine.h"
#include "face_config.h"
static constexpr float MATCH_THRESHOLD = 0.363f;
static float cosine(
const std::vector<float>& a,
const std::vector<float>& b)
{
if (a.size() != b.size())
return 0.0f;
float score = 0.0f;
for (std::size_t i = 0; i < a.size(); ++i)
score += a[i] * b[i];
return score;
}
static void print_faces(
const std::string& image_path,
const std::vector<Face>& faces)
{
std::cout << "\n" << image_path << "\n";
std::cout << "Faces: " << faces.size() << "\n";
for (std::size_t i = 0; i < faces.size(); ++i) {
const Face& face = faces[i];
std::cout
<< "\nFace " << i
<< " score=" << face.score
<< "\n bbox="
<< face.x1 << " "
<< face.y1 << " "
<< face.x2 << " "
<< face.y2
<< "\n";
std::cout << " landmarks:\n";
for (int j = 0; j < 5; ++j) {
std::cout
<< " "
<< j
<< ": "
<< face.kps[j].x
<< " "
<< face.kps[j].y
<< "\n";
}
}
}
int main(int argc, char** argv)
{
if (argc != 2 && argc != 3) {
std::cerr
<< "Uso:\n"
<< " " << argv[0]
<< " image.jpg\n"
<< " " << argv[0]
<< " reference.jpg query.jpg\n";
return 1;
}
const std::string root =
FACE_CLUSTERING_ROOT;
const std::string scrfd_model =
root + "/models/SCRFD_500M_KPS_640.rknn";
const std::string sface_model =
root + "/models/face_recognition_sface_2021dec.rknn";
FaceEngine engine;
if (!engine.init(scrfd_model, sface_model))
return 1;
std::vector<Face> faces1;
int image1_width = 0;
int image1_height = 0;
if (!engine.process_image(
argv[1],
faces1,
image1_width,
image1_height)) {
return 1;
}
print_faces(argv[1], faces1);
if (argc == 2)
return 0;
std::vector<Face> faces2;
int image2_width = 0;
int image2_height = 0;
if (!engine.process_image(
argv[2],
faces2,
image2_width,
image2_height)) {
return 1;
}
print_faces(argv[2], faces2);
std::cout
<< "\n========================================\n"
<< "COSINE SIMILARITY\n"
<< "========================================\n";
for (std::size_t i = 0; i < faces1.size(); ++i) {
for (std::size_t j = 0; j < faces2.size(); ++j) {
const float similarity = cosine(
faces1[i].embedding,
faces2[j].embedding);
std::cout
<< "ref[" << i
<< "] vs query[" << j
<< "] = "
<< similarity;
if (similarity >= MATCH_THRESHOLD)
std::cout << " MATCH";
std::cout << "\n";
}
}
return 0;
}

254
src/face_scan.cc Normal file
View file

@ -0,0 +1,254 @@
#include <cmath>
#include <cstddef>
#include <iomanip>
#include <iostream>
#include <string>
#include <vector>
#include "face_engine.h"
#include "face_config.h"
static constexpr std::size_t EMBEDDING_SIZE = 128;
static void write_json_string(const std::string& value)
{
std::cout << '"';
for (unsigned char ch : value) {
switch (ch) {
case '"':
std::cout << "\\\"";
break;
case '\\':
std::cout << "\\\\";
break;
case '\b':
std::cout << "\\b";
break;
case '\f':
std::cout << "\\f";
break;
case '\n':
std::cout << "\\n";
break;
case '\r':
std::cout << "\\r";
break;
case '\t':
std::cout << "\\t";
break;
default:
if (ch < 0x20) {
const char* hex = "0123456789abcdef";
std::cout
<< "\\u00"
<< hex[(ch >> 4) & 0x0f]
<< hex[ch & 0x0f];
} else {
std::cout << static_cast<char>(ch);
}
}
}
std::cout << '"';
}
static bool finite_face(const Face& face)
{
if (!std::isfinite(face.score) ||
!std::isfinite(face.x1) ||
!std::isfinite(face.y1) ||
!std::isfinite(face.x2) ||
!std::isfinite(face.y2)) {
return false;
}
for (const Point& point : face.kps) {
if (!std::isfinite(point.x) ||
!std::isfinite(point.y)) {
return false;
}
}
if (face.embedding.size() != EMBEDDING_SIZE)
return false;
for (float value : face.embedding) {
if (!std::isfinite(value))
return false;
}
return true;
}
static void write_face_json(
const Face& face,
std::size_t face_index)
{
const float width = face.x2 - face.x1;
const float height = face.y2 - face.y1;
std::cout << "{";
std::cout
<< "\"face_index\":"
<< face_index
<< ",";
std::cout
<< "\"score\":"
<< face.score
<< ",";
std::cout << "\"bbox\":{";
std::cout
<< "\"x\":"
<< face.x1
<< ",";
std::cout
<< "\"y\":"
<< face.y1
<< ",";
std::cout
<< "\"width\":"
<< width
<< ",";
std::cout
<< "\"height\":"
<< height;
std::cout << "},";
std::cout << "\"landmarks\":[";
for (int i = 0; i < 5; ++i) {
if (i != 0)
std::cout << ",";
std::cout
<< "["
<< face.kps[i].x
<< ","
<< face.kps[i].y
<< "]";
}
std::cout << "],";
std::cout << "\"embedding\":[";
for (std::size_t i = 0;
i < face.embedding.size();
++i) {
if (i != 0)
std::cout << ",";
std::cout << face.embedding[i];
}
std::cout << "]";
std::cout << "}";
}
int main(int argc, char** argv)
{
if (argc != 2) {
std::cerr
<< "Uso: "
<< argv[0]
<< " image.jpg\n";
return 2;
}
const std::string root =
FACE_CLUSTERING_ROOT;
const std::string scrfd_model =
root + "/models/SCRFD_500M_KPS_640.rknn";
const std::string sface_model =
root + "/models/face_recognition_sface_2021dec.rknn";
FaceEngine engine;
if (!engine.init(scrfd_model, sface_model)) {
std::cerr
<< "Impossibile inizializzare FaceEngine\n";
return 3;
}
std::vector<Face> faces;
int image_width = 0;
int image_height = 0;
if (!engine.process_image(
argv[1],
faces,
image_width,
image_height)) {
std::cerr
<< "Scansione fallita: "
<< argv[1]
<< "\n";
return 4;
}
for (std::size_t i = 0; i < faces.size(); ++i) {
if (!finite_face(faces[i])) {
std::cerr
<< "Risultato non valido per face_index="
<< i
<< "\n";
return 5;
}
}
std::cout << std::setprecision(9);
std::cout << "{";
std::cout << "\"image\":{";
std::cout << "\"path\":";
write_json_string(argv[1]);
std::cout
<< ",\"width\":"
<< image_width
<< ",\"height\":"
<< image_height;
std::cout << "},";
std::cout << "\"faces\":[";
for (std::size_t i = 0; i < faces.size(); ++i) {
if (i != 0)
std::cout << ",";
write_face_json(faces[i], i);
}
std::cout << "]";
std::cout << "}\n";
return 0;
}

24
test/README.md Normal file
View file

@ -0,0 +1,24 @@
# Immagini di test
Inserire in questa directory immagini locali per verificare:
- rilevamento dei volti;
- bounding box;
- landmark;
- generazione degli embedding;
- cosine similarity;
- output JSON.
Esempio:
```bash
./bin/face_recognition test/reference.jpg test/query.jpg
```
Oppure:
```bash
./bin/face_scan test/image.jpg
```
Le immagini sono escluse dal repository tramite `.gitignore`.