• Feature extraction: use the same neural network to extract features of two face images
• Comparison: use cosine similarity or normalized L2 distance
• Training face recognition models: adopt margins in Cross Entropy loss to maximize identity separability

Jiankang Deng, Jia Guo, Jing Yang, Niannan Xue, Irene Kotsia, and Stefanos Zafeiriou. ArcFace: Additive angular margin loss for deep face recognition. T-PAMI, 2022.

• How to design a secure face recognition system?

• Security: more secure under attacks
• Efficiency: faster inference with FHE
• Performance: better verification or identification accuracy

• Security: only feature protection
• Efficiency: fast to compute inner-product of two vectors under FHE
• Performance: preserve performance thanks to negligible FHE noise

• Is it secure if we only protect features?

• If the client is corrupted by an adversarial, templates on the database would be stolen!!!

• step 1: generate adversarial samples (features) and send them to the server
• step 2: obtain similarity scores from the server
• step 3: solve a linear system to recover templates

• Why feature-encryption is not secure?

• The server takes outsourced features as input
• The client is easy to manipulate these features

• A secure solution: end-to-end encrypted face recognition (CryptoFace) across all stages!

• If the client is adversarial, it may attempt to infer encrypted features stored on the server.

• The client cannot access to features or networks.

• If the server is adversarial, it may attempt to collect biometric data from the client.

• The server cannot access to the secret key.

• Security: provide security for face data and database
• Efficiency: it is slow to evaluate CNNs on encrypted images
• Performance: it is hard to preserve performance due to FHE-compatible CNNs

• To solve efficiency and performance challenges, we should co-design FHE-compatible CNNs and inference system.

• Offline Enrollment: face image and identity
• Online Verification: probe face image and claimed identity

• How to design FHE-compatible convolutional neural networks?

• The objective is to reduce the complexity of rotation and multiplication.

• Packing for convolutional layers: MPCNN [1]
• Polynomial approximation for ReLU: AESPA low-degree Hermite polynomial [2]

• [1] Eunsang Lee, Joon-Woo Lee, Junghyun Lee, Young-Sik Kim, Yongjune Kim, Jong-Seon No, Woosuk Choi. Low-Complexity Deep Convolutional Neural Networks on Fully Homomorphic Encryption Using Multiplexed Parallel Convolutions. ICML 2022.
• [2] Jaiyoung Park, Michael Jaemin Kim, Wonkyung Jung, Jung Ho Ahn. AESPA: Accuracy Preserving Low-degree Polynomial Activation for Fast Private Inference.

• AESPA block only consumes 8 multiplicative levels.
• CryptoFaceNet block is a shifted AESPA block and fuses polynomial coefficients to convolutional weights. CryptoFaceNet reduces levels by 2.

• Shallow patch CNNs (PCNNs) consume less levels.
• The mixture of PCNNs can be evaluated in parallel under FHE.

7.2x speedup (2.7 hours → 23 mins), while preserving accuracy (89.64 vs 89.42)

reduce memory footprint by 17G

near-constant latency across different resolutions