PhD Computer Vision Engineer - Real-time Face Filters & Video Editor (iOS)
Posted 2026-05-06Project Overview
I am developing a telemedicine platform where doctors record educational video content for patients. Doctors need TikTok-quality cosmetic filters (Velvet Vibe, Cinnamon, Skin Smoothing, etc.) to make their videos look professional and engaging.
Filters are for doctors's content creation, NOT medical diagnosis.
Filters will not be used for the consultations, and the non-healthcare user of this platform will not have access to record videos.
- Technical Requirements
- Deep understanding of 3D face models (FLAME, 3DMM, or similar)
- Experience with Perspective‑n‑Point (PnP) and pose estimation
- Strong iOS native development (Swift, Metal, Core Image)
- Experience with real-time face filters (MediaPipe, ARKit, or custom)
- Portfolio demonstrating TikTok-style cosmetic filters
- What Is Already Implemented and Working
- On-device FLAME / Core ML model loading and decoding
- 468 projected facial landmarks
- Dense FLAME mesh generation (5,023 vertices, 9,976 triangles)
- Vision-based face detection and tracking
- Head-pose estimation (PnP) with fallback modes
- Native mesh wireframe generation and JS overlay rendering
- 12 filter definitions (Core Image)
- What Is NOT Yet Implemented (The Gap)
- Accurate mesh-to-face projection across all head poses
- Feature-locked filters that precisely track lips, eyes, jawline
- TikTok-quality cosmetic filter application (Velvet Vibe, Cinnamon, Skin Smoothing, etc.)
Current status and problems
right now, you do have a real face detection / face tracking / face mesh pipeline, but you do not yet have true facial identity recognition. In other words: the app can find and track a face, estimate landmarks, and draw a FLAME mesh, but it is not recognizing “who” the person is.
Face Detection / Mesh The active iPhone pipeline is in FaceMeshCoreML.swift and is driven from FeedVideoRecording.tsx.
- What is already implemented there:
- On-device FLAME/Core ML loading and decoding.
- 468 projected facial landmarks.
- Dense FLAME mesh generation from 5023 vertices and 9976 triangle indices.
- Vision-based face detection plus VNTrackObjectRequest tracking between full detections.
- Bounding-box smoothing and pose smoothing.
- Head-pose estimation with PnP, plus fallback projection modes when pose fit is poor.
- Native mesh wireframe generation and JS overlay rendering in the camera screen.
- Debug/status plumbing so the app can show face count, mesh presence, mesh bounds, and source.
So the mesh system is real and already fairly advanced. The remaining issue is accuracy of projection/alignment, not “missing mesh technology.”
Facial Filters The filter system is also implemented, but it is much more approximate than the mesh system.
- What is already implemented:
- Native filter processing in FaceMeshCoreML.swift.
- A live iPhone preview path in FeedVideoRecording.tsx that now sends filter intensity again and can show the returned processedFrame.
- A JS live overlay tint/glow tied to the tracked face frame in FeedVideoRecording.tsx (line 2508).
- Native image filters applied with Core Image in FaceMeshCoreML.swift (line 3494).
- But the important limitation is:
- The filters are not placed from the FLAME triangle mesh.
- They are not feature-accurate for lips/eyes/eyelids.
- They currently use a soft radial face mask built from either landmark bounds or a Vision face box:
- FaceMeshCoreML.swift (line 3445)
- FaceMeshCoreML.swift (line 3464)
- So today’s filters are basically:
- face-area tinting / texture blending
- approximate face-region masking
- not precise makeup placement
- What Is Not Implemented Yet
- Identity recognition of a specific person.
- Feature-locked filters that hug lips, eyelids, nostrils, jawline, etc.
- Mesh-driven filter placement.
- Clean, production-grade FLAME-to-face alignment across all poses.
- What You Will Deliver
- Production-ready FLAME-to-face projection across all head poses
- Feature-locked filters that accurately track lips, eyes, jawline
- All 12 cosmetic filters working at 30+ FPS on iPhone
- Full integration into existing React Native / Swift pipeline
Source code ownership for DOCITOKI