Skip to content

Commit

Permalink
20231016 add readme
Browse files Browse the repository at this point in the history
  • Loading branch information
der3318 committed Oct 16, 2023
1 parent 1d3fc21 commit e80e856
Show file tree
Hide file tree
Showing 3 changed files with 69 additions and 0 deletions.
69 changes: 69 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@

## ⌨️ Head Pose Keyboard Mapping

![java](https://img.shields.io/badge/java-17+-blue.svg)
[![webcam](https://img.shields.io/badge/com.github.sarxos:webcam--capture-0.3.12-yellow.svg)](https://github.com/sarxos/webcam-capture)
[![ort](https://img.shields.io/badge/onnxruntime-1.16.0-pink.svg)](https://github.com/microsoft/onnxruntime)
![portable](https://img.shields.io/badge/portable-win%20x64-green.svg)
![license](https://img.shields.io/badge/license-MIT%20%28inherited%29-blueviolet.svg)

A Java program that can trigger keyboard actions using head poses streamed from PC camera, leveraging the end-to-end landmark detection ONNX model (ref: [atksh/onnx-facial-lmk-detector](https://github.com/atksh/onnx-facial-lmk-detector)):

![DemoUsage.gif](https://github.com/der3318/head-mapped-key/blob/main/imgs/DemoUsage.gif)


### How to Run

| Using Portable Version |
| :- |
| For Windows AMD64 devices, an all-in-one app package [head-mapped-key-portable-x64.zip](https://github.com/der3318/head-mapped-key/releases/download/2023.10.16/head-mapped-key-portable-x64.zip) is available. Unzip and double-click `launch.bat` to turn camera on and start the tool. |


| Using JRE 17+ |
| :- |
| Download fat JAR [head-mapped-key-all.jar](https://github.com/der3318/head-mapped-key/releases/download/2023.10.16/head-mapped-key-all.jar) and run `java -Duser.language=en -Dfile.encoding=UTF8 -jar head-mapped-key-all.jar` in CLI. |


| Using Gradle Wrapper |
| :- |
| Clone the repository and use `gradlew run` command. |


Debug messages will be dumpped directly to the console. Press CTRL+C to stop listening. A JVM shutdown hook will help release resouces gracefully, including camera handle.

Also noted that the responsiveness of the tracking mechanism highly depends on the actual CPU speed. For example, on a 4-Core 1.90GHz Intel laptop, each frame takes about 900ms to process, which makes the overall experience a bit lagging.


### Customizing Settings

The editable `app.properties` (plain text file) provides basic values that can be consumed during runtime:

```txt
# preferred camera (will be prioritized if any)
camera.keyword=Camera Front
# how many ms should the worker wait to take next webcam capture
worker.delay=100
# https://docs.oracle.com/en/java/javase/17/docs/api/java.desktop/java/awt/event/KeyEvent.html
keyname.head.left=VK_LEFT
keyname.head.right=VK_RIGHT
```

Key mappings based on personal need would be the most useful one.


### How to Build & Redistribute

To compile the sources, JDK 17+ should be either accessible via environment variable `JAVA_HOME`, or the Java excutables are avaiable under `PATH`. This is the only prerequisite.

Command `gradlew shadowJar` will download Gradle v8.3 and use it to build the redistributable JAR: `build/libs/head-mapped-key-all.jar`, including all the dependencies.


### Inference Logic Behind the Scene

Nothing fancy. Given the horizontal locations of eyes and nose, the head turn can be roughly inferred.

![SampleFace.gif](https://github.com/der3318/head-mapped-key/blob/main/imgs/SampleFace.jpg)

For instance, `(Nose.X - LeftEye.X) : (RightEye.X - Nose.X) = (3 or higher) : 1` implies the head is facing relatively left (mirrored) to the sensor, and vice versa. Otherwise, the pose is considered centered.
Binary file added imgs/DemoUsage.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added imgs/SampleFace.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit e80e856

Please sign in to comment.