A C# Project to detect facial landmarks by using Intel RealSense SR300

1 Deploy the running environment

1.1 Install the driver for Intel RealSense SR300

The Intel RealSense has already been stopped supporting, however there are still many existing material could be used. Before we start our project, the first thing we want to do is to download the driver of SR300 from their website.

driver

You will want to install it like below. For one thing that you need to attention is that please make sure the RealSense has already connected to your computer. If there appears any connection problem, you may want to restart your computer to make sure it works.

dcm

1.2 Install the SDK for Intel RealSense SR300

After installing the driver for SR300, you will want to install the SDK for further work. From their website we can see that there exist both R3 and R2 SDK, please download the full SDK R2. Because the Face Tracking for R3 on their web site is in-completed also there exists many problems. After the preceding procedure please reboot your computer to apply all changes. SDK

1.3 Run the Sample program

After installing the SDK, samples folder will be automatically generated on your desktop folder.

SampleFolder

Please open the sample browser and tried some samples to make sure your computer support the SR300

sampleBrowser

2 Create Your Own C# Project

2.1 Create a WPF HelloWorld project

In this time, we recommend you to use Visual Studio 2015 to create your own project, since the SDK and the reference offered by Intel was completed before Visual Studio 2017.

First you will want to create a new project by click the new Project.

VShomePage In this time, we want to create a WPF project, the selection you want to click wpfThe project structure looks like below

helloworld

Before we run our HelloWorld Program, we need to configure our target CPU, since the RealSense are suggested to work with 6 generation CPU, we need to change the target CPU to X64 by doing the following steps

cpu

selectspu

x64

Then you could run your first HelloWorld Program by clicking the start button.

helloworld

2.2 Add the essential references for from SDK

Then we need to add the references from RealSense SDK so that the project we created could work with RealSense.

Please right click the references at the right panel of Visual Studio,

references

Then add new references. You will want to browse to you local folder, find the “libpxcclr.dll” in the path “C:\Program Files (x86)\Intel\RSSDK\bin\x64” and add this dll file.

references

After these, you need to close your Visual Studio and access “C:\Program Files (x86)\Intel\RSSDK\bin\x64” then copy the “libpxccpp2c.dll” file to your project folder “C:\Users\UserName\Documents\Visual Studio 2015\Projects\HelloWorld\HelloWorld\bin\Debug” and “C:\Users\UserName\Documents\Visual Studio 2015\Projects\HelloWorld\HelloWorld\bin\x64\Debug”

2.3 Capture Your Facial Landmark

2.3.1 Create a button

button

Find the button element from toolbox in left Panel. Simply drag it in to your main window View

buttonInMainWindow

​ Your XAML code of the button may look like below

1
2
3
<Grid>
<Button x:Name="button" Content="Button" HorizontalAlignment="Left" Margin="153,97,0,0" VerticalAlignment="Top" Width="75"/>
</Grid>

Double click the button in MainWindow it will automatically generate the C# code for you. Your C# code will look like below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
namespace HelloWorld
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
public MainWindow()
{
InitializeComponent();
}
private void button_Click(object sender, RoutedEventArgs e)
{
}
}
}

2.3.2 Initialize your Realsense

Add the following code before the MainWindow() function

1
2
3
4
5
6
7
8
9
10
private PXCMSenseManager senseManager;
private PXCMFaceModule faceModule;
private PXCMFaceConfiguration faceConfig;
private PXCMFaceData faceData;
public MainWindow()
{
......
}

Then add the following code inside your MainWindow() function

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
public MainWindow()
{
InitializeComponent();
senseManager = PXCMSenseManager.CreateInstance();
senseManager.EnableFace();
senseManager.Init();
faceModule = senseManager.QueryFace();
faceConfig = faceModule.CreateActiveConfiguration();
faceConfig.landmarks.maxTrackedFaces = 2;
faceConfig.landmarks.isEnabled = true;
faceConfig.detection.maxTrackedFaces = 2;
faceConfig.detection.isEnabled = true;
faceConfig.EnableAllAlerts();
faceConfig.ApplyChanges();
}

Run your project. You will see the green light in your RealSense is on, which means your codes work.

2.3.3 Create a thread to capture facial data

Add these three lines on top of your C# file.

1
2
3
using System.Threading;
using System.Threading.Tasks;
using System.IO;

Then create a ProcessingThread() function to capture the facial data

1
2
3
4
private void ProcessingThread()
{
}

Add the following while loop to refresh the data from the camera

1
2
3
4
5
6
7
8
9
10
11
private void ProcessingThread()
{
// Start AcquireFrame/ReleaseFrame loop
while (senseManager.AcquireFrame(true) >= pxcmStatus.PXCM_STATUS_NO_ERROR)
{
// Release the frame
if (faceData != null) faceData.Dispose();
senseManager.ReleaseFrame();
}
}

Add the following code inside while loop to capture facial module

1
2
3
4
5
6
7
8
9
10
11
12
13
while (senseManager.AcquireFrame(true) >= pxcmStatus.PXCM_STATUS_NO_ERROR)
{
faceModule = senseManager.QueryFace();
if (faceModule != null)
{
// Retrieve the most recent processed data
faceData = faceModule.CreateOutput();
faceData.Update();
}
// Release the frame
if (faceData != null) faceData.Dispose();
senseManager.ReleaseFrame();
}

Then add the following code inside the while loop to traverse all the faces in faceData, and create the instance of landmark

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
while (senseManager.AcquireFrame(true) >= pxcmStatus.PXCM_STATUS_NO_ERROR)
{
faceModule = senseManager.QueryFace();
if (faceModule != null)
{
// Retrieve the most recent processed data
faceData = faceModule.CreateOutput();
faceData.Update();
}
if (faceData != null)
{
Int32 nfaces = faceData.QueryNumberOfDetectedFaces();
for (Int32 i = 0; i < nfaces; i++)
{
PXCMFaceData.Face face = faceData.QueryFaceByIndex(i);
PXCMFaceData.LandmarksData landmarkData = face.QueryLandmarks();
}
}
// Release the frame
if (faceData != null) faceData.Dispose();
senseManager.ReleaseFrame();
}

Add the following codes inside your face loop to traverse all the landmark

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
for (Int32 i = 0; i < nfaces; i++)
{
PXCMFaceData.Face face = faceData.QueryFaceByIndex(i);
PXCMFaceData.LandmarksData landmarkData = face.QueryLandmarks();
if (landmarkData != null)
{
PXCMFaceData.LandmarkPoint[] landmakrPoints;
landmarkData.QueryPoints(out landmakrPoints);
for (int j = 0; j < landmakrPoints.Length; j++)
{
}
}
}

Add the following codes inside the landmark loop to export the coordinate

1
2
3
4
5
6
7
8
9
10
for (int j = 0; j < landmakrPoints.Length; j++)
{
using (StreamWriter sw = File.AppendText(filePath))
{
sw.WriteLine(i + "," + (j + 1) + "," + landmakrPoints[j].image.x.ToString() + "," + landmakrPoints[j].image.y.ToString());
}
}

You will find that there exist an error on the filePath, then we will construct it.

2.3.4 Construct the FilePath

In this project, the filePath is consist of

  1. the folder path that your want your data stored in
  2. the filename
  3. the extension

In this time, we will store our data as a Comma separated values file on our desktop and the extension should be “.csv”. The file name will be “landMark” going with the created time.

We first create these two strings to store the time and the path, and add these two lines before the MainWindow() function.

1
2
private string time = DateTime.Now.ToString("M_d_hh_mm_ss");
private string filePath = @"C:\Users\csplayground\Desktop\landMark";

Then we add these two lines in MainWindow() to construct the final filePath

1
2
3
4
5
6
7
public MainWindow()
{
InitializeComponent();
filePath += time;
filePath += ".csv";
......
}

After that, add these lines at the beginning of the ProcessingThread() function to set the table header of data

1
2
3
4
5
6
7
8
9
10
private void ProcessingThread()
{
using (StreamWriter sw = File.AppendText(filePath))
{
sw.WriteLine("User ID" + "," + "landmarkIndex" + "," + "X" + "," + "Y");
}
......
}

2.3.5 Invoke the Thread

We then invoke the thread in the button_click() by adding these two lines

1
2
3
4
5
private void button_Click(object sender, RoutedEventArgs e)
{
processingThread = new Thread(new ThreadStart(ProcessingThread));
processingThread.Start();
}

3 Run your Program and see the Exported data

Here is the full code of the C# file, check if there is any missing and run it. Click the button and there will generate the exported data on your desktop.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
namespace RsNoUIWPF
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
private Thread processingThread;
private PXCMSenseManager senseManager;
private PXCMFaceModule faceModule;
private PXCMFaceConfiguration faceConfig;
private PXCMFaceData faceData;
private string time = DateTime.Now.ToString("M_d_hh_mm_ss");
private string filePath = @"C:\Users\csplayground\Desktop\landMark";
DateTime startTime;
public MainWindow()
{
InitializeComponent();
filePath += time;
filePath += ".csv";
startTime = DateTime.Now;
// Instantiate and initialize the SenseManager
senseManager = PXCMSenseManager.CreateInstance();
senseManager.EnableFace();
senseManager.Init();
faceModule = senseManager.QueryFace();
faceConfig = faceModule.CreateActiveConfiguration();
faceConfig.landmarks.maxTrackedFaces = 4;
faceConfig.landmarks.isEnabled = true;
faceConfig.detection.maxTrackedFaces = 4;
faceConfig.detection.isEnabled = true;
faceConfig.EnableAllAlerts();
faceConfig.ApplyChanges();
}
private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs e)
{
processingThread.Abort();
if (faceData != null) faceData.Dispose();
faceConfig.Dispose();
senseManager.Dispose();
}
private void ProcessingThread()
{
using (StreamWriter sw = File.AppendText(filePath))
{
sw.WriteLine("User ID" + "," + "landmarkIndex" + "," + "X" + "," + "Y");
}
// Start AcquireFrame/ReleaseFrame loop
while (senseManager.AcquireFrame(true) >= pxcmStatus.PXCM_STATUS_NO_ERROR)
{
// Retrieve face data
faceModule = senseManager.QueryFace();
if (faceModule != null)
{
// Retrieve the most recent processed data
faceData = faceModule.CreateOutput();
faceData.Update();
}
if (faceData != null)
{
Int32 nfaces = faceData.QueryNumberOfDetectedFaces();
for (Int32 i = 0; i < nfaces; i++)
{
PXCMFaceData.Face face = faceData.QueryFaceByIndex(i);
PXCMFaceData.LandmarksData landmarkData = face.QueryLandmarks();
if (landmarkData != null)
{
PXCMFaceData.LandmarkPoint[] landmakrPoints;
landmarkData.QueryPoints(out landmakrPoints);
for (int j = 0; j < landmakrPoints.Length; j++)
{
using (StreamWriter sw = File.AppendText(filePath))
{
sw.WriteLine(i + "," + (j + 1) + "," + landmakrPoints[j].image.x.ToString() + "," + landmakrPoints[j].image.y.ToString());
}
}
using (StreamWriter sw = File.AppendText(filePath))
{
sw.WriteLine("--------------------------");
}
}
}
}
// Release the frame
if (faceData != null) faceData.Dispose();
senseManager.ReleaseFrame();
}
}
private void button_Click(object sender, RoutedEventArgs e)
{
processingThread = new Thread(new ThreadStart(ProcessingThread));
processingThread.Start();
}
}
}