summaryrefslogtreecommitdiffstats
path: root/report/wiimote_ir.tex
blob: f311ee306ec726403fbff4707c3dc8d568e7761c (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

\subsection{Wiimote IR input}

As mentioned in the introduction, we used the Nintendo Wii controller for both head tracking and 3D (mouse) input. In both applications we made use of the infra-red sensing ability of the controller. A Wiimote has a build in infra-red camera that is used to track up to four infra-red sources at a time. An on board image processing chip processes the images acquired by the camera and outputs the coordinates of the infra-red sources in camera coordinates, which are reported to the client (pc) via Bluetooth at a frequency of 100Hz. Several sources report that the infra-red camera has a horizontal and vertical resolution of $ 1024 $ by $ 768 $ pixels with a viewing angle of $ 40^{\circ} $ and $ 30^{\circ} $ respectively. However, the author of the wiimote interface library that we used, claimed that his measurements indicated that the reported coordinates never exceeded the range $ [0,1015]\times[0,759] $. Since there are no official specifications of the hardware inside the Wii controller we assumed the resolution of the camera to be $1016 \time 760$.

In order to use the wiimote as a 3D mouse, two infra-red sources/beacons are required to be positioned a certain distance apart, above or below the display. The camera coordinates of these beacons are used to calculate world coordinates (in $ \mathbb{R}^3 $) of the mouse cursor. The beacon shipped with the Wii console is misleadingly called the 'sensor bar'. It is a plastic bar that houses two groups of five infra-red LEDs positioned at the tips of the bar. The distance between the LED groups is approximately $ 20.5 $ cm. Other, after market sensor bars, that we used during testing and development of the software, contained two groups of three LEDs each.  

To controll the 3D mouse cursor, the user points the Wii remote at the display such that both the infra-red beacons are registered by the wiimote's camera. The xy position of the mouse cursor is set by pointing the wiimote at the desired location. The position of the cursor in the z direction can be controlled by moving the remote toward or away from the screen.

% In order to use the wiimote as a 3D mouse input device, the user points the wii remote at the display such that the infra-red camera registers the beacons. The camera coordinates of the beacons determine the position of the mouse pointer in screen coordinates. The depth of the mouse pointer is changed by moving the remote toward or away from the display, the distance between the two beacons in camera coordinates is used to calculate the distance of the wiimote relative to the sensor bar.

\subsubsection{Mapping the camera coordinates to 3D}

In the ideal situation the wiimote reports the 2D camera coordinates of two points, one coordinate for the left beacon $ g_l $ and one for the right beacon $ g_r $. As mentioned before, the camera coordinates are assumed to be in the range $ [0,1015]\times[0,759] $, the origin is in the lower left corner. %For convenience we define $ C_w = 1016 $ and $ C_h = 760 $ to be the width and height of the camera resolution in pixels and $ S_w $ and $ S_h $ to be the width and height of the screen/application window in pixels.

In short, the average between $ g_l $ and $ g_r $ is mapped to the x and y-coordinate of the cursor and the distance $ |g_l - g_r| $ is mapped to the z-coordinate of the mouse cursor. But before we get to that we first need to pre-process the raw camera coordinates.

The first pre-process step inverts the coordinates of $ g_l $ and $ g_r $. This is required to compensate for the fact the camera is being moved instead of the beacons. When the camera is pointing upwards, the beacons are visible in the bottom of the image, when the camera is pointing downwards the beacons are visible in the top of image. The same holds for the left and right.

The next step compensates for the rotation  of the wiimote over its z-axis, which causes the coordinates of the measured points to be rotated.  If the user is holding the wiimote on its side (rotated $90^{\circ}$) and points to the right, the cursor will move down, instead of the expected direction. As a solution, the angle of the line $ \overline{g_l g_r} $ with the x-axis is calculated and is used to counter rotate $ g_l $ and $ g_r $ over the center of the camera. This rather simplistic solution works when the sensor bar is assumed to be aligned with the screen's x-axis. In order to determine whether the Wii remote is rotated over $ 180^{\circ} $ one can use the accelerometer data, but note that this will only give a reliable orientation estimate when the wiimote is not accelerating. 

Let $ b'_r $ and $ b'_l $ be the corrected camera coordinates of the beacons.

For the virtual world we use a left handed coordinate system with the origin in the lower left corner, such that we look down the negative z-axis. The movement of the 3D cursor is confined by a bounding box $ \mathcal{B} $, the box is defined by two points $ p_{min}, p_{max} \in \mathbb{R}^3 $, the corners with minimum and maximum coordinates.

The cursor's world

To compute the position of the cursor in world coordinates, we map the wiimote position to a cursor position $ c_{rel} [0,1]\times[0,1]\times[0,1] $, relative to the dimensions of $ \mathcal{B} $. The final world coordinates of the mouse cursor $ c_{world} $ are computed as follows:

\[ c_{world} = p_{min} + c_{rel}(p_{max}-p_{min}) \]

The relative x and y coordinates are trivially computed by averaging $ B_{lc} $ and $ B_{lr} $, yielding the coordinates of the point at the center of the line segment $ \overline{B_{lc}B_{rc}} $ and dividing the x and y coordinates by maximum camera coordinates.

The relative z coordinate is determined by distance between the wii remote and sensor bar.

Let $ \Delta $ be the distance between the LED groups and $ g_l $ and $ g_r $ in $ $ be the camera coordinates of the left and right LED group respectively. The distance $ d $ between the wiimote and the sensor bar can be calculated as shown in Figure \ref{fig:wiimote_dist_calc}. The angle $ \theta $ is computed as the distance between $ g_l $ and $ g_r $ multiplied by the angle per pixel: 
\[ \theta = |g_l-g_r|\theta_{pix} \]
Because the ratio in camera pixels and the ratio in viewing angles is equal we don't have to distinguish between vertical and horizontal pixel viewing angles and can define the angle per pixel as:
\[ \theta_{pix} = \frac{\pi}{180}\times\frac{40}{1016} \] 
Assuming that the wiimote is positioned perpendicular to the sensor bar, the distance can be calculated by solving:
\[ d = \frac{\frac{1}{2}\Delta}{\mathsf{tan}(\frac{1}{2}\theta)} \]

\begin{figure}[!h]
\begin{center}
\includegraphics[scale=1]{figs/wiimote_dist_calc}
\end{center}
\caption{The diagram for the computation of the distance between the Wii remote and the sensor bar}
\label{fig:wiimote_dist_calc}
\end{figure}

Because the wiimote never is perpendicular to the sensor bar, the distance will be an estimate. If the angle between the wiimote and the sensor bar deviates from $ 90^{\circ} $, being an angle $ \alpha $ in the range $ (0, 180) $, then the measured distance $ |g_l-g_r| $ is a factor $ \mathsf{sin}(\alpha) $ from the actual distance that would be perceived from a perpendicular viewing angle where $ \alpha = 90^{\circ} $. The computed distance $ d $ will therefore be larger then the actual distance. However, the angle $ \alpha $ can be measured when using customized sensor bar with three LED groups with equally space in between. The angle can then be computed from the ratio between $ | g_l - g_c | $ and $ | g_r - g_c | $, where $ g_c $ are de camera coordinates of the center LED group. %a third beacon positioned at equal distances in between the other two. %Our 3D mouse implementation does not require such precise distance calculation, because the  


%We are going to compute the position of the cursor relative to the dimensions of .

%to move the cursor further in the y-direction, the camera  have to be inverted before the are mapped to the xy-plane. This is because the 

%In order to determine the xy-postion for a simple 2D mouse cursor, one would simply take the average of $ g_l $ and $ g_l $ to get the center of $ \overline{g_l g_r} $ and convert that to screen coordinates by multiplying by the 2D vector $ (\frac{S_w}{C_w}, \frac{S_h}{C_w}) $. 

% the of the cursor we take the average of 
%beacons as two points in camera coordinates. Let these two dots be $ L_c $ and $ R_c $, the coordinates of the left and right beacons in camera coordinates. The camera coordinates origin $ (0,0) $ is located in the lower left corner. 
%One problem of the multiple led beacons in the sensorbars is that at close range the 


% noise in ir dot coordinates

% grouping / smoothing