Navigation in Kinect applications cannot be the same as other traditional applications with small buttons that are made to save safe and also Tabs that can allow a user to navigate with the application. Kinect approach is different and if you are new to Kinect, you probably came across the UI navigation problem as I did before, but was able to resolve the problem through friends who are also doing Kinect for Windows Development.
Objective
In this article I wanted to show you on how to create a basic
navigation system in wpf that uses hand gestures to select menu items.
Final Output of our
Application
Figure 1.1
What you see is our final navigational example. So
basically there are three buttons that we will use as our menus, so when a user
select any of the buttons, using the codingforfun sdk, we are able to show animation
that is defined in the namespaces as depicted below
Code Explanation
xmlns:Controls="clr-namespace:Coding4Fun.Kinect.Wpf.Controls;assembly=Coding4Fun.Kinect.Wpf"
And the hand in your application will be
represented by a hoverbutton from the sdk
HoverButton
<Controls:HoverButton Margin="0" Padding="0" x:Name="kinectButton" ImageSize="50"
ImageSource="/WpfApplication1;component/Images/myhand.png"
ActiveImageSource="/WpfApplication1;component/Images/myhand.png"
TimeInterval="4000" Panel.ZIndex="1000" Canvas.Left="0" Canvas.Top="0" /> The Hoverbutton has an Interval property that
allows to slowdown or speed up the animation that tells us the selection is
about to happen. Also note there is an Image called “myhand.png” that is the
image that is drawn or used when the user will move his hands. One other thing there is a function that Tracks
the hands, one can use the left hand or Right hand, in my previous article,
there was a part where we set the default Cursor, its either you select left or
Right, but in this case you can use both hands interchangeable.
//get the hand closest to the Kinect sensor
private static Joint GetPrimaryHand(Skeleton skeleton)
{
Joint primaryHand = new Joint();
if (skeleton != null)
{
primaryHand = skeleton.Joints[JointType.HandLeft];
Joint rightHand = skeleton.Joints[JointType.HandRight];
if (rightHand.TrackingState != JointTrackingState.NotTracked)
{
if (primaryHand.TrackingState == JointTrackingState.NotTracked)
{
primaryHand = rightHand;
}
else
{
if (primaryHand.Position.Z > rightHand.Position.Z)
{
primaryHand = rightHand;
}
}
}
}
return primaryHand;
}
Hand Tracking
Also you can track if the hand is over the
button so that you can start the animation that will tell the user that he or
she is about to select the button
//detect if hand is overlapping over any button
private bool isHandOver(FrameworkElement hand, List<Button> buttonslist)
{
var handTopLeft = new Point(Canvas.GetLeft(hand), Canvas.GetTop(hand));
var handX = handTopLeft.X + hand.ActualWidth / 2;
var handY = handTopLeft.Y + hand.ActualHeight / 2;
foreach (Button target in buttonslist)
{
if (target != null)
{
Point targetTopLeft = new Point(Canvas.GetLeft(target), Canvas.GetTop(target));
if (handX > targetTopLeft.X &&
handX < targetTopLeft.X + target.Width &&
handY > targetTopLeft.Y &&
handY < targetTopLeft.Y + target.Height)
{
selected = target;
return true;
}
}
}
return false;
}
Skeleton Tracking
Another thing
is that , is that Kinect Allow maximum of two skeleton tracking , when tracking
a Skeleton , you can decide to track all the skeleton or take one , in this
function i track the skeleton closest to Kinect.
//get the skeleton closest to the Kinect sensor
private static Skeleton GetPrimarySkeleton(Skeleton[] skeletons)
{
Skeleton skeleton = null;
if (skeletons != null)
{
for (int i = 0; i < skeletons.Length; i++)
{
if (skeletons[i].TrackingState == SkeletonTrackingState.Tracked)
{
if (skeleton == null)
{
skeleton = skeletons[i];
}
else
{
if (skeleton.Position.Z > skeletons[i].Position.Z)
{
skeleton = skeletons[i];
}
}
}
}
}
return skeleton;
}
Hand tracking using Hoverbutton
In order to track or be able to use gesture on the button, I must
tell the sdk that I am going to use those buttons, else nothing will happen to
them when the hand is over the button, this helps the method that tracks if the
hand is over the button, if the button was not registered, it will not see it
as a button that is going to be used by our hoverbutton, this should be
initialized on the loading of the form
//initialize buttons to be checked
private void InitializeButtons()
{
buttons = new List<Button> { GAME1, GAME2, GAME3 };
} And also we must subscribe to the button events of the hoverbutton
kinectButton.Click += new RoutedEventHandler(kinectButton_Click);
The above subscription must be done in the Form load
void kinectButton_Click(object sender, RoutedEventArgs e)
{
selected.RaiseEvent(new RoutedEventArgs(Button.ClickEvent, selected));
}
As you can see here if the hoverbutton is now selected and the
animation is done loading, then this part clicks the button that you have
selected, if your button is not in the list of the array buttons, then you will
notice no animation.
Kinect Events
Every page that you navigate from, you need to unsubscribe to the Kinect
Events, else you will faced with a strange behavior that you can explain, if
you don’t, your hand will still have access to pages that you navigated away
from, and instead of clicking the current buttons it will be able to click
buttons in the pages that does not exist in the current page. So it is
important you call this before you navigate away from the page.
private void UnregisterEvents()
{
KinectSensor.KinectSensors.StatusChanged -= KinectSensors_StatusChanged;
this.Kinect.SkeletonFrameReady -= Kinect_SkeletonFrameReady;
this.Kinect.ColorFrameReady -= Kinect_ColorFrameReady;
}
You can do something like this
private void GAME3_Click(object sender, RoutedEventArgs e)
{
UnregisterEvents();
(Application.Current.MainWindow.FindName("_mainFrame") as Frame).Source = new Uri("Game3.xaml", UriKind.Relative);
}
There is also another function that tracks your hand and display
it or update your Page or form with the position of the hand
//track and display hand
private void TrackHand(Joint hand)
{
if (hand.TrackingState == JointTrackingState.NotTracked)
{
kinectButton.Visibility = System.Windows.Visibility.Collapsed;
}
else
{
kinectButton.Visibility = System.Windows.Visibility.Visible;
DepthImagePoint point = this.Kinect.CoordinateMapper.MapSkeletonPointToDepthPoint(hand.Position, DepthImageFormat.Resolution640x480Fps30);
handX = (int)((point.X * LayoutRoot.ActualWidth / this.Kinect.DepthStream.FrameWidth) -
(kinectButton.ActualWidth / 2.0));
handY = (int)((point.Y * LayoutRoot.ActualHeight / this.Kinect.DepthStream.FrameHeight) -
(kinectButton.ActualHeight / 2.0));
Canvas.SetLeft(kinectButton, handX);
Canvas.SetTop(kinectButton, handY);
if (isHandOver(kinectButton, buttons)) kinectButton.Hovering();
else kinectButton.Release();
if (hand.JointType == JointType.HandRight)
{
kinectButton.ImageSource = "/WpfApplication1;component/Images/myhand.png";
kinectButton.ActiveImageSource = "/RVI_Education;component/Images/myhand.png";
}
else
{
kinectButton.ImageSource = "/WpfApplication1;component/Images/myhand.png";
kinectButton.ActiveImageSource = "/WpfApplication1;component/Images/myhand.png";
}
}
}
Basic Navigation
Ok that was just basics of Kinect, now let us come to Navigation.
To be able to do navigation in WPF you need to create a main window that will
contain a navigational Frame, this frame can be adjusted so that the contents
will fit in nicely, think of it like something like Iframe or MasterPage with
Content Pages in Asp.net. So the Page should not contain a Frame but a window,
so this will must be in full screen so that you can have nice view of your app
and it only serve pages. So in your application you will have one Window and
other will be pages. So when I say “Pages” I don’t mean asp.net pages or
Silverlight Pages, but these are WPF pages, so when you run your application it
does not mean you will see an address bar, no. The following is an example of
your window or Shell that will host the frame.
<Window x:Class="WpfApplication1.MainWindow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
Title="Splash" Height="673.2" Width="1304.4" >
<Grid x:Name="LayoutRoot">
<DockPanel>
<DockPanel.Background>
<LinearGradientBrush EndPoint="0.5,1" StartPoint="0.5,0">
<GradientStop Color="Black" Offset="0"/>
<GradientStop Color="#FFCC33D3" Offset="1"/>
</LinearGradientBrush>
</DockPanel.Background>
<Frame x:Name="_mainFrame" NavigationUIVisibility="Hidden" />
</DockPanel>
</Grid>
</Window>
So nothing is in here except the docked frame with properties that
hides the visibility of the navigation buttons. That is because we will use our
own buttons to navigate, these were made for normal traditional applications
that uses mouse to move between pages. Now this Window does not load anything
until you tell the frame to load something, so in the window loaded or on the
constructor you can load the Page that has menu
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
namespace WpfApplication1
{
///
/// Interaction logic for MainWindow.xaml
///
public partial class MainWindow : Window
{
public MainWindow()
{
InitializeComponent();
this.WindowState = System.Windows.WindowState.Maximized;
this.WindowStyle = System.Windows.WindowStyle.None;
if (Generics.LoadingStatus == 0)
{
_mainFrame.Source = new Uri("MainMenu.xaml", UriKind.Relative);
Generics.LoadingStatus = 1;
}
}
}
}
And the page that has menu will need to unsubscribe to its Kinect event
before moving to other pages, and also other pages must do the same.
As depicted in Figure 1.1, you will see the application and select
e.g. “GAME1” via your hand gesture, and it will take you to the “
Game1” page as
depicted in figure 1.2
Figure 1.2
There is a button that is also hooked to our hoverbutton that when
you select, it will clear its event and take you back to the main screen as depicted
below
private void BACKHOME_Click(object sender, RoutedEventArgs e)
{
UnregisterEvents();
(Application.Current.MainWindow.FindName("_mainFrame") as Frame).Source = new Uri("MainMenu.xaml", UriKind.Relative);
}
Figure 1.3
Conclusion
This is a simple basic navigation in WPF/Kinect. I have attached
an example project that you can use as a framework, if you have any questions
you can reply to this article or if you are viewing this article from Channel9,
I will see the comment or question and reply.
Thank you again for visiting DotNetFunda.com.
Vuyiswa Maseko