Basic Nagivation in Kinect using WPF

Vuyiswamb
Posted by in Kinect category on for Beginner level | Points: 250 | Views : 18136 red flag

Navigation in Kinect applications cannot be the same as other traditional applications with small buttons that are made to save safe and also Tabs that can allow a user to navigate with the application. Kinect approach is different and if you are new to Kinect, you probably came across the UI navigation problem as I did before, but was able to resolve the problem through friends who are also doing Kinect for Windows Development.


 Download source code for Basic Nagivation in Kinect using WPF

Objective 

 
In this article I wanted to show you on how to create a basic navigation system in wpf that uses hand gestures to select menu items. 
 

Final Output of our Application



Figure 1.1
 
What you see is our final navigational example. So basically there are three buttons that we will use as our menus, so when a user select any of the buttons, using the codingforfun sdk, we are able to show animation that is defined in the namespaces as depicted below

Code Explanation

 
xmlns:Controls="clr-namespace:Coding4Fun.Kinect.Wpf.Controls;assembly=Coding4Fun.Kinect.Wpf" 
 
And the hand in your application will be represented by a hoverbutton from the sdk 
 

HoverButton

  
    <Controls:HoverButton Margin="0" Padding="0" x:Name="kinectButton" ImageSize="50"                                           
                                ImageSource="/WpfApplication1;component/Images/myhand.png"  
                                ActiveImageSource="/WpfApplication1;component/Images/myhand.png" 
                                TimeInterval="4000"  Panel.ZIndex="1000" Canvas.Left="0" Canvas.Top="0" /> 

The Hoverbutton has an Interval property that allows to slowdown or speed up the animation that tells us the selection is about to happen. Also note there is an Image called “myhand.png” that is the image that is drawn or used when the user will move his hands. 
One other thing there is a function that Tracks the hands, one can use the left hand or Right hand, in my previous article, there was a part where we set the default Cursor, its either you select left or Right, but in this case you can use both hands interchangeable.
      
  //get the hand closest to the Kinect sensor 
        private static Joint GetPrimaryHand(Skeleton skeleton) 
        { 
            Joint primaryHand = new Joint(); 
            if (skeleton != null) 
            { 
                primaryHand = skeleton.Joints[JointType.HandLeft]; 
                Joint rightHand = skeleton.Joints[JointType.HandRight]; 
                if (rightHand.TrackingState != JointTrackingState.NotTracked) 
                { 
                    if (primaryHand.TrackingState == JointTrackingState.NotTracked) 
                    { 
                        primaryHand = rightHand; 
                    } 
                    else 
                    { 
                        if (primaryHand.Position.Z > rightHand.Position.Z) 
                        { 
                            primaryHand = rightHand; 
                        } 
                    } 
                } 
            } 
            return primaryHand; 
        } 

 

Hand Tracking

 
Also you can track if the hand is over the button so that you can start the animation that will tell the user that he or she is about to select the button
    
 //detect if hand is overlapping over any button 
        private bool isHandOver(FrameworkElement hand, List<Button> buttonslist) 
        { 
            var handTopLeft = new Point(Canvas.GetLeft(hand), Canvas.GetTop(hand)); 
            var handX = handTopLeft.X + hand.ActualWidth / 2; 
            var handY = handTopLeft.Y + hand.ActualHeight / 2; 
  
            foreach (Button target in buttonslist) 
            { 
  
                if (target != null) 
                { 
                    Point targetTopLeft = new Point(Canvas.GetLeft(target), Canvas.GetTop(target)); 
                    if (handX > targetTopLeft.X && 
                        handX < targetTopLeft.X + target.Width && 
                        handY > targetTopLeft.Y && 
                        handY < targetTopLeft.Y + target.Height) 
                    { 
                        selected = target; 
                        return true; 
                    } 
                } 
            } 
            return false; 
        }   
 

Skeleton Tracking

 
 Another thing is that , is that Kinect Allow maximum of two skeleton tracking , when tracking a Skeleton , you can decide to track all the skeleton or take one , in this function i track the skeleton closest to Kinect.
 
   
     //get the skeleton closest to the Kinect sensor 
        private static Skeleton GetPrimarySkeleton(Skeleton[] skeletons) 
        { 
            Skeleton skeleton = null; 
            if (skeletons != null) 
            { 
                for (int i = 0; i < skeletons.Length; i++) 
                { 
                    if (skeletons[i].TrackingState == SkeletonTrackingState.Tracked) 
                    { 
                        if (skeleton == null) 
                        { 
                            skeleton = skeletons[i]; 
                        } 
                        else 
                        { 
                            if (skeleton.Position.Z > skeletons[i].Position.Z) 
                            { 
                                skeleton = skeletons[i]; 
                            } 
                        } 
                    } 
                } 
            } 
            return skeleton; 
        }   
  

Hand tracking using Hoverbutton

 
In order to track or be able to use gesture on the button, I must tell the sdk that I am going to use those buttons, else nothing will happen to them when the hand is over the button, this helps the method that tracks if the hand is over the button, if the button was not registered, it will not see it as a button that is going to be used by our hoverbutton, this should be initialized on the loading of the form
 
//initialize buttons to be checked 
        private void InitializeButtons() 
        { 
            buttons = new List<Button> { GAME1, GAME2, GAME3 }; 
        } 
        
And also we must subscribe to the button events of the hoverbutton
 
kinectButton.Click += new RoutedEventHandler(kinectButton_Click); 

 
The above subscription must be done in the Form load
 
 
        void kinectButton_Click(object sender, RoutedEventArgs e) 
        { 
            selected.RaiseEvent(new RoutedEventArgs(Button.ClickEvent, selected)); 
        } 

 
As you can see here if the hoverbutton is now selected and the animation is done loading, then this part clicks the button that you have selected, if your button is not in the list of the array buttons, then you will notice no animation.
 
 

Kinect Events

 
Every page that you navigate from, you need to unsubscribe to the Kinect Events, else you will faced with a strange behavior that you can explain, if you don’t, your hand will still have access to pages that you navigated away from, and instead of clicking the current buttons it will be able to click buttons in the pages that does not exist in the current page. So it is important you call this before you navigate away from the page.
 
 
       private void UnregisterEvents() 
        { 
           KinectSensor.KinectSensors.StatusChanged -= KinectSensors_StatusChanged; 
           this.Kinect.SkeletonFrameReady -= Kinect_SkeletonFrameReady; 
           this.Kinect.ColorFrameReady -= Kinect_ColorFrameReady; 
             
        } 

 
You can do something like this
 
    private void GAME3_Click(object sender, RoutedEventArgs e) 
        { 
            UnregisterEvents(); 
            (Application.Current.MainWindow.FindName("_mainFrame") as Frame).Source = new Uri("Game3.xaml", UriKind.Relative); 
        } 
 
There is also another function that tracks your hand and display it or update your Page or form with the position of the hand   
  //track and display hand 
        private void TrackHand(Joint hand) 
        { 
            if (hand.TrackingState == JointTrackingState.NotTracked) 
            { 
                kinectButton.Visibility = System.Windows.Visibility.Collapsed; 
            } 
            else 
            { 
                kinectButton.Visibility = System.Windows.Visibility.Visible; 
  
                DepthImagePoint point = this.Kinect.CoordinateMapper.MapSkeletonPointToDepthPoint(hand.Position, DepthImageFormat.Resolution640x480Fps30); 
                handX = (int)((point.X * LayoutRoot.ActualWidth / this.Kinect.DepthStream.FrameWidth) - 
                    (kinectButton.ActualWidth / 2.0)); 
                handY = (int)((point.Y * LayoutRoot.ActualHeight / this.Kinect.DepthStream.FrameHeight) - 
                    (kinectButton.ActualHeight / 2.0)); 
                Canvas.SetLeft(kinectButton, handX); 
                Canvas.SetTop(kinectButton, handY); 
  
                if (isHandOver(kinectButton, buttons)) kinectButton.Hovering(); 
                else kinectButton.Release(); 
                if (hand.JointType == JointType.HandRight) 
                { 
                    kinectButton.ImageSource = "/WpfApplication1;component/Images/myhand.png"; 
                    kinectButton.ActiveImageSource = "/RVI_Education;component/Images/myhand.png"; 
                } 
                else 
                { 
                    kinectButton.ImageSource = "/WpfApplication1;component/Images/myhand.png"; 
                    kinectButton.ActiveImageSource = "/WpfApplication1;component/Images/myhand.png"; 
                } 
            } 
        } 

 

Basic Navigation

 
Ok that was just basics of Kinect, now let us come to Navigation. To be able to do navigation in WPF you need to create a main window that will contain a navigational Frame, this frame can be adjusted so that the contents will fit in nicely, think of it like something like Iframe or MasterPage with Content Pages in Asp.net. So the Page should not contain a Frame but a window, so this will must be in full screen so that you can have nice view of your app and it only serve pages. So in your application you will have one Window and other will be pages. So when I say “Pages” I don’t mean asp.net pages or Silverlight Pages, but these are WPF pages, so when you run your application it does not mean you will see an address bar, no. The following is an example of your window or Shell that will host the frame.
<Window x:Class="WpfApplication1.MainWindow" 
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" 
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" 
      Title="Splash"  Height="673.2" Width="1304.4" > 
     
    <Grid x:Name="LayoutRoot"> 
        <DockPanel> 
            <DockPanel.Background> 
                <LinearGradientBrush EndPoint="0.5,1" StartPoint="0.5,0"> 
                    <GradientStop Color="Black" Offset="0"/> 
                    <GradientStop Color="#FFCC33D3" Offset="1"/> 
                </LinearGradientBrush> 
            </DockPanel.Background> 
            <Frame x:Name="_mainFrame"    NavigationUIVisibility="Hidden" /> 
        </DockPanel> 
    </Grid> 
</Window> 
   
So nothing is in here except the docked frame with properties that hides the visibility of the navigation buttons. That is because we will use our own buttons to navigate, these were made for normal traditional applications that uses mouse to move between pages. Now this Window does not load anything until you tell the frame to load something, so in the window loaded or on the constructor you can load the Page that has menu
 
using System; 
using System.Collections.Generic; 
using System.Linq; 
using System.Text; 
using System.Threading.Tasks; 
using System.Windows; 
using System.Windows.Controls; 
using System.Windows.Data; 
using System.Windows.Documents; 
using System.Windows.Input; 
using System.Windows.Media; 
using System.Windows.Media.Imaging; 
using System.Windows.Navigation; 
using System.Windows.Shapes; 
  
namespace WpfApplication1 
{ 
    ///  
    /// Interaction logic for MainWindow.xaml 
    ///  
    public partial class MainWindow : Window 
    { 
        public MainWindow() 
        { 
            InitializeComponent(); 
            this.WindowState = System.Windows.WindowState.Maximized; 
            this.WindowStyle = System.Windows.WindowStyle.None; 
  
            if (Generics.LoadingStatus == 0) 
            { 
                _mainFrame.Source = new Uri("MainMenu.xaml", UriKind.Relative); 
                Generics.LoadingStatus = 1; 
            } 
        } 
  
    } 
} 

 
And the page that has menu will need to unsubscribe to its Kinect event before moving to other pages, and also other pages must do the same.
 
As depicted in Figure 1.1, you will see the application and select e.g. “GAME1” via your hand gesture, and it will take you to the “Game1” page as depicted in figure 1.2 
 

Figure 1.2
 
There is a button that is also hooked to our hoverbutton that when you select, it will clear its event and take you back to the main screen as depicted below
 
 
     private void BACKHOME_Click(object sender, RoutedEventArgs e) 
        { 
            UnregisterEvents(); 
            (Application.Current.MainWindow.FindName("_mainFrame") as Frame).Source = new Uri("MainMenu.xaml", UriKind.Relative); 
        } 
  

 

Figure 1.3
  

Conclusion

 
This is a simple basic navigation in WPF/Kinect. I have attached an example project that you can use as a framework, if you have any questions you can reply to this article or if you are viewing this article from Channel9, I will see the comment or question and reply.  
 
Thank you again for visiting DotNetFunda.com. 
 
Vuyiswa Maseko
 
Page copy protected against web site content infringement by Copyscape

About the Author

Vuyiswamb
Full Name: Vuyiswa Maseko
Member Level: NotApplicable
Member Status: Member,MVP,Administrator
Member Since: 7/6/2008 11:50:44 PM
Country: South Africa
Thank you for posting at Dotnetfunda [Administrator]
http://www.Dotnetfunda.com
Vuyiswa Junius Maseko is a Founder of Vimalsoft (Pty) Ltd (http://www.vimalsoft.com/) and a forum moderator at www.DotnetFunda. Vuyiswa has been developing for 16 years now. his major strength are C# 1.1,2.0,3.0,3.5,4.0,4.5 and vb.net and sql and his interest were in asp.net, c#, Silverlight,wpf,wcf, wwf and now his interests are in Kinect for Windows,Unity 3D. He has been using .net since the beta version of it. Vuyiswa believes that Kinect and Hololen is the next generation of computing.Thanks to people like Chris Maunder (codeproject), Colin Angus Mackay (codeproject), Dave Kreskowiak (Codeproject), Sheo Narayan (.Netfunda),Rajesh Kumar(Microsoft) They have made vuyiswa what he is today.

Login to vote for this post.

Comments or Responses

Login to post response

Comment using Facebook(Author doesn't get notification)