Virtual Controllers For Touch Based Devices (pt.1)

As I’m working on a series of articles on Pushbutton Engine I thought I’d use that as a basis for testing some virtual controllers I have be playing about with. For one, it saves me for having to create the majority of the scaffold code and concentrate on the actual controllers and how they integrate. One thing I was acutely aware of was keeping the integration of the controller as generic as possible with the idea that it could be used in any game, regardless of the code that supports it.

I also made a few assumptions, firstly I used my thumbs as the basis for determining the size of the controller’s thumb stick ‘head’ – I have quite big thumbs so I assumed that this would suffice for most people. The second one was more technical. As most PC games tend to rely heavily on keyboard input (and Flash web games even more so) for controlling your ship / player; I opted to emulate the 4 arrow keys – Up, Down, Left and Right instead of using co-ordinate translation*. There were a few reasons for key input emulation over co-ordinate translation. The first one I have already outlined. The second was to do with ease of conversion; as most Flash games use the keyboard already adding in the virtual controller shouldn’t require a major update to any pre-existing code. Lastly, there is the testing and debugging.

*I am working on a version that uses co-ordinate translation and will post it up soon.

Designing The Controller
The first thing to consider is the actual visual appearance of your controller – in my case this was going to be a thumb stick like those found on the Playstation or XBox controllers. For ease of mapping the users touch input on the thumb stick you really need two elements. First the thumb area, i.e. the place where the user places their thumb; secondly you need a surround. The surround is useful for two reasons. It provides a visual indication to the user on how far the thumb stick moves as well as allowing you to use this as the boundaries of your entire controller for the purpose of determining where the users thumb actually is and more importantly when they lift their thumb up. You can see a simple design for the entire thumb stick controller (including both the thumb and surround) in the image on the right.

Interestingly you don’t actually need to have any visuals if you don’t want them. Some games forego them and just map the users thumb location. This can be confusing for some so depending on type of users you are expecting to attract to your game may determine whether you can remove the visual elements entirely. If in doubt make the controllers slightly transparent.

As their is a visual aspect to this controller it makes sense to leverage Flash Professional CS5 – you can do this all in Flash Builder if you wish, but it makes the initial set up easier using a more visual tool. Create a new FLA file and save it somewhere on your machine with the file name VirtualControllers. Next create a new ActionScript 3.0 class, call it ThumbStick.as and save it in the same directory as the one that contains VirtualControllers.fla. I use a package structure for my classes so I have nested folders as the image shows. You don’t have to do this if you don’t want to and in this article you’ll be using a flat package structure.

Before we start writing code though you need to create a symbol in the Library for the controller, and import the image assets for both the surround and the thumb stick (which you can download via the link at the bottom of this article, or create your own).

Create a new symbol (Insert > New Symbol…)
Call this new symbol ThumbStick
Locate the downloaded assets for the surround and the thumb stick and import them in to your library (File > Import > Import to library)
Flash Pro CS5 will automatically create two new symbols (Symbol1 and Symbol2) in your Symbol library
Open the Symbol library (Window > Library)
Select Symbol1 and rename it based on the graphic it contains (it’ll either be the surround or the thumb image)
Repeat this process for Symbol2

Once you have renamed the symbols you need to edit them both to make sure that the enclosed image is centred in the middle of the symbol, as well as making sure that they are set as MovieClip symbols not Graphic symbols (which is what they will be by default).

Right click on the Surround symbol in your library
Select Properties from the context menu
When the properties dialog opens change the top Type drop down from Graphic to MovieClip
Click OK to apply and close this dialog window
Repeat this for the Thumb symbol too
With the Thumb symbol still selected double click it to enter edit mode
The stage should update with the image asset placed approximately in the middle of the screen
Select the image (a bounding blue box should appear)
From the Properties panel (Window > Properties) set the X and Y values to -45, this will centre the Thumb image as it’s 90×90 pixels
Repeat this for the Surround symbol, except set the X and Y values to -75 as the Surround image is 150×150. Again this will centre it

Now you have the main elements of your Thumb Stick controller defined you’ll need to combine them. To do this first edit the symbol you created called ThumbStick

Double click on the ThumbStick symbol in your Library to enter edit mode
Rename the default layer in your Timeline (Window > Timeline) to Thumb by right clicking on the layer and selecting Properties from the context menu
A dialog will open where you can set the new layer name. You can also just double click the layer to edit it directly
Create a new layer in the Timeline panel (click the New Layer icon in the bottom left of the Timeline) and name it Surround
Make sure you still have the Surround layer selected
Drag the Surround symbol from your library and make sure it is centred in the middle of your symbol
With the Surround symbol still selected open the Properties panel and give it an instance name of surround
Once you are happy with its position select the Thumb layer
With the Thumb layer selected drag the Thumb symbol on to the stage of the ThumbStick symbol, making sure it’s on the Thumb layer
Again centre the Thumb symbol so it is in the exact centre (note the cross hair in the middle of the symbols in the image)
And as with the Surround symbol, open the Properties panel (making sure it is still selected) and give the Thumb symbol the instance name of thumb

Quick Tip
If the bitmap assets are looking a bit ‘jaggy’ you can always apply smoothing to them by right clicking on each image in the library and selecting Properties from the context menu. When the dialog opens just tick the Smoothing checkbox and click OK to apply.

Now you have the FLA and ActionScript files saved we can get about creating the initial code. The first thing you need to do is make sure your class extends Sprite (by default it doesn’t extend anything). You’re also going to need some private variables to store some of the controllers info in. To that end add the following (or just copy and paste the code below):

Defining The Controller
Now that you have a design, and you have created your actual controller symbol, you need to start thinking about adding the code to actually control it when the user interacts with it. The best approach is to break this process down initially as a bulleted list of its functional parts. The list will likely look something like this:

Move thumb stick in a circular motion
Limit thumb stick movement to within the surround
Return thumb stick to centre of controller when the user removes their thumb
Dispatch an event when the user moves the thumb stick

Obviously you are free to add more items to the list but these four cover the basic functionality. With that in mind lets start coding…

Writing The Code
The first thing you need to do is make sure that your ThumbStick class extends Sprite – by default Flash Professional creates non-inheriting classes. Now your code will look like this (I’ve applied a bit of formatting to my structure in case yours is formatted slightly differently)

package  
{
	import flash.display.Sprite;
 
	public class ThumbStick extends Sprite 
	{
		public function ThumbStick() 
		{
 
		}
	}
}â€¨

Looking back at the four functional points I laid down earlier, the first related to moving the thumb stick in a circular motion let’s start by moving the thumb stick. This is a fairly complex process so I’m going to save that for part 2 of this article. However what you are going to do is set the controller so when you press down on the thumb stick it will drag in the direction of your thumb movement and on release it will re-centre itself back to the middle of the surround

To do this you need to add a couple of event listeners. These will deal with responding to the touch events when the user presses down with their thumb as well as when they remove it. Even though you can directly interact with the symbols you have within your controller (as you gave them instance names), the code hinting in Flash Professional’s code editor doesn’t kick in – so I tend to make a private variable that I map to the symbol to make this easier.

The code below has the handlers included as well as the event listeners so you can see the flow and structure (there is a problem with this code and I’ll explain what it is in a moment)

package  
{
	import flash.display.Sprite;
	import flash.events.TouchEvent;
 
	public class ThumbStick extends Sprite 
	{
		private var _thumb	:Sprite;
		private var _surround	:Sprite;
 
		public function ThumbStick() 
		{
			_thumb = thumb;
			_surround = surround;
 
			_thumb.addEventListener(TouchEvent.TOUCH_BEGIN, onThumbDown);
			_thumb.addEventListener(TouchEvent.TOUCH_END, onThumbUp);
			_surround.addEventListener(TouchEvent.TOUCH_END, onThumbUp);
		}
 
		protected function onThumbDown(e:TouchEvent):void
		{
			trace("Thumb Down");
		}
 
		protected function onThumbUp(e:TouchEvent):void
		{
			trace("Thumb Up");
		}
	}
}â€¨

So what’s the problem then? Well it has to do with testing. You see if you run this from inside Flash Professional you won’t be able to test the TouchEvents as there is no equivalent on a normal desktop PC. Not particularly useful if you don’t have a physical device at hand to test on (at this point I will emphasise how important it is to test you games , applications etc on the target hardware at some point).

One approach is to check the type of hardware you are running on and adjusting the code flow based on that by using the Capabilities() class. So if I update the code above further I can test to see if this is running on an ARM based processor or not (i.e most Smart phones and tablets). I can then tailor how the code is initialized.

package  
{
	import flash.display.Sprite;
	import flash.events.TouchEvent;
	import flash.system.Capabilities;
	import flash.events.MouseEvent;
 
	public class ThumbStick extends Sprite 
	{
		private var _thumb	:Sprite;
		private var _surround	:Sprite;
 
		public function ThumbStick() 
		{
			_thumb = thumb;
			_surround = surround;
 
			if(Capabilities.cpuArchitecture == "ARM")
			{
				_thumb.addEventListener(TouchEvent.TOUCH_BEGIN, onThumbDown);
				_thumb.addEventListener(TouchEvent.TOUCH_END, onThumbUp);
				_surround.addEventListener(TouchEvent.TOUCH_END, onThumbUp);
			}
			else
			{
				_thumb.addEventListener(MouseEvent.MOUSE_DOWN, onTestThumbDown);	
				_thumb.addEventListener(MouseEvent.MOUSE_UP, onTestThumbUp);	
				_surround.addEventListener(MouseEvent.MOUSE_UP, onTestThumbUp);
			}
		}
 
		protected function onThumbDown(e:TouchEvent):void
		{
			trace("Thumb Down");			
		}
 
		protected function onThumbUp(e:TouchEvent):void
		{
			trace("Thumb Up");			
		}
 
		// These are just for testing within Flash Pro		
		protected function onTestThumbDown(e:MouseEvent):void
		{
			trace("Mouse Down");
		}
 
		protected function onTestThumbUp(e:MouseEvent):void
		{
			trace("Mouse Up");
		}
	}
}â€¨

This is useful, but I tend to remove all of the non essential code just before I test on the actual device – after all why include code you will never execute? Moving on…

THe next functionality you need to implement is the actual ability to move the thumb stick. This is pretty easy to do and I’ve just created two methods to control this – one to deal with the actual movement when the user is touching the thumb stick, and one to reposition the thumb stick back to the centre of the surround

â€¨protected function initDrag():void
{
	_thumb.startDrag();
}
 
protected function resetThumb():void
{
	_thumb.stopDrag();	
	_thumb.x = 0;
	_thumb.y = 0;
}

To access these methods you need to place a call to initDrag() inside your event handlers that respond to the user placing their thumb on the actual thumb stick – onThumbDown() and onTestThumbDown(). Likewise the resetThumb() method needs to be called when the user removes their thumb from the controller, so you need to add a call to it within the onThumbUp() and onTestThumbUp() methods.

If you test it now it should function in a similar manner to the example below. Clicking and dragging the thumb stick moves it (try the example below). Releasing the mouse button will centre it back to its original position in the middle of the surround – note that while you can click and drag the thumb stick with your mouse, but it isn’t confined by the boundaries of the surround. This isn’t exactly what we want. Worry not though, you’ll be addressing that next.

[kml_flashembed publishmethod=”dynamic” fversion=”10.0.0″ replaceId=”virtcontroller1″ movie=”http://blog.flashgen.com/swfs/VirtualControllers1.swf” width=”200″ height=”200″ targetclass=”flashmovie”]

[/kml_flashembed]
Setting Controller Boundaries
As I mentioned earlier, this gives you basic drag functionality within your virtual controller and repositions the thumb stick on releasing the mouse button. However it doesn’t restrict the movement to within the actual controller surround. So with out further ado let’s jump right in the deep end and start with some Trigonometry :p. I’m not going to give you a trig primer here, but I will explain where applicable what is actually happening – if you want to know more there are many good resources online (a simple google search should suffice)

The first thing you need to do is add in a new event listener to monitor the thumb stick as you move it so it can be restricted when it hits the desired position at the boundaries of the controller surround. Now, normally you’d probably use the MouseEvent.MOUSE_MOVE event to monitor the movement of the thumb stick. However this event is quite expensive in terms of event dispatching and usage compared to the Event.ENTER_FRAME event. With that in mid you need to add the following line to end of the initDrag() method

1	addEventListener(Event.ENTER_FRAME, onThumbDrag);â€¨

Likewise you need to remove this event when the user stops interacting with the controller. To achieve this you just need to remove the event listener in the resetThumb() method. Like the initDrag() method add the following line to the beginning of the resetThumb() method. That way the event is removed straight away and any other house keeping routines can be processed once this has happened

1	removeEventListener(Event.ENTER_FRAME, onThumbDrag);â€¨

You’ll notice that both lines of code refer to an as yet unimplemented method called onThumbDrag(). This method is where all of calculations relating to the thumb stick’s movement boundaries are set. Below is the entire onThumbDrag() method code you’ll need to implement in your code.

protected function onThumbDrag(e:Event):void
{
	// Store the current x/y of the knob
	var _currentX		:Number = _thumb.x;
	var _currentY		:Number = _thumb.y;
 
	// Store the registration point of the surrounding 'joystick holder'
	var _registrationX	:Number = _surround.x;
	var _registrationY	:Number = _surround.y;
 
	// Subtract the two from each other to get the actual x/y
	var _actualX	:Number = _currentX - _registrationX;
	var _actualY	:Number =  _currentY - _registrationY;
 
	// Calculate the degrees for use when creating the zones.
	_degrees = Math.round(Math.atan2(_actualY, _actualX) * 180/Math.PI);
 
	// Calculate the radian value of the knobs current position
	var _angle 		:Number = _degrees * (Math.PI / 180);
 
	// As we want to lock the orbit of the knob we need to calculate x/y at the maximum distance
	var _maxX		:Number = Math.round((_radius * Math.cos(_angle)) + _registrationX);
	var _maxY		:Number = Math.round((_radius * Math.sin(_angle)) + _registrationY);
 
	// Check to make sure that the value is positive or negative 
	if(_currentX > 0 && _currentX > _maxX || _currentX < 0 && _currentX < _maxX)
		_thumb.x = _maxX;
 
	if(_currentY > 0 && _currentY > _maxY || _currentY < 0 && _currentY < _maxY)
		_thumb.y = _maxY;
}â€¨

Before I go through the various aspects of this method you’ve probably noticed that there are two variable fields being referenced beyond all of the local variables. They are _degrees and _radius. Both of which are defined below

1 2	private var _degrees :Number; private var _radius :Number = 25;â€¨

The _degrees variable I’ll cover later on, but the _radius variable deals with the amount of movement within the boundary of your controller. The reason it is set to 25 here is that the thumb stick is 50×50 pixels therefore half of that is 25 and the surround is 150×150 pixels – again half of that is 75. If you add 25 to the thumbs 50×50 dimensions you 75. Simple really. So when the user drags it the most it can move is to the actual edge of the surround. If you want it to move beyond the edge slightly just increase the value. Likewise if you want it to move a shorter distance reduce this amount.

The Math ‘Bit’
That’s all well and good in theory, but how does that actually get applied in the code above? Well let’s start at the beginning of the method and look at each block of code so you can get a clearer understanding of the actual flow. First up you have a set of local storage variables. These just contain references to the position of the thumb stick and the surround:

// Store the current x/y of the knob
var _currentX		:Number = _thumb.x;
var _currentY		:Number = _thumb.y;
 
// Store the registration point of the surrounding 'joystick holder'
var _registrationX	:Number = _surround.x;
var _registrationY	:Number = _surround.y;â€¨

Next up are the variables that store the actual X/Y position within the controller itself by subtracting the thumb stick’s position from the surround’s position.

1
2
3

â€¨// Subtract the two from each other to get the actual x/y
var _actualX	:Number = _currentX - _registrationX;
var _actualY	:Number =  _currentY - _registrationY;

Next up is the _degrees variable. As I mentioned earlier I’ll be going in to more detail on what you’ll be using this for shortly. All you need to know at this point is that it takes the values of _actualX and _actualY and converts them to radians (using the Math.atan() method) which are them in turn converted in to degrees – i.e. a value from 0 to 180 and -179 to -1

1 2	â€¨// Calculate the degrees for use when creating the zones. _degrees = Math.round(Math.atan2(_actualY, _actualX) * 180/Math.PI);

The _angles variable just converts the _degrees value back in to radian. While I appreciate that I could have simplified this code and just used Math.atan() like the _degrees variable. This way you can at least see how to convert between degrees and radians (should you ever need to)

1 2	// Calculate the radian value of the knobs current position var _angle :Number = _degrees * (Math.PI / 180);â€¨

You’re almost at the end of this process so hang on in there. There are two parts to restricting the thumb sticks X and Y position. First of all you need to calculate the maximum distance the thumb stick can move. The second part then checks to see if the thumb stick’s actual X or Y values are within or beyond those values. If they are beyond those values it automatically sets the X or Y of the thumb stick to the maximum distance for that property if outside of the boundary. To calculate the distance isn’t that obvious as you are measuring the edge of a circle. That’s where the trigonometry I mentioned comes in. By calculating the two sides of a right angle triangle within the controller surround you can easily then work out the hypotenuse (longest side) of the triangle from the centre. And that is all that is happening here:

1
2
3

// As we want to lock the orbit of the knob we need to calculate x/y at the maximum distance
var _maxX		:Number = Math.round((_radius * Math.cos(_angle)) + _registrationX);
var _maxY		:Number = Math.round((_radius * Math.sin(_angle)) + _registrationY);â€¨

As I said the second part then actually checks the position of the thumb stick and if it is beyond the maximum position it resets it to that maximum:

// Check to make sure that the value is positive or negative
if(_currentX > 0 && _currentX > _maxX || _currentX < 0 && _currentX < _maxX)
	_thumb.x = _maxX;
 
if(_currentY > 0 && _currentY > _maxY || _currentY < 0 && _currentY < _maxY)
	_thumb.y = _maxY;â€¨

With this code in place you can now test your controller. Now it is restricted to the orbit that defined within the onThumbDrag() method, so it doesn’t move beyond the bounds of the actual surround just like the version below

[kml_flashembed publishmethod=”dynamic” fversion=”10.0.0″ replaceId=”virtcontroller2″ movie=”http://blog.flashgen.com/swfs/VirtualControllers2.swf” width=”200″ height=”200″ targetclass=”flashmovie”]

[/kml_flashembed]
Where’s The Thumb Again?
One thing you will have already noticed is that if your mouse moves outside of the thumb stick symbol, when you release it, the thumb stick doesn’t return to the centred position in the middle of the surround. This is because there is no event registered to capture the release of the thumb stick if you accidentally move out of it. This needs resolving as it is highly likely that your user will move their thumbs beyond the bounds of the thumb stick when playing your game (it’s just one of those things as they aren’t looking at the controller and as it’s virtual there isn’t any real feedback that they have extended beyond the desired bounds).

To fix this you could just register the stage and respond when the users releases the thumb stick. This is fine desktop / web apps as they don’t generally support multi-touch so you can only interact with one element at a time with the mouse. On devices this isn’t the case. Because you can have multiple touch points if you listened for the release via the stage if you have a virtual button – for firing your ships guns for example – then every time you lifted your finger off the button it would reset the thumb stick. Not ideal really.

A better way would be to create a invidsible area that sits behind the actual controller graphics and listens for the release. That way it doesn’t matter if the user moves off the actual thumb stick symbol as this area would still respond to the release. Now one thing you would need to decide upon if you chose to implement this is how big should this ‘run off’ area be? Well that can be a bit subjective, but if you think about how the user will hold their device to use the controller then that should give you a good starting point.

For starters you don’t need it to cover a massive area – if fact I would say no more than a third of the width of the screen (working on the premise that the game is probably displayed as landscape). After all it’s unlikely the user will move their thumb a big distance given that they will be holding the device at the same time. If in doubt, test it out until you get a happy medium.

In the example below I have updated the ThumbStick symbol in Flash Professional by adding another layer called ‘Boundary’ and placed a new 25×25 pixel filled (red) circlular symbol in it. I gave it the instance name of boundary and made sure the layer was underneath all of the other layers in the ThumbStick symbol.

Obviously 25×25 pixels in size isn’t going to be that useful. However once you map the symbol to a variable in your ThumbStick() class you can easily set its dimensions. That way you could easily add in a getter/setter that would allow you to dynamically set the dimensions of the controller. In the code below I’ve just set this boundary symbol’s width and height with hard values within the constructor.

private var _thumb		:Sprite;
private var _surround	:Sprite;
private var _boundary	:Sprite;
 
private var _degrees	:Number;
private var _radius		:Number = 25;	
 
public function ThumbStick() 
{
	_thumb = thumb;
	_surround = surround;
	_boundary = boundary;
	_boundary.width = 200;
	_boundary.height = 200;
 
	if(Capabilities.cpuArchitecture == "ARM")
	{
		_thumb.addEventListener(TouchEvent.TOUCH_BEGIN, onThumbDown);
		_thumb.addEventListener(TouchEvent.TOUCH_END, onThumbUp);
		_surround.addEventListener(TouchEvent.TOUCH_END, onThumbUp);
		_boundary.addEventListener(TouchEvent.TOUCH_END, onThumbUp);
	}
	else
	{
		_thumb.addEventListener(MouseEvent.MOUSE_DOWN, onTestThumbDown);
		_thumb.addEventListener(MouseEvent.MOUSE_UP, onTestThumbUp);
		_surround.addEventListener(MouseEvent.MOUSE_UP, onTestThumbUp);
		_boundary.addEventListener(MouseEvent.MOUSE_UP, onTestThumbUp);
	}
}â€¨

I’ve left the boundary symbols alpha value at 1 so you can see it (as shown in the SWF below – it’s the red circle :p). Try it out thius time and even though you may drag off the actual thumb stick, as long as ou are still over the boundary area it will still reset the thumb stick back to the centre once released. Obviously if the boundary area isn’t big enough you can enlarge it as you feel is necessary – it doesn’t even need to be circular. A rectangle would work just as well.
[kml_flashembed publishmethod=”dynamic” fversion=”10.0.0″ replaceId=”virtcontroller3″ movie=”http://blog.flashgen.com/swfs/VirtualControllers3.swf” width=”200″ height=”200″ targetclass=”flashmovie”]

[/kml_flashembed]
Summary
In this article you got a good foundation in how to approach creating a virtual controller that can be used on touch based devices (and we’ll be looking in to that part of the process in more detail in part 2). You also saw how to restrict the movement of the thumb stick to within the actual controller surround; as well as what should happen if the user accidentally moves their thumb off of the thumb stick and then removes their thumb from the screen.

With the foundation set you can now leverage this to add further enhancements such as dispatching events when the user moves the controller so that you can wire it in to your game easily.

You can read the second part of this article here: Virtual Controllers For Touch Based Devices (pt.2)

Related Files
VirtualController_pt1.zip

Virtual Controllers For Touch Based Devices (pt.1)

2 Comments

Leave a Reply