aspose file tools*
The moose likes Android and the fly likes Android multi-touch.  How exactly does it work? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Java 8 in Action this week in the Java 8 forum!
JavaRanch » Java Forums » Mobile » Android
Bookmark "Android multi-touch.  How exactly does it work?" Watch "Android multi-touch.  How exactly does it work?" New topic
Author

Android multi-touch. How exactly does it work?

Stephen Bell
Ranch Hand

Joined: May 11, 2013
Posts: 36
OK Guys, I was wondering if someone would be kind enough to explain exactly how multi-touch works in Android? I've read various blogs and tuts online (including the official developers guide) but I still can't make head nor tails of it.

If I press the screen with one finger, no problem, I also know how to detect a non-primary finger touching the screen.

But if I have say, 2 fingers on the screen and I remove one of them, how can I determine where the remaining finger is on the screen? This is partly what I don't understand.

Where does event.getPointerCount, event.getActionIndex(); & event.getActionMasked(); come into it?

Here is my code: Please see the MotionEvent.ACTION_POINTER_UP: section, this is the part I don't understand - I need to know where the remaining fingers are.

Would be grateful for any help - thanks!!

Code

Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3967
    
  17

The MotionEvent keeps track of all the fingers/pointers on the screen. You can keep track the fingers using the getX(int) and getY(int) methods. The integer parameter is the index of the point being tracked. If you want to get the location of the fingers that were not released as part of the ACTION_POINTER_UP action, then you would:
1) Call the getPointerCount() to determine the total number of fingers touching the display. The indeces will be from 0 to the count-1
2) Determine the index of the finger which was released. Use the [tt]getActionIndex()
method for that.
3) Loop through all the other indeces using getX(index) and getY(index)

Note: the index of each pointer is not kept constant, so the first finger you put down will not always be index 0, and the third finger to touch the screen won't always be index 2. If you need consistency by finger, then you need to use the pointer id to get the index that you care about.

Stephen Bell wrote:...I also know how to detect a non-primary finger touching the screen.

I think you should disabuse yourself of the notion of a 'primary' finger. The ACTION_DOWN is performed the first time a finger touches the screen (call it A). Then ACTION_POINTER_DOWN if a second finger touches (call it B). If the user releases finger A, then the ACTION_POINTER_UP is sent, because there is still one remaining finger left. And the ACTION_UP is sent when finger B is removed. The ACTION_DOWN and ACTION_UP are used to signify the first finger and last finger, which may not be the same finger. So there is no such thing as a 'primary' finger.

Steve
Stephen Bell
Ranch Hand

Joined: May 11, 2013
Posts: 36
Hi Steve,

Thanks for this, I've implemented what I think is the correct way of doing this after reading your post but still have a couple of issues - would be grateful if you could advise of what I'm doing wrong!!

I've posted my code below, what happens is that if I press the left button, sprite goes left. If I the press the right (with the left still presses) then it goes right.

If I then release the right button (with the left still pressed from the very beginning, it goes left) but......

Instead of releasing the right button as above, I release the left button with the remaining finger on the right control, the sprite goes left!! I've looked over and over my code but I can't figure out why this could be, I think everything looks OK (I'm checking coordinates of all remaining fingers against the controls).

Would really appreciate if you can spot the error in my code! Thanks again, appreciate it!

Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3967
    
  17

The finder which produced the ACTION_POINTER_UP still counts as a pointer for that event, so its index is included in the pointers you loop through. My guess is when you release out-of-order you end up getting the released finger's index after the other other fingers. This is why I said:

Steve Luke wrote:2) Determine the index of the finger which was released. Use the [tt]getActionIndex() method for that.
3) Loop through all the other indeces using getX(index) and getY(index)


You haven't done anything to avoid processing the released finger as one that is still pressed. So I would suggest you do that.

Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3967
    
  17

On the otherhand, I can't help but think this might be easier/more efficient/better if you put touch listeners on the individual buttons, and let the buttons decide when they are pushed, and when they aren't, and handle the cases as such. Maybe something like this:

Stephen Bell
Ranch Hand

Joined: May 11, 2013
Posts: 36
Thanks again, I thought that when lifting a finger, the index assigned to it was always the most recent (so, three fingers on-screen, any of them is lifted, and 2 is returned).

I can see that I was incorrect in assuming this.

I don't understand therefore, how to discount the finger that was lifted from the screen from my loop?

Any idea how I would achieve this?

Thanks again.

PS thanks for your other suggestion. Unfortunately, I'm not using 'buttons' as such. I'm using graphics that I draw and place on the screen, so I have to do this via screen-coordinates.
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3967
    
  17

Sorry about mis-reading your logic, I didn't notice you were using both < and pointerCount-1. So I thought you were processing all pointers.

Still, I wouldn't make any supposition about pointer index. Rather, get the index of the action, loop through all pointers, and work on the ones that are != the one that is UP. Example:


If you have three pointers and two buttons, where are the two remaining pointers located? What happens if one of them is on the same button the released finger is on? Or if they are on different buttons, which direction should the movement go?
Stephen Bell
Ranch Hand

Joined: May 11, 2013
Posts: 36
Ah, I see - I never thought of that - I will give it a try and let you know how I get on. You're right about not making suppositions regarding the Pointer Index!

Cheers.
Stephen Bell
Ranch Hand

Joined: May 11, 2013
Posts: 36
Success! Thanks very much for this it works, and I have a better understanding now of how multi-touch works on Android! :-)
Stephen Bell
Ranch Hand

Joined: May 11, 2013
Posts: 36
Can I ask just one other thing please? How does one go about detecting motion on a secondary pointer? There is no "MotionEvent.ACTION_POINTER_MOVE".

So, when I'm doing something like this:



This works great as long as the finger is the 1st one down, but if it's not, it doesn't pick up the movement (it still monitors the 1st finger only).

if I log event.getActionIndex() it always returns 0 regardless of how many fingers are down (I'm guessing because ACTION_MOVE is only for the initial pointer?

:-(

Cheers
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3967
    
  17

The MOVE action isn't pointer-specific, so if you move all three fingers in a three finger gesture you don't get three events to process. Instead, you need to either get the position of the pointer you care about (by tracking its pointer ID) or you need to cycle through all the pointers to process them all.

I think in your case it would be to process all the pointers and check their current position, but am not positive.
Stephen Bell
Ranch Hand

Joined: May 11, 2013
Posts: 36
Hmmmm.... I don't seem to be able to use the pointer ID as it's always returns '0', regardless of how many fingers there are.

So I can't differentiate between fingers/pointers.

Cheers though for your help
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3967
    
  17

No, you get the pointer ids when the _DOWN actions happen, and you store them. Then you do findPointerIndex(pointerId) using the id of the pointer you want info about.

For example, if I wanted to track the last pointer, and make sure that the last pointer down is the one whose button matters, then I would use a stack holding the pointer ids. Each _DOWN event, I would put the new ID on the stack. Then each move I would peek at the top of the stack, get its location, and do the related action. Each _UP event I would remove the id from the stack, then peek at the top of the stack again and do what is there.

If you want to find a particular pointer that moves and do whatever it moves to, you would need to track the pointers and their last button. Then each move you would get each pointer, find which button they are on now, and 'move' it from the old one to the new one if they aren't the same. This wouldn't need a stack: it could be a Map<Integer,Direction> (or button, or something).

Will post again with some example code shortly
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3967
    
  17

Okay, so this is probably more than you need and may not fit into exactly what you are doing. But I was hacking around since the post were you said you weren't using buttons, and I thought 'if this were me, I would have my own class to help encapsulate the 'button' concept, even if it is not the Android Button.' So I made this:

Now, for this case, I think the only part that is necessary is the fact that there is a public boolean isEventInbounds(int x, int y) method which you can call to check if that button is pressed. I then modified my first bit of code snippet to use this class:


Then I did some other modifications of my original code:
1) I added a Map to track the Pointer IDs and what button they were last pushing
2) some methods for encapsulating the effects of left/right buttons
3) the OnTouch() method which determines what to do.

For #1, and #2, and #3 before adding the MOVE action, I had this code:

Lots of code, but the important part is that the handleRightButtonPush() and handleLeftButtonPush() take care of the DOWN and UP actions for each button, delegating to rightButtonUp() rightButtonDown(), leftButtonUp, rightButtonDown() as needed, and the ddddButtonUp/Down() methods take care of adding and removing pointers from the map.

So we have a Map<Integer, Button> which has all the Pointer IDs we care about, and what button they were last associated with. Now we get a MOVE action and need to:
1) Figure out where each pointer is located.
2) Figure out if a pointer left its last known location, and if so register that fact.
3) Figure out if a Pointer entered a new Button location, and if so, register that fact.

To do this, I thought I would have to account for pointers which where down, but not over one of the buttons. So I added the following code to track a 'NONE' button. Basically the same stuff as for the left and right buttons:


Now, I have something to put in the Map so I don't lose track of wandering pointers. Finally I can add MOVE to my onTouch method:


So the first half of the method determines if it is a MOVE action. If it is, we will handle it here. If not, then we will delegate to the handleddddButtonPush() methods, like before (but this time, also adding the NONE button push).

In the MOVE section of the method, I iterate all the pointers in the Map - the Map must contain all and only the Pointers I care about since I add/remove them whenever a DOWN/UP action occurs. So I iterate over each one, get its last known button, and check if it still sits on the same button. If it does, no further action is needed. If it does not, I check which button it does sit on and make it do that button's action.

None of the is compiled or tested, so take it as a little more than pseudo code. Also note that I think the way I handle the map in the iterator may lead to a ConcurrentModificationException, but I will let you handle that if it happens.

And like I said, this is so much different than what you have you may just want to take the Map or List idea and ignore 90% of the code
Stephen Bell
Ranch Hand

Joined: May 11, 2013
Posts: 36
Wow this is an incredibly detailed piece of code, thanks so much for this. It's gonna take me a while to read through it and understand what does what!

You're right about having a separate class, or at the very least I should have methods like you have for checking that left / right etc have been pressed instead of repeating the same code in every case.

Thanks again, I'm gonna get reading :-)
Stephen Bell
Ranch Hand

Joined: May 11, 2013
Posts: 36
Hi Steve, I've looked over the code multiple times but I think it's over my head! I can't seem to follow what's happening in it :-/ think I need something simpler lol

I did manage to 'kind of' get this working using a simple for-loop like so:



This works in that it will follow my finger movement whether initial finger or subsequent finger(s), however, the one problem is that if both fingers are held on the buttons (one on the left button and one on the right button), then the sprite is randomly turning left and right (presumably, because tiny movements are being detected and thus it's firing the move code and setting the sprite's direction when it doesn't need to be set).

So I guess my the code I've used is way too simple for this. Guess I have to find middle ground. Any idea how I can stop this happening? I'm guessing saving the position on a 'stack' like you advised, but I just don't get how to achieve this still? I'll keep looking over the code you wrote but if you could possibly break down that particular part I would be really grateful - thank you!
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3967
    
  17

I didn't use a Stack, that was an example for needing 'last finger down' but that isn't your scenario. You need to know only when a pointer (any pointer) moves from one button to the next. So you need to track the pointer and the button it is associated with. That is the job of a Map. Do you know what a Map is?

I will show you the code again, but simpler, and only for the Left button. To simplify things, I will get rid of the Button class. Instead I will use an enum to hold direction. You can use whatever you want but you need to use something that is safe, and enums are safe, easy, and descriptive. That is why I am using them:

I then have a Map which holds the known pointers and their last known direction:

I have a couple of methods to start and stop movement in the Left direction:

I then have methods which handle when the Left button is pushed (DOWN action) and released (UP action):

I have a method I extracted from the onTouch method to handle just the actions associated with the Left button:

And I have a method I extracted from the onTouch method which handles all non-MOVE actions and determines if the Left button's method should be called:

Finally, I have the onTouch method which determines if the non-MOVE action method should be called:

You should realize that these methods would get called in the reverse order as listed here. The onTouch() is called by the OS, which calls handleNonMoveAction() which calls handleLeftButtonPush() which calls leftButtonDown() which adds to the Map and calls goLeft(). I wrote the methods from the simplest and most specific to the most general.

So, if you put all that code in your project, and fix any compile errors, what you should get is in the simple case (with no finger movement) when the users touches and releases buttons with multiple fingers, the Left movement should be tracked properly. Additionally, behind the scenes, a Map is maintained so we know where the pointers are pointing to.

I think you need to be able to follow that part before continuing onto the MOVE. To prove that you should implement the Right side of the motion. Perhaps take care of the warning I put in the in the stopLeft() method's comments.

The MOVE action I have handled in its own method, like this:

I tried to be descriptive of what each part does. Note that this, too, just implements the Left side of things, but assumes the rightButtonDown() method has been created. To integrate this code, you will also nee to change the onTouch() method to:

The difference there should be pretty obvious to you.
Stephen Bell
Ranch Hand

Joined: May 11, 2013
Posts: 36
I've not used maps in Java before, but.............. I've picked out concepts from your code and spent all day writing a routine to track the pointers and I'm finally there :-)

Well, it appears to work anyway so that's always a major plus!

I definitely now have a better understanding of how multi-touch works on Android, it's definitely more complicated than I initially thought.

Again, thanks so much for your help with this - much appreciated!!
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3967
    
  17

Great, I am glad I could help.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Android multi-touch. How exactly does it work?
 
Similar Threads
Collision Detection nightmare
is there a batter way of doing this or a right way of doing it?
Building a better animation infrastructure
Enforcing Input Resolution
Android: Handling touch events - pinch zoom on gallery