How are Project Iris initiated keypresses different from other software generated keypresses?

Post Reply
annaek
Posts: 8
Joined: Mon May 23, 2016 4:46 am

How are Project Iris initiated keypresses different from other software generated keypresses?

Post by annaek » Fri Oct 14, 2016 11:33 pm

I've got a kind of strange problem that I'm trying to debug. I'm going to list the technical questions (as best I can, at least) first. Then I'll put the context below that.

I'm trying to understand how these keystrokes (generated by looking at an interactor) are appearing to the target application. Are they one "keypress" event? A "key down" event and "key up" event? Are there any interesting timing details? What would distinguish them from, for example, keystrokes sent by the Windows on-screen keyboard? (No, they aren't the same. Because they are not giving me the same behavior!)

Context: I've been experimenting with text entry using the EyeX. Obviously traditional on-screen keyboards are quite difficult because of lack of precision from the eye tracker, so I've been investigating all sorts of alternatives. One tool that I thought looked promising was EdgeWrite (http://depts.washington.edu/ewrite/), specifically the four key version. (The one intended for eye tracking doesn't seem to run on 64-bit Windows 7. And I actually really liked the idea of putting interactors at the four corners of my screen and allowing me to keep my text editor in the center anyway.)

That executable (4 button EdgeWrite) runs just fine on my computer and accepts input from my physical keyboard as expected. It also works just fine when the input is coming from the Windows on-screen keyboard. But, as soon as I trigger a single keypress from an interactor, I get a really strange bug. EdgeWrite registers that I pressed to the key, but something about that keypress is confusing its adaptive timeout algorithm. It never times out (and actually sends the character to the text editor). It will continue accepting more keypresses, including from the physical keyboard, but none of that will actually get it "unstuck" and allow a character to be entered into the text editor. (This suggests that maybe EdgeWrite is looking for some "end of key press" event that it never receives or perhaps is calculating that the keystroke has infinite length....)

It doesn't matter whether IRIS is running or not. The bug only appears when I actually trigger a relevant keystroke from an interactor. This makes me think that the problem has something to do with the characteristics of that keystroke, not some strange interference with, say, control of the system focus.

I am pretty sure that the bug is actually in EdgeWrite and not Project IRIS, but I would still like to understand it so that I can see if I can work around the problem. And it does seem that there is something different about this particular type of software generated input which is at least contributing to the problem. Any insight would be greatly appreciated!

Also potentially relevant: I disabled UAC some time ago. I'm running Windows 7 64 bit. I've been using plain old Microsoft Notepad as my text editor for these tests.
User avatar
Xcessity
Site Admin
Posts: 218
Joined: Sat May 16, 2015 2:23 pm
Location: Graz
Contact:

Re: How are Project Iris initiated keypresses different from other software generated keypresses?

Post by Xcessity » Mon Oct 17, 2016 10:19 am

Hi,

keyboard inputs are chained through several system layers when generated by physical hardware. You usually have a process like this:
  1. hardware interrupt generated by the keyboard
  2. the physical key number (=scan code) is interpreted by the keyboard driver and is translated to a virtual key number. Mapping depends on the localized keyboard mapping.
  3. the physical key and virtual key get sent to the application in focus as seperate windows message
Usually 99% of all applications work with windows messages specifying the virtual key press/release. It really depends though how the application that is listening to keyboard events interprets the messages.

Here are some things that we can try using macros.
  • increase the time between key up/down message
  • try to send send scan codes instead of virtual keys
Here are the macros. If you have problems using those macros, see this post.

So this first macro introduces a 200ms timeout between the key down/up actions. Just in case your app is not able to interpret instant key down/up messages.

Code: Select all

<MACRO id="W_KeyHold">
    <START>
      <KEYINPUT behaviour="PRESS" type="VIRTUAL_CODE" code="0x57"/>
    </START>

    <MAIN repeat="1">
    	<WAIT time="200"/>
    </MAIN>

    <END>
      <KEYINPUT behaviour="RELEASE" type="VIRTUAL_CODE" code="0x57"/>
    </END>
  </MACRO>
This macro sends a key input as scan code. The scancode is the hardware related key number. Some apps listen to those messages instead of virtual keys:

Code: Select all

<MACRO id="W_KeyHold">
    <START>
      <KEYINPUT behaviour="PRESS" type="SCAN_CODE" code="0x11"/>
    </START>

    <MAIN repeat="1">
    	<WAIT time="200"/>
    </MAIN>

    <END>
      <KEYINPUT behaviour="RELEASE" type="SCAN_CODE" code="0x11"/>
    </END>
  </MACRO>
annaek
Posts: 8
Joined: Mon May 23, 2016 4:46 am

Re: How are Project Iris initiated keypresses different from other software generated keypresses?

Post by annaek » Mon Oct 17, 2016 4:32 pm

Thanks! The scan code macro did the trick!
Post Reply