The talking (and listening) Gecko

Hi, as I mentioned in my first post, I need some samples of how people would give commands to their desktops in a natural language, for now just English. I guess it'll be easier if I could give some sort of a questionnaire. In it are a few desktop activities that one usually performs on a desktop, if you were to do instruct your computer to do these activities how would you formulate the sentence? You can take specific instances to create samples, for eg. for the first question you can prepare at least three sample commands, one for playing music by artist ("Coldplay" for instance), one for an album and one for a genre.

It'll be a most valued contribution, as it will help improve the strategy used in analyzing natural language commands. Please email your results to mohit.verma.in@gmail.com with subject "gsoc help" or post your samples as comments here.

Questionnaire:

1. You want to play music (you may mention the media player or not), based on artist, genre, album etc.

2. You want to install/upgrade a software.

3. You want to initiate an IM conversation with a friend (you may mention the protocol or not, for eg. yahoo chat).

4. You want to locate a file/folder in your home directory.

5. You want to find an application which does a particular task (probably mentioned in its description).

6. You want to browse a website (possibly present in the history or in bookmarks).

(If you feel there are other common desktop activities, then please mention them as well in the sample submitted).

Here's the basic layout of the system.

The layer will accept inputs in two forms-

1. Regular text
2. Speech

The speech input will first have to be converted to text
using a speech recognition system called Sphinx. Since this
conversion is usually error prone, the text will be enhanced
using knowledge of the system.

After this, it can be handled in a similar way to regular text input.

At the first phase, a parser will generate a tree and tags for
a given user command. For this, a statistical parser written in
Java at the Stanford University Natural Language Group will be used, it can be checked here.

After this, the analyzer will try to determine the kind of action the user wants to perform and then the application specific interpreter will try to find the arguments in the natural language text, for example if a user wants to play some music, the title, artist, genre etc will probably be mentioned in the text, that will have to be mined.

Several times, the system will not be completely sure of the result generated, hence user recommendation will be taken to improve the accuracy.

Hi all.. Hmm well, through this blog I hope to communicate to everyone interested, the progress of my GSoC Project, still untitlted (you could help there) project. The idea is to make a functional "Natural Language + Voice User Interface for openSUSE Desktop" (the abstract can be viewed here).

So here I am with a replaced motherboard and upgraded RAM in my laptop, all gung-ho to start my first dream project.. And at this very moment, I need some help :P

The project is about making a software layer which lets a computer understand a user's commands in a natural language (through text or speech), which is why I'd like some people to give in a sample of their natural language commands. To give a clue, if I could give commands to your computer in English, I'd say things like this --

1. "Privately message Nihar Joshi on Google Talk"
2. "Enqueue all the Coldplay songs in the player"

I do have a basic strategy in place to understand such commands, the first step being a parser, after which an application-specific analyzer will convert such a command to a bash command (which can be executed directly on a Linux platform). However, since commands in a natural language like English can be of a very wide variety and my strategy might be suffering from a lack of perspective, I'd like to take inputs from several sources to test my strategy.

Imagine that you do have such a system working on your computer, what kind of commands would you like to give to your computer in English? Please don't restrict yourself to my examples and try to think of all routine desktop activities you perform on your computer. Should you prefer anonymity you can mail me your list at mohit.verma.in@gmail.com with subject "GSoC help".

Cheers and looking forward to your support, (support open source, thats what good guys do :P )