The talking (and listening) Gecko

Its been quite a while since I last blogged about the project status, and now a good summer of coding has come to an end (according to the program timeline). I should thank Mr. Stephen Shaw "decriptor" (for being a "kewl" mentor, Mr. Pascal Bleser "yaloki" for mavenizing the code, among other things and Mr. Bryen Yunashko "suseROCKS" for getting my project selected and finding a mentor for me. (It was fun to have three openSUSE board members involved in the project)

I would like to report my works done till the "firm" pen (or pencil) down date - 17/08/2009. The code and other things can be accessed at http://code.google.com/p/vaani

As mentioned in the proposal, the softwar consists primarily of two parts-

*Part 1: The text NLP part - which analyzes text inputs and tries to find common desktop activities that the user might be trying to convey through it.

*Part 2: The speech analyzer part - which converts an audio input to text, and lets the first part complete the rest of the process.

Part 1 (mostly present in vaani.shabd package) is fairly complete, currently it has the following plugins -

1. Instant message plugin - analyzes purple buddy list information, and uses dbus to open new chat windows in Pidgin (an Empathy plugin can be extended easily).

2. Application plugin - which right now collects information from the .Desktop files, and tries to find the required application based on the text.

3. Search plugin - this performs searches using the beagle-query command (to be upgraded to use beagle-dbus soon).

The framework is fairly clean, and new plugins can be added easily.

About the 2nd part (vaani.swar package), the approach was to have a grammar for each plugin, and then the Recognizer would use all of these grammars to convert speech commands to text. Right now, grammars for the instant message and application plugin are ready, however the 2nd part isn't functional yet, owing to some problems with grammar compilation by the sphinx system. Effort is currently been put into making it work asap.

The 0.1 release can be downloaded from here, although checking out from svn would be a better option. Also, we need to package the code soon, currently the best way to hack it is by opening the project in an IDE (I wrote in Netbeans) Please try, suggestions/contributions/criticism are always welcome.

I think one of the most fun things I've got to learn during Summer of Code is d-bus. I had to learn it to open instant message conversations in the Pidgin module. And I guess I can present a noobish tutorial to do something simple in Java using d-bus. Before you start, you can refer to this excellent manual here.

Pre-requisites-

1. you need JVM and JDK (the version I used was openJDK 1.6 taken from the standard repository)

2. dbus-java (which requires you to install libmatthew as well).

The task at hand is to make an empty note in Tomboy using a java program.

Step-1 Extend the required interface for the function you're looking to implement (you can use a software called d-feet which can help you analyze various buses, object paths and interface names)


package org.gnome.Tomboy;

import org.freedesktop.dbus.DBusInterface;
import org.freedesktop.dbus.DBusInterfaceName;

/**
*
* @author sourcemorph
*/

@DBusInterfaceName("org.gnome.Tomboy.RemoteControl")
public interface RemoteControl extends DBusInterface {

public String CreateNote();
}


[The annotation is important, and you need to mention the corresponding object path of the interface you're extending, here I have declared just one function that we needed, but you can choose from the available functions from d-feet].

Step-2 Write a main class to get an object of this interface type and execute the function.


/**
*
* @author sourcemorph
*/

import org.freedesktop.dbus.DBusConnection;
import org.freedesktop.dbus.exceptions.DBusException;
import org.gnome.Tomboy.RemoteControl;

public class NewClass {

private static String ObjectPath = "/org/gnome/Tomboy/RemoteControl";
private static String ServiceBusName = "org.gnome.Tomboy";
private static DBusConnection conn;

public NewClass() {
try {
conn = DBusConnection.getConnection(DBusConnection.SESSION);
RemoteControl c = (RemoteControl) conn.getRemoteObject(ServiceBusName, ObjectPath);
c.CreateNote();
} catch(DBusException ex) {
ex.printStackTrace();
}
}

public static void main(String [] args) {
NewClass n = new NewClass();
}
}


Fairly simple. By the way its something really admirable about D-Bus because now my Java code can interact with processes which could have been coded in C sharp, Python etc. I am going to call methods on the Pidgin interface through Java code, and learning D-Bus was totally worth the effort.

PS: thanks to my mentor Stephen Shaw for patiently referring me to the README files when dbus-java wasn't compiling.. :) and also for d-feet, its awesome!

Hi, after a week of contemplation (:P), when my laptop was away with the nice service center guys (:D) , I finally figured out that "vaani" (means sound in Hindi) seems to be a cool enough title. The project has been hosted at code.google.com/p/vaani, and part of the initial code has been uploaded. You can check it out using svn (though its in considerably bad shape right now).

The package structure is--

1. sourcemorph.nlp.vaani -- for this project
2. sourcemorph.nlp.shabd -- for nl text to bash command ("shabd" means word in Hindi)
3. sourcemorph.nlp.swar -- for speech to nl text ("swar" roughly means voice in Hindi).