kml2ov2

My day job is developing C++ code for TomTom HOME. My evening job the coming weeks will be to write an add-on for TomTom HOME; kml2ov2. This add-on will be able to convert Google Earth points (.kml) to TomTom Overlay points (.ov2) and if possible TomTom Itinerary/Route (.itn) files.

I made a commitment to write the kml2ov2 add-on outside office hours. It has to be finished within half a year. It wouldn’t be much of a commitment otherwise :-). My goal is to document the development process of this add-on and make available its source code.

If you want to contribute, please drop a message at this e-mail address.

Disclaimer

There is no guarantee that TomTom will support kml2ov2.

Background Information

A third-party, command-line tool called POICommand (POICmd) can be found on hfrmobile.com. It would be nice to have this kind of functionality integrated into TomTom HOME. Another third-party, web-based tool on gps-data-team.com exports OV2 files to Google Earth files. These two tools provide an excelent starting point.

On hfrmobile.com, there are some sample OV2 files like the Badner Bahn. Download it, unpack it, and enter the resulting Badner_Bahn.ov2 file path in the gps-data-team.com exporter. The result is a file called Badner_Bahn.kml.kml. Rename the extension to .xml and view the content with your favorite XML viewer.

One thing to note in the produced kml file is the XML namespace for the kml tag; <kml xmlns="http://earth.google.com/kml/2.0">[…]</kml>. The file does not start with a XML header like <?xml version="1.0" encoding="UTF-8"?>. Encodings are important. So, note to self; add the XML header. As with most XML namespaces, there is no document at the URI. But remove the 2.0 part and you will get forwarded to the KML documentation. Okay, time to stop writing and to start learning.

2008-03-15

OV2 files are binary. This means the I (need to write) an application to view the content in a meaningful manner. Finding the OV2 file-format specification is hard. Googling for "TomTom OV2 specification" does yield some results, but you have to dig around until you find something useful. The first promissing reference is about the TomTom Navigator SDK, Version 2.0 build 162 This is quite a dusty reference (2002): it talks about Navigator 1.5 and Palmtop B.V.

[Update: Old it may be, the OV2 file format is explained in detail, as a later found out when I took the time to read the document. So the information wasn't that hard to find after all.]

Next reference; How to convert POI databases to TomTom Navigator format. There are two applications in the zip file; DUMPOV2 version 1.22 and MAKEOV2 version 1.1. The accompanying text file mentions that the latest version of these applications can be found at www.palmtop.nl/ce/support/nav_poi.html, but that link does not resolve to a location at www.tomtom.com.

DUMPOV2 and MAKEOV2 are probably written by someone who has a high managerial position in TomTom right now. So let me go on the record here and state these apps are the best command-line tools to ever written to handle OV2 files ;-) However, Windows applications do not reuse of command-line tools like is custom for Unix applications. Being an employee of TomTom, it would be possible to gain access to the source code. Right now, that feels like cheating, so lets Google some more to find the OV2 file-format specification.

GPS::Poi is a Perl extension for 'ov2' file extension of POI (Point of Interest) for TomTom GPS. The Perl source code is included. Great, now I need to learn Perl too :-) My interpretation is as follows.

// An OV2 file consists of POI records. Each record has the following data format. // 1 BYTE, char, POI status ('0' or '2') // 4 BYTES, long, denotes length of the POI record. // 4 BYTES, long, longitude * 100000 // 4 BYTES, long, latitude * 100000 // x BYTES, string, label for POI, x == total length – (1 + 3 * 4) including terminating 0.

What I am missing is an encoding parameter. This means that OV2 files are limited to Latin 1. Welcome to the pre-Unicode world.

Let's write an application to test the above specification. To keep it simple, we will just import an OV2 file and display the POIs in a list. Below is a screenshot of the today's result. For some reason, the name of the Visual Studio 2005 solution file became PoiView. As it turns out, that name is in use by bluescreen.de. To avoid problems, I will not make this code available.

The OV2 file used as input can be found on hfrmobile.com.

After some more Googling using the keywords "TomTom OV2 SDK", I found a newer reference of the TomTom Navigator SDK, Version 3.0 build 193. It still feels a bit outdated (2004). But maybe the SDK has super stable API (as it should be). :-) It confirms the above POI record, but there are also two other codes, which turns out to be the so-called 'type' byte.

// 0 : Deleted POI. // 1 : Skipper (contains bounding rectangle for all POIs in the file). // 2 : Simple POI. // 3 : Extended POI.

The Skipper record is a bit dangerous because the length refers to the file, not the record. Odd. But that is why documentation is important. :-)

Turns out the SDK document is indeed outdated. There is a newer version available. From tomtom.com, goto TomTom PRO, software development, TomTom Navigator SDK, and click on "Click here to download the TomTom NAVIGATOR SDK manual" (Note to the TomTom web developers: Permalinks would be hugh improvement to simply indexing by search engines!) This time the document takes the form of a collection of web pages. The title of the document is "TomTom Software Development Kit, Version 6.0 (for Pocket PC)."

The good news is that there are still only four types of POI records in OV2 files. With this information, I can update the code of my prototype.

2008-03-16

KML files are XML files and support Unicode through utf-8 encoding. Because utf-8 is a super set of Latin 1, it is a good choice when writing OV2 files. Below there is today's screenshot of the kml2ov2 prototype when importing the Badner_Bahn.kml from hfrmobile.com.

As a further test, I read in the Badner_Bahn.kml.kml file produces by the gps-data-team.com export of the Badner_Bahn.ov2 file. The result is below.

Note the missing ß and other German characters (the URL in the Placement.name does not make things more readable.) A lookup of the file in an XML editor shows that the mistake in not in my KML importer.

Now that I can read OV2 file and import KML files, it is time to stop prototyping (writing OV2 is not that difficult). The next task is to read up on how to develop extensions for Mozilla applications.

Lesson #1) Add-ons is the popular name for an extension.
Lesson #2) Extension are not plugins.

I plan to write the kml2ov2 extension in JavaScript. Since that is not my native programming language, it may take a while before I can report some progress.

2008-03-17

The Building an Extension sample is very readable. The first thing that I like to do is inspect TomTom HOME by installing, what the sample calls, the 'Analytical Tools for Debugging'; The DOM Inspector and Venkman.

The aforementioned link to Venkman directly installs to Firefox when you click the install button. In my profile there are now three extension of which one has today's date; {f13b157f-b174-47e7-a34d-4815ddfdfeb8}. Inspecting the install manifest (rdf file), there are all kinds of applications IDs, none of which matches TomTom HOME's application ID. Come to think of it, what is TomTom HOME's application ID? The application.ini file in the xul subfolder directly under the installation folder provides the answer; home2@tomtom.com. TomTom HOME 2.2.2.83 does not offer an entry in the menu to install an extension. So, I copy the Venkman extension from the extensions subfolder in my Firefox profile to the extension folder in my TomTom HOME profile. The following lines need to be added to the install manifest of Venkman.

 <em:targetApplication> <Description> <em:id>home2@tomtom.com</em:id> <em:minVersion>2.2.2.83</em:minVersion> <em:maxVersion>2.2.2.83</em:maxVersion> </Description> </em:targetApplication>

I restart TomTOM HOME and the extension is installed as shown in the screenshot below.

The installation may be successful, nothing gets added to TomTom HOME's menubar. In FireFox, the 'Tools' item in the menubar gets an extra entry called 'JavaScript Debugger'. TomTom HOME also has the 'Tools' item in the menubar, but the entry 'JavaScript Debugger' is nowhere to be found. This is of course not a surprise because I installed an extension for FireFox. The following four overlay entries are found in the chrome.manifest of Venkman.

overlay chrome://communicator/content/tasksOverlay.xul chrome://venkman/content/venkman-overlay.xul overlay chrome://browser/content/browser.xul chrome://venkman/content/venkman-overlay.xul overlay chrome://messenger/content/messenger.xul chrome://venkman/content/venkman-overlay.xul overlay chrome://calendar/content/calendar.xul chrome://venkman/content/venkman-overlay.xul

The second one applies to FireFox.

Wladimir Palant has gotten wind of my plans (okay, I wrote him an e-mail :-) As he writes, at TomTom we use ported versions of Venkman and the DOM Inspector. If we want people to start developing extensions for TomTom HOME, we need to make these available somehow.

The next task is to take a look at possible overlay points inside TomTom HOME. To do this, expand the tthome.jar file. Windows Vista does allow you to expand into install folder of TomTom HOME. As workaround, expand the files in a different location, say tthome in Documents. For inspection purposes, that suffices.

The chrome://tthome/content/ui/overlays/menubarOverlay.xul files contains the menu that is overlay for <menubar id="nav-menubar"> item in chrome://tthome/content/ui/mainwin/mainWin.xul (and chrome://tthome/content/ui/license/eulaWin.xul). As a starting point of my extension, I can add a menu entry in the 'Tools' section. After that is done, I need to find out how to add a dialog or something to enter the KML file.

2008-03-18

Wladimir published the ported versions of Venkman and DOM Inspector. When you unzip these add-ons in the extenstions folder in your TomTom HOME profile, you will see a new menu item called 'Debug' in the menubar with two entries.

Now that the development environment is working, it is time to think of design. Specifically, what will be the merge point? In other words, which part of TomTom HOME am I going to extend?

The most logical merge point has to be the 'Points of Interest' screen. Given the elements that are on that screen, I guess a button with the text 'Import' next to the 'Search' button is a probable solution. The screenshot below shows how this will look like. What do you think? Is it okay?

Please note the number of pages in 'Points of Interest' screen: 46 pages! Most of the items here are contributions from the community. WOW!

To create this screenshot, unzipped tthome.jar into a subfolder called 'tthome', modified three files, and simply started TomTom HOME.

xul\chrome\tthome\content\ui\subtask\shopSelect\serverList.xul <button id="search" label="&search.label;" disabled="true"/> <button id="import" label="&import.label;" disabled="false"/>  xul\chrome\tthome\locale\en-GB\ui\subtask\shopSelect\serverList.dtd <!ENTITY search.label "Search"> <!ENTITY import.label "Import">  xul\chrome\chrome.manifest replaced "jar:tthome.jar!" by "tthome" (without the quotes)

The disadvantage of using the serverList.xul screen is that it used for all content items. For most content items, the 'Import' button will be disabled. It is unlike that someday somebody writes an importer for maps. :-)

If you look closely at the above screenshot, you can see that I downloaded the 'Volkswagen, The Netherlands' POIs (installed it, and later removed it, that is why it says that the Location is 'My computer'. Of course, I could not resist to import the OV2 file into my kml2ov2 prototype. It failed to import. The reason was that I assumed that there could be one skipper record. After some modifications, I found out that there are several skipper records in this OV2 file.

There is also a POI with is coordinate swapped.

This tells me that there is a need for a tool to manipulate OV2 files, standalone from TomTom HOME. What do you think?

The next task is to design the screen that does the import of the KML file and allows the user to install the resulting OV2 file. Or maybe we should install it directly to the device by placing it in the folder of the active map?

Comments?

2008-03-22

Following the guidelines in the Building an Extension sample, it was not that difficult to create an add-on that produces the exact same result as the before. You can pick up the add-on here. To install it, you need to place it in a subfolder called "kml2ov2@rolandboon.net" (without the quotes) in your TomTom HOME profile. Please make sure that TomTom HOME is not running when you 'install' the add-on. The reason is that XULRunner does not like manual modifications of the content in the extensions subfolder, sometimes resulting in a crash of the application. Just a reminder, the import button is still non-functional.

TomTom has a site dedicated for Content Creation. Following the documention link, I ended up Add-To-TomTom Services. It describes how to create POI sets (OV2files, using makeov2.exe) and Itineraries (ITN files, using text files). The description for the latter mentions that itinerary files are plain ASCII. The example, however, contains "Gué (Le) (Vendeuvre−Du−Poitou)". The small e with acute accent begs the question whether the text file can contain characters from Code Page 437. Ignoring the answer, the documentation was helpful in writing an itinerary file importer for the kml2ov2-prototype.

2008-03-23

My goal for today was to open a new dialog after pressing the 'Import' button. This dialog contains a textbox to enter the file name and a browse button to find the file using the file picker. To this end, I added the following files; importPanel.xul to define the UI elements, importPanel.js to handle the events of the UI elements, importPanel.dtd for the label on the UI elements, and importPanel.properties for the strings needed by the event handlers.

To open the new dialog, you need to call the following method.

// @brief Open a dialog. // @param url The location of the XUL file of the dialog. // @param loadCallback Function to call upon initialization. // @param unloadCallback Function to call when user quits dialog. // @param allowDupe Can there be more than one dialog with the same |url|. // @return The new layer (iframe). Dialogs.addLayer : function(url, loadCallback, unloadCallback, allowDupe)

The explanation of the parameters is my interpretation (the comments are stripped from the JavaScript code for the release version.) The result is the following call in the event handler for the 'Import' button.

const Kml2ov2 = { onImport : function(aTextbox) { var url = "chrome://kml2ov2/content/dialogs/importPanel.xul"; top.gMainWindow.dialogs.addLayer(url, function(page) {}, // TODO: initialize function(result) {} // TODO: Add the imported POI set to the list. ) } }

I did not add the fourth parameter because the default value (undefined) suits me just fine.

By reading the JavaScript code of TomTom HOME 2, I learned that it dialogs are derived from ttDialog. You need the correct list of JavaScript and CSS files to get that working. It is best to simply copy that list from another dialog.

Designing the XUL was a bit of a challenge. It took quite a bit of experimenting, and lots of starts of the application to verify the result. What was very helpful was to use the following command line "xulrunner\xulrunner.exe -app xul\application -jsconsole" (without the quotes.) At least that way, does one get a hint that a runtime error has occured.

Next goal; open the file and read it as an XML document to get the POI set.

2008-03-24

Once the POI set is read, we can 1) either install the POI set on the (active) TomTom device, or 2) store the POI set on the PC and let TomTom HOME handle the installation later. Applications like TYRE write directly on a TomTom device. This is possible because a TomTom device behaves like a mass-storage device. The application can enumerate the disk drives in a PC, and identify the TomTom device by looking for signature files (like 'ttgo.bif' or 'system'.) However, there are TomTom devices that do not behave like one mass-storage device but as *two* mass-storage devices; the TomTom GO520 and GO720 have internal memory which can be extended by inserting an SD card. The order in which internal memory and the SD card reported themselves to the PC is not guaranteed. The risk is that the POI ends up on wrong 'disk'.

Because writing to a TomTom device is risky, I was very happy to see that TYRE 2.51 (build 9) lets you copy the POI set to TomTom HOME. TYRE manages to find my download folder out of the box. Nice! One remark though; Unlike TomTom HOME, TYRE stores all POI sets directly inside the download folder. That is dangerous because it could lead to name clashing.

TomTom HOME stores POI sets in the download folder in a subfolder called "poi" (without the quotes) and creates a subfolder per POI set to avoid name clashing. The basic ingredients are a table-of-content file (TOC) file and a cabinet (CAB) file that contains the zipped content.

As far as I know there is no public document that describes the TOC data format. Here is the minimum description. TOC file are XML files and the outmost tag is TomTomToc. The inner node of TomTomHOME determines the type of content. For POI sets, use Poi, for a itineries use Itinerary. One level deeper, the following nodes describe the content.

Id: Unique Identifier.
Version: The version number. Use 1 as the default.{
Name: Name. Requires the attribute language='en-GB'.
Target: Lists the TomTom devices that understand the content. Use tomtom for all PNDs.
Location: file URL or relative path to the cab file.
Size: total size of the unpacked content.

The relative path for Location is prefered because it allows reallocating the TOC and CAB files without modification. TomTom HOME uses Id to create a subfolder under POI to store the TOC and CAB file. There should be no name clashing because the Id needs to be (locally) unique.

Final note. The value for size should be the unpacked size. HOME Services uses that value to verify whether a content item fits on the device. Unfortunately, TomTom HOME 2.2.2.38 writes the packed size during 'sharing content'. This is not correct.

Next goal; study TomTom HOME's JavaScript code to find out how to create a POI set in the download folder and how to add the new POI set to the list of POI sets.

2008-03-28

The last two nights, I've been wrestling with JavaScript, Mozilla components, and E4X (XML). There were several challenges to overcome. First, reading an XML file with a namespace requires adding a namespace scope for each node. Fortunately, my colleague Sergi pointed me to the relevant documentation. Second challenge was writing the POI labels to a binary file. The only way to get this working was using the "US-ASCII" charset for a nsIScriptableUnicodeConverter. The "Latin-1" and "ISO-8859-1" charset did not import correctly as OV2. The result is that all non-ASCII characters are converted to one or more question marks. That looks bad but I have not found a better conversion. The third and last challenge was to put the OV2 file and the corresponding bitmap into a zipped archive. This task was actually the simplest; nsIZipWriter does all the work.

The current implementation works but it is not very efficient. It reads the whole KML file first into a string and then converts it into an XML object. Next it converts the XML to a POI set and than that POI set is converted into an array of values later converted into bytes when writing a binary file. Please look up the code in kml2ov2.xpi.

The next task was to write a TOC file. To create the content, I simply concatenated strings. I expect that the xml.toXMLString() will automatically do the UTF-8 conversion. As a starting point, I used the TOC file that I got when downloading a POI set using TomTom HOME. It contains a "HOME2" section which I did not include for reasons explained below. For the Badner Bahn KML file, used previously, the resulting TOC could like as follows.

<TomTomToc version="1.0"> <Poi> <Id>Badner_Bahn</Id> <Version>1</Version> <Name>Badner Bahn</Name> <Description>Badner Bahn Stationen</Description> <Size>1403</Size> <Location content="main">Badner_Bahn.cab</Location> <Picture>_icon.png</Picture> <Preview>_preview.png</Preview> </Poi> </TomTomToc>

The import dialog needs additional fields to enter the necessary data, as can be seen in the screenshot below.

Somewhat to my surprise, TomTom HOME does not display the 'icon' and the 'preview', see screenshot below.

As it turns out, TomTom HOME only shows these images when they are listed in the "HOME2" section and they are always downloaded (ah, that is why the feature is still in Beta phase). To verify this, share your POI set by uploading it to TomTom. This will add the "HOME2" section to the TOC file. Since the Badner Bahn POI set is not my creation, I did upload it myself. Even though these 'Preview' and 'Picture' fields are non-functional in TomTom HOME 2.2.2.38, a future probably will, so I will keep them.

From the above screenshots, you could have noticed that I changed the merge point. I selected the menu as a merge point because I was unable to get the new POI set inserted into the list directly. Also, this way, it is not necessary to connect the TomTom device to test the add-on. I wanted to add a menu item in the 'File' menu, but I was unable to get it working. If you can demonstrate how to do this, please e-mail me.

At this point, we have a completely funtional kml2ov2 add-on. If you want to give it a try, grap it here. Because TomTom HOME 2.2.2.38 has no add-on installer, you need to manually extract/unzip the kml2ov2.xpi file in the kml2ov2@rolandboon.net subfolder in the extensions folder of you profile.

The current version (1.1) writes the POI set to the download folder. After that, you need to install the POI set using the existing functionality of TomTom HOME 2.2.2.38. Please be aware that kml2ov2 currently does not give feedback whether the import was successful, or if it failed, why it failed.

A next version of kml2ov2 will allow installing it to the current/active map when a TomTom device is connected. Also, the option to export to a TomTom Itinerary/Route (.itn) file should be included. Furthermore, I got a request to support the import of other formats than KML. I will be happy to include it. However, some help how to select file types UI/design-wise would be appreciated. Finally, as said before, the error handling needs some attention.

Some Information about TomTom HOME

TomTom HOME uses the XULRunner as the application runtime. XULRunner is a Mozilla runtime package that runs XUL+XPCOM applications. One of its nicest features, in my opinion, is that it enables add-ons and plug-ins. The user-interface of TomTom HOME is a combination of XUL and JavaScript code. If you want to take a look at that code, simply extract from the tthome.jar file with zip tool like 7zip. The UI team strips out the comments (with a Perl script, if I remember correctly) so the code may look a bit sparse with odd indentations.

The C++ code written for TomTom HOME gets compiled into serveral DLLs. The most significant DLL is HOMEServices.dll. This DLL provides the so-called HOME Services that are used by the JavaScript code of the user-interface. HomeServices.dll uses several other DLLs like HomeBase.dll, MapShare.dll and three wxWidgets V2.8.7 DLLs called wxbase28u_net_vc_custom.dll, wxbase28u_vc_custom.dll, wxmsw28u_core_vc_custom.dll. TomTom HOME 1 was built upon wxWidgets and TomTom HOME 2 reuses its non-UI code.

The latest TomTom HOME version to date (March 14th, 2008) is 2.2.2.83. Example add-ons for this version are the Map Share™-statusindicator and various Emulator versions corresponding to TomTom Navigator versions (the application that runs on the device). Googling for "TomTom HOME add-ons" does not yet lead to the right web page which means that the word is not out yet. The developer of AdBlock Plus for Firefox, Wladimir Palant, also works for the TomTom HOME team. You might want to watch his blog. Lord knows I do :-)

Some Information about Roland Boon

Before TomTom, I worked as a software developer/consultant by Atos Origin, Logica (when its name was LogicaCMG and before that CMG), Tech5, and Imtech (Turnkiek Technical Systems, actually, assimilated by ImTech). My favorite language is still C#. Hopefully one day, I can show you some screenshots of BFC running on an XBox 360.