Yesterday @simcup wrote on twitter about that he is currently downloading the whole Jamendo catalog of Creative Commons music.

Although I already knew Jamendo it never occurred to be to download their whole catalog. Since I am a fan of choice I immediately thought about how I could download the catalog too. Since the only clue was a cryptic uri-like text how to achieve that it suddenly sounded like a great idea to write a universal tool and release it as open-source. This tool should allow users to download the whole catalog and keep their local jamendo mirror in sync with the server. So anytime new artists, albums or tracks are added the user does not need to download them all again.

So the only thing I had as a starting point was that cryptic uri pointing me to something I’ve never heard of called Rythmbox. Turns out that this is a GNOME music player application which has Jamendo integration. After some clueless poking around I decided to take a look at the source of Rythmbox, especially the Jamendo module.

This module is written in python and quite clean to read. And just by looking at the first lines I came across the interesting fact that there is a almost daily updated XML dump of the Jamendo catalog available from Jamendo. Hurray! Since Jamendo wants developers to interact with the platform they decided to put a documentation online which allows anyone to write tools and stream and download tracks. After all the clues I found I finally ended up on this page.

So there are the catalog download, track stream and torrent uris necessary to download the catalog. Now the only thing that is needed is a tool which parses the XML and creates a nice folder structure for us.

Parsing XML in C# (my prefered programming language) is easy. Basically you can use a tool called XSD.exe and let it generate first the XSD from the XML and then ready-to-use C# classes from that XSD.

After doing all that actually reading the whole catalog into a useable form breaks down to just three lines of code:

Isn’t it great how modern frameworks take away the complexity of such tasks. At this point I’ve already parsed the whole catalog into my tool and only wrote three lines of code. The rest was generated automatically for me. The best of all – this also works on non-windows operating systems when you use mono.

When the XML data is parsed and available in a nice data structure it’s easy to iterate through all artists, all albums and all tracks and then download the actual mp3 or ogg. And that’s basically what my tool does. It takes the XML, parses it, and downloads. It will check before downloading if the track already exists and will only download those added since the last run.

Additionally since I am deeply involved into the development of the GraphDB graph database at sones I want to make use of the Jamendo data and the graph structure it poses. Since the directory structure my tool is generating is only one aspect how you could possibly look at the data it’s quite interesting to demonstrate the capabilities of GraphDB based on that data.

The idea behind the graph representation of the data is that you could start from almost any starting point imaginable. No matter if you you start from a single track and drill up into genre and artists, or if you start at a location and drill down to tracks.

So what the Downloader does in matters of GraphDB integration is that it outputs a GraphQL script which can be imported into an instance of GraphDB.

The sourcecode of my tool is available on github and released unter the BSD license – feel free to play with it and to contribute.

Source 1: http://www.jamendo.com
Source 2: https://github.com/bietiekay/JAMENDOwnloader

Oh boy, it seems that Apple just screwed up big time when it comes to data privacy. Obviously everytime someone attaches an iOS device like the iPhone to a PC or Mac and it does a backup run this backup includes the location data of that iPhone of the last several months. Impressive logging on the one hand and a shame that they did not talk about that in public upfront on the other hand.

There’s a great tool available on GitHub which uses OpenStreetMap to visualize the logged data – it creates a quite impressive graphical representation of where I was the last 6 months…

Source 1: http://petewarden.github.com/iPhoneTracker/

Low Latency Network Audio was a dream for the past years (see an article of 2005 and 2008) and with AirPlay it’s finally there.

I am using the Apple AirPlay technology for several years now… after it got implemented into iOS it’s just fantastic to have the option to have whatever sound source I want to playing loud and clear in any room I want to…

Okay it’s not quite as sophisticated as the sonos solution regarding the control of multiple music sources in multiple rooms but it get’s the job done in an apartment.

So back to the topic: Apple integrated the AirPlay technology into their wireless base station “AirPort Express”. Basically AirPlay is a piece of software which receives an encrypted audio stream over the network and outputs the stream to the SPDIF or audio jack.

Back in 2005 there already was an emulator of this protocol called “Fairport” but Apple decided to encrypt the AirPlay traffic. This led to the problem that the encryption key was unkown because it’s baked into the AirPort Express firmware. And this is where the good news start:

“My girlfriend moved house, and her Airport Express no longer made it with her wireless access point. I figured it’d be easy to find an ApEx emulator – there are several open source apps out there to play to them. However, I was disappointed to find that Apple used a public-key crypto scheme, and there’s a private key hiding inside the ApEx. So I took it apart (I still have scars from opening the glued case!), dumped the ROM, and reverse engineered the keys out of it.”

So to keep things short: Someone got an AirPort Express, dumped the firmware, extracted the AirPlay encryption keys and wrote an emulator of the AirPlay protocol which uses the key. Voilá!

ShairPort is available in source code on the site of the guy and obviously it’s unsure if Apple will react by changing the encryption key in the future. But for the time being it works as advertised:

I took one of my computers and followed the instructions to update perl, install Macports and then run ShairPort. So when ShairPort is run it looks not as appealing as expected:

Notably  it uses IPv6 to communicate between iTunes and ShairPort… Oh I almost forgot to show how it looks in iTunes:

On another side note: It works on Linux, Windows and Mac OS X

Source 1: Apple AirPlay
Source 2: Sonos
Source 3: Apple AirPort Express
Source 4: ShairPort

I’ve worked on my little holiday project for a while now and it’s making great progress. Since logging is working for almost two weeks now I got some data that should be visualized. One main goal of the project is to have  a great UI to browse the sensor data.

So almost two weeks into the project I’ve started to learn JavaScript  

The logging server now included an internal http server which serves some pages and RESTful services already. One of those services is the sensor data service which can be asked to output JSON formatted sensor data. If you take that data using jQuery and the flot jQuery plugin you’ll get something like that:

Source: http://github.com/bietiekay/hacs

This EzControl XS1 device is a complex thing. And currently I am playing with more than 10 sensors and more than 10 actuators. Since poking around with such a device will most certainly lead to a condition where that configuration might get lost (like a power down for more than 30 minutes).

Therefore I was in need of a backup and restore tool. Because there isn’t one I had to write one myself. Here it is:

I can haz backup tool

My tool is available as opensource as part of the h.a.c.s. toolkit here. Enjoy!

Source 1: https://github.com/bietiekay/hacs/wiki/H.a.c.s.-toolkit
Source 2: http://github.com/bietiekay/hacs/

My holiday project is progressing: Today it was hardware delivery day!

So this is the hardware which is ready to be used:

• 1x EzControl XS1 controller
• 2x Temperature and Humidity sensor
• 8x Remote Power Switch

The EzControl XS1 is easy to use as far as I had the time to give it a try. After the network setup the XS1 offers a simple web interface and REST service. Built upon that REST service there is also a configuration application and a visualization application available. Those two applications are apparently built using the GWT framework.

I poked around a bit with the sensor and actor configuration screens and everything just worked. Those applications are great for the easy tasks. And for everything else hacs is what is going to be the tool of choice (to be written).

Source 1: http://www.ezcontrol.de
Source 2: http://github.com/bietiekay/hacs
Source 3: http://code.google.com/intl/de-DE/webtoolkit/overview.html

Hurray! One of those EzControl XS1 plus some sensors and actors is on the way to me. So I can finally start the little holiday project which will be called “HACS” (Home Automation Control Server).

The source code and documentation repository is up on GitHub as of now – you can access it here: https://github.com/bietiekay/hacs

If you are interested in working on that project – drop a comment.

In the last 10+ years I was fiddling with different home automation concepts. Mostly without broad use cases because at that time no one seemed to be interested in having sensors and actors like crazy at home. In fact not that many people seem to care these days.

Having more and more hardware and software around us creates the use cases for a broader audience people like me have for 10+ years. Mainstream is a bitch for nerds

That said I found a nice plastic box I want to use in a winter project. This plastic box is called “EzControl XS1”. It comes with several visible and “invisible” interfaces.

The visible and obvious ones are: power, 100 mbit ethernet, sd card slot. So it takes some power and does something on the network. The not so obvious and therefore “invisible” interfaces are the most interesting ones: the EzControl XS1 comes with the ability to send and receive on 433 Mhz and 868 Mhz.

Yes that are the ranges used by switchable and dimable power sockets, temperature sensor and AMR. The EzControl XS1 is not that cheap (coming at 189 Euros for the base version and additional 65 Euros per upgrade option). I do not own one yet so it’s the plan to acquire at least one and start of with dimable power sockets and add more sensors and actors on the way

One great feature of the EzControl XS1 is the embedded WebServer with which the users application (the one I want to write) can interact using a HTTP/JSON Protocol. Oh dear: Sensor data and Actor control using JSON. How great is that!

There is some example code available (even a proprietary iPad/iPhone client) but since I want to have some custom features I do not currently see to be available in software I am going to write a set of tools which will get and protocol sensor data and run scripts to controls actors. Oh it’ll be all available as open source (license not yet chosen).

P.S.: If some one from Rose+Herleth is reading this and wants to help – send me a test unit

Source 1: http://www.ezcontrol.de (in german though)
Source 2: http://en.wikipedia.org/wiki/Automatic_meter_reading
Source 3: http://www.ezcontrol.de/content/view/12/31/

Oh what a nice n3rd toy this would be. Rumors say it will be available soon for under $30. And for those who right now think: “What the hell is this?” – This is a coffee mug in the shape of a quite expensive canon lens. In fact I already heard of that idea more than a year ago and wrote about it here. At this time there were only hopes that it would be produced.

Source: http://www.petapixel.com/2010/03/06/canon-lens-mug-purchased-in-canada/

As you may know, my team and I are developing a graph database. A graph database is a database which is able to handle such things as the following:

social graph

So instead of tables with rows and columns, a graph database concentrates on objects and the connections between them and is therefore forming a graph which can be queried, traversed, whatever-you-might-want-to-do.

Lately more and more companies start realizing that their demand for storing unstructured data is growing. Reflecting on unstructured data, I always think of data which cannot single-handedly be mapped in columns and rows (e.g. tables). Normally complex relations between data are represented in relation-tables only containing this relational information. The complexity to query these data structures is humongous as the table based database needs to ‘calculate’ (JOINs, …) the relations every time they are queried. Even though modern databases cache these calculations the costs in terms of memory and cpu time are huge.

Graph databases more or less try to represent this graph of objects and edges (as the relations are called there) as native as possible. The sones GraphDB we have been working on for the last 5 years does exactly that: It stores and queries a data structure which represents a graph of objects. Our approach is to give the user a simple and easy to learn query language and handle all the object storage and object management tasks in a fully blown object oriented graph database developed from the scratch.

Since not everybody seems to have heard of graph databases, we thought it might be a good idea to lower barriers by providing personalized test instances. Everyone can get one of these without the need to install anything – a working AJAX/Javascript compatible browser will suit all needs. (get your instance here.)

Of course the user can choose between different ways to access the database test instance (like SOAP and REST) but the one we just released only needs a browser.

The sones GraphDB WebShell – as we call it – resembles a command line interface. The user can type a query and it is instantly executed on the database server and the results are presented in either a xml, json or text format.

Granted – the interested user needs to know about the query language and the possible usage scenarios. Everyone can access a long and a short documentation here.

Source 1: http://en.wikipedia.org/wiki/Social_graph
Source 2: http://www.sones.com
Source 3: Long documentation
Source 4: Short documentation

I am proud to anounce that there’s a video publicly available which shows parts and projects Microsoft Research is working on currently. It’s great to see theses projects, concepts and ideas become publicly available one by one:

“Craig Mundie, chief research and strategy officer of Microsoft, presents “Rethinking Computing,” a look a how software and information technology can help solve the most pressing global challenges we face today. Part of UW’s Computer Science and Engineering’s Distinguished Lecture Series, Mundie demonstrates a number of current and future-looking technologies that show how computer science is changing scientific exploration and discovery in exciting ways. He discusses the role of new science in solving the global energy crisis, and answer questions from the audience.”

Source: http://www.uwtv.org/programs/displayevent.aspx?rID=30363&fID=6021

If you – like me – want to migrate from dasBlog to WordPress by using the great BlogML Export and Import you might want to take this advice:

After exporting the BlogML you should replace all “ ”s in the XML file. If you don’t do that WordPress won’t import anything in the article after the “ "s.

So here we are on a new blog engine. It took me the better part of two days to do the Migration of 2,869 posts and 2,732 comments, a lot of pictures and movie files.

I will write an article on this but for now only two captures images from the migration:

yeah PHP rocks!

 
had to do some regex action to do the url rewrites

I ran into some strange problems with a notebook that leaded to sound drop-outs or things like sluggish UI and HDD performance. So I tried almost everything troubleshootig the problem. That worked for some problems but there are occasions when I want to have a more systematic approach to those kinds of hardware / driver related problems.

One tool that can help to find hardware / driver problems is the DPC Latency Checker. This tool measures and displays the latency of your system. All you have to do is watch as the measurements scroll by and remove / disable one device after another from your machine. As soon as the latency turns green again there’s a high probability that the device you removed last has a problem of some kind.

On my machine everything is in the greens now – after some BIOS and driver updates. If your system has some issues you would see something like that:

 
(courtesy of Gnawgnu’s Realm)

Source 1: http://www.thesycon.de/deu/latency_check.shtml
Source 2: http://gnawgnu.blogspot.com/2009/01/dell-latitude-e6400-sound-problem-fixed.html

Today we had a great meeting with SciEngines. These guys offer a great platform for everything that needs massive parallelism and IO bandwidth scalability. They even brought a small copacobana cluster to our headquater.

Source: http://www.sciengines.com

It’s great to finally have the .NET sourcecode for debugging purposes – inconveniently it’s in a format you might have your difficulties just browsing along. A little tool is here to help!

After you installed, let’s say the WCF sourcecode and debug symbols you get a directory structure similar to this:

This source.zip.tmp file holds the whole sourcecode as one big package. It can’t be unpacked – even one would suggest that by just looking at that .zip ending in the name of the file.

Instead this is a plain-text file of a certain yet simple format. I wrote me a little tool to unpack this file into it’s original files and directories.

You can get the little tool, including sourcecode, here: UnpackMSSources.zip

To start the magic, you would like to go to the command line and start the tool with two parameters. Parameter 1 is the path and filename of the source.zip.tmp file. Parameter 2 is the part of the Path that needs to be cut-off. For the WCF Sources it’s “/DEVDIV/depot/DevDiv/releases/Orcas/SP/ndp/cdf/src/” for example.

The tool will then start to whirl through the file and extract all the files it founds into directories it’s creating along the way. After some seconds you would end with a directory tree like this:

Have fun!

Source 1: http://referencesource.microsoft.com/netframework.aspx
Source 2: http://www.schrankmonster.de/content/binary/UnpackMSSources.zip

“”Being a geek means being so interested in something that you don’t care whether or not it’s cool.”

THE SOCIETY FOR GEEK ADVANCEMENT was founded upon the principles that we should all embrace our inner and outer geek and have fun while doing it. As individuals who love learning, innovating and believe in possibility as well as change, the second step of responsibility is to “be the geek that keeps on giving”. As a member of SGA, we work together as a global community to provide the tools and help others realize their true potential too!”