Last Friday I gave a talk on how we use HBase at Mendeley, which goes into more detail on the work I’ve been doing over the last year. It’s a summary of how the datamining team started out using MySQL and how and why we moved to HBase for most of our data storage and processing. It also describes some of the work we do in the datamining team.
You can find more details of the talk and a video of the presentation here http://lanyrd.com/2010/huguk7/sxbt/
The afternoon generally went very well, I met many intresting people in London using Hadoop and HBase. There were also some people from California stopping over in London from a conference in Brussels. So it was great to meet people like Jonathon Gray who is one of the HBase committers working at Facebook and learning how they are using it in their new messaging service, Stack who is another HBase commiter from StumbleUpon, and Tom White who started the whole Hadoop world off. It’s a great community and an intresting time to be part of it.
For more from this event you can see all the slides here http://lanyrd.com/2010/huguk7/ and learn more about the UK Hadoop users group I’ll now be runnning http://huguk.org/
Published June 21, 2010
It’s been a long time since my last post, I’ve mostly been busy down in London working at Mendeley and it’s gone very quickly. This has recently lead me to the Science Hack Day last weekend where I met some great people, learnt many things and hack on some YQL tables for Mendeley’s new api for their research data.
YQL is quite a cool piece of technology which basically lets you treat the web as a big SQL table, so you can effectivly join twitter to a google search, then grab some usage statistics from the Mendeley api for example.
I’ve put the Mendeley YQL tables up online here to play with in the YQL console here. With this you can do queries like :-
SELECT * FROM mendeley.search WHERE query = “information retrieval”
And then start joining our different api calls together like this :-
SELECT title, year, stats FROM mendeley.details WHERE id IN (select id from mendeley.tags(50) where tag = “genetics”) | sort (field=”year”, descending=”true”)
I’m trying to get them in the community tables in github soon so that they can be more easily used by everyone, and I don’t have to keep them up on dropbox (even how amazing that is for rapid prototyping of YQL tables).
We’ve just got this Samsung LCD TV at home and I’ve been seeing what it can do so I’ll give it a quick review here. Visually it looks very clear and sharp, even from some SD sources, so it’s quite impressive. It also comes with the HTZ310 Home Theatre System which is a up-scaling DVD player and 5.1 surround system in-one. The player looks great with DVDs and can also play mpeg-4 from discs or usb sticks (but not usb drives…). The included speakers sound really good and it supports both Dolby Digital and DTS signals.
The inputs on the TV consist of 4 HDMI, 1 Component and 2 Scart, although only one of the scarts are RGB the other is just compositive. The TV has optical and phono outputs for sound but the optical output only sends a stereo signal and not Dolby Digital or DTS that can come in via the HDMI ports. This is quite annoying as an advantage of HDMI is that it can carry both video and sound but this TV won’t output the sound part of the signal. This means if you have more than one source of digital sound, Cable + Console, then you need another device to switch between them. As it comes with the Home Theatre System that only has one optical input this would need to be a optical switch, though if you go another amp that supported pass through video you could use that to switch both the video and audio instead.
The TV also has DLNA support so it can connect to media servers on the network and play music, pictures and videos from them. We did this before using XBMC which meant that it had quite a lot to live up to in terms of UI and format support. Unfortunately the format support is quite limited, partly due to the DLNA protocol, so I got SD videos and pictures to work well but HD videos and music were far more complicated. As there was not much to be done to improve this we’ve decided to create a HTPC using an old PC to play media back. I’m thinking of using a HD 2600 Pro agp card with HDMI output to do AVC and VC-1 decoding and the LC12B case from SilverStone. Then on top of this we can use the windows version of XBMC which seems to be moving very fast and still has the best UI, although Windows Media Centre 7 is also tempting. I’ll post about how that goes once it’s up and running.