smallroomsoftware.com

Finally, I have Most Blue Skies running reasonably well on one of our servers. Here's the "live" world view:

... with some including current "bluest" sky here.... At the moment, the server struggles to run enough sky calculations to keep the simulation up-to-date. I think it runs about 10 per second. That's on a dual 2Ghz G5 Xserve. We currently have 500,000 places in the database to test which will take about 14 hours to run through. Lets say half of those are in the dark and half or the remainder are under cloud. That's still several hours worth of calculations. So I need to either learn Fortran and improve the SMARTS software for batch processing (not that likely), buy more servers or code some kind of Most Blue Skies @ Home system where the public or sponsors can donate some processing power to the project. Maybe I can find a Fortran to Java translator. That might be a way forward...

With literally minutes to spare we installed most blue skies. Josh heroically persevered with our light and it now seems to work well. The Koreans really know how to celebrate and art show! Opening events for the show were quite astonishing, involving ambassadors, riot police, prime ministers, secret agents, helicopters, blimps, rock a big concerts, several pop stars.

I'm now in Japan. I caught the 'beetle' high speed ferry from Busan and the shinkansen up to Kyoto. Today I go to Tokyo!

Most Blue Skies is progressing nicely! After an incredibly frustrating week or so battling with odd data formats we had a few breakthroughs and we finally have our 'core' data downloading and being used in the sky colour calculations. Josh has also checked in some nice code to convert the spectrum that SMARTS gives us to a RGB value. I hooked it all together today and it looks great!

For those of you who are interested in using MODIS Level 2 data with GIS applications (and can hopefully find this page via Google), I can probably offer some advice. As you'll know most GIS software doesn't support the HDF format that the MODIS data comes in so you need to find some tools to convert it into something you can work with. Finding the right tool, however, is a real pain. Also MODIS use a slightly extended version of HDF (just to make things even more difficult) so you need to find tools that are specially written to support MODIS data. The Level 1 and 2 products store data in 'swath' format. This basically means raw instrument readings each with a longitude and latitude coordinate attached. You need to therefore reproject and interpolate this data into some kind of grid in some projection that your GIS software/libraries can handle. Don't bother trying to use HEG to produce GeoTIFFs unless you are using certain data sets from certain products that don't have fill values and meet a bunch of idea criteria which I can't remember. The problem with HEG is that it doesn't use nearest neighbour interpolation when reprojecting the MODIS 'swath' to a geographically projected grid (even though you might tell it to). So this messes up scientific data sets where the values shouldn't be interpolated, e.g. bitfields (e.g. cloud mask data). On top of that, HEG doesn't honour the fill values and interpolates between these fill values and the actual data so you get even more bad data coming out.

In the end we HDFLook to convert each swath to a gridded linear projection in a new HDF file. It seems to do a very good job (although I'm having some problems with swaths that cross the international date line, not sure why). HDFLook can also produce a GeoTIFF but it seems to do a worse job than HEG at choosing the right data type for the TIFF data (you can store floats in a TIFF apparently). It always produces an RGB tiff which isn't very useful if your SDS uses floats. So I just produce a gridded HDF file with HDFLook. Then I've been using the Java HDF API to read the data directly from the new HDF file and build a coverage with GeoTools. I wanted to avoid calling any JNI code from the main app so I wrote a separate simple program to read the HDF files and spit out the values. Ask me if you want the Java code I used to do this (it's not particularly pretty).

Okay, so this what I think the sky looks like in Afghanistan today! (See previous post for what the hell I'm going on about...)

So it looks good. Some things are working.

Hugo sent me this link yesterday. Nice dancing! Made me want to pack up and just go somewhere. Hmm, well, maybe soon.

So in the last week or two I switched from working on "Departure Lounge" over to working on "Most Blue Skies". This is an art project that will be exhibited at the Busan Biennale 2006 in South Korea. This is mostly Lise's project but Josh is providing a lot of help, saving me from prolonged frustration on numerous occasions.

Although smaller in scale, the project shares a lot of similarities with Black Shoals (which was just a lot of fun and is one of my fondest memories). Both projects involve the processing of large amounts of data and turning it into some visual form. In this case we're processing atmospheric data in order to show the visitor where in the world they would have to go if they wanted to see the bluest sky at that very moment. We will also show what shade of blue they would see! The colour will be projected in some way onto a big square in the middle of the room and if you go behind the square you will see all the amazing technology that goes into producing that single lovely colour. As you might expect, calculating the colour of the sky in real-time over the whole of the world is turning out to be pretty tricky. We had to find algorithms for simulating radiation from the sun given some atmospheric parameters (took a while) and then try to find data sources for those parameters and try to get data that is as up to date as possible. Of course you have to grapple with mismatched units, satellites (or shuttles!) not flying where you want them to fly, millions of different co-ordinate system, file formats and access methods and also people insisting of writing their code in Fortran and still thinking in terms of punch cards (seriously!).

The other day we went to talk to the Atmospheric Physics department at Imperial about Blue Skies (but mostly we talked about Black Shoals because that actually exists ;-). The place seemed deserted when we arrived and we figured no-one would come but then everyone appeared (I think they thought it would be cooler that in their offices (it was a hot day)). Unfortunately it wasn't, but by that time they were trapped and could not escape from our gripping presentation! Anyway, a few people seemed to get quite into it and we got some good suggestions. Also happened to meet the cousin of someone Markus knows who works in bio-informatics. Small world.

We now have just four weeks to finish (I'm being optimistic and not counting a week of mad overnight in-gallery hacking in Korea). Pretty scary really! So I'd better get back to my GeoTiffs and aerosol optical depths...

Today Josh, Lise and I went the Met Office in Exeter to see if we could find some scientists willing to help up with Most Blue Skies project. It was nice to visit the west-country. I really think people get friendlier as you go West.

We also saw a sign which told us that the BIKINI state of the Met Office is...

There is an increased likelihood of an attack, but no defined target. It can also mean "potential terrorist threat".

So there you go. Today I was just two BIKINI steps away from all out war!