smallroomsoftware.com

Might save someone an hour or so. Just a simple Java class to invoke the command-line xgrid client on OS X. You can add files to send with the job. There's a pure java xgridclient library but it seems to leak a thread every time a job is submitted (somewhere in the BEEP implementation). I emailed the author but got no reply so I wrote this simple class to invoke Apple's client.

Usage example:

XGridJob job = new XGridJob("host.domain",
                            "script.sh");
job.addFile(new File("script.sh"));
job.submit();
while (!job.resultsAvailable()) {
    Thread.sleep(500);
}
String results = job.getResults();
job.dispose();

Download the class...

Most Blue Skies is progressing nicely! After an incredibly frustrating week or so battling with odd data formats we had a few breakthroughs and we finally have our 'core' data downloading and being used in the sky colour calculations. Josh has also checked in some nice code to convert the spectrum that SMARTS gives us to a RGB value. I hooked it all together today and it looks great!

For those of you who are interested in using MODIS Level 2 data with GIS applications (and can hopefully find this page via Google), I can probably offer some advice. As you'll know most GIS software doesn't support the HDF format that the MODIS data comes in so you need to find some tools to convert it into something you can work with. Finding the right tool, however, is a real pain. Also MODIS use a slightly extended version of HDF (just to make things even more difficult) so you need to find tools that are specially written to support MODIS data. The Level 1 and 2 products store data in 'swath' format. This basically means raw instrument readings each with a longitude and latitude coordinate attached. You need to therefore reproject and interpolate this data into some kind of grid in some projection that your GIS software/libraries can handle. Don't bother trying to use HEG to produce GeoTIFFs unless you are using certain data sets from certain products that don't have fill values and meet a bunch of idea criteria which I can't remember. The problem with HEG is that it doesn't use nearest neighbour interpolation when reprojecting the MODIS 'swath' to a geographically projected grid (even though you might tell it to). So this messes up scientific data sets where the values shouldn't be interpolated, e.g. bitfields (e.g. cloud mask data). On top of that, HEG doesn't honour the fill values and interpolates between these fill values and the actual data so you get even more bad data coming out.

In the end we HDFLook to convert each swath to a gridded linear projection in a new HDF file. It seems to do a very good job (although I'm having some problems with swaths that cross the international date line, not sure why). HDFLook can also produce a GeoTIFF but it seems to do a worse job than HEG at choosing the right data type for the TIFF data (you can store floats in a TIFF apparently). It always produces an RGB tiff which isn't very useful if your SDS uses floats. So I just produce a gridded HDF file with HDFLook. Then I've been using the Java HDF API to read the data directly from the new HDF file and build a coverage with GeoTools. I wanted to avoid calling any JNI code from the main app so I wrote a separate simple program to read the HDF files and spit out the values. Ask me if you want the Java code I used to do this (it's not particularly pretty).

Okay, so this what I think the sky looks like in Afghanistan today! (See previous post for what the hell I'm going on about...)

So it looks good. Some things are working.

Hugo sent me this link yesterday. Nice dancing! Made me want to pack up and just go somewhere. Hmm, well, maybe soon.

So in the last week or two I switched from working on "Departure Lounge" over to working on "Most Blue Skies". This is an art project that will be exhibited at the Busan Biennale 2006 in South Korea. This is mostly Lise's project but Josh is providing a lot of help, saving me from prolonged frustration on numerous occasions.

Although smaller in scale, the project shares a lot of similarities with Black Shoals (which was just a lot of fun and is one of my fondest memories). Both projects involve the processing of large amounts of data and turning it into some visual form. In this case we're processing atmospheric data in order to show the visitor where in the world they would have to go if they wanted to see the bluest sky at that very moment. We will also show what shade of blue they would see! The colour will be projected in some way onto a big square in the middle of the room and if you go behind the square you will see all the amazing technology that goes into producing that single lovely colour. As you might expect, calculating the colour of the sky in real-time over the whole of the world is turning out to be pretty tricky. We had to find algorithms for simulating radiation from the sun given some atmospheric parameters (took a while) and then try to find data sources for those parameters and try to get data that is as up to date as possible. Of course you have to grapple with mismatched units, satellites (or shuttles!) not flying where you want them to fly, millions of different co-ordinate system, file formats and access methods and also people insisting of writing their code in Fortran and still thinking in terms of punch cards (seriously!).

The other day we went to talk to the Atmospheric Physics department at Imperial about Blue Skies (but mostly we talked about Black Shoals because that actually exists ;-). The place seemed deserted when we arrived and we figured no-one would come but then everyone appeared (I think they thought it would be cooler that in their offices (it was a hot day)). Unfortunately it wasn't, but by that time they were trapped and could not escape from our gripping presentation! Anyway, a few people seemed to get quite into it and we got some good suggestions. Also happened to meet the cousin of someone Markus knows who works in bio-informatics. Small world.

We now have just four weeks to finish (I'm being optimistic and not counting a week of mad overnight in-gallery hacking in Korea). Pretty scary really! So I'd better get back to my GeoTiffs and aerosol optical depths...