I happened to be online when The CRU email hack story broke and immediately downloaded a copy of the zip-file from a Russian server, all 63mb of it. Part of Friday was spent poking through the Mail folder, finding tantalising snippets, and being frustrated by the lack of context that made it difficult to interpret some of them.Then I had to go away and it was difficult to keep in touch with the developing story, so on Monday I started a long catch-up process that is hardly completed now.

The following are just a few notes. For a anyone who is still unaware of what is in these emails which have become so controversial, there is no better place to look than on Bishop Hill’s excellent blog here .

When unzipped, the file which mysteriously turned up in Russia amounts to nearly 170mb. Inside FOI2009.zip is a folder named FOIA containing two subdirectories: Mail and Documents.  Freedom of information seemed to have been very much on someones mind, either the person who named the main download file or Phil Jones of CRU who named the main folder within it or perhaps both. If Jones did give the main folder its name then I wonder why he call it FOIA (Freedom of Informaton Act) when a great deal of the contents does not refer to this piece of legislation? Certainly there is much material here that it would have been most embarrassing to have to release under the Act. Is it possible that Jones kept sensitive – perhaps even incriminating  – emails carefully in one place so that deleting a single folder would purge his computer of potentially dangerous material if danger loomed? This seems unlikely, as he would be aware that backups are likely to exist elsewhere beyond his reach.

The sizes of the two sub-folders within the FOIA folder are 9.7mb and 158mb.  The one called  ‘Mail’ contains 1073 files, but this does not represent the number of emails. Many contain strings of emails received and sent as well as copies of emails that are being referred to. So there are many more than 1073 emails in total.

I do not know how many words are involved, but assuming an average of a hundred words per email – and most are longer than that – that makes over a hundred thousand words. This is a vast amount of verbiage and most of it is ‘insider-speak’; exchanges between fellow professionals who know each other and the subject matter that is being discussed very well. Much of it is likely to be opaque to even a well-informed outsider. The task of making a detailed analysis of just the material in the Mail  folder is enormous and, even if this is split up between various people with the necessary expertise, it is likely to take weeks or months. In the meantime, what is emerging at the moment are a few obviously sensational quotes. Only a systematic analysis will fully reveal what is there.

The Documents folder has eleven sub-folders, most of them hierarchical, and the largest one, named Code, is over 30mb. That is an enormous amount of code. Already there are estimates on the net that suggest that it could run to millions of lines. Continue reading »