Ever since I nervously downloaded a file named FOI2009.zip from a rather dodgy sounding Russian server one November night last year, there has been a nagging question troubling me. When the terms of reference for the Russell Inquiry were announce it seemed certain that, whatever shortcomings this so-called independent inquiry might have, an answer to that question must be forthcoming,
Examine the hacked e-mail exchanges, other relevant e-mail exchanges and any other information held at CRU …..
Review CRU‘s compliance or otherwise with the University‘s policies and practices regarding requests under the Freedom of Information Act (?the FoIA‘) and the Environmental Information Regulations (?the EIR‘) for the release of data.
The FOI2009.zip file contained the Climategate files, and the question was, where did it come from? I did not, of course, expect that the Russell Inquiry would reveal the precise circumstances of the leak or hack unless the Police investigation was over. However determining whether a folder with the name FOI2009, or one with the same contents as the file on the Russian server, existed on a University of East Anglia computer must surely be one of the main stepping-stones to unravelling just what the great scientific scandal known as Climategate was all about. If it did exist on a CRU server, then there would be many questions to ask about how and why it was created. No credible investigation of the university’s handing of FOI requests could be carried out otherwise.
From what I can see in the Russell Report, that mystery still persists.
The FOI2009.zip file contains two folders, Mail and Documents, the former containng the notorious emails and the latter consisting of a great deal of research material. As anyone who has made up a .zip file will know, most applications that do this assign a default name to the new file that is the same as the file or folder that is being zipped, but with the .zip file extension. So it would seem likely that the FOI2009.zip file may share it’s name with its parent folder.
Here is what Chapter 4 of the Russell Report, Context of the E-Mails, has to say about the content of the Mail folder in FOI2009, but without any mention of the filename:
2. The information released comprises a very small (less than 0.3%) subset of files which were held on the back-up server at CRU, which include e-mails and other documents – such as text files, Word documents, Excel spreadsheets, PDF documents, and computer code.
3. The focus of this Chapter is e-mails which spanned the period 7th March 1996 to 12th November 2009. The ‘primary’ e-mails number 1073 in total with 166 authors. There are more e-mails and authors if the associated e-mail chains are included. When printed on A4 paper the e-mails run to 3,375 pages and contain many embedded duplicates. Self-evidently each of the primary e-mails was either sent by or received by CRU members, but this is not the case for many of the associated e-mail chains.
And:
1. Recognising that the e-mails improperly released into the public domain represent only a tiny fraction (less than 0.3%) of the e-mails archived by the key individuals in the CRU, the Review team sought to set these in context. The backup server (CRUBACK3) had been taken as evidence by the police as part of their own investigation and was held by police contracted forensic investigators. A full context could only be established by some form of access to the information held on this server. In seeking to gain this access a number of legal issues arose, notably that: [my emphasis]
There can be no doubt that the amount of data that confronted the Russell panel when they began their work was daunting. According to the information available in this section of the report, (and in a report form an ‘independent’ consultant on the inquiry website; see below) there must have been getting on for half a million email files on the server. Anyone who has tried to cope with the 1073 files that were released may feel some sympathy for them, but that’s the way the cookie crumbled and the TOR of the inquiry make it quite clear that they were to investigate whatever information was available on the backup server.
Systems analysts and networking people commenting on blogs seem to think that it is unlikely that a hacker or whistle blower could have spent sufficient time logged on to the UEA server to extract the contents of the mail folder from the vast quantity of files available there without being detected. It seems very much more likely then, that someone quickly snatched a folder, or folders, that looked as if their contents might be compromising. That folder may have had the name FOI2009, or it is possible that two folders were found that had the same content as the sub-folders in FOI2009.zip on the Russian server.
It is reasonable to suppose that an inquiry with the TOR that SIr Muir Russell was given would be able to answer this question. However monumental a detailed examination of the content of all the emails on the backup server might be, discovering whether FOI2009, or its constituent sub-folders, were present on the server – and just there for the taking – would seem fairly straightforward.
Steve McIntyre has an excellent post at Climate Audit entitled, The Botched Examination of the Back-Up Server This is concerned with the failure of the Russell Inquiry to make a proper search of the backup server that the emails came from, but in a rather different context to what I am discussing here. Nevertheless his analysis of what happened is very useful. It points to a certain lack of rigour, or even vigour, on the part of the review panel.
In Chapter 4 of the report, Context of the Emails, we find that:
10. The presumption is that e-mails were selected to support a particular viewpoint. Recognising that they were a tiny fraction of those archived, the Review Team sought to learn more about the full contents of the back-up server. This attempt, summarised in Appendix 6, was largely unsuccessful due to the sheer scale of the task and ongoing police investigation.
To say that this exercise was ‘largely unsuccessful’ exhibits a level of understatement that is breathtaking. The evidence in the Russell Report, and in the ‘independent’ analyst’s report, makes it clear that they failed to learn anything at all from the backup server because they didn’t even find out what was on it, let alone attempt any analysis.
There is no doubt that searching CRUBACK3 posed problems for the inquiry panel, and that these extended beyond the mere volume of data. The server and its contents were evidence in an ongoing police investigation and in the possession of Norfolk Police (Appendix 6, para 1), but as Steve M points out in his post, the Russell Inquiry seem to have been in no hurry to negotiate means of access to the data with the police that would not prejudice a possible prosecution.
The delay does not seem to have been the fault of the police. Although Sir Muir Russell was appointed, and his terms of reference set out, on 3rd December 2009, and the first meeting of his panel was in early January, no decision was taken to contact the police and get to grips with the material on the server until 30th March, four months later. The target date for publishing the report was ‘Spring 2010′.
The Russell Inquiry’s terms of reference (see above) make it absolutely clear that examining information on the server was one of the primary tasks with which they were charged. And so was considering the way in which the university had dealt with FOI requests. As we will see, the origins of FOI2009 has an important bearing on this.
Eventually, it was agreed that the police would arrange for the data to be extracted from the server and that it would then be examined by a specialist forensic computer analyst in secure premises. This caution when dealing with evidence in a criminal inquiry is quite understandable, but what happened next was less so.
Instead of the Russell Inquiry appointing and commissioning an analyst, this was done by the UEA, the institution that was being investigated. The universtiy’s choice was Professor Peter Sommer who has connections with the London School of Economics and the Open University. I have no reason to suppose that Professor Sommer is anything other than a first rate expert in his field and a person of unimpeachable integrity. On the other hand, it is rather strange that an ‘independent’ inquiry panel, that might expect their modus operandi exposed to very critical scrutiny by sceptics, would allow the organisation under investigation to appoint a consultant to carry out such a vital task, and that the said consultant should be an academic when it is academics who were being investigated. But then things get even stranger.
On 26th March, prior to the decision to contact the police, Sir Muir Russell and another member of the panel had a meeting with Professor Edward Acton, the Vice Chancellor of the university. This is what the minutes say:
5. We explained that we were considering whether it would be practicable to have some work done to search the CRU server (held by the police) with a view to obtaining a fuller understanding of the basis of the selection of the e-mails that were the subject of the unauthorised release. Professor Acton was strongly supportive. He was concerned that due attention be given to the fact that this could raise Data Protection issues because more personal data, possibly relating to more people, might be involved; and the sheer practical burden of handling the possible outputs of the work had to be considered. We agreed to reflect on those concerns and return to the matter, possibly with a more focused proposition, when Jim Norton and I made our next visit the following week.
The Russell Report gives a little more information about the legal concerns:
In the opinion of UEA‘s legal advisers, unconstrained access to the contents of e-mails on the server by the Review would raise potential privacy and data protection issues.
This flags up another question. The inquiry panel would seem to have felt the need to approach the Vice Chancellor before searching the server although it was in their terms of reference, set out by the university, to do so. Why would they do this? Evidently Acton’s response was laudable enthusiasm swifty followed by dumping a couple of obstacles in the way. Note that the concerns about data protection were not raised by the review panel, but by the university of East Anglia’s solicitors. I’ll come back to this later.
Turning to the analyst’s report mentioned in the quotation from the Russell Report (see above) this makes interesting reading. Here is the first sentence:
In November 2009 a number of emails relating to the work of the Climatic Research Unit (CRU) at the University of East Anglia (UEA) appeared on various websites and were subjected to hostile interpretation on various “blog” sites and in the mainstream media.
Am I alone in thinking that there is something strange in an ‘independent’ forensics expert, who has been brought in solely for the purpose of ferreting out data on a backup server, should feel the need characterise the initial reaction to the Climategate emails as ‘hostile interpretation’, or to enclose the word blog in quotes as thought it was emitting a bad smell? Sorry, that’s a bit of a digression, but it did rather catch my eye.
Here are a few more extracts from Professor Sommer’s report to the panel:
I have at this stage no knowledge of the technical means by which the emails were acquired from the CRU.
…..
I have been supplied by the University with a “thumb drive” said to contain copies of all the emails known to have been published on the websites. I have also been supplied by Norfolk Police with three further “thumb drives” containing the emails extracted from the back-up server associated with the computers of the researchers in question. The extraction was carried out by Qinetiq, as contractors to Norfolk Police.
…..
The material has been given a very high level of security classification which requires that I work at secure facilities and follow particular protocols which, for example, preclude computers being left to run unattended or overnight and at weekends. These procedures, while providing a very high standard of protection to the data, are also very time consuming, particularly in the light of the need for the Review to conclude its work in a timely manner.
…..
The emails as provided to me are in the format of an email program called “Thunderbird” and before they can be searched require indexing. There are large numbers of un-indexed emails and time constraints in preparing this initial report preclude indexing and any form of sophisticated analysis.
…..
I am told there are some 1,073 primary emails and their associated threads in the published material, which itself amounts to 18.7Mb.[1]
From this it appears that Sommer had not been briefed about how the hack or leak was carried out, nor had he formed any opinion on his own initiative. Perhaps this is not surprising in view of the ongoing police investigation. On the other hand his description of the version of the ‘hacked’ emails that was provided on a ‘thumb drive’ by UEA does seem rather cautious, particularly as it is not clear whether the data was in it’s original form as leaked on the net, which was of course a sub-directory of FOI2009.zip. Surely he would want to see this file? The last extract from Somer’s report makes it clear that the university did not give him a copy of FOI2009.zip as, if they had, it would not have been necessary to have tell him that there were 1073 email files; he could have done a file count himself in less than a second.
The working conditions imposed by the police were certainly inconvenient, but understandable. It would have been irresponsible of them to take any risk that might compromise what is apparently a major inquiry that has involved help from specialists in other forces. Indexing and searching large volumes of data – and other parts of Sommer’s report make it clear they are large – is time-consuming even on a fast computer. Not being able to leave machines running unattended while this takes place was certainly a problem in terms of manpower, but by no means an insuperable one if there was a will to solve it.
What really stands out in the analyst’s report is that the data was not extracted by Sommer, but by Qinetic, on someones instructions, but we are not told who. So what were those instructions, and did anyone ask Qinetiq to keep an eye open for a folder with a name like FOI2009, or a couple of folders that contained the same data as the Mail and Data sub-folders in FOI2009.zip of dubious Russian server fame? Neither the Russell report nor Sommer’s report seem to give a clue.
Here is how the ‘independent analyst’ concludes his report:
The processes of analysis to identify (and then review) additional email traffic which might be associated with the issues which are the subject of the allegations which have been levelled against CRU, is likely to take at least several weeks. It would be for the Review Team and the University to determine whether the cost, inevitable time delays and (at this time) uncertain outcomes could be justified. Until the material is subjected to a much downgraded security level, the likely position will be that the University and its appointed team will not be able to carry out any meaningful analysis.
And let’s take another look at what the Russell report has to say:
10. The presumption is that e-mails were selected to support a particular viewpoint. Recognising that they were a tiny fraction of those archived, the Review Team sought to learn more about the full contents of the back-up server. This attempt, summarised in Appendix 6, was largely unsuccessful due to the sheer scale of the task and ongoing police investigation.
So the Russell Inquiry decided to call it a day, in spite of examination of the data on CRUBACK3 being one of the primary requirements in their terms of reference. Evidently it would just take to long and be too much trouble. And then there is the terrible problem of the UEA’s legal adviser’s concerns about data protection legislation. Did the review panel get independent legal advice on this from a source free from the very obvious conflict of interest that the university lawyers would have? There is nothing in the report to suggest that they did.
So the Russell report throws no light whatever on where FOI2009 came from and that matters a very great deal because of the obligation that the TOR put them under to consider the university’s handling of FOI requests. If that folder, or any of its contents was present on the backup server when the hack or leak took place, then it obviously means that the content was compiled by someone at the university. Why would anyone make up such folders containing apparently embarrassing material? Surely that is something that an inquiry charged with reporting on the university’s handling of FOI requests would want to know.
Looking at the problem from another standpoint, if the Russell Inquiry was, as some sceptics suspect, an exercise in clearing the CRU’s name, then if they had established that the information that was leaked or hacked had not already been gathered together on the server, then surely that would be something worth mentioning?
Perhaps the police told the inquiry panel that even to mention FOI2009 would prejudice their investigations, but there is no caveat in the report to the effect that there were aspects of their inquiry that they were unable to mention for legal reasons. There is no such caveat.
So the mystery of FOI2009 not only remains, but it has been extended and deepened. Not only do we not know whether this folder, or its sub-folders, were lifted complete from CRUBACK3, but there is the added conundrum of why the Russell Inquiry seems to have had no great interest in finding out.
One last thought. While the police inquiry continues, and they have possession of CRUBACK3, the integrity of the data on it is beyond dispute and available for inspection. What happens when the inquiry ends?
[1] I’m no propellerhead, but the Mail folder from FOI2009 takes up less than half that space on my HDD. I realise that this may just be the difference between the files being in .txt format rather and the original Thunderbird format, but I would be interested in any views on this.
In his book The Climate Files, Fred Pearce speculates about FOI2009; he writes that the material from the CRU backup server was copied, according to an analysis by Charles Arthur, the Guardian’s technology editor, over a number of weeks – he mentions three dates in 2009: 30th September, 10th October and 16th November. Pearce thinks it is still unclear as to whether this was after a security breach (i.e., infiltration by an outsider) or whether an insider had left the material (inadvertently or deliberately) in a location where it could have been accessed and copied by an outsider.
This sentence is interesting: “It may be relevant that all the copying apparently took place in the final weeks of consideration by the university’s information officer of McIntyre’s request for the CRU’s crown jewels, and that the last addition to the file of leaked material was made the day before the decision letter was written.” Pearce also mentions a theory advanced by Charles Rotter (Charles the Moderator, at WUWT) that the files may have been left carelessly on a FTP server and this “can lead to unintentional sharing with the rest of the internet.”
Maybe this comes close to the truth of the matter; perhaps material related to FOI requests was gathered together, put into a directory (called “FOI2009”) and inadvertently stored where it could be accessed by an outsider. Someone on the outside with an interest in climate matters, and in what could be gleaned from the UEA’s web pages, was poking around (“hacking” would be too extreme a term for this) and happened on the directory – material labelled “FOI2009” would have been of obvious interest to this person. And the rest, as they say, is history.
Pearce writes that “if his [Charles the Moderator’s] theory is true, then the university will be left looking rather foolish. There will be no-one to arrest.” Given the likely embarrassment for the UEA if this turns out to be an own goal, as it were (even if there was no “mole”), and the closeness with which the Muir Russell people worked with the UEA administrators, could it account for the lack of probing into this matter?
Alex:
Thanks for that. I’ve now ordered a copy of Pearce’s book.
Your précis makes it even more unacceptable that the review panel did not examine the backup server. Why did the Russell panel not follow the same trail as the Guardian?
If the content in FOI2009 was compiled at UEA, under that name or any other, then explanations are needed. And if there is evidence suggesting that the content was compiled in instalments over several months, is there any reason to believe that the process was complete at the time that Climategate broke? How much more similar material may be present on the server?
Russell, Acton and Davies really do have to answer questions about this and it is likely to be left to the sceptical blogosphere to ensure that this happens. There are a few atraws in the wind that will be the subject of a post shortly.
Incidentally, according to information obtained from Norfolk Police by Bishop Hill their investigation has cost nearly £55,000 so far. Where would that leave UEA if the leak resulted from their own carelessness rather than an offence being committed?
In view of the uncertainty surrounding just exactly how and by whom the leaked emails were accessed in the first place, it would be unwise to either attribute this to an outside “hacker” or an inside “whistle-blower”.
The best is simply to refer to them as “leaked emails”, until this is cleared up (if it ever is).
Max
Hi TonyN, from the press briefing on 11 Feb 2010 it seems clear to me that the panel were going to read through ‘tens of thousands’ of emails in accordance with point 1 of the Overview
At the end of the briefing 2 interesting points are made by Muir Russel
– the private character of the (undisclosed)emails is not relevant because the questions have become public questions.
– The UEA gave The Independent Climate Change Email Review permission to use any emails it wanted.
http://www.cce-review.org/
(Feb briefing: bottom link)
harold:
The recording of the press conference makes entertaining listening in the light of all that has happened since. Never can so many good intentions have been abandoned in the course of producing an ‘independent’ review.
Russell is also very clear, in response to a couple of rather good questions from Roger Harrabin of the BBC, that although UEA will fund the inquiry they will have no input whatsoever. So it’s a bit funny that Russell felt the need to go and see Acton before even considering examining CRUBACK3 and the university took it upon themselves to hire the independent consultant who decided that analysing the emails on the backup server would be far too difficult.
It is also clear that, at the time of the press conference, Russell saw no time constraints on when he reported, merely saying that he hoped that there would be ‘preliminary conclusions’ by the spring and he had no idea how long it would all take; that depended on how much work they needed to do. I wonder when things changed and who changed them?
Professor Sommer’s use of quotation marks around the terms ‘thumb drive’ and ‘Thunderbird’ suggests both lack of familiarity and a certain discomfiture, as thought they’re not the sort of things he would like his wife or servants to come into contact with.
I’m not a regular Thunderbird user, but I’m pretty sure it indexes files automatically – it is the professor’s reluctance to sully his hands with such a program that is the obstacle, but if that’s his attitude, why take the job?
I don’t suppose he even does his own typing.
Does anyone other than the police know whether the e-mails in the file FOIA2009 are still present on the computers of the staff of CRU? i.e. were they copied or were they removed? Has anyone asked this question?
Geoff:
I know that David Holland has had an FOI request for other information about the emails turned down by UEA because they say the the Police are holding the server as evidence and they do not have access to it. I assume that any request concerning FOI2009 would meet with the same response.
This is a bundle of mysteries which are only likely to be solved if the HoC Select Committee are prepared to grill Acton, Davies, Russell and possibly Sommer. I seem to remember that being less than cooperative with a select committee can lead to tears before bedtime.
Do you have any ideas?
Just to clarify my #7: if the e-mails are still in place, (on the computers of the correspondents, not on the back up server) wouldn’t that eliminate the possibility that they had been assembled by an insider with the aim of swiftly deleting them, if necessary? (or is there some clever programme that would instruct the computer to eliminate them? But in that case, it surely wouldn’t be necessary to assemble them in a file).
If they are still there, that would suggest, though not prove, that they had been secretly assembled by a “hacker”. This question could be simply answered by consulting the computer of any one of the CRU “team”. I don’t see how that would prejudice the police enquiry. The police know the answer, and the assembler of FOIA2009 knows that they know.
Geoff:
As the only computer that seems to have been impounded is CRUBACK3 I assume that the police have reason to believe that that was the only one compromised, and that is where the data came from. But you are raising an interesting point. The fact that the task of analysing the data on the backup server was so daunting is no reason for not having a look round the HDDs on the other computers used by those implicated in Climategate used, if the Russell inquiry wanted to find out if there was any more compromising evidence of malpractice. Did they just assume that post-climategate everyone must have had a spring clean perhaps?
I did wonder if FOI2009 was assembled as a reference by, or for, the FOI department, along the lines of ‘This material is really sensitive, watch out if there is an FOI request that would include any of it’. What is aggravating is that while we may speculate, the HoC committee is in a position to find dout and get explanations from those involved.
TonyN: I haven’t sifted through the evidence as carefully as you have. Please say if I’m missing something. Surely a “post-Climategate clean-up” would have been impossible, given that Jones swiftly confirmed the e-mails as genuine?
Either:
1) the 1000+ e-mails were stolen one by one and placed in a file, bizarrely labelled FOI2009, by the illegal hacker (the equivalent of a burglar spending hours in the jeweller’s shop picking the best items, and then, for a laugh, taking it to the fence in a bag marked “Swag”)
Or:
2) They were assembled and placed in a file in situ, from which they were hacked or leaked. In which case, it is important to know whether they were deleted from their original computers, or simply copied. In either case, it must have been a concerted effort, and the days of work involved were presumably delegated to a subordinate.
The police must know which is the case. If it was (1), and therefore the work of an external hacker, the police are in a position to exonerate publicly both Jones, who is psychologically fragile, and Briffa, who is recovering from a serious illness. They haven’t done so.
Geoff:
I think that you are far from missing anything and it’s a very nice point; using the term in its true sense.
No doubt there are strict constraints on what the police can put in the public domain prior to a prosecution.
I don’t like to get into this discussion because it just looks really squalid at how, what should be, private emails are being picked over.
Nevertheless I’ll just point out that files can be copied, or copied then the original deleted (cut), with equal ease.
The emails may have been in one file which may have been renamed, even on the original server, either before it was copied or afterwards. They may have been gathered from separate sources, and again put into one file. Or, the one file could be one used for backup purposes. The names or the number of files doesn’t really cast any light on whether it was done locally or remotely.
PeterM
You are right in writing
But I believe (like you apparently do, as well) that the jury is still out on whether the e-mails were leaked by an inside “whistleblower” or by an outside “hacker”.
Suffice it to say that they were “leaked”.
Was the “leak” of the e-mails more “squalid” than the “contents” or “implications”?
That is probably a matter of opinion, which is heavily flavored by one’s conclusions on “dangerous AGW”.
As has been said:
Max
tempterrain,
Why do you say “should be private emails”? The Information Commissioner has ruled that everything I asked for should have been disclosed under EIR and by implication it should have been in a publication scheme.
I find it odd that anyone might think emails written by public servants at the taxpayers expense and on the taxpayers business might be private.
Russell para 22
When was this? How old is Thunderbird?
How many of us regularly delete emails other than spam? I was involved with an outfit of similar size to CRU who had not deleted anything since the beginning of email.
Suppose someone just started in mid 2009 going through to delete things and suppose they all ended up in easily identified files eg .del. Suppose there was a back door into the open source software – or someone had fitted one. The hacker could be anywhere.
The more interesting question is who cares? Why after all this time is the police investigation not closed? The answer might have nothing to do with CRU.
The analyst’s report says that each of the CRU staff’s mailboxes had three area’s designated by Thunderbird: Inbox, Sent, and Trash which is assumed to be deleted emails. Surely they might have had a quick poke around the Trash boxes if nothing else.
And just when is ‘originally’ in your quote, and just why does the report not say something like ‘prior to 19?? CRU had no central backup’? And is there any evidence in the minutes that the review panel spoke to CRU’s very own IT manager. And with such a small staff, why did CRU require 40% of an IT manager’s time?
And why are we all still posing questions about this when there are supposed to have been three independent and transparent inquiries? And who, if anyone, is finally going to ensure that the people who either know the answers, or are in a position to find out, have to come up with some credible explanations? Surely one of the easiest tasks facing Russell was solving at least some of these mysteries. Why was this not done? Are those Trash boxes loaded with yet more compromising material? One could go on for ever.
David Holland,
“I find it odd that anyone might think emails written by public servants at the taxpayers expense and on the taxpayers business might be private.”
You might try suggesting that line of argument, if you’re curious about anyone’s email account, to your local MP, or MI5 even, and see how far it gets you! I should think your reply may well be a four letter word followed by ‘off’!
At this point, after more months have passed, it seems an inescapable conclusion that the FOI2009 files were assembled by a clerk working with a script (probably Perl) that extracted emails from “Thunderbird” (actually a common Unix-format) mail files based on pattern matches, determined the date, named the extract file and touched it with the original date.
We now have a detailed account of the sequence of outstanding FOIA requests (both McIntyre and Bishop Hill have covered the subject). Has anyone attempted to examine the body of material to try to determine whether they were assembled in response to one of the outstanding requests?
It is also clear, at least to me, that they were painstakingly assembled, then simply abandoned in place when the request was finally refused, and released by a disgusted insider at CRU who had been aware of the whole process.
Craig Goodrich, #19:
I don’t know whether anyone has attempted what you are suggesting, but it’s an interesting idea. I’ll make some enquiries.
The point never to loose sight of is that the Russell inquiry TOR make it quite clear that they were expected to examine the data on the server that was not released. They did not do so for reasons that raise very serious questions about the rigour of the investigation, as Steve McIntyre and I have both pointed out.
None of this would be a mystery now if Russell had done his job properly.