Nathaniel goes on to say that “… there’s a simple solution that should satisfy you theorists nicely: JOIN THE EXPERIMENT! …”.

A flaw in Nathaniel’s solution is that not every theorist/analyst will get to be affiliated with the experiment collaboration.

It seems to me that a more comprehensive, even simpler, solution would be to make the data public, in a format that is the work-product of Nathaniel and his fellow experimenters, by a paper authored by Nathaniel and his fellow experimenters.
Then, any theorist/analyst (whether or not affiliated) should cite that paper, so that Nathaniel et al would have a very high citation rating.

Further, if any theorist/analyst might ask Nathaniel et al for help in understanding the data, Nathaniel et al should be listed as coauthors for providing such help.

I have tried to follow that spirit in stuff that I have written. For example, in my writings about Fermilab T-quark data, I give explicit credit to Erich Ward Varnes whose 1997 UC Berkely PhD thesis contained data that I found very useful.

Tony Smith
http://www.valdostamuseum.org/hamsmith/

After working for six years on MINOS, I will get ONE (count ‘em) paper. OK, I’ll be fair. Three papers. Out of two hundred authors. If the data were made available publicly, then this paper wouldn’t even get cited… some theorist would come along, do a slightly more sophisticated analysis, and I the paper wouldn’t even get cited.

Even worse, to make the data public we now have to publish the methods and documentation how to use the data (which will NEVER just be a list of 4-vectors; there are correlations and resolution functions on every experiment) and that will take the experimentalists even more work.

Don’t get me wrong.. I love what I do. But I slave over computer code, measure crosstalk, invent calibration sources, crawl under dusty machines, travel, travel, travel, sit on interminably phone calls (every day) so that I can get those few weeks of analysing the data before everyone else. Now I can’t even do that?

Happily, there’s a simple solution that should satisfy you theorists nicely: JOIN THE EXPERIMENT! I need a three more people in my calibration group to measure attenuation curves. I need two more to get automated processing running and document things. We need people to think deeply about statistics, and to make sure our MC models are good. We need people who understand the theory well to suggest what fits to make and the best way of presenting the data. But, of course, that’s a lot of work, so not many of you take us up on the offer.

—Nathaniel

I think the biggest problem with releasing data from the LHC experiments would be the shear volume. How much would CMS or ATLAS record in a day? What kind of background reduction are people expecting here?

let me comment this phrase of yours:

“In principle I’m in favor of releasing the data, in practice I doubt that it would work. Without an intimate knowledge of the idiosyncrasies of the detector, too many spurious results would be hard to resist.”

My idea is that the release of data is really similar to release of software code in open source projects (my professional field).

For example, for software products like Netscape/Mozilla/Firefox, the code was originally proprietary and secret; then, there was the decision to release the code of the product itself. The idea was to create a large community of developers, able to contribute to the improvement of the product.

However, for more than one year after the release of the code, the contribution from the developers outside the original development team was minimal. There was a lot of interest from other developers, but they were not able to provide any significant change to the code.

The reason was understood shortly after. The code itself was only part of the knowledge that had built the product. Each line of code was the result of several decisions made by the developers, and contained assumptions there were not easy to make explicit.

In short, the code was the result of a long and complex process, but in order to contribute to the code, you had first to became part of the process. Only after the assumptions became clearer, it was possible for other people to make significant contributions.

Which is the relation I see with LHC data ?

Data are the result of complex processes, where there is a lot of hidden knowledge that is necessary in order to understand what a number really mean in a certain context. People outside the process cannot understand what the raw data can really mean, without a proper understanding of the process itself.

However, if the parallel I have made is anything significat, IT IS useful to make data available, as far as you understand that you have to make clear which is process through which they are produced and elaborated.

Then, other people can make useful proposal on how to improve the understanding of data. In fact, making the process public has significatively improved the process itself.

Again turning to the analogy with cosmology/astrophysics, I suspect many theorists in cosmology (myself included) are learning more about Bayesian statistics, priors, Markov Chains and all the rest of it than we would have ever dreamed. And I can certainly point to some deeply flawed papers in the literature that might never have seen the light if the raw data was not freely available. However, theorists *can* learn this stuff, and don’t like to look silly in public, so they have plenty of motivation for doing so.

In the end the theorists will either learn enough of the subtleties to do it themselves, or work with experimentalists who know how to perform the relevant analyses. Many (most?) papers are flawed in some way, and the community would react to the flurry of theorist-written data-driven papers by hiking its overall level of skepticism a notch or two. Just as it did with the arrival of the Arxiv, which does an end-run around peer review (for what that is worth, but don’t get me started)

As Sean pointed out above, one side-effect of the present system is that it is very hard for particle theorists to collaborate with experimentalists, if the whole collaboration needs to sign off on papers that any single member writes (and this is *after* the data is in the public domain). Again speaking from my own experience, my foray into the world of data-analysis has largely been conducted in collaboration with someone who understood the issues involved at the outset (although not an “experimentalist” in the strict sense of the term), and it is a singularly productive mode of collaboration. To the extent that the “rules” of experimental particle physics discourage this sort of collaboration they are clearly ounter-productive.

Secondly, the cosmological community has benefitted greatly from the development of the Cosmomc package which greatly simplifies the Monte Carlo Markov Chain analyses of cosmological data. (It is not theorist-proof however, as I have seen several publicly displayed figures that showed chains which, to my now practiced eye, were clearly unconverged). My guess is that if more experimental particle physicss data was made publicly available it would seed a small industry in the development of software tools that facilitated its analysis.

The data belongs to taxpayers, so far as I’m concerned.

In my experience experimentalists are extremely suspicious of results, since they know what actually goes into making them — hence the tradition of requiring confirmation from an independent experiment for discovery claims. Without some kind of (at least short-term) data encapsulation, problems will arise: imagine experiments looking at each other’s data! (Related reasons are behind the rise of “blind analyses,” where a collaboration hides its data from itself, for fear that it will find what it wants to find.) Even if only a small fraction of a collaboration reads a paper thoroughly, that’s still an awful lot of experience-years.

And finally, a point that vaguely amuses me: we are used to a feedback system where either (a) theorists predict something, experiment finds it, theorists claim vindication because the prediction was ante hoc instead of post hoc, or (b) experiment finds something unexpected, everyone scrambles to see how models could accomodate this result, some things can’t and are excluded. What happens to this waltz if the theorists get to look at the data at the same time the experimentalists do?

[Note: none of this implies that I don't think processed HEP data should be released after a (longish) while, or that short-term data release might not be a good thing if we find ourselves with a one-detector ILC.]

The main problem is that in order to make all the data public, you have to understand all the data in a very global way.

Of course, you know at least one experimentalist who claims to take such a global, um, vista.

Some of these issues exist for astronomical archives. For example, some of the instrumental idiosyncrasies of the Hubble Space Telescope become better understood over time, and the post-processing and corrections get updated (or new correction stages are introduced). The archive implements an “on-the-fly” recalibration system, so if you request data from the archive, they are always processed with the latest approved algorithms and calibration files. In a sense, the data are continually “re-published”, and it’s up to the archive users to make sure that the data they retrieved a year ago haven’t been made obsolete by significant improvements in the calibration since then (this is quite rare in practice).

Of course, the fact that the HST data come in small, discrete chunks (be they images or spectra) makes it easier to implement such a system. But “publishing” data from an instrument does not have to be a one-time, can’t-go-back-and-fix-it affair.

So an interesting question is whether something like this might be true for LHC datasets. Not being a high-energy physicist in any way, shape, or form, I have no idea. (Thomas Dent’s comment suggests that perhaps there could be such cases.)

The main problem is that in order to make all the data public, you have to understand all the data in a very global way. This isn’t the way particle physics of this scale is traditionally done. On any given analysis, you have some touchstones to ensure your sanity and some control regions to ensure your methods and some signal regions to measure or search for something. And all this is on just a small portion of the data. Generally, any given analysis will have some object ID requirements and some fudge factors, such as the probability for a jet to fake an electron or the k-factor applied to the leading order Z+3jet x-section. While these are somewhat standardized across analyses, not everybody working on every analysis will use the same thing. For example, not everybody on an experiment will agree on what objects are in a given event. If I’m measuring the W mass in the W->e nu channel, my definition of an electron is going to be much tighter (rightly so) than if I’m looking for ZZ->eeee where not only do I have four electrons, but I also have two mass constraints.

Another problem is that a list of four vectors from ATLAS will not mean the same thing as a list of four vectors from CMS. So then, not only do you have to have the data for each experiment, but also the monte carlo.

Finally, this is all a moving target. Algorithms change. Object ID changes as parts of the detector are better understood. At what point do you publish the data? When the experiment is over and no new changes are going to be made? As soon as it’s well understood? Once you publish it, do you get to go back and change it? What happens if someone produces a bogus result on old data?

A few more points. “Experimental papers written by theorists will never be believed” - maybe true, but this doesn’t prove that data should be secret. In fact the more true it is, the bigger disincentive there is for theorists to encounter raw data and the more likely that experimenters will have the field to themselves.

“Theorists might scoop experimental grad students” … this seems in contradiction with the previous point. If experimentalists are so much better at dealing properly with detectors, backgrounds, etc. then there is no way they can get scooped because you need to control those aspects before making any credible claim of discovery. Something is a bit odd if a major scientific discovery has to be delayed until one student has the time to write up a thesis. But I don’t think experiments really work like that.

There is no real theory-experiment conflict. The experimental work is for example designing and implementing a trigger and doing detector simulations for some type of signal and calibrating the detector and background once the machine is running, and without this there would be nothing at all for theorists to use. Morally, any ‘discovery’ paper should credit those experimentalists who were crucial to the existence of the data. But traditionally, individuals are *not* credited, the experimental collaboration publishes collectively and is cited collectively and no-one knows exactly who did what. The question is, how is the experiment run (democracy? benevolent dictator?), who decides when and how and by whom data analysis is done, how do individual experimenters get credit for their own work apart from word-of-mouth? These are not questions which involve theorists unless they are really competent to do data analysis. I don’t think experimental secrecy will solve any problems.

How about if the equations of GR or supersymmetry were kept secret from anyone who wasn’t a theorist…

Not really. The final production data consists of photons, electrons, muons, taus, and jets, constructed out of all of the things recorded in the detector.

Of course, this means some set of tracks and energy deposits and so on that have passed certain identification criteria, and one must understand fake rates and so on.

Nonetheless, the final objects of an analysis are, at some approximate level, just four-vectors labelled as a certain particle type, with extra supplementary information.

Of course, outsiders analyzing data are always troublesome: see e.g. de Boer’s claims of dark matter discovery in EGRET data. On the other hand, after some reasonable length of time, it seems wise to make the data public. (How many interesting LEP analyses remain that aren’t happening because experimenters have [reasonably] moved on?)

This is not always true. Radio interferometry is conceptually quite intricate and requires that a fair amount of thinking be done in Fourier space. GLAST, scheduled to be launched next year, is a stack of particle detectors and calorimeters.

In fact I think that people tend to underestimate the extent to which the experimental collaborations will sweep up all the low-hanging fruit, when it comes to interpreting surprises in the data. These folks know about all the major models, and they’ll definitely put a lot of work into matching the data to the theories before they ever release any results. Which is okay — theorists, instead of ambulance-chasing, will be left to do the hard work of puzzling out the results that don’t fit into any of the popular models lying around.

One related (and, I would think, simpler to solve) problem is the closed nature of the collaboration-based publication process. Given all the blessing and godparenting and so on that must take place before an analysis sees the light of day, it’s hard-to-impossible for an experimentalist to actually collaborate directly with a theorist on attacking some particularly interesting puzzle. And that is just a shame.