Wake Up and Smell the Coffee: Evaluation Methodology for the 21st Century.
Evaluation methodology underpins all innovation in experimental computer science. It requires relevant workloads, appropriate experimental design, and rigorous analysis. Unfortunately, methodology is not keeping pace with the changes in our field. The rise of managed languages such as Java, C#, and...
| Publicado en: | Communications of the ACM Vol. 51; no. 8; pp. 83 - 90 |
|---|---|
| Autores principales: | , , , , , , , , , , , , , , , , , , , |
| Formato: | Artículo |
| Publicado: |
Association for Computing Machinery
Aug2008
|
| Materias: | |
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=hlh&AN=33662987&site=ehost-live header: @attributes: shortDbName: hlh uiTerm: 33662987 longDbName: Humanities International Complete uiTag: AN controlInfo: bkinfo: jinfo: jid: 00010782 ACM jtl: Communications of the ACM issn: 00010782 maglogo: N pubinfo: dt: Aug2008 vid: 51 iid: 8 pid: 68 pub: Association for Computing Machinery artinfo: ui: 33662987 10.1145/1378704.1378723 ppf: 83 ppct: 7 formats: tig: atl: Wake Up and Smell the Coffee: Evaluation Methodology for the 21st Century. aug: au: Blackburn, Stephen M. McKinley, Kathryn S. Garner, Robin Hoffmann, Chris Khan, Asjad M. Bentzur, Rotem Diwan, Amer Feinberg, Daniel Frampton, Daniel Guyer, Samuel Z. Hirzel, Martin Hosking, Antony Jump, Maria Han Lee Eliot, J. Moss, B. Phansalkar, Aashish Stefanović, Darko VanDrunen, Thomas Von Dincklage, Daniel affil: Australian National University. University of Texas, Austin. University of Massachusetts, Amherst. University of New Mexico. University of Colorado. Tufts University. Wheaton College. Purdue University. su: Java programming language Object-oriented programming languages Programming languages Application software Computer software sug: subj: Java programming language Object-oriented programming languages Programming languages Application software Computer software ab: Evaluation methodology underpins all innovation in experimental computer science. It requires relevant workloads, appropriate experimental design, and rigorous analysis. Unfortunately, methodology is not keeping pace with the changes in our field. The rise of managed languages such as Java, C#, and Ruby in the past decade and the imminent rise of commodity multicore architectures for the next decade pose new methodological challenges that are not yet widely understood. This paper explores the consequences of our collective inattention to methodology on innovation, makes recommendations for addressing this problem in one domain, and provides guidelines for other domains. We describe benchmark suite design, experimental design, and analysis for evaluating Java applications. For example, we introduce new criteria for measuring and selecting diverse applications for a benchmark suite. We show that the complexity and nondeterminism of the Java runtime system make experimental design a first-order consideration, and we recommend mechanisms for addressing complexity and nondeterminism. Drawing on these results, we suggest how to adapt methodology more broadly. To continue to deliver innovations, our field needs to significantly increase participation in and funding for developing sound methodological foundations. pubtype: Periodical doctype: Article src: R language: English refInfo: copyright: @attributes: flag: Y dt: @attributes: year: 2008 holdings: @attributes: islocal: N |
|---|