diff --git a/common/modules/reprod-bad-sota.org b/common/modules/reprod-bad-sota.org index ad0c5e4..c38ac74 100644 --- a/common/modules/reprod-bad-sota.org +++ b/common/modules/reprod-bad-sota.org @@ -1,101 +1,101 @@ #+COLUMNS: %40ITEM %10BEAMER_env(Env) %9BEAMER_envargs(Env Args) %10BEAMER_act(Act) %4BEAMER_col(Col) %10BEAMER_extra(Extra) %8BEAMER_opt(Opt) #+INCLUDE: "prelude.org" :minlevel 1 * The state of Software reproducibility :PROPERTIES: :CUSTOM_ID: main :END: ** Software and reproducibility :PROPERTIES: :CUSTOM_ID: sw-reproducibility :END: *** A fundamental question How are we doing, regarding reproducibility, in \emph{Software}? *** The case of Computer Systems Research A field with Computer experts ... we have high expectations!\\ Christian Collberg set out to check them. *** Measuring Reproducibility in Computer Systems Research :B_alertblock: :PROPERTIES: :BEAMER_env: alertblock :END: Long and detailed technical report, March 2014\\ \url{http://reproducibility.cs.arizona.edu/v1/tr.pdf} ** Collberg's report from the trenches :PROPERTIES: :CUSTOM_ID: collbergmethod :END: *** Analysis of 613 papers :B_block:BMCOL: :PROPERTIES: :BEAMER_col: 0.45 :BEAMER_env: block :END: - 8 ACM conferences: ASPLOS’12, CCS’12, OOPSLA’12, OSDI’12, PLDI’12, SIGMOD’12, SOSP’11, VLDB’12 - 5 journals: TACO’9, TISSEC’15, TOCS’30, TODS’37, TOPLAS’34 all very practical oriented **** The basic question can we get the code to build and run? *** The workflow :B_block:BMCOL: :PROPERTIES: :BEAMER_col: 0.40 :BEAMER_env: block :END: #+BEGIN_EXPORT latex \includegraphics[height=.7\textheight]{collberg-workflow} #+END_EXPORT ** The result :PROPERTIES: :CUSTOM_ID: collbergfindings :END: #+BEGIN_EXPORT latex \begin{center} - \includegraphics[width=\extblockscale{.9\textwidth}]{collberg-outcome} + \includegraphics[width=\extblockscale{.7\textwidth}]{collberg-outcome-new} \end{center} #+END_EXPORT -*** This can be debated (see http://cs.brown.edu/~sk/Memos/Examining-Reproducibility/), but... :B_alertblock: +*** ... :B_alertblock: :PROPERTIES: :BEAMER_env: alertblock :END: - \hfill ... that's a whopping 81% of \alert{non reproducible} works! + \hfill ... that's a whopping 40% of \alert{non reproducible} works! ** In Software Engineering :PROPERTIES: :CUSTOM_ID: swe-reprod :END: Even higher expectations, and yet similarly disappointing results http://fr.slideshare.net/carloghezzi18/icse-2009-keynote-15919951 *** Reference journal ACM Transactions on Software Engineering and Methodology (TOSEM) - analysis by Carlo Ghezzi, in 2009, of TOSEM from 2001 to 2006 - 60% of papers refer to a tool - 20% only are /installable/ *** Reference conference International Conference on Software Engineering (ICSE) - analysis by Zannier, Melrik, Maurer 2006 - complete absence of replication studies # # Artefact evaluation # #+INCLUDE: "../../common/modules/sourcecode-reproductibility.org::#artefactevaluation" :minlevel 2 #+INCLUDE: "../../common/modules/sourcecode-reproductibility.org::#opensourcereprod" :minlevel 2 #+INCLUDE: "../../common/modules/source-code-different-short.org::#main" :only-contents t :minlevel 2 ** The reasons (or, ``the dog ate my program'') :PROPERTIES: :CUSTOM_ID: collbergreasons :END: *** Why so much software fails to pass the test? Many issues, nice anecdotes, and it finally boils down to - /Availability/ - /Traceability/ - Environment - Automation (do /you/ use continuous integration?) - Documentation - Understanding (including free/open source software) #+BEAMER: \pause *** The first two are important /software preservation issues/ :PROPERTIES: :BEAMER_act: +- :END: Yes, code is fragile:\\ \mbox{}\hfill it can be destroyed, and we can lose trace of it