diff --git a/common/modules/reprod-bad-sota.org b/common/modules/reprod-bad-sota.org index c38ac74..bc1d333 100644 --- a/common/modules/reprod-bad-sota.org +++ b/common/modules/reprod-bad-sota.org @@ -1,101 +1,101 @@ #+COLUMNS: %40ITEM %10BEAMER_env(Env) %9BEAMER_envargs(Env Args) %10BEAMER_act(Act) %4BEAMER_col(Col) %10BEAMER_extra(Extra) %8BEAMER_opt(Opt) #+INCLUDE: "prelude.org" :minlevel 1 * The state of Software reproducibility :PROPERTIES: :CUSTOM_ID: main :END: ** Software and reproducibility :PROPERTIES: :CUSTOM_ID: sw-reproducibility :END: *** A fundamental question How are we doing, regarding reproducibility, in \emph{Software}? *** The case of Computer Systems Research A field with Computer experts ... we have high expectations!\\ Christian Collberg set out to check them. *** Measuring Reproducibility in Computer Systems Research :B_alertblock: :PROPERTIES: :BEAMER_env: alertblock :END: Long and detailed technical report, March 2014\\ \url{http://reproducibility.cs.arizona.edu/v1/tr.pdf} ** Collberg's report from the trenches :PROPERTIES: :CUSTOM_ID: collbergmethod :END: *** Analysis of 613 papers :B_block:BMCOL: :PROPERTIES: :BEAMER_col: 0.45 :BEAMER_env: block :END: - 8 ACM conferences: ASPLOS’12, CCS’12, OOPSLA’12, OSDI’12, PLDI’12, SIGMOD’12, SOSP’11, VLDB’12 - 5 journals: TACO’9, TISSEC’15, TOCS’30, TODS’37, TOPLAS’34 all very practical oriented **** The basic question can we get the code to build and run? *** The workflow :B_block:BMCOL: :PROPERTIES: :BEAMER_col: 0.40 :BEAMER_env: block :END: #+BEGIN_EXPORT latex \includegraphics[height=.7\textheight]{collberg-workflow} #+END_EXPORT ** The result :PROPERTIES: :CUSTOM_ID: collbergfindings :END: #+BEGIN_EXPORT latex \begin{center} - \includegraphics[width=\extblockscale{.7\textwidth}]{collberg-outcome-new} + \includegraphics[width=\extblockscale{1\textwidth}]{collberg-outcome-new} \end{center} #+END_EXPORT *** ... :B_alertblock: :PROPERTIES: :BEAMER_env: alertblock :END: \hfill ... that's a whopping 40% of \alert{non reproducible} works! ** In Software Engineering :PROPERTIES: :CUSTOM_ID: swe-reprod :END: Even higher expectations, and yet similarly disappointing results http://fr.slideshare.net/carloghezzi18/icse-2009-keynote-15919951 *** Reference journal ACM Transactions on Software Engineering and Methodology (TOSEM) - analysis by Carlo Ghezzi, in 2009, of TOSEM from 2001 to 2006 - 60% of papers refer to a tool - 20% only are /installable/ *** Reference conference International Conference on Software Engineering (ICSE) - analysis by Zannier, Melrik, Maurer 2006 - complete absence of replication studies # # Artefact evaluation # #+INCLUDE: "../../common/modules/sourcecode-reproductibility.org::#artefactevaluation" :minlevel 2 #+INCLUDE: "../../common/modules/sourcecode-reproductibility.org::#opensourcereprod" :minlevel 2 #+INCLUDE: "../../common/modules/source-code-different-short.org::#main" :only-contents t :minlevel 2 ** The reasons (or, ``the dog ate my program'') :PROPERTIES: :CUSTOM_ID: collbergreasons :END: *** Why so much software fails to pass the test? Many issues, nice anecdotes, and it finally boils down to - /Availability/ - /Traceability/ - Environment - Automation (do /you/ use continuous integration?) - Documentation - Understanding (including free/open source software) #+BEAMER: \pause *** The first two are important /software preservation issues/ :PROPERTIES: :BEAMER_act: +- :END: Yes, code is fragile:\\ \mbox{}\hfill it can be destroyed, and we can lose trace of it diff --git a/common/modules/reprod-digital-age.org b/common/modules/reprod-digital-age.org index d3a7325..fef815f 100644 --- a/common/modules/reprod-digital-age.org +++ b/common/modules/reprod-digital-age.org @@ -1,59 +1,59 @@ #+COLUMNS: %40ITEM %10BEAMER_env(Env) %9BEAMER_envargs(Env Args) %10BEAMER_act(Act) %4BEAMER_col(Col) %10BEAMER_extra(Extra) %8BEAMER_opt(Opt) #+INCLUDE: "prelude.org" :minlevel 1 * Reproducibility :PROPERTIES: :CUSTOM_ID: main :END: ** Reproducibility in the digital age :PROPERTIES: :CUSTOM_ID: reprod :END: For an experiment involving software, we need - open access :: to the scientific article describing it - open data sets :: used in the experiment - source code :: of all the components - environment :: of execution - stable references :: between all this #+beamer: \pause *** Remark The first two items are already widely discussed!\\ \mbox{}\\ \flushright ... what about /software/? -** Software is Knowledge +** Software is knowledge :PROPERTIES: :CUSTOM_ID: softwareknowledge :END: *** Software is /an essential component/ of modern scientific research :B_picblock: # Deep knowledge embodied in complex software systems :PROPERTIES: :BEAMER_opt: pic=papermountain,width=.25\linewidth :BEAMER_env: picblock :BEAMER_act: +- :END: Top 100 papers (Nature, October 2014)\\ #+BEGIN_QUOTE [...] the vast majority describe experimental methods or software that have become essential in their fields.\\ #+END_QUOTE http://www.nature.com/news/the-top-100-papers-1.16224 # Why using the Source? ** Use the Source, Luke! :PROPERTIES: :CUSTOM_ID: opensourcereprod :END: Some people claim that having (all) the source of the code used in an experiment is /not worth the effort/ (see ``Replicability is not Reproducibility: Nor is it Good Science'', Chris Drummond, ICML 2009) #+BEAMER: \pause *** Sure, diversity \emph{is} important, but: - Source code is like the proof used in a theorem: can we really accept /Fermat statements/ like ``the details are omitted due to lack of space''? - modern complex systems makes even the simplest experiment depend on a wealth of components and configuration options - access to /all/ the source code is not just necessary to /reproduce/, it is also useful to /evolve and modify/, to /build new experiments/ from the old ones # Why using the Source? #+INCLUDE: "../../common/modules/sourcecode-reproductibility.org::#opensourcereprod" :minlevel 2