Class Journal Week 8

Mahrad Saeedi

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?

Nicole Anguiano

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?

Nanguiano (talk) 15:06, 20 October 2015 (PDT)

Jake Woodlee

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • From what I saw, Baggerly and Coombs identified a lack of transparency in procedure, improper labeling, and an eventual discrepancy in the statistical analysis of the paper. The best practice violated in DataONE were not labeling data properly and an inadequate procedure section. Baggerly said it was a common problem for the statistical analysis to be inadequately written out in a step by step format with incomplete documentation. And while the review board mentioned this they still failed to act.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • He recommends the origin of the paper, the raw data included, any code used to process the data, descriptions of non scriptable steps, and descriptions of planned design are all necessary for clinical trials and should be included in a reproducible scientific paper. He notes that if there is a more standard procedure reproduction would be easier. In the DataONE powerpoint there is mention of consistent data labels and formats that would contribute to reproducible research.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • I am glad that he was so persistent on pushing his review of the paper through all the criticism. I also think it is a good thing he is such a proponent for reproducible research because it is the only think science can really rely on to verify a claim. Also, I thought it was interesting how he mentioned that more complex technologies call for more complex research reproduction tools.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The values are not exactly the same. This is definitely due to how we processed the data and performed our analysis. They used the program "SAM" while we used GenMAPP and to some extent, Excel. These differences in procedure caused the difference in conclusion, a further and deeper analysis of our procedures will get to the bottom of this. I am not confident I could reproduce their data completely. I might be able to completely figure it out given enough time, I'm just not sure how I would react to the software they used.

Jwoodlee (talk) 15:35, 22 October 2015 (PDT)

Emily Simso

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • The most common issues are that, first, documentation is often poor in clinical trials, thus not explaining what researchers did in their work. There are also then problems with intuition, since people assume things about the data. Overall, Baggerly and Coombs stress that lack of thoroughness with data and research leads to complications further on.
   • The violated best practices according to DataONE were: consistency, being descriptive, and lacking data or information.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Dr. Baggerly recommends that groups use the same standards for reports, templates are reused, there is a report structure for approval, and use executive summaries for complete documentation.
   • This connects to DataONE because they stress reproducible research through appropriate file types, consistent formatting, clear definitions, and understanding how databases are set up before using them, amongst other points.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • I think watching Dr. Baggerly's talk helped explain how something like the Duke case could happen, because there are so many details in data and research. It seems that mistakes could easily be covered up simply because there is a culture of not documenting every aspect of your work. I think that this needs to change so that future clinical trials are more closely regulated.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The values did not match between the Merrell et al. analysis and my analysis. This is probably because they are much more trained in this field and have more resources at their disposal. They were also able to perform more tests on the data, whereas we only had the given values. While I think we are able to get fairly accurate results, we are not close enough to the experiment to get the exact same results.

Emilysimso (talk) 20:49, 25 October 2015 (PDT)

Veronica Pacheco

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • Alot of the data being analyzed had discrepancies. They noticed problems in the data and they predict that they switched their input giving a much different resistance number than the actual resistance number. They figured this out by using a different study.They also made the point that there other genes that wasn't making sense in their data. 14/19 genes were accounted for by cross referencing them with another paper but that still leaves the other 5 genes. At this point, the contacted the magazine to report these findings. They said the most common mistakes are simple. Some of these mistakes include experimental design, mixing up sample labels, gene labels and group labels. DataOne explains that organization, consistency, and description is key to practicing good data preservation skills.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • The ask for labeling the columns to tell which sample is which and DataOne emphasizes this point as well. They also ask to provide the code so that it is clear when trying to reproduce the results in a given experiment.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • I like how at the end, he tells the audience what Coombs and him do as part of their protocol.For example, the use literate programming like Sweave. Overall, I liked how this assignment was structured. Reading the case first, then hearing this talk on how they went about figuring out the issues was a neat experience.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The values did not match. I think it has to do with the accuracy of their method.They were able to use SAM which is Statistical Analysis for Microarrays and it is probably more advanced and accurate than using the pvalues in Excel. Although Excel is a great tool.

Vpachec3 (talk) 21:59, 25 October 2015 (PDT)

Kevin Wyllie

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • Baggerly points out confounding in experimental design (mixing up interpretation of results: sensitive versus resistant), mixing up sample/gene/group labels, and incomplete documentation (which hides the previously-mentioned mistakes). DataONE mentions that data should be stored in a format which allows it to be used by any application. This was violated when the researchers added a column name to the gene ID’s, which tricked their code into offsetting each gene by one.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Baggerly recommends appropriately labeling data, and providing code (provenance) so that it can be tested by third parties. Baggerly thinks these things should be required before beginning clinical trials. DataONE recommends using “descriptive column names” as well as dataset provenance.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • Dr. Baggerly speaks quite fast and uses some terminology that is unfamiliar to me, so after watching the video twice, I’m still not entirely sure which “errors” he is implying Dr. Potti committed intentionally (if any). To me, this contrasts with the 60 Minutes segment which seemed to mention solely the deliberate manipulation of the data. The off-by-one index error, for example, seems like it could have been an honest mistake (not to say this would relieve Potti of culpability), as I can’t imagine how that would actually add to the (false) significance of the results.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The values did not match, likely because different statistical methods were used. After reading the paper, I can't say I think the authors provided sufficient information to carry out an identical analysis. In fact, the paper itself mentions very little statistical analysis at all. Baggerly and DataONE would not approve.

Brandon Klein

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • Baggerly and Coombes identified a concerning amount of issues with the data and analysis present in Dr. Potti's paper. Much of the data used for the analysis had consistent off-by-one index errors due to inappropriately omitting column headings, which in turn mismatched data and offset p-values. Further, some data sets appeared in replicate and were inconsistent with themselves, while others were integrated to explain biology but not used in the actual analysis. If this was not concerning enough, some critical labels such as resistant and sensitive were swapped in the data analysis. When challenged, the team at Duke even released a validation data set that contained entirely incorrect and misleading samples.
   • The issues that Dr. Baggerly addressed in his lecture violated various best practices enumerated by DataONE. The Duke team did not maintain dataset provenance (data transformations and duplications were common), ensure data compatibility (off-by-one index error), or use reproducible workflows (many steps were poorly documented or omitted). In addition to this, there were issues with the consistency of the data (e.g. switched labels) and insufficient documentation, particularly when the Duke team was attempting to validate their research in the face of concerns (much of the validation work was incorrect and not released).
   • Dr. Baggerly claimed issues such as off-by-one index errors, confounding experimental design, mixing up labels, and incomplete documentation are unfortunately more common than we would like to admit.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Dr. Baggerly introduced a series of recommendations for producing reproducible research that he and Coombs had been using to the audience. To begin, he suggested that all papers should include data, code, descriptions of nonscriptable steps, descriptions of planned design if used, and should maintain the provenance of data. This both largely automates and elucidates the data analysis methods, enabling it to be reproducible. Further, steps such as including literature programming (Dr. Baggerly mentioned using Sweave so that anyone could run his code through R and get the same results), reusing templates, reporting structure, providing summaries, and including appendices are all ways to further ensure that your research is reproducible. These suggestions are consistent with the DataONE best practices, such as mantaining dataset provenance, documenting manipulations/assumptions, and particularly using reproducible workflows (DataONE explicitly suggests automating the integration process as much as possible, which Dr. Baggerly stressed).
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • Although I had initially been appalled by the 60 Minutes exposé on this case, watching Dr. Baggerly's lecture still managed to floor me. I had not realized how dramatic and blatant some of the data manipulations were. The fact that entirely incorrect datasets were used, treatment groups were inverted, and data was even replicated/mislabeled to generate a specific result demonstrates a clear intent to fake results. And this was no minor faking of data either; the Duke team had to actively and aggressively make these alterations to the data. This kind of behavior, particularly in the realm of medicine and clinical trials, is extremely concerning to me. Further, I had not realized how difficult it was for Baggelry and Coombes to draw attention to the manipulations made by the Duke team. There should be a system in place to make it easier to expose research fraud. Research publications should be focused on simply presenting discoveries as opposed to finding and supporting sensational stories while turning a blind eye to potential manipulation.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The "sanity check" that I performed comparing my microarray analysis to the results published by Merrell et al. (2002) did not entirely match up. Slightly less than one-third of the genes they reported as having exhibited significant changes in expression did not exhibit statistically significant expression changes in my analysis. However, this likely occurred because I used a different method to determine significance than they did, assessing p-values instead of running a two-class Statistical Analysis for Microarrays (SAM) analysis. Because the criteria used for judging significance was different, it makes why not all of my results matched theirs. Although the paper is slightly vague when discussing the specific statistical methods they used, I believe they included enough information regarding their methods and software tools to reproduce their results (although minor forensic bioinformatics may be necessary to bridge slight gaps in the documentation).

-- Bklein7 (talk) 18:22, 26 October 2015 (PDT)

Josh Kuroda

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • One of the main issues that Baggerly and Coombs brought up had do to with proper documentation, since the absence of this would require "forensic bioinformatics" so that one may infer what was done to obtain the results. One recurring practice that was violated was the practice to correctly label data points. In one case, 43 samples were mislabled, and 16 samples were not matched at all. Complete documentation is also a practice that was violated, and Baggerly claims that both incomplete documentation and labeling issues are the most common. Mixing up sample, gene and group labels are the most simple mistakes, but happen too often.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Dr. Baggerly recommends literate programming, reusing templates, report structure, executive summaries, and appendices as elements to the bigger picture of reproducible research. These mostly overlap with what DataONE recommends, since DataONE says that consistently formatted data, documentation of assumptions, reproducible workflows, and compatibility ensuring is vital to good data management.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • In my opinion, many of his points are valid, but I feel like the Duke case is more of a lesson for bioethics than reinforcing reproducible research. In my opinion, the Duke case shows why researchers cannot simply wish for success, since the venture for truth and cures is a journey that is littered with failures and lessons learned.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • No, the values did not match. This is probably because of the differences in data analysis techniques between our assignment and their research. I am not completely sure, but I think we would be able to reproduce their results to a certain degree of accuracy, mostly because of the fact that we would most likely be able to fill in any gaps in information or instruction now that we have a good understanding of how to analyze microarray data.

--Jkuroda (talk) 21:37, 26 October 2015 (PDT)

Anu Varshneya

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • Baggerly and Coombs brought attention to several different issues regarding issues with indexing, headings, mismatched data, and a lack of documentation which makes it harder to reveal the issues with the previous steps. DataONE stressed the importance of storing data in a consistent format that is usable in several programs and applications, and being descriptive and explicit with all data. This best practice was clearly violated in Dr. Potti's data by adding an extra column to his data that caused everything to be misread, which prevented the data from providing accurate data.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Baggerly recommends descriptive labels on data, and providing code or any other testing procedure to ensure that all data can be tested for reproducibility and accuracy. These recommendations are similar to the recommendations made by DataONE which include reproducible research through easily accessed data, consistency in format, and descriptive labels and definitions regarding data and databases used.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • Dr. Baggerly's talk helped clarify how the mistakes made in Dr. Potti's research could have happened, and how they could be prevented in the future. His talk was insightful in regards to proper practices in data analysis, but it doesn't answer questions regarding how intentional Dr. Potti's data discrepancies were.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • Not all of the genes that Merrell et al. (2002) presented were significant in my data. This is most likely due to differences in data analysis tools - we used Excel to compute t-tests and p values while they used a Statistical Analysis of Microarrays (SAM) analysis. Because the SAM analysis analyzed the data and had different requirements for statistical difference, it is understandable that we had different results. Though the paper was pretty vague regarding the exact methods used, I think it is possible to partially reproduce their findings, but only after some guessing regarding the exact steps they too.

-- Anuvarsh (talk) 22:50, 26 October 2015 (PDT)