LMU BioDB 2015 - User contributions [en]

Gene Database Project

2015-12-18T22:13:33Z

Dondi: /* Individual Deliverable */ Eliminate CATME requirement.

{{Gene Database Project Links}}

== Deliverables ==

* You will give a final group [[Gene_Database_Project_Deliverables#PowerPoint_Presentation | PowerPoint presentation]] in class during the final exam period on '''Tuesday, December 15, 2:00pm'''.
* Final due date for all other deliverables is no later than '''Friday, December 18, 4:30pm''' (according to official final exam schedule for fall, 2015).
* The deliverables should be uploaded and organized onto one group wiki page, or alternately, delivered on digital media to either Dr. Dahlquist or Dr. Dionisio.
* Each member of the group will be assigned the same grade for the group project.
* Detailed specifications, particularly for the [[Gene Database Project Deliverables#Group Report|group report]] and [[Gene Database Project Deliverables#Individual Assessment and Reflection|individual assessment and reflection]], are given on the [[Gene Database Project Deliverables]] page.

=== Group Deliverables ===

* GenMAPP Gene Database for assigned species (''.gdb'')
* ReadMe file to accompany the Gene Database (''.pdf'')
** Sample ReadMe in Word format: [[Media:ReadMe_Vc-Std_External_20131122.zip | ReadMe_Vc-Std_External_20131122.zip]]
** [https://github.com/lmu-bioinformatics/xmlpipedb/blob/readme/GenMAPP%20Gene%20Databases/V.%20cholerae/V.%20cholerae%2020101022/ReadMe.md Sample ReadMe in markdown (a work in progress)]
** Include Gene Database Schema diagram in ReadMe
*** Sample schema in Adobe Illustrator format: [[Media:GenMAPP_schema_generic_bacteria_20151210.zip | GenMAPP_schema_generic_bacteria_20151210.zip]] 
*** Sample schema in jpeg format: [[Media:GenMAPP_schema_generic_bacteria_20151210.jpg | GenMAPP_schema_generic_bacteria_20151210.jpg]]
* Gene Database Testing Report for final submitted Gene Database (print from wiki to ''.pdf'' file)
* Processed and analyzed DNA microarray dataset (''.xls'' or ''.xlsx'')
* Data file used for import into GenMAPP (''.txt'' or ''.csv'')
* GenMAPP Expression Dataset file (''.gex'')
* Exceptions file of data imported into GenMAPP (''.EX.txt'')
* Raw MAPPFinder results files (''-GO.txt'')
* ''.gmf'' file
* Filtered MAPPFinder Results (''.xls'' or ''.xlsx'')
* Sample MAPP file of a relevant biological pathway for your species (''.mapp'')
* [[Gene Database Project Deliverables#Group Report | Group Report]] describing the creation of the Gene Database and the biological analysis of the data (''.doc'', ''.docx'', or ''.pdf'')
* PowerPoint presentation (''.ppt'', ''.pptx'', or ''.pdf'', given on Tuesday, December 15)

=== Individual Deliverable ===

The individual deliverable is an [[Gene Database Project Deliverables#Individual Assessment and Reflection|assessment and reflection]] on the process (either wiki or email, depending on your preference; see note in the linked section):
* Statement of work
* Assessment of the work done
* What was learned


== Team Journal Entries ==

* Each team will write a combined journal entry for each week with contributions from all members.
* [[Week 10]] Creation of page and combined annotated bibliography (midnight 11/10)
* [[Week 11]] (midnight 11/17)
* [[Week 12]] (midnight 11/24)
* [[Week 14]] (midnight 12/8)
* [[Week 15]] (midnight 12/15)

== Groups ==

The project groups are:
# Team 1 - '''''Heavy Metal HaterZ''''' ''Shewanella oneidensis''
#* Coder: Mary Alverson
#* Quality Assurance: Josh Kuroda
#* GenMAPP Users: Ron Legaspi & Emily Simso
#* Project Manager: Ron Legaspi
# Team 2 - '''Class Whoopers''' ''Bordetella pertussis'' (Whooping Cough)
#* Coder: Nicole Anguiano
#* Quality Assurance: Brandon Klein
#* GenMAPP Users: Lena Olufson & Mahrad Saeedi
#* Project Manager: Nicole Anguiano
# Team 3 - '''GÉNialOMICS''' - ''Burkholderia cenocepacia''
#* Coder: Anu Varshneya
#* Quality Assurance: Brandon Litvak
#* GenMAPP Users: Veronica Pacheco & Kevin Wyllie
#* Project Manager: Anu Varshneya
# Team 4 - ''' Oregon Trail Survivors''' ''Shigella flexneri''
#* Coder: Jake Woodlee
#* Quality Assurance: Trixie Roque
#* GenMAPP Users: Erich Yanoschik & Kristin Zebrowski
#* Project Manager: Kristin Zebrowski

== Roles (Guilds) ==

As the project moves forward, we will use class time for team meetings. In addition, we will also have guild meetings where students sharing the same role can work together on common issues. Each student has been assigned a primary role in the project by the instructors (see above).

=== [[Project Manager]] (PM) ===

The project manager makes sure that individuals are fulfilling their roles and performing the tasks on time.

=== [[GenMAPP User]] (GU) ===

The GenMAPP user runs GenMAPP with the microarray dataset (Expression Dataset Manager, MAPPFinder, GenMAPP MAPP). His or her tasks generally correspond to the work done during the DNA microarray analysis exercise ([http://www.openwetware.org/wiki/BIOL398-01/S10:Sample_Microarray_Analysis_Vibrio_cholerae part 1], [http://www.openwetware.org/wiki/BIOL367/F10:GenMAPP_and_MAPPFinder_Protocols part 2]), as applicable to the team’s chosen bacterium.

=== [[Quality Assurance]] (QA) ===

The Quality Assurance team member is the resident expert on species ID systems and formats. He or she should be proficient with ''XMLPipeDB Match'', SQL queries in PostgreSQL, Microsoft Excel, and Microsoft Access to navigate through the data and find missing IDs, discrepancies, sanity checks, etc.

=== [[Coder]] (C) ===

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Project Milestones ==

Specific project milestones are found on the individual guild pages.

=== Overall Flow ===

A successful project will have the following steps from end to end. Depending on the data and/or other issues that are encountered, one or more of these steps may go through iterations, repetitions, or refinements.

[[Image:ProjectWorkflow.png|450px|thumb|right|Overall Flow]]

# Following the steps shown in the [[Running GenMAPP Builder]] instructions, download the UniProt XML proteome set and GOA (GO association) files for your species.
#* Make sure that you are choosing the correct strain/subspecies as the microarray data you have.
#* Note the date of download and the version of the file on your Gene Database Testing Report.
# Download GO terms from [http://geneontology.org/page/download-ontology#Legacy_Downloads Gene Ontology].
#* You will need the OBO-XML format, the "obo-xml.gz" file.
#* Note the date of download and the version of the file on your Gene Database Testing Report.
# Create the GenMAPP Builder tables in PostgreSQL.
# Load files into PostgreSQL database via GenMAPP Builder.
# Export into a GenMAPP Gene Database.
# Inspect/vet/validate Gene Database.
#* Create a [[Gene Database Testing Report Sample | Gene Database Testing Report]] each time a new export is run.
#* Compare to the model organism database for your species
# Find an original journal article describing a DNA microarray experiment performed on your species.
#* Download the microarray data for this article.
#* Consult with Dr. Dahlquist about the proper steps to take to process the data (normalization, statistical analysis) and perform the analysis.
# Run GenMAPP using the Gene Database.
#* Microarray data (import using Expression Dataset Manager)
#* MAPPFinder
#* Place genes on MAPP and draw pathway

=== Guild Milestones ===

These are links to the respective milestone lists in the guild pages:
* [[Project Manager#Milestones|Project Manager]]
* [[GenMAPP User#Milestones|GenMAPP User]]
* [[Quality Assurance#Milestones|Quality Assurance]]
* [[Coder#Milestones|Coder]]

{{Gene Database Project Links}}

[[Category:Group Projects]]

Gene Database Project

2015-11-02T07:32:39Z

Dondi: /* Deliverables */ Additional link to deliverables page.

{{Gene Database Project Links}}

== Deliverables ==

* You will give a final group PowerPoint presentation in class on '''Tuesday, December 15, 2:00pm'''.
* Final due date for all other deliverables is no later than '''Friday, December 18, 4:30pm''' (according to official final exam schedule for fall, 2015).
* The deliverables should be uploaded and organized onto one group wiki page, or alternately, delivered on digital media to either Dr. Dahlquist or Dr. Dionisio.
* Each member of the group will be assigned the same grade for the group project.
* Detailed specifications, particularly for the [[Gene Database Project Deliverables#Group Report|group report]] and [[Gene Database Project Deliverables#Individual Assessment and Reflection|individual assessment and reflection]], are given on the [[Gene Database Project Deliverables]] page.

=== Group Deliverables ===

* GenMAPP Gene Database for assigned species (''.gdb'')
* ReadMe file to accompany the Gene Database (''.pdf'')
** Sample ReadMe in Word format: [[Media:ReadMe_Vc-Std_External_20131122.zip | ReadMe_Vc-Std_External_20131122.zip]]
** Include Gene Database Schema diagram in ReadMe
*** Sample schema in Adobe Illustrator format: [[Media:Vibrio_schema_20101022.zip | Vibrio_schema_20101022.zip]]
* Gene Database Testing Report for final submitted Gene Database (print from wiki to ''.pdf'' file)
* Processed and analyzed DNA microarray dataset (''.xls'')
* GenMAPP Expression Dataset file (''.gex'')
* Filtered MAPPFinder Results (''.xls'')
* Sample MAPP file of a relevant biological pathway for your species (''.mapp'')
* [[Gene Database Project Deliverables#Group Report | Group Report]] describing the creation of the Gene Database and the biological analysis of the data (''.doc'' or ''.pdf'')
* PowerPoint presentation (''.ppt'', given on Tuesday, December 15)

=== Individual Deliverable ===

The individual deliverable is an [[Gene Database Project Deliverables#Individual Assessment and Reflection|assessment and reflection]] on the process (either wiki or email, depending on your preference; see note in the linked section):
* Statement of work
* Assessment of the work done
* What was learned
* Team evaluation via the [https://www.catme.org CATME] survey
*: ''You will, or should have already, received an email message from the CATME system with login instructions on your '''LMU email account''' (not necessarily the preferred email address that you listed in the wiki).''

== Team Journal Entries ==

* Each team will write a combined journal entry for each week with contributions from all members.
* [[Week 10]] Creation of page (midnight 11/10)
* [[Week 11]] (midnight 11/17)
* [[Week 12]] (midnight 11/24)
* [[Week 14]] (midnight 12/8)
* [[Week 15]] (midnight 12/15)

== Groups ==

The project groups are:
# Team 1
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:
# Team 2
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:
# Team 3
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:
# Team 4
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:

== Roles (Guilds) ==

As the project moves forward, we will use class time for team meetings. In addition, we will also have guild meetings where students sharing the same role can work together on common issues. Each student has been assigned a primary role in the project by the instructors (see above).

=== [[Project Manager]] (PM) ===

The project manager makes sure that individuals are fulfilling their roles and performing the tasks on time.

=== [[GenMAPP User]] (GU) ===

The GenMAPP user runs GenMAPP with the microarray dataset (Expression Dataset Manager, MAPPFinder, GenMAPP MAPP). His or her tasks generally correspond to the work done during the DNA microarray analysis exercise ([http://www.openwetware.org/wiki/BIOL398-01/S10:Sample_Microarray_Analysis_Vibrio_cholerae part 1], [http://www.openwetware.org/wiki/BIOL367/F10:GenMAPP_and_MAPPFinder_Protocols part 2]), as applicable to the team’s chosen bacterium.

=== [[Quality Assurance]] (QA) ===

The Quality Assurance team member is the resident expert on species ID systems and formats. He or she should be proficient with ''XMLPipeDB Match'', SQL queries in PostgreSQL, Microsoft Excel, and Microsoft Access to navigate through the data and find missing IDs, discrepancies, sanity checks, etc.

=== [[Coder]] (C) ===

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Project Milestones ==

Specific project milestones are found on the individual guild pages.

=== Overall Flow ===

A successful project will have the following steps from end to end. Depending on the data and/or other issues that are encountered, one or more of these steps may go through iterations, repetitions, or refinements.

[[Image:ProjectWorkflow.png|450px|thumb|right|Overall Flow]]

# Following the steps shown in the [[Running GenMAPP Builder]] instructions, download the UniProt XML proteome set and GOA (GO association) files for your species.
#* Make sure that you are choosing the correct strain/subspecies as the microarray data you have.
#* Note the date of download and the version of the file on your Gene Database Testing Report.
# Download GO terms from [http://geneontology.org/page/download-ontology#Legacy_Downloads Gene Ontology].
#* You will need the OBO-XML format, the "obo-xml.gz" file.
#* Note the date of download and the version of the file on your Gene Database Testing Report.
# Create the GenMAPP Builder tables in PostgreSQL.
# Load files into PostgreSQL database via GenMAPP Builder.
# Export into a GenMAPP Gene Database.
# Inspect/vet/validate Gene Database.
#* Create a [[Gene Database Testing Report Sample | Gene Database Testing Report]] each time a new export is run.
#* Compare to the model organism database for your species
# Find an original journal article describing a DNA microarray experiment performed on your species.
#* Download the microarray data for this article.
#* Consult with Dr. Dahlquist about the proper steps to take to process the data (normalization, statistical analysis) and perform the analysis.
# Run GenMAPP using the Gene Database.
#* Microarray data (import using Expression Dataset Manager)
#* MAPPFinder
#* Place genes on MAPP and draw pathway

=== Guild Milestones ===

These are links to the respective milestone lists in the guild pages:
* [[Project Manager#Milestones|Project Manager]]
* [[GenMAPP User#Milestones|GenMAPP User]]
* [[Quality Assurance#Milestones|Quality Assurance]]
* [[Coder#Milestones|Coder]]

{{Gene Database Project Links}}

[[Category:Group Projects]]

Gene Database Project Deliverables

2015-11-02T07:29:42Z

Dondi: Include list of files in deliverables page too.

{{Gene Database Project Links}}

== Group Report ==

These guidelines are based on the [https://peerj.com/about/author-instructions/ Instructions for Authors] issued by the [https://peerj.com/computer-science/ PeerJ Computer Science] journal. We have made this choice so that, if a group report is considered to be of sufficient quality, we can pursue publication of this report in ''PeerJ Computer Science'' as smoothly as possible. If there are formatting or detail questions that are not covered here, visit the [https://peerj.com/about/author-instructions/ Instructions for Authors] and follow their guidance.

* The report should be written with contributions from all group members.
* Submit as ''.docx'' or ''.pdf'' file.

=== Style Sheet ===

Use the following guidelines when formatting your report:
* 2.54 cm (1 in) margins on all sides
* Double-spaced
* 12 point Times/Times New Roman font
* Number the pages on the lower-right corner
* Use left justification (“jagged” on the right side)

=== Cover Page ===

Include the following information in a standalone cover page:
* A descriptive title for your project
* The names of the team members
* The course number and title of the class
* The date of submission

=== Abstract ===

Provide an abstract of no more than 500 words.

=== Introduction ===

The introduction gives the background information necessary to understand your report. The introduction should be in the form of a logical argument that “funnels” from broad to narrow:

<gallery mode="nolines" widths=322px heights=256px>
Funnel.png
</gallery>

* States importance of the problem
Why is this species important?
* States what is known about the problem
- Give an overview of what is known about your species' genome from your [[Week 11|journal club outline and presentation]].
- Introduce the DNA microarray experiment that was performed on your species.
* States what is unknown about the problem
You want to analyze the data with GenMAPP/MAPPFinder, but can't because there is no Gene Database for your species.
* States clues that suggest how to approach the unknown
Introduce XMLPipeDB and GenMAPP Builder as the answer to this problem.
* States the question the paper is trying to address
In this case you want to discover new information about the microarray data using GenMAPP.

=== Materials & Methods ===

This section will summarize the entire workflow for the project. This needs to be a '''''narrative description''''' of what your team actually did.
# Download the UniProt XML proteome set and GOA (GO association) files for your species.
#* Note the date of download and the version of the files.
# Download GO terms from in the OBO-XML format.
#* Note the date of download and the version of the files.
# Create the GenMAPP Builder tables in PostgreSQL.
# Load files into PostgreSQL database via GenMAPP Builder.
# Export into a GenMAPP Gene Database.
# Inspect/vet/validate Gene Database.
# Prepare microarray data (organize, normalize, perform statistical analysis)
# Run GenMAPP using the Gene Database.
#* Microarray data (import using Expression Dataset Manager)
#* Run MAPPFinder analysis
#* Place genes on MAPP and draw pathway

=== Results ===

This section will summarize the results of the project. This section will include figures, tables, and '''''narrative description''''' of the results shown in those figures and tables. You should include:
* Gene Database Schema
* Gene Database Testing Report on final version of Gene Database
* Report on quantity and identity of gene IDs that did not make it into the database
*# Gene IDs that were not in the XML source at all
*# Gene IDs that were in the XML source but did not get imported into Postgres
*# Gene IDs that were in Postgres but did not get exported to the GenMAPP Gene Database
* Report on what changes need to be made to the GenMAPP Builder code in order to to accommodate the second and third type of missing gene IDs
** If your team was able to make such changes, report on the results of these changes
* Report results of the DNA microarray analysis
** How many genes were significantly increased and decreased in the dataset? (including your criteria for a significant increase or decrease)
** Table of MAPPFinder results (from .xls)
** GenMAPP MAPP of a pathway relevant to your results

=== Discussion ===

* How well did the GenMAPP Builder process work for your species (just comment on the technical aspects here, you will discuss the teamwork/process aspects in your individual assessment).
* Discuss the statistical analysis and MAPPFinder results for your microarray dataset. Compare it to what was reported in the original paper from which you got the microarray data.

=== Conclusions ===

Summarize the overall project and your findings. How closely do your findings correspond to the original study? Are there significant differences? Did you discover anything new?

=== Acknowledgments ===

''(Optional):'' Write a short paragraph acknowledging the assistance of anyone who is not a member of your team.

=== References ===

Cite and list your references using [https://peerj.com/about/author-instructions/#reference-format PeerJ’s recommended reference format]. This format is very similar to APA style and should feel familiar if you have written research papers before.

To minimize busy work, the PeerJ website includes links to downloadable style files for [https://www.zotero.org/styles/?q=peerj Zotero] and [http://endnote.com/downloads/style/peerj EndNote], if you use either system for managing and rendering references.

== Group Files and Datasets ==

* GenMAPP Gene Database for assigned species (''.gdb'')
* ReadMe file to accompany the Gene Database (''.pdf'')
** Sample ReadMe in Word format: [[Media:ReadMe_Vc-Std_External_20131122.zip | ReadMe_Vc-Std_External_20131122.zip]]
** Include Gene Database Schema diagram in ReadMe
*** Sample schema in Adobe Illustrator format: [[Media:Vibrio_schema_20101022.zip | Vibrio_schema_20101022.zip]]
* Gene Database Testing Report for final submitted Gene Database (print from wiki to ''.pdf'' file)
* Processed and analyzed DNA microarray dataset (''.xls'')
* GenMAPP Expression Dataset file (''.gex'')
* Filtered MAPPFinder Results (''.xls'')
* Sample MAPP file of a relevant biological pathway for your species (''.mapp'')
* PowerPoint presentation (''.ppt'', given on Tuesday, December 15)

== Individual Assessment and Reflection ==

Each person on the team will complete an assessment and reflection ''individually''. If you are comfortable with making this assessment publicly available, you may write it up as a wiki page. If you prefer to communicate your assessment privately, then email this to both Drs. Dahlquist and Dionisio.

=== Statement of Work ===

* Describe exactly what you did on the project.
* Provide references or links to artifacts of your work, such as:
** Wiki pages
** Other files or documents
** Code or scripts

=== Assessment of Project ===

* Give an objective assessment of the success of your project workflow and teamwork.
* What worked and what didn't work?
* What would you do differently if you could do it all over again?
* Evaluate the Gene Database Project and Group Report in the following areas:
*# Content: What is the quality of the work?
*# Organization: Comment on the organization of the project and of your group's wiki pages.
*# Completeness: Did your team achieve all of the project objectives? Why or why not?

=== Reflection on the Process ===

* What did you learn?
** With your head (biological or computer science principles)
** With your heart (personal qualities and teamwork qualities that make things work or not work)?
** With your hands (technical skills)?
* What lesson will you take away from this project that you will still use a year from now?

{{Gene Database Project Links}}

[[Category:Group Projects]]

Template:Gene Database Project Links

2015-11-02T07:26:06Z

Dondi: Add placeholders for team links.

{| class="wikitable" style="width: 100%; text-align: center"
! colspan="8"|Gene Database Project Links
|-
! rowspan="2"|[[Gene Database Project|Overview]]
! rowspan="2"| [[Gene Database Project Deliverables|Deliverables]]
| rowspan="2"|[https://peerj.com/about/author-instructions/#reference-format Reference Format]
! Guilds
| [[Project Manager]]
| [[GenMAPP User]]
| [[Quality Assurance]]
| [[Coder]]
|-
! Teams
| [[Team 1]]
| [[Team 2]]
| [[Team 3]]
| [[Team 4]]
|}

Main Page

2015-11-02T07:24:14Z

Dondi: /* Final Group Project */ Substitute links with template.

__NOTOC__
<div style="text-align: center; font-size: 175%">Loyola Marymount University</div>
<div style="text-align: center; font-size: 250%; line-height: 1.25em">'''BIOL/CMSI 367-01: Biological Databases'''</div>
<div style="text-align: center; font-size: 175%; margin-bottom: 0.85em">Fall 2015</div>

<div class="toc-container">__TOC__</div>

== Announcements ==

* The Microsoft Excel portion of the ID investigation protocol has been added to the wiki: [[Using Microsoft Excel to Compare ID Lists]]. The link has also been added to the course schedule below for Week 9.
:Please note that this version is similar, but not identical, to the process shown in class ([[User:Dondi|Dondi]] misremembered the function to use; it was supposed to be <code>match</code>, as shown in the wiki page, and not <code>lookup</code>). Furthermore, instructions for exporting from PostgreSQL (you might recall that the numbers did not match there either) are included. —''[[User:Dondi|Dondi]] ([[User talk:Dondi|talk]]) 18:49, 29 October 2015 (PDT)''
*'''''TriBeta Halloween Party and Costume Contest:''''' Friday, October 30, 6:00 PM, LSB Terrace, $3 (to cover food). RSVP to ralkhaya at lion dot lmu dot edu. Come see if you can beat me in the costume contest! ''— [[User:Kdahlquist|Kdahlquist]] ([[User talk:Kdahlquist|talk]]) 09:46, 28 October 2015 (PDT)''
* An initial round of feedback to the [[Week 6]] assignment has been posted to your respective user talk pages. The database part is not finished, but I figured some feedback and basics on the best practice stuff will be useful as you prepare your [[Week 8]] work (as in, get used to making the best practices routine so you can avoid nagging deductions week after week). —''[[User:Dondi|Dondi]] ([[User talk:Dondi|talk]]) 21:30, 25 October 2015 (PDT)''
* To report issues with the configuration of the computers in Seaver 120, please record it on [[Talk:Software_Configuration | this wiki page.]] The complete list of software that should be installed for this course is found [[Software Configuration | here]]. Note that this configuration is Windows-specific. Some software may not be available for a Mac or may look differently.

=== Upcoming Seminars ===

* '''Chemistry & Biochemistry Department Seminar:''' Thursday, October 29, 11:50 AM, Seaver 200, Presentation by first candidate for the Biochemistry Faculty Search.
* '''Biology Department Seminar:''' Friday, October 30, 2:00 PM, LSB Auditorium, Dr. Ashley Robart, LMU Biology, ''MALE QUALITY INFLUENCES PARENTAL CARE, BUT NOT FEMALE COMPETITION, IN A MONOGAMOUS, BIPARENTAL FISH.''
* '''Noyce Poster Session:''' Friday, October 30, 4:00 – 6:00 PM, Hannon Courtyard of the LSB (area just outside the west entrance). The NOYCE Summer Interns will be presenting the posters they made highlighting the research they have done. The Poster Session will be a fantastic opportunity to talk to the presenters and ask them all about their research! To receive the extra credit, fill out one of the "seminar" sheets available according to the instructions and turn it in to a professor before you leave the poster session.
* '''Chemistry & Biochemistry Department Seminar:''' Thursday, November 5, 11:50 AM, Seaver 200, Presentation by second candidate for the Biochemistry Faculty Search.
* '''Biology Department Seminar:''' Friday, November 6, 2:00 PM, LSB Auditorium, Dr. Wes Dowd, LMU Biology.
* '''Chemistry & Biochemistry Department Seminar:''' Tuesday, November 10, 11:50 AM, Seaver 200, Presentation by third candidate for the Biochemistry Faculty Search.
* '''Chemistry & Biochemistry Department Seminar:''' Thursday, November 12, 11:50 AM, Seaver 200, Presentation by fourth candidate for the Biochemistry Faculty Search.
* '''Biology Department Seminar:''' Friday, December 4, 2:00 PM, LSB Auditorium, Dr. Deepa Dabir, LMU Biology.



== Instructors ==
* [[User:Kdahlquist|Kam D. Dahlquist]]
* [[User:Dondi|John David N. Dionisio]]

== Students ==

{| style="margin-left: auto; margin-right: auto"
| style="padding-right: 1em" | [[User: Malverso | Mary Alverson]]
| style="padding-right: 1em" | [[User:Rlegaspi | Ron Legaspi]]
| style="padding-right: 1em" | [[User:Troque | Trixie Roque]]
| style="padding-right: 1em" | [[User:Jwoodlee | Jake Woodlee]]
|-
| style="padding-right: 1em" | [[User:Nanguiano | Nicole Anguiano]]
| style="padding-right: 1em" | [[User:Blitvak | Brandon Litvak]]
| style="padding-right: 1em" | [[user:Msaeedi23 | Mahrad Saeedi]]
| style="padding-right: 1em" | [[User:Kwyllie | Kevin Wyllie]]
|-
| style="padding-right: 1em" | [[User:Bklein7 | Brandon Klein]]
| style="padding-right: 1em" | [[User:Lenaolufson | Elena Olufson]]
| style="padding-right: 1em" | [[User:Emilysimso | Emily Simso]]
| style="padding-right: 1em" | [[User:Eyanosch | Erich Yanoschik]]
|-
| style="padding-right: 1em" | [[User:Jkuroda | Josh Kuroda]]
| style="padding-right: 1em" | [[User:Vpachec3 | Veronica Pacheco]]
| style="padding-right: 1em" | [[User:Anuvarsh | Anindita Varshneya]]
| style="padding-right: 1em" | [[User:Kzebrows | Kristin Zebrowski]]
|}

{{ Gene Database Project Links }}

== Course Schedule ==

The current course schedule is shown below. In addition to the listed in-class agendas, the following guidelines are also notable:

* Unless otherwise stated on the schedule, your weekly journals/assignments are due on Tuesdays at midnight (Monday night/Tuesday morning). In cases where subsequent revisions or corrections will be accepted, the instructors will provide feedback and submission deadlines on a per-assignment basis.
* Reading assignments should be completed ''before'' coming to class.
* Dr. Dionisio’s office hours ({{ Dondi Office Hours }}) can be viewed as unofficial lab sessions: use them for IT help or desktop support

=== Part 1: Building Blocks (Genetic Code and Manipulating Text) ===

{| class="schedule-table"
! #
! Date
! Reading
! Agenda
! Journal
|-
! 1
| Tuesday<br/>9/1/2015
|
| Syllabus walkthrough
* ''my.cs.lmu.edu'' account: Visit https://secure.cs.lmu.edu to get a computer science account if you don’t already have one
* Class wiki account: Set up a username and password for this wiki site

Pairwise interviews (report back after each bullet)
* Basic acquaintance information
* Like a cell/not like a cell; like a computer/not like a computer

| rowspan="2" | [[Week 1]]

(due at midnight 9/8)

[[Class_Journal_Week_1 | Class Journal Week 1]]
|-
!
| Thursday<br/>9/3/2015
|
[http://dl.acm.org/ft_gateway.cfm?id=1378711 ''Voices of Computing''] and [http://dl.acm.org/ft_gateway.cfm?id=1272529 ''Computing is a Natural Science''] by Peter Denning (these links should be accessed from within LMU to get the full article)

[https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&mode=reset Chapter 1 of ''On Becoming a Biologist''] by John Janovy, Jr. (on MyLMUConnect)
| Discuss the [[Week 1]] assignment (accounts should be ready to wiki at this point)
* Quick wiki overview
* User wiki page setup

{{Course Schedule Divider}}
|-
! 2
| Tuesday<br/>9/8/2015
| [http://www.ncbi.nlm.nih.gov/books/NBK21134/#A5234 Ch. 1.1 ''Genomes 2'']

[http://www.nature.com/nature/dna50/watsoncrick.pdf Watson & Crick (1953)]

[http://www.nature.com/nature/dna50/franklingosling.pdf Franklin & Gosling (1953)]

| DNA
* DNA structure and Genetic Code
* [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&content_id=_1103141_1 Slides shown in class on MyLMU Connect]
| rowspan="2" | [[Week 2]]

(due at midnight 9/15)

[[Class Journal Week 2]]
|-
!
| Thursday<br/>9/10/2015
| On [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&mode=reset MyLMUConnect]:

Nirenberg (2004) ''Deciphering the Genetic Code''

Kaji & Kaji (2004) ''Setting the Record Straight''

Moody (2004) ''Digital Code of Life'', Ch. 1

Hayes (2004) ''Ode to the Code''

[http://www.ncbi.nlm.nih.gov/books/NBK21121/ Brown, T.A. (2002) ''Genomes 2'', Ch. 3, especially section 3.3.2]
| Central dogma
* [http://www.youtube.com/watch?v=u9dhO0iCLww Berkeley translation video]
* This [http://www.dnai.org/a/index.html web site] has links to animations of the replication, transcription, and translation processes.
* Slides shown in class
* [[Media:BIOL367_Fall2015_GeneticCode-GeneExpression.pdf | Genetic Code-Gene Expression Pencil Exercise]]

{{Course Schedule Divider}}
|-
! 3
| Tuesday<br/>9/15/2015
| [http://www.bloomberg.com/graphics/2015-paul-ford-what-is-code/ Ford (2015) ''What is Code?'']
* Emphasis on first two sections (“The Man in the Taupe Blazer” and “Let’s Begin”), but feel free to read further

[http://myweb.lmu.edu/dondi/share/intro/wheres-my-stuff.pdf Where's my Stuff?]

[[Introduction to the Command Line]]
| An overview of computers, networks, files, and databases
* Useful help resource: http://explainshell.com
* Manipulating DNA sequences on the command line
* [[Using the XMLPipeDB Match Utility]] 
| rowspan="2" | [[Week 3]]

(due at midnight 9/22)

[[Class Journal Week 3]]
|-
!
| Thursday<br/>9/17/2015
|
| Command line lab session
* Hands-on practice
* Discuss [[Week 3]] assignment tasks

{{Course Schedule Divider}}
|-
! 4
| Tuesday<br/>9/22/2015
| [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&mode=reset Moody (2004) Chapter 6 (on MyLMU Connect)]

[http://www.nature.com/news/2010/100623/pdf/4651000a.pdf Science After the Sequence]

[http://www.nature.com/news/2010/100623/full/4651000a.html Nature Special: Human Genome at 10]

[http://nar.oxfordjournals.org/content/43/D1/D1.full Introduction to ''NAR'' Database Issue]

[http://scienceblogs.com/digitalbio/2015/01/30/bio-databases-2015/ Commentary on ''NAR'' Database Issue]

| Genome sequencing and annotation

Introduction to biological databases
* [https://mylmuconnect.blackboard.com/bbcswebdav/pid-1146580-dt-content-rid-1944592_1/courses/M_46692.201530/20150922_HGP_IntroBiolDB.pdf Slides shown in class (MyLMU Connect)]
* [http://www.dataone.org/sites/all/documents/L01_DataManagement.pptx DataONE: Why Data Management]
* [http://www.dataone.org/sites/all/documents/L02_DataSharing.pptx DataONE: Data Sharing]
| rowspan="2" | [[Week 4]]

(due at midnight 9/29)

[[Class Journal Week 4]]
|-
!
| Thursday<br/>9/24/2015
| [[More Text Processing Features]]

[[How to Read XML Files]]
| Command line lab session
* Review assignment issues
* Discuss [[Week 4]] computer portion


{{Course Schedule Divider}}
|-
! 5
| Tuesday<br/>9/29/2015
| [http://myweb.lmu.edu/dondi/share/db/relational1.pdf The Relational Data Model: Structure]

[http://myweb.lmu.edu/dondi/share/db/sql.pdf An Overview of SQL]

| Introduction to relational databases
* Case study from one of the NAR databases: http://www.rhesusbase.org 
* Relating XML structure to relational structure

Test your understanding: http://xkcd.com/327/
| rowspan="2" | [[Week 5]]

(due at midnight 10/6)

[[Class Journal Week 5]]
|-
!
| Thursday<br/>10/1/2015
| Additional background and details can be found in Chapters 1, 2, and 6 of ''A First Course in Database Systems'' (on reserve at the library).
| SQL and working with relational databases continued
* [[PostgreSQL Tutorial]]

{{Course Schedule Divider}}
|-
! style="background-color: #ddf" | 6
| style="background-color: #eef; border: none" | Tuesday<br/>10/6/2015
| style="background-color: #eef; border: none" |
| style="background-color: #eef; border: none" | President Synder’s Inauguration Day
| rowspan="2" | [[Week 6]]

(due at midnight 10/13)

[[Class Journal Week 6]]
|-
!
| Thursday<br/>10/8/2015
|
| Database presentations
# Nicole, Erich, Kristen, Josh
# Mary, Emily, Brandon L., Kevin
# Lena, Jake, Anu, Veronica
# Ron, Mahrad, Brandon K., Trixie

[[Presentation Rubric]]

[[Media:PresentationGuidelines.ppt | PowerPoint Guidelines]]

|}

=== Part 2: Going Deeper (Gene Expression Data and Relational Databases) ===

{| class="schedule-table"
! #
! Date
! Reading
! Agenda
! Journal
|-
! 7
| Tuesday<br/>10/13/2015
| [http://www.ncbi.nlm.nih.gov/books/NBK26818/#A1633 Alberts ''et al''. (2002) ''Molecular Biology of the Cell'', Ch. 8: Microarrays]

[http://www.nature.com/ng/journal/v21/n1s/full/ng0199supp_33.html Brown & Botstein (1999) "Microarrays" ''Nature Genetics'']

Campbell & Heyer Chapter 4 (on [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1 MyLMU Connect])

DeRisi et al. (1997) [http://www.sciencemag.org/content/278/5338/680.full ''Science'' 278: 680-686.]

| Introduction to DNA microarray data, Gene Ontology, and GenMAPP/MAPPFinder
* [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&content_id=_1103141_1 Slides on MyLMU Connect]
* [http://www.dataone.org/sites/all/documents/L04_DataEntryManipulation.pptx DataONE: Data Entry and Manipulation]
* [http://www.bio.davidson.edu/courses/genomics/chip/chip.html Microarray animation]

| rowspan="2" | [[Week 7]]

(due at midnight 10/20)

[[Class Journal Week 7]]
|-
!
| Thursday<br/>10/15/2015
| [http://www.nature.com/nature/journal/v417/n6889/full/nature00778.html Merrell et al. (2002) "''Vibrio cholerae'' microarray data" ''Nature'']
| [http://www.openwetware.org/wiki/BIOL398-01/S10:Sample_Microarray_Analysis_Vibrio_cholerae DNA Microarray Analysis activity part 1]

Test your understanding: [http://xkcd.com/882/ http://xkcd.com/882/]

{{Course Schedule Divider}}
|-
! 8
| Tuesday<br/>10/20/2015
| [http://www.nature.com/ng/journal/v25/n1/full/ng0500_25.html Ashburner et al. (2000) "Gene Ontology" ''Nature Genetics'']

[http://www.genomebiology.com/content/4/1/R7 Doniger et al. (2003) "MAPPFinder" ''Genome Biology'']
| [http://www.openwetware.org/wiki/BIOL367/F10:GenMAPP_and_MAPPFinder_Protocols DNA Microarray Analysis activity part 2]
| rowspan="2" | [[Week 8]]

(due at midnight 10/27)

[[Class Journal Week 8]]
|-
!
| Thursday<br/>10/22/2015
|
| Work session to complete DNA Microarray Analysis [http://www.openwetware.org/wiki/BIOL398-01/S10:Sample_Microarray_Analysis_Vibrio_cholerae part 1] and [http://www.openwetware.org/wiki/BIOL367/F10:GenMAPP_and_MAPPFinder_Protocols part 2]

{{Course Schedule Divider}}
|-
! 9
| Tuesday<br/>10/27/2015
|
|
* Introduction to XMLPipeDB: export a Gene Database for ''Vibrio cholerae''
** [[Running GenMAPP Builder]]
* [http://www.opensource.org/ Open Source] review
* Group requests due midnight Saturday, October 31
| rowspan="2" | [[Week 9]]

(due at midnight 11/3)

[[Class Journal Week 9]]
|-
!
| Thursday<br/>10/29/2015
| [http://nar.oxfordjournals.org/content/40/D1/D565.full Dimmer et al. (2012) "UniProt-GOA Database" ''Nucleic Acids Research'']

[http://www.nature.com/nature/journal/v406/n6795/full/406477A0.html Heidelberg et al. (2000) "''Vibrio cholerae'' Genome Sequence" ''Nature'']
|
Integrity analysis of Vibrio Gene Database
* [[How Do I Count Thee? Let Me Count The Ways]]
* [[Using Microsoft Excel to Compare ID Lists]]
* [[Gene Database Testing Report Sample | Complete your Gene Database Testing Report for ''Vibrio cholerae'']]
|}

=== Part 3: Integrating for Research (Gene Database Project) ===

{| class="schedule-table"
! #
! Date
! Reading
! Agenda
! Journal
|-
! 10
| Tuesday<br/>11/3/2015
|
|
* Annotated bibliography of genomics papers for your species
* ''Course LibGuide:'' http://libguides.lmu.edu/BIOL367
** Select genome paper for journal club
** Select microarray paper
| rowspan="2" | [[Week 10]]

(due at midnight 11/10)


|-
!
| Thursday<br/>11/5/2015
|
| Begin [[Gene_Database_Project | group projects]]

{{Course Schedule Divider}}
|-
! 11
| Tuesday<br/>11/10/2015
|
| Guild meetings and work session
* Export of Gene Database for your group's species
* Document ID types for your species
* Begin DNA microarray analysis
| rowspan="2" | [[Week 11]]

(due at midnight 11/17)
|-
!
| Thursday<br/>11/12/2015
|
|

* Status report
* Work session

{{Course Schedule Divider}}
|-
! 12
| Tuesday<br/>11/17/2015
|
| First Set of Journal Club Presentations
# TBD
# TBD
# TBD
# TBD

[[Presentation Rubric]]

[[Media:PresentationGuidelines.ppt | PowerPoint Guidelines]]
| rowspan="2" | [[Week 12]]

(due at midnight 11/24)

|-
!
| Thursday<br/>11/19/2015
|
|
* Status report
* Work session

{{Course Schedule Divider}}
|-
! 13
| Tuesday<br/>11/24/2015
|
| Second Set of Journal Club Presentations
# TBD
# TBD
# TBD
# TBD

[[Presentation Rubric]]

[[Media:PresentationGuidelines.ppt | PowerPoint Guidelines]]
| '''''No Week 13 assignment'''''

|-
! style="background: #ded" |
| style="background: #efe; border: none" | Thursday<br/>11/26/2015
| style="background: #efe; border: none" |
| style="background: #efe; border: none" | Thanksgiving—no class
| style="background: #efe; border: none" |
{{Course Schedule Divider}}
|-
! 14
| Tuesday<br/>12/1/2015
|
|
* Status report
* Work session
| rowspan="2" | [[Week 14]]

(due at midnight 12/8)
|-
!
| Thursday<br/>12/3/2015
|
|
* Status report
* Work session
{{Course Schedule Divider}}
|-
! 15
| Tuesday<br/>12/8/2015
|
|
* Status report
* Work session
| rowspan="2" | [[Week 15]]

(due at midnight 12/15)
|-
!
| Thursday<br/>12/10/2015
|
|
* Status report
* Work session
{{Course Schedule Divider}}
|-
! style="background: #edd" | F
| style="background: #fee; border: none" | Tuesday<br/>12/15/2015
| style="background: #fee; border: none" |
| style="background: #fee; border: none" | '''Final project presentations 2:00-4:00 PM'''
| style="background: #fee; border: none" |
|-
!
| Friday<br/>12/18/2015 4:30 PM
|
| '''Project deliverables due 4:30 PM'''
|
|}

== Course Information ==

=== Instructors ===

==== [[User:Kdahlquist | Kam D. Dahlquist, Ph.D.]] ====
: http://myweb.lmu.edu/kdahlqui
: '''Phone:''' (310) 338-7697
: '''E-mail:''' kdahlquist at lmu dot edu
: '''Office:''' Life Sciences Building 289
: '''Office Hours:''' Tuesdays 9:00-11:00 AM, Wednesdays 1:30-3:30 PM and by appointment. I keep a sign-up sheet on my office door to facilitate appointment-making.

==== [[User:Dondi|John David N. Dionisio, Ph.D.]] ====
: http://myweb.lmu.edu/dondi
: '''Phone:''' (310) 338-5782
: '''E-mail:''' dondi at lmu dot edu
: '''Office:''' Doolan 106
: '''Office Hours:''' {{Dondi Office Hours}}

=== Prerequisites/Recommended Background ===

To take this course, you must have upper division standing in the Seaver College of Science and Engineering. Otherwise, there are no strict prerequisites. Backgrounds in biology and computer science, as well as prior experience with database or information management applications, may be helpful but not necessary.

=== Class Meetings and Attendance ===
TR 2:40–3:55pm, Seaver 120 

This is a hands-on, participatory course, thus attendance at all class meetings is required. Each student is allowed two “sick” days (automatically excused absences) during the semester. Further unexcused absences from class will result in a 5% deduction from the overall course grade for each absence. Every effort should be made to attend class on oral presentation days as the content of that day's class is dependent on student participation. Unexcused absences from an oral presentation will result in a grade of zero for the presentation. The instructors should be notified as soon as possible, electronically or by phone, of the reasons for all absences.

Note that the last day to add or drop a class without a grade of W is September 4. The withdrawal or credit/no-credit status deadline is November 6.

=== Mutual Responsibilities ===

This course is designed to foster your development as a biologist and computer scientist and to give you an authentic research experience. We will be engaged together in discovering, examining, and practicing the personal qualities, technical skills, and community standards of the scientific community. While you are ultimately responsible for your own learning, you are not alone. Our class constitutes a team where we will be learning from each other. The role of the instructors is to provide the expert coaching to support and assist you on your journey. All of the exercises, readings, assignments, and policies detailed below have been designed with this purpose in mind.

=== Classroom Conduct ===

We are all responsible for maintaining a classroom and laboratory environment that is safe and conducive to learning. As such, we will observe the following:

# As an LMU Lion, by the [http://studentaffairs.lmu.edu/media/lmustudentaffairs/administration/judicialaffairs/Lion%27s%20Code.pdf Lion’s code], you are pledged to join the discourse of the academy with honesty of voice and integrity of scholarship and to show respect for staff, professors, and other students.
# You are responsible for your own learning and for being a good class citizen.
# Class will start promptly on time.
# You are expected to come to class having done the assigned reading and preparatory work so that you are ready to participate in discussions and to perform the laboratory exercises.
# You are expected to bring the required materials to each class session.
# Cell phones, pagers, and other communication or music devices must be turned off and put away out of sight. Your own laptop and/or tablet may be used to conduct the class exercises; however, if they are being used for other purposes and become distracting to you or others, you will be asked to put them away.
# All students are governed by [http://studentaffairs.lmu.edu/media/lmustudentaffairs/administration/judicialaffairs/Loyola-Marymount-University-Community-Standards-2015-2016.pdf LMU Community Standards Publication]. Disruptive behavior which is persistent or significantly interferes with classroom activities may be subject to disciplinary action. A student may be referred to the Office of Student Judicial Affairs if his or her behavior constitutes a violation of the conduct code.

=== Course Website ===

This is the course web site and wiki, hosted by http://xmlpipedb.cs.lmu.edu/biodb/fall2015/. You will need to [https://xmlpipedb.cs.lmu.edu/biodb/fall2015/index.php?title=Special:UserLogin&returnto=Main+Page&type=signup register] to be able to edit the wiki and complete coursework. Updates to the course schedule and electronic copies of all handouts, assignments, and readings will be posted to this site. You will also use the site to keep an electronic lab notebook/journal for the course. In addition, students have been automatically enrolled in [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&mode=reset BIOL/CMSI 367-01 on MyLMUConnect]. The two cross-listed sections are using the same site, which is listed as "Biological Databases Fall 2015" with no department or course number. The MyLMUConnect site will be used for materials that cannot be made public on this wiki, including grades.

=== E-mail Communication ===

At times we will communicate with the entire class using campus e-mail systems, so it is essential that you regularly check your lion.lmu.edu e-mail address or forward your lion account e-mail to your preferred e-mail address. Messages sent to the instructors at night or on the weekend will be answered the next school day. Please CC both instructors on all e-mail messages related to this class.

=== Required Materials ===

==== Texts ====

There is no required text to purchase for the course; materials will be put on reserve at Hannon Library or will be available online on this wiki or the MyLMUConnect site. Specific reading assignments are given on the course schedule and should be completed before coming to class.
* This text has been placed on reserve at the library: Jeffrey Ullman and Jennifer Widom, ''A First Course in Database Systems'', Third Edition. Prentice Hall, 2008.
* Assorted handouts, articles, and sample code will be distributed throughout the semester.
* Additional information is also available on the web; do not hesitate to look for further sources of information regarding the concepts, techniques, tools, and paradigms that we will discuss.

==== Materials (must be brought to each class meeting) ====

* 3-ring binder with all course handouts
* Pen, pencil, extra paper
* USB flash drive to store data
* [https://xmlpipedb.cs.lmu.edu/biodb/fall2015/index.php/Special:UserLogin/signup Account for this wiki]
* [https://secure.cs.lmu.edu/ Computer Science account]
* [http://www.github.com GitHub account]

=== Course Description ===

The disciplines of biology and computer science come together in bioinformatics, where computational tools are needed to manage and analyze the flood of data coming from new genomics technologies. Biological databases form a significant part of this young and exciting field. This course introduces students to both the biology and computer science expertise needed to understand, use, and develop biological databases. Biology topics include the fundamentals of genetics, molecular biology, and biochemistry needed to understand the data stored in biological databases, as well as the biotechnologies used to gather these data in a high-throughput manner. Computer science topics include what biological databases are, why they are important (and needed), and the challenges that arise in compiling them effectively. Biology and computer science lectures on topics that are relevant to biological databases are coupled with hands-on experience with a variety of software packages ranging from bioinformatics utilities to general-purpose database and software development tools. After learning how to use a biological database, students will be asked to build one of their own.

==== Course Objectives and Learning Outcomes ====

This course is built upon L. Dee Fink’s ''taxonomy of significant learning'', as applied to biological databases. Long after the course concludes, our hope is that:
* You understand how biological information is encoded in the genome and can apply this knowledge to a variety of biological tasks and problems
* You understand the core concepts, structure, and functions of a database, ranging from individual files to a full relational database management system, and can perform useful tasks with such data
* You show discipline and proficiency in day-to-day science and engineering best practices, such as maintaining journals and notebooks, managing your files and code, and critically evaluating scientific and technical information
* You recognize and care about how the biological and technological issues presented in this course relate to and affect society, our daily lives, and ourselves
* You have some skills and tools for “leaving your comfort zone,” flourishing outside of it, and learning more about biology and computer science on your own
* You learn how to communicate and work effectively with colleagues from different disciplines

==== University Core Curriculum ====

This course fulfills the following requirements in the University Core Curriculum:
* Integrations: Interdisciplinary Connections
* Upper Division Information Literacy Flag
* Upper Division Oral Communication Flag

=== Course Work and Grading ===

Letter grades are determined as follows: ≥ 90% gets an A– or better; ≥ 80% gets a B– or better; ≥ 70% gets a C– or better. The instructors may curve grades upward based on qualitative considerations such as degree of difficulty, effort, class participation, time constraints, and overall attitude throughout the course. Grades are never curved downward. Current grades will be made available at MyLMU Connect (the system formerly known as Blackboard).

Your work in this course will be assessed in five areas:

{| style="border: 1px dashed rgb(47, 111, 171); background: rgb(249, 249, 249); margin: 1em 0; padding: 1em" cellpadding="5ex"
| Weekly electronic lab notebook/journal assignments, individual (10 points each)
| style="text-align: right" | 140
| points
|-
| Weekly electronic lab notebook/journal assignments, shared (3 points each)
| style="text-align: right" | 42
| points
|-
| Oral presentations
| style="text-align: right" | 105
| points 
|-
| Information literacy (additional points added to two journal assignments)
| style="text-align: right" | 24
| points 
|-
| Final Project Deliverables (including written report)
| style="text-align: right" | 175
| points
|-
| '''Total'''
| style="text-align: right" | '''486'''
| '''points'''
|}

==== Electronic Laboratory Notebook/Journal ====

One of the most important skills you can develop as a scientist is keeping an excellent laboratory notebook. For computational research, the equivalent of the biology paper-based lab notebook is documentation of your “workflow”. For this course you will practice documentation skills by keeping an electronic lab notebook or journal. The technology we will use is this wiki, that we will create and edit during the semester. You will create an individual user page and make weekly entries that the instructors will read and grade. You will use the wiki to complete the assignments as well. The following guidelines apply:
* Your weekly journal entry is typically due every midnight on Tuesday PST (''Monday night/Tuesday morning''); consult the schedule for specific due dates for each assignment.
* Each weekly assignment has an individual component and a shared component. You will earn 10 points per weekly submission for the individual journal entry and 3 points per submission for the shared journal entry. '''''Late journal entries will be accepted up to one week later for up to half credit.'''''
* The instructors will read and comment on how to improve your journal entries on your user talk pages.
* Depending on the type of assignment for that week, you may be given the opportunity to make improvements to previous journal entries as the semester progresses.
* Generally, your journal entries will consist of:
** Your electronic laboratory notebook and other documentation for hands-on exercises and projects
** Answers to any specific questions posed in the exercise
** Reflection on your learning
* For most weeks in the semester, you will be assigned a "homework partner" from the complementary biology or computer science discipline. You will be expected to consult with your partner, sharing your domain expertise, in order to complete the assignment. However, unless otherwise stated, each partner must submit his or her own work as the individual journal entry (direct copies of each other's work is not allowed).

==== Oral Presentations ====

You will give three oral presentations in this course. The first two will be in the format of a “Journal Club” presentation where students will present and lead discussion of research articles from the primary literature. The third will be a research presentation on your final project. '''''Because that day’s class content is dependent upon each student being ready to present and lead discussion, late journal club presentations will not be accepted. An unexcused absence from a journal club presentation will result in a grade of zero for the presentation.'''''

==== Final Group Project ====

{{ Gene Database Project Links }}

In addition to the research presentation, the culmination of your final project will be the preparation of a written laboratory report in the style of a manuscript that could be submitted to a peer-reviewed journal, specifically [https://peerj.com/computer-science/ PeerJ Computer Science]. Specific instructions are posted [[Gene Database Project Deliverables|here]]. '''''The Final Written Report cannot be accepted any later than Friday, December 18 at 4:30 PM. The Final Written Report must be completed to receive a passing grade in the course.'''''

==== Extra Credit ====

Students may accumulate up to 12.5 points toward their final grade in extra credit by attending Department seminars in Biology or Electrical Engineering & Computer Science and completing the seminar sheets. Each seminar attended is worth 2.5 points with up to 5 seminars (12.5 points) total. You must attend the entire seminar from start to finish and personally turn in your seminar sheet to a faculty member at the end of the seminar.

Certain, non-Biology/Computer Science Department seminars may be approved in advance for extra credit at the instructors’ discretion. To receive credit for these seminars, you must turn in a one-page '''''hard copy''''' of your summary of the seminar '''''in class, within one week''''' of the date of the seminar or they will not count as extra credit.

=== Work Load Expectations ===

In line with [http://www.lmu.edu/Assets/LMU+Credit+Hour+Policy_Final.pdf LMU’s Credit Hour Policy], the work load expectation for this course is that for every one hour (50 minutes) of classroom instruction, you will complete a minimum of two hours of out-of-class student work each week. This is a 3-unit course with 3 hours (150 minutes) of instruction per week. Thus the expectation is that you will complete 6 hours of work outside of class per week.

=== University Policy on Academic Honesty ===

Loyola Marymount University expects high standards of honesty and integrity from all members of its community. All students are expected to follow the LMU honor code. As stated in the LMU Undergraduate Bulletin, “Examples of academic dishonesty include, but are not limited to, the following: all acts of cheating on assignments or examinations, or facilitating other students’ cheating; plagiarism; fabrication of data, including the use of false citations; improper use of non-print media; unauthorized access to computer accounts or files or other privileged information and improper use of Internet sites and resources.” Attempting to gain extra credit for a seminar that you did not attend or assisting another student to do this are examples of academic dishonesty and will be considered a violation of the Honor Code. [http://bulletin.lmu.edu/content.php?catoid=6&navoid=250#LMU_Honor_Code_and_Process Click here for an online version of the LMU Honor Code and Process].

You are required to sign the Honor Code Agreement for this course.

==== Academic Honesty Resources ====
* [[Media:BIOL367_Fall2015_HonorCodeAgreement.pdf|Honor Code Agreement]]
* [http://bulletin.lmu.edu/content.php?catoid=6&navoid=250#LMU_Honor_Code_and_Process LMU Honor Code and Process].
* [http://www.bio.davidson.edu/dept/plagiarism.html The Davidson College Department of Biology Statement on Plagiarism]
* [[Media:BIOL367_Fall2015_GuidelinesforLiteratureCitations.pdf | Guidelines for Literature Citations in a Scientific Paper]]
** [http://libguides.lmu.edu/content.php?pid=472661&sid=3874648 APA Style for References]
* [[Media:Avoiding_plagiarism.pptx | Academic Resource Center presentation on plagiarism]] (PowerPoint presentation)
* [http://myweb.lmu.edu/dondi/share/misc/plagiarism.pdf Notes on plagiarism from an electrical engineering & computer science perspective]

=== Americans with Disabilities Act—Special Accommodations ===

Students with special needs who require reasonable modifications, special assistance, or accommodations in this course should promptly direct their request to the Disability Support Services (DSS) Office. Any student who currently has a documented disability (ADHD, Autism Spectrum Disorder, Learning, Physical, or Psychiatric) needing academic accommodations should contact the DSS Office (Daum Hall 2nd floor, 310-338-4216) as early in the semester as possible. All discussions will remain confidential. Please visit [http://www.lmu.edu/dss www.lmu.edu/dss] for additional information. In addition, please schedule an appointment with the instructors early in the semester to discuss any accommodations for this course for which you have been approved.

=== Revision Notice ===

If necessary, this syllabus and its contents are subject to revision; students are responsible for any changes or modifications announced in class. The most current version of this information resides on this page, the course web site at http://xmlpipedb.cs.lmu.edu/biodb/fall2015/.

Gene Database Project Deliverables

2015-11-02T07:18:02Z

Dondi: /* Style Sheet */ Add more PeerJ guidelines.

{{Gene Database Project Links}}

== Group Report ==

These guidelines are based on the [https://peerj.com/about/author-instructions/ Instructions for Authors] issued by the [https://peerj.com/computer-science/ PeerJ Computer Science] journal. We have made this choice so that, if a group report is considered to be of sufficient quality, we can pursue publication of this report in ''PeerJ Computer Science'' as smoothly as possible. If there are formatting or detail questions that are not covered here, visit the [https://peerj.com/about/author-instructions/ Instructions for Authors] and follow their guidance.

* The report should be written with contributions from all group members.
* Submit as ''.docx'' or ''.pdf'' file.

=== Style Sheet ===

Use the following guidelines when formatting your report:
* 2.54 cm (1 in) margins on all sides
* Double-spaced
* 12 point Times/Times New Roman font
* Number the pages on the lower-right corner
* Use left justification (“jagged” on the right side)

=== Cover Page ===

Include the following information in a standalone cover page:
* A descriptive title for your project
* The names of the team members
* The course number and title of the class
* The date of submission

=== Abstract ===

Provide an abstract of no more than 500 words.

=== Introduction ===

The introduction gives the background information necessary to understand your report. The introduction should be in the form of a logical argument that “funnels” from broad to narrow:

<gallery mode="nolines" widths=322px heights=256px>
Funnel.png
</gallery>

* States importance of the problem
Why is this species important?
* States what is known about the problem
- Give an overview of what is known about your species' genome from your [[Week 11|journal club outline and presentation]].
- Introduce the DNA microarray experiment that was performed on your species.
* States what is unknown about the problem
You want to analyze the data with GenMAPP/MAPPFinder, but can't because there is no Gene Database for your species.
* States clues that suggest how to approach the unknown
Introduce XMLPipeDB and GenMAPP Builder as the answer to this problem.
* States the question the paper is trying to address
In this case you want to discover new information about the microarray data using GenMAPP.

=== Materials & Methods ===

This section will summarize the entire workflow for the project. This needs to be a '''''narrative description''''' of what your team actually did.
# Download the UniProt XML proteome set and GOA (GO association) files for your species.
#* Note the date of download and the version of the files.
# Download GO terms from in the OBO-XML format.
#* Note the date of download and the version of the files.
# Create the GenMAPP Builder tables in PostgreSQL.
# Load files into PostgreSQL database via GenMAPP Builder.
# Export into a GenMAPP Gene Database.
# Inspect/vet/validate Gene Database.
# Prepare microarray data (organize, normalize, perform statistical analysis)
# Run GenMAPP using the Gene Database.
#* Microarray data (import using Expression Dataset Manager)
#* Run MAPPFinder analysis
#* Place genes on MAPP and draw pathway

=== Results ===

This section will summarize the results of the project. This section will include figures, tables, and '''''narrative description''''' of the results shown in those figures and tables. You should include:
* Gene Database Schema
* Gene Database Testing Report on final version of Gene Database
* Report on quantity and identity of gene IDs that did not make it into the database
*# Gene IDs that were not in the XML source at all
*# Gene IDs that were in the XML source but did not get imported into Postgres
*# Gene IDs that were in Postgres but did not get exported to the GenMAPP Gene Database
* Report on what changes need to be made to the GenMAPP Builder code in order to to accommodate the second and third type of missing gene IDs
** If your team was able to make such changes, report on the results of these changes
* Report results of the DNA microarray analysis
** How many genes were significantly increased and decreased in the dataset? (including your criteria for a significant increase or decrease)
** Table of MAPPFinder results (from .xls)
** GenMAPP MAPP of a pathway relevant to your results

=== Discussion ===

* How well did the GenMAPP Builder process work for your species (just comment on the technical aspects here, you will discuss the teamwork/process aspects in your individual assessment).
* Discuss the statistical analysis and MAPPFinder results for your microarray dataset. Compare it to what was reported in the original paper from which you got the microarray data.

=== Conclusions ===

Summarize the overall project and your findings. How closely do your findings correspond to the original study? Are there significant differences? Did you discover anything new?

=== Acknowledgments ===

''(Optional):'' Write a short paragraph acknowledging the assistance of anyone who is not a member of your team.

=== References ===

Cite and list your references using [https://peerj.com/about/author-instructions/#reference-format PeerJ’s recommended reference format]. This format is very similar to APA style and should feel familiar if you have written research papers before.

To minimize busy work, the PeerJ website includes links to downloadable style files for [https://www.zotero.org/styles/?q=peerj Zotero] and [http://endnote.com/downloads/style/peerj EndNote], if you use either system for managing and rendering references.

== Individual Assessment and Reflection ==

Each person on the team will complete an assessment and reflection ''individually''. If you are comfortable with making this assessment publicly available, you may write it up as a wiki page. If you prefer to communicate your assessment privately, then email this to both Drs. Dahlquist and Dionisio.

=== Statement of Work ===

* Describe exactly what you did on the project.
* Provide references or links to artifacts of your work, such as:
** Wiki pages
** Other files or documents
** Code or scripts

=== Assessment of Project ===

* Give an objective assessment of the success of your project workflow and teamwork.
* What worked and what didn't work?
* What would you do differently if you could do it all over again?
* Evaluate the Gene Database Project and Group Report in the following areas:
*# Content: What is the quality of the work?
*# Organization: Comment on the organization of the project and of your group's wiki pages.
*# Completeness: Did your team achieve all of the project objectives? Why or why not?

=== Reflection on the Process ===

* What did you learn?
** With your head (biological or computer science principles)
** With your heart (personal qualities and teamwork qualities that make things work or not work)?
** With your hands (technical skills)?
* What lesson will you take away from this project that you will still use a year from now?

{{Gene Database Project Links}}

[[Category:Group Projects]]

Gene Database Project Deliverables

2015-11-02T07:16:43Z

Dondi: /* References */ Last sentence was extraneous.

{{Gene Database Project Links}}

== Group Report ==

These guidelines are based on the [https://peerj.com/about/author-instructions/ Instructions for Authors] issued by the [https://peerj.com/computer-science/ PeerJ Computer Science] journal. We have made this choice so that, if a group report is considered to be of sufficient quality, we can pursue publication of this report in ''PeerJ Computer Science'' as smoothly as possible. If there are formatting or detail questions that are not covered here, visit the [https://peerj.com/about/author-instructions/ Instructions for Authors] and follow their guidance.

* The report should be written with contributions from all group members.
* Submit as ''.docx'' or ''.pdf'' file.

=== Style Sheet ===

Use the following guidelines when formatting your report:
* 2.5 cm (approx. 1 in) margins on all sides
* Double-spaced
* 12 point Times/Times New Roman font
* Number the pages

=== Cover Page ===

Include the following information in a standalone cover page:
* A descriptive title for your project
* The names of the team members
* The course number and title of the class
* The date of submission

=== Abstract ===

Provide an abstract of no more than 500 words.

=== Introduction ===

The introduction gives the background information necessary to understand your report. The introduction should be in the form of a logical argument that “funnels” from broad to narrow:

<gallery mode="nolines" widths=322px heights=256px>
Funnel.png
</gallery>

* States importance of the problem
Why is this species important?
* States what is known about the problem
- Give an overview of what is known about your species' genome from your [[Week 11|journal club outline and presentation]].
- Introduce the DNA microarray experiment that was performed on your species.
* States what is unknown about the problem
You want to analyze the data with GenMAPP/MAPPFinder, but can't because there is no Gene Database for your species.
* States clues that suggest how to approach the unknown
Introduce XMLPipeDB and GenMAPP Builder as the answer to this problem.
* States the question the paper is trying to address
In this case you want to discover new information about the microarray data using GenMAPP.

=== Materials & Methods ===

This section will summarize the entire workflow for the project. This needs to be a '''''narrative description''''' of what your team actually did.
# Download the UniProt XML proteome set and GOA (GO association) files for your species.
#* Note the date of download and the version of the files.
# Download GO terms from in the OBO-XML format.
#* Note the date of download and the version of the files.
# Create the GenMAPP Builder tables in PostgreSQL.
# Load files into PostgreSQL database via GenMAPP Builder.
# Export into a GenMAPP Gene Database.
# Inspect/vet/validate Gene Database.
# Prepare microarray data (organize, normalize, perform statistical analysis)
# Run GenMAPP using the Gene Database.
#* Microarray data (import using Expression Dataset Manager)
#* Run MAPPFinder analysis
#* Place genes on MAPP and draw pathway

=== Results ===

This section will summarize the results of the project. This section will include figures, tables, and '''''narrative description''''' of the results shown in those figures and tables. You should include:
* Gene Database Schema
* Gene Database Testing Report on final version of Gene Database
* Report on quantity and identity of gene IDs that did not make it into the database
*# Gene IDs that were not in the XML source at all
*# Gene IDs that were in the XML source but did not get imported into Postgres
*# Gene IDs that were in Postgres but did not get exported to the GenMAPP Gene Database
* Report on what changes need to be made to the GenMAPP Builder code in order to to accommodate the second and third type of missing gene IDs
** If your team was able to make such changes, report on the results of these changes
* Report results of the DNA microarray analysis
** How many genes were significantly increased and decreased in the dataset? (including your criteria for a significant increase or decrease)
** Table of MAPPFinder results (from .xls)
** GenMAPP MAPP of a pathway relevant to your results

=== Discussion ===

* How well did the GenMAPP Builder process work for your species (just comment on the technical aspects here, you will discuss the teamwork/process aspects in your individual assessment).
* Discuss the statistical analysis and MAPPFinder results for your microarray dataset. Compare it to what was reported in the original paper from which you got the microarray data.

=== Conclusions ===

Summarize the overall project and your findings. How closely do your findings correspond to the original study? Are there significant differences? Did you discover anything new?

=== Acknowledgments ===

''(Optional):'' Write a short paragraph acknowledging the assistance of anyone who is not a member of your team.

=== References ===

Cite and list your references using [https://peerj.com/about/author-instructions/#reference-format PeerJ’s recommended reference format]. This format is very similar to APA style and should feel familiar if you have written research papers before.

To minimize busy work, the PeerJ website includes links to downloadable style files for [https://www.zotero.org/styles/?q=peerj Zotero] and [http://endnote.com/downloads/style/peerj EndNote], if you use either system for managing and rendering references.

== Individual Assessment and Reflection ==

Each person on the team will complete an assessment and reflection ''individually''. If you are comfortable with making this assessment publicly available, you may write it up as a wiki page. If you prefer to communicate your assessment privately, then email this to both Drs. Dahlquist and Dionisio.

=== Statement of Work ===

* Describe exactly what you did on the project.
* Provide references or links to artifacts of your work, such as:
** Wiki pages
** Other files or documents
** Code or scripts

=== Assessment of Project ===

* Give an objective assessment of the success of your project workflow and teamwork.
* What worked and what didn't work?
* What would you do differently if you could do it all over again?
* Evaluate the Gene Database Project and Group Report in the following areas:
*# Content: What is the quality of the work?
*# Organization: Comment on the organization of the project and of your group's wiki pages.
*# Completeness: Did your team achieve all of the project objectives? Why or why not?

=== Reflection on the Process ===

* What did you learn?
** With your head (biological or computer science principles)
** With your heart (personal qualities and teamwork qualities that make things work or not work)?
** With your hands (technical skills)?
* What lesson will you take away from this project that you will still use a year from now?

{{Gene Database Project Links}}

[[Category:Group Projects]]

Gene Database Project Deliverables

2015-11-02T07:14:46Z

Dondi: /* References */ Link to Zotero and EndNote PeerJ style files.

{{Gene Database Project Links}}

== Group Report ==

These guidelines are based on the [https://peerj.com/about/author-instructions/ Instructions for Authors] issued by the [https://peerj.com/computer-science/ PeerJ Computer Science] journal. We have made this choice so that, if a group report is considered to be of sufficient quality, we can pursue publication of this report in ''PeerJ Computer Science'' as smoothly as possible. If there are formatting or detail questions that are not covered here, visit the [https://peerj.com/about/author-instructions/ Instructions for Authors] and follow their guidance.

* The report should be written with contributions from all group members.
* Submit as ''.docx'' or ''.pdf'' file.

=== Style Sheet ===

Use the following guidelines when formatting your report:
* 2.5 cm (approx. 1 in) margins on all sides
* Double-spaced
* 12 point Times/Times New Roman font
* Number the pages

=== Cover Page ===

Include the following information in a standalone cover page:
* A descriptive title for your project
* The names of the team members
* The course number and title of the class
* The date of submission

=== Abstract ===

Provide an abstract of no more than 500 words.

=== Introduction ===

The introduction gives the background information necessary to understand your report. The introduction should be in the form of a logical argument that “funnels” from broad to narrow:

<gallery mode="nolines" widths=322px heights=256px>
Funnel.png
</gallery>

* States importance of the problem
Why is this species important?
* States what is known about the problem
- Give an overview of what is known about your species' genome from your [[Week 11|journal club outline and presentation]].
- Introduce the DNA microarray experiment that was performed on your species.
* States what is unknown about the problem
You want to analyze the data with GenMAPP/MAPPFinder, but can't because there is no Gene Database for your species.
* States clues that suggest how to approach the unknown
Introduce XMLPipeDB and GenMAPP Builder as the answer to this problem.
* States the question the paper is trying to address
In this case you want to discover new information about the microarray data using GenMAPP.

=== Materials & Methods ===

This section will summarize the entire workflow for the project. This needs to be a '''''narrative description''''' of what your team actually did.
# Download the UniProt XML proteome set and GOA (GO association) files for your species.
#* Note the date of download and the version of the files.
# Download GO terms from in the OBO-XML format.
#* Note the date of download and the version of the files.
# Create the GenMAPP Builder tables in PostgreSQL.
# Load files into PostgreSQL database via GenMAPP Builder.
# Export into a GenMAPP Gene Database.
# Inspect/vet/validate Gene Database.
# Prepare microarray data (organize, normalize, perform statistical analysis)
# Run GenMAPP using the Gene Database.
#* Microarray data (import using Expression Dataset Manager)
#* Run MAPPFinder analysis
#* Place genes on MAPP and draw pathway

=== Results ===

This section will summarize the results of the project. This section will include figures, tables, and '''''narrative description''''' of the results shown in those figures and tables. You should include:
* Gene Database Schema
* Gene Database Testing Report on final version of Gene Database
* Report on quantity and identity of gene IDs that did not make it into the database
*# Gene IDs that were not in the XML source at all
*# Gene IDs that were in the XML source but did not get imported into Postgres
*# Gene IDs that were in Postgres but did not get exported to the GenMAPP Gene Database
* Report on what changes need to be made to the GenMAPP Builder code in order to to accommodate the second and third type of missing gene IDs
** If your team was able to make such changes, report on the results of these changes
* Report results of the DNA microarray analysis
** How many genes were significantly increased and decreased in the dataset? (including your criteria for a significant increase or decrease)
** Table of MAPPFinder results (from .xls)
** GenMAPP MAPP of a pathway relevant to your results

=== Discussion ===

* How well did the GenMAPP Builder process work for your species (just comment on the technical aspects here, you will discuss the teamwork/process aspects in your individual assessment).
* Discuss the statistical analysis and MAPPFinder results for your microarray dataset. Compare it to what was reported in the original paper from which you got the microarray data.

=== Conclusions ===

Summarize the overall project and your findings. How closely do your findings correspond to the original study? Are there significant differences? Did you discover anything new?

=== Acknowledgments ===

''(Optional):'' Write a short paragraph acknowledging the assistance of anyone who is not a member of your team.

=== References ===

Cite and list your references using [https://peerj.com/about/author-instructions/#reference-format PeerJ’s recommended reference format]. That website includes links to downloadable style files for [https://www.zotero.org/styles/?q=peerj Zotero] and [http://endnote.com/downloads/style/peerj EndNote], if you use either system for managing references.

Be consistent with your format for the in text citations and for your references list at the end.

== Individual Assessment and Reflection ==

Each person on the team will complete an assessment and reflection ''individually''. If you are comfortable with making this assessment publicly available, you may write it up as a wiki page. If you prefer to communicate your assessment privately, then email this to both Drs. Dahlquist and Dionisio.

=== Statement of Work ===

* Describe exactly what you did on the project.
* Provide references or links to artifacts of your work, such as:
** Wiki pages
** Other files or documents
** Code or scripts

=== Assessment of Project ===

* Give an objective assessment of the success of your project workflow and teamwork.
* What worked and what didn't work?
* What would you do differently if you could do it all over again?
* Evaluate the Gene Database Project and Group Report in the following areas:
*# Content: What is the quality of the work?
*# Organization: Comment on the organization of the project and of your group's wiki pages.
*# Completeness: Did your team achieve all of the project objectives? Why or why not?

=== Reflection on the Process ===

* What did you learn?
** With your head (biological or computer science principles)
** With your heart (personal qualities and teamwork qualities that make things work or not work)?
** With your hands (technical skills)?
* What lesson will you take away from this project that you will still use a year from now?

{{Gene Database Project Links}}

[[Category:Group Projects]]

Template:Gene Database Project Links

2015-11-02T07:12:33Z

Dondi: Switch to PeerJ format.

{| class="wikitable" style="width: 100%; text-align: center"
! colspan="8"|Gene Database Project Links
|-
! [[Gene Database Project|Overview]]
! [[Gene Database Project Deliverables|Deliverables]]
| [https://peerj.com/about/author-instructions/#reference-format Reference Format]
! Guilds
| [[Project Manager]]
| [[GenMAPP User]]
| [[Quality Assurance]]
| [[Coder]]
|}

Gene Database Project Deliverables

2015-11-02T07:10:40Z

Dondi: /* References */ Adopt PeerJ’s reference format.

{{Gene Database Project Links}}

== Group Report ==

These guidelines are based on the [https://peerj.com/about/author-instructions/ Instructions for Authors] issued by the [https://peerj.com/computer-science/ PeerJ Computer Science] journal. We have made this choice so that, if a group report is considered to be of sufficient quality, we can pursue publication of this report in ''PeerJ Computer Science'' as smoothly as possible. If there are formatting or detail questions that are not covered here, visit the [https://peerj.com/about/author-instructions/ Instructions for Authors] and follow their guidance.

* The report should be written with contributions from all group members.
* Submit as ''.docx'' or ''.pdf'' file.

=== Style Sheet ===

Use the following guidelines when formatting your report:
* 2.5 cm (approx. 1 in) margins on all sides
* Double-spaced
* 12 point Times/Times New Roman font
* Number the pages

=== Cover Page ===

Include the following information in a standalone cover page:
* A descriptive title for your project
* The names of the team members
* The course number and title of the class
* The date of submission

=== Abstract ===

Provide an abstract of no more than 500 words.

=== Introduction ===

The introduction gives the background information necessary to understand your report. The introduction should be in the form of a logical argument that “funnels” from broad to narrow:

<gallery mode="nolines" widths=322px heights=256px>
Funnel.png
</gallery>

* States importance of the problem
Why is this species important?
* States what is known about the problem
- Give an overview of what is known about your species' genome from your [[Week 11|journal club outline and presentation]].
- Introduce the DNA microarray experiment that was performed on your species.
* States what is unknown about the problem
You want to analyze the data with GenMAPP/MAPPFinder, but can't because there is no Gene Database for your species.
* States clues that suggest how to approach the unknown
Introduce XMLPipeDB and GenMAPP Builder as the answer to this problem.
* States the question the paper is trying to address
In this case you want to discover new information about the microarray data using GenMAPP.

=== Materials & Methods ===

This section will summarize the entire workflow for the project. This needs to be a '''''narrative description''''' of what your team actually did.
# Download the UniProt XML proteome set and GOA (GO association) files for your species.
#* Note the date of download and the version of the files.
# Download GO terms from in the OBO-XML format.
#* Note the date of download and the version of the files.
# Create the GenMAPP Builder tables in PostgreSQL.
# Load files into PostgreSQL database via GenMAPP Builder.
# Export into a GenMAPP Gene Database.
# Inspect/vet/validate Gene Database.
# Prepare microarray data (organize, normalize, perform statistical analysis)
# Run GenMAPP using the Gene Database.
#* Microarray data (import using Expression Dataset Manager)
#* Run MAPPFinder analysis
#* Place genes on MAPP and draw pathway

=== Results ===

This section will summarize the results of the project. This section will include figures, tables, and '''''narrative description''''' of the results shown in those figures and tables. You should include:
* Gene Database Schema
* Gene Database Testing Report on final version of Gene Database
* Report on quantity and identity of gene IDs that did not make it into the database
*# Gene IDs that were not in the XML source at all
*# Gene IDs that were in the XML source but did not get imported into Postgres
*# Gene IDs that were in Postgres but did not get exported to the GenMAPP Gene Database
* Report on what changes need to be made to the GenMAPP Builder code in order to to accommodate the second and third type of missing gene IDs
** If your team was able to make such changes, report on the results of these changes
* Report results of the DNA microarray analysis
** How many genes were significantly increased and decreased in the dataset? (including your criteria for a significant increase or decrease)
** Table of MAPPFinder results (from .xls)
** GenMAPP MAPP of a pathway relevant to your results

=== Discussion ===

* How well did the GenMAPP Builder process work for your species (just comment on the technical aspects here, you will discuss the teamwork/process aspects in your individual assessment).
* Discuss the statistical analysis and MAPPFinder results for your microarray dataset. Compare it to what was reported in the original paper from which you got the microarray data.

=== Conclusions ===

Summarize the overall project and your findings. How closely do your findings correspond to the original study? Are there significant differences? Did you discover anything new?

=== Acknowledgments ===

''(Optional):'' Write a short paragraph acknowledging the assistance of anyone who is not a member of your team.

=== References ===

Cite and list your references using [https://peerj.com/about/author-instructions/#reference-format PeerJ’s recommended reference format]. That website includes links to downloadable style files for Zotero and EndNote, if you use either system for managing references.

Be consistent with your format for the in text citations and for your references list at the end.

== Individual Assessment and Reflection ==

Each person on the team will complete an assessment and reflection ''individually''. If you are comfortable with making this assessment publicly available, you may write it up as a wiki page. If you prefer to communicate your assessment privately, then email this to both Drs. Dahlquist and Dionisio.

=== Statement of Work ===

* Describe exactly what you did on the project.
* Provide references or links to artifacts of your work, such as:
** Wiki pages
** Other files or documents
** Code or scripts

=== Assessment of Project ===

* Give an objective assessment of the success of your project workflow and teamwork.
* What worked and what didn't work?
* What would you do differently if you could do it all over again?
* Evaluate the Gene Database Project and Group Report in the following areas:
*# Content: What is the quality of the work?
*# Organization: Comment on the organization of the project and of your group's wiki pages.
*# Completeness: Did your team achieve all of the project objectives? Why or why not?

=== Reflection on the Process ===

* What did you learn?
** With your head (biological or computer science principles)
** With your heart (personal qualities and teamwork qualities that make things work or not work)?
** With your hands (technical skills)?
* What lesson will you take away from this project that you will still use a year from now?

{{Gene Database Project Links}}

[[Category:Group Projects]]

Gene Database Project Deliverables

2015-11-02T07:02:35Z

Dondi: /* Group Report */ Make the format closer to PeerJ instructions.

{{Gene Database Project Links}}

== Group Report ==

These guidelines are based on the [https://peerj.com/about/author-instructions/ Instructions for Authors] issued by the [https://peerj.com/computer-science/ PeerJ Computer Science] journal. We have made this choice so that, if a group report is considered to be of sufficient quality, we can pursue publication of this report in ''PeerJ Computer Science'' as smoothly as possible. If there are formatting or detail questions that are not covered here, visit the [https://peerj.com/about/author-instructions/ Instructions for Authors] and follow their guidance.

* The report should be written with contributions from all group members.
* Submit as ''.docx'' or ''.pdf'' file.

=== Style Sheet ===

Use the following guidelines when formatting your report:
* 2.5 cm (approx. 1 in) margins on all sides
* Double-spaced
* 12 point Times/Times New Roman font
* Number the pages

=== Cover Page ===

Include the following information in a standalone cover page:
* A descriptive title for your project
* The names of the team members
* The course number and title of the class
* The date of submission

=== Abstract ===

Provide an abstract of no more than 500 words.

=== Introduction ===

The introduction gives the background information necessary to understand your report. The introduction should be in the form of a logical argument that “funnels” from broad to narrow:

<gallery mode="nolines" widths=322px heights=256px>
Funnel.png
</gallery>

* States importance of the problem
Why is this species important?
* States what is known about the problem
- Give an overview of what is known about your species' genome from your [[Week 11|journal club outline and presentation]].
- Introduce the DNA microarray experiment that was performed on your species.
* States what is unknown about the problem
You want to analyze the data with GenMAPP/MAPPFinder, but can't because there is no Gene Database for your species.
* States clues that suggest how to approach the unknown
Introduce XMLPipeDB and GenMAPP Builder as the answer to this problem.
* States the question the paper is trying to address
In this case you want to discover new information about the microarray data using GenMAPP.

=== Materials & Methods ===

This section will summarize the entire workflow for the project. This needs to be a '''''narrative description''''' of what your team actually did.
# Download the UniProt XML proteome set and GOA (GO association) files for your species.
#* Note the date of download and the version of the files.
# Download GO terms from in the OBO-XML format.
#* Note the date of download and the version of the files.
# Create the GenMAPP Builder tables in PostgreSQL.
# Load files into PostgreSQL database via GenMAPP Builder.
# Export into a GenMAPP Gene Database.
# Inspect/vet/validate Gene Database.
# Prepare microarray data (organize, normalize, perform statistical analysis)
# Run GenMAPP using the Gene Database.
#* Microarray data (import using Expression Dataset Manager)
#* Run MAPPFinder analysis
#* Place genes on MAPP and draw pathway

=== Results ===

This section will summarize the results of the project. This section will include figures, tables, and '''''narrative description''''' of the results shown in those figures and tables. You should include:
* Gene Database Schema
* Gene Database Testing Report on final version of Gene Database
* Report on quantity and identity of gene IDs that did not make it into the database
*# Gene IDs that were not in the XML source at all
*# Gene IDs that were in the XML source but did not get imported into Postgres
*# Gene IDs that were in Postgres but did not get exported to the GenMAPP Gene Database
* Report on what changes need to be made to the GenMAPP Builder code in order to to accommodate the second and third type of missing gene IDs
** If your team was able to make such changes, report on the results of these changes
* Report results of the DNA microarray analysis
** How many genes were significantly increased and decreased in the dataset? (including your criteria for a significant increase or decrease)
** Table of MAPPFinder results (from .xls)
** GenMAPP MAPP of a pathway relevant to your results

=== Discussion ===

* How well did the GenMAPP Builder process work for your species (just comment on the technical aspects here, you will discuss the teamwork/process aspects in your individual assessment).
* Discuss the statistical analysis and MAPPFinder results for your microarray dataset. Compare it to what was reported in the original paper from which you got the microarray data.

=== Conclusions ===

Summarize the overall project and your findings. How closely do your findings correspond to the original study? Are there significant differences? Did you discover anything new?

=== Acknowledgments ===

''(Optional):'' Write a short paragraph acknowledging the assistance of anyone who is not a member of your team.

=== References ===

Please use the [[Guidelines for Literature Citations in a Scientific Paper]]. The [http://en.wikipedia.org/wiki/APA_style#Citation APA format] is also very similar to these guidelines and will be acceptable. Be consistent with your format for the in text citations and for your references list at the end.

== Individual Assessment and Reflection ==

Each person on the team will complete an assessment and reflection ''individually''. If you are comfortable with making this assessment publicly available, you may write it up as a wiki page. If you prefer to communicate your assessment privately, then email this to both Drs. Dahlquist and Dionisio.

=== Statement of Work ===

* Describe exactly what you did on the project.
* Provide references or links to artifacts of your work, such as:
** Wiki pages
** Other files or documents
** Code or scripts

=== Assessment of Project ===

* Give an objective assessment of the success of your project workflow and teamwork.
* What worked and what didn't work?
* What would you do differently if you could do it all over again?
* Evaluate the Gene Database Project and Group Report in the following areas:
*# Content: What is the quality of the work?
*# Organization: Comment on the organization of the project and of your group's wiki pages.
*# Completeness: Did your team achieve all of the project objectives? Why or why not?

=== Reflection on the Process ===

* What did you learn?
** With your head (biological or computer science principles)
** With your heart (personal qualities and teamwork qualities that make things work or not work)?
** With your hands (technical skills)?
* What lesson will you take away from this project that you will still use a year from now?

{{Gene Database Project Links}}

[[Category:Group Projects]]

Main Page

2015-11-02T06:33:09Z

Dondi: /* Final Group Project */ Fix dangling links, mention PeerJ Computer Science.

__NOTOC__
<div style="text-align: center; font-size: 175%">Loyola Marymount University</div>
<div style="text-align: center; font-size: 250%; line-height: 1.25em">'''BIOL/CMSI 367-01: Biological Databases'''</div>
<div style="text-align: center; font-size: 175%; margin-bottom: 0.85em">Fall 2015</div>

<div class="toc-container">__TOC__</div>

== Announcements ==

* The Microsoft Excel portion of the ID investigation protocol has been added to the wiki: [[Using Microsoft Excel to Compare ID Lists]]. The link has also been added to the course schedule below for Week 9.
:Please note that this version is similar, but not identical, to the process shown in class ([[User:Dondi|Dondi]] misremembered the function to use; it was supposed to be <code>match</code>, as shown in the wiki page, and not <code>lookup</code>). Furthermore, instructions for exporting from PostgreSQL (you might recall that the numbers did not match there either) are included. —''[[User:Dondi|Dondi]] ([[User talk:Dondi|talk]]) 18:49, 29 October 2015 (PDT)''
*'''''TriBeta Halloween Party and Costume Contest:''''' Friday, October 30, 6:00 PM, LSB Terrace, $3 (to cover food). RSVP to ralkhaya at lion dot lmu dot edu. Come see if you can beat me in the costume contest! ''— [[User:Kdahlquist|Kdahlquist]] ([[User talk:Kdahlquist|talk]]) 09:46, 28 October 2015 (PDT)''
* An initial round of feedback to the [[Week 6]] assignment has been posted to your respective user talk pages. The database part is not finished, but I figured some feedback and basics on the best practice stuff will be useful as you prepare your [[Week 8]] work (as in, get used to making the best practices routine so you can avoid nagging deductions week after week). —''[[User:Dondi|Dondi]] ([[User talk:Dondi|talk]]) 21:30, 25 October 2015 (PDT)''
* To report issues with the configuration of the computers in Seaver 120, please record it on [[Talk:Software_Configuration | this wiki page.]] The complete list of software that should be installed for this course is found [[Software Configuration | here]]. Note that this configuration is Windows-specific. Some software may not be available for a Mac or may look differently.

=== Upcoming Seminars ===

* '''Chemistry & Biochemistry Department Seminar:''' Thursday, October 29, 11:50 AM, Seaver 200, Presentation by first candidate for the Biochemistry Faculty Search.
* '''Biology Department Seminar:''' Friday, October 30, 2:00 PM, LSB Auditorium, Dr. Ashley Robart, LMU Biology, ''MALE QUALITY INFLUENCES PARENTAL CARE, BUT NOT FEMALE COMPETITION, IN A MONOGAMOUS, BIPARENTAL FISH.''
* '''Noyce Poster Session:''' Friday, October 30, 4:00 – 6:00 PM, Hannon Courtyard of the LSB (area just outside the west entrance). The NOYCE Summer Interns will be presenting the posters they made highlighting the research they have done. The Poster Session will be a fantastic opportunity to talk to the presenters and ask them all about their research! To receive the extra credit, fill out one of the "seminar" sheets available according to the instructions and turn it in to a professor before you leave the poster session.
* '''Chemistry & Biochemistry Department Seminar:''' Thursday, November 5, 11:50 AM, Seaver 200, Presentation by second candidate for the Biochemistry Faculty Search.
* '''Biology Department Seminar:''' Friday, November 6, 2:00 PM, LSB Auditorium, Dr. Wes Dowd, LMU Biology.
* '''Chemistry & Biochemistry Department Seminar:''' Tuesday, November 10, 11:50 AM, Seaver 200, Presentation by third candidate for the Biochemistry Faculty Search.
* '''Chemistry & Biochemistry Department Seminar:''' Thursday, November 12, 11:50 AM, Seaver 200, Presentation by fourth candidate for the Biochemistry Faculty Search.
* '''Biology Department Seminar:''' Friday, December 4, 2:00 PM, LSB Auditorium, Dr. Deepa Dabir, LMU Biology.



== Instructors ==
* [[User:Kdahlquist|Kam D. Dahlquist]]
* [[User:Dondi|John David N. Dionisio]]

== Students ==

{| style="margin-left: auto; margin-right: auto"
| style="padding-right: 1em" | [[User: Malverso | Mary Alverson]]
| style="padding-right: 1em" | [[User:Rlegaspi | Ron Legaspi]]
| style="padding-right: 1em" | [[User:Troque | Trixie Roque]]
| style="padding-right: 1em" | [[User:Jwoodlee | Jake Woodlee]]
|-
| style="padding-right: 1em" | [[User:Nanguiano | Nicole Anguiano]]
| style="padding-right: 1em" | [[User:Blitvak | Brandon Litvak]]
| style="padding-right: 1em" | [[user:Msaeedi23 | Mahrad Saeedi]]
| style="padding-right: 1em" | [[User:Kwyllie | Kevin Wyllie]]
|-
| style="padding-right: 1em" | [[User:Bklein7 | Brandon Klein]]
| style="padding-right: 1em" | [[User:Lenaolufson | Elena Olufson]]
| style="padding-right: 1em" | [[User:Emilysimso | Emily Simso]]
| style="padding-right: 1em" | [[User:Eyanosch | Erich Yanoschik]]
|-
| style="padding-right: 1em" | [[User:Jkuroda | Josh Kuroda]]
| style="padding-right: 1em" | [[User:Vpachec3 | Veronica Pacheco]]
| style="padding-right: 1em" | [[User:Anuvarsh | Anindita Varshneya]]
| style="padding-right: 1em" | [[User:Kzebrows | Kristin Zebrowski]]
|}

{{ Gene Database Project Links }}

== Course Schedule ==

The current course schedule is shown below. In addition to the listed in-class agendas, the following guidelines are also notable:

* Unless otherwise stated on the schedule, your weekly journals/assignments are due on Tuesdays at midnight (Monday night/Tuesday morning). In cases where subsequent revisions or corrections will be accepted, the instructors will provide feedback and submission deadlines on a per-assignment basis.
* Reading assignments should be completed ''before'' coming to class.
* Dr. Dionisio’s office hours ({{ Dondi Office Hours }}) can be viewed as unofficial lab sessions: use them for IT help or desktop support

=== Part 1: Building Blocks (Genetic Code and Manipulating Text) ===

{| class="schedule-table"
! #
! Date
! Reading
! Agenda
! Journal
|-
! 1
| Tuesday<br/>9/1/2015
|
| Syllabus walkthrough
* ''my.cs.lmu.edu'' account: Visit https://secure.cs.lmu.edu to get a computer science account if you don’t already have one
* Class wiki account: Set up a username and password for this wiki site

Pairwise interviews (report back after each bullet)
* Basic acquaintance information
* Like a cell/not like a cell; like a computer/not like a computer

| rowspan="2" | [[Week 1]]

(due at midnight 9/8)

[[Class_Journal_Week_1 | Class Journal Week 1]]
|-
!
| Thursday<br/>9/3/2015
|
[http://dl.acm.org/ft_gateway.cfm?id=1378711 ''Voices of Computing''] and [http://dl.acm.org/ft_gateway.cfm?id=1272529 ''Computing is a Natural Science''] by Peter Denning (these links should be accessed from within LMU to get the full article)

[https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&mode=reset Chapter 1 of ''On Becoming a Biologist''] by John Janovy, Jr. (on MyLMUConnect)
| Discuss the [[Week 1]] assignment (accounts should be ready to wiki at this point)
* Quick wiki overview
* User wiki page setup

{{Course Schedule Divider}}
|-
! 2
| Tuesday<br/>9/8/2015
| [http://www.ncbi.nlm.nih.gov/books/NBK21134/#A5234 Ch. 1.1 ''Genomes 2'']

[http://www.nature.com/nature/dna50/watsoncrick.pdf Watson & Crick (1953)]

[http://www.nature.com/nature/dna50/franklingosling.pdf Franklin & Gosling (1953)]

| DNA
* DNA structure and Genetic Code
* [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&content_id=_1103141_1 Slides shown in class on MyLMU Connect]
| rowspan="2" | [[Week 2]]

(due at midnight 9/15)

[[Class Journal Week 2]]
|-
!
| Thursday<br/>9/10/2015
| On [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&mode=reset MyLMUConnect]:

Nirenberg (2004) ''Deciphering the Genetic Code''

Kaji & Kaji (2004) ''Setting the Record Straight''

Moody (2004) ''Digital Code of Life'', Ch. 1

Hayes (2004) ''Ode to the Code''

[http://www.ncbi.nlm.nih.gov/books/NBK21121/ Brown, T.A. (2002) ''Genomes 2'', Ch. 3, especially section 3.3.2]
| Central dogma
* [http://www.youtube.com/watch?v=u9dhO0iCLww Berkeley translation video]
* This [http://www.dnai.org/a/index.html web site] has links to animations of the replication, transcription, and translation processes.
* Slides shown in class
* [[Media:BIOL367_Fall2015_GeneticCode-GeneExpression.pdf | Genetic Code-Gene Expression Pencil Exercise]]

{{Course Schedule Divider}}
|-
! 3
| Tuesday<br/>9/15/2015
| [http://www.bloomberg.com/graphics/2015-paul-ford-what-is-code/ Ford (2015) ''What is Code?'']
* Emphasis on first two sections (“The Man in the Taupe Blazer” and “Let’s Begin”), but feel free to read further

[http://myweb.lmu.edu/dondi/share/intro/wheres-my-stuff.pdf Where's my Stuff?]

[[Introduction to the Command Line]]
| An overview of computers, networks, files, and databases
* Useful help resource: http://explainshell.com
* Manipulating DNA sequences on the command line
* [[Using the XMLPipeDB Match Utility]] 
| rowspan="2" | [[Week 3]]

(due at midnight 9/22)

[[Class Journal Week 3]]
|-
!
| Thursday<br/>9/17/2015
|
| Command line lab session
* Hands-on practice
* Discuss [[Week 3]] assignment tasks

{{Course Schedule Divider}}
|-
! 4
| Tuesday<br/>9/22/2015
| [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&mode=reset Moody (2004) Chapter 6 (on MyLMU Connect)]

[http://www.nature.com/news/2010/100623/pdf/4651000a.pdf Science After the Sequence]

[http://www.nature.com/news/2010/100623/full/4651000a.html Nature Special: Human Genome at 10]

[http://nar.oxfordjournals.org/content/43/D1/D1.full Introduction to ''NAR'' Database Issue]

[http://scienceblogs.com/digitalbio/2015/01/30/bio-databases-2015/ Commentary on ''NAR'' Database Issue]

| Genome sequencing and annotation

Introduction to biological databases
* [https://mylmuconnect.blackboard.com/bbcswebdav/pid-1146580-dt-content-rid-1944592_1/courses/M_46692.201530/20150922_HGP_IntroBiolDB.pdf Slides shown in class (MyLMU Connect)]
* [http://www.dataone.org/sites/all/documents/L01_DataManagement.pptx DataONE: Why Data Management]
* [http://www.dataone.org/sites/all/documents/L02_DataSharing.pptx DataONE: Data Sharing]
| rowspan="2" | [[Week 4]]

(due at midnight 9/29)

[[Class Journal Week 4]]
|-
!
| Thursday<br/>9/24/2015
| [[More Text Processing Features]]

[[How to Read XML Files]]
| Command line lab session
* Review assignment issues
* Discuss [[Week 4]] computer portion


{{Course Schedule Divider}}
|-
! 5
| Tuesday<br/>9/29/2015
| [http://myweb.lmu.edu/dondi/share/db/relational1.pdf The Relational Data Model: Structure]

[http://myweb.lmu.edu/dondi/share/db/sql.pdf An Overview of SQL]

| Introduction to relational databases
* Case study from one of the NAR databases: http://www.rhesusbase.org 
* Relating XML structure to relational structure

Test your understanding: http://xkcd.com/327/
| rowspan="2" | [[Week 5]]

(due at midnight 10/6)

[[Class Journal Week 5]]
|-
!
| Thursday<br/>10/1/2015
| Additional background and details can be found in Chapters 1, 2, and 6 of ''A First Course in Database Systems'' (on reserve at the library).
| SQL and working with relational databases continued
* [[PostgreSQL Tutorial]]

{{Course Schedule Divider}}
|-
! style="background-color: #ddf" | 6
| style="background-color: #eef; border: none" | Tuesday<br/>10/6/2015
| style="background-color: #eef; border: none" |
| style="background-color: #eef; border: none" | President Synder’s Inauguration Day
| rowspan="2" | [[Week 6]]

(due at midnight 10/13)

[[Class Journal Week 6]]
|-
!
| Thursday<br/>10/8/2015
|
| Database presentations
# Nicole, Erich, Kristen, Josh
# Mary, Emily, Brandon L., Kevin
# Lena, Jake, Anu, Veronica
# Ron, Mahrad, Brandon K., Trixie

[[Presentation Rubric]]

[[Media:PresentationGuidelines.ppt | PowerPoint Guidelines]]

|}

=== Part 2: Going Deeper (Gene Expression Data and Relational Databases) ===

{| class="schedule-table"
! #
! Date
! Reading
! Agenda
! Journal
|-
! 7
| Tuesday<br/>10/13/2015
| [http://www.ncbi.nlm.nih.gov/books/NBK26818/#A1633 Alberts ''et al''. (2002) ''Molecular Biology of the Cell'', Ch. 8: Microarrays]

[http://www.nature.com/ng/journal/v21/n1s/full/ng0199supp_33.html Brown & Botstein (1999) "Microarrays" ''Nature Genetics'']

Campbell & Heyer Chapter 4 (on [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1 MyLMU Connect])

DeRisi et al. (1997) [http://www.sciencemag.org/content/278/5338/680.full ''Science'' 278: 680-686.]

| Introduction to DNA microarray data, Gene Ontology, and GenMAPP/MAPPFinder
* [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&content_id=_1103141_1 Slides on MyLMU Connect]
* [http://www.dataone.org/sites/all/documents/L04_DataEntryManipulation.pptx DataONE: Data Entry and Manipulation]
* [http://www.bio.davidson.edu/courses/genomics/chip/chip.html Microarray animation]

| rowspan="2" | [[Week 7]]

(due at midnight 10/20)

[[Class Journal Week 7]]
|-
!
| Thursday<br/>10/15/2015
| [http://www.nature.com/nature/journal/v417/n6889/full/nature00778.html Merrell et al. (2002) "''Vibrio cholerae'' microarray data" ''Nature'']
| [http://www.openwetware.org/wiki/BIOL398-01/S10:Sample_Microarray_Analysis_Vibrio_cholerae DNA Microarray Analysis activity part 1]

Test your understanding: [http://xkcd.com/882/ http://xkcd.com/882/]

{{Course Schedule Divider}}
|-
! 8
| Tuesday<br/>10/20/2015
| [http://www.nature.com/ng/journal/v25/n1/full/ng0500_25.html Ashburner et al. (2000) "Gene Ontology" ''Nature Genetics'']

[http://www.genomebiology.com/content/4/1/R7 Doniger et al. (2003) "MAPPFinder" ''Genome Biology'']
| [http://www.openwetware.org/wiki/BIOL367/F10:GenMAPP_and_MAPPFinder_Protocols DNA Microarray Analysis activity part 2]
| rowspan="2" | [[Week 8]]

(due at midnight 10/27)

[[Class Journal Week 8]]
|-
!
| Thursday<br/>10/22/2015
|
| Work session to complete DNA Microarray Analysis [http://www.openwetware.org/wiki/BIOL398-01/S10:Sample_Microarray_Analysis_Vibrio_cholerae part 1] and [http://www.openwetware.org/wiki/BIOL367/F10:GenMAPP_and_MAPPFinder_Protocols part 2]

{{Course Schedule Divider}}
|-
! 9
| Tuesday<br/>10/27/2015
|
|
* Introduction to XMLPipeDB: export a Gene Database for ''Vibrio cholerae''
** [[Running GenMAPP Builder]]
* [http://www.opensource.org/ Open Source] review
* Group requests due midnight Saturday, October 31
| rowspan="2" | [[Week 9]]

(due at midnight 11/3)

[[Class Journal Week 9]]
|-
!
| Thursday<br/>10/29/2015
| [http://nar.oxfordjournals.org/content/40/D1/D565.full Dimmer et al. (2012) "UniProt-GOA Database" ''Nucleic Acids Research'']

[http://www.nature.com/nature/journal/v406/n6795/full/406477A0.html Heidelberg et al. (2000) "''Vibrio cholerae'' Genome Sequence" ''Nature'']
|
Integrity analysis of Vibrio Gene Database
* [[How Do I Count Thee? Let Me Count The Ways]]
* [[Using Microsoft Excel to Compare ID Lists]]
* [[Gene Database Testing Report Sample | Complete your Gene Database Testing Report for ''Vibrio cholerae'']]
|}

=== Part 3: Integrating for Research (Gene Database Project) ===

{| class="schedule-table"
! #
! Date
! Reading
! Agenda
! Journal
|-
! 10
| Tuesday<br/>11/3/2015
|
|
* Annotated bibliography of genomics papers for your species
* ''Course LibGuide:'' http://libguides.lmu.edu/BIOL367
** Select genome paper for journal club
** Select microarray paper
| rowspan="2" | [[Week 10]]

(due at midnight 11/10)


|-
!
| Thursday<br/>11/5/2015
|
| Begin [[Gene_Database_Project | group projects]]

{{Course Schedule Divider}}
|-
! 11
| Tuesday<br/>11/10/2015
|
| Guild meetings and work session
* Export of Gene Database for your group's species
* Document ID types for your species
* Begin DNA microarray analysis
| rowspan="2" | [[Week 11]]

(due at midnight 11/17)
|-
!
| Thursday<br/>11/12/2015
|
|

* Status report
* Work session

{{Course Schedule Divider}}
|-
! 12
| Tuesday<br/>11/17/2015
|
| First Set of Journal Club Presentations
# TBD
# TBD
# TBD
# TBD

[[Presentation Rubric]]

[[Media:PresentationGuidelines.ppt | PowerPoint Guidelines]]
| rowspan="2" | [[Week 12]]

(due at midnight 11/24)

|-
!
| Thursday<br/>11/19/2015
|
|
* Status report
* Work session

{{Course Schedule Divider}}
|-
! 13
| Tuesday<br/>11/24/2015
|
| Second Set of Journal Club Presentations
# TBD
# TBD
# TBD
# TBD

[[Presentation Rubric]]

[[Media:PresentationGuidelines.ppt | PowerPoint Guidelines]]
| '''''No Week 13 assignment'''''

|-
! style="background: #ded" |
| style="background: #efe; border: none" | Thursday<br/>11/26/2015
| style="background: #efe; border: none" |
| style="background: #efe; border: none" | Thanksgiving—no class
| style="background: #efe; border: none" |
{{Course Schedule Divider}}
|-
! 14
| Tuesday<br/>12/1/2015
|
|
* Status report
* Work session
| rowspan="2" | [[Week 14]]

(due at midnight 12/8)
|-
!
| Thursday<br/>12/3/2015
|
|
* Status report
* Work session
{{Course Schedule Divider}}
|-
! 15
| Tuesday<br/>12/8/2015
|
|
* Status report
* Work session
| rowspan="2" | [[Week 15]]

(due at midnight 12/15)
|-
!
| Thursday<br/>12/10/2015
|
|
* Status report
* Work session
{{Course Schedule Divider}}
|-
! style="background: #edd" | F
| style="background: #fee; border: none" | Tuesday<br/>12/15/2015
| style="background: #fee; border: none" |
| style="background: #fee; border: none" | '''Final project presentations 2:00-4:00 PM'''
| style="background: #fee; border: none" |
|-
!
| Friday<br/>12/18/2015 4:30 PM
|
| '''Project deliverables due 4:30 PM'''
|
|}

== Course Information ==

=== Instructors ===

==== [[User:Kdahlquist | Kam D. Dahlquist, Ph.D.]] ====
: http://myweb.lmu.edu/kdahlqui
: '''Phone:''' (310) 338-7697
: '''E-mail:''' kdahlquist at lmu dot edu
: '''Office:''' Life Sciences Building 289
: '''Office Hours:''' Tuesdays 9:00-11:00 AM, Wednesdays 1:30-3:30 PM and by appointment. I keep a sign-up sheet on my office door to facilitate appointment-making.

==== [[User:Dondi|John David N. Dionisio, Ph.D.]] ====
: http://myweb.lmu.edu/dondi
: '''Phone:''' (310) 338-5782
: '''E-mail:''' dondi at lmu dot edu
: '''Office:''' Doolan 106
: '''Office Hours:''' {{Dondi Office Hours}}

=== Prerequisites/Recommended Background ===

To take this course, you must have upper division standing in the Seaver College of Science and Engineering. Otherwise, there are no strict prerequisites. Backgrounds in biology and computer science, as well as prior experience with database or information management applications, may be helpful but not necessary.

=== Class Meetings and Attendance ===
TR 2:40–3:55pm, Seaver 120 

This is a hands-on, participatory course, thus attendance at all class meetings is required. Each student is allowed two “sick” days (automatically excused absences) during the semester. Further unexcused absences from class will result in a 5% deduction from the overall course grade for each absence. Every effort should be made to attend class on oral presentation days as the content of that day's class is dependent on student participation. Unexcused absences from an oral presentation will result in a grade of zero for the presentation. The instructors should be notified as soon as possible, electronically or by phone, of the reasons for all absences.

Note that the last day to add or drop a class without a grade of W is September 4. The withdrawal or credit/no-credit status deadline is November 6.

=== Mutual Responsibilities ===

This course is designed to foster your development as a biologist and computer scientist and to give you an authentic research experience. We will be engaged together in discovering, examining, and practicing the personal qualities, technical skills, and community standards of the scientific community. While you are ultimately responsible for your own learning, you are not alone. Our class constitutes a team where we will be learning from each other. The role of the instructors is to provide the expert coaching to support and assist you on your journey. All of the exercises, readings, assignments, and policies detailed below have been designed with this purpose in mind.

=== Classroom Conduct ===

We are all responsible for maintaining a classroom and laboratory environment that is safe and conducive to learning. As such, we will observe the following:

# As an LMU Lion, by the [http://studentaffairs.lmu.edu/media/lmustudentaffairs/administration/judicialaffairs/Lion%27s%20Code.pdf Lion’s code], you are pledged to join the discourse of the academy with honesty of voice and integrity of scholarship and to show respect for staff, professors, and other students.
# You are responsible for your own learning and for being a good class citizen.
# Class will start promptly on time.
# You are expected to come to class having done the assigned reading and preparatory work so that you are ready to participate in discussions and to perform the laboratory exercises.
# You are expected to bring the required materials to each class session.
# Cell phones, pagers, and other communication or music devices must be turned off and put away out of sight. Your own laptop and/or tablet may be used to conduct the class exercises; however, if they are being used for other purposes and become distracting to you or others, you will be asked to put them away.
# All students are governed by [http://studentaffairs.lmu.edu/media/lmustudentaffairs/administration/judicialaffairs/Loyola-Marymount-University-Community-Standards-2015-2016.pdf LMU Community Standards Publication]. Disruptive behavior which is persistent or significantly interferes with classroom activities may be subject to disciplinary action. A student may be referred to the Office of Student Judicial Affairs if his or her behavior constitutes a violation of the conduct code.

=== Course Website ===

This is the course web site and wiki, hosted by http://xmlpipedb.cs.lmu.edu/biodb/fall2015/. You will need to [https://xmlpipedb.cs.lmu.edu/biodb/fall2015/index.php?title=Special:UserLogin&returnto=Main+Page&type=signup register] to be able to edit the wiki and complete coursework. Updates to the course schedule and electronic copies of all handouts, assignments, and readings will be posted to this site. You will also use the site to keep an electronic lab notebook/journal for the course. In addition, students have been automatically enrolled in [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&mode=reset BIOL/CMSI 367-01 on MyLMUConnect]. The two cross-listed sections are using the same site, which is listed as "Biological Databases Fall 2015" with no department or course number. The MyLMUConnect site will be used for materials that cannot be made public on this wiki, including grades.

=== E-mail Communication ===

At times we will communicate with the entire class using campus e-mail systems, so it is essential that you regularly check your lion.lmu.edu e-mail address or forward your lion account e-mail to your preferred e-mail address. Messages sent to the instructors at night or on the weekend will be answered the next school day. Please CC both instructors on all e-mail messages related to this class.

=== Required Materials ===

==== Texts ====

There is no required text to purchase for the course; materials will be put on reserve at Hannon Library or will be available online on this wiki or the MyLMUConnect site. Specific reading assignments are given on the course schedule and should be completed before coming to class.
* This text has been placed on reserve at the library: Jeffrey Ullman and Jennifer Widom, ''A First Course in Database Systems'', Third Edition. Prentice Hall, 2008.
* Assorted handouts, articles, and sample code will be distributed throughout the semester.
* Additional information is also available on the web; do not hesitate to look for further sources of information regarding the concepts, techniques, tools, and paradigms that we will discuss.

==== Materials (must be brought to each class meeting) ====

* 3-ring binder with all course handouts
* Pen, pencil, extra paper
* USB flash drive to store data
* [https://xmlpipedb.cs.lmu.edu/biodb/fall2015/index.php/Special:UserLogin/signup Account for this wiki]
* [https://secure.cs.lmu.edu/ Computer Science account]
* [http://www.github.com GitHub account]

=== Course Description ===

The disciplines of biology and computer science come together in bioinformatics, where computational tools are needed to manage and analyze the flood of data coming from new genomics technologies. Biological databases form a significant part of this young and exciting field. This course introduces students to both the biology and computer science expertise needed to understand, use, and develop biological databases. Biology topics include the fundamentals of genetics, molecular biology, and biochemistry needed to understand the data stored in biological databases, as well as the biotechnologies used to gather these data in a high-throughput manner. Computer science topics include what biological databases are, why they are important (and needed), and the challenges that arise in compiling them effectively. Biology and computer science lectures on topics that are relevant to biological databases are coupled with hands-on experience with a variety of software packages ranging from bioinformatics utilities to general-purpose database and software development tools. After learning how to use a biological database, students will be asked to build one of their own.

==== Course Objectives and Learning Outcomes ====

This course is built upon L. Dee Fink’s ''taxonomy of significant learning'', as applied to biological databases. Long after the course concludes, our hope is that:
* You understand how biological information is encoded in the genome and can apply this knowledge to a variety of biological tasks and problems
* You understand the core concepts, structure, and functions of a database, ranging from individual files to a full relational database management system, and can perform useful tasks with such data
* You show discipline and proficiency in day-to-day science and engineering best practices, such as maintaining journals and notebooks, managing your files and code, and critically evaluating scientific and technical information
* You recognize and care about how the biological and technological issues presented in this course relate to and affect society, our daily lives, and ourselves
* You have some skills and tools for “leaving your comfort zone,” flourishing outside of it, and learning more about biology and computer science on your own
* You learn how to communicate and work effectively with colleagues from different disciplines

==== University Core Curriculum ====

This course fulfills the following requirements in the University Core Curriculum:
* Integrations: Interdisciplinary Connections
* Upper Division Information Literacy Flag
* Upper Division Oral Communication Flag

=== Course Work and Grading ===

Letter grades are determined as follows: ≥ 90% gets an A– or better; ≥ 80% gets a B– or better; ≥ 70% gets a C– or better. The instructors may curve grades upward based on qualitative considerations such as degree of difficulty, effort, class participation, time constraints, and overall attitude throughout the course. Grades are never curved downward. Current grades will be made available at MyLMU Connect (the system formerly known as Blackboard).

Your work in this course will be assessed in five areas:

{| style="border: 1px dashed rgb(47, 111, 171); background: rgb(249, 249, 249); margin: 1em 0; padding: 1em" cellpadding="5ex"
| Weekly electronic lab notebook/journal assignments, individual (10 points each)
| style="text-align: right" | 140
| points
|-
| Weekly electronic lab notebook/journal assignments, shared (3 points each)
| style="text-align: right" | 42
| points
|-
| Oral presentations
| style="text-align: right" | 105
| points 
|-
| Information literacy (additional points added to two journal assignments)
| style="text-align: right" | 24
| points 
|-
| Final Project Deliverables (including written report)
| style="text-align: right" | 175
| points
|-
| '''Total'''
| style="text-align: right" | '''486'''
| '''points'''
|}

==== Electronic Laboratory Notebook/Journal ====

One of the most important skills you can develop as a scientist is keeping an excellent laboratory notebook. For computational research, the equivalent of the biology paper-based lab notebook is documentation of your “workflow”. For this course you will practice documentation skills by keeping an electronic lab notebook or journal. The technology we will use is this wiki, that we will create and edit during the semester. You will create an individual user page and make weekly entries that the instructors will read and grade. You will use the wiki to complete the assignments as well. The following guidelines apply:
* Your weekly journal entry is typically due every midnight on Tuesday PST (''Monday night/Tuesday morning''); consult the schedule for specific due dates for each assignment.
* Each weekly assignment has an individual component and a shared component. You will earn 10 points per weekly submission for the individual journal entry and 3 points per submission for the shared journal entry. '''''Late journal entries will be accepted up to one week later for up to half credit.'''''
* The instructors will read and comment on how to improve your journal entries on your user talk pages.
* Depending on the type of assignment for that week, you may be given the opportunity to make improvements to previous journal entries as the semester progresses.
* Generally, your journal entries will consist of:
** Your electronic laboratory notebook and other documentation for hands-on exercises and projects
** Answers to any specific questions posed in the exercise
** Reflection on your learning
* For most weeks in the semester, you will be assigned a "homework partner" from the complementary biology or computer science discipline. You will be expected to consult with your partner, sharing your domain expertise, in order to complete the assignment. However, unless otherwise stated, each partner must submit his or her own work as the individual journal entry (direct copies of each other's work is not allowed).

==== Oral Presentations ====

You will give three oral presentations in this course. The first two will be in the format of a “Journal Club” presentation where students will present and lead discussion of research articles from the primary literature. The third will be a research presentation on your final project. '''''Because that day’s class content is dependent upon each student being ready to present and lead discussion, late journal club presentations will not be accepted. An unexcused absence from a journal club presentation will result in a grade of zero for the presentation.'''''

==== Final Group Project ====

* Overall Description of the [[Gene Database Project]], including entire list of deliverables
* [[Gene Database Project Deliverables]]

{| style="width: 50%"
| '''Teams'''
| '''Guilds'''
|-
| [[Team1]]
| [[Project Manager]]
|-
| [[Team2]]
| [[GenMAPP User]]
|-
| [[Team3]]
| [[Quality Assurance]]
|-
| [[Team4]]
| [[Coder]]
|}

In addition to the research presentation, the culmination of your final project will be the preparation of a written laboratory report in the style of a manuscript that could be submitted to a peer-reviewed journal, specifically [https://peerj.com/computer-science/ PeerJ Computer Science]. Specific instructions are posted [[Gene Database Project Deliverables|here]]. '''''The Final Written Report cannot be accepted any later than Friday, December 18 at 4:30 PM. The Final Written Report must be completed to receive a passing grade in the course.'''''

==== Extra Credit ====

Students may accumulate up to 12.5 points toward their final grade in extra credit by attending Department seminars in Biology or Electrical Engineering & Computer Science and completing the seminar sheets. Each seminar attended is worth 2.5 points with up to 5 seminars (12.5 points) total. You must attend the entire seminar from start to finish and personally turn in your seminar sheet to a faculty member at the end of the seminar.

Certain, non-Biology/Computer Science Department seminars may be approved in advance for extra credit at the instructors’ discretion. To receive credit for these seminars, you must turn in a one-page '''''hard copy''''' of your summary of the seminar '''''in class, within one week''''' of the date of the seminar or they will not count as extra credit.

=== Work Load Expectations ===

In line with [http://www.lmu.edu/Assets/LMU+Credit+Hour+Policy_Final.pdf LMU’s Credit Hour Policy], the work load expectation for this course is that for every one hour (50 minutes) of classroom instruction, you will complete a minimum of two hours of out-of-class student work each week. This is a 3-unit course with 3 hours (150 minutes) of instruction per week. Thus the expectation is that you will complete 6 hours of work outside of class per week.

=== University Policy on Academic Honesty ===

Loyola Marymount University expects high standards of honesty and integrity from all members of its community. All students are expected to follow the LMU honor code. As stated in the LMU Undergraduate Bulletin, “Examples of academic dishonesty include, but are not limited to, the following: all acts of cheating on assignments or examinations, or facilitating other students’ cheating; plagiarism; fabrication of data, including the use of false citations; improper use of non-print media; unauthorized access to computer accounts or files or other privileged information and improper use of Internet sites and resources.” Attempting to gain extra credit for a seminar that you did not attend or assisting another student to do this are examples of academic dishonesty and will be considered a violation of the Honor Code. [http://bulletin.lmu.edu/content.php?catoid=6&navoid=250#LMU_Honor_Code_and_Process Click here for an online version of the LMU Honor Code and Process].

You are required to sign the Honor Code Agreement for this course.

==== Academic Honesty Resources ====
* [[Media:BIOL367_Fall2015_HonorCodeAgreement.pdf|Honor Code Agreement]]
* [http://bulletin.lmu.edu/content.php?catoid=6&navoid=250#LMU_Honor_Code_and_Process LMU Honor Code and Process].
* [http://www.bio.davidson.edu/dept/plagiarism.html The Davidson College Department of Biology Statement on Plagiarism]
* [[Media:BIOL367_Fall2015_GuidelinesforLiteratureCitations.pdf | Guidelines for Literature Citations in a Scientific Paper]]
** [http://libguides.lmu.edu/content.php?pid=472661&sid=3874648 APA Style for References]
* [[Media:Avoiding_plagiarism.pptx | Academic Resource Center presentation on plagiarism]] (PowerPoint presentation)
* [http://myweb.lmu.edu/dondi/share/misc/plagiarism.pdf Notes on plagiarism from an electrical engineering & computer science perspective]

=== Americans with Disabilities Act—Special Accommodations ===

Students with special needs who require reasonable modifications, special assistance, or accommodations in this course should promptly direct their request to the Disability Support Services (DSS) Office. Any student who currently has a documented disability (ADHD, Autism Spectrum Disorder, Learning, Physical, or Psychiatric) needing academic accommodations should contact the DSS Office (Daum Hall 2nd floor, 310-338-4216) as early in the semester as possible. All discussions will remain confidential. Please visit [http://www.lmu.edu/dss www.lmu.edu/dss] for additional information. In addition, please schedule an appointment with the instructors early in the semester to discuss any accommodations for this course for which you have been approved.

=== Revision Notice ===

If necessary, this syllabus and its contents are subject to revision; students are responsible for any changes or modifications announced in class. The most current version of this information resides on this page, the course web site at http://xmlpipedb.cs.lmu.edu/biodb/fall2015/.

Coder

2015-11-02T02:45:03Z

Dondi: /* Updating and Committing Code */ Adapt update/commit steps for git.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] (which, at this writing, is ''JDK 8u65'')
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used:
#** Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Follow the instructions in the [[#GenMAPP Builder Project Setup and Initial Build|GenMAPP Builder Project Setup and Initial Build]] section of this wiki page in order to:
#* Set up a functioning Eclipse development environment for your branch of GenMAPP Builder.
#* Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

Follow the instructions in the [[#Adding a Species Profile to GenMAPP Builder|Adding a Species Profile to GenMAPP Builder]] section of this wiki page in order to:
* Add a ''species profile'' to the GenMAPP Builder code base.
* Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
* Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
** ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder. You can only run ''GenMAPP'' and ''MAPPFinder'' on Windows, but you can build and run ''GenMAPP Builder'' on any platform that supports PostgreSQL and JDK 8.

Although there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Prerequisites ===

# Make sure that you have already accomplished the [[#Milestone 1: Version Control Setup|version control setup milestone (Milestone 1)]].
# Make sure that you have already downloaded and installed the software mentioned in [[#Milestone 2: “Developer Rig” Setup and Initial As-Is Build|Milestone 2]] (first item).

=== GitHub Repository Clone Setup ===

# Determine the desired location (on your development computer) for your local copy of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub repository].
# <code>cd</code> to this location.
# Clone the repository:<pre>git clone https://github.com/lmu-bioinformatics/xmlpipedb.git</pre>
# <code>cd</code> into the clone folder <code>xmlpipedb</code>:<pre>cd xmlpipedb</pre>
# Switch to your branch:<pre>git checkout <your-branch-name></pre>
# You should see a message like this: <pre>Branch <your-branch-name> set up to track remote branch <your-branch-name> from origin. Switched to a new branch '<your-branch-name>'</pre>
#: …this is OK and expected.

=== Eclipse Workspace Setup ===

[[Image:Initial-eclipse-workspace.png|thumb]]
# Run Eclipse.
# When Eclipse asks you to select a workspace, click the ''Browse…'' and navigate to your repository clone folder. Select it and click on ''Open''.
# Verify that your repository clone folder is what is listed as the ''Workspace:''. Click on ''OK''.
# You will see an introductory display with assorted menu items. Click the ''Workbench'' button (upper-right corner, with a 3D arrow for an icon).
# If everything went well, you should now see an empty developer area, showing your repository clone folder in the window title with a ''Project Explorer'' tab on the left (see screenshot for an example).

=== Java Project Setup ===

# Right-click anywhere within the empty ''Project Explorer'' tab and choose '''New > Project…''' from the menu that appears.
# Choose ''Java Project'' from the list of “wizards” and click on the ''Next >'' button.
# On the next panel, enter <code>gmbuilder</code> as the ''Project name:''. Note how the ''Location:'' field underneath, although disabled, should show your cloned repository folder location with ''/gmbuilder'' appended to it.
# Verify that the ''JRE'' section is showing Java 8 or 1.8. If not, talk to Dr. Dionisio.
# Click on the ''Finish'' button (yes, there is no need for further configuration).
# If Eclipse asks you whether you want to open the “Java perspective,” respond with ''Yes''. You may check the ''Remember my decision'' checkbox if you wish.
# You should now see a ''gmbuilder'' project folder in the ''Project Explorer'' tab.
# If the ''gmbuilder'' project folder icon shows a red ''x'' icon, do the following:
#* Right-click on the ''gmbuilder'' item and choose '''Properties''' from the menu that appears.
#* Click on the ''Java Compiler'' item. ''Click on the item name itself, not its triangle.''
#* You should see that ''Compiler compliance level:'' is set to ''1.6''. If so, uncheck the ''Enable project specific settings'' checkbox.
#* Click the ''Apply'' button. If Eclipse asks you whether it is OK to rebuild the project, click ''Yes''.
#* Click on ''OK'' to dismiss the ''Properties'' dialog.
#* See if the red ''x'' icon has disappeared.
# If the red ''x'' icon persists, please show your setup to fellow guild members or Dr. Dionisio for troubleshooting.

=== Initial Build ===

{{ Build Instructions }}

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose '''New > Class''' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose '''Team > Synchronize Workspace''' from the menu that appears.
#* If Eclipse asks you whether it is OK to enter the ''Team Synchronizing'' perspective, respond with ''Yes''. You may check the ''Remember my decision'' checkbox so that this does not happen again.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of left-pointing, blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Pull'' from the menu that appears. You can also click the ''Pull'' button from the toolbar of the ''Synchronize'' tab.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective and run a quick ''clean, dist'' cycle to make sure that there are no errors.
# If everything works out, go to ''Synchronize Workspace'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Synchronize'' tab with no incoming code. At this point, go ahead and commit the right-pointing gray-arrowed files by right-clicking on them (or their containing folder) and choosing ''Commit...''.
#* ''If you see files that you did not consciously edit being marked for commit, check with Dr. Dionisio first.'' These may be build products or local settings files, neither of which should be committed.
# Verify that the files you want to commit are checked in the ensuing ''Commit Changes'' dialog.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# When you are ready, you may choose either ''Commit and Push'' or just ''Commit''.
#* ''Commit'' will preserve a version of these files, but not yet update the GitHub website. This allows you to keep on working without others being affected by what you do (yet).
#* ''Commit and Push'' will both save a version of your files ''and'' send the latest changes to GitHub. Your changes will then be visible to others.
#* You may ''Push'' as a separate step; this is available either from the '''Team''' right-click menu or the toolbar of the ''Synchronize'' tab.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
{{ Build Instructions }}
# Run PostgreSQL (''pgAdmin III'') and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Quality Assurance

2015-11-02T02:33:41Z

Dondi: Move tally engine customization to the Quality Assurance guild page.

{{Gene Database Project Links}}

The Quality Assurance team member is the resident expert on species ID systems and formats. He or she should be proficient with ''XMLPipeDB Match'', SQL queries in PostgreSQL, Microsoft Excel, and Microsoft Access to navigate through the data and find missing IDs, discrepancies, sanity checks, etc.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 1: Initial Database Export ===

# (with Coders) Get a full import-export cycle done.
# (with Coders) Decide on a file/version management scheme/system.
# Learn the ID systems:
#* Systems that are the same for each species (hint: guild members help each other out by posting the relevant information on this page)
#** UniProt
#** RefSeq
#** GeneID (EntrezGene from NCBI)
#** GO
#* The OrderedLocusNames for your species

=== Milestone 2: ID Pattern Definition and Verification ===

# Characterize regular expression patterns to detect the IDs ([[How Do I Count Thee? Let Me Count The Ways|for filtering then counting]]).
#* XMLPipeDB Match utility
#* Direct SQL queries in PostgreSQL
#* For example, the ''Vibrio'' IDs were of the form <code>VC####</code> or <code>VC_####</code>; how would you express that in Match or as an SQL query?
#* Table inspection/filtering/sorting in Microsoft Access
#* If needed, side-by-side sorted comparisons in Microsoft Excel ([[Using Microsoft Excel to Compare ID Lists|as described here]])
# Document/log all work done, problems encountered, and how they were resolved.

=== Milestone 3: Tally Engine Configuration ===

Along with your Coder, customize the Tally Engine setup for your species as specified in [[#Customizing the IDs that the Tally Engine Counts|the steps described below]]. You will want to add, at the very least, the ''ordered locus'' IDs for your species.

=== Milestone 4: Final Documentation ===

# Document the relational database schema for the gene database.
# Create the ReadMe with comparisons to MOD for your species.

== Customizing the IDs that the Tally Engine Counts ==

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; the QA team member should be the authority on this.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Work with the Coder to open the Eclipse project for your team’s branch of GenMAPP Builder.
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T02:29:51Z

Dondi: Point milestone 3 instructions to the detailed technical section.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] (which, at this writing, is ''JDK 8u65'')
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used:
#** Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Follow the instructions in the [[#GenMAPP Builder Project Setup and Initial Build|GenMAPP Builder Project Setup and Initial Build]] section of this wiki page in order to:
#* Set up a functioning Eclipse development environment for your branch of GenMAPP Builder.
#* Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

Follow the instructions in the [[#Adding a Species Profile to GenMAPP Builder|Adding a Species Profile to GenMAPP Builder]] section of this wiki page in order to:
* Add a ''species profile'' to the GenMAPP Builder code base.
* Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
* Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
** ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder. You can only run ''GenMAPP'' and ''MAPPFinder'' on Windows, but you can build and run ''GenMAPP Builder'' on any platform that supports PostgreSQL and JDK 8.

Although there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Prerequisites ===

# Make sure that you have already accomplished the [[#Milestone 1: Version Control Setup|version control setup milestone (Milestone 1)]].
# Make sure that you have already downloaded and installed the software mentioned in [[#Milestone 2: “Developer Rig” Setup and Initial As-Is Build|Milestone 2]] (first item).

=== GitHub Repository Clone Setup ===

# Determine the desired location (on your development computer) for your local copy of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub repository].
# <code>cd</code> to this location.
# Clone the repository:<pre>git clone https://github.com/lmu-bioinformatics/xmlpipedb.git</pre>
# <code>cd</code> into the clone folder <code>xmlpipedb</code>:<pre>cd xmlpipedb</pre>
# Switch to your branch:<pre>git checkout <your-branch-name></pre>
# You should see a message like this: <pre>Branch <your-branch-name> set up to track remote branch <your-branch-name> from origin. Switched to a new branch '<your-branch-name>'</pre>
#: …this is OK and expected.

=== Eclipse Workspace Setup ===

[[Image:Initial-eclipse-workspace.png|thumb]]
# Run Eclipse.
# When Eclipse asks you to select a workspace, click the ''Browse…'' and navigate to your repository clone folder. Select it and click on ''Open''.
# Verify that your repository clone folder is what is listed as the ''Workspace:''. Click on ''OK''.
# You will see an introductory display with assorted menu items. Click the ''Workbench'' button (upper-right corner, with a 3D arrow for an icon).
# If everything went well, you should now see an empty developer area, showing your repository clone folder in the window title with a ''Project Explorer'' tab on the left (see screenshot for an example).

=== Java Project Setup ===

# Right-click anywhere within the empty ''Project Explorer'' tab and choose '''New > Project…''' from the menu that appears.
# Choose ''Java Project'' from the list of “wizards” and click on the ''Next >'' button.
# On the next panel, enter <code>gmbuilder</code> as the ''Project name:''. Note how the ''Location:'' field underneath, although disabled, should show your cloned repository folder location with ''/gmbuilder'' appended to it.
# Verify that the ''JRE'' section is showing Java 8 or 1.8. If not, talk to Dr. Dionisio.
# Click on the ''Finish'' button (yes, there is no need for further configuration).
# If Eclipse asks you whether you want to open the “Java perspective,” respond with ''Yes''. You may check the ''Remember my decision'' checkbox if you wish.
# You should now see a ''gmbuilder'' project folder in the ''Project Explorer'' tab.
# If the ''gmbuilder'' project folder icon shows a red ''x'' icon, do the following:
#* Right-click on the ''gmbuilder'' item and choose '''Properties''' from the menu that appears.
#* Click on the ''Java Compiler'' item. ''Click on the item name itself, not its triangle.''
#* You should see that ''Compiler compliance level:'' is set to ''1.6''. If so, uncheck the ''Enable project specific settings'' checkbox.
#* Click the ''Apply'' button. If Eclipse asks you whether it is OK to rebuild the project, click ''Yes''.
#* Click on ''OK'' to dismiss the ''Properties'' dialog.
#* See if the red ''x'' icon has disappeared.
# If the red ''x'' icon persists, please show your setup to fellow guild members or Dr. Dionisio for troubleshooting.

=== Initial Build ===

{{ Build Instructions }}

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose '''New > Class''' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
{{ Build Instructions }}
# Run PostgreSQL (''pgAdmin III'') and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T02:23:56Z

Dondi: /* Creating a Distribution */ Switch to template.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] (which, at this writing, is ''JDK 8u65'')
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used:
#** Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Follow the instructions in the [[#GenMAPP Builder Project Setup and Initial Build|GenMAPP Builder Project Setup and Initial Build]] section of this wiki page in order to:
#* Set up a functioning Eclipse development environment for your branch of GenMAPP Builder.
#* Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder. You can only run ''GenMAPP'' and ''MAPPFinder'' on Windows, but you can build and run ''GenMAPP Builder'' on any platform that supports PostgreSQL and JDK 8.

Although there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Prerequisites ===

# Make sure that you have already accomplished the [[#Milestone 1: Version Control Setup|version control setup milestone (Milestone 1)]].
# Make sure that you have already downloaded and installed the software mentioned in [[#Milestone 2: “Developer Rig” Setup and Initial As-Is Build|Milestone 2]] (first item).

=== GitHub Repository Clone Setup ===

# Determine the desired location (on your development computer) for your local copy of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub repository].
# <code>cd</code> to this location.
# Clone the repository:<pre>git clone https://github.com/lmu-bioinformatics/xmlpipedb.git</pre>
# <code>cd</code> into the clone folder <code>xmlpipedb</code>:<pre>cd xmlpipedb</pre>
# Switch to your branch:<pre>git checkout <your-branch-name></pre>
# You should see a message like this: <pre>Branch <your-branch-name> set up to track remote branch <your-branch-name> from origin. Switched to a new branch '<your-branch-name>'</pre>
#: …this is OK and expected.

=== Eclipse Workspace Setup ===

[[Image:Initial-eclipse-workspace.png|thumb]]
# Run Eclipse.
# When Eclipse asks you to select a workspace, click the ''Browse…'' and navigate to your repository clone folder. Select it and click on ''Open''.
# Verify that your repository clone folder is what is listed as the ''Workspace:''. Click on ''OK''.
# You will see an introductory display with assorted menu items. Click the ''Workbench'' button (upper-right corner, with a 3D arrow for an icon).
# If everything went well, you should now see an empty developer area, showing your repository clone folder in the window title with a ''Project Explorer'' tab on the left (see screenshot for an example).

=== Java Project Setup ===

# Right-click anywhere within the empty ''Project Explorer'' tab and choose '''New > Project…''' from the menu that appears.
# Choose ''Java Project'' from the list of “wizards” and click on the ''Next >'' button.
# On the next panel, enter <code>gmbuilder</code> as the ''Project name:''. Note how the ''Location:'' field underneath, although disabled, should show your cloned repository folder location with ''/gmbuilder'' appended to it.
# Verify that the ''JRE'' section is showing Java 8 or 1.8. If not, talk to Dr. Dionisio.
# Click on the ''Finish'' button (yes, there is no need for further configuration).
# If Eclipse asks you whether you want to open the “Java perspective,” respond with ''Yes''. You may check the ''Remember my decision'' checkbox if you wish.
# You should now see a ''gmbuilder'' project folder in the ''Project Explorer'' tab.
# If the ''gmbuilder'' project folder icon shows a red ''x'' icon, do the following:
#* Right-click on the ''gmbuilder'' item and choose '''Properties''' from the menu that appears.
#* Click on the ''Java Compiler'' item. ''Click on the item name itself, not its triangle.''
#* You should see that ''Compiler compliance level:'' is set to ''1.6''. If so, uncheck the ''Enable project specific settings'' checkbox.
#* Click the ''Apply'' button. If Eclipse asks you whether it is OK to rebuild the project, click ''Yes''.
#* Click on ''OK'' to dismiss the ''Properties'' dialog.
#* See if the red ''x'' icon has disappeared.
# If the red ''x'' icon persists, please show your setup to fellow guild members or Dr. Dionisio for troubleshooting.

=== Initial Build ===

{{ Build Instructions }}

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
{{ Build Instructions }}
# Run PostgreSQL (''pgAdmin III'') and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T02:21:38Z

Dondi: /* Initial Build */ Switch to template.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] (which, at this writing, is ''JDK 8u65'')
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used:
#** Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Follow the instructions in the [[#GenMAPP Builder Project Setup and Initial Build|GenMAPP Builder Project Setup and Initial Build]] section of this wiki page in order to:
#* Set up a functioning Eclipse development environment for your branch of GenMAPP Builder.
#* Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder. You can only run ''GenMAPP'' and ''MAPPFinder'' on Windows, but you can build and run ''GenMAPP Builder'' on any platform that supports PostgreSQL and JDK 8.

Although there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Prerequisites ===

# Make sure that you have already accomplished the [[#Milestone 1: Version Control Setup|version control setup milestone (Milestone 1)]].
# Make sure that you have already downloaded and installed the software mentioned in [[#Milestone 2: “Developer Rig” Setup and Initial As-Is Build|Milestone 2]] (first item).

=== GitHub Repository Clone Setup ===

# Determine the desired location (on your development computer) for your local copy of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub repository].
# <code>cd</code> to this location.
# Clone the repository:<pre>git clone https://github.com/lmu-bioinformatics/xmlpipedb.git</pre>
# <code>cd</code> into the clone folder <code>xmlpipedb</code>:<pre>cd xmlpipedb</pre>
# Switch to your branch:<pre>git checkout <your-branch-name></pre>
# You should see a message like this: <pre>Branch <your-branch-name> set up to track remote branch <your-branch-name> from origin. Switched to a new branch '<your-branch-name>'</pre>
#: …this is OK and expected.

=== Eclipse Workspace Setup ===

[[Image:Initial-eclipse-workspace.png|thumb]]
# Run Eclipse.
# When Eclipse asks you to select a workspace, click the ''Browse…'' and navigate to your repository clone folder. Select it and click on ''Open''.
# Verify that your repository clone folder is what is listed as the ''Workspace:''. Click on ''OK''.
# You will see an introductory display with assorted menu items. Click the ''Workbench'' button (upper-right corner, with a 3D arrow for an icon).
# If everything went well, you should now see an empty developer area, showing your repository clone folder in the window title with a ''Project Explorer'' tab on the left (see screenshot for an example).

=== Java Project Setup ===

# Right-click anywhere within the empty ''Project Explorer'' tab and choose '''New > Project…''' from the menu that appears.
# Choose ''Java Project'' from the list of “wizards” and click on the ''Next >'' button.
# On the next panel, enter <code>gmbuilder</code> as the ''Project name:''. Note how the ''Location:'' field underneath, although disabled, should show your cloned repository folder location with ''/gmbuilder'' appended to it.
# Verify that the ''JRE'' section is showing Java 8 or 1.8. If not, talk to Dr. Dionisio.
# Click on the ''Finish'' button (yes, there is no need for further configuration).
# If Eclipse asks you whether you want to open the “Java perspective,” respond with ''Yes''. You may check the ''Remember my decision'' checkbox if you wish.
# You should now see a ''gmbuilder'' project folder in the ''Project Explorer'' tab.
# If the ''gmbuilder'' project folder icon shows a red ''x'' icon, do the following:
#* Right-click on the ''gmbuilder'' item and choose '''Properties''' from the menu that appears.
#* Click on the ''Java Compiler'' item. ''Click on the item name itself, not its triangle.''
#* You should see that ''Compiler compliance level:'' is set to ''1.6''. If so, uncheck the ''Enable project specific settings'' checkbox.
#* Click the ''Apply'' button. If Eclipse asks you whether it is OK to rebuild the project, click ''Yes''.
#* Click on ''OK'' to dismiss the ''Properties'' dialog.
#* See if the red ''x'' icon has disappeared.
# If the red ''x'' icon persists, please show your setup to fellow guild members or Dr. Dionisio for troubleshooting.

=== Initial Build ===

{{ Build Instructions }}

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Main Page

2015-11-02T02:20:43Z

Dondi: /* Students */ Add gene database project links for easy access.

__NOTOC__
<div style="text-align: center; font-size: 175%">Loyola Marymount University</div>
<div style="text-align: center; font-size: 250%; line-height: 1.25em">'''BIOL/CMSI 367-01: Biological Databases'''</div>
<div style="text-align: center; font-size: 175%; margin-bottom: 0.85em">Fall 2015</div>

<div class="toc-container">__TOC__</div>

== Announcements ==

* The Microsoft Excel portion of the ID investigation protocol has been added to the wiki: [[Using Microsoft Excel to Compare ID Lists]]. The link has also been added to the course schedule below for Week 9.
:Please note that this version is similar, but not identical, to the process shown in class ([[User:Dondi|Dondi]] misremembered the function to use; it was supposed to be <code>match</code>, as shown in the wiki page, and not <code>lookup</code>). Furthermore, instructions for exporting from PostgreSQL (you might recall that the numbers did not match there either) are included. —''[[User:Dondi|Dondi]] ([[User talk:Dondi|talk]]) 18:49, 29 October 2015 (PDT)''
*'''''TriBeta Halloween Party and Costume Contest:''''' Friday, October 30, 6:00 PM, LSB Terrace, $3 (to cover food). RSVP to ralkhaya at lion dot lmu dot edu. Come see if you can beat me in the costume contest! ''— [[User:Kdahlquist|Kdahlquist]] ([[User talk:Kdahlquist|talk]]) 09:46, 28 October 2015 (PDT)''
* An initial round of feedback to the [[Week 6]] assignment has been posted to your respective user talk pages. The database part is not finished, but I figured some feedback and basics on the best practice stuff will be useful as you prepare your [[Week 8]] work (as in, get used to making the best practices routine so you can avoid nagging deductions week after week). —''[[User:Dondi|Dondi]] ([[User talk:Dondi|talk]]) 21:30, 25 October 2015 (PDT)''
* To report issues with the configuration of the computers in Seaver 120, please record it on [[Talk:Software_Configuration | this wiki page.]] The complete list of software that should be installed for this course is found [[Software Configuration | here]]. Note that this configuration is Windows-specific. Some software may not be available for a Mac or may look differently.

=== Upcoming Seminars ===

* '''Chemistry & Biochemistry Department Seminar:''' Thursday, October 29, 11:50 AM, Seaver 200, Presentation by first candidate for the Biochemistry Faculty Search.
* '''Biology Department Seminar:''' Friday, October 30, 2:00 PM, LSB Auditorium, Dr. Ashley Robart, LMU Biology, ''MALE QUALITY INFLUENCES PARENTAL CARE, BUT NOT FEMALE COMPETITION, IN A MONOGAMOUS, BIPARENTAL FISH.''
* '''Noyce Poster Session:''' Friday, October 30, 4:00 – 6:00 PM, Hannon Courtyard of the LSB (area just outside the west entrance). The NOYCE Summer Interns will be presenting the posters they made highlighting the research they have done. The Poster Session will be a fantastic opportunity to talk to the presenters and ask them all about their research! To receive the extra credit, fill out one of the "seminar" sheets available according to the instructions and turn it in to a professor before you leave the poster session.
* '''Chemistry & Biochemistry Department Seminar:''' Thursday, November 5, 11:50 AM, Seaver 200, Presentation by second candidate for the Biochemistry Faculty Search.
* '''Biology Department Seminar:''' Friday, November 6, 2:00 PM, LSB Auditorium, Dr. Wes Dowd, LMU Biology.
* '''Chemistry & Biochemistry Department Seminar:''' Tuesday, November 10, 11:50 AM, Seaver 200, Presentation by third candidate for the Biochemistry Faculty Search.
* '''Chemistry & Biochemistry Department Seminar:''' Thursday, November 12, 11:50 AM, Seaver 200, Presentation by fourth candidate for the Biochemistry Faculty Search.
* '''Biology Department Seminar:''' Friday, December 4, 2:00 PM, LSB Auditorium, Dr. Deepa Dabir, LMU Biology.



== Instructors ==
* [[User:Kdahlquist|Kam D. Dahlquist]]
* [[User:Dondi|John David N. Dionisio]]

== Students ==

{| style="margin-left: auto; margin-right: auto"
| style="padding-right: 1em" | [[User: Malverso | Mary Alverson]]
| style="padding-right: 1em" | [[User:Rlegaspi | Ron Legaspi]]
| style="padding-right: 1em" | [[User:Troque | Trixie Roque]]
| style="padding-right: 1em" | [[User:Jwoodlee | Jake Woodlee]]
|-
| style="padding-right: 1em" | [[User:Nanguiano | Nicole Anguiano]]
| style="padding-right: 1em" | [[User:Blitvak | Brandon Litvak]]
| style="padding-right: 1em" | [[user:Msaeedi23 | Mahrad Saeedi]]
| style="padding-right: 1em" | [[User:Kwyllie | Kevin Wyllie]]
|-
| style="padding-right: 1em" | [[User:Bklein7 | Brandon Klein]]
| style="padding-right: 1em" | [[User:Lenaolufson | Elena Olufson]]
| style="padding-right: 1em" | [[User:Emilysimso | Emily Simso]]
| style="padding-right: 1em" | [[User:Eyanosch | Erich Yanoschik]]
|-
| style="padding-right: 1em" | [[User:Jkuroda | Josh Kuroda]]
| style="padding-right: 1em" | [[User:Vpachec3 | Veronica Pacheco]]
| style="padding-right: 1em" | [[User:Anuvarsh | Anindita Varshneya]]
| style="padding-right: 1em" | [[User:Kzebrows | Kristin Zebrowski]]
|}

{{ Gene Database Project Links }}

== Course Schedule ==

The current course schedule is shown below. In addition to the listed in-class agendas, the following guidelines are also notable:

* Unless otherwise stated on the schedule, your weekly journals/assignments are due on Tuesdays at midnight (Monday night/Tuesday morning). In cases where subsequent revisions or corrections will be accepted, the instructors will provide feedback and submission deadlines on a per-assignment basis.
* Reading assignments should be completed ''before'' coming to class.
* Dr. Dionisio’s office hours ({{ Dondi Office Hours }}) can be viewed as unofficial lab sessions: use them for IT help or desktop support

=== Part 1: Building Blocks (Genetic Code and Manipulating Text) ===

{| class="schedule-table"
! #
! Date
! Reading
! Agenda
! Journal
|-
! 1
| Tuesday<br/>9/1/2015
|
| Syllabus walkthrough
* ''my.cs.lmu.edu'' account: Visit https://secure.cs.lmu.edu to get a computer science account if you don’t already have one
* Class wiki account: Set up a username and password for this wiki site

Pairwise interviews (report back after each bullet)
* Basic acquaintance information
* Like a cell/not like a cell; like a computer/not like a computer

| rowspan="2" | [[Week 1]]

(due at midnight 9/8)

[[Class_Journal_Week_1 | Class Journal Week 1]]
|-
!
| Thursday<br/>9/3/2015
|
[http://dl.acm.org/ft_gateway.cfm?id=1378711 ''Voices of Computing''] and [http://dl.acm.org/ft_gateway.cfm?id=1272529 ''Computing is a Natural Science''] by Peter Denning (these links should be accessed from within LMU to get the full article)

[https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&mode=reset Chapter 1 of ''On Becoming a Biologist''] by John Janovy, Jr. (on MyLMUConnect)
| Discuss the [[Week 1]] assignment (accounts should be ready to wiki at this point)
* Quick wiki overview
* User wiki page setup

{{Course Schedule Divider}}
|-
! 2
| Tuesday<br/>9/8/2015
| [http://www.ncbi.nlm.nih.gov/books/NBK21134/#A5234 Ch. 1.1 ''Genomes 2'']

[http://www.nature.com/nature/dna50/watsoncrick.pdf Watson & Crick (1953)]

[http://www.nature.com/nature/dna50/franklingosling.pdf Franklin & Gosling (1953)]

| DNA
* DNA structure and Genetic Code
* [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&content_id=_1103141_1 Slides shown in class on MyLMU Connect]
| rowspan="2" | [[Week 2]]

(due at midnight 9/15)

[[Class Journal Week 2]]
|-
!
| Thursday<br/>9/10/2015
| On [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&mode=reset MyLMUConnect]:

Nirenberg (2004) ''Deciphering the Genetic Code''

Kaji & Kaji (2004) ''Setting the Record Straight''

Moody (2004) ''Digital Code of Life'', Ch. 1

Hayes (2004) ''Ode to the Code''

[http://www.ncbi.nlm.nih.gov/books/NBK21121/ Brown, T.A. (2002) ''Genomes 2'', Ch. 3, especially section 3.3.2]
| Central dogma
* [http://www.youtube.com/watch?v=u9dhO0iCLww Berkeley translation video]
* This [http://www.dnai.org/a/index.html web site] has links to animations of the replication, transcription, and translation processes.
* Slides shown in class
* [[Media:BIOL367_Fall2015_GeneticCode-GeneExpression.pdf | Genetic Code-Gene Expression Pencil Exercise]]

{{Course Schedule Divider}}
|-
! 3
| Tuesday<br/>9/15/2015
| [http://www.bloomberg.com/graphics/2015-paul-ford-what-is-code/ Ford (2015) ''What is Code?'']
* Emphasis on first two sections (“The Man in the Taupe Blazer” and “Let’s Begin”), but feel free to read further

[http://myweb.lmu.edu/dondi/share/intro/wheres-my-stuff.pdf Where's my Stuff?]

[[Introduction to the Command Line]]
| An overview of computers, networks, files, and databases
* Useful help resource: http://explainshell.com
* Manipulating DNA sequences on the command line
* [[Using the XMLPipeDB Match Utility]] 
| rowspan="2" | [[Week 3]]

(due at midnight 9/22)

[[Class Journal Week 3]]
|-
!
| Thursday<br/>9/17/2015
|
| Command line lab session
* Hands-on practice
* Discuss [[Week 3]] assignment tasks

{{Course Schedule Divider}}
|-
! 4
| Tuesday<br/>9/22/2015
| [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&mode=reset Moody (2004) Chapter 6 (on MyLMU Connect)]

[http://www.nature.com/news/2010/100623/pdf/4651000a.pdf Science After the Sequence]

[http://www.nature.com/news/2010/100623/full/4651000a.html Nature Special: Human Genome at 10]

[http://nar.oxfordjournals.org/content/43/D1/D1.full Introduction to ''NAR'' Database Issue]

[http://scienceblogs.com/digitalbio/2015/01/30/bio-databases-2015/ Commentary on ''NAR'' Database Issue]

| Genome sequencing and annotation

Introduction to biological databases
* [https://mylmuconnect.blackboard.com/bbcswebdav/pid-1146580-dt-content-rid-1944592_1/courses/M_46692.201530/20150922_HGP_IntroBiolDB.pdf Slides shown in class (MyLMU Connect)]
* [http://www.dataone.org/sites/all/documents/L01_DataManagement.pptx DataONE: Why Data Management]
* [http://www.dataone.org/sites/all/documents/L02_DataSharing.pptx DataONE: Data Sharing]
| rowspan="2" | [[Week 4]]

(due at midnight 9/29)

[[Class Journal Week 4]]
|-
!
| Thursday<br/>9/24/2015
| [[More Text Processing Features]]

[[How to Read XML Files]]
| Command line lab session
* Review assignment issues
* Discuss [[Week 4]] computer portion


{{Course Schedule Divider}}
|-
! 5
| Tuesday<br/>9/29/2015
| [http://myweb.lmu.edu/dondi/share/db/relational1.pdf The Relational Data Model: Structure]

[http://myweb.lmu.edu/dondi/share/db/sql.pdf An Overview of SQL]

| Introduction to relational databases
* Case study from one of the NAR databases: http://www.rhesusbase.org 
* Relating XML structure to relational structure

Test your understanding: http://xkcd.com/327/
| rowspan="2" | [[Week 5]]

(due at midnight 10/6)

[[Class Journal Week 5]]
|-
!
| Thursday<br/>10/1/2015
| Additional background and details can be found in Chapters 1, 2, and 6 of ''A First Course in Database Systems'' (on reserve at the library).
| SQL and working with relational databases continued
* [[PostgreSQL Tutorial]]

{{Course Schedule Divider}}
|-
! style="background-color: #ddf" | 6
| style="background-color: #eef; border: none" | Tuesday<br/>10/6/2015
| style="background-color: #eef; border: none" |
| style="background-color: #eef; border: none" | President Synder’s Inauguration Day
| rowspan="2" | [[Week 6]]

(due at midnight 10/13)

[[Class Journal Week 6]]
|-
!
| Thursday<br/>10/8/2015
|
| Database presentations
# Nicole, Erich, Kristen, Josh
# Mary, Emily, Brandon L., Kevin
# Lena, Jake, Anu, Veronica
# Ron, Mahrad, Brandon K., Trixie

[[Presentation Rubric]]

[[Media:PresentationGuidelines.ppt | PowerPoint Guidelines]]

|}

=== Part 2: Going Deeper (Gene Expression Data and Relational Databases) ===

{| class="schedule-table"
! #
! Date
! Reading
! Agenda
! Journal
|-
! 7
| Tuesday<br/>10/13/2015
| [http://www.ncbi.nlm.nih.gov/books/NBK26818/#A1633 Alberts ''et al''. (2002) ''Molecular Biology of the Cell'', Ch. 8: Microarrays]

[http://www.nature.com/ng/journal/v21/n1s/full/ng0199supp_33.html Brown & Botstein (1999) "Microarrays" ''Nature Genetics'']

Campbell & Heyer Chapter 4 (on [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1 MyLMU Connect])

DeRisi et al. (1997) [http://www.sciencemag.org/content/278/5338/680.full ''Science'' 278: 680-686.]

| Introduction to DNA microarray data, Gene Ontology, and GenMAPP/MAPPFinder
* [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&content_id=_1103141_1 Slides on MyLMU Connect]
* [http://www.dataone.org/sites/all/documents/L04_DataEntryManipulation.pptx DataONE: Data Entry and Manipulation]
* [http://www.bio.davidson.edu/courses/genomics/chip/chip.html Microarray animation]

| rowspan="2" | [[Week 7]]

(due at midnight 10/20)

[[Class Journal Week 7]]
|-
!
| Thursday<br/>10/15/2015
| [http://www.nature.com/nature/journal/v417/n6889/full/nature00778.html Merrell et al. (2002) "''Vibrio cholerae'' microarray data" ''Nature'']
| [http://www.openwetware.org/wiki/BIOL398-01/S10:Sample_Microarray_Analysis_Vibrio_cholerae DNA Microarray Analysis activity part 1]

Test your understanding: [http://xkcd.com/882/ http://xkcd.com/882/]

{{Course Schedule Divider}}
|-
! 8
| Tuesday<br/>10/20/2015
| [http://www.nature.com/ng/journal/v25/n1/full/ng0500_25.html Ashburner et al. (2000) "Gene Ontology" ''Nature Genetics'']

[http://www.genomebiology.com/content/4/1/R7 Doniger et al. (2003) "MAPPFinder" ''Genome Biology'']
| [http://www.openwetware.org/wiki/BIOL367/F10:GenMAPP_and_MAPPFinder_Protocols DNA Microarray Analysis activity part 2]
| rowspan="2" | [[Week 8]]

(due at midnight 10/27)

[[Class Journal Week 8]]
|-
!
| Thursday<br/>10/22/2015
|
| Work session to complete DNA Microarray Analysis [http://www.openwetware.org/wiki/BIOL398-01/S10:Sample_Microarray_Analysis_Vibrio_cholerae part 1] and [http://www.openwetware.org/wiki/BIOL367/F10:GenMAPP_and_MAPPFinder_Protocols part 2]

{{Course Schedule Divider}}
|-
! 9
| Tuesday<br/>10/27/2015
|
|
* Introduction to XMLPipeDB: export a Gene Database for ''Vibrio cholerae''
** [[Running GenMAPP Builder]]
* [http://www.opensource.org/ Open Source] review
* Group requests due midnight Saturday, October 31
| rowspan="2" | [[Week 9]]

(due at midnight 11/3)

[[Class Journal Week 9]]
|-
!
| Thursday<br/>10/29/2015
| [http://nar.oxfordjournals.org/content/40/D1/D565.full Dimmer et al. (2012) "UniProt-GOA Database" ''Nucleic Acids Research'']

[http://www.nature.com/nature/journal/v406/n6795/full/406477A0.html Heidelberg et al. (2000) "''Vibrio cholerae'' Genome Sequence" ''Nature'']
|
Integrity analysis of Vibrio Gene Database
* [[How Do I Count Thee? Let Me Count The Ways]]
* [[Using Microsoft Excel to Compare ID Lists]]
* [[Gene Database Testing Report Sample | Complete your Gene Database Testing Report for ''Vibrio cholerae'']]
|}

=== Part 3: Integrating for Research (Gene Database Project) ===

{| class="schedule-table"
! #
! Date
! Reading
! Agenda
! Journal
|-
! 10
| Tuesday<br/>11/3/2015
|
|
* Annotated bibliography of genomics papers for your species
* ''Course LibGuide:'' http://libguides.lmu.edu/BIOL367
** Select genome paper for journal club
** Select microarray paper
| rowspan="2" | [[Week 10]]

(due at midnight 11/10)


|-
!
| Thursday<br/>11/5/2015
|
| Begin [[Gene_Database_Project | group projects]]

{{Course Schedule Divider}}
|-
! 11
| Tuesday<br/>11/10/2015
|
| Guild meetings and work session
* Export of Gene Database for your group's species
* Document ID types for your species
* Begin DNA microarray analysis
| rowspan="2" | [[Week 11]]

(due at midnight 11/17)
|-
!
| Thursday<br/>11/12/2015
|
|

* Status report
* Work session

{{Course Schedule Divider}}
|-
! 12
| Tuesday<br/>11/17/2015
|
| First Set of Journal Club Presentations
# TBD
# TBD
# TBD
# TBD

[[Presentation Rubric]]

[[Media:PresentationGuidelines.ppt | PowerPoint Guidelines]]
| rowspan="2" | [[Week 12]]

(due at midnight 11/24)

|-
!
| Thursday<br/>11/19/2015
|
|
* Status report
* Work session

{{Course Schedule Divider}}
|-
! 13
| Tuesday<br/>11/24/2015
|
| Second Set of Journal Club Presentations
# TBD
# TBD
# TBD
# TBD

[[Presentation Rubric]]

[[Media:PresentationGuidelines.ppt | PowerPoint Guidelines]]
| '''''No Week 13 assignment'''''

|-
! style="background: #ded" |
| style="background: #efe; border: none" | Thursday<br/>11/26/2015
| style="background: #efe; border: none" |
| style="background: #efe; border: none" | Thanksgiving—no class
| style="background: #efe; border: none" |
{{Course Schedule Divider}}
|-
! 14
| Tuesday<br/>12/1/2015
|
|
* Status report
* Work session
| rowspan="2" | [[Week 14]]

(due at midnight 12/8)
|-
!
| Thursday<br/>12/3/2015
|
|
* Status report
* Work session
{{Course Schedule Divider}}
|-
! 15
| Tuesday<br/>12/8/2015
|
|
* Status report
* Work session
| rowspan="2" | [[Week 15]]

(due at midnight 12/15)
|-
!
| Thursday<br/>12/10/2015
|
|
* Status report
* Work session
{{Course Schedule Divider}}
|-
! style="background: #edd" | F
| style="background: #fee; border: none" | Tuesday<br/>12/15/2015
| style="background: #fee; border: none" |
| style="background: #fee; border: none" | '''Final project presentations 2:00-4:00 PM'''
| style="background: #fee; border: none" |
|-
!
| Friday<br/>12/18/2015 4:30 PM
|
| '''Project deliverables due 4:30 PM'''
|
|}

== Course Information ==

=== Instructors ===

==== [[User:Kdahlquist | Kam D. Dahlquist, Ph.D.]] ====
: http://myweb.lmu.edu/kdahlqui
: '''Phone:''' (310) 338-7697
: '''E-mail:''' kdahlquist at lmu dot edu
: '''Office:''' Life Sciences Building 289
: '''Office Hours:''' Tuesdays 9:00-11:00 AM, Wednesdays 1:30-3:30 PM and by appointment. I keep a sign-up sheet on my office door to facilitate appointment-making.

==== [[User:Dondi|John David N. Dionisio, Ph.D.]] ====
: http://myweb.lmu.edu/dondi
: '''Phone:''' (310) 338-5782
: '''E-mail:''' dondi at lmu dot edu
: '''Office:''' Doolan 106
: '''Office Hours:''' {{Dondi Office Hours}}

=== Prerequisites/Recommended Background ===

To take this course, you must have upper division standing in the Seaver College of Science and Engineering. Otherwise, there are no strict prerequisites. Backgrounds in biology and computer science, as well as prior experience with database or information management applications, may be helpful but not necessary.

=== Class Meetings and Attendance ===
TR 2:40–3:55pm, Seaver 120 

This is a hands-on, participatory course, thus attendance at all class meetings is required. Each student is allowed two “sick” days (automatically excused absences) during the semester. Further unexcused absences from class will result in a 5% deduction from the overall course grade for each absence. Every effort should be made to attend class on oral presentation days as the content of that day's class is dependent on student participation. Unexcused absences from an oral presentation will result in a grade of zero for the presentation. The instructors should be notified as soon as possible, electronically or by phone, of the reasons for all absences.

Note that the last day to add or drop a class without a grade of W is September 4. The withdrawal or credit/no-credit status deadline is November 6.

=== Mutual Responsibilities ===

This course is designed to foster your development as a biologist and computer scientist and to give you an authentic research experience. We will be engaged together in discovering, examining, and practicing the personal qualities, technical skills, and community standards of the scientific community. While you are ultimately responsible for your own learning, you are not alone. Our class constitutes a team where we will be learning from each other. The role of the instructors is to provide the expert coaching to support and assist you on your journey. All of the exercises, readings, assignments, and policies detailed below have been designed with this purpose in mind.

=== Classroom Conduct ===

We are all responsible for maintaining a classroom and laboratory environment that is safe and conducive to learning. As such, we will observe the following:

# As an LMU Lion, by the [http://studentaffairs.lmu.edu/media/lmustudentaffairs/administration/judicialaffairs/Lion%27s%20Code.pdf Lion’s code], you are pledged to join the discourse of the academy with honesty of voice and integrity of scholarship and to show respect for staff, professors, and other students.
# You are responsible for your own learning and for being a good class citizen.
# Class will start promptly on time.
# You are expected to come to class having done the assigned reading and preparatory work so that you are ready to participate in discussions and to perform the laboratory exercises.
# You are expected to bring the required materials to each class session.
# Cell phones, pagers, and other communication or music devices must be turned off and put away out of sight. Your own laptop and/or tablet may be used to conduct the class exercises; however, if they are being used for other purposes and become distracting to you or others, you will be asked to put them away.
# All students are governed by [http://studentaffairs.lmu.edu/media/lmustudentaffairs/administration/judicialaffairs/Loyola-Marymount-University-Community-Standards-2015-2016.pdf LMU Community Standards Publication]. Disruptive behavior which is persistent or significantly interferes with classroom activities may be subject to disciplinary action. A student may be referred to the Office of Student Judicial Affairs if his or her behavior constitutes a violation of the conduct code.

=== Course Website ===

This is the course web site and wiki, hosted by http://xmlpipedb.cs.lmu.edu/biodb/fall2015/. You will need to [https://xmlpipedb.cs.lmu.edu/biodb/fall2015/index.php?title=Special:UserLogin&returnto=Main+Page&type=signup register] to be able to edit the wiki and complete coursework. Updates to the course schedule and electronic copies of all handouts, assignments, and readings will be posted to this site. You will also use the site to keep an electronic lab notebook/journal for the course. In addition, students have been automatically enrolled in [https://mylmuconnect.blackboard.com/webapps/blackboard/content/listContentEditable.jsp?content_id=_1103141_1&course_id=_62090_1&mode=reset BIOL/CMSI 367-01 on MyLMUConnect]. The two cross-listed sections are using the same site, which is listed as "Biological Databases Fall 2015" with no department or course number. The MyLMUConnect site will be used for materials that cannot be made public on this wiki, including grades.

=== E-mail Communication ===

At times we will communicate with the entire class using campus e-mail systems, so it is essential that you regularly check your lion.lmu.edu e-mail address or forward your lion account e-mail to your preferred e-mail address. Messages sent to the instructors at night or on the weekend will be answered the next school day. Please CC both instructors on all e-mail messages related to this class.

=== Required Materials ===

==== Texts ====

There is no required text to purchase for the course; materials will be put on reserve at Hannon Library or will be available online on this wiki or the MyLMUConnect site. Specific reading assignments are given on the course schedule and should be completed before coming to class.
* This text has been placed on reserve at the library: Jeffrey Ullman and Jennifer Widom, ''A First Course in Database Systems'', Third Edition. Prentice Hall, 2008.
* Assorted handouts, articles, and sample code will be distributed throughout the semester.
* Additional information is also available on the web; do not hesitate to look for further sources of information regarding the concepts, techniques, tools, and paradigms that we will discuss.

==== Materials (must be brought to each class meeting) ====

* 3-ring binder with all course handouts
* Pen, pencil, extra paper
* USB flash drive to store data
* [https://xmlpipedb.cs.lmu.edu/biodb/fall2015/index.php/Special:UserLogin/signup Account for this wiki]
* [https://secure.cs.lmu.edu/ Computer Science account]
* [http://www.github.com GitHub account]

=== Course Description ===

The disciplines of biology and computer science come together in bioinformatics, where computational tools are needed to manage and analyze the flood of data coming from new genomics technologies. Biological databases form a significant part of this young and exciting field. This course introduces students to both the biology and computer science expertise needed to understand, use, and develop biological databases. Biology topics include the fundamentals of genetics, molecular biology, and biochemistry needed to understand the data stored in biological databases, as well as the biotechnologies used to gather these data in a high-throughput manner. Computer science topics include what biological databases are, why they are important (and needed), and the challenges that arise in compiling them effectively. Biology and computer science lectures on topics that are relevant to biological databases are coupled with hands-on experience with a variety of software packages ranging from bioinformatics utilities to general-purpose database and software development tools. After learning how to use a biological database, students will be asked to build one of their own.

==== Course Objectives and Learning Outcomes ====

This course is built upon L. Dee Fink’s ''taxonomy of significant learning'', as applied to biological databases. Long after the course concludes, our hope is that:
* You understand how biological information is encoded in the genome and can apply this knowledge to a variety of biological tasks and problems
* You understand the core concepts, structure, and functions of a database, ranging from individual files to a full relational database management system, and can perform useful tasks with such data
* You show discipline and proficiency in day-to-day science and engineering best practices, such as maintaining journals and notebooks, managing your files and code, and critically evaluating scientific and technical information
* You recognize and care about how the biological and technological issues presented in this course relate to and affect society, our daily lives, and ourselves
* You have some skills and tools for “leaving your comfort zone,” flourishing outside of it, and learning more about biology and computer science on your own
* You learn how to communicate and work effectively with colleagues from different disciplines

==== University Core Curriculum ====

This course fulfills the following requirements in the University Core Curriculum:
* Integrations: Interdisciplinary Connections
* Upper Division Information Literacy Flag
* Upper Division Oral Communication Flag

=== Course Work and Grading ===

Letter grades are determined as follows: ≥ 90% gets an A– or better; ≥ 80% gets a B– or better; ≥ 70% gets a C– or better. The instructors may curve grades upward based on qualitative considerations such as degree of difficulty, effort, class participation, time constraints, and overall attitude throughout the course. Grades are never curved downward. Current grades will be made available at MyLMU Connect (the system formerly known as Blackboard).

Your work in this course will be assessed in five areas:

{| style="border: 1px dashed rgb(47, 111, 171); background: rgb(249, 249, 249); margin: 1em 0; padding: 1em" cellpadding="5ex"
| Weekly electronic lab notebook/journal assignments, individual (10 points each)
| style="text-align: right" | 140
| points
|-
| Weekly electronic lab notebook/journal assignments, shared (3 points each)
| style="text-align: right" | 42
| points
|-
| Oral presentations
| style="text-align: right" | 105
| points 
|-
| Information literacy (additional points added to two journal assignments)
| style="text-align: right" | 24
| points 
|-
| Final Project Deliverables (including written report)
| style="text-align: right" | 175
| points
|-
| '''Total'''
| style="text-align: right" | '''486'''
| '''points'''
|}

==== Electronic Laboratory Notebook/Journal ====

One of the most important skills you can develop as a scientist is keeping an excellent laboratory notebook. For computational research, the equivalent of the biology paper-based lab notebook is documentation of your “workflow”. For this course you will practice documentation skills by keeping an electronic lab notebook or journal. The technology we will use is this wiki, that we will create and edit during the semester. You will create an individual user page and make weekly entries that the instructors will read and grade. You will use the wiki to complete the assignments as well. The following guidelines apply:
* Your weekly journal entry is typically due every midnight on Tuesday PST (''Monday night/Tuesday morning''); consult the schedule for specific due dates for each assignment.
* Each weekly assignment has an individual component and a shared component. You will earn 10 points per weekly submission for the individual journal entry and 3 points per submission for the shared journal entry. '''''Late journal entries will be accepted up to one week later for up to half credit.'''''
* The instructors will read and comment on how to improve your journal entries on your user talk pages.
* Depending on the type of assignment for that week, you may be given the opportunity to make improvements to previous journal entries as the semester progresses.
* Generally, your journal entries will consist of:
** Your electronic laboratory notebook and other documentation for hands-on exercises and projects
** Answers to any specific questions posed in the exercise
** Reflection on your learning
* For most weeks in the semester, you will be assigned a "homework partner" from the complementary biology or computer science discipline. You will be expected to consult with your partner, sharing your domain expertise, in order to complete the assignment. However, unless otherwise stated, each partner must submit his or her own work as the individual journal entry (direct copies of each other's work is not allowed).

==== Oral Presentations ====

You will give three oral presentations in this course. The first two will be in the format of a “Journal Club” presentation where students will present and lead discussion of research articles from the primary literature. The third will be a research presentation on your final project. '''''Because that day’s class content is dependent upon each student being ready to present and lead discussion, late journal club presentations will not be accepted. An unexcused absence from a journal club presentation will result in a grade of zero for the presentation.'''''

==== Final Group Project ====

* Overall Description of the [[Gene Database Project]], including entire list of deliverables
* [[Gene Database Project Report Guidelines]]

{| style="width: 50%"
| '''Teams'''
| '''Guilds'''
|-
| [[Team1]]
| [[Project Manager]]
|-
| [[Team2]]
| [[GenMAPP User]]
|-
| [[Team3]]
| [[Quality Assurance]]
|-
| [[Team4]]
| [[Coder]]
|}

In addition to the research presentation, the culmination of your final project will be the preparation of a written laboratory report in the style of a manuscript that could be submitted to a peer-reviewed journal. Specific instructions are posted [[Gene Database Project Report Guidelines | here]]. '''''The Final Written Report cannot be accepted any later than Friday, December 18 at 4:30 PM. The Final Written Report must be completed to receive a passing grade in the course.'''''

==== Extra Credit ====

Students may accumulate up to 12.5 points toward their final grade in extra credit by attending Department seminars in Biology or Electrical Engineering & Computer Science and completing the seminar sheets. Each seminar attended is worth 2.5 points with up to 5 seminars (12.5 points) total. You must attend the entire seminar from start to finish and personally turn in your seminar sheet to a faculty member at the end of the seminar.

Certain, non-Biology/Computer Science Department seminars may be approved in advance for extra credit at the instructors’ discretion. To receive credit for these seminars, you must turn in a one-page '''''hard copy''''' of your summary of the seminar '''''in class, within one week''''' of the date of the seminar or they will not count as extra credit.

=== Work Load Expectations ===

In line with [http://www.lmu.edu/Assets/LMU+Credit+Hour+Policy_Final.pdf LMU’s Credit Hour Policy], the work load expectation for this course is that for every one hour (50 minutes) of classroom instruction, you will complete a minimum of two hours of out-of-class student work each week. This is a 3-unit course with 3 hours (150 minutes) of instruction per week. Thus the expectation is that you will complete 6 hours of work outside of class per week.

=== University Policy on Academic Honesty ===

Loyola Marymount University expects high standards of honesty and integrity from all members of its community. All students are expected to follow the LMU honor code. As stated in the LMU Undergraduate Bulletin, “Examples of academic dishonesty include, but are not limited to, the following: all acts of cheating on assignments or examinations, or facilitating other students’ cheating; plagiarism; fabrication of data, including the use of false citations; improper use of non-print media; unauthorized access to computer accounts or files or other privileged information and improper use of Internet sites and resources.” Attempting to gain extra credit for a seminar that you did not attend or assisting another student to do this are examples of academic dishonesty and will be considered a violation of the Honor Code. [http://bulletin.lmu.edu/content.php?catoid=6&navoid=250#LMU_Honor_Code_and_Process Click here for an online version of the LMU Honor Code and Process].

You are required to sign the Honor Code Agreement for this course.

==== Academic Honesty Resources ====
* [[Media:BIOL367_Fall2015_HonorCodeAgreement.pdf|Honor Code Agreement]]
* [http://bulletin.lmu.edu/content.php?catoid=6&navoid=250#LMU_Honor_Code_and_Process LMU Honor Code and Process].
* [http://www.bio.davidson.edu/dept/plagiarism.html The Davidson College Department of Biology Statement on Plagiarism]
* [[Media:BIOL367_Fall2015_GuidelinesforLiteratureCitations.pdf | Guidelines for Literature Citations in a Scientific Paper]]
** [http://libguides.lmu.edu/content.php?pid=472661&sid=3874648 APA Style for References]
* [[Media:Avoiding_plagiarism.pptx | Academic Resource Center presentation on plagiarism]] (PowerPoint presentation)
* [http://myweb.lmu.edu/dondi/share/misc/plagiarism.pdf Notes on plagiarism from an electrical engineering & computer science perspective]

=== Americans with Disabilities Act—Special Accommodations ===

Students with special needs who require reasonable modifications, special assistance, or accommodations in this course should promptly direct their request to the Disability Support Services (DSS) Office. Any student who currently has a documented disability (ADHD, Autism Spectrum Disorder, Learning, Physical, or Psychiatric) needing academic accommodations should contact the DSS Office (Daum Hall 2nd floor, 310-338-4216) as early in the semester as possible. All discussions will remain confidential. Please visit [http://www.lmu.edu/dss www.lmu.edu/dss] for additional information. In addition, please schedule an appointment with the instructors early in the semester to discuss any accommodations for this course for which you have been approved.

=== Revision Notice ===

If necessary, this syllabus and its contents are subject to revision; students are responsible for any changes or modifications announced in class. The most current version of this information resides on this page, the course web site at http://xmlpipedb.cs.lmu.edu/biodb/fall2015/.

Template:Build Instructions

2015-11-02T02:19:30Z

Dondi: Templatize the build instructions because they appear in multiple places.

# Open the ''gmbuilder'' project by clicking on the gray triangle to the left of its name.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right-click on ''build.xml'' and choose '''Run As > Ant Build...''' (the one with the ellipses) from the menu that appears.
# In the ''Edit Configuration'' dialog that appears, uncheck ''dist'' if it is already checked.
# Check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit. You will see some messages scroll up on the ''Console'' tab if that tab is visible.
# When it is done, right-click on the ''gmbuilder'' project folder and choose '''Refresh''' from the menu that appears (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.

Coder

2015-11-02T02:18:31Z

Dondi: /* GenMAPP Builder Project Setup and Initial Build */ Write out build instructions.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] (which, at this writing, is ''JDK 8u65'')
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used:
#** Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Follow the instructions in the [[#GenMAPP Builder Project Setup and Initial Build|GenMAPP Builder Project Setup and Initial Build]] section of this wiki page in order to:
#* Set up a functioning Eclipse development environment for your branch of GenMAPP Builder.
#* Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder. You can only run ''GenMAPP'' and ''MAPPFinder'' on Windows, but you can build and run ''GenMAPP Builder'' on any platform that supports PostgreSQL and JDK 8.

Although there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Prerequisites ===

# Make sure that you have already accomplished the [[#Milestone 1: Version Control Setup|version control setup milestone (Milestone 1)]].
# Make sure that you have already downloaded and installed the software mentioned in [[#Milestone 2: “Developer Rig” Setup and Initial As-Is Build|Milestone 2]] (first item).

=== GitHub Repository Clone Setup ===

# Determine the desired location (on your development computer) for your local copy of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub repository].
# <code>cd</code> to this location.
# Clone the repository:<pre>git clone https://github.com/lmu-bioinformatics/xmlpipedb.git</pre>
# <code>cd</code> into the clone folder <code>xmlpipedb</code>:<pre>cd xmlpipedb</pre>
# Switch to your branch:<pre>git checkout <your-branch-name></pre>
# You should see a message like this: <pre>Branch <your-branch-name> set up to track remote branch <your-branch-name> from origin. Switched to a new branch '<your-branch-name>'</pre>
#: …this is OK and expected.

=== Eclipse Workspace Setup ===

[[Image:Initial-eclipse-workspace.png|thumb]]
# Run Eclipse.
# When Eclipse asks you to select a workspace, click the ''Browse…'' and navigate to your repository clone folder. Select it and click on ''Open''.
# Verify that your repository clone folder is what is listed as the ''Workspace:''. Click on ''OK''.
# You will see an introductory display with assorted menu items. Click the ''Workbench'' button (upper-right corner, with a 3D arrow for an icon).
# If everything went well, you should now see an empty developer area, showing your repository clone folder in the window title with a ''Project Explorer'' tab on the left (see screenshot for an example).

=== Java Project Setup ===

# Right-click anywhere within the empty ''Project Explorer'' tab and choose '''New > Project…''' from the menu that appears.
# Choose ''Java Project'' from the list of “wizards” and click on the ''Next >'' button.
# On the next panel, enter <code>gmbuilder</code> as the ''Project name:''. Note how the ''Location:'' field underneath, although disabled, should show your cloned repository folder location with ''/gmbuilder'' appended to it.
# Verify that the ''JRE'' section is showing Java 8 or 1.8. If not, talk to Dr. Dionisio.
# Click on the ''Finish'' button (yes, there is no need for further configuration).
# If Eclipse asks you whether you want to open the “Java perspective,” respond with ''Yes''. You may check the ''Remember my decision'' checkbox if you wish.
# You should now see a ''gmbuilder'' project folder in the ''Project Explorer'' tab.
# If the ''gmbuilder'' project folder icon shows a red ''x'' icon, do the following:
#* Right-click on the ''gmbuilder'' item and choose '''Properties''' from the menu that appears.
#* Click on the ''Java Compiler'' item. ''Click on the item name itself, not its triangle.''
#* You should see that ''Compiler compliance level:'' is set to ''1.6''. If so, uncheck the ''Enable project specific settings'' checkbox.
#* Click the ''Apply'' button. If Eclipse asks you whether it is OK to rebuild the project, click ''Yes''.
#* Click on ''OK'' to dismiss the ''Properties'' dialog.
#* See if the red ''x'' icon has disappeared.
# If the red ''x'' icon persists, please show your setup to fellow guild members or Dr. Dionisio for troubleshooting.

=== Initial Build ===

# Open the ''gmbuilder'' project by clicking on the gray triangle to the left of its name.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right-click on ''build.xml'' and choose '''Run As > Ant Build...''' (the one with the ellipses) from the menu that appears.
# In the ''Edit Configuration'' dialog that appears, uncheck ''dist'' if it is already checked.
# Check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit. You will see some messages scroll up on the ''Console'' tab if that tab is visible.
# When it is done, right-click on the ''gmbuilder'' project folder and choose '''Refresh''' from the menu that appears (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T02:11:56Z

Dondi: /* Java Project Setup */ Update the project setup process.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] (which, at this writing, is ''JDK 8u65'')
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used:
#** Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Follow the instructions in the [[#GenMAPP Builder Project Setup and Initial Build|GenMAPP Builder Project Setup and Initial Build]] section of this wiki page in order to:
#* Set up a functioning Eclipse development environment for your branch of GenMAPP Builder.
#* Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder. You can only run ''GenMAPP'' and ''MAPPFinder'' on Windows, but you can build and run ''GenMAPP Builder'' on any platform that supports PostgreSQL and JDK 8.

Although there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Prerequisites ===

# Make sure that you have already accomplished the [[#Milestone 1: Version Control Setup|version control setup milestone (Milestone 1)]].
# Make sure that you have already downloaded and installed the software mentioned in [[#Milestone 2: “Developer Rig” Setup and Initial As-Is Build|Milestone 2]] (first item).

=== GitHub Repository Clone Setup ===

# Determine the desired location (on your development computer) for your local copy of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub repository].
# <code>cd</code> to this location.
# Clone the repository:<pre>git clone https://github.com/lmu-bioinformatics/xmlpipedb.git</pre>
# <code>cd</code> into the clone folder <code>xmlpipedb</code>:<pre>cd xmlpipedb</pre>
# Switch to your branch:<pre>git checkout <your-branch-name></pre>
# You should see a message like this: <pre>Branch <your-branch-name> set up to track remote branch <your-branch-name> from origin. Switched to a new branch '<your-branch-name>'</pre>
#: …this is OK and expected.

=== Eclipse Workspace Setup ===

[[Image:Initial-eclipse-workspace.png|thumb]]
# Run Eclipse.
# When Eclipse asks you to select a workspace, click the ''Browse…'' and navigate to your repository clone folder. Select it and click on ''Open''.
# Verify that your repository clone folder is what is listed as the ''Workspace:''. Click on ''OK''.
# You will see an introductory display with assorted menu items. Click the ''Workbench'' button (upper-right corner, with a 3D arrow for an icon).
# If everything went well, you should now see an empty developer area, showing your repository clone folder in the window title with a ''Project Explorer'' tab on the left (see screenshot for an example).

=== Java Project Setup ===

# Right-click anywhere within the empty ''Project Explorer'' tab and choose '''New > Project…''' from the menu that appears.
# Choose ''Java Project'' from the list of “wizards” and click on the ''Next >'' button.
# On the next panel, enter <code>gmbuilder</code> as the ''Project name:''. Note how the ''Location:'' field underneath, although disabled, should show your cloned repository folder location with ''/gmbuilder'' appended to it.
# Verify that the ''JRE'' section is showing Java 8 or 1.8. If not, talk to Dr. Dionisio.
# Click on the ''Finish'' button (yes, there is no need for further configuration).
# If Eclipse asks you whether you want to open the “Java perspective,” respond with ''Yes''. You may check the ''Remember my decision'' checkbox if you wish.
# You should now see a ''gmbuilder'' project folder in the ''Project Explorer'' tab.
# If the ''gmbuilder'' project folder icon shows a red ''x'' icon, do the following:
#* Right-click on the ''gmbuilder'' item and choose '''Properties''' from the menu that appears.
#* Click on the ''Java Compiler'' item. ''Click on the item name itself, not its triangle.''
#* You should see that ''Compiler compliance level:'' is set to ''1.6''. If so, uncheck the ''Enable project specific settings'' checkbox.
#* Click the ''Apply'' button. If Eclipse asks you whether it is OK to rebuild the project, click ''Yes''.
#* Click on ''OK'' to dismiss the ''Properties'' dialog.
#* See if the red ''x'' icon has disappeared.
# If the red ''x'' icon persists, please show your setup to fellow guild members or Dr. Dionisio for troubleshooting.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T01:57:42Z

Dondi: /* Eclipse Workspace Setup */ Spell out Eclipse workspace setup.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] (which, at this writing, is ''JDK 8u65'')
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used:
#** Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Follow the instructions in the [[#GenMAPP Builder Project Setup and Initial Build|GenMAPP Builder Project Setup and Initial Build]] section of this wiki page in order to:
#* Set up a functioning Eclipse development environment for your branch of GenMAPP Builder.
#* Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder. You can only run ''GenMAPP'' and ''MAPPFinder'' on Windows, but you can build and run ''GenMAPP Builder'' on any platform that supports PostgreSQL and JDK 8.

Although there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Prerequisites ===

# Make sure that you have already accomplished the [[#Milestone 1: Version Control Setup|version control setup milestone (Milestone 1)]].
# Make sure that you have already downloaded and installed the software mentioned in [[#Milestone 2: “Developer Rig” Setup and Initial As-Is Build|Milestone 2]] (first item).

=== GitHub Repository Clone Setup ===

# Determine the desired location (on your development computer) for your local copy of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub repository].
# <code>cd</code> to this location.
# Clone the repository:<pre>git clone https://github.com/lmu-bioinformatics/xmlpipedb.git</pre>
# <code>cd</code> into the clone folder <code>xmlpipedb</code>:<pre>cd xmlpipedb</pre>
# Switch to your branch:<pre>git checkout <your-branch-name></pre>
# You should see a message like this: <pre>Branch <your-branch-name> set up to track remote branch <your-branch-name> from origin. Switched to a new branch '<your-branch-name>'</pre>
#: …this is OK and expected.

=== Eclipse Workspace Setup ===

[[Image:Initial-eclipse-workspace.png|thumb]]
# Run Eclipse.
# When Eclipse asks you to select a workspace, click the ''Browse…'' and navigate to your repository clone folder. Select it and click on ''Open''.
# Verify that your repository clone folder is what is listed as the ''Workspace:''. Click on ''OK''.
# You will see an introductory display with assorted menu items. Click the ''Workbench'' button (upper-right corner, with a 3D arrow for an icon).
# If everything went well, you should now see an empty developer area, showing your repository clone folder in the window title with a ''Project Explorer'' tab on the left (see screenshot for an example).

=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

File:Initial-eclipse-workspace.png

2015-11-02T01:56:36Z

Dondi: An empty Eclipse workspace.

An empty Eclipse workspace.

Coder

2015-11-02T01:49:58Z

Dondi: /* GitHub Repository Clone Setup */ Spell out clone and branch setup.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] (which, at this writing, is ''JDK 8u65'')
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used:
#** Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Follow the instructions in the [[#GenMAPP Builder Project Setup and Initial Build|GenMAPP Builder Project Setup and Initial Build]] section of this wiki page in order to:
#* Set up a functioning Eclipse development environment for your branch of GenMAPP Builder.
#* Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder. You can only run ''GenMAPP'' and ''MAPPFinder'' on Windows, but you can build and run ''GenMAPP Builder'' on any platform that supports PostgreSQL and JDK 8.

Although there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Prerequisites ===

# Make sure that you have already accomplished the [[#Milestone 1: Version Control Setup|version control setup milestone (Milestone 1)]].
# Make sure that you have already downloaded and installed the software mentioned in [[#Milestone 2: “Developer Rig” Setup and Initial As-Is Build|Milestone 2]] (first item).

=== GitHub Repository Clone Setup ===

# Determine the desired location (on your development computer) for your local copy of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub repository].
# <code>cd</code> to this location.
# Clone the repository:<pre>git clone https://github.com/lmu-bioinformatics/xmlpipedb.git</pre>
# <code>cd</code> into the clone folder <code>xmlpipedb</code>:<pre>cd xmlpipedb</pre>
# Switch to your branch:<pre>git checkout <your-branch-name></pre>
# You should see a message like this: <pre>Branch <your-branch-name> set up to track remote branch <your-branch-name> from origin. Switched to a new branch '<your-branch-name>'</pre>
#: …this is OK and expected.

=== Eclipse Workspace Setup ===

# Run Eclipse.
# Establish an Eclipse workspace for the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB repository].


=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T01:43:11Z

Dondi: /* Milestone 2: “Developer Rig” Setup and Initial As-Is Build */ Revert to un-numbered bullets.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] (which, at this writing, is ''JDK 8u65'')
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used:
#** Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Follow the instructions in the [[#GenMAPP Builder Project Setup and Initial Build|GenMAPP Builder Project Setup and Initial Build]] section of this wiki page in order to:
#* Set up a functioning Eclipse development environment for your branch of GenMAPP Builder.
#* Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder. You can only run ''GenMAPP'' and ''MAPPFinder'' on Windows, but you can build and run ''GenMAPP Builder'' on any platform that supports PostgreSQL and JDK 8.

Although there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Prerequisites ===

# Make sure that you have already accomplished the [[#Milestone 1: Version Control Setup|version control setup milestone (Milestone 1)]].
# Make sure that you have already downloaded and installed the software mentioned in [[#Milestone 2: “Developer Rig” Setup and Initial As-Is Build|Milestone 2]] (first item).

=== GitHub Repository Clone Setup ===

# Determine the desired location (on your development computer) for your local copy of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub repository].
# <code>cd</code> to this location.
# Clone the repository:
git clone https://github.com/lmu-bioinformatics/xmlpipedb.git

=== Eclipse Workspace Setup ===

# Run Eclipse.
# Establish an Eclipse workspace for the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB repository].


=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T01:42:31Z

Dondi: /* Prerequisites */ Expressly point out Milestone 1.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] (which, at this writing, is ''JDK 8u65'')
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used:
#*# Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.
#*# Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#*# If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Follow the instructions in the [[#GenMAPP Builder Project Setup and Initial Build|GenMAPP Builder Project Setup and Initial Build]] section of this wiki page in order to:
#* Set up a functioning Eclipse development environment for your branch of GenMAPP Builder.
#* Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder. You can only run ''GenMAPP'' and ''MAPPFinder'' on Windows, but you can build and run ''GenMAPP Builder'' on any platform that supports PostgreSQL and JDK 8.

Although there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Prerequisites ===

# Make sure that you have already accomplished the [[#Milestone 1: Version Control Setup|version control setup milestone (Milestone 1)]].
# Make sure that you have already downloaded and installed the software mentioned in [[#Milestone 2: “Developer Rig” Setup and Initial As-Is Build|Milestone 2]] (first item).

=== GitHub Repository Clone Setup ===

# Determine the desired location (on your development computer) for your local copy of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub repository].
# <code>cd</code> to this location.
# Clone the repository:
git clone https://github.com/lmu-bioinformatics/xmlpipedb.git

=== Eclipse Workspace Setup ===

# Run Eclipse.
# Establish an Eclipse workspace for the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB repository].


=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T01:41:18Z

Dondi: Work-in-progress: Eclipse/git setup from scratch.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] (which, at this writing, is ''JDK 8u65'')
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used:
#*# Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.
#*# Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#*# If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Follow the instructions in the [[#GenMAPP Builder Project Setup and Initial Build|GenMAPP Builder Project Setup and Initial Build]] section of this wiki page in order to:
#* Set up a functioning Eclipse development environment for your branch of GenMAPP Builder.
#* Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder. You can only run ''GenMAPP'' and ''MAPPFinder'' on Windows, but you can build and run ''GenMAPP Builder'' on any platform that supports PostgreSQL and JDK 8.

Although there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Prerequisites ===

# Make sure that you have already accomplished the [[#Milestone 1: Version Control Setup|version control setup milestone]].
# Make sure that you have already downloaded and installed the software mentioned in [[#Milestone 2: “Developer Rig” Setup and Initial As-Is Build|Milestone 2]] (first item).

=== GitHub Repository Clone Setup ===

# Determine the desired location (on your development computer) for your local copy of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub repository].
# <code>cd</code> to this location.
# Clone the repository:
git clone https://github.com/lmu-bioinformatics/xmlpipedb.git

=== Eclipse Workspace Setup ===

# Run Eclipse.
# Establish an Eclipse workspace for the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB repository].


=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T01:24:31Z

Dondi: /* Milestone 2: “Developer Rig” Setup and Initial As-Is Build */ Clarify what’s in the detailed setup section.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

''Additional technical details for accomplishing this milestone are listed in the [[#GenMAPP Builder Project Setup and Initial Build|GenMAPP Builder Project Setup and Initial Build]] section of this wiki page.''

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8]
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder.

While there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Software to Install ===

# While you can only run GenMAPP and MAPPFinder on Windows, you can build and run GenMAPP Builder on any platform: Windows, Linux, or Mac OS X. You need to download and install the Java Development Kit from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html (which, at this writing, is ''JDK 8u65'').
# Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.

=== Eclipse Project Setup ===

# Make sure that you have already accomplished the [[#Milestone 1: Version Control Setup|version control setup milestone]].
# Run Eclipse.
# Establish an Eclipse workspace for the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB repository].


=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T01:22:35Z

Dondi: /* Initial Code Checkout */ Change title and reference Milestone 1.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

''More specific and technical details for accomplishing this milestone are listed in the [[#GenMAPP Builder Project Setup and Initial Build|GenMAPP Builder Project Setup and Initial Build]] section of this wiki page.''

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] and [http://ant.apache.org ant]
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder.

While there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Software to Install ===

# While you can only run GenMAPP and MAPPFinder on Windows, you can build and run GenMAPP Builder on any platform: Windows, Linux, or Mac OS X. You need to download and install the Java Development Kit from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html (which, at this writing, is ''JDK 8u65'').
# Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.

=== Eclipse Project Setup ===

# Make sure that you have already accomplished the [[#Milestone 1: Version Control Setup|version control setup milestone]].
# Run Eclipse.
# Establish an Eclipse workspace for the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB repository].


=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T01:21:11Z

Dondi: /* Milestone 2: “Developer Rig” Setup and Initial As-Is Build */ Make hash symbol invisible.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

''More specific and technical details for accomplishing this milestone are listed in the [[#GenMAPP Builder Project Setup and Initial Build|GenMAPP Builder Project Setup and Initial Build]] section of this wiki page.''

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] and [http://ant.apache.org ant]
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder.

While there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Software to Install ===

# While you can only run GenMAPP and MAPPFinder on Windows, you can build and run GenMAPP Builder on any platform: Windows, Linux, or Mac OS X. You need to download and install the Java Development Kit from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html (which, at this writing, is ''JDK 8u65'').
# Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.

=== Initial Code Checkout ===

# Get an account from https://github.com and send your account name to Dr. Dionisio, for inclusion as an XMLPipeDB developer.
# Run Eclipse.
# Establish an Eclipse workspace for the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB repository].


=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T01:19:17Z

Dondi: /* Milestone 1: Version Control Setup */ More details.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB project on GitHub].
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
#* The easiest way to do this is via the ''Branch'' dropdown menu on the [https://github.com/lmu-bioinformatics/xmlpipedb GitHub project website for XMLPipeDB].
# ''(with QA)'' Commit and push relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch
#* You can always verify what is publicly visible on your branch by visiting the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB GitHub website], choosing your branch from the ''Branch'' dropdown menu, then inspecting the code that is visible there.

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

''More specific and technical details for accomplishing this milestone are listed in [[#GenMAPP Builder Project Setup and Initial Build]].''

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] and [http://ant.apache.org ant]
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder.

While there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Software to Install ===

# While you can only run GenMAPP and MAPPFinder on Windows, you can build and run GenMAPP Builder on any platform: Windows, Linux, or Mac OS X. You need to download and install the Java Development Kit from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html (which, at this writing, is ''JDK 8u65'').
# Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.

=== Initial Code Checkout ===

# Get an account from https://github.com and send your account name to Dr. Dionisio, for inclusion as an XMLPipeDB developer.
# Run Eclipse.
# Establish an Eclipse workspace for the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB repository].


=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T01:16:13Z

Dondi: /* Milestones */ Version control setup should come before developer rig and initial build.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the XMLPipeDB project.
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
# ''(with QA)'' Commit relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch

=== Milestone 2: “Developer Rig” Setup and Initial As-Is Build ===

''More specific and technical details for accomplishing this milestone are listed in [[#GenMAPP Builder Project Setup and Initial Build]].''

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] and [http://ant.apache.org ant]
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder.

While there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Software to Install ===

# While you can only run GenMAPP and MAPPFinder on Windows, you can build and run GenMAPP Builder on any platform: Windows, Linux, or Mac OS X. You need to download and install the Java Development Kit from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html (which, at this writing, is ''JDK 8u65'').
# Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.

=== Initial Code Checkout ===

# Get an account from https://github.com and send your account name to Dr. Dionisio, for inclusion as an XMLPipeDB developer.
# Run Eclipse.
# Establish an Eclipse workspace for the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB repository].


=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T01:13:00Z

Dondi: /* Milestones */ Name milestones and separate working environment from developer rig.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 0: Working Environment Setup ===

''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.''

* PostgreSQL (http://www.postgresql.org)
* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
* GenMAPP (http://genmapp.org)
** We will be using GenMAPP and MAPPFinder version 2.1.
*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
*** Follow the instructions in the installer.
*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
* XMLPipeDB ''match'' utility for counting IDs in XML files
* Microsoft Access or any other tool that can read ''.mdb'' files

=== Milestone 1: “Developer Rig” Setup and Initial As-Is Build ===

# Install core software for developing, building, and testing prototype versions of GenMAPP Builder:
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] and [http://ant.apache.org ant]
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* ''XMLPipeDB Match'' utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used
#** Eclipse includes ''ant'' so you do not need a separate ''ant'' installation unless you plan to build GenMAPP Builder outside of Eclipse
#** If you want to use ''ant'' outside Eclipse, please visit http://ant.apache.org.
# Build your own copy of GenMAPP Builder from scratch.
# ''(with QA)'' Get a full import-export cycle done.
# ''(with QA)'' Decide on a file/version management scheme/system.

=== Milestone 2: Version Control Setup ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the XMLPipeDB project.
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Create a GitHub branch of ''xmlpipedb'' for your team.
# ''(with QA)'' Commit relevant source data to the ''GenMAPP Gene Databases'' folder of your GitHub branch

As needed, coders may arrange for a walkthrough or other help session with Dr. Dionisio if there are any issues with the procedures on this guild page.

=== Milestone 3: Species Profile Creation ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the ''OrderedLocusNames'' record of the Systems table.
# Customize the ''Link'' field in the ''OrderedLocusNames'' record of the ''Systems table'' to hold a URL query with <code>~</code> standing in for the gene ID.
#* ''(with QA)'' The URL would need to be determined first, of course.

=== Milestone 4: Species Export Customization ===

# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.
# Implement this export behavior.
# As needed, commit and push your work to your GitHub branch.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.
# Document/log all work done, problems encountered, and how they were resolved.
# When your work is complete, issue a GitHub ''pull request'' to merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder.

While there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Software to Install ===

# While you can only run GenMAPP and MAPPFinder on Windows, you can build and run GenMAPP Builder on any platform: Windows, Linux, or Mac OS X. You need to download and install the Java Development Kit from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html (which, at this writing, is ''JDK 8u65'').
# Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.

=== Initial Code Checkout ===

# Get an account from https://github.com and send your account name to Dr. Dionisio, for inclusion as an XMLPipeDB developer.
# Run Eclipse.
# Establish an Eclipse workspace for the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB repository].


=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Gene Database Project

2015-11-02T00:42:37Z

Dondi: /* GenMAPP User (GU) */ Synchronize overview sentence with guild page.

{{Gene Database Project Links}}

== Deliverables ==

* You will give a final group PowerPoint presentation in class on '''Tuesday, December 15, 2:00pm'''.
* Final due date for all other deliverables is no later than '''Friday, December 18, 4:30pm''' (according to official final exam schedule for fall, 2015)
* The deliverables should be uploaded and organized onto one group wiki page, or alternately, delivered on digital media to either Dr. Dahlquist or Dr. Dionisio.
* Each member of the group will be assigned the same grade for the group project.

=== Group Deliverables ===

* GenMAPP Gene Database for assigned species (''.gdb'')
* ReadMe file to accompany the Gene Database (''.pdf'')
** Sample ReadMe in Word format: [[Media:ReadMe_Vc-Std_External_20131122.zip | ReadMe_Vc-Std_External_20131122.zip]]
** Include Gene Database Schema diagram in ReadMe
*** Sample schema in Adobe Illustrator format: [[Media:Vibrio_schema_20101022.zip | Vibrio_schema_20101022.zip]]
* Gene Database Testing Report for final submitted Gene Database (print from wiki to ''.pdf'' file)
* Processed and analyzed DNA microarray dataset (''.xls'')
* GenMAPP Expression Dataset file (''.gex'')
* Filtered MAPPFinder Results (''.xls'')
* Sample MAPP file of a relevant biological pathway for your species (''.mapp'')
* [[Gene Database Project Deliverables#Group Report | Group Report]] describing the creation of the Gene Database and the biological analysis of the data (''.doc'' or ''.pdf'')
* PowerPoint presentation (''.ppt'', given on Tuesday, December 15)

=== Individual Deliverable ===

The individual deliverable is an [[Gene Database Project Deliverables#Individual Assessment and Reflection|assessment and reflection]] on the process (either wiki or email, depending on your preference; see note in the linked section):
* Statement of work
* Assessment of the work done
* What was learned
* Team evaluation via the [https://www.catme.org CATME] survey
*: ''You will, or should have already, received an email message from the CATME system with login instructions on your '''LMU email account''' (not necessarily the preferred email address that you listed in the wiki).''

== Team Journal Entries ==

* Each team will write a combined journal entry for each week with contributions from all members.
* [[Week 10]] Creation of page (midnight 11/10)
* [[Week 11]] (midnight 11/17)
* [[Week 12]] (midnight 11/24)
* [[Week 14]] (midnight 12/8)
* [[Week 15]] (midnight 12/15)

== Groups ==

The project groups are:
# Team 1
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:
# Team 2
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:
# Team 3
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:
# Team 4
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:

== Roles (Guilds) ==

As the project moves forward, we will use class time for team meetings. In addition, we will also have guild meetings where students sharing the same role can work together on common issues. Each student has been assigned a primary role in the project by the instructors (see above).

=== [[Project Manager]] (PM) ===

The project manager makes sure that individuals are fulfilling their roles and performing the tasks on time.

=== [[GenMAPP User]] (GU) ===

The GenMAPP user runs GenMAPP with the microarray dataset (Expression Dataset Manager, MAPPFinder, GenMAPP MAPP). His or her tasks generally correspond to the work done during the DNA microarray analysis exercise ([http://www.openwetware.org/wiki/BIOL398-01/S10:Sample_Microarray_Analysis_Vibrio_cholerae part 1], [http://www.openwetware.org/wiki/BIOL367/F10:GenMAPP_and_MAPPFinder_Protocols part 2]), as applicable to the team’s chosen bacterium.

=== [[Quality Assurance]] (QA) ===

The Quality Assurance team member is the resident expert on species ID systems and formats. He or she should be proficient with ''XMLPipeDB Match'', SQL queries in PostgreSQL, Microsoft Excel, and Microsoft Access to navigate through the data and find missing IDs, discrepancies, sanity checks, etc.

=== [[Coder]] (C) ===

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Project Milestones ==

Specific project milestones are found on the individual guild pages.

=== Overall Flow ===

A successful project will have the following steps from end to end. Depending on the data and/or other issues that are encountered, one or more of these steps may go through iterations, repetitions, or refinements.

[[Image:ProjectWorkflow.png|450px|thumb|right|Overall Flow]]

# Following the steps shown in the [[Running GenMAPP Builder]] instructions, download the UniProt XML proteome set and GOA (GO association) files for your species.
#* Make sure that you are choosing the correct strain/subspecies as the microarray data you have.
#* Note the date of download and the version of the file on your Gene Database Testing Report.
# Download GO terms from [http://geneontology.org/page/download-ontology#Legacy_Downloads Gene Ontology].
#* You will need the OBO-XML format, the "obo-xml.gz" file.
#* Note the date of download and the version of the file on your Gene Database Testing Report.
# Create the GenMAPP Builder tables in PostgreSQL.
# Load files into PostgreSQL database via GenMAPP Builder.
# Export into a GenMAPP Gene Database.
# Inspect/vet/validate Gene Database.
#* Create a [[Gene Database Testing Report Sample | Gene Database Testing Report]] each time a new export is run.
#* Compare to the model organism database for your species
# Find an original journal article describing a DNA microarray experiment performed on your species.
#* Download the microarray data for this article.
#* Consult with Dr. Dahlquist about the proper steps to take to process the data (normalization, statistical analysis) and perform the analysis.
# Run GenMAPP using the Gene Database.
#* Microarray data (import using Expression Dataset Manager)
#* MAPPFinder
#* Place genes on MAPP and draw pathway

=== Guild Milestones ===

These are links to the respective milestone lists in the guild pages:
* [[Project Manager#Milestones|Project Manager]]
* [[GenMAPP User#Milestones|GenMAPP User]]
* [[Quality Assurance#Milestones|Quality Assurance]]
* [[Coder#Milestones|Coder]]

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T00:41:45Z

Dondi: Add overview sentence.

{{Gene Database Project Links}}

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 1 ===

# For your team: set up the working environment, if necessary (already done on the machines in the Seaver 120 computer lab)
#: ''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.
#* PostgreSQL (http://www.postgresql.org)
#* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
#** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
#** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
#* GenMAPP (http://genmapp.org)
#** We will be using GenMAPP and MAPPFinder version 2.1.
#*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
#*** Follow the instructions in the installer.
#*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
#* XMLPipeDB ''match'' utility for counting IDs in XML files
#* Microsoft Access or any other tool that can read ''.mdb'' files
# Set up the development environment
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] and [http://ant.apache.org ant]
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* XMLPipeDB Match utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used
# (with QA) Get a full import-export cycle done.
# (with QA) Decide on a file/version management scheme/system.
# (with QA) Commit relevant source data to your GitHub branch
# Document/log all work done, problems encountered, and how they were resolved.

=== Milestone 2 ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the XMLPipeDB project.
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Use ''ant'' to build your own copy of GenMAPP Builder from scratch.
# Create a GitHub branch of ''xmlpipedb'' for your team.
# As needed, coders should arrange for a walkthrough or other help session with Dr. Dionisio.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.

=== Milestone 3 ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the OrderedLocusNames record of the Systems table.
# Customize the Link field in the OrderedLocusNames record of the Systems table to hold a URL query with <code>~</code> standing in for the gene ID.
#* The URL would need to be determined first, of course.
# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.

=== Milestone 4 ===

# Implement the export behavior defined in [[#Milestone 3|Milestone 3]].
# As needed, commit and push your species profile to your GitHub branch.
# When your work is complete, merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder.

While there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Software to Install ===

# While you can only run GenMAPP and MAPPFinder on Windows, you can build and run GenMAPP Builder on any platform: Windows, Linux, or Mac OS X. You need to download and install the Java Development Kit from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html (which, at this writing, is ''JDK 8u65'').
# Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.

=== Initial Code Checkout ===

# Get an account from https://github.com and send your account name to Dr. Dionisio, for inclusion as an XMLPipeDB developer.
# Run Eclipse.
# Establish an Eclipse workspace for the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB repository].


=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Quality Assurance

2015-11-02T00:41:14Z

Dondi: Add dropped italics.

Gene Database Project

2015-11-02T00:40:25Z

Dondi: /* Coder (C) */ Add version control to coder overview.

{{Gene Database Project Links}}

== Deliverables ==

* You will give a final group PowerPoint presentation in class on '''Tuesday, December 15, 2:00pm'''.
* Final due date for all other deliverables is no later than '''Friday, December 18, 4:30pm''' (according to official final exam schedule for fall, 2015)
* The deliverables should be uploaded and organized onto one group wiki page, or alternately, delivered on digital media to either Dr. Dahlquist or Dr. Dionisio.
* Each member of the group will be assigned the same grade for the group project.

=== Group Deliverables ===

* GenMAPP Gene Database for assigned species (''.gdb'')
* ReadMe file to accompany the Gene Database (''.pdf'')
** Sample ReadMe in Word format: [[Media:ReadMe_Vc-Std_External_20131122.zip | ReadMe_Vc-Std_External_20131122.zip]]
** Include Gene Database Schema diagram in ReadMe
*** Sample schema in Adobe Illustrator format: [[Media:Vibrio_schema_20101022.zip | Vibrio_schema_20101022.zip]]
* Gene Database Testing Report for final submitted Gene Database (print from wiki to ''.pdf'' file)
* Processed and analyzed DNA microarray dataset (''.xls'')
* GenMAPP Expression Dataset file (''.gex'')
* Filtered MAPPFinder Results (''.xls'')
* Sample MAPP file of a relevant biological pathway for your species (''.mapp'')
* [[Gene Database Project Deliverables#Group Report | Group Report]] describing the creation of the Gene Database and the biological analysis of the data (''.doc'' or ''.pdf'')
* PowerPoint presentation (''.ppt'', given on Tuesday, December 15)

=== Individual Deliverable ===

The individual deliverable is an [[Gene Database Project Deliverables#Individual Assessment and Reflection|assessment and reflection]] on the process (either wiki or email, depending on your preference; see note in the linked section):
* Statement of work
* Assessment of the work done
* What was learned
* Team evaluation via the [https://www.catme.org CATME] survey
*: ''You will, or should have already, received an email message from the CATME system with login instructions on your '''LMU email account''' (not necessarily the preferred email address that you listed in the wiki).''

== Team Journal Entries ==

* Each team will write a combined journal entry for each week with contributions from all members.
* [[Week 10]] Creation of page (midnight 11/10)
* [[Week 11]] (midnight 11/17)
* [[Week 12]] (midnight 11/24)
* [[Week 14]] (midnight 12/8)
* [[Week 15]] (midnight 12/15)

== Groups ==

The project groups are:
# Team 1
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:
# Team 2
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:
# Team 3
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:
# Team 4
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:

== Roles (Guilds) ==

As the project moves forward, we will use class time for team meetings. In addition, we will also have guild meetings where students sharing the same role can work together on common issues. Each student has been assigned a primary role in the project by the instructors (see above).

=== [[Project Manager]] (PM) ===

The project manager makes sure that individuals are fulfilling their roles and performing the tasks on time.

=== [[GenMAPP User]] (GU) ===

The GenMAPP user runs GenMAPP with the microarray dataset (Expression Dataset Manager, MAPPFinder, GenMAPP MAPP).

=== [[Quality Assurance]] (QA) ===

The Quality Assurance team member is the resident expert on species ID systems and formats. He or she should be proficient with ''XMLPipeDB Match'', SQL queries in PostgreSQL, Microsoft Excel, and Microsoft Access to navigate through the data and find missing IDs, discrepancies, sanity checks, etc.

=== [[Coder]] (C) ===

The coder is the resident expert on the technology being used—assorted software, file management, version control, some troubleshooting, some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Project Milestones ==

Specific project milestones are found on the individual guild pages.

=== Overall Flow ===

A successful project will have the following steps from end to end. Depending on the data and/or other issues that are encountered, one or more of these steps may go through iterations, repetitions, or refinements.

[[Image:ProjectWorkflow.png|450px|thumb|right|Overall Flow]]

# Following the steps shown in the [[Running GenMAPP Builder]] instructions, download the UniProt XML proteome set and GOA (GO association) files for your species.
#* Make sure that you are choosing the correct strain/subspecies as the microarray data you have.
#* Note the date of download and the version of the file on your Gene Database Testing Report.
# Download GO terms from [http://geneontology.org/page/download-ontology#Legacy_Downloads Gene Ontology].
#* You will need the OBO-XML format, the "obo-xml.gz" file.
#* Note the date of download and the version of the file on your Gene Database Testing Report.
# Create the GenMAPP Builder tables in PostgreSQL.
# Load files into PostgreSQL database via GenMAPP Builder.
# Export into a GenMAPP Gene Database.
# Inspect/vet/validate Gene Database.
#* Create a [[Gene Database Testing Report Sample | Gene Database Testing Report]] each time a new export is run.
#* Compare to the model organism database for your species
# Find an original journal article describing a DNA microarray experiment performed on your species.
#* Download the microarray data for this article.
#* Consult with Dr. Dahlquist about the proper steps to take to process the data (normalization, statistical analysis) and perform the analysis.
# Run GenMAPP using the Gene Database.
#* Microarray data (import using Expression Dataset Manager)
#* MAPPFinder
#* Place genes on MAPP and draw pathway

=== Guild Milestones ===

These are links to the respective milestone lists in the guild pages:
* [[Project Manager#Milestones|Project Manager]]
* [[GenMAPP User#Milestones|GenMAPP User]]
* [[Quality Assurance#Milestones|Quality Assurance]]
* [[Coder#Milestones|Coder]]

{{Gene Database Project Links}}

[[Category:Group Projects]]

Quality Assurance

2015-11-02T00:39:25Z

Dondi: Provide overview and milestone names.

{{Gene Database Project Links}}

The Quality Assurance team member is the resident expert on species ID systems and formats. He or she should be proficient with XMLPipeDB Match, SQL queries in PostgreSQL, Microsoft Excel, and Microsoft Access to navigate through the data and find missing IDs, discrepancies, sanity checks, etc.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 1: Initial Database Export ===

# (with Coders) Get a full import-export cycle done.
# (with Coders) Decide on a file/version management scheme/system.
# Learn the ID systems:
#* Systems that are the same for each species (hint: guild members help each other out by posting the relevant information on this page)
#** UniProt
#** RefSeq
#** GeneID (EntrezGene from NCBI)
#** GO
#* The OrderedLocusNames for your species

=== Milestone 2: ID Pattern Definition and Verification ===

# Characterize regular expression patterns to detect the IDs ([[How Do I Count Thee? Let Me Count The Ways|for filtering then counting]]).
#* XMLPipeDB Match utility
#* Direct SQL queries in PostgreSQL
#* For example, the ''Vibrio'' IDs were of the form <code>VC####</code> or <code>VC_####</code>; how would you express that in Match or as an SQL query?
#* Table inspection/filtering/sorting in Microsoft Access
#* If needed, side-by-side sorted comparisons in Microsoft Excel ([[Using Microsoft Excel to Compare ID Lists|as described here]])
# Document/log all work done, problems encountered, and how they were resolved.

=== Milestone 3: Tally Engine Configuration ===

Along with your Coder, customize the Tally Engine setup for your species as specified in [[Coder#Customize_the_IDs_that_the_Tally_Engine_Counts|these coder steps]]. You will want to add, at the very least, the ''ordered locus'' IDs for your species.

=== Milestone 4: Final Documentation ===

# Document the relational database schema for the gene database.
# Create the ReadMe with comparisons to MOD for your species.

{{Gene Database Project Links}}

[[Category:Group Projects]]

GenMAPP User

2015-11-02T00:36:47Z

Dondi: Give some overviews and names to milestones.

{{Gene Database Project Links}}

The GenMAPP user runs GenMAPP with the microarray dataset (Expression Dataset Manager, MAPPFinder, GenMAPP MAPP). His or her tasks generally correspond to the work done during the DNA microarray analysis exercise ([http://www.openwetware.org/wiki/BIOL398-01/S10:Sample_Microarray_Analysis_Vibrio_cholerae part 1], [http://www.openwetware.org/wiki/BIOL367/F10:GenMAPP_and_MAPPFinder_Protocols part 2]), as applicable to the team’s chosen bacterium.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

The milestones do not necessarily correspond to particular weeks; instead they are sets of tasks grouped together.

=== Milestone 1: Startup from Original Microarray Paper and Data ===

# Find and acquire the microarray paper for your team (done [[Week 10]]).
#* Provide the proper bibliographic reference for your paper.
#* Link to the HTML version of the paper on the journal website.
#* Upload the PDF version of the paper to the wiki ''(if no copyright restrictions)''; otherwise provide a download link to the publisher site.
# Download microarray data in its “rawest” form that you can find. (Consult with Dr. Dahlquist about this.)
# Verify that the gene IDs in the microarray data match the chosen species and '''''strain''''' that is being used to create the ''.gdb''.
#* This task needs to be done in conjunction with the Quality Assurance person and Coder.
# File management: on your team's home page
#* Link to the source of the microarray data
#* Upload the microarray data files to the wiki

=== Milestone 2: Data Preparation ===

# Read the microarray paper to understand the experiment.
# Create a table or list that shows the correspondence between the samples in the experiment and the files you have downloaded.
# Determine how many biological or technical replicates, and which samples were labeled with Cy3 or Cy5.
# Create a Master Raw Data file that contains the IDs and columns of data required for further analysis.
# Consult with Dr. Dahlquist on how to process the data (normalization, statistics).

=== Milestone 3: On-going Analysis Cycle ===

This milestone represents the on-going work of the GenMAPP user, as the project gradually converges toward a final version of its gene database and compares its own results to the results in the original microarray paper.

# Perform the statistical analysis in Excel.
# Format the gene expression data for import into GenMAPP.
# Import data into GenMAPP, create ColorSets, and run MAPPFinder.
# Document and take notes on test runs with GenMAPP.
#* Use the ''EX.txt'' file to help the Coder/Quality Assurance team members to validate the ''.gdb''.
# Do a journal club outline of the paper so that you can use it in the Discussion section of your group report and your final presentation.
# Create a ''.mapp'' file showing one pathway that is changed in your data.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Project Manager

2015-11-02T00:30:53Z

Dondi: Add overview sentence.

{{Gene Database Project Links}}

The project manager makes sure that individuals are fulfilling their roles and performing the tasks on time.

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 1: Project “Scaffolding” ===

This milestone pertains to setting up an initial schedule and any resources that your team will use for the duration of the project. It will be useful to get an overview of every team member’s own milestones so that you have an accurate big picture view.

# In consultation with your team, work backward from the final deadline to set intermediate deadlines for each deliverable. In particular you need to set deadlines for what you will accomplish by the journal deadline for [[Week 12]], [[Week 14]], and [[Week 15]].
# Organize management tools for your team:
#* Workflow narratives
#* Action items
#* Testing results/reports
#** Bugs/feature requests
#** Question/answer sequences

=== Milestone 2: Periodic Updates ===

Not as much a milestone as an on-going task, once the project is up and running the Project Manager is responsible for keeping track of everyone’s progress.

# Get periodic updates on progress; in particular, the project’s “place” in the overall flow should be known at all times (transparency). You and your team will be giving a status report in class for the rest of the semester.
# Monitor the status of the report-in-progress and other related documentation.
# Coordinate team decisions and action items addressing any unforeseen delays or roadblocks.



{{Gene Database Project Links}}

[[Category:Group Projects]]

Project Manager

2015-11-02T00:29:03Z

Dondi: /* Milestones */ Give names to milestones.

{{Gene Database Project Links}}

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 1: Project “Scaffolding” ===

This milestone pertains to setting up an initial schedule and any resources that your team will use for the duration of the project. It will be useful to get an overview of every team member’s own milestones so that you have an accurate big picture view.

# In consultation with your team, work backward from the final deadline to set intermediate deadlines for each deliverable. In particular you need to set deadlines for what you will accomplish by the journal deadline for [[Week 12]], [[Week 14]], and [[Week 15]].
# Organize management tools for your team:
#* Workflow narratives
#* Action items
#* Testing results/reports
#** Bugs/feature requests
#** Question/answer sequences

=== Milestone 2: Periodic Updates ===

Not as much a milestone as an on-going task, once the project is up and running the Project Manager is responsible for keeping track of everyone’s progress.

# Get periodic updates on progress; in particular, the project’s “place” in the overall flow should be known at all times (transparency). You and your team will be giving a status report in class for the rest of the semester.
# Monitor the status of the report-in-progress and other related documentation.
# Coordinate team decisions and action items addressing any unforeseen delays or roadblocks.



{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-02T00:10:51Z

Dondi: /* Milestone 1 */ Clarify that working environment is already done for the lab and update GenMAPP link.

{{Gene Database Project Links}}

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 1 ===

# For your team: set up the working environment, if necessary (already done on the machines in the Seaver 120 computer lab)
#: ''Because the machines in the Seaver 120 computer lab have already been set up for this process, the information below is listed primarily for documentation and troubleshooting purposes.
#* PostgreSQL (http://www.postgresql.org)
#* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
#** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
#** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
#* GenMAPP (http://genmapp.org)
#** We will be using GenMAPP and MAPPFinder version 2.1.
#*** This version is now called "GenMAPP Classic" and can be downloaded [https://github.com/GenMAPPCS/genmapp from GitHub].
#*** Follow the instructions in the installer.
#*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
#* XMLPipeDB ''match'' utility for counting IDs in XML files
#* Microsoft Access or any other tool that can read ''.mdb'' files
# Set up the development environment
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] and [http://ant.apache.org ant]
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* XMLPipeDB Match utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used
# (with QA) Get a full import-export cycle done.
# (with QA) Decide on a file/version management scheme/system.
# (with QA) Commit relevant source data to your GitHub branch
# Document/log all work done, problems encountered, and how they were resolved.

=== Milestone 2 ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the XMLPipeDB project.
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Use ''ant'' to build your own copy of GenMAPP Builder from scratch.
# Create a GitHub branch of ''xmlpipedb'' for your team.
# As needed, coders should arrange for a walkthrough or other help session with Dr. Dionisio.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.

=== Milestone 3 ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the OrderedLocusNames record of the Systems table.
# Customize the Link field in the OrderedLocusNames record of the Systems table to hold a URL query with <code>~</code> standing in for the gene ID.
#* The URL would need to be determined first, of course.
# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.

=== Milestone 4 ===

# Implement the export behavior defined in [[#Milestone 3|Milestone 3]].
# As needed, commit and push your species profile to your GitHub branch.
# When your work is complete, merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder.

While there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Software to Install ===

# While you can only run GenMAPP and MAPPFinder on Windows, you can build and run GenMAPP Builder on any platform: Windows, Linux, or Mac OS X. You need to download and install the Java Development Kit from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html (which, at this writing, is ''JDK 8u65'').
# Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.

=== Initial Code Checkout ===

# Get an account from https://github.com and send your account name to Dr. Dionisio, for inclusion as an XMLPipeDB developer.
# Run Eclipse.
# Establish an Eclipse workspace for the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB repository].


=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-01T23:54:44Z

Dondi: /* Creating a Distribution */ Note on setting VERSION string.

{{Gene Database Project Links}}

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 1 ===

# For your team: set up the working environment, if necessary (already done on the machines in the Seaver 120 computer lab)
#* PostgreSQL (http://www.postgresql.org)
#* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
#** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
#** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
#* GenMAPP (http://genmapp.org)
#** We will be using GenMAPP and MAPPFinder version 2.1.
#*** This version is now called "GenMAPP Classic" and can be downloaded [http://www.genmapp.org/download_v2.1.php from this page].
#*** Follow the instructions in the installer.
#*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
#* XMLPipeDB ''match'' utility for counting IDs in XML files
#* Microsoft Access or any other tool that can read ''.mdb'' files
# Set up the development environment
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] and [http://ant.apache.org ant]
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* XMLPipeDB Match utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used
# (with QA) Get a full import-export cycle done.
# (with QA) Decide on a file/version management scheme/system.
# (with QA) Commit relevant source data to your GitHub branch
# Document/log all work done, problems encountered, and how they were resolved.

=== Milestone 2 ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the XMLPipeDB project.
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Use ''ant'' to build your own copy of GenMAPP Builder from scratch.
# Create a GitHub branch of ''xmlpipedb'' for your team.
# As needed, coders should arrange for a walkthrough or other help session with Dr. Dionisio.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.

=== Milestone 3 ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the OrderedLocusNames record of the Systems table.
# Customize the Link field in the OrderedLocusNames record of the Systems table to hold a URL query with <code>~</code> standing in for the gene ID.
#* The URL would need to be determined first, of course.
# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.

=== Milestone 4 ===

# Implement the export behavior defined in [[#Milestone 3|Milestone 3]].
# As needed, commit and push your species profile to your GitHub branch.
# When your work is complete, merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder.

While there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Software to Install ===

# While you can only run GenMAPP and MAPPFinder on Windows, you can build and run GenMAPP Builder on any platform: Windows, Linux, or Mac OS X. You need to download and install the Java Development Kit from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html (which, at this writing, is ''JDK 8u65'').
# Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.

=== Initial Code Checkout ===

# Get an account from https://github.com and send your account name to Dr. Dionisio, for inclusion as an XMLPipeDB developer.
# Run Eclipse.
# Establish an Eclipse workspace for the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB repository].


=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create by setting the <code>VERSION</code> string (located at approximately line 83) to a sufficiently descriptive value.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Coder

2015-11-01T09:42:35Z

Dondi: Initial transcription of this page.

{{Gene Database Project Links}}

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 1 ===

# For your team: set up the working environment, if necessary (already done on the machines in the Seaver 120 computer lab)
#* PostgreSQL (http://www.postgresql.org)
#* GenMAPP Builder (https://github.com/lmu-bioinformatics/xmlpipedb)
#** Current version is [https://github.com/lmu-bioinformatics/xmlpipedb/releases/tag/gmbuilder-3.0.0-build-5 3.0.0 build 5]
#** Requires [http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html Java 8 Runtime Environment]
#* GenMAPP (http://genmapp.org)
#** We will be using GenMAPP and MAPPFinder version 2.1.
#*** This version is now called "GenMAPP Classic" and can be downloaded [http://www.genmapp.org/download_v2.1.php from this page].
#*** Follow the instructions in the installer.
#*** During installation, the installer will open a window called the GenMAPP Data Acquisition Tool. It will not function because it cannot connect to the server. This is OK.
#* XMLPipeDB ''match'' utility for counting IDs in XML files
#* Microsoft Access or any other tool that can read ''.mdb'' files
# Set up the development environment
#* Java developer tools: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html JDK 8] and [http://ant.apache.org ant]
#* A ''git'' client (for interacting with GitHub)
#* Any tool that can unpack .gz and .zip files (we are using [http://www.7-zip.org/ 7-zip] on the Seaver 120 machines)
#* XMLPipeDB Match utility
#* Development environment: while any will do, [http://www.eclipse.org Eclipse] is the specific one that most XMLPipeDB developers have used
# (with QA) Get a full import-export cycle done.
# (with QA) Decide on a file/version management scheme/system.
# (with QA) Commit relevant source data to your GitHub branch
# Document/log all work done, problems encountered, and how they were resolved.

=== Milestone 2 ===

# Get a GitHub account and pass it to Dr. Dionisio so that you can be added as a developer of the XMLPipeDB project.
#* Once you are set up as a developer, you can clone and push your GenMAPP Builder source code.
# Use ''ant'' to build your own copy of GenMAPP Builder from scratch.
# Create a GitHub branch of ''xmlpipedb'' for your team.
# As needed, coders should arrange for a walkthrough or other help session with Dr. Dionisio.
# Additional milestones will depend on how the rest of the project goes, and the bugs/features generated by that work.

=== Milestone 3 ===

# Add a ''species profile'' to the GenMAPP Builder code base.
# Customize the species profile with the species name in the OrderedLocusNames record of the Systems table.
# Customize the Link field in the OrderedLocusNames record of the Systems table to hold a URL query with <code>~</code> standing in for the gene ID.
#* The URL would need to be determined first, of course.
# Based on observations from the GenMAPP User and QA, determine and document (as thoroughly as possible) any other modified export behavior that GenMAPP Builder will have to manifest for this species.

=== Milestone 4 ===

# Implement the export behavior defined in [[#Milestone 3|Milestone 3]].
# As needed, commit and push your species profile to your GitHub branch.
# When your work is complete, merge your branch into the main development line.

== GenMAPP Builder Project Setup and Initial Build ==

This section of the page seeks to provide a guide for building new versions of GenMAPP Builder.

While there are many ways to update and maintain GenMAPP Builder code, for uniformity these instructions will assume the use of [http://www.eclipse.org Eclipse] for viewing, modifying, and updating GenMAPP Builder. The main benefit of Eclipse is that it is largely a one-stop shop for performing all of these tasks.

The instructions listed in this [[#Setup|Setup]] section need only be performed once. Once done correctly, you will primarily be doing what is described in the [[#Common Tasks|Common Tasks]] section.

=== Software to Install ===

# While you can only run GenMAPP and MAPPFinder on Windows, you can build and run GenMAPP Builder on any platform: Windows, Linux, or Mac OS X. You need to download and install the Java Development Kit from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html (which, at this writing, is ''JDK 8u65'').
# Download and install Eclipse from its [http://www.eclipse.org/downloads download web site]. Either '''Eclipse IDE for Java Developers''' or '''Eclipse IDE for Java EE Developers''' will work.

=== Initial Code Checkout ===

# Get an account from https://github.com and send your account name to Dr. Dionisio, for inclusion as an XMLPipeDB developer.
# Run Eclipse.
# Establish an Eclipse workspace for the [https://github.com/lmu-bioinformatics/xmlpipedb XMLPipeDB repository].


=== Java Project Setup ===

# Make sure that Eclipse is using a '''''JDK''''' (Java Development Kit) and not a ''JRE'' (Java Runtime Environment). To verify this, go to ''Window'' > ''Preferences'', click on ''Java'', click on ''Installed JREs'', and make sure that the checked environment has '''JDK''' in it. If not, you may need to add the environment (on Windows, it lives in <code>C:\Program Files\Java</code>) then check on it.
# Upon a successful checkout, you should have a ''gmbuilder'' Java project in Eclipse. Make sure that you are in the ''Java'' perspective by choosing ''Window'' > ''Open Perspective'' > ''Java'' (or choosing ''Java'' from ''Other...'' if ''Java'' is not already in the ''Open Perspective'' submenu).
# Double-click on the ''gmbuilder'' folder (or whatever you called it) to view its contents.
# Double-click on the ''lib'' folder.
# Shift-click and select all files in the ''lib'' folder and then control-click (or, on a Mac, Command-click) on every file inside the ''lib'' folder whose name does not end in ''.jar''.
# Once all of these files are selected, right-click on one of them and choose ''Build Path'' > ''Add to Build Path'' from the popup menu that comes up.
# The ''src'' folder should look different from the other folders in that it has a little brown square badge on its upper-right corner. If not, right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# Do the same to the ''test'' folder: right-click on it and choose ''Build Path'' > ''Use as Source Folder'' from the popup menu that comes up.
# If you see any red ''x'' icons appear, something has not been set up right. Contact other guild members or Dr. Dionisio for troubleshooting if you get stuck.

== Adding a Species Profile to GenMAPP Builder ==

All of this work happens in the ''Java'' perspective, so switch to that first if you’re not already there.

=== Create the Species Profile ===

# Expose the contents of the ''src'' folder.
# Right-click on the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package and choose ''New > Class'' from the popup menu.
# In the dialog that appears, enter the following:
#* '''Name:''' <code>''name-of-your-species-without-spaces''UniProtSpeciesProfile</code> (in ''camel case'': no spaces, capitalizing the first letters of each word)
#* '''Superclass:''' <code>edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles.UniProtSpeciesProfile</code> (you can also click on ''Browse...'' to navigate to this if you don’t feel like typing)
# Click ''Finish''. There should now be a new ''.java'' file within the ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'' package (the one you just created).

=== Customize the Species Profile ===

* Open the file that you have just created. It should appear in the editor area of Eclipse.
* Override the method that supplies the name of the species and the description of the profile: add the following constructor block right below the ''public class'' line in the new file. Remember to customize according to your particular species; the portions that need to be customized are highlighted in asterisks.
public ***NameOfYourSpecies***UniProtSpeciesProfile() {
super("***Genus species***",
***taxonIDOfYourSpecies***,
"This profile customizes the GenMAPP Builder export for " +
"***Genus species***" +
" data loaded from a UniProt XML file.");
}
* To customize the species profile with the species name in the OrderedLocusNames record of the Systems table as well as a link query for that same record, add the following method block right below the constructor block that you added above. Again, the key information to customize is highlighted in asterisks.
@Override
public TableManager getSystemsTableManagerCustomizations(TableManager tableManager, DatabaseProfile dbProfile) {
super.getSystemsTableManagerCustomizations(tableManager, dbProfile);
tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Species", "|" + getSpeciesName() + "|" }
});

tableManager.submit("Systems", QueryType.update, new String[][] {
{ "SystemCode", "N" },
{ "Link", "***species-specific-database-link***" }
});

return tableManager;
}
* Note the '''species-specific-database-link''' placeholder above. This is a species-specific URL that returns a web page describing a gene for that species. It should look like a standard URL, with the tilde ('''~''') standing in for the gene ID. For example, the link for ''Vibrio cholerae'' is <code>http://bacteria.ensembl.org/Multi/Search/Results?species=all;idx=;q=~;site=ensemblunit</code>. The link for ''Plasmodium falciparum'' is <code>http://plasmodb.org/plasmo/showRecord.do?name=GeneRecordClasses.GeneRecordClass&project_id=PlasmoDB&source_id=~</code>. Work with your GenMAPP User and/or QA to determine the appropriate URL for your species.
* Your code may have a red error badge at this point; assuming you typed everything in exactly, the fix for this is to choose ''Organize Imports'' from the ''Source'' menu. If the red error badge persists, make sure that you typed everything in correctly.
* Save the file and see if these changes worked (see below).

Additional customization, particularly with regard to the exported data, will depend on the species. Communicate with your QA to see if additional customization is needed. If the additional customization is not too complicated, you might be able to do the work yourself with some instructions. However, if the customization is too difficult, Dr. Dionisio will probably be the one to do the work.

=== Customize the IDs that the Tally Engine Counts ===

This step is technically optional, in that it does not affect the overall import/export process. However, it ''does'' help you to get an idea of how well the IDs from the UniProt XML file are being brought into the relational database.

# First, determine which IDs (outside of the defaults that the tally engine already counts) you would like to count. At a minimum, this includes at least the ''ordered locus'' IDs from the ''gene/name'' tag in the UniProt XML file. There may be more; consult with your QA.
# For each of these IDs, determine the following:
#* Where in the XML file they can be found, in terms of which XML tags
#* Where in the relational database they can be found, in terms of which relational tables
# Under ''edu.lmu.xmlpipedb.gmbuilder.resource.properties'', open ''gmbuilder.properties''.
# Locate the block of text below (it’s near the bottom). You will insert the customizations that will be described right above this block.
#
# wizard.properties
#

* First, mark out the section that denotes the customization for your species:
# Species name
* Next, rewrite your species name without spaces and all lowercase (e.g., ''Plasmodium falciparum'' becomes ''plasmodiumfalciparum''). Specify the number of additional custom IDs to count as follows, where ''speciesname'' is your no-space, all-lowercase species name, and ''#'' represents the actual number of IDs:
speciesname_level_amount=#
* Now, ''for each custom ID'', you need to specify three things: an ''element'', a ''query'', and a ''name''. Each of these items is numbered, starting from 0. Each item number is called a ''level''.
*# The ''element'' states where you expect an ID to be found in the UniProt XML file. It starts with ''uniprot/entry'', then continues with additional tags as needed. After the tag, you may specify, separated by ampersands (''&''s), any specific attributes that you would like to choose.
*# The ''query'' states the SQL query that you would use to count the IDs in the relational database. The query would be exactly as you would type it if you were entering it directly into the relational database.
*# The ''name'' is a simple label: this is how you would like to identify this ID in the final Tally Engine table.
* You can write these in any order, though existing customizations group them by ''element'', ''query'', and ''name''. For example, if your species is ''speciesname'' and you only need to count ordered locus IDs, you would add:
# Species name
speciesname_level_amount=1

speciesname_element_level0=uniprot/entry/gene/name&type&ordered locus

speciesname_query_level0=select count(*) from genenametype where type = 'ordered locus';

speciesname_query_level0=Ordered Locus
* Note how the ''element'' ends with ''name&type&ordered locus'', because the ''name'' tag in the UniProt XML file will have different types (e.g., “primary”, “ORF”, “synonym”, “ordered locus”, etc.). For ''ordered locus'' IDs, we only want to count the ''name'' IDs whose type is “ordered locus”.

Once you are done with these customizations, you can test your work by building a new version of GenMAPP Builder, connecting to a relational database that already has imported data (or importing data first if needed), then running the Tally Engine. The resulting table should include, in addition to the defaults that you have seen before, the new IDs that you have added.

=== Add the Species Profile to the Catalog of Known Species Profiles ===

The last step involves actually making GenMAPP Builder ''know'' that your new species profile exists. This involves a change in an existing file:
* Under ''edu.lmu.xmlpipedb.gmbuilder.databasetoolkit.profiles'', open ''UniProtDatabaseProfile.java''.
* Near the top of the file is a block that looks like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile() });
* What you want to do is add the species profile that you just created to this block. If your species profile is called ''MySpecialUniProtSpeciesProfile'', your modified code should look like this:
super("org.uniprot.uniprot.Uniprot",
"This profile defines the requirements "
+ "for any UniProt centric gene database.",
new SpeciesProfile[] {
new EscherichiaColiUniProtSpeciesProfile(),
new ArabidopsisThalianaUniProtSpeciesProfile(),
new PlasmodiumFalciparumUniProtSpeciesProfile(),
new VibrioCholeraeUniprotSpeciesProfile(),
new MySpecialUniProtSpeciesProfile() });
* Essentially, you need to add an item to the comma-separated list, beginning with ''new'', followed by the species profile name, finally followed by ''()''.
* Save your changes, do ''Organize Imports'' to eliminate any red errors, and try a test build!

=== Build, Test, and Possibly Commit ===

# Create a new distribution of GenMAPP Builder based on [[#Creating a Distribution|Creating a Distribution]].
# Perform a new export run with this version of GenMAPP Builder (you can skip the import steps and use the same PostgreSQL database if it’s available).
# Check the ''Systems'' table in the resulting ''.gdb'' to see if it contains the custom information:
#* Open the ''.gdb'' in Microsoft Access, then open the ''Systems'' table.
#* Look for the record for ''OrderedLocusNames''. Your species name should appear under the ''Species'' column and your link URL should appear under the ''Link'' column.
# If all goes well, commit your code as described in [[#Updating and Committing Code|Updating and Committing Code]]. You have now officially contributed to the XMLPipeDB project ''':)'''

== Common Tasks ==

The tasks in this section reflect the typical development cycle.

==== Updating and Committing Code ====

# Right-click on the ''gmbuilder'' project folder and choose ''Synchronize Repository...'' from the popup menu.
# You will be switched to the ''Team Synchronization'' perspective.
# The presence of blue-arrowed files means that the server has new updates for you to download. Right-click on the ''gmbuilder'' project folder and choose ''Update'' from the popup menu.
# It is good “developer etiquette” to build a new distribution from scratch when you’ve received updates prior to committing your own changes. Thus, after the update, return to the ''Java'' perspective, do a ''build.xml'' > ''clean'' followed by a ''build.xml'' > ''dist''.
# If everything works out, do ''Team Synchronize...'' again. If there are new updates (in the tiny amount of time since you last updated!), test things again.
# Eventually, you will see a ''Team Synchronize...'' with no incoming code. At this point, go ahead and commit the gray-arrowed files by right-clicking on them and choosing ''Commit...''.
# Just like with the wiki, it is good developer etiquette to describe briefly the nature of the changes that you are committing.
# Even if you have nothing to commit, it is still a good idea to invoke ''Team Synchronize...'' regularly so that you are kept up-to-date with regard to files that others may be committing.

==== Creating a Distribution ====

To create your own version of GenMAPP Builder based on the code you have in Eclipse (which may contain some new changes/customizations that you would like to test), follow these steps:

# Switch to Eclipse’s Java perspective.
# Edit the ''GenMAPPBuilder.java'' source code to identify the distribution that you are about to create.
# Within the ''gmbuilder'' Java project is a file called ''build.xml''. It should have an icon that appears to include an ant.
# Right click on ''build.xml'' and choose ''Run As'' > ''Ant Build...'' (the one with the ellipses) from the popup menu that appears.
# In the ''Edit Configuration'' dialog that appears, check on the ''clean'' and ''dist'' items in the ''Targets'' tab. The ''Target execution order'' section near the bottom of the dialog should say ''clean, dist''.
# Click the ''Run'' button. The computer will work for a bit.
# When it is done, right-click on the ''gmbuilder'' project folder and choose ''Refresh'' (''F5'' is its keyboard shortcut).
# You should see a ''dist'' folder appear inside the ''gmbuilder'' project folder.
# This is your personally-built copy of ''GenMAPP Builder''. Its contents correspond to the extracted contents of the ''gmbuilder-3.0.0-build-5.zip'' file that was downloaded in class.
# Run ''pgAdmin III'' and start a database, then run this copy of ''GenMAPP Builder'' as you would the “released” copy. The program should behave just like the one that you downloaded and have been using.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Quality Assurance

2015-11-01T09:12:52Z

Dondi: Initial transcription of this page.

{{Gene Database Project Links}}

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 1 ===

# (with Coders) Get a full import-export cycle done.
# (with Coders) Decide on a file/version management scheme/system.
# Learn the ID systems:
#* Systems that are the same for each species (hint: guild members help each other out by posting the relevant information on this page)
#** UniProt
#** RefSeq
#** GeneID (EntrezGene from NCBI)
#** GO
#* The OrderedLocusNames for your species

=== Milestone 2 ===

# Characterize regular expression patterns to detect the IDs ([[How Do I Count Thee? Let Me Count The Ways|for filtering then counting]]).
#* XMLPipeDB Match utility
#* Direct SQL queries in PostgreSQL
#* For example, the ''Vibrio'' IDs were of the form <code>VC####</code> or <code>VC_####</code>; how would you express that in Match or as an SQL query?
#* Table inspection/filtering/sorting in Microsoft Access
#* If needed, side-by-side sorted comparisons in Microsoft Excel ([[Using Microsoft Excel to Compare ID Lists|as described here]])
# Document/log all work done, problems encountered, and how they were resolved.

=== Milestone 3 ===

Along with your Coder, customize the Tally Engine setup for your species as specified in [[Coder#Customize_the_IDs_that_the_Tally_Engine_Counts|these coder steps]]. You will want to add, at the very least, the ''ordered locus'' IDs for your species.

=== Milestone 4 ===

# Document the relational database schema for the gene database.
# Create the ReadMe with comparisons to MOD for your species.

{{Gene Database Project Links}}

[[Category:Group Projects]]

GenMAPP User

2015-11-01T09:07:55Z

Dondi: Initial transcription of this page.

{{Gene Database Project Links}}

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

The milestones do not necessarily correspond to particular weeks; instead they are sets of tasks grouped together.

=== Milestone 1 ===
# Find and acquire the microarray paper for your team (done [[Week 10]]).
#* Provide the proper bibliographic reference for your paper.
#* Link to the HTML version of the paper on the journal website.
#* Upload the PDF version of the paper to the wiki ''(if no copyright restrictions)''; otherwise provide a download link to the publisher site.
# Download microarray data in its “rawest” form that you can find. (Consult with Dr. Dahlquist about this.)
# Verify that the gene IDs in the microarray data match the chosen species and '''''strain''''' that is being used to create the ''.gdb''.
#* This last task needs to be done in conjunction with the Quality Assurance person and Coder.
# File management: on your team's home page
#* Link to the source of the microarray data
#* Upload the microarray data files to the wiki

=== Milestone 2 ===

# Read the microarray paper to understand the experiment.
# Create a table or list that shows the correspondence between the samples in the experiment and the files you have downloaded.
# Determine how many biological or technical replicates, and which samples were labeled with Cy3 or Cy5.
# Create a Master Raw Data file that contains the IDs and columns of data required for further analysis.
# Consult with Dr. Dahlquist on how to process the data (normalization, statistics).

=== Additional Milestones ===

# Perform the statistical analysis in Excel.
# Format the gene expression data for import into GenMAPP.
# Import data into GenMAPP, create ColorSets, and run MAPPFinder.
# Document and take notes on test runs with GenMAPP.
#* Use the ''EX.txt'' file to help the Coder/Quality Assurance team members to validate the ''.gdb''.
# Do a journal club outline of the paper so that you can use it in the Discussion section of your group report and your final presentation.
# Create a ''.mapp'' file showing one pathway that is changed in your data.

{{Gene Database Project Links}}

[[Category:Group Projects]]

Project Manager

2015-11-01T09:03:52Z

Dondi: Initial transcription of this page.

{{Gene Database Project Links}}

== Guild Members ==

* Species 1:
* Species 2:
* Species 3:
* Species 4:

== Milestones ==

=== Milestone 1 ===

# In consultation with your team, work backward from the final deadline to set intermediate deadlines for each deliverable. In particular you need to set deadlines for what you will accomplish by the journal deadline for [[Week 12]], [[Week 14]], and [[Week 15]].
# Organize management tools for your team:
#* Workflow narratives
#* Action items
#* Testing results/reports
#** Bugs/Feature requests
#** Question/answer sequences

=== Milestone 2 ===
# Get periodic updates on progress; in particular, the project’s “place” in the overall flow should be known at all times (transparency). You and your team will be giving a status report in class for the rest of the semester.
# Monitor the status of the report-in-progress and other related documentation.



{{Gene Database Project Links}}

[[Category:Group Projects]]

Gene Database Project

2015-11-01T09:00:52Z

Dondi: /* Project Milestones */ Integrate overall flow diagram.

{{Gene Database Project Links}}

== Deliverables ==

* You will give a final group PowerPoint presentation in class on '''Tuesday, December 15, 2:00pm'''.
* Final due date for all other deliverables is no later than '''Friday, December 18, 4:30pm''' (according to official final exam schedule for fall, 2015)
* The deliverables should be uploaded and organized onto one group wiki page, or alternately, delivered on digital media to either Dr. Dahlquist or Dr. Dionisio.
* Each member of the group will be assigned the same grade for the group project.

=== Group Deliverables ===

* GenMAPP Gene Database for assigned species (''.gdb'')
* ReadMe file to accompany the Gene Database (''.pdf'')
** Sample ReadMe in Word format: [[Media:ReadMe_Vc-Std_External_20131122.zip | ReadMe_Vc-Std_External_20131122.zip]]
** Include Gene Database Schema diagram in ReadMe
*** Sample schema in Adobe Illustrator format: [[Media:Vibrio_schema_20101022.zip | Vibrio_schema_20101022.zip]]
* Gene Database Testing Report for final submitted Gene Database (print from wiki to ''.pdf'' file)
* Processed and analyzed DNA microarray dataset (''.xls'')
* GenMAPP Expression Dataset file (''.gex'')
* Filtered MAPPFinder Results (''.xls'')
* Sample MAPP file of a relevant biological pathway for your species (''.mapp'')
* [[Gene Database Project Deliverables#Group Report | Group Report]] describing the creation of the Gene Database and the biological analysis of the data (''.doc'' or ''.pdf'')
* PowerPoint presentation (''.ppt'', given on Tuesday, December 15)

=== Individual Deliverable ===

The individual deliverable is an [[Gene Database Project Deliverables#Individual Assessment and Reflection|assessment and reflection]] on the process (either wiki or email, depending on your preference; see note in the linked section):
* Statement of work
* Assessment of the work done
* What was learned
* Team evaluation via the [https://www.catme.org CATME] survey
*: ''You will, or should have already, received an email message from the CATME system with login instructions on your '''LMU email account''' (not necessarily the preferred email address that you listed in the wiki).''

== Team Journal Entries ==

* Each team will write a combined journal entry for each week with contributions from all members.
* [[Week 10]] Creation of page (midnight 11/10)
* [[Week 11]] (midnight 11/17)
* [[Week 12]] (midnight 11/24)
* [[Week 14]] (midnight 12/8)
* [[Week 15]] (midnight 12/15)

== Groups ==

The project groups are:
# Team 1
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:
# Team 2
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:
# Team 3
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:
# Team 4
#* Coder:
#* Quality Assurance:
#* GenMAPP User:
#* Project Manager:

== Roles (Guilds) ==

As the project moves forward, we will use class time for team meetings. In addition, we will also have guild meetings where students sharing the same role can work together on common issues. Each student has been assigned a primary role in the project by the instructors (see above).

=== [[Project Manager]] (PM) ===

The project manager makes sure that individuals are fulfilling their roles and performing the tasks on time.

=== [[GenMAPP User]] (GU) ===

The GenMAPP user runs GenMAPP with the microarray dataset (Expression Dataset Manager, MAPPFinder, GenMAPP MAPP).

=== [[Quality Assurance]] (QA) ===

The Quality Assurance team member is the resident expert on species ID systems and formats. He or she should be proficient with ''XMLPipeDB Match'', SQL queries in PostgreSQL, Microsoft Excel, and Microsoft Access to navigate through the data and find missing IDs, discrepancies, sanity checks, etc.

=== [[Coder]] (C) ===

The coder is the resident expert on the technology being used—assorted software, file management, some troubleshooting, possibly some programming. He or she coordinates with Drs. Dahlquist and Dionisio in extending GenMAPP Builder code and making new versions. GenMAPP Builder is written in Java and is built on open source pure-Java libraries. Source code is hosted on GitHub and built using Apache’s ''ant'' utility.

== Project Milestones ==

Specific project milestones are found on the individual guild pages.

=== Overall Flow ===

A successful project will have the following steps from end to end. Depending on the data and/or other issues that are encountered, one or more of these steps may go through iterations, repetitions, or refinements.

[[Image:ProjectWorkflow.png|450px|thumb|right|Overall Flow]]

# Following the steps shown in the [[Running GenMAPP Builder]] instructions, download the UniProt XML proteome set and GOA (GO association) files for your species.
#* Make sure that you are choosing the correct strain/subspecies as the microarray data you have.
#* Note the date of download and the version of the file on your Gene Database Testing Report.
# Download GO terms from [http://geneontology.org/page/download-ontology#Legacy_Downloads Gene Ontology].
#* You will need the OBO-XML format, the "obo-xml.gz" file.
#* Note the date of download and the version of the file on your Gene Database Testing Report.
# Create the GenMAPP Builder tables in PostgreSQL.
# Load files into PostgreSQL database via GenMAPP Builder.
# Export into a GenMAPP Gene Database.
# Inspect/vet/validate Gene Database.
#* Create a [[Gene Database Testing Report Sample | Gene Database Testing Report]] each time a new export is run.
#* Compare to the model organism database for your species
# Find an original journal article describing a DNA microarray experiment performed on your species.
#* Download the microarray data for this article.
#* Consult with Dr. Dahlquist about the proper steps to take to process the data (normalization, statistical analysis) and perform the analysis.
# Run GenMAPP using the Gene Database.
#* Microarray data (import using Expression Dataset Manager)
#* MAPPFinder
#* Place genes on MAPP and draw pathway

=== Guild Milestones ===

These are links to the respective milestone lists in the guild pages:
* [[Project Manager#Milestones|Project Manager]]
* [[GenMAPP User#Milestones|GenMAPP User]]
* [[Quality Assurance#Milestones|Quality Assurance]]
* [[Coder#Milestones|Coder]]

{{Gene Database Project Links}}

[[Category:Group Projects]]

File:ProjectWorkflow.png

2015-11-01T09:00:10Z

Dondi: Overall flow diagram for gene database project.

Overall flow diagram for gene database project.

Template:Gene Database Project Links

2015-11-01T07:11:12Z

Dondi: Add citation guidelines.

{| class="wikitable" style="width: 100%; text-align: center"
! colspan="8"|Gene Database Project Links
|-
! [[Gene Database Project|Overview]]
! [[Gene Database Project Deliverables|Deliverables]]
| [[Guidelines for Literature Citations in a Scientific Paper|Citation Guidelines]]
! Guilds
| [[Project Manager]]
| [[GenMAPP User]]
| [[Quality Assurance]]
| [[Coder]]
|}