Difference between revisions of "Bklein7 Week 12"

From LMU BioDB 2015
Jump to: navigation, search
(added gene database testing report)
(fixed headings)
Line 84: Line 84:
  
 
==Gene Database Testing Report==
 
==Gene Database Testing Report==
−
==Export Information==
+
===Export Information===
  
 
Version of GenMAPP Builder:
 
Version of GenMAPP Builder:
Line 118: Line 118:
 
Note:
 
Note:
  
−
==TallyEngine==
+
===TallyEngine===
  
 
* Run the TallyEngine in GenMAPP Builder and record the number of records for UniProt and GO in the XML data and in the Postgres databases.
 
* Run the TallyEngine in GenMAPP Builder and record the number of records for UniProt and GO in the XML data and in the Postgres databases.
Line 125: Line 125:
 
** For more information, [[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | see this page.]]
 
** For more information, [[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | see this page.]]
  
−
== Using XMLPipeDB match to Validate the XML Results from the TallyEngine==
+
=== Using XMLPipeDB match to Validate the XML Results from the TallyEngine===
  
 
[[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | Follow the instructions found on this page to run XMLPipeDB match.]]
 
[[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | Follow the instructions found on this page to run XMLPipeDB match.]]
Line 131: Line 131:
 
Are your results the same as you got for the TallyEngine?  Why or why not?
 
Are your results the same as you got for the TallyEngine?  Why or why not?
  
−
== Using SQL Queries to Validate the PostgreSQL Database Results from the TallyEngine==
+
=== Using SQL Queries to Validate the PostgreSQL Database Results from the TallyEngine===
  
 
For more information, [[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | see this page.]]
 
For more information, [[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | see this page.]]
Line 145: Line 145:
 
Are your results the same as reported by the TallyEngine?  Why or why not?
 
Are your results the same as reported by the TallyEngine?  Why or why not?
  
−
==OriginalRowCounts Comparison==
+
===OriginalRowCounts Comparison===
  
 
Within the .gdb file, look at the OriginalRowCounts table to see if the database has the expected tables with the expected number of records. Compare the tables and records with a benchmark .gdb file.
 
Within the .gdb file, look at the OriginalRowCounts table to see if the database has the expected tables with the expected number of records. Compare the tables and records with a benchmark .gdb file.
Line 155: Line 155:
 
Note:
 
Note:
  
−
==Visual Inspection==
+
===Visual Inspection===
  
 
Perform visual inspection of individual tables to see if there are any problems.
 
Perform visual inspection of individual tables to see if there are any problems.
Line 164: Line 164:
 
Note:
 
Note:
  
−
==.gdb Use in GenMAPP==
+
===.gdb Use in GenMAPP===
  
 
<!--Need to add more instructions here.-->
 
<!--Need to add more instructions here.-->
Line 174: Line 174:
 
Note:
 
Note:
  
−
===Putting a gene on the MAPP using the GeneFinder window===
+
====Putting a gene on the MAPP using the GeneFinder window====
  
 
* In the main GenMAPP Drafting Board window, left-click on the icon for "Gene" in the upper left corner of the window.  Click on the Drafting Board to place the Gene on the MAPP.  Now, right-click on the gene to access the GeneFinder window.  Type or paste a gene ID into the Gene ID field.  Select the appropriate Gene ID system from the drop-down menu and click the Search button.  For example, for ''Vibrio cholerae'', you could search for the ID "VC0028", which is an OrderedLocusNames ID.  Once the ID has been found, click the OK button to return to the Drafting Board window.
 
* In the main GenMAPP Drafting Board window, left-click on the icon for "Gene" in the upper left corner of the window.  Click on the Drafting Board to place the Gene on the MAPP.  Now, right-click on the gene to access the GeneFinder window.  Type or paste a gene ID into the Gene ID field.  Select the appropriate Gene ID system from the drop-down menu and click the Search button.  For example, for ''Vibrio cholerae'', you could search for the ID "VC0028", which is an OrderedLocusNames ID.  Once the ID has been found, click the OK button to return to the Drafting Board window.
Line 182: Line 182:
 
Note:
 
Note:
  
−
===Creating an Expression Dataset in the Expression Dataset Manager===
+
====Creating an Expression Dataset in the Expression Dataset Manager====
  
 
* How many of the IDs were imported out of the total IDs in the microarray dataset? How many exceptions were there? Look in the EX.txt file and look at the error codes for the records that were not imported into the Expression Dataset.  Do these represent IDs that were present in the UniProt XML, but were somehow not imported? or were they not present in the UniProt XML?
 
* How many of the IDs were imported out of the total IDs in the microarray dataset? How many exceptions were there? Look in the EX.txt file and look at the error codes for the records that were not imported into the Expression Dataset.  Do these represent IDs that were present in the UniProt XML, but were somehow not imported? or were they not present in the UniProt XML?
Line 188: Line 188:
 
Note:
 
Note:
  
−
===Coloring a MAPP with expression data===
+
====Coloring a MAPP with expression data====
  
 
Note:
 
Note:
  
−
===Running MAPPFinder===
+
====Running MAPPFinder====
  
 
Note:
 
Note:
  
−
== Compare Gene Database to Outside Resource==
+
=== Compare Gene Database to Outside Resource===
  
 
'''''Note:''''' This section applies to the Group Final Project and does not need to be completed for the [[Week 9]] assignment.  ''&mdash; [[User:Kdahlquist|Kdahlquist]] ([[User talk:Kdahlquist|talk]]) 15:46, 2 November 2015 (PST)''
 
'''''Note:''''' This section applies to the Group Final Project and does not need to be completed for the [[Week 9]] assignment.  ''&mdash; [[User:Kdahlquist|Kdahlquist]] ([[User talk:Kdahlquist|talk]]) 15:46, 2 November 2015 (PST)''

Revision as of 23:35, 19 November 2015

Files Asked for in the Gene Database Testing Report

For convenience, all of the files explicitly asked for in the "Gene Database Testing Report" section were compressed together in this file: File:Bklein7 Week9 Files.zip.

Pre-requisites

The following set of software was used in the creation and testing of the Vibrio cholerae gene database:

  1. 7-ziptool that for unpacking .gz and .zip files
  2. PostgreSQL on Windows (version 9.4.x)
  3. GenMAPP Builder
  4. Java JDK 1.8 64-bit
  5. GenMAPP 2
  6. XMLPipeDB match utility for counting IDs in XML files
  7. Microsoft Access for reading .mdb files

Gene Database Creation

Downloading Data Source Files and GenMAPP Builder

  • I download the UniProt XML, GOA, and GO OBO-XML files for Bordetella Pertussis along with the GenMAPP Builder program.
    • All files were saved to the folder Bklein7_CW on my computer's ThawSpace.
    • Files that required extraction were unzipped using 7-zip.
    • Data files that remained in a folder after unzipping were removed from their folders to facilitate organization and command line processing.

UniProt XML

GOA

GO OBO-XML

Downloaded GenMAPP Builder

  1. I downloaded the GenMAPP Builder zip folder: Download gmbuilder-3.0.0-build-5.zip.
  2. I extracted the GenMAPP Builder folder using 7-zip.

Creating the New Database in PostgreSQL

  • I launched pgAdmin III and connected to the PostgreSQL 9.4 server (localhost:5432).
    • On this server, I created a new database: bpertussis_cw20151119_gmb3build5.
    • I opened the SQL Editor tab to use an XMLPipeDB query to create the tables in the database.
      • I clicked on the Open File icon and selected the file gmbuilder.sql. This imported a series of SQL commands into the editor tab.
      • I clicked on the Execute Query icon to run this command.
      • In viewing the schema for this database, I confirmed that there were 167 tables after running the above command.

Configuring GenMAPP Builder to Connect to the PostgreSQL Database

  • To begin, I launched gmbuilder.bat.
  • I selected the "Configure Database" option and entered the following information into the fields below:
    • Host or address: localhost
    • Port number: 5432
    • Database name: bpertussis_cw20151119_gmb3build5
    • Username: postgres
    • Password: Welcome1

Importing Data into the PostgreSQL Database

  • The downloaded data files for Bordetella pertussis were specified and imported into the database by clicking on the following buttons:
    • Selected File > Import UniProt XML...
    • Selected File > Import GO OBO-XML...
    • Clicked OK to the message asking to process the GO data.
    • Selected File > Import GOA...

Exporting a GenMAPP Gene Database (.gdb)

  • I selected File > Export to GenMAPP Gene Database... to begin the export process.
  • I typed my name in the owner field (Brandon Klein).
  • I selectedVibrio cholerae as the gene database species and then clicked Next.
  • The database was saved as Vc-Std_BK_20151027.
  • I checked the boxes for exporting all Molecular Function, Cellular Component, and Biological Process Gene Ontology Terms.
  • Finally, I clicked the "Next" button to begin the export process.

Gene Database Testing Report

Export Information

Version of GenMAPP Builder:

Computer on which export was run:

Postgres Database name:

UniProt XML filename (give filename and upload and link to compressed file):

  • UniProt XML version (The version information can be found at the UniProt News Page):
  • UniProt XML download link:
  • Time taken to import:
    • Note:

GO OBO-XML filename (give filename and upload and link to compressed file):

  • GO OBO-XML version (The version information can be found in the file properties after the file downloaded from the GO Download page has been unzipped):
  • GO OBO-XML download link:
  • Time taken to import:
  • Time taken to process:
    • Note:

GOA filename (give filename and upload and link to compressed file):

  • GOA version (News on this page records past releases; current information can be found in the Last modified field on the FTP site):
  • GOA download link:
  • Time taken to import:
    • Note:

Name of .gdb file (give filename and upload and link to compressed file):

  • Time taken to export:
    • Start time:
    • End time:

Note:

TallyEngine

  • Run the TallyEngine in GenMAPP Builder and record the number of records for UniProt and GO in the XML data and in the Postgres databases.
    • Choose the menu item Tallies > Run XML and Database Tallies for UniProt and GO...
    • Take a screenshot of the results. Upload the image to the wiki and display it on this page.
    • For more information, see this page.

Using XMLPipeDB match to Validate the XML Results from the TallyEngine

Follow the instructions found on this page to run XMLPipeDB match.

Are your results the same as you got for the TallyEngine? Why or why not?

Using SQL Queries to Validate the PostgreSQL Database Results from the TallyEngine

For more information, see this page.

You can also look for counts at the SQL level, using some variation of a select count(*) query. This requires some knowledge of which table received what data. Here’s an initial tip: the gene/name tags in the XML file land in the genenametype table. A query on this table counting values from this table that were marked as ordered locus in the XML file matching the pattern VC_[0-9][0-9][0-9][0-9] would look like this: 

select count(*) from genenametype where type = 'ordered locus' and value ~ 'VC_[0-9][0-9][0-9][0-9]';

In pgAdmin III, you can issue these queries by clicking on the pencil/SQL icon in the toolbar, typing the query into the SQL Editor tab, then clicking on the green triangular Play button to run.

Pgadminiii-query.png

Are your results the same as reported by the TallyEngine? Why or why not?

OriginalRowCounts Comparison

Within the .gdb file, look at the OriginalRowCounts table to see if the database has the expected tables with the expected number of records. Compare the tables and records with a benchmark .gdb file.

Benchmark .gdb file:

Copy the OriginalRowCounts table from the benchmark and new gdb and paste them here:

Note:

Visual Inspection

Perform visual inspection of individual tables to see if there are any problems.

  • Look at the Systems table. Is there a date in the Date field for all gene ID systems present in the database?
  • Open the UniProt, RefSeq, and OrderedLocusNames tables. Scroll down through the table. Do all of the IDs look like they take the correct form for that type of ID?

Note:

.gdb Use in GenMAPP

While the above sections perform quality assurance on the exported Gene Database via verifying ID counts, the "proof in the pudding" is to actually use the Gene Database in GenMAPP. You can follow the instructions in Part 2 of the Vibrio cholerae Microarray Data Analysis to verify that the Gene Database works in GenMAPP. In this case, the emphasis is not on the findings of the data analysis itself, but that the Gene Database functions appropriate in GenMAPP.

For assistance with using the GenMAPP program, the GenMAPP Help is very extensive. To access it within GenMAPP, go to the menu item Help > GenMAPP Help and either browse or search for your topic of interest.

Note:

Putting a gene on the MAPP using the GeneFinder window

  • In the main GenMAPP Drafting Board window, left-click on the icon for "Gene" in the upper left corner of the window. Click on the Drafting Board to place the Gene on the MAPP. Now, right-click on the gene to access the GeneFinder window. Type or paste a gene ID into the Gene ID field. Select the appropriate Gene ID system from the drop-down menu and click the Search button. For example, for Vibrio cholerae, you could search for the ID "VC0028", which is an OrderedLocusNames ID. Once the ID has been found, click the OK button to return to the Drafting Board window.
    • For the Final Project, you will need to try a sample ID from each of the gene ID systems, not just OrderedLocusNames.
  • Open the Backpage by left-clicking on the gene box on the Drafting Board to see if all of the cross-referenced IDs that are supposed to be there are there.

Note:

Creating an Expression Dataset in the Expression Dataset Manager

  • How many of the IDs were imported out of the total IDs in the microarray dataset? How many exceptions were there? Look in the EX.txt file and look at the error codes for the records that were not imported into the Expression Dataset. Do these represent IDs that were present in the UniProt XML, but were somehow not imported? or were they not present in the UniProt XML?

Note:

Coloring a MAPP with expression data

Note:

Running MAPPFinder

Note:

Compare Gene Database to Outside Resource

Note: This section applies to the Group Final Project and does not need to be completed for the Week 9 assignment. — Kdahlquist (talk) 15:46, 2 November 2015 (PST)

The OrderedLocusNames IDs in the exported Gene Database are derived from the UniProt XML. It is a good idea to check your list of OrderedLocusNames IDs to see how complete it is using the original source of the data (the sequencing organization, the MOD, etc.) Because UniProt is a protein database, it does not reference any non-protein genome features such as genes that code for functional RNAs, centromeres, telomeres, etc.

Note:

Links

Assignments Pages

Individual Journal Entries

Shared Journal Entries

Retrieved from "https://xmlpipedb.lmucs.io/biodb/fall2015/index.php?title=Bklein7_Week_12&oldid=6549"