Difference between revisions of "Bklein7 Week 12"

From LMU BioDB 2015
Jump to: navigation, search
(added export details)
(updated gene database testing report)
Line 123: Line 123:
 
** End time: 4:46 PM
 
** End time: 4:46 PM
 
** Elapsed time: 40 minutes
 
** Elapsed time: 40 minutes
−
Note: All export windows remained open when I returned to check the export status. Therefore, no interruptions occurred during the export process.
+
Note: All export windows remained open when I returned to check the export status. No interruptions occurred during the export process.
  
 
===TallyEngine===
 
===TallyEngine===
  
−
* Run the TallyEngine in GenMAPP Builder and record the number of records for UniProt and GO in the XML data and in the Postgres databases.
+
* I ran the TallyEngine in GenMAPP Builder and specified the following files:
−
** Choose the menu item Tallies > Run XML and Database Tallies for UniProt and GO...
+
**XML- uniprot-proteome%3AUP000002676_cw20151119.xml
−
** Take a screenshot of the results.  Upload the image to the wiki and display it on this page.
+
**GO- go_daily-termdb_cw20151119.obo-xml
−
** For more information, [[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | see this page.]]
+
*Results:
−
 
+
**[[File:TallyEngine cw20151119.png]]
 +
**All tally results were consistent across both files.
 
=== Using XMLPipeDB match to Validate the XML Results from the TallyEngine===
 
=== Using XMLPipeDB match to Validate the XML Results from the TallyEngine===
−
 
+
The following functions were performed using the Windows command line (cmd).
−
[[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | Follow the instructions found on this page to run XMLPipeDB match.]]
+
*I entered my project folder using the following command:
−
 
+
cd /d T:\Bklein7_CW
−
Are your results the same as you got for the TallyEngine?  Why or why not?
+
*I used XMLPipeDB match to identify matches of any ordered locus name following the pattern "BP####" in the UniProt XML file. The command sequence used is as follows:\
 +
java -jar xmlpipedb-match-1.1.1.jar "BP[0-9][0-9][0-9][0-9]" < "uniprot-proteome%3AUP000002676_cw20151119.xml"
 +
*Match Results:
 +
**[[File:XMLPipeDBMatch cw20151119.png]]
 +
**The total number of unique matches listed above, 3438, differs from the Order Locus Names count of 3435 produced by the Tally Engine. Thus, 3 gene IDs present in the original XML file were not imported into ''bpertussis-std_cw20151119.gdb''.
  
 
=== Using SQL Queries to Validate the PostgreSQL Database Results from the TallyEngine===
 
=== Using SQL Queries to Validate the PostgreSQL Database Results from the TallyEngine===
 +
I ran a SQL query designed to match the pattern BP####:
  
−
For more information, [[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | see this page.]]
+
  select count (*) from genenametype where type = 'ordered locus' and value ~ 'BP[0-9][0-9][0-9][0-9]';
−
 
+
−
You can also look for counts at the SQL level, using some variation of a ''select count(*)'' query.  This requires some knowledge of which table received what data.  Here&rsquo;s an initial tip: the ''gene/name'' tags in the XML file land in the ''genenametype'' table.  A query on this table counting values from this table that were marked as ''ordered locus'' in the XML file matching the pattern ''VC_[0-9][0-9][0-9][0-9]'' would look like this:
+
−
 
+
−
  select count(*) from genenametype where type = 'ordered locus' and value ~ 'VC_[0-9][0-9][0-9][0-9]';
+
−
 
+
−
In ''pgAdmin III'', you can issue these queries by clicking on the pencil/SQL icon in the toolbar, typing the query into the ''SQL Editor'' tab, then clicking on the green triangular ''Play'' button to run.
+
−
 
+
−
[[Image:Pgadminiii-query.png]]
+
  
−
Are your results the same as reported by the TallyEngine?  Why or why not?
+
Results:
 +
*[[File:SQLQuery cw20151119.png]]
 +
* The number of unique matches yielded by this SQL query, 3435, matched that produced by the Tally Engine. However, this count was also 3 less than that yielded by the XMLPipeDB Match result reported above. This further indicates that there was an error present in importing all gene IDs from the original XML file.
  
 
===OriginalRowCounts Comparison===
 
===OriginalRowCounts Comparison===

Revision as of 23:09, 23 November 2015

TO DO

  • Check Export!
  • ZIP and upload files used in creating the GDB
  • Create a new page for the gene testing report (_cw20151119) and link to it from this page
  • Follow instructions on the Coder page and update electronic lab notebook
    • Consider dividing this page into coder and QA sections

Files Asked for in the Gene Database Testing Report

For convenience, all of the files explicitly asked for in the "Gene Database Testing Report" section were compressed together in this file:

Pre-requisites

The following set of software was used in the creation and testing of the Vibrio cholerae gene database:

  1. 7-ziptool that for unpacking .gz and .zip files
  2. PostgreSQL on Windows (version 9.4.x)
  3. GenMAPP Builder
  4. Java JDK 1.8 64-bit
  5. GenMAPP 2
  6. XMLPipeDB match utility for counting IDs in XML files
  7. Microsoft Access for reading .mdb files

Gene Database Creation

Downloading Data Source Files and GenMAPP Builder

  • I download the UniProt XML, GOA, and GO OBO-XML files for Bordetella Pertussis along with the GenMAPP Builder program.
    • All files were saved to the folder Bklein7_CW on my computer's ThawSpace.
    • Files that required extraction were unzipped using 7-zip.
    • Data files that remained in a folder after unzipping were removed from their folders to facilitate organization and command line processing.

UniProt XML

GOA

GO OBO-XML

Downloaded GenMAPP Builder

  1. I downloaded the GenMAPP Builder zip folder: Download gmbuilder-3.0.0-build-5.zip.
  2. I extracted the GenMAPP Builder folder using 7-zip.

Creating the New Database in PostgreSQL

  • I launched pgAdmin III and connected to the PostgreSQL 9.4 server (localhost:5432).
    • On this server, I created a new database: bpertussis_cw20151119_gmb3build5.
    • I opened the SQL Editor tab to use an XMLPipeDB query to create the tables in the database.
      • I clicked on the Open File icon and selected the file gmbuilder.sql. This imported a series of SQL commands into the editor tab.
      • I clicked on the Execute Query icon to run this command.
      • In viewing the schema for this database, I confirmed that there were 167 tables after running the above command.

Configuring GenMAPP Builder to Connect to the PostgreSQL Database

  • To begin, I launched gmbuilder.bat.
  • I selected the "Configure Database" option and entered the following information into the fields below:
    • Host or address: localhost
    • Port number: 5432
    • Database name: bpertussis_cw20151119_gmb3build5
    • Username: postgres
    • Password: Welcome1

Importing Data into the PostgreSQL Database

  • The downloaded data files for Bordetella pertussis were specified and imported into the database by clicking on the following buttons:
    • Selected File > Import UniProt XML...
    • Selected File > Import GO OBO-XML...
    • Clicked OK to the message asking to process the GO data.
    • Selected File > Import GOA...

Exporting a GenMAPP Gene Database (.gdb)

  • I selected File > Export to GenMAPP Gene Database... to begin the export process.
  • I typed my name in the owner field (Brandon Klein).
  • I selected "Bordetella pertussis (strain Tohama I/ATCC BAA-589/NCTC 13251), Taxon ID 257313" as the gene database species and then clicked Next.
  • The database was saved as bpertussis-std_cw20151119.
  • I checked the boxes for exporting all Molecular Function, Cellular Component, and Biological Process Gene Ontology Terms.
  • Finally, I clicked the "Next" button to begin the export process.

Gene Database Testing Report

Export Information

Version of GenMAPP Builder: Version 3.0.0 Build 5

Computer on which export was run: Seaver 120- Last computer on the right in the row farthest from the front of the room

Postgres Database name: bpertussis_cw20151119_gmb3build5

UniProt XML filename (give filename and upload and link to compressed file):

  • UniProt XML version (The version information can be found at the UniProt News Page):
  • UniProt XML download link:
  • Time taken to import: 2.60 minutes
    • Note: The import time was similar to that of V. cholerae in Week 9 (2.92 minutes). No interruptions occurred during this process.

GO OBO-XML filename (give filename and upload and link to compressed file):

  • GO OBO-XML version (The version information can be found in the file properties after the file downloaded from the GO Download page has been unzipped):
  • GO OBO-XML download link:
  • Time taken to import: 6.99 minutes
  • Time taken to process: 4.48 minutes
    • Note: The import and processing times were similar to those for V. cholerae in Week 9 (6.88 minutes and 4.49 minutes respectively). No interruptions occurred during these processes.

GOA filename (give filename and upload and link to compressed file):

  • GOA version (News on this page records past releases; current information can be found in the Last modified field on the FTP site):
  • GOA download link:
  • Time taken to import: 0.04 minutes
    • Note: The import time was similar to that of V. cholerae in Week 9 (0.06 minutes). No interruptions occurred during this process.

Name of .gdb file (give filename and upload and link to compressed file):

  • Time taken to export:
    • Start time: 4:06 PM
    • End time: 4:46 PM
    • Elapsed time: 40 minutes

Note: All export windows remained open when I returned to check the export status. No interruptions occurred during the export process.

TallyEngine

  • I ran the TallyEngine in GenMAPP Builder and specified the following files:
    • XML- uniprot-proteome%3AUP000002676_cw20151119.xml
    • GO- go_daily-termdb_cw20151119.obo-xml
  • Results:
    • TallyEngine cw20151119.png
    • All tally results were consistent across both files.

Using XMLPipeDB match to Validate the XML Results from the TallyEngine

The following functions were performed using the Windows command line (cmd).

  • I entered my project folder using the following command:
cd /d T:\Bklein7_CW
  • I used XMLPipeDB match to identify matches of any ordered locus name following the pattern "BP####" in the UniProt XML file. The command sequence used is as follows:\
java -jar xmlpipedb-match-1.1.1.jar "BP[0-9][0-9][0-9][0-9]" < "uniprot-proteome%3AUP000002676_cw20151119.xml"
  • Match Results:
    • XMLPipeDBMatch cw20151119.png
    • The total number of unique matches listed above, 3438, differs from the Order Locus Names count of 3435 produced by the Tally Engine. Thus, 3 gene IDs present in the original XML file were not imported into bpertussis-std_cw20151119.gdb.

Using SQL Queries to Validate the PostgreSQL Database Results from the TallyEngine

I ran a SQL query designed to match the pattern BP####: 

select count (*) from genenametype where type = 'ordered locus' and value ~ 'BP[0-9][0-9][0-9][0-9]';

Results:

  • SQLQuery cw20151119.png
  • The number of unique matches yielded by this SQL query, 3435, matched that produced by the Tally Engine. However, this count was also 3 less than that yielded by the XMLPipeDB Match result reported above. This further indicates that there was an error present in importing all gene IDs from the original XML file.

OriginalRowCounts Comparison

Within the .gdb file, look at the OriginalRowCounts table to see if the database has the expected tables with the expected number of records. Compare the tables and records with a benchmark .gdb file.

Benchmark .gdb file:

Copy the OriginalRowCounts table from the benchmark and new gdb and paste them here:

Note:

Visual Inspection

Perform visual inspection of individual tables to see if there are any problems.

  • Look at the Systems table. Is there a date in the Date field for all gene ID systems present in the database?
  • Open the UniProt, RefSeq, and OrderedLocusNames tables. Scroll down through the table. Do all of the IDs look like they take the correct form for that type of ID?

Note:

.gdb Use in GenMAPP

While the above sections perform quality assurance on the exported Gene Database via verifying ID counts, the "proof in the pudding" is to actually use the Gene Database in GenMAPP. You can follow the instructions in Part 2 of the Vibrio cholerae Microarray Data Analysis to verify that the Gene Database works in GenMAPP. In this case, the emphasis is not on the findings of the data analysis itself, but that the Gene Database functions appropriate in GenMAPP.

For assistance with using the GenMAPP program, the GenMAPP Help is very extensive. To access it within GenMAPP, go to the menu item Help > GenMAPP Help and either browse or search for your topic of interest.

Note:

Putting a gene on the MAPP using the GeneFinder window

  • In the main GenMAPP Drafting Board window, left-click on the icon for "Gene" in the upper left corner of the window. Click on the Drafting Board to place the Gene on the MAPP. Now, right-click on the gene to access the GeneFinder window. Type or paste a gene ID into the Gene ID field. Select the appropriate Gene ID system from the drop-down menu and click the Search button. For example, for Vibrio cholerae, you could search for the ID "VC0028", which is an OrderedLocusNames ID. Once the ID has been found, click the OK button to return to the Drafting Board window.
    • For the Final Project, you will need to try a sample ID from each of the gene ID systems, not just OrderedLocusNames.
  • Open the Backpage by left-clicking on the gene box on the Drafting Board to see if all of the cross-referenced IDs that are supposed to be there are there.

Note:

Creating an Expression Dataset in the Expression Dataset Manager

  • How many of the IDs were imported out of the total IDs in the microarray dataset? How many exceptions were there? Look in the EX.txt file and look at the error codes for the records that were not imported into the Expression Dataset. Do these represent IDs that were present in the UniProt XML, but were somehow not imported? or were they not present in the UniProt XML?

Note:

Coloring a MAPP with expression data

Note:

Running MAPPFinder

Note:

Compare Gene Database to Outside Resource

Note: This section applies to the Group Final Project and does not need to be completed for the Week 9 assignment. — Kdahlquist (talk) 15:46, 2 November 2015 (PST)

The OrderedLocusNames IDs in the exported Gene Database are derived from the UniProt XML. It is a good idea to check your list of OrderedLocusNames IDs to see how complete it is using the original source of the data (the sequencing organization, the MOD, etc.) Because UniProt is a protein database, it does not reference any non-protein genome features such as genes that code for functional RNAs, centromeres, telomeres, etc.

Note:

Links

Assignments Pages

Individual Journal Entries

Shared Journal Entries

Retrieved from "https://xmlpipedb.lmucs.io/biodb/fall2015/index.php?title=Bklein7_Week_12&oldid=6774"