Gene Database Testing Report- cw20151119

Files Asked for in the Gene Database Testing Report

For convenience, all of the files explicitly asked for in the "Gene Database Testing Report" section were compressed together in this file:

Pre-requisites

The following set of software was used in the creation and testing of the Vibrio cholerae gene database:

1. 7-ziptool that for unpacking .gz and .zip files
2. PostgreSQL on Windows (version 9.4.x)
3. GenMAPP Builder
4. Java JDK 1.8 64-bit
5. GenMAPP 2
6. XMLPipeDB match utility for counting IDs in XML files
7. Microsoft Access for reading .mdb files

Gene Database Creation

Downloading Data Source Files and GenMAPP Builder

• I download the UniProt XML, GOA, and GO OBO-XML files for Bordetella Pertussis along with the GenMAPP Builder program.
  • All files were saved to the folder Bklein7_CW on my computer's ThawSpace.
  • Files that required extraction were unzipped using 7-zip.
  • Data files that remained in a folder after unzipping were removed from their folders to facilitate organization and command line processing.

UniProt XML

• I went to the UniProt Complete Proteomes page.
  • From there, I navigated to the complete proteome download page for Bordetella pertussis (strain Tohama I / ATCC BAA-589 / NCTC 13251).
  • I clicked on the "Download" button at the top of the page above and selected the following options:
    • "Download all"
    • "XML" from the "Format" drop-down menu
    • "Compressed" format
  • I extracted the file using 7-zip.

GOA

• UniProt-GOA files can be downloaded from the UniProt-GOA ftp site.
• Within the above site, I navigated to the for Bordetella pertussis strain Tohama I.
  • This text file was automatically opened by my browser. Therefore, I had to manually download the file.

GO OBO-XML

• I downloaded the GO OBO-XML formatted file from the Gene Ontology legacy download page.
• I extracted the file using 7-zip.

Downloaded GenMAPP Builder

1. I downloaded the GenMAPP Builder zip folder: Download gmbuilder-3.0.0-build-5.zip.
2. I extracted the GenMAPP Builder folder using 7-zip.

Creating the New Database in PostgreSQL

• I launched pgAdmin III and connected to the PostgreSQL 9.4 server (localhost:5432).
  • On this server, I created a new database: bpertussis_cw20151119_gmb3build5.
  • I opened the SQL Editor tab to use an XMLPipeDB query to create the tables in the database.
    • I clicked on the Open File icon and selected the file gmbuilder.sql. This imported a series of SQL commands into the editor tab.
    • I clicked on the Execute Query icon to run this command.
    • In viewing the schema for this database, I confirmed that there were 167 tables after running the above command.

Configuring GenMAPP Builder to Connect to the PostgreSQL Database

• To begin, I launched gmbuilder.bat.
• I selected the "Configure Database" option and entered the following information into the fields below:
  • Host or address: localhost
  • Port number: 5432
  • Database name: bpertussis_cw20151119_gmb3build5
  • Username: postgres
  • Password: Welcome1

Importing Data into the PostgreSQL Database

• The downloaded data files for Bordetella pertussis were specified and imported into the database by clicking on the following buttons:
  • Selected File > Import UniProt XML...
  • Selected File > Import GO OBO-XML...
  • Clicked OK to the message asking to process the GO data.
  • Selected File > Import GOA...

Exporting a GenMAPP Gene Database (.gdb)

• I selected File > Export to GenMAPP Gene Database... to begin the export process.
• I typed my name in the owner field (Brandon Klein).
• I selected "Bordetella pertussis (strain Tohama I/ATCC BAA-589/NCTC 13251), Taxon ID 257313" as the gene database species and then clicked Next.
• The database was saved as bpertussis-std_cw20151119.
• I checked the boxes for exporting all Molecular Function, Cellular Component, and Biological Process Gene Ontology Terms.
• Finally, I clicked the "Next" button to begin the export process.

Gene Database Testing Report

Export Information

Version of GenMAPP Builder: Version 3.0.0 Build 5

Computer on which export was run: Seaver 120- Last computer on the right in the row farthest from the front of the room

Postgres Database name: bpertussis_cw20151119_gmb3build5

UniProt XML filename (give filename and upload and link to compressed file):

• UniProt XML version (The version information can be found at the UniProt News Page):
• UniProt XML download link:
• Time taken to import: 2.60 minutes
  • Note: The import time was similar to that of V. cholerae in Week 9 (2.92 minutes). No interruptions occurred during this process.

GO OBO-XML filename (give filename and upload and link to compressed file):

• GO OBO-XML version (The version information can be found in the file properties after the file downloaded from the GO Download page has been unzipped):
• GO OBO-XML download link:
• Time taken to import: 6.99 minutes
• Time taken to process: 4.48 minutes
  • Note: The import and processing times were similar to those for V. cholerae in Week 9 (6.88 minutes and 4.49 minutes respectively). No interruptions occurred during these processes.

GOA filename (give filename and upload and link to compressed file):

• GOA version (News on this page records past releases; current information can be found in the Last modified field on the FTP site):
• GOA download link:
• Time taken to import: 0.04 minutes
  • Note: The import time was similar to that of V. cholerae in Week 9 (0.06 minutes). No interruptions occurred during this process.

Name of .gdb file (give filename and upload and link to compressed file):

• Time taken to export:
  • Start time: 4:06 PM
  • End time: 4:46 PM
  • Elapsed time: 40 minutes

Note: All export windows remained open when I returned to check the export status. No interruptions occurred during the export process.

TallyEngine

• I ran the TallyEngine in GenMAPP Builder and specified the following files:
  • XML- uniprot-proteome%3AUP000002676_cw20151119.xml
  • GO- go_daily-termdb_cw20151119.obo-xml
• Results:
  • 
  • All tally results were consistent across both files.

Using XMLPipeDB match to Validate the XML Results from the TallyEngine

The following functions were performed using the Windows command line (cmd).

• I entered my project folder using the following command:

cd /d T:\Bklein7_CW

• I used XMLPipeDB match to identify matches of any ordered locus name following the pattern "BP####" in the UniProt XML file. The command sequence used is as follows:\

java -jar xmlpipedb-match-1.1.1.jar "BP[0-9][0-9][0-9][0-9]" < "uniprot-proteome%3AUP000002676_cw20151119.xml"

• Match Results:
  • 
  • The total number of unique matches listed above, 3438, differs from the Order Locus Names count of 3435 produced by the Tally Engine. Thus, 3 gene IDs present in the original XML file were not imported into bpertussis-std_cw20151119.gdb.

Using SQL Queries to Validate the PostgreSQL Database Results from the TallyEngine

I ran a SQL query designed to match the pattern BP####:

select count (*) from genenametype where type = 'ordered locus' and value ~ 'BP[0-9][0-9][0-9][0-9]';

Results:

• 
• The number of unique matches yielded by this SQL query, 3435, matched that produced by the Tally Engine. However, this count was also 3 less than that yielded by the XMLPipeDB Match result reported above. This further indicates that there was an error present in importing all gene IDs from the original XML file.

OriginalRowCounts Comparison

I opened the gene database file bpertussis-std_cw20151119 in Microsoft Access and assessed the "OriginalRowCounts" table to see if the expected tables were listed with the expected number of records. The contents of this table were compared to the OriginalRowCounts table of an existing .gdb file created during Week 9.

Benchmark .gdb file: Vc-Std_20151027_TR

"OriginalRowCounts" table from the benchmark and new gdb:

• 
• All 52 tables present in the 2015 Vibrio cholerae database were also present in the B. pertussis database tagged _cw20151119. This confirmed that all expected tables were successfully created.
• Further, the "OrderedLocusNames" count of 3435 generated by the Tally Engine was confirmed to reflect that actual contents of the database.

Note: The "OriginalRowCounts" tables were too large to screenshot. To circumvent this problem and facilitate the comparison, I copied the "OriginalRowCounts" tables from both gene databases into an Excel file and zoomed out. The above screenshot was taken from this Excel file. The "OrderedLocusNames" row counts are highlighted in yellow.

Visual Inspection

I visually inspected individual tables within bpertussis-std_cw20151119 using Microsoft Access.

• Systems Table
  • 35 gene ID systems were listed, 11 of which listed the appropriate import date (11/19/2015)
    • All gene ID systems relevant to B. pertussis were listed. This includes: EMBL, EnsemblBacteria, GeneID, GeneOntology, InterPro, OrderedLocusNames, Pfam, RefSeq, and UniProt.
    • This result corresponded with that of the benchmark .gdb file listed in the "OriginalRowCounts Comparison" section.
• UniProt Table
  • This table contained 3258 entries with 6 character IDs.
  • All ID's in the UniProt table conform to the following pattern:
  • There are no apparent issues with the 3258 entries. However, it is curious that only 3258 out of 3438 IDs identified through XMLPipeDB Match made it into the database. This suggests the need for gmbuilder coding changes.
• RefSeq Table
  • This table contained 6627 entries. All IDs began with one of three prefixes: "NP_", "YP_", or "WP_". The meanings of these prefixes can be found in the RefSeq documentation found here.
    • "NP_" and "YP_" Prefixes
      • Refer to proteins. There are 3410 "NP_" IDs and 7 "YP_" IDs.
    • "WP_" Prefixes
      • Refer to "autonomous non-redundant proteins that are not yet directly annotated on a genome". There were 3210 IDs with the "WP_" prefixes.
    • Overall, every entry in the ID column was an expected value.
• OrderedLocusNames Table
  • This table contained 3435 entries (consistent with Tally Engine & SQL counts).
  • The IDs were copied into an Excel document for analysis:
    • 3434 IDs conformed to the pattern "BP####"
    • 1 ID was unique: "BP3167.1"

Note:

.gdb Use in GenMAPP

• By following the instructions in Part 2 of the Vibrio cholerae Microarray Data Analysis and looking at Brandon's Week 9 individual journal assignment, I was able to verify that our Gene Database works in GenMAPP.
• I was able to open the GenMAPP program on the computer, and then I went to Data -> Choose Gene Database -> and selected the cw20151119 gdb file.
  • There were no problems thus far as our database was able to load into the program.

Putting a gene on the MAPP using the GeneFinder window

• In the main GenMAPP Drafting Board window, I left-clicked on the icon for "Gene" in the upper left corner of the window.
• I clicked on the Drafting Board to place the Gene on the MAPP.
• Then I right-clicked on the gene to access the GeneFinder window.
• T typed in the gene Id BP3723 into the Gene ID field and selected "OrderedLocusName" for the Gene ID system.
• When the gene was found this is the back page that popped up:

• • Not all of the expected cross-referenced IDs were present.
• In the main GenMAPP Drafting Board window, I left-clicked on the icon for "Gene" in the upper left corner of the window.
• I clicked on the Drafting Board to place the Gene on the MAPP.
• Then I right-clicked on the gene to access the GeneFinder window.
• T typed in the gene Id BP2980 into the Gene ID field and selected "OrderedLocusName" for the Gene ID system.
• When the gene was found this is the back page that popped up:

• • Not all of the expected cross-referenced IDs were present.

• • For the Final Project, you will need to try a sample ID from each of the gene ID systems, not just OrderedLocusNames.
• Open the Backpage by left-clicking on the gene box on the Drafting Board to see if all of the cross-referenced IDs that are supposed to be there are there.

Note:

Creating an Expression Dataset in the Expression Dataset Manager

• How many of the IDs were imported out of the total IDs in the microarray dataset? How many exceptions were there? Look in the EX.txt file and look at the error codes for the records that were not imported into the Expression Dataset. Do these represent IDs that were present in the UniProt XML, but were somehow not imported? or were they not present in the UniProt XML?

Note:

Coloring a MAPP with expression data

Note:

Running MAPPFinder

Note:

Compare Gene Database to Outside Resource

Note: This section applies to the Group Final Project and does not need to be completed for the Week 9 assignment. — Kdahlquist (talk) 15:46, 2 November 2015 (PST)

The OrderedLocusNames IDs in the exported Gene Database are derived from the UniProt XML. It is a good idea to check your list of OrderedLocusNames IDs to see how complete it is using the original source of the data (the sequencing organization, the MOD, etc.) Because UniProt is a protein database, it does not reference any non-protein genome features such as genes that code for functional RNAs, centromeres, telomeres, etc.

Note: