Difference between revisions of "Blitvak Week 9"

Revision as of 23:17, 31 October 2015

Initial Preparations

List of required programs was initially sourced from the Running GenMAPP Builder page

One of the final goals is to perform an export of a Vibrio cholerae GenMAPP Gene Database

In preparation for this assignment, it was ensured that these programs were installed on a Windows workstation:

• 7-zip for the unpacking of compressed files
• PostgreSQL
• GenMAPP Builder
• Java JDK 1.8 64-bit (download jdk-8u65-windows-x64.exe)
• GenMAPP 2 (download GenMAPPv2Setup.exe)
• XMLPipeDB match utility
• A program that is able to read .mdb files (such as Microsoft Access)

Downloading the Required Files

List of required files was initially sourced from the Running GenMAPP Builder page

Retrieving the UniProt XML file, Performed on 10/27

• The UniProt Complete Proteomes page was entered
• The Superkingdom Bacteria was selected and Reference proteome was clicked on (72 results currently)
• "vibrio cholerae" was added to the search bar and search was clicked upon. The results were filtered to the only proteome of interest (Vibrio cholerae serotype O1 (strain ATCC 39315 / El Tor Inaba N16961))
• The result was clicked upon and, on the result page, UniProtKB was clicked upon in the "Map to" section (on left of the page)
• On the UniProtKB results page, Download was clicked; in the box that appeared, download all was selected, the format was set to XML, and the file was set to be compressed.

Retrieving the GOA file, Performed on 10/27

• The UniProt-GOA ftp site was entered
• The link to the "proteomes" directory was clicked in the main directory
• In "proteomes", the file V_cholerae_ATCC_39315.goa was searched for, found, and downloaded by using right-click with "Save link as"

Retrieving the GO OBO-XML file, Performed on 10/27

• The GO OBO-XML file was downloaded from the Gene Ontology download page
• "obo-xml.gz" was selected/clicked under Legacy Downloads

Downloading/Updating GenMAPP Builder,Performed on 10/27

• The files were downloaded from the XMLPipeDB releases page on GitHub
• Download link for gmbuilder-3.0.0-build-5 (the version used in this assignment)
  • All of the downloaded files, if compressed, were extracted using 7-Zip. All required files were placed in one folder

Gene Database Testing Report

Export Information

Version of GenMAPP Builder: 3.0.0-build-5

Computer on which export was run: Workstation in Seaver 120

Postgres Database name: V.cholerae_20151027_gmb3build5

UniProt XML filename: compressed uniprot-organism%3A243277_BL_20151027.xml

• UniProt XML version (found out using the UniProt News Page: UniProt release 2015_10
• UniProt XML download link: http://www.uniprot.org/uniprot/?query=organism:243277#
• Time taken to import: 2.99 minutes
  • Note: Import times were as expected; nothing unusual occurred during the import process.

GO OBO-XML filename: compressed BL_20151027_go_daily-termdb.obo-xml

• GO OBO-XML version (derived from the date modified on the file, itself): Date modified - 10/27/2015 2:24 AM
• GO OBO-XML download link: GO website
• Time taken to import: 6.98 minutes
• Time taken to process: 4.54 minutes
  • Note: Import times were as expected; nothing unusual occurred during the import process.

GOA filename: compressed 46.V_cholerae_ATCC_39315_BL_20151027.goa

• GOA version (taken from here: Last Update - 13-Oct-2015 07:31
• GOA download link: link
• Time taken to import: 0.06
  • Note: Import times were as expected; nothing unusual occurred during the import process.

Name of .gdb file: compressed Vc-Std_20151027_BL.gdb

• Time taken to export: one hour and nineteen minutes
  • Start time: 3:56
  • End time: 5:15

Note: Export times were as expected; nothing unusual occurred during the export process. Program windows, however, were closed by the time the export product was observed; the time taken to export was found out by observing the time on the Date Modified part of the produced file. The found end time was in line with the times found by others (ranged around an hour and 15 minutes)

TallyEngine

• Run the TallyEngine in GenMAPP Builder and record the number of records for UniProt and GO in the XML data and in the Postgres databases.
  • Choose the menu item Tallies > Run XML and Database Tallies for UniProt and GO...
  • Take a screenshot of the results. Upload the image to the wiki and display it on this page.

• • For more information, see this page.

Using XMLPipeDB match to Validate the XML Results from the TallyEngine

Follow the instructions found on this page to run XMLPipeDB match.

Are your results the same as you got for the TallyEngine? Why or why not?

• 2738 unique matches were found through XMLPipeDB match

Using SQL Queries to Validate the PostgreSQL Database Results from the TallyEngine

For more information, see this page.

You can also look for counts at the SQL level, using some variation of a select count(*) query. This requires some knowledge of which table received what data. Here’s an initial tip: the gene/name tags in the XML file land in the genenametype table. A query on this table counting values from this table that were marked as ordered locus in the XML file matching the pattern VC_[0-9][0-9][0-9][0-9] would look like this:

select count(*) from genenametype where type = 'ordered locus' and value ~ 'VC_[0-9][0-9][0-9][0-9]';

In pgAdmin III, you can issue these queries by clicking on the pencil/SQL icon in the toolbar, typing the query into the SQL Editor tab, then clicking on the green triangular Play button to run.

Are your results the same as reported by the TallyEngine? Why or why not?

• 2737 unique matches were found in pgAdmin III (postgres db)

OriginalRowCounts Comparison

Within the .gdb file, look at the OriginalRowCounts table to see if the database has the expected tables with the expected number of records. Compare the tables and records with a benchmark .gdb file.

Benchmark .gdb file:

7664 unique matches found using Microsoft Access

Copy the OriginalRowCounts table from the benchmark and new gdb and paste them here:

Note: Some IDs have underscores, some have four numbers, and some have the letter A

Visual Inspection

Perform visual inspection of individual tables to see if there are any problems.

• Look at the Systems table. Is there a date in the Date field for all gene ID systems present in the database?
• Open the UniProt, RefSeq, and OrderedLocusNames tables. Scroll down through the table. Do all of the IDs look like they take the correct form for that type of ID?

Note: select count(*) from genenametype where type = 'ordered locus' and value ~ 'VC_A?[0-9][0-9][0-9][0-9]';

.gdb Use in GenMAPP

Note:

Putting a gene on the MAPP using the GeneFinder window

• Try a sample ID from each of the gene ID systems. Open the Backpage and see if all of the cross-referenced IDs that are supposed to be there are there.

Note:

Creating an Expression Dataset in the Expression Dataset Manager

• How many of the IDs were imported out of the total IDs in the microarray dataset? How many exceptions were there? Look in the EX.txt file and look at the error codes for the records that were not imported into the Expression Dataset. Do these represent IDs that were present in the UniProt XML, but were somehow not imported? or were they not present in the UniProt XML?

Note:

Coloring a MAPP with expression data

Note:

Running MAPPFinder

Note:

Compare Gene Database to Outside Resource

The OrderedLocusNames IDs in the exported Gene Database are derived from the UniProt XML. It is a good idea to check your list of OrderedLocusNames IDs to see how complete it is using the original source of the data (the sequencing organization, the MOD, etc.) Because UniProt is a protein database, it does not reference any non-protein genome features such as genes that code for functional RNAs, centromeres, telomeres, etc.

Note:

Downloaded on 10/27

• The complete proteome for V. cholerae was downloaded from UniProtKB in the XML format
• The GOA file for V. cholerae was downloaded from this link
• The GO OBO-XML formatted file for V. cholerae was downloaded from GO website
• The most recent version (3.0.0, build 5) of gmBuilder was downloaded from GitHub

Brandon Litvak
BIOL 367, Fall 2015