Difference between revisions of "Bklein7 Week 9"

From LMU BioDB 2015
Jump to: navigation, search
(added to sql database import section)
(finished writing the gene database creation section)
Line 73: Line 73:
  
 
*The downloaded data files for ''Vibrio cholerea'' were specified and imported into the database by clicking on the following buttons:
 
*The downloaded data files for ''Vibrio cholerea'' were specified and imported into the database by clicking on the following buttons:
−
* Selected File > Import UniProt XML...
+
** Selected File > Import UniProt XML...
−
* Selected File > Import GO OBO-XML...
+
** Selected File > Import GO OBO-XML...
−
* Clicked OK to the message asking to process the GO data.
+
** Clicked OK to the message asking to process the GO data.
−
* Selected File > Import GOA...
+
** Selected File > Import GOA...
  
 
===Exporting a GenMAPP Gene Database (.gdb)===
 
===Exporting a GenMAPP Gene Database (.gdb)===
  
−
* Selected File > Export to GenMAPP Gene Database...
+
* I selected File > Export to GenMAPP Gene Database... to begin the export process.
−
* Typed my name in the owner field (Brandon Klein).
+
* I typed my name in the owner field (Brandon Klein).
−
* GenMAPP Builder scans your PostgreSQL database to see what species are available.  Click on the species that you would like to export, then click ''Next'' to continue.
+
* I selected''Vibrio cholerae'' as the gene database species and then clicked ''Next''.
−
* Create GenMAPP Database: click on the "Save GenMAPP Database File As..." button.
+
* The database was saved as ''Vc-Std_BK_20151027''.
−
** In the Save dialog box that appears, navigate to the "T:" drive, and then modify the default file name by appending your initials.  Click the "Save" button
+
* I checked the boxes for exporting all Molecular Function, Cellular Component, and Biological Process Gene Ontology Terms.
−
* Leave the boxes checked for exporting all Molecular Function, Cellular Component, and Biological Process Gene Ontology Terms.
+
* Finally, I clicked the "Next" button to begin the export process.
−
* Click the "Next" button to begin teh export process.
+
−
 
+
−
==Importing Data into the PostgreSQL Database==
+
−
 
+
−
* Select File > Import UniProt XML...
+
−
** Navigate to the UniProt XML file that you extracted previously and click the Open button.
+
−
** This should take about 5-10 minutes, but may take longer depending on the size of the file, processor speed, and available memory of the machine.  When the process has completed, record the elapsed time from the message window that appears.
+
−
***Import Time: 2.92 minutes
+
−
* Select File > Import GO OBO-XML...
+
−
** Navigate to the GO OBO-XML file that you extracted previously.  Click the Open button.
+
−
** This should take about 5-10 minutes, but may take longer depending on the size of the file, processor speed, and available memory of the machine.  When the process has completed, record the elapsed time from the message window that appears.
+
−
***Import Time: 6.88 minutes
+
−
* Click OK to the message asking you to process the GO data.
+
−
** This should take about 5-10 minutes, but may take longer depending on the size of the file, processor speed, and available memory of the machine.  When the process has completed, record the elapsed time from the message window that appears.
+
−
***Processing Time: 4.49 minutes
+
−
* Select File > Import GOA...
+
−
** Navigate to the GOA file that you downloaded previously and click the Import button.  This process should only take a minute or so.
+
−
***Import Time: 0.06 minutes
+
−
==Exporting a GenMAPP Gene Database (.gdb)==
+
−
 
+
−
* Select File > Export to GenMAPP Gene Database...
+
−
* Type a name in the Owner field (or else it won't let you export).
+
−
** When doing the individual exercise for the [[Week 9]] assignment, use your own name.  When doing this for your team project, use your team's name.
+
−
* GenMAPP Builder scans your PostgreSQL database to see what species are available.  Click on the species that you would like to export, then click ''Next'' to continue.
+
−
* Create GenMAPP Database: click on the "Save GenMAPP Database File As..." button.
+
−
** In the Save dialog box that appears, navigate to the "T:" drive, and then modify the default file name by appending your initials.  Click the "Save" button
+
−
* Leave the boxes checked for exporting all Molecular Function, Cellular Component, and Biological Process Gene Ontology Terms.
+
−
* Click the "Next" button to begin teh export process.
+
−
** Record the starting and ending times from the black console window.  This will take 1-2 hours for a typical bacterial genome, depending on the size of the database, the processor speed, and available memory.  Large eukaryotic genomes (like ''Arabidopsis thaliana'') or genomes with many GO annotations (like ''Saccharomyces cerevisiae'') can take much longer, in the range of 12-24 hours.
+
−
**NOTE:  the progress bar is not accurate.
+
−
***Start Time: 4:44 PM (restarted export after class)
+
−
***End Time: 6:11 PM
+
−
***Elapsed Time: 1 hour, 27 minutes
+
−
 
+
  
 
==Gene Database Testing Report==
 
==Gene Database Testing Report==

Revision as of 07:34, 2 November 2015

Screenshots

TallyresultsBK1029.png PostgresIDsBK1029.png PostgresIDsBK1029 UPDATED.png XmlpipedbmatchoutputBK1029.png XmlpipedbmatchoutputBK1029 Updated.png

Pre-requisites

The following set of software was used in the creation and testing of the 'Vibrio cholerae gene database:

  1. 7-ziptool that for unpacking .gz and .zip files
  2. PostgreSQL on Windows (version 9.4.x)
  3. GenMAPP Builder
  4. Java JDK 1.8 64-bit
  5. GenMAPP 2
  6. XMLPipeDB match utility for counting IDs in XML files
  7. Microsoft Access for reading .mdb files

Gene Database Creation

Downloading Data Source Files and GenMAPP Builder

  • I download the UniProt XML, GOA, and GO OBO-XML files for Vibrio choleraealong with the GenMAPP Builder program.
    • All files were saved to the folder Bklein7_Week9 on my computer's ThawSpace.
    • Files that required extraction were unzipped using 7-zip.
    • Data files that remained in a folder after unzipping were removed from their folders to facilitate organization and command line processing.

UniProt XML

GOA

GO OBO-XML

Downloaded GenMAPP Builder

  1. I downloaded the GenMAPP Builder zip folder: Download gmbuilder-3.0.0-build-5.zip.
  2. I extracted the GenMAPP Builder folder using 7-zip.

Creating the New Database in PostgreSQL

  • I launched pgAdmin III and connected to the PostgreSQL 9.4 server (localhost:5432).
    • On this server, I created a new database: V.cholerae_20151027_gmb3build5.
    • I opened the SQL Editor tab to use an XMLPipeDB query to create the tables in the database.
      • I clicked on the Open File icon and selected the file gmbuilder.sql. This imported a series of SQL commands into the editor tab.
      • I clicked on the Execute Query icon to run this command.
      • In viewing the schema for this database, I confirmed that there were 167 tables after running the above command.

Configuring GenMAPP Builder to Connect to the PostgreSQL Database

  • To begin, I launched gmbuilder.bat.
  • I selected the "Configure Database" option and entered the following information into the fields below:
    • Host or address: localhost
    • Port number: 5432
    • Database name: V.cholerae_20151027_gmb3build5
    • Username: postgres
    • Password: Welcome1

Importing Data into the PostgreSQL Database

  • The downloaded data files for Vibrio cholerea were specified and imported into the database by clicking on the following buttons:
    • Selected File > Import UniProt XML...
    • Selected File > Import GO OBO-XML...
    • Clicked OK to the message asking to process the GO data.
    • Selected File > Import GOA...

Exporting a GenMAPP Gene Database (.gdb)

  • I selected File > Export to GenMAPP Gene Database... to begin the export process.
  • I typed my name in the owner field (Brandon Klein).
  • I selectedVibrio cholerae as the gene database species and then clicked Next.
  • The database was saved as Vc-Std_BK_20151027.
  • I checked the boxes for exporting all Molecular Function, Cellular Component, and Biological Process Gene Ontology Terms.
  • Finally, I clicked the "Next" button to begin the export process.

Gene Database Testing Report

Export Information

Version of GenMAPP Builder: Version 3.0.0 Build 5

Computer on which export was run: Seaver 120- Last computer on the right in the row closest to the front of the room

Postgres Database name: V.cholerae_20151027_gmb3build5

UniProt XML filename: uniprot-organism-243277_BK_20151027.xml

  • UniProt XML version: UniProt release 2015_10 (version information can be accessed on the UniProt News Page)
  • UniProt XML download link: [[1]]
  • Time taken to import: 2.92 minutes
    • Note:

GO OBO-XML filename: go_daily-termdb_BK_20151027.obo-xml

  • GO OBO-XML version: File last modified on ‎Tuesday, ‎October ‎27, ‎2015, ‏‎2:24:02 AM (accessed from the file properties window)
  • GO OBO-XML download link: [2]
  • Time taken to import: 6.88 minutes
  • Time taken to process: 4.49 minutes
    • Note:

GOA filename: 46.V_cholerae_ATCC_39315_BK_20151027.goa

  • GOA version: File last modified on Wednesday, October 14, 2015 (accessed on the UniProt-GOA news page)
  • GOA download link: [3]
  • Time taken to import: 0.06 minutes
    • Note:

Name of .gdb file: Vc-Std_BK_20151027.gdb

  • Time taken to export: 1 hour, 27 minutes
    • Start time: 4:44 PM (restarted export after class)
    • End time: 6:11 PM

Note:

TallyEngine

  • Run the TallyEngine in GenMAPP Builder and record the number of records for UniProt and GO in the XML data and in the Postgres databases.
    • Choose the menu item Tallies > Run XML and Database Tallies for UniProt and GO...
    • Take a screenshot of the results. Upload the image to the wiki and display it on this page.
    • For more information, see this page.

Using XMLPipeDB match to Validate the XML Results from the TallyEngine

Follow the instructions found on this page to run XMLPipeDB match.

Are your results the same as you got for the TallyEngine? Why or why not?

Using SQL Queries to Validate the PostgreSQL Database Results from the TallyEngine

For more information, see this page.

You can also look for counts at the SQL level, using some variation of a select count(*) query. This requires some knowledge of which table received what data. Here’s an initial tip: the gene/name tags in the XML file land in the genenametype table. A query on this table counting values from this table that were marked as ordered locus in the XML file matching the pattern VC_[0-9][0-9][0-9][0-9] would look like this: 

select count(*) from genenametype where type = 'ordered locus' and value ~ 'VC_[0-9][0-9][0-9][0-9]';

In pgAdmin III, you can issue these queries by clicking on the pencil/SQL icon in the toolbar, typing the query into the SQL Editor tab, then clicking on the green triangular Play button to run.

Pgadminiii-query.png

Are your results the same as reported by the TallyEngine? Why or why not?

OriginalRowCounts Comparison

Within the .gdb file, look at the OriginalRowCounts table to see if the database has the expected tables with the expected number of records. Compare the tables and records with a benchmark .gdb file.

Benchmark .gdb file:

Copy the OriginalRowCounts table from the benchmark and new gdb and paste them here:

Note:

Visual Inspection

Perform visual inspection of individual tables to see if there are any problems.

  • Look at the Systems table. Is there a date in the Date field for all gene ID systems present in the database?
  • Open the UniProt, RefSeq, and OrderedLocusNames tables. Scroll down through the table. Do all of the IDs look like they take the correct form for that type of ID?

Note:

.gdb Use in GenMAPP

Note:

Putting a gene on the MAPP using the GeneFinder window

  • Try a sample ID from each of the gene ID systems. Open the Backpage and see if all of the cross-referenced IDs that are supposed to be there are there.

Note:

Creating an Expression Dataset in the Expression Dataset Manager

  • How many of the IDs were imported out of the total IDs in the microarray dataset? How many exceptions were there? Look in the EX.txt file and look at the error codes for the records that were not imported into the Expression Dataset. Do these represent IDs that were present in the UniProt XML, but were somehow not imported? or were they not present in the UniProt XML?

Note:

Coloring a MAPP with expression data

Note:

Running MAPPFinder

Note:


Compare Gene Database to Outside Resource

The OrderedLocusNames IDs in the exported Gene Database are derived from the UniProt XML. It is a good idea to check your list of OrderedLocusNames IDs to see how complete it is using the original source of the data (the sequencing organization, the MOD, etc.) Because UniProt is a protein database, it does not reference any non-protein genome features such as genes that code for functional RNAs, centromeres, telomeres, etc.

Note:

Checking the Quality of your Exported Gene Database

  • It is a good idea to check the quality of your exported Gene Database to make sure that all of the data from the XML files made it into the PostgreSQL database and was then exported to the GenMAPP Gene Database. We have created a Gene Database Testing Report Sample to help guide you through this process.

Tally Engine

The first tool, called the Tally Engine, can be used for verifying that certain data from the XML file transferred consistently into the PostgreSQL database upon import. The Tally Engine can be found in GenMAPP Builder itself.

  1. Run PostgreSQL (via pgAdmin III on Windows) and make sure that your database is up and running.
  2. Run GenMAPP Builder and make sure that it is connected to the database (via Configure Database...).
  3. After performing an import, choose Run XML and Database Tallies for UniProt and GO....
  4. Choose the UniProt and GO files that you imported.
  5. You should see a table for selected data items, and how many of each were found.

Tally-results.png

Under the hood, the Tally Engine bases its XML counts on certain XML tags, and bases its database counts on SQL queries using count. This tool is thus primarily useful for making sure that the “raw” import worked without any errors or glitches.

My Tally Results:

TallyresultsBK1029.png

XMLPipeDB Match

XMLPipeDB Match is useful for counting data in files. Thus, in our context, you would use XMLPipeDB Match to tally stuff in XML files, with greater flexibility than with Tally Engine.

You will have to use XMLPipeDB Match from the command line. In addition, you can use this in any platform (as you have seen). Download the application from the XMLPipeDB SourceForge site and take note of the location of the xmlpipedb-match-1.1.1.jar. Then, on the command line (Terminal for Linux and Mac OS X, cmd on Windows), cd to the folder containing the XML file that you would like to check. Use XMLPipeDB Match as follows, with the parts in parentheses varying depending on your specific setup, desired pattern, and file being scanned: 

java -jar (location-of-jar) "(pattern)" < (XML file)

On a Windows machine, with XMLPipeDB Match and a Vibrio cholerae XML file located on the Desktop, scanning for IDs of the form VC_####, where # represents a digit from 0 to 9, one would type, after cd-ing to the Desktop: 

java -jar xmlpipedb-match-1.1.1.jar "VC_[0-9][0-9][0-9][0-9]" < uniprot-taxonomy%3A243277.xml

As you have seen before, this will give you a list of unique matches, with a total number at the bottom.

The trick with XMLPipeDB Match is to use the patterns well: with the database project, you will mainly be matching IDs. A desired count is an XMLPipeDB Match result whose matched ID pattern corresponds to the number of IDs found by the Tally Engine.

SQL

You can also look for counts at the SQL level, using some variation of a select count(*) query. This requires some knowledge of which table received what data. Here’s an initial tip: the gene/name tags in the XML file land in the genenametype table. A query on this table counting values from this table that were marked as ordered locus in the XML file matching the pattern VC_[0-9][0-9][0-9][0-9] would look like this: 

select count(*) from genenametype where type = 'ordered locus' and value ~ 'VC_[0-9][0-9][0-9][0-9]';

In pgAdmin III, you can issue these queries by clicking on the pencil/SQL icon in the toolbar, typing the query into the SQL Editor tab, then clicking on the green triangular Play button to run.

Pgadminiii-query.png

Microsoft Access

For the GenMAPP Gene Database, you can open the .gdb in Microsoft Access and navigate its tables to find counts for various IDs. Opening the table, noting its size, and doing some sorting may help. You can also look at the OriginalRowCounts table for a summary of totals.

Again, the ideal situation is a correspondence in these numbers with what you found in XML and the relational database.

Back to the Command Line

Amidst all this, you can still use grep and wc on the command line for some basic counting. Just remember that these tools work on a line-by-line basis; useful in some cases, but not useful in others.

You can use grep and wc with the various files on the my.cs.lmu.edu server by using the curl -O command shown in the Week 6 assignment. Upload your data files to the wiki, place media links to them on your wiki page, then mouse over those live links to capture their URL (like this—look at the source to see the wiki markup), then use curl -O (whatever-the-url-is) while ssh-ed to my.cs.lmu.edu to bring that file into the server.

  • If the file is a .zip file, you can use unzip at the command line to unzip it.
  • If the file is a .gz file, you can use gunzip at the command line to uncompress that one.


Links

Assignments Pages

Individual Journal Entries

Shared Journal Entries

Retrieved from "https://xmlpipedb.lmucs.io/biodb/fall2015/index.php?title=Bklein7_Week_9&oldid=4945"