Week 9 E-notes Eyanosch - Revision history

Eyanosch at 17:42, 31 October 2015

2015-10-31T17:42:41Z

Eyanosch: Importing the Uniprot and GO OBO files

2015-10-27T22:39:14Z

Importing the Uniprot and GO OBO files

Eyanosch at 22:14, 27 October 2015

2015-10-27T22:14:33Z

Eyanosch: diagram of Dr. Dhalquists explanation in class

2015-10-27T22:10:22Z

diagram of Dr. Dhalquists explanation in class

Eyanosch: added the export information, taking notes on actions taken in class

2015-10-27T21:50:46Z

added the export information, taking notes on actions taken in class

New page

Notes for week 9

==Export Information==

Version of GenMAPP Builder:

Computer on which export was run:

Postgres Database name:

UniProt XML filename (give filename and upload and link to compressed file):
* UniProt XML version (The version information can be found at [http://uniprot.org/news the UniProt News Page]):
* UniProt XML download link:
* Time taken to import:
** Note:

GO OBO-XML filename (give filename and upload and link to compressed file):
* GO OBO-XML version (The version information can be found in the file properties after the file downloaded from the [http://beta.geneontology.org/page/download-ontology GO Download page] has been unzipped):
* GO OBO-XML download link:
* Time taken to import:
* Time taken to process:
** Note:

GOA filename (give filename and upload and link to compressed file):
* GOA version (News on [http://www.ebi.ac.uk/GOA/ this page] records past releases; current information can be found in the Last modified field on the [ftp://ftp.ebi.ac.uk/pub/databases/GO/goa/proteomes/ FTP site]):
* GOA download link:
* Time taken to import:
** Note:

Name of .gdb file (give filename and upload and link to compressed file):
* Time taken to export:
** Start time:
** End time:

Note:

==TallyEngine==

* Run the TallyEngine in GenMAPP Builder and record the number of records for UniProt and GO in the XML data and in the Postgres databases.
** Choose the menu item Tallies > Run XML and Database Tallies for UniProt and GO...
** Take a screenshot of the results. Upload the image to the wiki and display it on this page.
** For more information, [[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | see this page.]]

== Using XMLPipeDB match to Validate the XML Results from the TallyEngine==

[[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | Follow the instructions found on this page to run XMLPipeDB match.]]

Are your results the same as you got for the TallyEngine? Why or why not?

== Using SQL Queries to Validate the PostgreSQL Database Results from the TallyEngine==

For more information, [[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | see this page.]]

You can also look for counts at the SQL level, using some variation of a ''select count(*)'' query. This requires some knowledge of which table received what data. Here’s an initial tip: the ''gene/name'' tags in the XML file land in the ''genenametype'' table. A query on this table counting values from this table that were marked as ''ordered locus'' in the XML file matching the pattern ''VC_[0-9][0-9][0-9][0-9]'' would look like this:

select count(*) from genenametype where type = 'ordered locus' and value ~ 'VC_[0-9][0-9][0-9][0-9]';

In ''pgAdmin III'', you can issue these queries by clicking on the pencil/SQL icon in the toolbar, typing the query into the ''SQL Editor'' tab, then clicking on the green triangular ''Play'' button to run.

[[Image:Pgadminiii-query.png]]

Are your results the same as reported by the TallyEngine? Why or why not?

==OriginalRowCounts Comparison==

Within the .gdb file, look at the OriginalRowCounts table to see if the database has the expected tables with the expected number of records. Compare the tables and records with a benchmark .gdb file.

Benchmark .gdb file:

Copy the OriginalRowCounts table from the benchmark and new gdb and paste them here:

Note:

==Visual Inspection==

Perform visual inspection of individual tables to see if there are any problems.

* Look at the Systems table. Is there a date in the Date field for all gene ID systems present in the database?
* Open the UniProt, RefSeq, and OrderedLocusNames tables. Scroll down through the table. Do all of the IDs look like they take the correct form for that type of ID?

Note:

==.gdb Use in GenMAPP==

Note:

===Putting a gene on the MAPP using the GeneFinder window===

* Try a sample ID from each of the gene ID systems. Open the Backpage and see if all of the cross-referenced IDs that are supposed to be there are there.

Note:

===Creating an Expression Dataset in the Expression Dataset Manager===

* How many of the IDs were imported out of the total IDs in the microarray dataset? How many exceptions were there? Look in the EX.txt file and look at the error codes for the records that were not imported into the Expression Dataset. Do these represent IDs that were present in the UniProt XML, but were somehow not imported? or were they not present in the UniProt XML?

Note:

===Coloring a MAPP with expression data===

Note:

===Running MAPPFinder===

Note:

== Compare Gene Database to Outside Resource==

The OrderedLocusNames IDs in the exported Gene Database are derived from the UniProt XML. It is a good idea to check your list of OrderedLocusNames IDs to see how complete it is using the original source of the data (the sequencing organization, the MOD, etc.) Because UniProt is a protein database, it does not reference any non-protein genome features such as genes that code for functional RNAs, centromeres, telomeres, etc.

Note:

← Older revision		Revision as of 22:39, 27 October 2015
Line 19:		Line 19:

	46.V_cholerae_ATCC_39315.goa '''13-Oct-2015 07:31''' 5.0M		46.V_cholerae_ATCC_39315.goa '''13-Oct-2015 07:31''' 5.0M
		+
		+	Imported the Uniprot XML file that had been unzipped. Record data for personal journal
		+
		+	Imported the GO OBO XML file that had been unzipped. Record data for personal journal
		+
		+	Imported the file that had been unzipped. Record data for personal journal

← Older revision		Revision as of 22:14, 27 October 2015
Line 17:		Line 17:

	#UniProt XML		#UniProt XML
		+
		+	46.V_cholerae_ATCC_39315.goa '''13-Oct-2015 07:31''' 5.0M

← Older revision		Revision as of 17:42, 31 October 2015
Line 25:		Line 25:

	Imported the file that had been unzipped. Record data for personal journal		Imported the file that had been unzipped. Record data for personal journal
		+
		+
		+
		+	==Export Information==
		+
		+	Version of GenMAPP Builder:
		+
		+	Computer on which export was run: Computer second to the front left of class. (to the right of Jake's class)
		+
		+	Postgres Database name: Vcholerae_20151027_gmb3build5
		+
		+	UniProt XML filename (give filename and upload and link to compressed file): Eyanosch\uniprot-organism%3A243277.xml
		+	* UniProt XML version (The version information can be found at [http://uniprot.org/news the UniProt News Page]):
		+	* UniProt XML download link:
		+	* Time taken to import: 3.08 Minutes
		+	** Note: I wasn't able to download the GM builder using Chrome FYI, earlier in the process
		+
		+	GO OBO-XML filename (give filename and upload and link to compressed file): go_daily-termdb.obo-xml
		+	* GO OBO-XML version (The version information can be found in the file properties after the file downloaded from the [http://beta.geneontology.org/page/download-ontology GO Download page] has been unzipped):
		+	* GO OBO-XML download link:
		+	* Time taken to import: 7.44 minutes
		+	* Time taken to process: 4.65 minutes
		+	** Note:
		+
		+	GOA filename (give filename and upload and link to compressed file): 46.V_cholerae_ATCC_39315.goa
		+	* GOA version (News on [http://www.ebi.ac.uk/GOA/ this page] records past releases; current information can be found in the Last modified field on the [ftp://ftp.ebi.ac.uk/pub/databases/GO/goa/proteomes/ FTP site]):
		+	* GOA download link:
		+	* Time taken to import: 0.07 minutes
		+	** Note:
		+
		+	Name of .gdb file (give filename and upload and link to compressed file): Vc-Std_20151027_EMY.gdb
		+	* Time taken to export: 1:17:43
		+	** Start time: 3:54:02
		+	** End time:5:11:45
		+
		+	Note:
		+
		+	==TallyEngine==
		+
		+	* Run the TallyEngine in GenMAPP Builder and record the number of records for UniProt and GO in the XML data and in the Postgres databases.
		+	** Choose the menu item Tallies > Run XML and Database Tallies for UniProt and GO...
		+	** [[File:DatabaseTallies EMY 20153110.jpg]]
		+	** For more information, [[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways \| see this page.]]
		+
		+	== Using XMLPipeDB match to Validate the XML Results from the TallyEngine==
		+
		+	[[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways \| Follow the instructions found on this page to run XMLPipeDB match.]]
		+
		+	*
		+
		+
		+	Are your results the same as you got for the TallyEngine? Why or why not?
		+
		+	== Using SQL Queries to Validate the PostgreSQL Database Results from the TallyEngine==
		+
		+	For more information, [[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways \| see this page.]]
		+
		+	You can also look for counts at the SQL level, using some variation of a ''select count(*)'' query. This requires some knowledge of which table received what data. Here’s an initial tip: the ''gene/name'' tags in the XML file land in the ''genenametype'' table. A query on this table counting values from this table that were marked as ''ordered locus'' in the XML file matching the pattern ''VC_[0-9][0-9][0-9][0-9]'' would look like this:
		+
		+	select count(*) from genenametype where type = 'ordered locus' and value ~ 'VC_[0-9][0-9][0-9][0-9]';
		+
		+	In ''pgAdmin III'', you can issue these queries by clicking on the pencil/SQL icon in the toolbar, typing the query into the ''SQL Editor'' tab, then clicking on the green triangular ''Play'' button to run.
		+
		+	[[Image:Pgadminiii-query.png]]
		+
		+	Are your results the same as reported by the TallyEngine? Why or why not?
		+
		+	==OriginalRowCounts Comparison==
		+
		+	Within the .gdb file, look at the OriginalRowCounts table to see if the database has the expected tables with the expected number of records. Compare the tables and records with a benchmark .gdb file.
		+
		+	Benchmark .gdb file:
		+
		+	Copy the OriginalRowCounts table from the benchmark and new gdb and paste them here:
		+
		+	Note:
		+
		+	==Visual Inspection==
		+
		+	Perform visual inspection of individual tables to see if there are any problems.
		+
		+	* Look at the Systems table. Is there a date in the Date field for all gene ID systems present in the database?
		+	* Open the UniProt, RefSeq, and OrderedLocusNames tables. Scroll down through the table. Do all of the IDs look like they take the correct form for that type of ID?
		+
		+	Note:
		+
		+	==.gdb Use in GenMAPP==
		+
		+	Note:
		+
		+	===Putting a gene on the MAPP using the GeneFinder window===
		+
		+	* Try a sample ID from each of the gene ID systems. Open the Backpage and see if all of the cross-referenced IDs that are supposed to be there are there.
		+
		+	Note:
		+
		+	===Creating an Expression Dataset in the Expression Dataset Manager===
		+
		+	* How many of the IDs were imported out of the total IDs in the microarray dataset? How many exceptions were there? Look in the EX.txt file and look at the error codes for the records that were not imported into the Expression Dataset. Do these represent IDs that were present in the UniProt XML, but were somehow not imported? or were they not present in the UniProt XML?
		+
		+	Note:
		+
		+	===Coloring a MAPP with expression data===
		+
		+	Note:
		+
		+	===Running MAPPFinder===
		+
		+	Note:
		+
		+
		+	== Compare Gene Database to Outside Resource==
		+
		+	The OrderedLocusNames IDs in the exported Gene Database are derived from the UniProt XML. It is a good idea to check your list of OrderedLocusNames IDs to see how complete it is using the original source of the data (the sequencing organization, the MOD, etc.) Because UniProt is a protein database, it does not reference any non-protein genome features such as genes that code for functional RNAs, centromeres, telomeres, etc.
		+
		+	Note:

@@ Line 2: / Line 2: @@
-Version of GenMAPP Builder:
+Uniprotxml <--> GOA <--> GO OBO-XML
-Postgres Database name:
+#UniProt XML
-−
-UniProt XML filename (give filename and upload and link to compressed file):
-* UniProt XML version (The version information can be found at [http://uniprot.org/news the UniProt News Page]):
-* UniProt XML download link:
-* Time taken to import:
-** Note:
-−
-GO OBO-XML filename (give filename and upload and link to compressed file):
-* GO OBO-XML version (The version information can be found in the file properties after the file downloaded from the [http://beta.geneontology.org/page/download-ontology GO Download page] has been unzipped):
-* GO OBO-XML download link:
-* Time taken to import:
-* Time taken to process:
-** Note:
-−
-GOA filename (give filename and upload and link to compressed file):
-* GOA version (News on [http://www.ebi.ac.uk/GOA/ this page] records past releases; current information can be found in the Last modified field on the [ftp://ftp.ebi.ac.uk/pub/databases/GO/goa/proteomes/ FTP site]):
-* GOA download link:
-* Time taken to import:
-** Note:
-−
-Name of .gdb file (give filename and upload and link to compressed file):
-* Time taken to export:
-** Start time:
-** End time:
-−
-Note:
-−
-==TallyEngine==
-−
-* Run the TallyEngine in GenMAPP Builder and record the number of records for UniProt and GO in the XML data and in the Postgres databases.
-** Choose the menu item Tallies > Run XML and Database Tallies for UniProt and GO...
-** Take a screenshot of the results.  Upload the image to the wiki and display it on this page.
-** For more information, [[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | see this page.]]
-−
-== Using XMLPipeDB match to Validate the XML Results from the TallyEngine==
-−
-[[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | Follow the instructions found on this page to run XMLPipeDB match.]]
-−
-Are your results the same as you got for the TallyEngine?  Why or why not?
-−
-== Using SQL Queries to Validate the PostgreSQL Database Results from the TallyEngine==
-−
-For more information, [[How_Do_I_Count_Thee%3F_Let_Me_Count_The_Ways | see this page.]]
-−
-You can also look for counts at the SQL level, using some variation of a ''select count(*)'' query.  This requires some knowledge of which table received what data.  Here&rsquo;s an initial tip: the ''gene/name'' tags in the XML file land in the ''genenametype'' table.  A query on this table counting values from this table that were marked as ''ordered locus'' in the XML file matching the pattern ''VC_[0-9][0-9][0-9][0-9]'' would look like this:
-−
- select count(*) from genenametype where type = 'ordered locus' and value ~ 'VC_[0-9][0-9][0-9][0-9]';
-−
-In ''pgAdmin III'', you can issue these queries by clicking on the pencil/SQL icon in the toolbar, typing the query into the ''SQL Editor'' tab, then clicking on the green triangular ''Play'' button to run.
-−
-[[Image:Pgadminiii-query.png]]
-−
-Are your results the same as reported by the TallyEngine?  Why or why not?
-−
-==OriginalRowCounts Comparison==
-−
-Within the .gdb file, look at the OriginalRowCounts table to see if the database has the expected tables with the expected number of records. Compare the tables and records with a benchmark .gdb file.
-−
-Benchmark .gdb file:
-−
-Copy the OriginalRowCounts table from the benchmark and new gdb and paste them here:
-−
-Note:
-−
-==Visual Inspection==
-−
-Perform visual inspection of individual tables to see if there are any problems.
-−
-* Look at the Systems table. Is there a date in the Date field for all gene ID systems present in the database?
-* Open the UniProt, RefSeq, and OrderedLocusNames tables. Scroll down through the table. Do all of the IDs look like they take the correct form for that type of ID?
-−
-Note:
-−
-==.gdb Use in GenMAPP==
-−
-Note:
-−
-===Putting a gene on the MAPP using the GeneFinder window===
-−
-* Try a sample ID from each of the gene ID systems. Open the Backpage and see if all of the cross-referenced IDs that are supposed to be there are there.
-−
-Note:
-−
-===Creating an Expression Dataset in the Expression Dataset Manager===
-−
-* How many of the IDs were imported out of the total IDs in the microarray dataset? How many exceptions were there? Look in the EX.txt file and look at the error codes for the records that were not imported into the Expression Dataset.  Do these represent IDs that were present in the UniProt XML, but were somehow not imported? or were they not present in the UniProt XML?
-−
-Note:
-−
-===Coloring a MAPP with expression data===
-−
-Note:
-−
-===Running MAPPFinder===
-−
-Note:
-−
-−
-== Compare Gene Database to Outside Resource==
-−
-The OrderedLocusNames IDs in the exported Gene Database are derived from the UniProt XML.  It is a good idea to check your list of OrderedLocusNames IDs to see how complete it is using the original source of the data (the sequencing organization, the MOD, etc.)  Because UniProt is a protein database, it does not reference any non-protein genome features such as genes that code for functional RNAs, centromeres, telomeres, etc.
-−
-Note: