Wed 17dec14
- website backup of T3
- new data types (continuous genotypes, non-SNP variation, imputed data)
- CSR Data File and Annotation File stored in directory raw/phonotype/CSR,
if file already exists then appended timestamp to file name
- How to automate the data usage policy table
- Minimum Information about Plant Phenotype Experiments or API standard
- marker annotation by blasting against wheat survey sequence
40881 entries for marker_annotations using only markers in loaded maps
- next meeting discuss PAG presentations (possible Mon Dec 22 and Wed Jan 7)
Wed 10dec14
- multidimensional phenotype data (James)
NIR spectral, weight, photo
plot level (maybe not in DB) and means
database structure? backup data?
- Manage Phenotype Trials (Vic)
- comments on GBS markers in T3
- Scab meeting problems with data template versions
- done: Stop version-checking the Trial Annotations template
- switching barley Line submission to the multi-Breeding-Program version
- Use a wheat-based template for Phenotype Results for BD and other wheat T3s.
- Format the Trial Mean row as General rather than Text to allow formulas.
discuss next week
- Add single-trial Phenotype Results template to the Data Submission page?
- wheat 820K (UK) and 660K (China) SNP arrays
Wed 03dec14
- done: blast database for wheat chromosome survey sequence in blast tool, note that when using new version (4A_v2) the version number is not used in contig name
- www.solvebio.com - private beta (no access now)
1. access to curated data library
2. api for datasets
- Search Trials by Experiment
- barley POPSEQ SNP matrix ftp://ftp.ipk-gatersleben.de/barley-popseq/snp_data
- barley BAC sequences http://harvest-web.org/hweb/
Wed 26nov14
I- Vic presenting Breeders Datafarm at Scab meeting Dec 7-9
- Vic questions about loading markers
- barley POPSEQ SNP, 11 million SNPs
filter to reduce data? (maf, markers missing data, lines missing data, contig length)
- GBS data requirements http://triticeaetoolbox.org/wheat/curator_data/gbs_import_instructions.php
- done: Phenotype Results from multiple trials in the same file
-----------
Wed 19nov14
- GBS documentation in T3
State: Can we define a protocol for loading?
!! Ask source if we should name/number the markers or if they want to.
!! Store full contig sequence in GG and link to it from Marker record.
!! Also link to GBrowse on URGI(?).
!! Instructions to data donors as in curator_data/gbs_import_instructions.php
!! Insist on simple 2D allele file like examples/GBS_Genotype_template.txt
!! Define A_allele in examples//Generic_SNP.txt. Make example for GBS.
*C,V Document what we want for input format.
Handling TBT, TagsByTaxa format
!! Avoid having to load marker sets in an order; reshuffle primary/synonyms
afterward in batch process.
?? Markerset, to allow user to receive synonym names instead of primary?
- Affymetrix Axiom 35K Wheat Breeders Array in T3?
*JL Ask users whether they want it, e.g. are using the array.
- Backup on BioHPC
Test transfer from malt worked. ssh to Globus source:dest:file. $80/Tb/yr
*C Do it. Disks -> James
- Loading Genotype Results (checking synonyms), parallel BLAST
- Phenome Project Unity
*** Suggest API project.
- How to deal with disease scores taken at multiple time points in T3/Oat
- oat GBS map, 45K markers
- wheat GBS: contig order from genome zipper, for Gina
*D Load the POPSEQ map from Mayer article?
- Allele Conflicts out of date, genotyping/sum_lines.php
-----------
Mon 10nov14: skipped
-----------
Mon 3nov14
- GBS roundup
- marker sequence synonyms using BLAST
- backups onto BioHPC: First 200 GB is free. James can use the NAS drives.
- PAG talks
- T3 manuscript
- Eduard's GBS pipeline
* - Trial Submission Form date
*C: Writeup of extracting context sequence of SNPs from Eduard's VCV
-----------
Mon 27oct14
- Network attached storage malware problem
- replacement, add firewall, or build custom box?
- $80/TB/year for storage on CBSU machine at BioHPC. One of our T3's is
about 100GB. *C: Ask Jarek Pillardy re their malware, rsync capability.
- Return to Amazon, through Doris Hagen. Possible?
- checking marker import sequence for matches using BLAST. Looks Good.
- Hadi Quesnesville, WheatIS, is building a list of nodes accessible from
WheatIS's search engine (Apache SolR). E.g. germplasm names. Wants a
statement of willingness. Saying Yes Contingent on the Specifics.
- new "CAP Data Programs" page
- editor for What's New, wheatplus/login/edit_whatsnew.php
git overwriting the whatsnew files of BD, Oat?
-----------
Tue 21oct14: T3/GG Coordination (last met 8sep14)
- RSA with UC Davis for Programmer. High salary. Yong has cleared with
HR. Primary job would be T3/GG. Develop some Requirements. Examples:
- GG API to be consumed by T3 to show marker info, maps. Propose to the
API group? Correspondences of marker names on the map. Rosetta-Stone
merging process?
- T3-like upload scripts for maps etc.
- Move T3 data to GG. Catalog what data types should move.
- oat map from GrainGenes to T3/Oat
- long sequences in GG, > 16M. Or links to MIPS?
- next meeting the day before Veterans Day, Nov 10? Nov 17?
-----------
Mon
20oct14
- Display Genotype trial page added select markers and compatibility
with large GBS table
*D new "CAP Data Programs" page,
http://malt.pw.usda.gov/t3/wheatplus/all_breed_css.php
- notifying users of the new Line Submission Form template
- download selected markers with annotation?
- separate Google Analytics for Oat vs Wheat vs Barley?
-----------
Wed 15oct14
- Data Programs and Breeding Programs on wheat database
!! A Breeding program can be the source of either Lines or Trials.
- GBS synonyms-by-sequence, 63 vs. 64 bp. Clay will use BLAST.
*** TCAP "Participant" from Saaten-Union? Clarify "Why register".
*** Non-participants *can* make their own Line Panels.
*C wheat POPSEQ map of 1.3M genome contigs. Barley too (Muehlbauer)?
*D - Ask Mayer which assembly version used for contig names.
- T3/GG coordination meeting next week
*** Select by Properties "Select a Category."
**D all_breed_css.php: Delete "CAP". Three tables, breeding/data/mapping
*D Add Genetic Characters for barley to menu, traitadd.php
*D barley Line Submission Form; .Send copy to Vic.
*D one Lines template for each species
*D one master git code repository for both wheat and barley
- Yung-Fen's CORE T3/Oat tutorial
- oat SNP map
- oat Line Submission Form and Genetic Characters
-----------
Mon 06oct14, Users
Gina, Brian Steffenson, James Clohessy, Jianli, Karen Beaubien, Mark
Sorrells, Peter Bradbury, Shiaoman, Trevor Rife, Jamie Sherman
GWAS: "linke", capitalize sentences in Download, "Mod. Aug. Design", indicate which fields are required
1. Changes to genotype data by experiment and GBS
2. Updates to experimental design page
3. Updates on integration with Android Field Book; plotid includes name of
Trial
4. Barley line template will allow multiple breeding programs
- Next meeting early December
*- Send Jamie a message re the new Barley lines template.
-----------
Mon 06oct14, T3 staff
- NAS installed (git, rsync, mysql)
- Offsite live T3 server? No, wait for Congress to get pissy.
- PAG 2015 presentations: API, ARS Database Workshop, Posters (Vic, Clay)
Computer Demo (Vic)
- T3 paper. HWW-90K wheat genotyping is online. GBS included, or later?
- synonym matching for 63-base sequences from Jesse's HWW. Vic figured it out.
SNP position outside the context sequence length. Clay: matching partial
lengths is too slow for website use, would need programmer to do on
commandline. Use /tmp/tht/ directory for "interactive" partial files?
- T3 User group today.
Mark "*U" in future for topics we want them to consider.
- Katherine Frels: Calculate CSR Indices for multiple timepoints or indices
at once?
- New Oat Curator Clare Saied starts tomorrow.
-----------
Mon 29sep14
- Bill Gates visit.
- T3 paper: Crop Science or The Plant Genome? Cornell Master not needed.
Now have Eduard's 90K HWW allele data.
- data loading
1) deleting marker data
2) marker import tool, does it find synonyms for Jesse's HWW
*C Matching is broken for 63-base sequences for SynOp from Jesse.
- User group Oct 6th
- experiment design; integrate with Android Fieldbook
- genotype data by GBS experiment
- documentation of Sandbox use for analyzing temporary data
- page to manage Lists (not yet)
- defined datasets with filters (not yet)
- diversity analysis to produce the most-diverse subset of a panel (not yet)
*J- wheat and barley POPSEQ maps from Gary Muehlbauer to anchor GBS SNPs
*** Does Eduard's vcf file come with a map???
- pages modified to work with new GBS table
GWAS, trials and traits, maps
- Line uid vs. name foulup in Quick Search, barley Line 6B01-2221 vs. 2717
***- Are markers with no map location valuable? (Vic) Yes
- Breeding vs Data program in Wizard and homepage search; Annotation template
*V- new template for barley Lines being tested by Vic
*V- new Show Alleles for Lines for a marker, being tested by Vic
- Clare starting October 6.
-----------
Mon 22sep14
- Bill Gates visit. Poster (with Lukas), demo.
- added Curate => Delete Trials and Experiments, including genotyping
experiments. Slow? Needs to rebuild allele caches.
- pages modified to work with new GBS table in MySQL
display genotype experiment
<http://malt.pw.usda.gov/t3/wheat/display_genotype.php?trial_code=HWWAMP_2014_GBS>
select markers
select lines by genotype experiment
download genotype and phenotype data
content status
- wizard page now shows breeding programs and data programs
* What does the code do? What do we want it to do?
- Cornell Master genotype data (should markers be synonyms of SynOp, use
same map or new map). Yes make synonyms.
- Marker report: List genotype experiments
V** Add genoexperiment to Alleles for All Lines for a marker, marker.inc
- NAS chassis is here.
- User Group meeting Oct 6
- Clare starts Oct 6 or 7
-----------
Mon 15sep14
- Eduard's lost HWW 90K data; GBS data 50K markers
- T3/Oat curator candidate selection. Consult Nick
- Cornell Master GBS
- Network attached storage device. Spare drive hot-swappable RAID5.
Locate in G-07. All 8 bays taken. Specd by Clay and James, separately.
Other expenditures: tablet, conference phone, CBSU installation/maintenance
= $3K
*D send recommendation for conference phone to JL
*C- T3 Manuscript: Big Data, GBS. Clay is loading GBS's manually instead of
web scripts. Table of Content Status will be changing when Eduard's GBS
*D,V Marker_types: Array SNPs, GBS SNPs, DArTs, Other. Which one is Wheat "SNP"?
"Array SNP" = Sequenom, Snapshot, Veracode, Illumina
*D- Excalibur_c10030_1305 in wheat has no data but says "Show line alleles" in
view.php?table=markers
- Bill Gates visit
- CORE data in Oat Chex from Yung-Fen. To delete: experiment CORE2010_Infinium
-----------
Mon 8sep14
- T3 and GrainGenes collaboration
- RSA with UC Davis for Programmer to be approved this week, could hire next
month. High salary.
- GG API to be consumed by T3 to show marker info, maps; move T3 data to GG.
- T3-like upload scripts for maps etc.
- CSS / logo
* Catalog what data types should move from T3 to GG.
- Link traits, germplasm, QTLs in GrainGenes to T3.
- Cornell Master codenames decoded by Lynn Veenstra. Straightforward.
- improvements on modified augmented design form
- GBS data loading: Vic has resolved Jesse's. Test for SNPs with position
outside the extent of the context sequence. Also Clay is working on
Eduard's GBS genotyping data. Markers not on production because of duplicated
sequences (Vic).
- Cleaning up / simplifying the set of marker types:
1. Illumine SNP chip
2. DArT
3. GBS (split between restriction enzyme and exome capture?)
4. Historical / Miscellaneous (less than, say, 500 markers)
*- genotype experiment page display_genotype.php:
- "Select these markers" to view which
- Show which marker types are included? No, the list of names
Upgrade PHP for better session handling? Don't store marker selections in
session? Save to database?
*D - footer too wide in BD
*D - merging marker types
-----------
Tue 2sep14
- Dropbox has first draft of the Manuscript.
- Viroblast does accept ambiguity codes: YGAYMGGGARGAGGGGGAYAAGGCGGTCKCMTCGGGGTACGCCGACCTCGTCTCTTTTGGGCGGCTCTTCCTGGCAAACCCYGACCTGCCGAAGAGGCTAGAGCTCAACGCGCCGCTTAA
**D Viroblast bug: no name shown. Use makeblastdb without -parse_seqids
- GBS data storage by experiment
- Download and Select-Lines-by-Genotype-Experiment updated.
- do we need imports tools for large GBS data sets on website or only as scripts?
- schedule user group meeting 2nd Monday of October
- Meet with GG on 2nd Monday of each month.
-----------
Mon 25aug14
- HWW AM panel: 90K data from Eduard, GBS data from Jesse, and Eduard (.vcf)
*V Ask Jesse for description of files at wheatgenetics.org/HWWAMP/
*C Status of parsing the .VCF?
*JL Followup Vic's Aug 20 email to Eduard.
- new GBS-handling system, wheatplus/downloads/downloads.php
- For Trial and Phenotype forms, use Data Program instead of Breeding Program?
*D Wizard should show both Data/Breeding (with Clay)
*D Paragraph glossary for Data vs. Breeding
- Allow a program to be both Data and Breeding
- all_breed_css.php: Make linkouts only for "data" and "mapping" programs?
- Sort into subsections by program type.
- questions from Yung-Fen Huang
*D- Edit Experiments shows breeding programs instead of data programs; login/edit_trials.php
- Home page: Add a Browser? Search by Location, Year
- How to cite T3?
- Dropbox has first draft of the Manuscript.
-----------
Mon 18aug14
- Quick Links - addition of Genotype Experiments
- GBS data storage - test site on malt:/cbirkett/t3/wheattest and
malt:/cbirkett/t3/wheatplus
- new submission template for barley Lines
- parallelize our GWAS analysis by chromosome?
-----------
Mon 11aug14
***- In Select Lines by Properties, "Year" is trial year, not creation. Is OK.
- hard drive failure on tcap machine
Dave H will set up a tcap virtual host on malt to warn users of downtime.
- SSL on login page
wheat:/etc/apache2/sites-available/squirrelmail: RewriteCond %{HTTPS} !^on$ [NC]
- GBS data storage
- Memory overload uploading the file. Log overflow with > 1000 errors.
- 1 experiment/table approach: filtering => finding mapped markers
=> allele retrieval. Omit steps 1 and 2, too slow. Conversion to
output format (A/B to ACTG or 0/1) could be slow.
***- Load Lines from multiple Breeding Programs in one file. Done for wheat.
*D- adding Genetic Characters for barley
*D- Load Phenotypes from multiple Trials in one file
- Move Sandboxes and Breeders' Datafarm to tcap for SSL?
**JL No, JL will certify malt.
- JL vacationing till 8/25. Peter too after next week.
- James:
- Pepsico oat server ordered.
*JC- input_file_log not being updated by seven scripts. Will fix.
***- development account for James on malt
***- API for merging T3s? Send apiary
info to James.
- Oat Curator position open, through Cornell, BS preferred, located Ithaca.
- GWAS cool result for manuscript: breeder plump grain linked to Ppd-H1, the
2/6 row gene
-----------
Mon 4aug14
- fixed: All Breeders Datafarm registrants auto-promoted to Curator
- SSL certificate on tcap; needs change in config.php
- backup of website on tcap; mysql slave system blocked internally
- better storage table for genotyping data (Clay): new mySQL tables or HDF5
Eduard's GBS data ca. 2 days to load.
Organize the markers by map position to allow GWAS by chromosome?
Wait for Back Office, don't try to use 1.3M markers in the same tools?
Develop Reduced Marker Sets?
*D WCSS1* is too many markers for Quick Search to deliver
*C Create a T3BigData db on malt.
- manuscript Tables and Figures
- Lines file submissions failing to notify Curator on Breeders Funnyfarm
- unclear things (Jean-Luc)
*D - Select Markers by Map Position should default to all-selected
* - Search for all Genotype experiments. (Phenotype Experiments too.)
*V - Eduard's outstanding HWW 90K dataset using different algorithm
*C - HWW vs 9K fails to cluster with default parameters, "No markers selected"
-----------
Mon 28jul14
- Brian Steffenson's Ethiopian barley data
- Select Lines by Genotype Experiment: by Platform instead of Lab
*D - Ethiopia genotype experiment isn't found; Data Program MN
- how to merge data from public/private T3 sites
- export data in T3 format
*D - import large omnibus files or multiple files at once. Consult IBP?
- track all loaded files (in public db too) and make available for download
What about manual Curator edits via the web forms? Log them too?
*J table input_file_log, curator_data/uploads/* files
- experiment design tablet integration (Clay)
use sandbox website: when importing to website ignore plot_id or
use production website: give users permission to curate for specific
*C Match Trial_Codes between T3 and Android; Use different ranges between
sandbox and production.
- Back Office as T3's genotype data store via API, T3 as a front office
- Eduard's wheat GBS genotyping data; three-allele SNPs, TAM_105
- Bill Gates, meet T3! Oct 1
- next User Group meeting
- PAG workshop
- loading lines from multiple breeding programs in the same file
*D - Loading line "TAM_105" when "TAM105" already exists silently equates them.
- oat CORE phenotype data being curated by Nick
- oat Line submission template
- better storage table for genotyping data (Clay): new mySQL tables or HDF5
-----------
Mon 21Jul14
- loading 1.3M WCSS1 markers
- how many at a time?
- names like WCSS1_contig3379368_7DL-569
- Clay is extracting the genotyping data for them.
- new Science paper this week
- report from AOWC (JL)
- new Line submission form
- Species required? As a Genetic Character (property)?
- new Genetic Characters
***- curator input form for Institutions
- oat consensus map from Jessica Schlueter; 10 biparentals, GBS, Illumina;
Add an oat sandbox?
COOL CORE Coming from Yung-Fen
- New oat breeder in Florida wants his own T3.
- Conversion from Agrobase to T3
*D Load multiple trials of phenotype data at once? Download in T3 format?
Collisions: lines, traits, markers
- T3/GrainGenes coordination document
- Experiment design tablet integration
use sandbox website and ignore plot_id or
use production website and allow user curation
- SSL certificate from Comodo; apache config needs update; send list of secure pages
- Vic's manuscript outline
- James's progress
- Breeders Database call for submissions
***D Ask Paul re Agrobase users
-----------
Mon 14Jul14
- changes to marker upload page (Clay)
***D revised wording
- American Oat Workers Conf (Dave, JL)
- Private Oat instance and admin support (James)
***D Bug Yong re cost estimate for support from Dave H.
***D curator training for James.
- T3 paper (Vic)
* answer Outline items
- removing saved session when logged in (JL)
- experiment design page, how can tablet device save measurements in
production website? (Clay)
- Allow user (PARTICIPANT) to own a fieldbook record in production database?
- Is BD notifying Vic of submitted files? Yes
- Trial annotations from BD historical nurseries have text description
of fertilization, irrigation, inoculation. Itemize? Use controlled
vocabulary to allow use as criteria for selecting Trials?
-----------
Mon 07jul14
- GBS data - when to order alleles alphabetically.
- Whenever the tag is not aligned to a reference sequence
- Should resolve the case where strand is unknown
- Duplicate markers (identical sequences) within a file are rejected. What
should be done? 1. Omit those markers? 2. Include extra context sequence
to find a difference? 3. Automatically synonymize? 4. Automatically just
make two separate markers?
*D- links from T3 to Barleymap http://floresta.eead.csic.es/barleymap
- "Manage" menu
- report on NABRW Big Data Workshop
- List genotype trials. On homepage
- T3 paper
- http://malt.pw.usda.gov/t3/wheatplus/curator_data/markers_upload_check.php
imperative vs. declarative
***D rewrite
- SSL Certification for tcap
- okay to use Adobe Connect tcapt3 for API group?
-----------
Wed 2jul14, NABRW Big Data Workshop
Homepage search menus are only Phenotype data. EurekaPanel of 404
markers to speed up association analysis. 'Compare Trials' very cool.
Does "Cluster 3D" use currently selected markers only? Karen thinks so.
'hclust' doesn't work, she says. 'pam' failed during class with R error
'duplicate row names not allowed': crossover of simultaneous tmp files?
Query time 115s for 776 lines. Even the mousedrag rotation is slow.
-----------
Mon 30jun14
- HWWGBS marker sequence (29K duplicates out of 82K SNPs) - Jesse
- curator_data/.htaccess to increase default PHP resource limits
*D: Move to php.ini
- API wants genotype data by experiment. Need to cache it this way?
- confer with Clay
- loading Eduard's SNP markers - Vic
*C: Ref vs Alt alleles: Ref always first in sequence [/]?
*C: Parse the genotype calls from the .vcf file for loading.
- *: Dropbox overload/lockup: delete big files
-----------
Mon 23jun14
1. HWWGBS marker sequence (29,865 duplicates out of 82,000 SNP markers) - Vic
- Sometimes there are two identical tags, including the SNP, with different
scores.
loading GBS data (HWW and Cornell Master) - Vic
- curator_data/GBS/ raw files
how to handle GBS tags with more than one SNP
2. Emergency planning and preparedness (federal shutdown or natural disaster)
- Clay: due by end of June. On Clay's machine, Albany NAS, ...
3. Updates to the download page - Clay
independent choice of phenotype or genotype
genotype using one genotype experiment
definition of terms
warnings if large download or bad filter selection
4. Other User Group "Easy"s: managing lists (Dave), JBrowse, Sandbox ->
Playground (Clay), saving datasets for retrieval, N-most-diverse lines (JL)
-----------
Mon 9jun14
- Gina's genotype calls, continuous values 0 to 2.0. "many SNP from the
same 64 bp read and they are often not redundant."
**store in a new column. Drop the old Illumina raw score data.
- Clay: Of 80,000 SNP context sequences in 1AL, 576 are duplicates within
64bp, 5 within 128bp.
- format for NAM progeny names?
- BLAST graphic fixed on tcap. Needed perl package XML::DOM.
- Review of agenda for user group meeting at 3pm
- INSTALL.html instructions being updated
-----------
Mon 2jun14
- agenda for user group conference call next week: Big Projects, Seattle
API report, demo Experimental Design
- Big Projects
1. predefined data sets including filtered lines and markers (suggested by Advisory Board) cf. panzea.org packaged datasets MEDIUM
2. single login using Google account UNCERTAIN
3. imputation of marker data SLOW
4. Automated validation -- T3 should perform checks on newly uploaded data
to identify possible errors (phenotypic outliers, unlikely marker scores
based on local haplotypes, ...). SLOW
5. Diversity panel selection -- T3 should be able to perform an analysis
that identifies the N most genetically diverse lines, based on marker
alleles, in the currently selected set of lines. QUICK
6. HDF5 storage of genotype data using byte storage and compression similar
to TASSEL ( == faster genotype data retrieval) SLOW
7. provide option to query and download genotype data by experiment not
consensus QUICK for download only, MEDIUM for analyze
8. manage lists - a page to view/edit/save/name selections QUICK
9. external access to data and tools (common API) (alternative:
structured data format on web pages (RDF, schema.org) SLOW
10. GBrowse, JBrowse QUICK for as-is, SLOW to add contigs and sequences
C*- final specification of GBS: Data Submission page templates DOIT
D*- links to external resources for Markers: e.g. Barley Give examples.
UNCERTAIN
*P- doing GWAS on iPlant: Semantic Web? Discovery Environment? Agave API?
cf. TASSEL UNCERTAIN
- formalizing Sandbox > Playground: tutorial? more functionality? QUICK
as-is, SLOW to save the Playground
- better integration with Fieldbook: login, API, drive everything
from the app SLOW
- Barley GBS project: GBrowse, naming markers by pseudo-molecule? Relation
to the published barley genome?
- Management Plan ideas from BackOffice project
*D- Cost of Private server for Oat T3 at Albany for Mike Gore, own box or T3's
Cornell CBSU: $1K install, $1K/year for own server. Consult with RLs.
(1/2-time curator from NAMA for public oat db, at Minnesota;
plus
Rines's ARS CRIS?)
*D- postfix for INSTALL.html
-----------
Tues 27May14
-follow up with Katmandoo about Seattle API meeting
-follow up with Trevor Rife
and Field Book development; API? Authentication?
*D Login to T3 with Google ID
-next user group conference call - June 2nd?
-experiment design page: wait for MADII version 2
-GBS data: naming, as contig#_chromosome_position; many multi-SNP tags
-PAG Asia
*D-.png BLAST graphic images not created on tcap, and Score links not working
-Breeders DB for Barley, for Karen Beaubien
-next meeting: Big Projects
-----------
Mon 19May14
- report from crop database API workshop (Dave M)
- Goal: Minimize effort duplication on software tools for breeding
- Customers: small breeding programs in south Africa and Asia
- uniform API to allow exchangeable plug-in tools like GS, cross planning
- At most 10,000 markers per dataset are useful.
- "Genomics Back Office" project
***- SSL configuration of email and logon (Dave H): MX record to graingenes.org
- experiment design page (Clay). "Manage" menu. Add a Trial. Design => Layout
R package "Agricole", MADII design from Trevor Amen (spelling?)
- Iowa State University (Lawrence Lab) national group of maize
researchers evaluating T3
-----------
Mon 12May14
- marker panels for PVP applications (Vic)
- request for database dump from evogene
- Mark has assigned codes for proprietary lines in Cornell Master
- reformatting marker sequences for synthetic (Vic)
*genotyping/show_lines.php should show which genotyping experiment
- API for T3
- T3 overview for Gates meeting
- Allele Conflicts in t3/wheatplus?
- Oat Pepsico project, private data (Gore, Mark)
-----------
Mon 5May14
**V - Do we need to alphabetize alleles for SynOp GBS data? (Vic)
- How to set an existing phenotype data point to 'missing'? (David M)
**D - Allow "--" to mean "set to missing". Document.
- GWAS
- added option for variance calculation method (Clay)
**C,JL - interpreting the Validation plot (Vic).
- Download page - added selection option for genotype data download (Clay)
- User group meeting May 12th (Trial Design demo, Clay)
- New template for phenotype data from multiple trials per file (DaveM)
- Methods section for manuscript (Vic)
- phenotypes for replicated checks not stored uniquely, auto-deletes (DaveM)
**D - Any existing cases of replicates? Don't support in future. (No delete)
- HWW = "Hard Red Winter Wheat" [Eduard]? "Hard White Wheat"? (DaveM)
- Trevor Rife here this week. Lab group Friday 10am.
-----------
Mon 28Apr14
-**D Google spider comes to visit. Need robots.txt
- T3 functional modules
-*** Changing Variety names of lines in T3 that are also in GRIN to a
lab-specific code names.
If we do this (i.e. create new unique names for those 17 lines with WB1
through WB17 as primary T3 names), we are going to have to edit and
re-upload all four of the phenotype trials and genotype data for the
LRpanel. Adult Leaf Rust Response
-**C Download page, genotype data optional
- Compare trials page, plots for more than 2 trials
- marker import file for Eduard's GBS data
-**JL Follow up email to Lee Hickey
interchange between DArTdb, Katmandoo, T3
DArTdb - internal LIMS at DArT
KDDart - data storage and integration platform hosted by DArT
Katmandoo - database and client software (trial management,
pedigree(beta), windows mobile(beta), crossing tool, molecular
marker, inventory.
Fieldscorer 4 Android - collecting trait data in field
-----------
Mon 21Apr14
- GBS data
*C a) Cornell Master - Code the names of the proprietary lines.
b) Eduard - extracting marker sequence, experiment description
*D- Description field for Experiments (experiment_set)
*D- Select Experiments. Maybe Select by Datatype
(Experiment/Marker/Line/etc.) like GrainGenes's Class Browser? Low priority.
*C- Karen: Compare multiple trials; resistant vs. susceptible checks
- Celeste's problems analyzing phenotypes for selected lines - changes
needed to download page, bug in wizard, better trial comparison plots
*C Option to download phenotype data only
- Kathryn, line name changes in leaf rust association panel
*D - questions sent to Eduard; bug him. Two weeks = 28apr
**J- POPSEQ RIL and DH population data, whole-genome sequence from Jesse
http://onlinelibrary.wiley.com/enhanced/doi/10.1111/tpj.12319/
**J- DArT markers, how to post use policy for the sequence information
*C- BLAST on T3, database update should be linked to the marker import script
- Lee Hickey here Thursday and Friday (Australian wheat guy)
*V- marker types: Illumina SNP, Sequenom SNP, DArT, GBS restriction
site, GBS sequence capture, Historical
*V- reloading GBS markers with new A/B definition, with genotype data changes
-----------
Mon 14Apr14
- GBS data
1. review page Marker Alleles and Sequences.
***D "-" => "N"; add marker_type
marker types: Illumina SNP, Sequenom SNP, DArT, GBS restriction site, GBS sequence capture, Historical
**D Query for all markers by type; query for all unused marker_types
**C add Platform to Experiment report
* editor for Genotype Experiments
2. review marker import Marker Upload page. checking same name
different sequence and possible errors
***C: Must update BLAST database whenever we load new markers.
3. loading .vcf genotype data from Eduard
***D: BLAST using ambiguity codes?
*Assign ambiguity codes for multi-SNP markers of type "GBS restriction site"
!Decision!: Don't try to match .vcf genomic-sequence-aligned SNPs to
restriction-site-anchored SNPs with no genome position.
**D Marker annotation of contig position, build
Marker name: contig3917765_1al-5481 or IWGSC2012_3917765_1al-5481
For the different versions of some chromosome-arms: IWGSC2012_4as_v2-1185913
Annotation: "IWGSC draft 2012", "Mayer et al., submitted", no link
- Stable Oats
- Trial and Experiment design web page
- SSL certificate for website
- "What's New" text overlapping menu "Allele Data Conflicts" (Dave M)
-----------
Mon 07Apr14
GBS data
1. primary/alternate (A/B) allele determination
*P2. Use Illumina rule for coding GBS data? No, alphabetical for 5'
restriction-site-anchored GBS tags.
*P Use ambiguity code for other SNPs in same tag.
Don't try to find matches between .vcf and 5'-RE-anchored tags.
Add markers.SNP-position-in-sequence field.
Compare Illumina vs GBS as Homologs of type Overlap?
*D Update http://malt.pw.usda.gov/t3/wheatplus/snps.php for GBS SNPs.
3. we should record the version of contig assembly
Jessie Poland - how does he pick reference sequence? First allele found.
Eduard Akhunov - how to interpret alternate alleles
T3 Oat loading -
*C Blank page loading genotype data.
Clone of T3 for Mike Pumphrey DRRW.
- Passworded. On malt, tcap, or iPlant. By late June.
developing training for T3 curators (Vic)
- Maize (Carolyn Lawrence) getting a demo tomorrow.
- Offer a PBTN workshop on setting up and curating a T3?
bug fixes to the Github T3 by Kenneth Chan, U. Queensland
Correcting bad genotype data for 2 lines from Araby (Vic)
Email problems on pw.usda.gov domain – authsmpt.com account works as a
temporary fix (Clay). Update from Dave Hane.
Hard drive for malt machine (Dave Hane)
New trial and field layout generator. (Clay)
T3 User group (May 12th)
-----------
Mon 31mar14
1. T3 wheat GBS. Lynn provided list of 60 germplasm lines that have
not been loaded into T3 and should be removed from genotype results
file.
Jessie Poland uses "H" for heterozygous - The import scripts has
been changed to accept this.
*JL We need to decide what rule we will use first. Try extended
Illumina rule
Eduard Akhunov - coordinate file from Shichen Wang. The alternate
allele has 2 values for 10% of the data. Should this be stored in the
database as is, changed to ambiguity code, or disregarded as bad data?
***D Dave M. will email Eduard to get more information
HWW – Test checking for synonyms (Vic)
**CB change summary into a 2x2 table (name match, seq match) (in
db, not in db), add download button for large imports
genotype results import is now quick for new data. I also have new
tool to check import file before loading. This will quickly tell you if
required lines or markers are have not been loaded.
**CB combine verification step into first stage of import
2. Automatic validation of marker data, what do we want?
3. Status of Peter Morrell annotations. We received annotations once
upon a time, and work has been done to integrate them as marker
properties. Do we have a regular pipeline to do this kind of think when
new markers come in? What is the integration with GrainGenes, which (I
think) should store these annotations?
4. T3 Oat
field layout import? not many trials have field information, when we
have the resources we can work on this
when more than one observation were provided for a given line of a
given trait (e.g., day to heading is "71/88" for "Florida 167"), would
you assign it as missing or keep it as is?
Known erroneous observations were discarded (kg/ha converted from
bu/a); correct conversion will be input into access database
(COOL_CORE).
Revisiting observations in the database: remove unnecessary ones
and convert some of them (if applicable) to meaningful one.
5. Can we add new hard drive to malt machine? I have had to disable
the updating of the sandbox websites because there is not enough disk
space on the malt machine. Dave H will work on this
6. Email broken on malt, wheat, and tcap machines. Can we setup a
commercial service?
-----------
Mon 23mar14
- GBS data loading
1. Mark Sorrells - ambiguity codes, marker names, loading lines
*JL Also ask Lynn if they have been using markers names
* 2. Eduard Akhunov - coordinate file from Shichen Wang (Dave): to Clay
3. HWW - (Vic)
- Experiment
design on T3 using agricolae
*JL i) List of experimental lines and list of checks; ii) Specify design type; iii) Specify parameters of design; iv) Specify field
size: could be incomplete block size + number of rows; v) Tool produces T3-format annotation file and field layout file, and Android field book file.
- ARS scientific computing assessment
1. ARS science DMZ
2. Hybrid Cloud
3. Data Commons
- COOL data in Oat Chex
1. 747 lines and 5000 markers loaded; ready for genotyping data
2. Trait discussion in Global Oat forum
* merging traits in Access? In T3 upload file?
*JL Gerry’s list of traits to be added; Redundant rows removed; What to do with disease trait scales?
3. Trials nearly ready to load
4. bushels as a measure of weight
- inquiry from CIMMYT, Celso Carreiro
*- iPlant backup server via mysqldump to DataStore
****- Put this agenda file on Dropbox.
-----------
Mon 17mar14
1. mysql replication through firewall. What ports are available?
Configuring mysql server. (David H)
2. how to deliver database content to other copies of T3? (Clay, Dave M)
* mysqldump without the user passwords
* how could we merge into existing data?
3. Analyze => Haplotype Data seems not to be showing the right
phenotype experiments. (Dave M)
* CAP Core, all trials, UCR-1H markers: only 3 lines scored? checks only?
4. GBS from Eduard (Dave M)
5. ASA CSSA SSSA National Meeting Nov 2- 5 in Long Beach , CA TCAP
Session by Kevin Smith (Jean-Luc)
6. ontologies: are they a good thing or a cumbersome thing, or both?
(Jean-Luc)
7. T3 paper (Vic); Crop Science "Review and Interpretation"
8. Marker upload script - the sequence match checking works with large
files (Clay)
9. Allele Conflicts analysis - changed to show between experiment
conflicts (count and percentage).
- requesting T3 users to cite it when publishing; notify us?
- curation of COOL data for Oat Chex
* - trait conversions, e.g. bu/A to kg/ha? 15 kg/bu
* - line names
** - load Trials (planting date conversions)
-----------
Mon 10mar14
- T3 user group call 3pm EST
- mysql replication through firewall. What ports are available? Configuring
mysql server (David H)
i*** iPlant IP range?
- submitting multiple files as multi-select and as .zip. (Dave M)
- Analyze => Haplotype Data seems not to be showing the right phenotype
experiments. (Dave M)
- GBS from Eduard (Dave M)
- CSR data from Sarah Grogan (Clay)
- Data Usage Policy (Dave M)
- White Paper from National Barley Improvement Committee, March 10-12
(Jean-Luc)
- ASA CSSA SSSA National Meeting Nov 2- 5 in Long Beach , CA TCAP Session by
Kevin Smith (Jean-Luc)
- done: download Cluster coordinates table
- MaizeGDB to get Ed Buckler's phenotype data; Maize meeting is soon
-----------
Mon 3mar14
1. Winnipeg Oat Conference (Dave M)
- DUDE queries, e.g. Extract Balanced Dataset
2. road map for additions to T3 (Jean-Luc)
**** - GBS: tell JL what we need from Eduard re VCF
- experiment design generator. See Cassavabase.
- imputation
- automated validation of import data (phenotype? genotype?)
- download genotype data for specific experiment (not consensus)
3. User group conference call next week (Jean-Luc)
4. Tassel v5 now works
with hapmap download (Clay)
5. genotype consensus data, identifying lines to delete, allele
conflicts (Clay, Jean-Luc, others)
- ask Mary and Tyler Gordon, Aberdeen
- query for lines 99% identical
6. lme4 package on tcap machine (David H)
7. SSL web certificate
8. mysql replication through firewall
- TCAP Scientific Advisory Board recommendations
A. funding for info systems detached from commodities and co-leveraged.
Gramene, Morrell SNP annotations, GrainGenes table from Selected Markers
**** B. Link to Data Usage Policy on homepage. Revise table labels. queue.php
prompt to add Toronto info as Comments.
**** - move Submit button on Data Submission page
C. My Project selections. My Archive (button on Download page)
- submitting multiple files as multi-select and as .zip.
- Analyze => Haplotype Data seems not to be showing the right
phenotype experiments.
- Greenhousebook App (Karen Beaubien)
- template Versions vs. file timestamps are a pain for new
installations
-----------
Mon 24feb14
- Lme4 package on tcap machine (David H); libppl, libgmp
- SSL certificate (Jean-Luc, David H): JL's court
1. Tasselhapmap file for SynOp High Density 2013 map: convert map
positions to integer.
2. "Open with Tassel" fixed by updating the .jnlp file. Also added
compatibility instructions on web page
Terry CS may know how to pass data file arguments to "Run Tassel"
3. Tassel v5 hapmap file needs markers sorted by name when position is
not unique.
4. iPlant backup server - discuss configuration and testing
- mysql replication
Mysql ports blocked at WRRC firewall, tho' open on OCIO's.
Replication requires separate innodb file for each table.
5. iPlant image available "CropDataFarm" (Dave M)
8. Loading phenotype results for multiple trials at once (Dave M)
* Remove '*Breeding Program Code' from template
* Change '*Line Name' to '*Line Name or Trial Statistic'; yellowbox
?: Other summary statistics supported? No
9. Question from Kathryn Turner on markers for "NSGC 9K spring"
Select Markers "Infinium 9K" doesn't list NSGC_wheat9K_spring.
- GrainGenes 3.0
-----------
Tue 18feb14
- backup server (iPlant/Amazon)
- information from iPlant WebEx (login, math libraries, etc)
- iPlant collaboration,
www.iplantcollaborative.org/forms/iplant-extended-collaborative-support
- marker import with sequence matching
- loading phenotype results into BDB from SAS output (Clay Sneller)
- interoperability with IBP - platform independent data (Dave Edwards)
- How to update/integrate multiple T3 databases (Scott Haley). Do we want to
1. develop ways to synchronize multiple databases
2. use one database and add more ways to protect private data
3. analyze data from multiple databases (*Simple Semantic Web?)*
* Export "T3 format". Accept more data in fewer files.
- problem loading lme4 R package
- Jeff's LPmerge is now published
- API for interfacing with Android Field Book
- https for login password security? SSL certificate
- Akhunov SynOp High Density map loaded, 39K markers
-----------
Mon 10feb14
1. Import marker information, check for sequence match. (Clay)
2. T3 server mirror - compare options on iPlant and Amazon (David M, Clay)
3. CSR - CropScan imports done. (Clay)
4. Loading Oat data into database
5. Map selection required for download page. Should this be optional?
(Jean-Luc)
6. NSGC data delivered as T3-ized names. How to get PI numbers? (Vic)
**** - Check synonyms for all GRIN accessions, > Vic
- Replace names using VLOOKUP or R, on download only.
*- Deleting genotype results for a set of lines in one experiment:
*- Downloading the genotype clustering results as a table (Tyler Gordon)
- Presentation to iPlant about T3 image "My Crop Datafarm", Wed 2pm EST
**** iPlant connection info to JL et al
- POOL's pedigree display: selectable depth, table instead of image
- https for login password security?
-----------
Mon 3feb14
1. Import marker information, check for sequence match.
2. Wild Oats Grainworld Monday, Feb 24 & Tues Feb 25 Winnipeg Manitoba
* Use T3-ized names. Trial annotations, means. DArT markers for UEMOPN lines.
* Gerry: loaded on wheat
3. Australia wheat T3 - visit to Cornell on April 24-25
4. For the main TCAP panels, there is a desire to have calculation of Q and
K matrices done centrally, so that everyone uses the same matrices. I
think we would have to ask the objective coordinators what they think.
5. Jorge wanting to have better access to the annotations that Peter
Morrell's student's software computes. (Jean-Luc)
* Load in GrainGenes.
* Ask Peter if he has for wheat.
* i_11_10815 Barley Physical Map dead on malt (not on tcap)
6. User group schedule for next few months. * Next: March 10*
a) Cropscan importing and other CSR features.
b) GBS data - data requirements, naming
c) use of Field Book App
d) how to present raw genotype data instead of consensus
7. T3 server mirror. Order new machine or use Cloud.
Supermicro prices are now no better than Dell.
* Jarek/Qi re CBSU experience on Cloud services
8. Setting up JBrowse on malt. (David M)
9. CSR - CropScan imports (Clay)
10. Android Field Book - app requirements for direct download and upload
from server
11. Map selection required for download page. Should this be optional?
(Jean-Luc)
12. NSGC data delivered as T3-ized names. How to get PI numbers? (Vic)
-----------
Mon 27jan14
1. Marker selection using pattern matching. I increased the performance of
this query by overwriting results of previous selection.
2. Import marker information, check for sequence match.
3. Australia wheat T3 - setup this winter then work with new
database/programmer in summer. Should someone go to Obregon in March to
talk about T3?
4. Global Oat Genetics Database - proposal
*- JL's UEOMOPN data from GrainGenes
- Gabe Gusmini, Pepsico, visiting next week
- metabolomics (Pepsi, Mike Gore, Mark, JL)
*5. Winnipeg Feb 27-28, Gabe Gusmini, Joe Lutz, Bruce Roskens
6. Being able to disaggregate markers scores by genotyping experiment
rather than having to take the "majority rules" value. Basically, Mary
Guttieri just thinks the majority rule method is a bad idea. To get
genotypes from specific genotyping experiments will of course complicate
the interface and make precalculation of the 2D alleles table difficult (?).
* Download Allele Data?
7. For the main TCAP panels, there is a desire to have calculation of Q
and K matrices done centrally, so that everyone uses the same matrices.
I think we would have to ask the objective coordinators what they think.
8. I think
I mentioned Jorge wanting to have better access to the
annotations that Peter Morrell's student's software computes.
9. User group schedule for next few months.
I can have something to present
on Cropscan importing and other CSR features. We could also discuss GBS
data and use of Field Book App.
10. Updates on email from T3 server and setting up mirror copy of T3 (David
H)
11. Setting up JBrowse on malt.
- trait name abbreviations (Clay Sneller, Mary Guttieri)
- submitting 270 Trials at once (Clay Sneller)
* summary of Allele Conflicts by number per Line and per Experiment
-----------
Tue 21jan14
1. Some discussion of the T3 business model.
2. business from Australia! Lee Hickey, wheat data on T3 or a T3 clone.
3. business from oats! (Pepsi, Gen Mills) CORE, Nick Tinker
4. central database with public data, peripheral ones with private:
how to update new public data into the privates?
5. notes from PAG
* requirements doc for merging private/public T3 data. Within the database
using groups? merging databases? repository of loadfiles?
* rename GBS markers iff > 63bp and same as a MAPPED marker
-----------
Mon 6jan14
- Tablet tools - added download of trait definition for Android Field Book
(Clay)
- Analysis Menu - added selection index to analyze menu (David M)
* implement Rank
**** Back button resets weights and reverse box.
- Calculate CSR index - added custom label when formula changed, reworked
options for saving the index calculation (Clay)
- GBS data from Edward - VCF file
- PAG - T3 posters
- fixed: checkline > 1 causes all phenotype data to be ignored
- fixed: blank Trait columns in phenotype data assigns values to wrong Traits
- plot-level-calculated means vs. Means file means
- Too many traits! Subcat by Trait vs by Trial
* display_phenotype.php *two* decimal points