export_gzipNexSON.json not returning internal node labels that have non-trivial info.

[mtholder]

Not sure if this is a bug.
It appears that http://www.reelab.net/phylografter/stree/edit/3
should have bootstrap proportions as internal node labels, but those numbers don't seem to make it through to the NexSON (so they are also missing in the phylesystem repo that was created from the phylografter nexson dump).

[rhr]

It seems that branch length is the only non-id node attribute currently being serialized into nexson. But the node table has fields for bootstrap_support, posterior_support, and (other_support, other_support_type). And age_min and age_max. Since Peter implemented this, I'll assign him to it.

[pmidford]

I want to be clear on the semantics for the ot:nodeLabelsRepresent - the intent appears to be that this applies only to interior nodes (and corresponds to the clade_labels_represent field in the stree table). So what ever is in the label field of a terminal node is passed through regardless of nodeLabelsRepresent? Would cladeLabelsRepresent be a better predicate in the NexSON?

[pmidford]

@rhr - yes, there are several things that are lost in NexSON export, because they were never defined to be in the NexSON spec - thus my ticket #115

[rhr]

@pmidford : right, terminal nodes are assumed to be be taxon labels of some sort. clade_labels_represent refers to internal node labels.

Not sure how to resolve this issue - or even if it should be resolved?

[pmidford]

Ok, here's a list of phylografter fields that have information to ought to be captured in the switchover to phylesystem. In several cases comment or label fields contain text that should be captured by a controlled vocabulary elsewhere (this isn't news, but there are additional fields where such text appears on occasion). Several of these probably ought to be in annotations rather than fields (e.g., last modified/last modified by). I propose that the most expedient to handle these would probably be to support them temporarily, but only as NexSON 0.0.0 vocabulary and let peyotl translate into annotations. They can be
later deprecated or just left as NexSON 0.0.0 only.

Per the discussion at the beginning of the monday call, I'm planning to use clade labels, rather than node labels since they only apply to internal nodes.

There are several fields that have never been used outside of study 20, which was a made-up test. I could implement them for phylografter and other NexSON processors could ignore them for now.

study

• ot:comment
• @Label
• ot:last_modified - ought to be converted to annotation
• ot:uploaded - ought to be converted to annotation

tree

• ot:tb_tree_id - treebase id for tree; there is data here
• ot:contributor - convert to annotation? (not always = study curator)
• ot:uploaded - convert to annotation?
• ot:branch_lengths_represent
• ot:branch_lengths_comment
• ot:clade_labels_represent
• ot:clade_labels_comment
• ot:author contributed - annotation?
• ot:comment

node

• ot:age, ot:age_min, ot:age_max - actually never appear in a non-test study

edge

• ot:bootstrap_support
• ot:posterior_support
• ot:other_support, ot:other_support_type - never appear in a non-test study

[mtholder]

Hi Peter,

I'm a little unhappy with the combination of
length+branch_lengths_represent
and
label+clade_labels_represent
for general information about the tree.

This seems fine for an import settings since newick has only a label
and a length field for each clade (or edge).

But it seems ugly to have a generic "label" and then a separate field
saying what that is in our data model.

It looks to me like we have (in the nexsons in phylesystem)
branchLengthModes of:
ot:changesCount
ot:substitutionCount
ot:time
ot:posteriorSupport
ot:bootstrapValues
It seems like the first 3 of these types should be emitted as @Length

• branchLengthMode, but the latter two should go in
  ot:posteriorSupport and ot:bootstrapValues fields in the edges of the
  tree. I say that because the first 3 are conventional lengths (with
  the branchLengthMode helping to diagnose the units of those), while
  the latter two are statistical support statements.

I should have noticed this earlier, but I missed it. I wonder if this
is a good time to fix this in the NexSON tags spec (before we start
tracking the history of studies "for real").

Does that make sense?

Mark

On Fri, Jun 6, 2014 at 12:52 PM, Peter Midford notifications@github.com wrote:

Ok, here's a list of phylografter fields that have information to ought to be captured in the switchover to phylesystem. In several cases comment or label fields contain text that should be captured by a controlled vocabulary elsewhere (this isn't news, but there are additional fields where such text appears on occasion). Several of these probably ought to be in annotations rather than fields (e.g., last modified/last modified by). I propose that the most expedient to handle these would probably be to support them temporarily, but only as NexSON 0.0.0 vocabulary and let peyotl translate into annotations. They can be
later deprecated or just left as NexSON 0.0.0 only.

Per the discussion at the beginning of the monday call, I'm planning to use clade labels, rather than node labels since they only apply to internal nodes.

There are several fields that have never been used outside of study 20, which was a made-up test. I could implement them for phylografter and other NexSON processors could ignore them for now.

study

ot:comment
label
ot:last_modified - ought to be converted to annotation
ot:uploadedDate - ought to be converted to annotation

tree

ot:tb_tree_id - treebase id for tree; there is data here
ot:contributor - convert to annotation? (not alway = study curator)
ot:uploaded - convert to annotation?
ot:branch_lengths_represent
ot:branch_lengths_comment
ot:clade_labels_represent
ot:clade_labels_comment
ot:author contributed - annotation?
ot:comment

node

ot:age, ot:age_min, ot:age_max - actually never appear in a non-test study

edge

ot:bootstrap_support
ot:posterior_support
ot:other_support, ot:other_support_type - never appear in a non-test study

—
Reply to this email directly or view it on GitHub.

Mark Holder

mtholder@gmail.com
mtholder@ku.edu
http://phylo.bio.ku.edu/mark-holder

Department of Ecology and Evolutionary Biology
University of Kansas
6031 Haworth Hall
1200 Sunnyside Avenue
Lawrence, Kansas 66045

lab phone: 785.864.5789

fax (shared): 785.864.5860

[pmidford]

Yes, I think the treatment of branch lengths is well established and
there aren't multiple fields for alternate branch length values (there's
one length, with different interpretations). Unlike branch lengths,
there are three fields for support statistics (bootstrap, posterior,
other statistic + name of other statistic), so we would lose information
by just saving on the label (in case of multiple statistics). The
clade_labels_represent could be considered as an option that pertains to
exporting to Newick, so not saving this into NexSON loses nothing for
synthesis.

So, this works for me.

Rick?

-Peter

On 6/6/14, 2:50 PM, Mark T. Holder wrote:

Hi Peter,

I'm a little unhappy with the combination of
length+branch_lengths_represent
and
label+clade_labels_represent
for general information about the tree.

This seems fine for an import settings since newick has only a label
and a length field for each clade (or edge).

But it seems ugly to have a generic "label" and then a separate field
saying what that is in our data model.

It looks to me like we have (in the nexsons in phylesystem)
branchLengthModes of:
ot:changesCount
ot:substitutionCount
ot:time
ot:posteriorSupport
ot:bootstrapValues
It seems like the first 3 of these types should be emitted as @Length

• branchLengthMode, but the latter two should go in
  ot:posteriorSupport and ot:bootstrapValues fields in the edges of the
  tree. I say that because the first 3 are conventional lengths (with
  the branchLengthMode helping to diagnose the units of those), while
  the latter two are statistical support statements.

I should have noticed this earlier, but I missed it. I wonder if this
is a good time to fix this in the NexSON tags spec (before we start
tracking the history of studies "for real").

Does that make sense?

Mark

On Fri, Jun 6, 2014 at 12:52 PM, Peter Midford
notifications@github.com wrote:

Ok, here's a list of phylografter fields that have information to
ought to be captured in the switchover to phylesystem. In several
cases comment or label fields contain text that should be captured by
a controlled vocabulary elsewhere (this isn't news, but there are
additional fields where such text appears on occasion). Several of
these probably ought to be in annotations rather than fields (e.g.,
last modified/last modified by). I propose that the most expedient to
handle these would probably be to support them temporarily, but only
as NexSON 0.0.0 vocabulary and let peyotl translate into annotations.
They can be
later deprecated or just left as NexSON 0.0.0 only.

Per the discussion at the beginning of the monday call, I'm planning
to use clade labels, rather than node labels since they only apply to
internal nodes.

There are several fields that have never been used outside of study
20, which was a made-up test. I could implement them for phylografter
and other NexSON processors could ignore them for now.

study

ot:comment
label
ot:last_modified - ought to be converted to annotation
ot:uploadedDate - ought to be converted to annotation

tree

ot:tb_tree_id - treebase id for tree; there is data here
ot:contributor - convert to annotation? (not alway = study curator)
ot:uploaded - convert to annotation?
ot:branch_lengths_represent
ot:branch_lengths_comment
ot:clade_labels_represent
ot:clade_labels_comment
ot:author contributed - annotation?
ot:comment

node

ot:age, ot:age_min, ot:age_max - actually never appear in a non-test
study

edge

ot:bootstrap_support
ot:posterior_support
ot:other_support, ot:other_support_type - never appear in a non-test
study

—
Reply to this email directly or view it on GitHub.

Mark Holder

mtholder@gmail.com
mtholder@ku.edu
http://phylo.bio.ku.edu/mark-holder

Department of Ecology and Evolutionary Biology
University of Kansas
6031 Haworth Hall
1200 Sunnyside Avenue
Lawrence, Kansas 66045

lab phone: 785.864.5789

fax (shared): 785.864.5860

—
Reply to this email directly or view it on GitHub
#121 (comment).

[pmidford]

Ok, I'd like some feedback on support statistics for edges. Here's an example edge element:
{

"@source": "node518",
"@about": "#edge519",
"@length": 0.000876523,
"@target": "node519",
"meta": [
    {
        "@datatype": "xsd:double",
        "@property": "ot:bootstrap_support",
        "@xsi:type": "nex:LiteralMeta",
        "$": 0.78
    }
],
"@id": "edge519"

},

I'm asking in particular about the xsd tagging and emitting the value as a float (rather than as a string). Lengths are already done this way, so I hope this won't cause a problem (but I can save as a string if that is the consensus choice.

[pmidford]

Final list -
Ok, here is the final list of tags I will be adding to resolve this (reflecting @mtholder's comments and adjusting the rendering to camel case). @blackrim will you be ready to handle these when phylografter starts writing these?

study

• @Label - not a meta tag
• ot:comment
• ot:lastModified - ought to be converted to annotation
• ot:uploaded - ought to be converted to annotation

tree

• ot:tbTreeId - treebase id for tree; there is data here
• ot:contributor - convert to annotation? (not always = study curator)
• ot:uploaded - convert to annotation?
• ot:branchLengthsComment
• ot:cladeLabelsComment
• ot:authorContributed - convert toannotation? value is xsd:boolean true (absent = false)
• ot:comment

node

node

• ot:age, ot:ageMin, ot:ageMax - supported, but only used in a test study, value is xsd:double

edge

• ot:bootstrapSupport - value is xsd:double
• ot:posteriorSupport - value is xsd:double
• ot:otherSupport, ot:otherSupportType - never appear in a non-test study

[mtholder]

great. thanks. can you update https://github.com/OpenTreeOfLife/phylesystem-api/wiki/NexSON ?

[pmidford]

Working on it now.
On 6/12/14, 11:43 AM, Mark T. Holder wrote:

great. thanks. can you update
https://github.com/OpenTreeOfLife/phylesystem-api/wiki/NexSON ?

—
Reply to this email directly or view it on GitHub
#121 (comment).