Should we exclude datasets where the simulated network has worse BIC than the raxml-ng best tree?

[lutteropp]

It sometimes happens that the simulated network is rejected by BIC, in favor of the best tree inferred by raxml-ng. What should we do with these datasets?

A) Include them in our experiments, wasting hours on NetRAX calls that will infer a tree anyway.
B) Re-simulate the dataset if this happens. Do not run NetRAX on it. Keep the problematic dataset in an extra folder (to use for post-analysis of simulation parameters).
C) Re-simulate the dataset if this happens. Do not run NetRAX on it. Discard the problematic dataset.

I vote for B. (Or C, if we are ok with "well, likely it was an ultrametric network with a fake reticulation again")

[lutteropp]

(reopened because this still happens from time to time. It was not just due to some bug)

[lutteropp]

Or we can go with A, add writing to a logfile to NetRAX, and then post-evaluate whether NetRAX did encounter the correct network and just discarded it due to BIC issues.

[stamatak]

That (A) sounds like a good idea, but I wouldn't call it BIC issues, maybe just too weak signal/not enough data to infer the network, as, if I remember correctly, you said that with longer MSAs it's easier to infer the network correctly.

…

On 18.01.21 15:26, Sarah Lutteropp wrote: Or we can go with A, add writing to a logfile to NetRAX, and then post-evaluate whether NetRAX did encounter the correct network and just discarded it due to BIC issues. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#40 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGXB6XSXBKDGDYZ24VBTJLS2QZG3ANCNFSM4WGPWYRA>.

-- Alexandros (Alexis) Stamatakis Research Group Leader, Heidelberg Institute for Theoretical Studies Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology www.exelixis-lab.org