Moderation
Experts in killer whale bioacoustics collaborate to ensure that predictions from the OrcaHello real time inference system are validated as efficiently as possible. The goal is semi-automated notifications that Southern Resident Killer Whales are audible (and therefore inferred to be present near an Orcasound hydrophone) as rapidly as possible, but also with high reliability (i.e. a false positive rate near zero).
During beta testing in 2020-2023, the moderators were:
- David Bain
- Scott Veirs
- Val Veirs
In 2024-5, we would like to add more moderators, ideally in a range of time zones around the globe. That way, if most of us are sleeping here in the Pacific Time zone, a domain expert on the other side of the planet might greatly reduce our collective response time! If you'd like to help, please read through the rest of this section and all of this wiki page first. Then send an email to info@orcasound.net or join the Orcasound Zulip to volunteer.
If you're an expert at listening for orcas, we'd welcome your help as a moderator. At a minimum, you should be able to differentiate between Southern Resident and Bigg's killer whales, humpbacks, and other common Salish Sea sounds. Ideally, you should also be able to infer which pod is the source of SRKW calls, and be familiar with at least 5-10 of the most common calls in the SRKW repertoire.
As of late 2022, moderators are notified of recent OrcaHello detections via a Mailman discussion list. You can subscribe to this listserv here -- lists.orcasound.net/listinfo.cgi/orcahello-detection-orcasound.net
To moderate a candidate, you will need to be added as an OrcaHello moderator. An Orcasound administrator with Azure access can help do this for you. Contact admin@orcasound.net to get started.
Anyone can receive these notification emails, but many of them are false positives, so if you aren't a moderator you may want to receive a daily digest, or you not want to subscribe at all. If you're interested in acoustic detections of SRKW calls that have been confirmed by a moderator, you may instead want to subscribe via other Orcasound notification channels.
This is the recommended procedure (to maximize reliability and minimize latency):
- As soon as possible after receiving a notification as a moderator, navigate to the OrcaHello moderator UI and authenticate.
- Review the most recent candidate(s) which by default will be at the top of the queue, or -- if many false positives occurred recently -- sort to review the highest-confidence recent candidate.
- Be sure to listen to the full 60-second candidate.
- If you find a true positive, check "Yes" in response to the question "Was there an SRKW call in this clip?" (see screenshot below)
- Add labels (using the procedure and standards described below)
- Add comments (a brief synopsis or notes beyond what the labels provide)
- Hit submit (which will issue a notification email to subscribers)
- At a minimum for true positives, add comma-delimited tags for pod(s) and any individuals known to be present (e.g. from visual IDs). Ideally, also label any call types that you recognize within the candidate (e.g.
S04, S16). - For false positives, or for "unknown" candidates (for which you cannot confidently assign a sound source), add labels for at least one of the model predictions (to start characterizing what caused the false positives) and any other signals of interest you hear in the candidate. Ideally, any label you add will be consistent with existing cloud of OrcaHello tags and any new labels that you create will aspire to be consistent with the annotation efforts of the broader Orcasound and bioacoustic communities (see e.g. the [signals-srkw public repo on Github])https://github.com/orcasound/signals-srkw).
If you are unsure if a label exists yet for a particular sound, you can browse the Orcasound signal dictionary for OrcaHello candidates (Google spreadsheet with ~100 tags in these broad classes: Biophony, Anthrophony, Geophony, General, and Systemic). To hear examples of particular sounds that have been labeled, browse the Tags section of the OrcaHello Dashboard; filter for All time and you can review and listen to all available samples for a particular tag. In the same spreadsheet, you will find other labels used by other annotation efforts, like HALLO and the BC Hydrophone Network, with which Orcasound is coordinating in pursuit of common metadata and annotation standards for orca bioacoustics.
To efficiently improve the OrcaHello system, as a moderator you can create Github issues in the aifororcas-livesystem repo for bugs or other problems you experience. For each issue, please use a label to characterize it for the dev team, e.g. "bug" (something's not working) and/or "moderatorportal."
Simple rules when deciding whether to file a bug or issue:
- If there was ever an "event" with whales of any kind but you did not receive an email - please file an issue.
- Create separate issues for separate hydrophones. For example, if on 11/2/21 there were 2 independent events on Bush Point and Port Townsend and you did not receive an email for either, please file 2 different issues.
- Create separate issues for different events happening on the same day. For example, if there were 2 different events -- one at 9 AM and one at 9 PM -- on the same hydrophone on the same day, file 2 separate issues.
- When in doubt, file separate issues!
What is useful when filing an issue is a full description of the bug with facts and pictures, minus any conjecture.
We'll work on creating an issue template later, but for now consider including, at least:
- Which hydrophone?
- When did the KW event start?
- When did the KW event end?
- What was the nature of the event? Faint calls, killer whales/ humpback?
- Was the container running? (If you have this info?)
You can also create issues with ideas for feature requests or enhancements to the existing moderator portal UI. Please use the "enhancement" label for these issues.