No data of specific projects

[PieterjanVerhelst]

I want to download data for several animal_projects to which I normally have access. However, for the following animal_project_codes no data is generated after running the code underneath:

• EMMN
• 2011_Loire
• 2004_Gudena
• life4fish

data <- get_acoustic_detections(my_con, scientific_name = "Anguilla anguilla",
                                animal_project_code = "2014_Nene",
                                limit = FALSE)

[damianooldoni]

Thanks @PieterjanVerhelst, I checked as well and I can reproduce what you said:

> data_anguilla <- get_acoustic_detections(scientific_name = "Anguilla anguilla",
+                                 animal_project_code = "EMMN",
+                                 limit = FALSE)
> nrow(data_anguilla)
[1] 0
> data <- get_acoustic_detections(animal_project_code = "EMMN", limit = FALSE)
> nrow(data)
[1] 0
> animals <- get_animals(animal_project_code = "EMMN")
> animals %>% list_values(column = scientific_name)
1 unique scientific_name values
[1] "Anguilla anguilla"

Based on the data retrieved with etn package it seems that there are tagged animals for EMMN but they have never been detected.

[PieterjanVerhelst]

So the detections are missing? Or how should I interpret this? Before the adjustments of the etn package, I was able to extract this data with the package.

[peterdesmet]

@PieterjanVerhelst, @aubrivliz and I updated/corrected some aspects of the views this morning. Can you install the latest version of the package (devtools::install_github("inbo/etn")) and try again. The problem might be resolved.

[PieterjanVerhelst]

@peterdesmet the issue persists; still no data coming out from the function-call.

[damianooldoni]

Indeed, I can confirm it.

[peterdesmet]

Thanks.

There are detections. You can find them by looking on acoustic_tag_id (@PieterjanVerhelst meaning you can use that for now). Apparently there is noanimal_id or animal_project_code associated. I have to figure out why.

emmn_animals <- get_animals(scientific_name = "Anguilla anguilla", animal_project_code = c("EMMN"))
emmn_acoustic_tag_id <- emmn_animals %>% list_values(acoustic_tag_id)
# 26 unique acoustic_tag_id values
get_acoustic_detections(acoustic_tag_id = emmn_acoustic_tag_id, limit = TRUE)
# A tibble: 100 × 20
#    detection_id date_time           tag_serial_number acoustic_tag_id animal_project_code animal_id
#           <int> <dttm>              <chr>             <chr>           <chr>                   <int>
#  1    118566863 2007-10-03 20:20:40 1078152           S256-234        NA                         NA
#  2    118566867 2007-10-03 20:22:58 1078152           S256-234        NA                         NA
#  3    118566879 2007-10-03 20:43:21 1078152           S256-234        NA                         NA
#  4    118566172 2007-10-03 22:56:37 1078152           S256-234        NA                         NA
#  5    118566173 2007-10-03 22:57:38 1078152           S256-234        NA                         NA
#  6    118566174 2007-10-03 23:03:32 1078152           S256-234        NA                         NA
#  7    118566177 2007-10-03 23:12:09 1078152           S256-234        NA                         NA
#  8    118566178 2007-10-03 23:12:37 1078152           S256-234        NA                         NA
#  9    118566179 2007-10-03 23:13:23 1078152           S256-234        NA                         NA
# 10    118566181 2007-10-03 23:14:42 1078152           S256-234        NA                         NA
# … with 90 more rows, and 14 more variables: scientific_name <chr>, acoustic_project_code <chr>,
#   receiver_id <chr>, station_name <chr>, deploy_latitude <dbl>, deploy_longitude <dbl>,
#   sensor_value <dbl>, sensor_unit <chr>, sensor2_value <dbl>, sensor2_unit <chr>,
#   signal_to_noise_ratio <int>, source_file <chr>, qc_flag <chr>, deployment_id <int>

[peterdesmet]

Ok, I see what the issue is, there are two tags:

tag_serial_number	acoustic_tag_id	acoustic_tag_id_alternative	manufacturer	tag_type + subtype
1078152	A69-1105-234	S256-23	VEMCO	acoustic > animal
73500234	S256-234	A69-1105-234	THELMA	acoustic-archival > sentinel

@PieterjanVerhelst, the one you want is the second one.

However, every detection is initially associated with both (duplicating the detection) and then filtered based on a number of fields. The 73500234 detections are filtered out because all detections fall after the battery_estimated_enddate, which is 2007-10-03.

---

@aubrivliz, this is exactly what you discussed earlier this morning. I would update the detections_limited view to add 2 years of wiggle room for battery_estimated_enddate:

AND (detection.datetime <= tag_device.battery_estimated_end_date + interval '2 year' OR tag_device.battery_estimated_end_date IS NULL)

[PieterjanVerhelst]

So if those 2 years of wiggle room are added, that means upon download I will get the duplicates as well? No problem as I can remove them afterwards, just to know, because I am downloading a huge dataset (already bouncing in memory error on the Rstudio server).

[peterdesmet]

It depends. If you are searching detections on a specific animal project code, then it will only return tags associated with animals from that project (and not other detections), thus not creating duplicates. You can always test if detection_id is unique in your results.

[PieterjanVerhelst]

aah ok, I am indeed downloading data based on animal_project_code. Let me know when I can run another test.

[peterdesmet]

@PieterjanVerhelst the wiggle room has been implemented and you can test again.

[PieterjanVerhelst]

Still no data, but when reinstalling etn I got the notification nothing changed. So no changes have been made to the package, but on database level?

[peterdesmet]

That is correct, only changes in the database.

So, no data for:

data <- get_acoustic_detections(con, scientific_name = "Anguilla anguilla",
                                 animal_project_code = "2011_Loire",
                                 limit = FALSE)

I get 25850 detections.

[PieterjanVerhelst]

Correct! However, I still get no data for the project life4fish.

[peterdesmet]

I'm confused by your "however" 😄. Are you getting data for 2011_Loire or not?

These are the Anguilla anguilla detections I get for:

EMMN         4393
2011_Loire   25850
2004_Gudena  497173
life4fish    0

[PieterjanVerhelst]

Yes, I indeed get data for 2011_Loire, as well as the other 2 projects you list 😃. But for some reason, life4fish returns no data.

[peterdesmet]

Indeed, same for me (no records for life4fish), so we can rule out an access rights issue. Try to see if you get records by searching on acoustic_tag_id?

[PieterjanVerhelst]

That also doesn't return data, for example:

data3 <- get_acoustic_detections(my_con, scientific_name = "Anguilla anguilla",
                                 acoustic_tag_id = "416kHz-41014",
                                 limit = FALSE)

Note that when I apply the function get_animals() animals from life4fish are returned.

[peterdesmet]

To truly test on acoustic_tag_id, you have to drop the filter on scientific_name. But it doesn't make any difference here:

 get_acoustic_detections(acoustic_tag_id = "416kHz-41014",  limit = FALSE)

... returns no data.

Is it possible this project did not generate detections for Anguilla anguilla?

[PieterjanVerhelst]

aah, but the only data that is included, is on Anguilla anguilla. So yes, it should contain data. I'll have a check if this data is in the ETN database itself.

[PieterjanVerhelst]

Update: In ETN it says there is no data available. @jreubens @aubrivliz could you check why the data from life4fish is not coming through?
This project involves Lotek equipment and there are issues with integrating such data in ETN. So probably there is a data issue.

[aubrivliz]

@peterdesmet @PieterjanVerhelst @jreubens
I am updating 100M+ detections with incorrect Lotek transmitter names. If the update is done I will check the data for life4fish

[PieterjanVerhelst]

@aubrivliz any news on the Lotek data? I tried to download the life4fish data but still no output.

[aubrivliz]

@PieterjanVerhelst all the detection data has been updated. But there is still no data avalaible for some reason.
I will investigate further and get back to you asap.

[aubrivliz]

@PieterjanVerhelst I've made some changes in the detections_limited view, I am getting now detection data for animal_project_code = 'life4fish'.

Can you verify?

[PieterjanVerhelst]

Yes! Now I can download the data and according to some quick checks I should have the data for all 146 eels in the project life4fish. Just FYI, I get 415,903 data records. I will have a more thorough check on Friday and will close this issue if no further problems arise.
Great job 👍 !