Installed but not creating directories

[CaiusX]

Hi @mcguirepr89

Thanks for putting this system together. I have run the installation script (both as normal and sudo) which completed, but does not appear to be creating the necessary directories - probably a permissions issue? I wasn't prompted for my pi user password in the installation & tried with the Public directory as the recordings path.

I can access the "BirdNET-system Extractions" page on http://localhost ; but any of the links from there generates a 404 error.

Looking forward to getting this running & beta testing. It would be great if it could also be on the BirdNET-Lite platform model with its 6,000 species (which recognizes species in my neck of the woods - Africa!!)

Full installation output below:

pi@raspberrypi:~/BirdNET-system $ ~/BirdNET-system/scripts/install_birdnet.sh
Would you like to fill out the configuration file interactively?y
The next few questions will populate the required configuration file.

Who will be the BirdNET user? (use 'whoami' if unsure) pi
What is the full path to your recordings directory (locally)? /home/pi/Public
What is the latitude where the recordings were made? -26
What is the longitude where the recordings were made? 28
Is this device also doing the recording? y
Checking for alsa-utils
ALSA-Utils installed
Installing birdnet_recording.service
Created symlink /etc/systemd/system/multi-user.target.wants/birdnet_recording.service → /etc/systemd/system/birdnet_recording.service.
Do you want this device to perform the extractions? y
Installing the extraction.service
Created symlink /etc/systemd/system/multi-user.target.wants/extraction.timer → /etc/systemd/system/extraction.timer.
Created symlink /etc/systemd/system/multi-user.target.wants/extraction.service → /etc/systemd/system/extraction.service.
Adding the species_updater.cron
Would you like to access the extractions via a web browser
*Note: It is recommended, (but not required), that you run the web
server on the same host that does the extractions. If the extraction
service and web server are on different hosts, the "By_Species" and
"Processed" symbolic links won't work. The "By-Date" extractions,
however, will work as expected.y
What URL would you like to publish the extractions to?
*Note: Set this to http://localhost if you do not want to make the
extractions publically available: http://localhost
Caddy is installed
Please set a password to protect your live stream: xxxxx

Please set the icecast password. Use only alphanumeric characters. xxxxx

/usr/bin/icecast2
Icecast2 is installed
icecast2.service is not a native service, redirecting to systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable icecast2
[ ok ] Starting icecast2 (via systemctl): icecast2.service.
Installing Live Stream service
Created symlink /etc/systemd/system/multi-user.target.wants/livestream.service → /etc/systemd/system/livestream.service.
Installing avahi-alias service
Created symlink /etc/systemd/system/multi-user.target.wants/avahi-alias@birdnetsystem.local.service → /etc/systemd/system/avahi-alias@.service.
Created symlink /etc/systemd/system/multi-user.target.wants/avahi-alias@birdlog.local.service → /etc/systemd/system/avahi-alias@.service.
Created symlink /etc/systemd/system/multi-user.target.wants/avahi-alias@extractionlog.local.service → /etc/systemd/system/avahi-alias@.service.
Created symlink /etc/systemd/system/multi-user.target.wants/avahi-alias@birdstats.local.service → /etc/systemd/system/avahi-alias@.service.
Created symlink /etc/systemd/system/multi-user.target.wants/birdnet_log.service → /etc/systemd/system/birdnet_log.service.
Created symlink /etc/systemd/system/multi-user.target.wants/extraction_log.service → /etc/systemd/system/extraction_log.service.
Created symlink /etc/systemd/system/multi-user.target.wants/birdstats.service → /etc/systemd/system/birdstats.service.
Do you have a free App key to receive mobile notifications via Pushed.co?n
Adding the cleanup.cron
Created symlink /etc/systemd/system/multi-user.target.wants/birdnet_analysis.service → /etc/systemd/system/birdnet_analysis.service.

This script will do the following:
#1: Install the following BirdNET system dependencies:

• ffmpeg
• libblas-dev
• liblapack-dev
• wget
• llvm-9
  Added avahi-aliases (mdns) for gotty/caddy pages #2: Creates a conda virtual environment for BirdNET
  switching to miniforge3 since it is much better maintained #3: Builds BirdNET in the 'birdnet' conda virtual environment
  Testing #4: Copies the systemd .service and .mount files and enables those chosen
  Welcome to BirdNET-system Discussions! #5: Adds cron environments and jobs chosen

Be sure you have read the software license before installing. This is
available in the BirdNET-system directory as LICENSE
If you DO NOT want to install BirdNET and the birdnet_analysis.service,
press Ctrl+C to cancel. If you DO wish to install BirdNET and the
birdnet_analysis.service, press ENTER to continue with the installation. Checking dependencies
libblas-dev is installed!
liblapack-dev is installed!
llvm-9 is installed!
ffmpeg is installed!
wget is installed!
Making symbolic link for llvm-config binary
\ Fetching the model
--2021-09-21 15:43:54-- https://tuc.cloud/index.php/s/m9smX4FkqmJaxLW/download
|134.109.133.44, 2001:638:911:b01:134:109:133:44
-connected.to tuc.cloud (tuc.cloud)|134.109.133.44|:443...
-200 OKequest sent, awaiting response...
Length: 307374723 (293M) [application/octet-stream]
Saving to: ‘model/BirdNET_Soundscape_Model.pkl’

model/BirdNET_Soundscape_Model.pkl 100%[=========================================================================================================================================>] 293.13M 7.70MB/s in 64s

2021-09-21 15:44:59 (4.55 MB/s) - ‘model/BirdNET_Soundscape_Model.pkl’ saved [307374723/307374723]

Fetching the Miniforge3 installation script

/ Installing Miniforge3 for aarch64
| Initializing Miniforge (conda)
Initializing the birdnet virtual environment with
- numba
- numpy
- scipy
- future
- python=3.7

• Activating new environment

• Upgrading pip, wheel, and setuptools
  \ Installing Librosa
  / Installing Theano
  | Installing Lasagne
  \ BirdNet is installed!!
  Enabling birdnet_analysis.service now
  BirdNET is enabled.

  To start the service manually, issue:
  'sudo systemctl start birdnet_analysis'
  To monitor the service logs, issue:
  'journalctl -fu birdnet_analysis'
  To stop the service manually, issue:
  'sudo systemctl stop birdnet_analysis'
  To stop and disable the service, issue:
  'sudo systemctl disable --now birdnet_analysis.service'

Visit http://birdnetsystem.local to see your extractions,
http://birdlog.local to see the log output of the birdnet_analysis.service,
http://extractionlog.local to see the log output of the extraction.service, and
http://birdstats.local to see the BirdNET-system Report

Would you like to run the BirdNET service now?y-- Logs begin at Tue 2021-09-21 15:23:21 SAST. --
Sep 21 15:35:05 raspberrypi systemd[1]: birdnet_analysis.service: Service RestartSec=3s expired, scheduling restart.
Sep 21 15:35:05 raspberrypi systemd[1]: birdnet_analysis.service: Scheduled restart job, restart counter is at 210.
Sep 21 15:35:05 raspberrypi systemd[1]: Stopped BirdNET Analysis.
Sep 21 15:35:05 raspberrypi systemd[1]: Started BirdNET Analysis.
Sep 21 15:35:05 raspberrypi birdnet_analysis.sh[5781]: find: ‘home/pi/recordings/September-2021/19-Sunday’: No such file or directory
Sep 21 15:35:05 raspberrypi birdnet_analysis.sh[5781]: find: ‘home/pi/recordings/September-2021/20-Monday’: No such file or directory
Sep 21 15:35:05 raspberrypi birdnet_analysis.sh[5781]: find: ‘home/pi/recordings/September-2021/21-Tuesday’: No such file or directory
Sep 21 15:35:05 raspberrypi systemd[1]: birdnet_analysis.service: Succeeded.

[mcguirepr89]

Greetings, @CaiusX -- Thank you so much for installing the BirdNET-system for the RPi4B. I know we'll get this working for you :)
Also, this installation output is very helpful and gets us started off on the right foot, so thanks for that, too!

I have a hunch as to what might be the hang up here and a little more info about the system to address the 404 responses you're getting on the http://localhost links.

My hunch: the installation went smoothly, but I'm unclear with documentation!

Looking at your installation, you opted for the recording service, which will create the "/home/pi/recordings/September-2021/21-Tuesday" directory once it starts creating sound files. (Notice that it is automatically also looking in the directories that would have been the previous two days of recording, but that is irrelevant here and I can provide more information about that if you are curious).

The birdnet_recording.service doesn't actually start up until you reboot the system. Once you do reboot the system, that directory will be created, and then the birdnet_analysis.service (which you see as the log output when answering "yes" to whether to start BirdNET service now) will start analyzing the files it finds.

The links on your http://localhost don't take you anywhere yet. The By_Date, By_Species, and Processed directories get created during extractions so that things stay organized. I will definitely add a step to the new installer that simply creates these directories so that they don't give that 404 response right after installation.

Since you mentioned running the installer with sudo, there are likely some configuration files and settings for the 'root' user on your system, which is almost definitely undesirable. I'm sorry that my lack of documentation and little scripting oversights have created the uncertainty around the installation's success. To remedy this, I recommend running the included uninstall.sh script twice, once as the pi use and once with sudo, to be sure that there are no lingering root user configurations, then rerun the installer using the steps below.
(Don't worry, the uninstaller only removes services, their configuration files, and the birdnet.conf file and /etc/birdnet/birdnet.conf symbolic link. It does NOT remove BirdNET, so it shouldn't be very time consuming to reconfigure the installation.)

Steps for you:

1. Run the uninstall.sh script as the pi user by issuing /usr/local/bin/uninstall.sh.
2. Step 1 will remove the uninstall.sh script from your $PATH, so running it again, this time with sudo, will have to point to the script within the BirdNET-system/scripts directory: sudo /home/pi/BirdNET-system/scripts/uninstall.sh
3. With all the service files and birdnet.conf wiped clean, you can reconfigure by running the install_services.sh script with sudo (I'm sorry if the sudo use seems arbitrary at this point, but I do promise there is a rhyme and reason to it all -- though I have completely rewritten the installer in the rpitesting branch that will get pulled into the BirdNET-system-for-raspi4 branch very soon.): sudo /home/pi/BirdNET-system/scripts/install_services.sh
4. Reboot. When the system comes back up, you can watch the birdnet_analysis.service log either from http://birdlog.local (if your router permits) or from the terminal with journalctl -fu birdnet_analysis.service or from the raspberry pi's web browser, http://localhost:8080.

Once your system has been recording for a little while and has heard a few birds, the extraction service will pull the extractions and place them in the By_Date and By_Species directories and will put the original recordings and their corresponding BirdNET.selections.txt files in the Processed directory (actually a symbolic link). This is the step that will discontinue the 404 responses and will then be filled with the goodies we love.

I most certainly have plans to get the BirdNET-LITE setup with its own RPi4 installer and companion system services like what I've put together for the main BirdNET repo.

Please let me know if my suggestions have addressed the issues and get your system working -- I am very eager to help and grateful for your feedback on your experience. I hope you will continue to share your insights, suggestions, successes and wishes with me for the BirdNET-system.

My best,

[CaiusX]

Running the scripts:

pi@raspberrypi:~ $ /usr/local/bin/birdnet_recording.sh
Recording WAVE '/home/pi/BirdSongs/%B-%Y/%d-%A/%F-birdnet-%I:%M%P.wav' : Signed 16 bit Little Endian, Rate 48000 Hz, Stereo
arecord: set_params:1345: Channels count non available

pi@raspberrypi:~ $ ffmpeg -loglevel 52 -ac 2 -f alsa -i dsnoop:CARD=USB,DEV=0 -acodec libmp3lame -b:a 320k -ac 2 -content_type 'audio/mpeg' -f mp3 icecast://source:rev222@localhost:8000/stream -re
ffmpeg version 4.1.6-1~deb10u1+rpt2 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 8 (Debian 8.3.0-6)
  configuration: --prefix=/usr --extra-version='1~deb10u1+rpt2' --toolchain=hardened --incdir=/usr/include/aarch64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --disable-mmal --enable-neon --enable-vout-drm --enable-v4l2-request --enable-libudev --libdir=/usr/lib/aarch64-linux-gnu --arch=arm64 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Splitting the commandline.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '52'.
Reading option '-ac' ... matched as option 'ac' (set number of audio channels) with argument '2'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'alsa'.
Reading option '-i' ... matched as input url with argument 'dsnoop:CARD=USB,DEV=0'.
Reading option '-acodec' ... matched as option 'acodec' (force audio codec ('copy' to copy stream)) with argument 'libmp3lame'.
Reading option '-b:a' ... matched as option 'b' (video bitrate (please use -b:v)) with argument '320k'.
Reading option '-ac' ... matched as option 'ac' (set number of audio channels) with argument '2'.
Reading option '-content_type' ... matched as AVOption 'content_type' with argument 'audio/mpeg'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'mp3'.
Reading option 'icecast://source:rev222@localhost:8000/stream' ... matched as output url.
Reading option '-re' ... matched as option 're' (read input at native frame rate) with argument '1'.
Trailing options were found on the commandline.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option loglevel (set logging level) with argument 52.
Successfully parsed a group of options.
Parsing a group of options: input url dsnoop:CARD=USB,DEV=0.
Applying option ac (set number of audio channels) with argument 2.
Applying option f (force format) with argument alsa.
Successfully parsed a group of options.
Opening an input file: dsnoop:CARD=USB,DEV=0.
ALSA lib confmisc.c:767:(parse_card) cannot find card 'USB'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such device
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such device
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such device
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such device
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM dsnoop:CARD=USB,DEV=0
[alsa @ 0x55bafec930] cannot open audio device dsnoop:CARD=USB,DEV=0 (No such device)
dsnoop:CARD=USB,DEV=0: Input/output error

[mcguirepr89]

Yep -- the sound card variable is the culprit -- there was a missing backslash in the installer which kept the sound card variable from having its ",DEV=0" truncated -- you can still see it in the service files. There may also be an issue with the number of channels your microphone supports compared to what the recording service expects to be available.

I'm such a dummy! Great catch with the /usr/ /user -- 'usr' -- though I've verified that that line can be completely wrong and the service doesn't care -- the issue is still elsewhere, but I've made the change in the repo for continuity -- that line will likely get removed in the next installer. Thanks again for the catch!

To change the bad parts of the livestream service file, run the following and then check to see that the changes took effect:

• sudo sed -i 's/dsnoop:CARD=USB,DEV=0/dsnoop:CARD=Device/' /etc/systemd/system/livestream.service
• sudo sed -i 's/usr/user/' /etc/systemd/system/livestream.service
• cat /etc/systemd/system/livestream.service just use this command to ensure that the "dsnoop...." part of the ExecStart line is corrected

And to test whether the number of channels are the issue, you can try:

• source /etc/birdnet/birdnet.conf && arecord -f S16_LE -c1 -r48000 -t wav --max-file-time 60 -D dsnoop:CARD=Device --use-strftime ${RECS_DIR}/%B-%Y/%d-%A/%F-birdnet-%I:%M%P.wav
  and if that fails, try:
• source /etc/birdnet/birdnet.conf && arecord -f S16_LE -c1 -r48000 -t wav --max-file-time 60 --use-strftime ${RECS_DIR}/%B-%Y/%d-%A/%F-birdnet-%I:%M%P.wav
  If either of the above commands does not complain, then the number of channels is also an issue, in which case I will adjust the recording service script to reflect the command that works.

The changes to the livestream service won't apply until a reboot, but will also depend on figuring out whether or not the number of channels your microphone supports is an issue. Let me know if the arecord . . . command works without complaining about the number of channels. We'll go from there. I'm sorry it has gotten so touch-and-go with the troubleshooting, but that's the way things often go during beta testing. Thanks again for everything

[mcguirepr89]

Hi Dennis -- I just wanted to update you that I have fixed the installer to address the issues with the REC_CARD variable. If you decided to try a fresh installation, I feel assured now that the REC_CARD variable will be appropriately set. This won't necessarily rule out a possible issue with the number of microphone channels, but would rule out issues with the REC_CARD variable.

[CaiusX]

Hi Patrick

Found some time to dabble with a clean installation:

I think the number of channels may be an issue, so changed -ac 2 to -ac 1 in:

ffmpeg -loglevel 52 -ac 1 -f alsa -i dsnoop:CARD=Device -acodec libmp3lame -b:a 320k -ac 1 -content_type 'audio/mpeg' -f mp3 icecast://source:xxxxxx@localhost:8000/stream -re

I can now access the stream on http://birdnetsystem.local/stream - some progress !!! (lags about 10 seconds)

BirdNET service doesnt seem to be running - possibly another service which needs channels updated?

I cant see the wood for the trees, so attached is a full log of all of the commands provided for testing thusfar on the current installation - hopefully useful in tracking down the problem/s.

Birdnet_System_Troubleshoot_26_09.txt

May be better for you to send me some targetted commands?

Best
Dennis

[mcguirepr89]

Hi Dennis -- thanks so much for the info. I also think the channels are the issue -- getting the live stream working with one channel seems to also prove that. The modified recording scripts also seemed to open a stream without complaining, so that's a good sign. The birdnet.conf has a strange "REC_CARD" setting, which I'm hoping you input manually since you said this was a clean install and I would not expect the installer to create what is shown there.

Depending on when you performed this most recent installation, there may be an installation log you can send me. The newest installer automatically spits out a compressed set of logs in the BirdNET-system directory called "logs.tar.gz" that you can send me. If you did the newest installation prior to that being incorporated into the installer, you can run the log dumper manually.

1. cd ~/BirdNET-system && git pull && ./scripts/dump_logs.sh

If I need more information than what the log dump provides, I will let you know.
Thank you

[CaiusX]

Hi Patrick

I had been messing around with the files - hence the REC_CARD strangeness, so have created a fresh installation, logs attached.
logs.tar.gz

Shout if you need anything else.

Best

[mcguirepr89]

Hi Dennis,
I've made a few improvements to the main installer based on what we've gone through to get to the issue:

1. The birdnet.conf file now has a variable called CHANNELS that is set during installation. A script reads the hardware parameters available for the sound card and sets the CHANNELS variable accordingly. The original plan to simply set all things to using one channel fell apart quickly when my microphone fussed just like yours when asked to use one channel. Thus, CHANNELS is set dynamically, but can be manually overridden after installation if the microphone supports more than one channel structure (no idea if that is even a thing, but I expect it may be?).

2. The livestream.service and birdnet_recording.service now read the REC_CARD and CHANNELS variables on each run, so that you can easily change and set those variables manually in the birdnet.conf file instead of needing to alter any service files.

   Likewise, you should be able to swap out the USB microphone pretty easily as a result, with the possible need to run the reconfigure_birdnet.sh script. Alternately, running the uninstall.sh script, creating a new birdnet.conf file (cp ~/BirdNET-system/birdnet.conf-defaults ~/BirdNET-system/birdnet.conf), editing it to the settings you used before, and running the installer (~/BirdNET-system/scripts/install_birdnet.sh) would certainly suffice to change microphones.

3. The installer creates a helpful little file alongside the installation log called soundcard_params.txt in the BirdNET-system directory which will tell users what their microphone is capable of doing, in case the basic settings need specific tweaking to work. If I had access to various USB microphones with different hardware parameters, I would be able to put something together to handle more types of hardware -- so you can imagine that I am very grateful to have your input on how BirdNET-system handles your hardware :)

Thanks a bunch for your help in troubleshooting -- I'm hopeful for this as the solution for you.

My best regards,

[CaiusX]

Hi Patrick

That seems to have done the trick!!

After re-installing for a North American location, the system operated flawlessly and recognised e-bird species accuarately (American Crow, Tufted Titmouse).

Running reconfigure_birdnet.sh script and changing to my location in a different recording directory and rebooting however threw up errors in the extraction service. I think that the birdnet.conf file gets corrupted by inserting additional numeric characters below the new "channels=1" line (line 200), in my case inserting a "1" on line 201. If I delete this line in the birdnet.conf the system starts working again with the new location.

I will continue with my "birder beta" testing (switch microphones, location etc) & report back.

In the meantime I think we can close this "not creating directories" question?

Best
Dennis

[mcguirepr89]

Hi Dennis -- I'm so glad that the most recent improvements (mostly) worked for you. I truly appreciate your time and feedback in testing the project. I will close this discussion, but please keep me in the loop for anything else you care to share about your experience with the system.

My best regards,
Patrick

PS -- May I add you as a beta testing on my list of locations BirdNET-system is listening?

[CaiusX]

Hi Patrick

Thank you for all the hard work and quick responses.

Definitely will keep you in the loop with comments and suggestions. Most of the ID's I'm getting here down south are false positives. I am going to open another thread when I've run it overnight and have more results, to encourage BirdNET-Lite system implementation (which I'm happy to try help out with, I got it running on my RPi a few months back with better accuracy rate) , or a larger BirdNET model (beyond the current 1,000 species, possibly alluded to by @kahst in kahst#20 (comment))?!.

Yes, put me on the list -

Regards

DennisC - Johannesburg, South Africa "Listening and learning since 09/2021".

[mcguirepr89]

I have that 6,000 species BirdNET-Lite model in my sights! I've been a bit busy working out the kinks in this repo, but plan to start building that out this week if I can, next week if I can't. I'll certainly keep you updated and hope you'll be among the first beta-testers for that fork :)

My best,