Incident 9th June 2019 10:30 am service

On 9th June 2019 10:30 am service, the In ear monitors system did not function. The team had to revert to the wedge monitors at short notice.

Symptoms

No sound on any channels of ME-1 units

Cause

There were two causes to the above issue. It's hard to know exactly how much each affected the other - in theory the system should have been able to carry on working on the secondary link only - maybe it has been for some time, I haven't been able to determine that.

Cause 1: Physical cable not connecting

The primary Dante network was connected from the Cisco Switch to the ME-U unit via the cable in the floor (known as a cable 'drop') numbered 21, which didn't seem to be working. No signal lights were visible at either end of the connection. Switching to drop 23 fixed the issue. It's hard to know how long this has been the case as it's possible it happened months ago, but the secondary link kept on working, or it's possible that without the primary link at start up, the secondary can never be initialised, and this was the actual cause of the issue.

Fix applied

• Use a different cable drop

Prevention

• Remember to regularly check that both primary and secondary networks are operating

Cause 2: IP addresses allocated by unknown transient DHCP server

(if the below makes no sense, see "more detailed explanation" below)

The primary Dante ports of the MY16-AUD cards in the M7CL had their IP addresses set to 192.168.2.* range even though they were set to automatically assigned IP addresses, and normally work on the link-local (169.254.*.*) address range. Presumably, at some point, the network was connected to a DHCP server by mistake, either by connecting it to one of the permanent networks, or by connecting it to a laptop with something like internet connection sharing enabled which allocated those addresses.

Fix applied

• Physically disconnect all the MY16-AUD connections from the primary network and reconnect after allowing a few seconds for them to believe they might be on a new network and might need a new ip address, then reconnect them. Power cycling the switch didn't work - they held on to their 192.168.2.* addresses.

Prevention

• Make sure we never connect the Dante networks to a DHCP service, e.g. by bridging it with the office network or possibly the dSNAKE network (if that uses DHCP).
• Get a DHCP server set up on the Dante networks so that if another DHCP server appears temporarily, the devices know that they already have DHCP, and won't go joining the wrong network.
• Set up some sort of network device monitoring to make sure only expected devices are connected to expected networks. This is much harder to do, unless it can be done using the Cisco switches we already have.
• Find any devices with DHCP servers issuing in the 192.168.2.* range and work out how they were connected.

More detailed explanation

The Internet Protocol (IP) is a set of standards that governs how computers on the internet can talk to each other. It's also possible to have a small Internet Protocol network that isn't part of the main internet (a private network), because it's not connected at all, or only via a gateway. This is how most e.g. home networks, office networks etc. are set up, with a gateway to the real internet in the Modem / Router. There are at least 5 such networks in Church:

• The church office network, which has a gateway to the global internet
• The A/V network (known as Galactica) which has a gateway to the above, and can therefore reach the global internet if needed. This is used by the touch screens and video switchers.
• 2 Dante networks (primary and secondary) which are not connected to the internet in any way, they only connect specific devices. This is used to send the audio signals from the Sound desk to the ME-U. If the primary network fails, the secondary network can take over automatically.
• The dSNAKE network that links the ME-1s to the ME-U, also not connected to the internet. It is connected to the Dante networks in the ME-U, but this is not an IP connection - the audio signals are read from Dante and encoded into the dSNAKE format.

Every computer on an IP network needs to have an IP address unique to it across the network. It looks like 4 numbers between 0 and 255, e.g. 216.58.213.14. For the global internet, these addresses can be basically anything, but a few groups of addresses are reserved for private network use only so that gateways on a don't get confused as to which data should be on which network. Some examples are addresses like 10.*.*.*, like 192.168.*.* or like 169.254.*.*.

In the Dante network, the IP address can either be set up manually, where you have to choose an IP address for each device by yourself, or automatically, where the device will:

• Ask the network if it has a DHCP server. If it does, this server will tell them an IP address to use. A gateway device will often perform this function, e.g. a home router, or on the office network it's one of the servers controlled by Angus.
• Failing that, it will randomly pick 2 numbers (e.g. x and y) and try the address 169.254.x.y (checking that no other devices have that particular address first). This is called link-local addressing, and is designed to be especially easy to use on networks with no gateway to the internet.

For some reason, the Sound desk side of the Dante network had got it's IP address from a DHCP server, and the ME-U side had got it's address from Link-local addressing. There should not be a DHCP server on the Dante network, and wasn't one when I was checking. So one must have been added by mistake at some point.