This week at the Imlay City site a power amplifier and it’s power supply had failed which runs a channel bank for the 800MHz voice system. Users in the area were unaffected and did not notice because the system still has excess capacity for daily operations. A spare for each have been put in and brought the site back up to full capacity less the other channel bank that failed a few months back.
Fortunately, these power amplifiers and their power supplies are still covered under our maintenance contract and news ones are being shipped out today.
Over the weekend a part of the microwave system failed. The path from Lapeer to Imlay City kept fading on the ‘A’ side. The ‘B’ side which is the backup to the ‘A’ side took over seamlessly and users were unaware of the problem. The system did as it was designed to do. ComSource came out and worked on Saturday and then came back on Monday to complete work. Initial checks showed no problems, but there was evidence of one. Special software had to be acquired to read parts of the system to find the problem component. It was found that a power amplifier was the cause. A spare was on hand from our purchase of parts a few years ago. Both the ‘A’ & ‘B’ sides of the path are up and running again. A new replacement spare will be purchased to have on hand.
On Monday some users in the Imlay City area experienced a very quick “out of range” alert on their radios while ComSource replaced the amplifier and switched the paths back and forth for testing purposes.
At the same time, but unrelated, our telephone system was experiencing problems as well. Many calls to the center, both 9-1-1 and non-emergency calls were coming in full of static and would often disconnect during the call. Our phone system provider ProTel was contacted and monitored the system. Nothing was found to be amiss within our system locally and it was determined to be an issue outside of the center and with AT&T. The problem was cleared up by Sunday.
Recently there was a channel bank in our current analog system that had a failure and had to be disabled. This channel bank is part of the voice simulcast system which pulls frequencies from the trunked system for transmitting. We have a spare board to replace it with, however in order to do so, the system would have to be put into failsoft – essentially that means turning the system off to make the change. As a public safety agency, we do not have the luxury of turning the system off for a repair of this nature. Since 1996 this is the first time one of these boards has failed.
Fortunately, we have plenty of excess capacity to handle all the users and routine radio traffic even with this one channel bank disabled. We believe that the system would have more than enough capacity to handle another major incident like the Champion Bus fire that occurred recently with the multitude of users that were present.
Though we have the capacity to continue uninterrupted operations, it illustrates the importance of replacing this analog system which is showing its age.
When users make a submission, we get the email results and then plot them on a publicly available Google map to show where the problems are occurring the most. We then, along with Blumerich and Motorola, will use the information to troubleshoot the issues with the paging system and hopefully if the millage passes tomorrow, will use the same information to design a new paging system and provide data for better tower placement.
The Zetron paging system was repaired by the company last week and returned to us. With Blumerich’s technical assistance it was brought back online late Friday afternoon. During the Zetron outage we set up the fire paging system to be able to tone out the EMS bases so they could still get notified for emergency runs. With that setup, we were only able to tone the bases and no backup personnel. Backup personnel are called out when all EMS rigs are on calls or otherwise unavailable. These personnel come into the base nearest the call for help and get an ambulance to respond.
With the system back the way it was before the lightning strike, we can tone the bases and backup personnel utilizing the alpha-numeric pagers and the Minitor V station pagers. EMS supervisors were unaffected during this time because they carry pagers operating on a cellular system from American Messaging which runs through the Internet. We have left the fire paging setup for EMS in place in the event that the Zetron goes out again. This way there will not be any setup time getting their station tones to go.
Additionally from the lightning strike, two dispatch servers’ Windows operating systems became corrupted. They didn’t crash until late Friday afternoon, the day after the storm. This can happen anytime there is a significant fluctuation in voltage to the server or any computer. Regardless of the tower and all connected equipment being properly grounded and on a UPS system (uninterruptible power supply) the lightning was more powerful than what the grounding could handle and obviously it ruined some things.
The first server to crash was the DSS Equature voice recording server. This machine records all telephone and radio traffic to and from the 9-1-1 center. Several attempts were made to restore the operating system using various repair software. They were unsuccessful and DSS was called in with a spare server. They changed out the servers and reconfigured the spare for our system and got recording back online. Unfortunately there were several hours where nothing was recorded.
Halfway through working on the DSS server the primary CAD (computer aided dispatch) server crashed. Thanks be to God for allowing us to successfully complete a prior project which gave us a backup CAD server with an instant failover software called Neverfail. When the primary server crashed, the backup server instantly assumed duty to become the primary CAD server. The dispatchers only noticed a small hiccup where they only had to restart the CAD application and continue working. Had we not had this Neverfail project complete, the dispatchers would be taking and dispatching calls by pen and paper. This method of dispatching is extremely inefficient and time consuming for this type of business. It wasn’t until today that the primary server was fully brought back online. The backup server has been running operations for the last two weeks. We don’t want to imagine our work being done on pen and paper for this long.
The lighting messed with the CAD server so much that its main board had to be replaced by an IBM technician. During the Neverfail project we signed a hardware and software maintenance agreement with IBM for $1,800 per year. It has already paid dividends as the board that was replaced had a cost of $3,000 and that doesn’t include the time and labor that the IBM tech had invested in the repair of the server. Once that was completed the RAID arrays (redundant array of independent disks) were reconfigured and the Windows operating system reloaded. After that, our CAD vendor VisionAIR had to load their software, manipulate the data from the backup server to the primary server and load and configure the Neverfail software again.
Yesterday afternoon a severe thunderstorm passed through Lapeer County and early on a bolt of lightning is believed to have struck the tower behind the 9-1-1 Center in Lapeer. A dispatcher witnessed an arch of electricity move from one of the stand-off antennas to a leg of the tower structure. The dispatcher also saw the tower glowing briefly. Subsequently, this lightning strike temporarily knocked out all communications in the center including sending the voice radio system into failsoft.
What is failsoft?
If the trunking system loses its control channel or has certain other failures, it is no longer able to operate in the trunking mode. So instead of going into a condition that stops all communication, the system enters Failsoft. In this state all transmitters (channels) turn on and operate in a ‘conventional’ repeater mode. The subscriber radios are able to recognize this state and switch to a predetermined frequency (one of the trunk system frequencies, but not the control channel frequency) depending on their selected talkgroup. In most systems several talkgroups will share a frequency. Some talkgroups may not be assigned a failsoft frequency and these talkgroups will cease to operate during the failsoft period. If a particular failsoft frequency has also failed, the talkgroups assigned to that frequency will also be off the air during failsoft.
The system quickly recovered and went back into normal operation. However, one system was unable to recover and that is the part of the paging system which dispatch’s EMS crews for Lapeer County EMS.
The old Zetron unit which runs the alpha-numeric pagers for Lapeer County EMS is no longer communicating with the paging servers. Several attempts to reset the unit and even change paging servers were unsuccessful. Another problem is that this unit has gone off of the Motorola maintenance contract several years ago. We do have replacement parts on hand for this unit, but working on it will cost the center time and labor. We will be contacting Zetron this morning to seek their assistance in diagnosing the problem and programming spare boards to be put into service.
As of yesterday, all bases of Lapeer County EMS (Imlay City, North Branch and two in Lapeer City) are operating on radio and telephone standby until further notice. It is not known when or even if this system can be repaired.
Fire department paging was unaffected and is in normal operation.
Additionally, the wireless communication link to Metamora Police which runs their AICS and Internet services went down during the lightning strike.
Further updates will be posted when information is available.
This past week we experienced another breakdown in the communications system. This time it is was on the voice side. At the Imlay City tower site voice channel bank number 5 had a power supply and power amplifier failure. Blumerich replaced both parts with new ones after having Motorola ship them with overnight delivery.
The good news out of this is that although there was a failure of equipment, no one noticed that there was a problem. This is because the voice system has plenty of capacity for traffic and although one of the voice banks went down, the system still had excess capacity to handle all of the radio traffic. The only time a failure like this would become noticeable would be if there was a huge influx of users in the area of the affected tower.
In the case of the Champion Bus fire a few months ago, had we lost a voice bank or two, there could have been a possibility of users getting a “bonk” when they try to transmit, indicating the system is too busy at that particular moment to handle their transmit request. The user would just have to keep trying every few seconds to get permission to make a transmission.
The power supply and power amplifier both are covered under our Motorola service contract.
So far in 2010, we are seeing more frequent breakdowns of various equipment at each of the sites than we have in the past. The age of the system is quickly catching up to it.