Amplifier & Power Supply Failure

This week at the Imlay City site a power amplifier and it’s power supply had failed which runs a channel bank for the 800MHz voice system. Users in the area were unaffected and did not notice because the system still has excess capacity for daily operations. A spare for each have been put in and brought the site back up to full capacity less the other channel bank that failed a few months back.

Fortunately, these power amplifiers and their power supplies are still covered under our maintenance contract and news ones are being shipped out today.

Power amplifier, exciter and power supply for a channel bank

Minor Microwave & Telephone Problems

Over the weekend a part of the microwave system failed. The path from Lapeer to Imlay City kept fading on the ‘A’ side. The ‘B’ side which is the backup to the ‘A’ side took over seamlessly and users were unaware of the problem. The system did as it was designed to do. ComSource came out and worked on Saturday and then came back on Monday to complete work. Initial checks showed no problems, but there was evidence of one. Special software had to be acquired to read parts of the system to find the problem component. It was found that a power amplifier was the cause. A spare was on hand from our purchase of parts a few years ago. Both the ‘A’ & ‘B’ sides of the path are up and running again. A new replacement spare will be purchased to have on hand.

Microwave Power Amplifier

On Monday some users in the Imlay City area experienced a very quick “out of range” alert on their radios while ComSource replaced the amplifier and switched the paths back and forth for testing purposes.

At the same time, but unrelated, our telephone system was experiencing problems as well. Many calls to the center, both 9-1-1 and non-emergency calls were coming in full of static and would often disconnect during the call. Our phone system provider ProTel was contacted and monitored the system. Nothing was found to be amiss within our system locally and it was determined to be an issue outside of the center and with AT&T. The problem was cleared up by Sunday.

Reduced System Capacity

Recently there was a channel bank in our current analog system that had a failure and had to be disabled. This channel bank is part of the voice simulcast system which pulls frequencies from the trunked system for transmitting. We have a spare board to replace it with, however in order to do so, the system would have to be put into failsoft – essentially that means turning the system off to make the change. As a public safety agency, we do not have the luxury of turning the system off for a repair of this nature. Since 1996 this is the first time one of these boards has failed.

Fortunately, we have plenty of excess capacity to handle all the users and routine radio traffic even with this one channel bank disabled. We believe that the system would have more than enough capacity to handle another major incident like the Champion Bus fire that occurred recently with the multitude of users that were present.

Though we have the capacity to continue uninterrupted operations, it illustrates the importance of replacing this analog system which is showing its age.

Missed Page Form & Map

Over the weekend we developed a web form for firefighters and EMS personnel to fill out any time they miss a page or experience a poor quality one.

The form is found on the Lapeer County Fire Association and Lapeer County EMS websites at www.lapeercofireassoc.org/missedpage.htm and www.lcems.org/missedpage.htm, respectively.

When users make a submission, we get the email results and then plot them on a publicly available Google map to show where the problems are occurring the most. We then, along with Blumerich and Motorola, will use the information to troubleshoot the issues with the paging system and hopefully if the millage passes tomorrow, will use the same information to design a new paging system and provide data for better tower placement.

The maps are located at www.lapeercofireassoc.org/pagemap.htm and www.lcems.org/pagemap.htm.

EMS Paging and Dispatch Servers Restored

The Zetron paging system was repaired by the company last week and returned to us. With Blumerich’s technical assistance it was brought back online late Friday afternoon. During the Zetron outage we set up the fire paging system to be able to tone out the EMS bases so they could still get notified for emergency runs. With that setup, we were only able to tone the bases and no backup personnel. Backup personnel are called out when all EMS rigs are on calls or otherwise unavailable. These personnel come into the base nearest the call for help and get an ambulance to respond.

With the system back the way it was before the lightning strike, we can tone the bases and backup personnel utilizing the alpha-numeric pagers and the Minitor V station pagers. EMS supervisors were unaffected during this time because they carry pagers operating on a cellular system from American Messaging which runs through the Internet. We have left the fire paging setup for EMS in place in the event that the Zetron goes out again. This way there will not be any setup time getting their station tones to go.

Additionally from the lightning strike, two dispatch servers’ Windows operating systems became corrupted. They didn’t crash until late Friday afternoon, the day after the storm. This can happen anytime there is a significant fluctuation in voltage to the server or any computer. Regardless of the tower and all connected equipment being properly grounded and on a UPS system (uninterruptible power supply) the lightning was more powerful than what the grounding could handle and obviously it ruined some things.

The first server to crash was the DSS Equature voice recording server. This machine records all telephone and radio traffic to and from the 9-1-1 center. Several attempts were made to restore the operating system using various repair software. They were unsuccessful and DSS was called in with a spare server. They changed out the servers and reconfigured the spare for our system and got recording back online. Unfortunately there were several hours where nothing was recorded.

Halfway through working on the DSS server the primary CAD (computer aided dispatch) server crashed. Thanks be to God for allowing us to successfully complete a prior project which gave us a backup CAD server with an instant failover software called Neverfail. When the primary server crashed, the backup server instantly assumed duty to become the primary CAD server. The dispatchers only noticed a small hiccup where they only had to restart the CAD application and continue working. Had we not had this Neverfail project complete, the dispatchers would be taking and dispatching calls by pen and paper. This method of dispatching is extremely inefficient and time consuming for this type of business. It wasn’t until today that the primary server was fully brought back online. The backup server has been running operations for the last two weeks. We don’t want to imagine our work being done on pen and paper for this long.

The lighting messed with the CAD server so much that its main board had to be replaced by an IBM technician. During the Neverfail project we signed a hardware and software maintenance agreement with IBM for $1,800 per year. It has already paid dividends as the board that was replaced had a cost of $3,000 and that doesn’t include the time and labor that the IBM tech had invested in the repair of the server. Once that was completed the RAID arrays (redundant array of independent disks) were reconfigured and the Windows operating system reloaded. After that, our CAD vendor VisionAIR had to load their software, manipulate the data from the backup server to the primary server and load and configure the Neverfail software again.

Lightning Strikes Lapeer 9-1-1 Tower

Yesterday afternoon a severe thunderstorm passed through Lapeer County and early on a bolt of lightning is believed to have struck the tower behind the 9-1-1 Center in Lapeer. A dispatcher witnessed an arch of electricity move from one of the stand-off antennas to a leg of the tower structure. The dispatcher also saw the tower glowing briefly. Subsequently, this lightning strike temporarily knocked out all communications in the center including sending the voice radio system into failsoft.

What is failsoft?

If the trunking system loses its control channel or has certain other failures, it is no longer able to operate in the trunking mode. So instead of going into a condition that stops all communication, the system enters Failsoft. In this state all transmitters (channels) turn on and operate in a ‘conventional’ repeater mode. The subscriber radios are able to recognize this state and switch to a predetermined frequency (one of the trunk system frequencies, but not the control channel frequency) depending on their selected talkgroup. In most systems several talkgroups will share a frequency. Some talkgroups may not be assigned a failsoft frequency and these talkgroups will cease to operate during the failsoft period. If a particular failsoft frequency has also failed, the talkgroups assigned to that frequency will also be off the air during failsoft.

 

The system quickly recovered and went back into normal operation. However, one system was unable to recover and that is the part of the paging system which dispatch’s EMS crews for Lapeer County EMS.

The old Zetron unit which runs the alpha-numeric pagers for Lapeer County EMS is no longer communicating with the paging servers. Several attempts to reset the unit and even change paging servers were unsuccessful. Another problem is that this unit has gone off of the Motorola maintenance contract several years ago. We do have replacement parts on hand for this unit, but working on it will cost the center time and labor. We will be contacting Zetron this morning to seek their assistance in diagnosing the problem and programming spare boards to be put into service.

As of yesterday, all bases of Lapeer County EMS (Imlay City, North Branch and two in Lapeer City) are operating on radio and telephone standby until further notice. It is not known when or even if this system can be repaired.

Fire department paging was unaffected and is in normal operation.

Additionally, the wireless communication link to Metamora Police which runs their AICS and Internet services went down during the lightning strike.

Further updates will be posted when information is available.

Voice System Breakdown

This past week we experienced another breakdown in the communications system. This time it is was on the voice side. At the Imlay City tower site voice channel bank number 5 had a power supply and power amplifier failure. Blumerich replaced both parts with new ones after having Motorola ship them with overnight delivery.

The good news out of this is that although there was a failure of equipment, no one noticed that there was a problem. This is because the voice system has plenty of capacity for traffic and although one of the voice banks went down, the system still had excess capacity to handle all of the radio traffic. The only time a failure like this would become noticeable would be if there was a huge influx of users in the area of the affected tower.

In the case of the Champion Bus fire a few months ago, had we lost a voice bank or two, there could have been a possibility of users getting a “bonk” when they try to transmit, indicating the system is too busy at that particular moment to handle their transmit request. The user would just have to keep trying every few seconds to get permission to make a transmission.

The power supply and power amplifier both are covered under our Motorola service contract.

So far in 2010, we are seeing more frequent breakdowns of various equipment at each of the sites than we have in the past. The age of the system is quickly catching up to it.

Thoughts on our communications system

Due to heightened sensitivity of the issues surrounding the paging system I am hearing reports of strange things occurring on folks pagers. Things such as one department being toned and a pager from another department opening up but not hearing any voice, pagers opening up when no one is being toned etc.. Many of these occurrences are not new and have been happening to some degree over the years and I would like to say a few things about that.

First, the Minitor V is a very sensitive device and has a history of opening up on “close” tones. Many of the departments in Lapeer County have tone sets that are very close to one another and on occasion will open up when it thinks it hears its own tone. A few years ago Motorola issued a software update to help control sensitivity of the device. If you think you have a pager that is still too sensitive you can bring it in and I can further adjust the sensitivity of it.

There is also the likelihood that there could be some other transmitting device in a frequency band near our assigned frequency that could be transmitting at high power and causing bleed over into our band and causing a pager to open up. As an example, I’ve had it happen before on my car stereo while listening to the radio that a person on CB with a linear amplifier (which is illegal to use) was pushing so much power on the citizens band that his transmission bled over the FM radio band and blasted me in my car right over the radio station.

Secondly, there are areas of the county that traditionally have poor coverage even though they are in relative close proximity of a tower. The likely cause for this is what is called “dead zones” in radio wave propagation combined with terrain features. This also accounts for why people will be well outside of the county and get a page and have it be crystal clear.

Thirdly, for some who keep their pager on an “open” status, on occasion you hear bursts of data. Those are a couple of different things. One kind is the alpha-numeric pages that Lapeer County EMS uses and formerly some MFR groups. Also the system itself sends out data bursts periodically to “poll” the sites. The system polls the sites to make sure they are either still online, offline or to find out if there is something going wrong that it needs to send out an alarm about.

Degraded Communications

 

There are just so many variables to look at in wireless communications from terrain features, vegetation, local weather conditions, space weather (i.e. solar flares, sun spots), buildings, orientation of the pager, where it is worn on the body, body type, direction the pager (or radio) is facing relative to the source signal, heavy machinery, high tension power lines, electronics and so on. All of these things and more play a role in how radio communications work, or don’t work for that matter. I think that many times, these weird things that happen are not always due to a problem with the system, but can be attributed to external factors interfering with our system.

Also important is the age of our system as we are all very aware of. Blumerich and I keep a very close eye on the system and we check it and test it weekly to make sure it is operating within specification. Even though when it is operating normally, there can still be issues with receiving pages that just cannot be explained other than looking at some of the other variables I mentioned already.

Regardless of all that, the looming age of the system is a big concern and every time we experience a problem with it, it further highlights the need to have it replaced with a new system and more towers installed to increase coverage in the deficient areas. Please continue to keep your ears tuned to any potential problems and let us know about them.

Paging System Update

This week Blumerich Communications (our radio shop) installed a power supply for what we call the “page bridge” which is the unit that failed in March that brought the entire system down. When that happened we installed the only spare power supply we had to bring the system back up and that left us without a spare backup.

Blumerich found another one on eBay and purchased it as is. They tested it and it seemed to work. Then they brought it out this week and installed it to make sure it worked on our system and it did. It ran a little hot (high voltage) but within specification. They took it back out and tuned it down and then put the original power supply back in.

So right now we have a spare power supply again for the page bridge. As we are well aware, it is a used part of equal age of the system and has no warranty. Therefore when and if it is needed, there is no guarantee that it will work for long. But at least we have a backup unit again.

Also we ordered two new power supplies from Motorola for the nucleolus that failed in Lapeer last weekend and we’ll have them in hand on Tuesday.

Paging System

On Saturday, June 19,  the paging system for fire departments and Lapeer County EMS partially went down in the area of the Lapeer tower. The Imlay City and North Branch tower areas were unaffected. It was a combination problem of a faulty power supply and a circuit board called a nucleolus, which is the “brain” of the paging system. Each site has its own nucleolus, which allows them to continue to operate though one site is down. Fortunately we had several spare power supplies and one spare nucleolus.

We felt due to the concentration of population surrounding the Lapeer tower as opposed to the North Branch tower that we were going to pull the nucleolus from North Branch and put it in service in Lapeer to get paging back online in the area. Blumerich Communications (our radio shop) went to North Branch and pulled the parts and brought them to Lapeer. The boards proved to be a bit different from each site and when inserted wouldn’t work right. By all indications, it was only the nucleolus that was bad as both power supplies were working and showing faults, and the two cards (the nucleolus and another card) were not coming up. Working with several cards and different technical methods took quite some time. We found that the  first spare power supply we pulled out of storage turned out to be a bad unit and didn’t work. We were able to insert the original nucleolus into the Lapeer site with a second spare power supply and that brought it back online.

During this time, it was discovered that the nucleolus from the North Branch site was going bad as it would only power up and then go into a reset loop. By this time it was 2am. Earlier in the evening dispatch made phone calls to all the departments advising them of the situation. Sunday morning we returned after getting some rest and took the spare nucleolus back to North Branch. The spare nucleolus needed programming for our system and Blumerich had copied the proper settings from the working board in Lapeer before going to North Branch. Once the spare was programmed and inserted into the system we did a test page with Burlington Fire and people in Tuscola County and Marlette as well as in Lapeer received the page. After the successful page with Burlington it was confirmed that all sites were back online. We directed an ‘all call’ page for the entire county around 1200 on Sunday to alert the fire service that the system was back online at 100% for all three sites.

Additionally, all the Lapeer County EMS bases were on radio standby initially and then only the North Branch EMS base was required to be on radio standby, once the problem was diagnosed. Now all the bases are back on routine paging for calls.

The nucleolus board is an obsolete part and went off the Motorola contract several years ago. Had we not had a spare on hand, it is unlikely that we would be able to get another one or get one repaired. We are going to try and find another one, as well as try and repair the bad one. The nucleolus is a critical part of the paging system.

Coincidentally, earlier in the day Saturday we had a high temperature alarm in the Imlay City tower site. The A/C had gone out and the threshold of heat inside the radio room was exceeded. We contacted the County Buildings and Grounds department and they went out and fixed the problem. Though it was a high temperature, the systems were unaffected. This problem was completely separate and unrelated to the paging problem.