Roaming Verizon ‘Test Guy’ Real Deal

“Can you hear me now?”

The oft-repeated catchphrase is probably listed on some Madison Avenue copywriter’s resume as a top accomplishment for changing the collective lexicon, at least temporarily.

Sporting black horn-rimmed glasses, “Test Man” is meant to personify Verizon Wireless’ constant testing of its network and its commitment to performance.

The latest commercials for Verizon – which purchased Alltel Corp. of Little Rock for $28.1 billion in June – show Test Man and his posse of Verizon technicians saving their customers from a host of creepy characters warning them about the horrors of cellular “dead zones.”

It turns out that those commercials aren’t far from the truth.

Cell phone companies actually employ test men and women who drive around searching for wireless dead zones. These technicians – driving about 31 million miles annually and making millions of test calls – are constantly monitoring the network.

“My testing is really the icing on the cake,” said Ken Lee, a real Verizon Wireless test man. “If our customers drop a call, Verizon records it automatically. My testing is a simulation of the customer. It is testing that goes above the norm. If we find a fault, we will send someone to fix it.”

Verizon Wireless has a team of about 90 specially equipped vehicles driven by test men and women across the country as part of its baseline program, which was created in 2000. The job of the test men, known more formally as baseline engineers, is to monitor and test the network to ensure proper function for their regions. Each vehicle automatically makes more than 3 million voice call attempts and more than 16 million data tests annually on its network.

In a 2007 white Chevrolet Uplander, Lee drives through north Texas, Oklahoma, Arkansas and Louisiana with the help of two other test drivers.

His daily testing routine consists of eight-hour and occasionally 10-hour days during the company’s parameter hours between 7 a.m. and 7 p.m. That 12-hour period was chosen for testing because that’s when the network is most heavily used.

“Probably I spend two-thirds of my time in the van and one-third of it processing the data,” Lee said.

Lee’s vehicle may look like a normal van on the outside, but the equipment inside is worth about $250,000.

Cell site antennas for several cell phone companies spring from the van’s roof. Square black patches that work as data antennas adorn the windows. No seats fill the back section of the van; instead, four metal boxes are strapped to the floor. Two boxes are filled with cell phones connected to 18 computers that are built on a single circuit board. The other boxes have air cards, devices found in laptops, PDAs and cell phones that allow wireless connection to the Internet. The testing equipment consists of a two-part system – voice call testing and data testing.

 

How It Works

The voice call testing system conducts 2.5-minute phone calls from Verizon cell phones as well as from its competitors. Then the phone waits 20 seconds before repeating the process. Each phone call contains a taped cell phone message that is answered at a test center. The equipment tests for two types of faults: drops and initiated attempts, known as IAs.

“Drops are when you call someone and you lose the call. An IA is when the call doesn’t go through,” Lee said. “I know when and where [the fault] happens.”

The data testing system consists of air cards downloading and uploading Internet files and Web sites. The system records the time it takes the transmission to download or upload.

“The servers upload and download a specific one-megabyte file,” Lee said. “It’s something not copyrighted, probably something like the Declaration of Independence. We choose that one file every time. It is all about consistency between all networks.”

Both systems are connected to two laptops mounted in Lee’s passenger seat. The computers are connected to GPS systems that show his location. Every hour Lee must stop testing so he can transfer the data from the voice and data test systems to his laptops.

On each laptop screen, a box chart represents each carrier company. Green boxes mean a call was successful, but a red box means a call was faulted, and inside it a specific code details what went wrong.

Because Verizon is the primary focus of Lee’s testing, it is measured for more data, including the signal strength and the amount of interference in an area.

“The machines record all the data. I don’t have any attention to [the laptops] while I am driving,” Lee said.

When the vehicle is in motion, the laptop screens display a light blue trail with a green dot showing where Lee is located. Dark blue dots, representing when a call has been made, slowly appear on the trail. Lee said the vehicle makes about 20 calls per hour. 

 

Ground Coverage

The Verizon test vehicles travel about 1 million miles across the country annually. All set routes, based on U.S. Census population figures, are driven every quarter. Lee said he usually averages about 500 miles per week of testing and another 300 miles traveling to where he needs to test. The route for the Little Rock area is about 1,000 miles.

Lee has only driven the Northwest Arkansas market a couple of times since Verizon launched its service in the area. The route takes him about two days and 375 miles, he said.

“We drive the same route every time,” Lee said. “If we drove a different route, it would be like comparing apples to oranges. All major roads are driven for places with a population of more than 1,500 people per square mile. We only cover major highways for places with under 1,500. We couldn’t possibly drive everywhere, even though I am sure my bosses would want me to.”

While the increase in gasoline prices has affected almost everyone else’s time behind the wheel, Lee said higher fuel costs haven’t altered his testing duties. “The testing is too important,” Lee said.

The cost of gas for the central Texas region’s three test men and 12 radio frequency engineers is about $10,000 a week.

“It’s a helluva lot of driving,” he said.

Currently, three Verizon radio frequency engineers, who design and fix wireless networks, are located in Little Rock, and pretty soon a test man will be joining them, Lee said. Lee is based in South Lake, Texas.

According to the Verizon Wireless Web site, the cell phone carrier was formed in 2000 and has invested more $45 billion since then in its national network and the addition of services. The wireless phone carrier, headquartered in Basking Ridge, N.J., has about 67.2 million customers.

“Once we have processed the data, then we spend money trying to fix [the network]. Verizon takes my data very seriously,” Lee said. “The real purpose of my job is to make sure that Verizon Wireless has the best service of all the carriers in Arkansas and it stays that way.”  

 

The Allover Network

Testing the networks of competitors helps companies ensure they are providing quality coverage that is equal to or better than that of other carriers. AT&T primarily tests its own network but also uses the help of third-party companies.

“Internally, we have 14 radio frequency engineers who drive and fix the network for us in Arkansas and Oklahoma. Any of our employees and customers can file an issue report,” said Larry Evans, vice president and general manager for AT&T wireless operations in Arkansas and Oklahoma. “Externally, we like to have a third-party opinion outside our business. That way we can get a better feel of how our performance compares to others.”

According to AT&T’s Web site, this year hundreds of test drivers will drive 30 million miles throughout the country to examine the quality of the wireless network.

Brian Wann is an AT&T radio frequency engineer and performance leader for the 3G (third generation) network, the latest AT&T network released in the market. He and his radio frequency department find network problems based on key performance indictors. Wann said the department is divided into two sections: design and performance.

The design section creates new cell sites for the area. The performance section monitors the network and looks for problems or alerts.

“As we drive through the network, we take the readings and match them against our statistics. The system is set up with an audio indicator that tells you when we are entering a new cell site,” Wann said.

The AT&T engineers use TEMS, an Ericsson product, to check the EDGE (Enhanced Data GSM Environment), AT&T’s Allover nationwide network and 3G networks. The phones are connected to the laptops, and the data can be viewed on the laptop or recorded in a log format. The system tests for signal strength and can indicate from which cell site the signal originated. Typically, one phone records data and the other sets up calls. Wann can test four phones at a time and conduct a laptop data test by using laptop cards.

“I can set up a call and let it go,” Wann said. “It is a good customer simulation because most customers drive while using their phones. It is on a loop so if it is dropped, it will retry the call again.”

Every day the engineers grade their network on accessibility (whether a call can be started) and retainability (whether the call can be sustained). Wann said they don’t drive the network daily but they drive it as much as they can. “We do have certain drives we must do, like when the network changes.”

Earlier this year, the company announced a $40 million investment in the Arkansas wireless network. This amount, added to the company’s three-year wireless investment, will total more than $125 million.

The 2008 investment will help fund the addition of 34 cell sites in the state. It also will enhance the network coverage for those in the Fort Smith, Hot Springs and Little Rock metro areas; Benton and Washington counties and in southwestern Arkansas and the Arkansas River Valley.

Between 2005 and the end of this year, AT&T will have invested more than $20 billion in its wireless network, the company says on its Web site. The Dallas-headquartered carrier serves about 71.4 million customers.