All the dope on
Boston people meters
Nielsen's Ken Wollenberg interprets the findings
By Kevin Downey
When Nielsen Media Research switches over to its first Local People Meter in Boston next May, it will be the beginning of the end in the biggest TV markets for the paper diary that has been in use for decades to collect demographic ratings on TV viewing. Nielsen has been demonstrating the system in Boston since April and began releasing TV-viewing data from the LPM shortly afterwards. The results were a long time in coming. Media researchers have complained for years that the diary used to collect demographic ratings was ineffective at capturing viewing. The pressure on Nielsen to update the system, which also includes set-top meters to measure household viewing in the biggest markets, got stronger as the sheer increase in the number of TV networks made tracking what people watch even more difficult. The advent of digital TV propelled what was concern into outright panic. But while the LPM is widely considered a major advance, many researchers have concerns about it, specifically regarding the data that are coming out of it. Results from the Boston LPM show that the number of households watching TV fell by as much as 18 percent compared to the previous HH meter/diary setup. And at the same time, the number of people watching, especially younger viewers, jumped by up to 31 percent. Ken Wollenberg, who is spearheading the introduction of the Local People Meter at Nielsen, discussed the LPM and the concerns of researchers with Media Life on Friday.
How do you know that the results from the Local People Meter are better than those from the diary?
I had a conversation with a group operator not long ago and we got off on this area. I told him that somebody had given me a book called, "I’ll Know It When I See It."
The whole discussion about quality is about how nebulous the definition of it is.
As a research company, we are somewhat blessed by absolute indicators of quality. They are things like cooperation rates, response rates, sampling error, and effective sampling base.
No matter how you look at the numbers for the people meter in Boston, it is a higher-quality measure.
The cooperation of basic homes, which are the ones originally designated when we draw the sample, is better. We know which households we want. If we were to get all of those households, our cooperation rate would be 100 percent. As of Sept. 23, the LPM has an initial cooperation rate of 52.1 percent. The set-tuning meter has an initial cooperation rate of 49.6 percent.
There’s a more important cooperation rate, which is ongoing cooperation. That says, in this moment of time, what percentage of homes are basics versus alternates. With the LPM, 48.0 percent of homes are basic compared to 40.7 percent with the set-tuning meter. That’s about a 20 percent increase.
The third measure, called the sample performance indicator, includes response rates and is based not only on installed homes but on the number of homes reporting in on a given day. The SPI for the People Meter is 35.6 percent versus 30.1 percent for the set-top meter.
Whatever measures we use, including the sample characteristics, are much better with the LPM. And the sampling error with the people meter is much lower.
So we have absolute ways of knowing which is the higher-quality measure.
What you’re saying then is that you know the results from the LPM are better because the data that are coming in are better?
All the statistical measures that you would use to understand quality are much higher in the LPM. That’s saying it very simply.
What analyses have you done to compare the new system to the current system? And how is Nielsen communicating those results to clients?
We’ve done any number of analyses, some of which I just referred to. But we also look at the numbers themselves.
From the beginning, we’ve looked at the HUT (households using television) and PUT (people using television) levels. We have done comparisons on the station level and at the cable programming service level.
Every day there is a set of overnight ratings that clients can access. The difference with the LPM overnights is that they can access not just households but also demographics.
There is a local market report produced every month in Boston. The reports come out once every month instead of six times a year as they do with the regular service.
And there was a September report from the LPM so that the stations and agencies could understand how the unusual events of Sept. 11 impacted viewing.
We also had a series of breakfasts and client meetings in Boston that all parties were invited to. And we’ve sent out a series of e-letters. So, it’s as open as open can be. The analysis is as complete as our clients want it to be.
Why are the HUTs lower with the LPM than they are with the meter/diary system?
In the second week of the demonstration period, in the May survey, there were some double-digit differences in the HUTs.
What got everybody’s attention is that you would think the two systems would produce the same HUT levels. The HUTs are just a matter of whether a set is turned on or off and what channel you’re tuned into.
We started doing a number of analyses to explain the differences. We looked at the characteristics of the two panels to see if there was something different, in terms of age, sex, geography, income, or occupation that would cause a different set of numbers.
We looked at what’s called cut-back samples, meaning looking at the 120 national survey homes to see if there were differences with the new homes that we hooked up. We looked at other markets. And we looked at the calibration of the meters.
We also looked at edit rules and we looked at the basic versus the alternate homes.
One thing that jumped out at us is that the alternates in the set-tuning meter panel had much higher tuning levels than the basic homes in the set-tuning meter panel. And they were higher than the homes in the LPM sample.
It wasn’t that the LPM was lower, it was that part of the set-tuning meter was higher. Since they had a much higher percentage of alternate homes, it brought up the question of whether that was driving it. But it bore no fruit; it seemed to be a coincidence.
What we have come to realize is that there were two areas that presented themselves. One was the occupational profile differences in the panel. There is a much higher percentage of people not in the labor force in the set-tuning meter panel.
We took a look at that and reran some of the data using that as a weighting criterion. We saw that that probably accounted for about 10 percent of the HUT differences.
We discovered something else that we call tuning-without-viewing. That’s when a person leaves the set on but is not in the room viewing. A person has a little remote control and you do two simple things. If you are assigned number 1 in your house, you push "1" and you push "OK."
Knowing that, think about tuning-without-viewing. As an example, you come home, you turn the set on, but you go off and do other things.
What we’re discovering is that there is a lower percentage of tuning-without-viewing in people meter households than we would have expected. When the set is turned on, it goes into HUT calculations. What we’re beginning to sense is that people in the people meter households are doing one of two things.
They are either not turning on the set. Or, when they turn on the set and then realize they are leaving the room, they shut their set off.
It’s not as though these people are watching and stations are losing viewers because of it. They are not in the room and they’re not paying attention.
We believe that tuning-without-viewing accounts for maybe as much as 60 to 70 percent of the HUT differences.
Between that and the occupation, it’s probably up to 85 percent of the difference in the HUT levels.
We can address both of these things, and we’ve put together research teams to look at them.
If you find there are some things that need to be adjusted with the LPM that corrects the HUTs, how can those fixes be made?
With the occupation data it could be as simple as creating weighting criteria to understand how to better recruit people.
Regarding tuning-without-viewing, it could be as simple as instructing people what to do, or maybe there’s a software change we need to make.
Even without addressing these two areas, though, there has been a narrowing of the differences between May and September results. One of the reasons, we suspect, is that on Sept. 20, we went from 420 to 600 households, and the profile of that replicate was a bit different from the rest.
Are the HUTs from the LPM accurate? And are the results surprising to you?
The results are not surprising because if you look back over time, whenever people meters have been introduced and compared to set-tuning meters there have been differences in the estimates.
That was true in the mid-80s when the national service went from set-tuning meters to people meters. It was true in the late-80s when Arbitron introduced ScanAmerica.
There is a track record for this.
What we’ve said is that one of the purposes of the demonstration period is for our clients to get used to the estimates, to understand the differences, and make whatever adjustments they need to their market estimates.
Another reason for the demonstration period is for us to understand the differences. We’ve spent a lot of time doing analyses to try to explain those differences.
You don’t know starting out which number is more accurate.
Why is the People Using Television number higher with the LPM when HUTs are lower?
The thinking behind that is that the People Meter does a much better job than the diary of registering difficult-to-reach demographics, like younger men and women, teens, and children.
The diary probably understates some of their viewing.
Tomorrow: Part II: All the dope on the Boston people meters
October 22, 2001 © 2001 Media Life
-Kevin Downey is a staff writer for Media Life.