Internet polls based on volunteer panels have quickly captured a significant slice of the UK polling market, based in large part on success at predicting the outcome of recent elections. However, opinion research is most usually conducted on a wide range of issues that cannot be measured against an election outcome and are only loosely linked to voting behaviour. This paper compares the results obtained from a representative sample of people interviewed by telephone with the internet-accessible population, those willing to join an internet panel and those who actually respond online. Across a range of subjects similar results are obtained, but on others differences emerge that suggest online panels cannot reliably replicate results obtained by more traditional research methods.
Introduction
The rapid diffusion of the internet in recent years has brought in its wake a new method of conducting opinion polls. In the UK this new approach has been championed in particular by YouGov, which in just two years has rapidly captured a significant proportion of the market. The company has done so not least because of its success at predicting the outcome of a number of recent events, including the 2002 London borough elections and the 2003 Scottish Parliamentary election, as well as the result of the Conservative leadership election and the first Pop Idol contest. Given the continuing difficulties that some polling companies have had in providing an accurate picture of vote intentions at elections (Crewe 2001), YouGov's recent record has proved to be a formidable selling-point.
This success has of course been achieved in the face of most conventional wisdom. YouGov's polls are not probability samples. Each YouGov poll is conducted among a subset of a panel of people who have visited the YouGov website and signed up to participate in the company's surveys. They are thus similar in character to the polls conducted in the US by Harris Interactive (Taylor 2000). The method means that those who do not have access to the internet (who still constitute around half of the UK population) and those internet users who have not enrolled to participate in YouGov's polls have no chance of being polled by the company.
But while success in predicting the outcome of elections may be necessary evidence of the ability of internet polling to provide an accurate picture of public attitudes, it is not on its own sufficient as most polls are conducted on subjects unconnected with vote intentions. Carefully designed and controlled comparisons between internet and probability samples in the US (where internet penetration is 50% higher than in the UK) have indicated that sometimes the two methods produce different results (Krosnick & Chang 2001; Smith 2003). So in this paper the authors report the results of research they have conducted in the UK to examine whether the results of internet polls based on a volunteer panel can be relied upon to provide accurate estimates of the distribution of public opinion as well as vote intentions. The key finding is that Couper's (2000) comment that 'the extent to which weighting the results from individual panels can reliably produce reasonable estimates is unknown' still has considerable validity.
The characteristics of internet polling
There are at least five characteristics of internet polling of the kind conducted by YouGov that might lead to results that are different from those obtained by other, more conventional methods such as telephone interviews undertaken using random digit dialling, the method that in the 1990s became the most common in conventional UK opinion polling.
Sampling bias
Not everyone has access to the internet. As already noted, only 50% or so of the UK population currently have access, and those who do tend to come from the younger and more affluent sections of the population.
Sampling frame
Probability sampling requires a sampling frame (such as the electoral register or postal address file) that makes it possible to undertake a random selection of persons to interview. No sampling frame of persons in the general population who have access to the internet is available. There are a number of ways of meeting this difficulty (Couper 2000). The one with which this paper is concerned is a non-probability method of recruiting a panel of volunteers via advertisements on selected websites, which is the method used by YouGov.
Response rates
While all modes of survey research have experienced declining response rates in recent years, most researchers who have conducted internet surveys in parallel with other methods have found that the response rates to internet surveys are usually lower than for other methods (Cook et al. 2000; Krosnick & Chang 2001; Vehovar 2003).
Conditioning and attrition
Those who are asked on a regular basis for their views are always liable to have those views influenced by the very fact that their opinions are regularly being solicited (Mitofsky 1999). Meanwhile, those who decide to maintain their membership of an internet panel may well be different from those who at some point opt to drop out (Gregory 2002)
Mode effects
The answers that people give to pollsters may well depend on how they are interviewed. It has, inter alia, been suggested that people are more likely to answer sensitive questions honestly in an online poll as there is no requirement to give a potentially embarrassing answer to an interviewer, that they are more likely to use the end-points of a five-point scale and that they are more likely to give apparently incorrect answers. However, the extent and even the consistency of these effects is far from clear (Taylor 2000; Krosnick & Chang 2001; Vehovar & Lozar Manfreda 2002; Smith 2003).
The focus in this paper is primarily on the possible impact of the first three of these characteristics. The authors are concerned to establish the extent of the potential bias that might be introduced into a YouGov-type internet poll by the fact that half the population is excluded, that people have to volunteer to participate and that even many of those who do volunteer may fail to complete any individual poll. In so doing the authors bear in mind the possible existence of mode effects, but their research does not encompass the possible impact of conditioning and attrition. This exclusion would not, however, appear to vitiate the conclusions that are drawn.
Data
Between October 2002 and August 2003 ICM set out to recruit a panel of people willing to join an internet polling panel via the company's telephone omnibus survey. Apart from basic demographic questions respondents were asked whether they had access to the internet and whether or not they would be prepared to join an internet polling panel (after being offered an incentive similar to that offered by YouGov of [pounds sterling]0.50 for every survey completed). In all, 71,018 interviews were contacted--a response rate of 25%. Of that total 61% (43,141 people) claimed to have access to the internet and 19% (13,501 people) agreed to join an internet polling panel. We can use these data to compare the demographic characteristics of, first, those who are accessible over the internet and, second, those who say they are willing to join an internet panel, with the characteristics of those who respond to a telephone poll. Meanwhile, the demographic representativeness of this telephone poll can be assessed by comparing its data with those collected by a random face-to-face probability sample, the National Readership Survey (NRS). (1)
In addition, the first 4014 people who were contacted as part of the panel building programme, that is those who were interviewed between 11 and 24 October 2002, were also asked a series of political and attitudinal questions. Of these 52% (2088 people) said that they had access to the internet, while 15% (588) were willing to join an internet panel. The authors use these data to compare the attitudes of their three groups of respondents. At the same time an attempt is made to verify the accuracy of the attitudinal profile obtained by the telephone sample as a whole by comparing it with the results obtained when some of the same questions were asked on an in-house face to face random probability sample of 2030 people undertaken by ICM for the Oxford Internet Institute. The response rate to this survey, which was conducted between 23 May and 28 June 2003, was 66%. (2)
This first exercise allows us to assess the impact on the results of sampling bias obtained by an internet poll and the strategy that is adopted to cope with the absence of a sampling frame. It does not, however, allow us to assess the possible impact of low response rates or indeed mode effects. To do this we need to conduct a survey online. Therefore, ICM attempted to recontact by phone 4011 of its potential internet panellists to ask them to go online to read an email link to an online survey and then complete the survey, which contained many of the same political and attitudinal questions that were asked in the earlier exercise. These 4011 were a subset of the 4616 people who had agreed to join an internet panel when they had first been contacted as part of the initial panel building exercise between December 2002 and March 2003. Of this group 2027 were successfully recontacted and eventually 1000 of them actually completed the survey at some point between 7 and 15 May 2003 after being offered an incentive that [pounds sterling]1 would be paid to the Red Cross for each complete interview. Although every effort was made by the use of quota controls to ensure that those who were successfully recontacted and asked to complete the online survey were demographically representative of the adult population, older people still proved to be poorly represented in the final achieved sample. Therefore between 1 and 10 September 2003 a further 153 potential panellists, all aged 55 plus, were successfully persuaded to complete the survey online, with those aged over 65 being offered the incentive of a [pounds sterling]5 donation to the Red Cross for doing so. These respondents were drawn from the additional potential panellists who had been identified between April and August 2003.
There are, of course, some differences between these two exercises and how a YouGov-style internet poll is conducted. For example, the fieldwork period for our online poll was longer than the 48 hours during which a YouGov poll is conducted. But that should help enhance the representativeness of our online exercise. Meanwhile, most importantly of all, both those who in the first exercise said that they were willing to join an internet panel and those in the second survey who actually completed an online survey were actively recruited (via random telephone dialling) rather than volunteers as is true of YouGov's panel. This, however, arguably meant that the methodology used to acquire these two groups was closer to being a probability sample of the internet accessible population than is true of a YouGov-style poll. So again the difference is one that would seem likely to reduce rather than increase the chances of the authors finding differences between internet polling and a conventional telephone sample (Flemming & Sonner 1999). If anything, then, the approach is a conservative one.
Representativeness
Table 1 examines how those who have access to the internet and those who are willing to participate in an internet panel differ from the general population. The second column shows the current social profile of all adults as measured by the NRS. The next column shows the unweighted profile of the sample achieved by telephone. The fourth and fifth columns show the demographic profile of two key subgroups in the telephone sample after that sample had been weighted to the demographic profile of the NRS, i.e. the subsamples that could have been obtained if the original telephone sample had matched NRS exactly. The first subgroup comprises those who in the initial telephone survey said they had access to the internet at home. The second consists of those who in that same survey said not only that they had access to the internet but also that they would be willing to join an internet panel. Meanwhile, the final column of Table 1 shows the unweighted profile of those who actually completed the online survey.
As we would expect, those who in our first sample said they had access to the internet at home were younger, more middle class and better educated than the general population. They were also more likely to be men. But those differences are exaggerated further among those who say that they are willing to join an internet panel. Those who are willing to participate in internet polls are it seems even less representative of the general population than are those with access to the internet in general. In other words there is a particularly high risk that among those groups where internet penetration is low, those who are willing to join an internet panel may prove to be atypical of all people in that group.
The results appear to reflect YouGov's experience, even though that company makes particular efforts to reach those belonging to groups with low levels of internet penetration (e.g. by advertising for volunteers on a horoscope website). Kellner (2003a) reports that YouGov's panel of 40,000 people contains 57% men while the authors' group of potential internet panellists comprises 56% men. And while the proportion of YouGov's sample aged 55 plus is, at 17.5%, somewhat higher than the 12% figure among the authors' potential panellists, it is still well short of the 33% that the NRS indicates that this group comprises in the population as a whole. Equally, the 39% of the authors' potential panellists who said that they read a tabloid newspaper is only a little lower than the 42% of YouGov's panellists who do so.
Those who completed the authors' online survey were in some respects at least somewhat more representative of the adult population than the authors' potential internet panellists, which of course was precisely the result that the quota controls were designed to achieve. Even so, they proved to be just as unrepresentative in respect of gender, car ownership, having been on a foreign holiday and education as were potential internet panellists. We should note too just how small a group those who actually completed the authors' online survey proved to be. Of the original 71,018 respondents only 19% agreed to participate in an internet panel. And then when the authors attempted to turn their potential internet panellists into online respondents only 25% actually completed the online survey. In other words the authors' online respondents represent a little under 5% of those who were successfully contacted by the telephone survey. In short, the authors' experience confirms that, even with handsome incentives and repeated reminders, internet surveys do indeed suffer from poor response rates.
Attitudes
So even actively recruited internet polls can it seems only access a small socially unrepresentative subset of the population. And, as Table 2 shows, their political attitudes are rather different too. The table compares the answers given by the same four groups that were identified in Table 1 to a set of questions regarding voting intentions, political attitudes and political knowledge. The subject-matter of these questions ranged across some of the key debates about public policy in the UK and full details of their wording are given in the appendix. And we can see, inter alia, potential internet panellists are markedly more interested in politics than are the general public, while they are also less in favour of wealth redistribution, less supportive of the death penalty but more favourable to joining the euro. They also have a greater propensity to support the Liberal Democrats. Moreover, most of these differences were even more marked for those who actually completed the online survey.
But a YouGov-style poll is not conducted simply by taking a random sample of those who have volunteered to complete its polls. Rather, a stratified sample is issued, a stratification that takes into account the known demographic characteristics of individual panellists. So while, say, older females may be underrepresented among YouGov's panel, the number asked to complete any individual survey will be proportional to their size in the general population. Moreover, the data that are eventually obtained can be weighted so they are representative of the known characteristics of the general population. So the key question to address is whether weighting strategies can be found that have the effect of bringing the results obtained by those of the respondents who say that they are willing to join an internet panel, and those who completed the online survey, into line with those of the original telephone sample. In short, can those who respond online be made to look like the rest of the population, or does the fact that someone is willing and able to participate in an internet poll make them unrepresentative of their fellow citizens regardless of what else we know about them (Best & Krueger 2002)? If they can be made to look similar we might conclude that YouGov-style internet polls can provide a reliable estimate of political attitudes as well as vote intentions. If not, then doubts must remain.
Weighting
The most obvious strategy that we can apply is to weight all four samples so that in each case their social profile matches the known characteristics of the adult population. To do this the authors deployed a 56-cell matrix comprising categories of gender, age, housing tenure and social grade within region together with separate cells for work status, car ownership and having taken a foreign holiday in the last three years. This is, in short, a demographic weighting schema that might commonly be applied to any opinion poll. The results for all four samples after applying this procedure are shown in Table 3.
Two key findings emerge. First, with just a few exceptions the figures for those who have access to the internet are almost identical to those for the telephone sample as a whole. The exceptions are that those with access to the internet are (unsurprisingly) more likely to say that they like to try new things, are less opposed to joining the euro and are less in favour of the death penalty. Still, for the most part it appears that if internet pollsters could acquire a random sample of all those with access to the internet then their weighted results could on most subjects be expected to be reasonable estimates of the distribution of opinion among adults as a whole.
However, the second key finding is that the figures for those who say that they are willing to join an internet panel are rather more likely to be different from those for the sample as a whole. Unlike those on the internet in general, those willing to join a panel are considerably more interested in and knowledgeable about politics, as evidenced by their greater willingness to keep up with political news, to know the name of the Foreign Secretary and to have firm views about politics (see also Krosnick & Chang 2001). In addition, they are rather keener on better public services even if it means more tax while they also share the tendency of all those on the internet of being more sympathetic to new things. And while the views of potential internet panellists are not statistically significantly different on the euro and the death penalty these items still exhibit a difference in the same direction as that for all those on the internet. Moreover, if we calculate the mean difference between the figure for potential internet panellists and that for the total telephone sample across all the items in the bottom section of Table 2 we find that at 5.6 the gap is rather larger than that for the difference between all those on the internet and the total sample (3.6).
Moreover, nearly all of the differences the authors have identified so far are also evident when we look at those who actually responded to the online survey. These respondents were more interested in politics, keener on the euro and more likely to be opposed to the death penalty. But there are other significant differences too. The online respondents were more likely to back the Liberal Democrats and less likely to support Labour, more likely to believe the monarchy is outdated and more likely to oppose the redistribution of wealth. Meanwhile, while the potential internet panellists were rather keener on better public services even if it meant more tax the online respondents were very much less inclined to take that view. It seems that the closer we approach the methodology of an internet poll, the more we depart from the results of a conventional telephone survey.
But conventional demographic weighting is not the only tactic that we can deploy. After all, some internet pollsters themselves deploy so-called 'propensity weighting' rather than just demographic weighting in order to try to correct for differences between the characteristics of those who participate in internet surveys and those of the general population (Taylor 2000; Terhanian et al. 2001; Terhanian & Bremer 2002). Meanwhile, opinion polls in the UK are now often weighted so that the proportion of people who say they voted for each party at the last election (past vote) matches some expected figure (though not necessarily simply the result of the last election because of errors in people's recall of their past vote) (Himmelweit et al. 1978; Curtice & Sparrow 1997). Perhaps once weighted in this way the differences in political attitudes between those inclined to respond to an internet survey and the total sample will disappear. Meanwhile, attitudes towards social issues such as the death penalty are often heavily influenced by educational background (Evans 2002), something not weighted for so far even though the authors have shown that those who participate in internet polls are more highly educated. So perhaps further weighting by the age that someone left full-time education would also help eliminate some of the differences in attitudes the authors have seen.
However, the authors' hopes are not fulfilled. Even when samples are weighted so that their reported past voting behaviour is the same, their current political attitudes prove to be rather different. While this weighting strategy does mean that the potential internet panellists and the online respondents are for the most part no longer significantly more knowledgeable and interested in politics, they continue to be more likely to say that they have firm views about the subject. Meanwhile, the potential internet panellists are still significantly keener on better public services (though also their reform) and more supportive of adopting the euro. And those who actually responded to the survey still exhibit particularly strong opposition to wealth redistribution and higher taxes, a greater tendency to believe that the monarchy is outdated, greater opposition to the death penalty and a greater willingness to support the Liberal Democrats, while at the same time also retaining many of the differences exhibited by those who were simply willing to join an internet panel.
The authors have, then, found that those willing to join an internet panel and those who respond online share an apparently greater level of support for the euro. Meanwhile, perusal of Table 4 indicates that they may well share distinctive views in respect of the death penalty too. These findings suggest that there are indeed differences between internet samples and telephone polls that cannot readily be corrected by the kinds of weighting strategies we have been able to apply. Moreover, that they occur both when questions have been administered by telephone and when they have been answered online indicates that they are not the result of mode effects. On the other hand, the very distinctive answers given by the online sample to questions on taxation and public services, income redistribution and the monarchy could perhaps be the product of mode effects. Perhaps, for example, those who answered online felt less social pressure to give what might be regarded as the socially desirable answer in support of better public services.
There is in fact another possible explanation. The online survey was administered after the introduction of an increase in national insurance in April 2003. In contrast, the telephone survey was conducted before hand (though well after the announcement in March 2002 that the increase was going to take place). So perhaps support for higher taxes had diminished among the population as a whole, and this was reflected in the online survey. However, this proves to be an inadequate explanation. For when the same question was administered by ICM to a telephone sample, again in June 2003, the difference between the proportion agreeing with better public services and the proportion disagreeing had only fallen from 48 points to 41.
But it appears that mode effects are an inadequate explanation too. In order to assess this possibility, ICM recontacted 30 of the online respondents, explained that we had failed to capture the information on tax and re-asked the question. Not a single respondent gave a different answer to the one they had given online.
Of course this still begs the question as to whether the estimates of the balance of opinion in the poll are reliable at least so far as what people are willing to tell an interviewer is concerned. To check that they are, the authors were able to include questions about the euro and the death penalty on the face-to-face random probability survey that ICM conducted for the Oxford Internet Institute. (3) As the results of this survey were weighted using exactly the same method as that used to produce the figures for Table 3, these are the relevant point of comparison. And on the euro the Oxford survey produced a 30-point majority for those who disagreed with its adoption, closer to the figure for the authors' total telephone sample than that for either the potential internet panellists or the online survey. Meanwhile, on the death penalty it produced an even bigger majority in favour of reintroduction--that is, 30 points--than our telephone sample obtained.
So it appears that if it were possible to contact a random sample of all those with access to the internet, the resulting data would, when appropriately weighted, often closely match those of a telephone survey. But people who say they are willing to join an internet polling panel are rather more likely to give answers that are different from those given by respondents to a telephone survey and those differences cannot be overcome by conventional weighting strategies. Thus, by implication, they also cannot be overcome by stratifying those respondents who are asked to complete any particular internet survey. Meanwhile, not only are those differences largely replicated when the same questions are asked online but further and sometimes rather puzzling differences also appear.
Real comparisons
These conclusions are based on the evidence of experiments. Perhaps in practice YouGov has found a means of overcoming the apparent problems the authors have uncovered. After all, as noted at the beginning of this paper, the authors actively recruited their online panellists whereas YouGov relies on volunteers. At the same time, also as already noted, YouGov makes particular efforts to advertise for volunteers on websites that are particularly likely to be accessed by those belonging to social groups that are less likely to have access to the internet. So perhaps the discrepancies uncovered by the authors' experiments do not appear in practice.
As far as voting intentions are concerned there have been some differences between ICM's telephone polls and those produced by YouGov. YouGov started polling voting intentions on a regular basis in January 2002 and over the next 12 months its estimate for both the Conservative and the Liberal Democrat share of the vote was on average just 0.6% higher than that obtained by ICM, while its estimate for Labour was 1.8% higher. Over the first ten months of 2003, however, the differences were rather wider, with YouGov's Conservative estimate on average higher by 2.3%, its Liberal Democrat vote 1.5% higher and its figure for Labour no less than 3.1% lower. (Details of the question wording used by the two companies to elicit voting intentions together with the wording used in all the other comparisons made in this section can be found in the appendix.)
As Table 5 shows, these discrepancies occur even though during 2003 YouGov has been weighting its data to a lower target Liberal Democrat past vote than ICM and a higher Labour one (Kellner 2003b), a difference that, other things being equal, would lead one to expect YouGov to produce a much higher Labour estimate and a lower Liberal Democrat figure than ICM. That YouGov still has some apparent tendency to produce a higher Liberal Democrat figure than ICM despite using a lower target past vote weight suggests that the danger indicated by the authors' online survey that such surveys will produce a higher Liberal Democrat estimate is a real one.
But what about the authors' principal interest in this paper--that is, on questions other than vote intentions? Is there any evidence that the differences found in the experiments have been reflected in reality? There is certainly such evidence on attitudes towards the euro, about which YouGov regularly asked people their attitudes between March 2002 and May 2003. As Figure 1 shows, throughout 2002 YouGov consistently obtained a lower level of opposition to joining the euro than either ICM or MORI, only coming into line in the spring of 2003. Quite why the discrepancy suddenly disappeared is not clear.
The authors' finding that those who participate in internet surveys are less likely to oppose the death penalty has also been replicated in real polls. Immediately after the Soham murders in August 2002, a YouGov poll found that only 53% wanted to reintroduce the death penalty for child murders. But when both ICM and NOP repeated the same question in a conventional telephone poll, just days later, ICM found 64% in favour, NOP 68%. (Moreover, both ICM and NOP were able to identify which of these respondents had access to the internet, and the balance of opinion among these was similar to that obtained by YouGov.) The same post-Soham YouGov survey also found just 37% in favour of reintroducing the death penalty for all murders, compared with the 53% figure among all those who participated in the first round of the data collection reported in this paper.
Finally, in April 2002 ICM and YouGov both conducted polls on the increase in national insurance contributions that had been announced in the previous month's budget. YouGov found 55% in favour of the increase in national insurance contributions while 31% were against and 18% undecided. But a poll undertaken two days later by ICM found 76% in favour and 19% opposed. Again, similar question wordings were used. The difference in results obtained mirrors that between the authors' online survey and the telephone sample.
One possible reason for the difference in attitudes to these tax increases is suggested by the answers to another question. Whereas only 47% of the people interviewed by ICM thought that they would be worse off as a result of the budget increases, 60% of YouGov's internet panel though that they would be out of pocket. This suggests that an online sample that is matched to the demographic characteristics of the whole adult population may still be more affluent (and therefore pay higher national insurance contributions) than those who participate in an equivalent telephone poll.
The similarity between the differences found in the present experiments and those found in real polls suggests that the differences the authors have found cannot be dismissed as a product of the particular methodology used to conduct the experiments reported here. ICM may have actively recruited people while YouGov relies on a volunteer panel but both methods, it seems, find a small subsample of the population who are keen to give their views in polls and have attitudes that are different in some respects from the wider population even after a wide range of weighting strategies, including weighting by past vote, have been deployed.
Conclusion
Despite the success of YouGov in anticipating election outcomes at recent elections in the UK, the present research suggests that there is still a need for considerable caution in assessing the ability of internet polls to provide reliable estimates of the distribution of political and social attitudes in the UK. The authors have found marked differences between the attitudes of those who respond to a conventional telephone poll and both those who say they would, and those who actually do, respond to an online poll. Moreover, these differences remain even when the samples are weighted so that they have the same past voting behavior as well as the same demographic characteristics.
This result should perhaps not come as a surprise. After all, the relationship between past vote and social and political attitudes other than vote intention can often be quite weak. This means that there is little theoretical reason to believe that such weighting should ensure that a non-probability internet poll's estimate of the distribution of attitudes is correct.
Moreover, the character of the differences the authors have found gives two particular reasons for concern. First, it appears that they are more the product of the distinctive characteristics of those who are willing to join an internet panel and respond to internet surveys than they are of the social skew of all those with access to the internet. This suggests that continued growth in internet access alone will not eliminate the difficulties assessed here and similar biases found in the US lend weight to this conclusion. What matters more is overcoming the problems caused by the lack of a sampling frame and low response rates. Second, the differences found have no discernible pattern to them. Why, when appropriately weighted, those who respond to an internet poll prove to have similar views to the rest of the population on proportional representation but not on the euro is hard to fathom. It certainly makes it hard to be sure when an internet poll is providing a reasonable estimate of public opinion in general and when not. When Fleming & Sonner (1999, p. 13) obtained similar results they wrote: 'The lack of predictable patterns to the differences raises important questions about the utility of internet polls to replace traditional telephone survey practices.' Those questions remain.
Appendix: Vote intention wording
ICM
The Conservatives, Labour, the Liberal Democrats, Plaid Cymru/SNP (where appropriate) and other parties would fight a new election in your area. If there were a general election tomorrow which party do you think you would vote for?
YouGov
If there were a general election tomorrow, which party would you vote for?
Answer options which appear on screen are:
* Conservative
* Labour
* Liberal Democrat
* Scottish National Party/Plaid Cymru
* Some other party
* Would not vote
* Don't know
Wording of attitudinal questions used in the ICM experiments
Respondents were invited to indicate their agreement or disagreement with the following options on a four-point scale from strongly agree to strongly disagree:
* Britain should adopt the euro
* I am interested in political news
* I tend to have quite firm views on political issues of the day
* We should spend more on public services even if my taxes have to rise
* We should replace the present voting system and introduce proportional representation
* The monarchy is an outdated institution
* Reform is the key to improved public services not more money
* We need to have much tighter asylum laws
* The government should redistribute income and wealth from the rich to the poor
* Britain should reintroduce the death penalty for murder
Wording used in April 2002 post-budget polls
ICM
The Chancellor increased NI [national insurance] contributions by 1% for employers and employees in the budget on Wednesday in order to provide extra money to fund the NHS [National Health Service]. Do you approve or disapprove of this increase?
YouGov
Here are some things the Chancellor announced in his budget. Please indicate for each one of them whether you approve or disapprove of it. To pay for the increase in NHS spending, personal tax allowances to be frozen and an increase of 1% in national insurance contributions, the increase to be paid for at all levels of earnings. Approve, neither approve nor disapprove, disapprove, don't know.
ICM
As a result of the budget on Wednesday, do you expect that you and your family will be much better off financially, a little better off, a little worse off or a lot worse off?
YouGov
Thinking now of you and your family, do you think the budget will leave you and your family a lot better off, a little better off, a little worse off, a lot worse off, not make much difference, don't know.
Wording used in polls on reintroduction of the death penalty
ICM
Do you support or oppose the reintroduction of the death penalty for people convicted of murdering children?
NOP
Would you support or oppose the reintroduction of the death penalty for people convicted of murdering children?
YouGov
Do you support the reintroduction of the death penalty for people convicted of murdering children?
ICM
Agree or disagree with the statement 'Britain should reintroduce the death penalty for murder'. See above wording in ICM experiments.
YouGov
Do you support the reintroduction of the death penalty for any person convicted of murder?
Wording used in euro polls
ICM
If there were to be a referendum, would you vote to join the European single currency (the euro) or would you vote not to join?
MORI
If there were a referendum now on whether Britain should be part of a single European currency, how would you vote?
YouGov
If there were to be a referendum today on whether to join the euro, how would you vote?
Table 1 Demographic characteristics of internet users and panellists Unweighted NRS 2001 Telephone Base 33,439 71,018 Sex Male 49 46 Female 51 54 Age 18-24 11 10 25-34 19 17 35-44 19 21 45-54 17 18 55-64 13 16 65 plus 20 19 Class AB 24 20 C1 28 31 C2 21 24 DE 28 24 Tenure Own outright 28 29 Mortgage 42 42 Rented/other 30 29 Work status Work full-time 46 45 Part-time 11 15 Not working 44 40 Cars No car 23 20 1 car 44 44 2 plus cars 34 35 Foreign holidays In last three years 60 63 Not in last three years 40 37 Education TEA 16 or under 58 49 Demographic weights on total sample Internet- Willing to accessible join internet subsample panel subsample Base 43,144 13,501 Sex Male 53 56 Female 47 44 Age 18-24 14 16 25-34 25 28 35-44 24 25 45-54 19 19 55-64 10 8 65 plus 7 4 Class AB 32 32 C1 34 36 C2 20 19 DE 14 13 Tenure Own outright 21 17 Mortgage 55 55 Rented/other 24 27 Work status Work full-time 60 62 Part-time 13 12 Not working 27 8 Cars No car 13 14 1 car 42 42 2 plus cars 45 44 Foreign holidays In last three years 71 75 Not in last three years 29 29 Education TEA 16 or under 35 32 Unweighted Responded online Base 1153 Sex Male 57 Female 43 Age 18-24 9 25-34 21 35-44 24 45-54 22 55-64 17 65 plus 8 Class AB * C1 * C2 * DE * Tenure Own outright 21 Mortgage 51 Rented/other 28 Work status Work full-time 56 Part-time 16 Not working 24 Cars No car 12 1 car 42 2 plus cars 46 Foreign holidays In last three years 75 Not in last three years 25 Education TEA 16 or under 33 * Data not collected Table 2 Comparing the attitudes of those on and not on the internet (unweighted data) Total Internet sample accessible Base 4014 2088 Vote intentions Conservative 19 19 Labour 29 29 Liberal Democrat 12 15 Other 4 4 Don't know 17 17 Will not vote 11 11 Refused 8 7 Among supporters, percentage certain to vote for that party Conservative 54 49 Labour 45 41 Liberal Democrat 31 29 Political knowledge Percentage who know that Foreign Secretary 37 40 is Jack Straw Underlying attitudes, percentage who agree minus percentage who disagree Like to keep up with political news 49 52 Have firm views on politics 34 36 Like to try new things 80 90 Other issues, percentage who agree minus percentage who disagree Better public services even if more tax 48 48 Redistribute wealth from rich to poor 20 11 Should reform public services not give more 51 48 money Introduce proportional representation 0 0 Monarchy outdated -21 -24 Asylum laws should be tightened 69 63 Adopt the euro -33 -22 Reintroduce death penalty for murder 11 -4 Willing to join Responded internet panel online Base 588 1153 Vote intentions Conservative 18 18 Labour 32 26 Liberal Democrat 18 25 Other 4 8 Don't know 14 13 Will not vote 10 4 Refused 4 5 Among supporters, percentage certain to vote for that party Conservative 53 * Labour 40 * Liberal Democrat 27 * Political knowledge Percentage who know that Foreign Secretary 45 * is Jack Straw Underlying attitudes, percentage who agree minus percentage who disagree Like to keep up with political news 57 63 Have firm views on politics 42 52 Like to try new things 95 * Other issues, percentage who agree minus percentage who disagree Better public services even if more tax 48 14 Redistribute wealth from rich to poor 12 -3 Should reform public services not give more 46 53 money Introduce proportional representation 4 5 Monarchy outdated -19 -7 Asylum laws should be tightened 60 61 Adopt the euro -16 -10 Reintroduce death penalty for murder -2 -7 * Not asked Table 3 Comparing demographically weighted samples Total Internet Willing to join Responded sample accessible internet panel online Vote intentions Conservative 18 17 18 19 Labour 29 30 34 25 * Liberal Democrat 12 14 15 23 * Other 4 4 6 8 * Don't know 16 16 15 13 Will not vote 12 11 8 7 * Refused 8 8 4 4 * Among supporters, percentage certain to vote for that party Conservative 54 54 56 [dagger] Labour 46 44 40 * [dagger] Liberal Democrat 33 38 * 44 * [dagger] Political knowledge Percentage who know that 37 39 45 * [dagger] Foreign Secretary is Jack Straw Underlying attitudes, percentage who agree minus percentage who disagree Like to keep up with 49 50 59 * 61 * political news Have firm views on 35 37 49 * 49 * politics Like to try new things 79 86 * 91 * [dagger] Other issues, percentage who agree minus percentage who disagree Better public services 49 50 58 * 16 * even if more tax Redistribute wealth from 20 19 25 2 * rich to poor Should reform public 50 52 56 57 services not give more money Introduce proportional 0 0 2 4 representation Monarchy outdated -21 -21 -25 -9 * Asylum laws should be 68 64 66 62 tightened Adopt the euro -34 -26 * -24 -17 * Reintroduce death 12 0 * 5 -6 * penalty for murder * Figure is statistically significantly different (using an appropriate t-test) from that for the total telephone sample after taking into account the effective sample size after weighting: for the attitudinal scales the test is based on the difference in mean score where strongly agree = 4, agree = 3, disagree = 2 and strongly disagree = 1 [dagger]Not asked Table 4 Weighting by past vote and education Total Willing to join Responded sample internet panel online Vote intentions Conservative 20 17 18 Labour 26 31 24 Liberal Democrat 13 13 20 * Other 5 7 8 * Don't know 16 19 15 Will not vote 12 9 9 * Refused 8 4 7 Among supporters, percentage certain to vote for that party Conservative 55 55 [dagger] Labour 46 42 [dagger] Liberal Democrat 35 44 * [dagger] Political knowledge Percentage who know that Foreign 37 41 [dagger] Secretary is Jack Straw Underlying attitudes, percentage who agree minus percentage who disagree Like to keep up with political news 48 55 53 Have firm views on politics 35 47 * 44 * Like to try new things 78 94 * [dagger] Other issues, percentage who agree minus percentage who disagree Better public services even if more 48 56 * 11 * tax Redistribute wealth from rich to 20 23 -2 * poor Should reform public services not 51 58 * 58 give more money Introduce proportional 0 6 0 representation Monarchy outdated -23 -23 -13 * Asylum laws should be tightened 70 71 71 Adopt the euro -36 -22 * -26 * Reintroduce death penalty for murder 16 8 7 * * Figure is statistically significantly different from that for the telephone sample: see Table 3 for further details [dagger]Not asked Table 5 Past vote weighting targets, ICM and YouGov, September 2003 2001 election: ICM past votes cast vote weight targets Conservative 33 30 Labour 42 47 Liberal Democrat 19 17 Other 7 6 YouGov past vote weight targets Difference Conservative 31 +1 Labour 53 +6 Liberal Democrat 12 -5 Other 4 -2
Acknowledgements
The authors are grateful to the Guardian newspaper for its financial support of much of the data collection on which this paper is based. They would also like to thank the Oxford Internet Institute for its help in establishing benchmark data from its own random probability survey.
[C] 2004 The Market Research Society
(1) Data from the 2001 NRS were provided by IPSOS-RSL. A random sample of adults aged 18 plus were interviewed during the course of the year. The sampling frame was the postcode address file.
(2) The survey interviewed persons aged 14 plus. The figures quoted in this paper are based only on the 1918 persons aged 18 plus. For further technical details about the survey see http//users.ox.ac.uk/~oxis/sample.htm.
(3) Note, however, that the questions were administered on this survey using a five-point scale rather than a four-point one. This of course means that, everything else being equal, we would expect smaller net percentages in favour of agreement or disagreement.
References
Best, S. & Krueger, B. (2002) New approaches to assessing public opinion: the prospects for electronic mail surveys, International Journal of Public Opinion Research, 14, pp. 73-92.
Cook, C., Heath, F. & Thompson, R. (2000) A meta-analysis of response rates in web- or internet-based surveys, Educational and Psychological Measurement, 60, pp. 821-836.
Couper, M. (2000) Web surveys: a review of issues and approaches, Public Opinion Quarterly, 64, pp. 464-494.
Crewe, I. (2001) The opinion polls: still biased to Labour, in P. Nortris (ed.), Britain Votes 2001. Oxford: Oxford University Press.
Curtice, J. & Sparrow, N. (1997) How accurate are traditional quota opinion polls?, Journal of the Market Research Society, 39, pp. 433-448.
Evans, G. (2002) In search of tolerance, in A. Park, J. Curtice, K. Thomson, L. Jarvis & C. Bromley (eds), British Social Attitudes: the 19th report. London: Sage.
Fleming, G. & Sonner, M. (1999) Can internet polling work? Strategies for conducting public opinion surveys online, Paper presented at the annual meeting of the American Association for Public Opinion Research, St Petersburg Beach, May.
Gregory, J. (2002) A Report on the People's Panel for the Office of Public Service Reform by the Office for National Statistics. Office for National Statistics.
Himmelweit, H., Biberian, M. & Stockdale, J. (1978) Memory for past vote: implications of a study in recall, British Journal of Political Science, 8, pp. 365-384.
Kellner, P. (2003a) First among equals, Research, September, pp. 34-35.
Kellner, P. (2003b) Personal communication to Nick Sparrow.
Krosnick, J. & Chang, L. (2001) A comparison of the random digit dialling telephone survey methodology with internet survey methodology as implemented by Knowledge Networks and Harris Interactive. Unpublished paper, Ohio State University.
Mitofsky, W. (1999) Pollsters.com, Public Perspective, June/July, pp. 24-26.
Smith, T. (2003) An experimental comparison of Knowledge Networks and the GSS, International Journal of Public Opinion Research, 15, pp. 167-179.
Taylor, H. (2000) Does internet research work?, International Journal of Market Research, 42, pp. 51-63.
Terhanian, G. & Bremer, J. (2002) A multi-method approach for reducing error in internet-based surveys of non-probability samples, Paper presented at the annual conference of the American Political Science Association, Boston, MA, August.
Terhanian, G., Marcus, S., Bremer, J. & Smith, R. (2001) Reducing error associated with non-probability sampling through propensity scores: evidence from election 2000, Paper presented at Joint Statistical Meeting 2001, Atlanta, GA, August.
Vehovar, V. (2003) Web surveys: a meta analysis. Presentation at the Institute of Mathematics, Tokyo. Available at http://websm.org.
Vehovar, V. & Lozar Manfreda, K. (2002) Mode effect in web surveys, Paper presented to the American Association for Public Opinion Research, St Petersburg Beach, May.
Nick Sparrow
ICM Research
John Curtice
Strathclyde University
No comments:
Post a Comment