Large experimental studies comparing 4 different voting systems in France 2007

The French Presidential election of 2007 was conducted using plurality voting (21-22 April) with a subsequent top-2-runoff round (5-6 May). There were 12 candidates on ballot. Meanwhile, three different scientific studies of alternative voting systems were underway, with real voters at official polling places in France – these studies were all conducted by academics with government funding and cooperation.

The result is an unprecedently high-quality, head-to-head, fair comparison of these different voting systems in action in real life.

#ranked #ballots
    1      30
    2      67
    3     163
    4      95
    5      78
    6      37
    7      17
    8       9
    9       3
   10       9
   11      15
all 12    370
total unspoiled  893 
mean #ranked 7.3

#approved #ballots  (# in 2002)
   0       120      36
   1       736     287
   2       905     569
   3       673     783
   4       264     492
   5        75     258
   6        23      94
   7        13      40
   8         1      16
   9         1       6
  10         1       1
 >10         1       5
total      2813    2587
mean#approved 2.33*  3.15

Who was the "right" winner?

All methods studied agreed that the winner should be either Sarkozy or Bayrou. Which? I want to break my usual policy and actually take a stance on that: Bayrou was better for France. I am not saying this because of any love or dislike of the policies or abilities of Bayrou or Sarkozy. I am saying it because:

1. Poll data we shall review on this page shows beyond doubt (99.99999999+% confidence) that Bayrou would have defeated Sarkozy in a head-to-head simple majority vote in April 2007.
2. Also, even if you may not agree with simple majority vote because voters care unequally... Bayrou also beats Sarkozy using both {0,1,2} and {0,1,2,3,4,5} range voting.
3. With the benefit of hindsight France later came to the conclusion even more strongly that Bayrou was better. Specifically as the next (2012) election and the end of Sarkozy's term both approached, pairwise polls showed Bayrou's advantage over Sarkozy had increased to a landslide 56:35 margin! (Despite the fact that Sarkozy then came 2nd in the official vote, and Bayrou 5th, due to the flawed and massively distortionary plurality+top-2 runoff voting system used by France!)

Notes on the table

The rank-order-ballot voters were told the voting system would be instant runoff voting. Hence, attempts to use those ballots to examine other rank-order voting systems such as Coombs elimination are (somewhat) less justified. The score-ballot voters were told the voting system would be median-based (Balinski & Laraki's "majority judgment") with no-score counting as zero, and employed a verbal not numerical value-scale. Hence, attempts to examine other score-based voting systems such as average-based range voting with no-score ignored, are (somewhat) less justified. However, median-based, average-based with no-score=zero (and scores 5,4,3,2,1,0), and average-based with no-score ignored, all returned the same finish order of their top 8 candidates. (But see below re geographic bias correction.) There were no Condorcet cycles hence there was an unambiguous finish order deducible from the rank-order ballots

with each finisher successively preferred over the next, pairwise, by a voter majority. An independent professional France-wide poll by Le Figaro Magazine on 14-15 March 2007 (telephone, 1000 sampled, age≥18) assessing the results of three possible head-to-head runoff rounds ("duel"s) found:

confirming the top three in this order. (The additive standard error should be less than ±1.58%, with a 54:46 duel yielding confidence of victory >99%, and a 60:40 duel yielding >99.999999%. At least 32 other such "Bayrou-Sarkozy duel" polls were conducted between 1 March and 21 April 2007 by many pollsters including BVA, Harris, TNS-Sofres, Ifop, and Ipsos. All 32 agreed Bayrou would defeat Sarkozy head-to-head, with Bayrou's vote share ranging between 51% and 60%. An Ifop 20 April poll, also age≥18 by telephone of approximately 1000 pollees, also considered all three duels, and again agreed Bayrou would defeat Royal by 58-42% and Sarkozy by 55-45%. The combination of all of these polls yields extremely high confidence, exceeding 99.99999999%, that Bayrou was the genuine Condorcet winner.) The Figaro poll also found the "second choices" of supporters of Bayrou, Royal, Sarkozy, and J.Le Pen, finding the interesting result that 1-5% of voters intended in round 2 to vote against their choice in the first round. The largest such liar-type Figaro found was Sarkozy voters, 5% of whom said they intended to vote Bayrou in round 2 (if it were a Sarkozy-vs-Bayrou runoff). Similarly 3% of Royal voters said they intended to vote Bayrou if available in the second round – but this was compensated by 3% of Bayrou voters who said they intended to vote Royal in the second round!

That Condorcet order was quite similar (although not exactly the same) as

but quite different from

The fact that Coombs and pseudo-Borda did not return ridiculous results is because these French rank-order voters evidently did not strategically/dishonestly demote frontrunners to last or nearly-last place. (At least, the fraction who did, was small.) But that kind of "strategic burial" behavior is very common (80-95%) in Australian rank-order voting, which would have been enough to make both Coombs and Borda yield completely ridiculous results. This difference between Australian and French voters is presumably due to either

1. The fact full ranking is compulsory in most of Australia,
2. The prevalence of party "how to vote" cards offering rank-order-advice from the Australian parties,
3. The greater experience of the Australians with rank-order voting.

It would have been ridiculous to try to use the Borda voting system because only 370 (38.5%) of the 960 rank-order ballots ranked all 12 candidates – Borda would have had an insane ≈60% spoilage rate. However, one can still deduce "pseudo-Borda" scores for each candidate by using the theorem that a candidate's Borda score is the sum of her pairwise vote-totals. If we employ the pairwise-table below and assume that the voters who did not express "A</>B" opinions would have (if forced) expressed those opinions in the same ratios as the voters who did express an opinion about A vs. B, then we get the pseudo-Borda score for each candidate by summing his row-numbers.

Total approval was 232.5%, i.e. the average ballot approved 2.33 out of the 12 candidates. This figure comes from table 15.15 page 382 of Baujard & Igersheim's Handbook paper, and is based on an "extrapolation to all France" of their data correcting for sampling and participation biases. In 2007 the most common number to approve was 2 (done by 32.2% of all ballots), but in the preceding 2002 election it was 3 (by 30.3%). Note that even the approval-winner (Bayrou) got below 50% approval. (And in the preceding 2002 election, Chirac got the most approval, with 36.7%, which was even further below 50%.) In the ranked-ballot election, 41% of valid ballots ranked all 12 candidates (the most common type) as the instructions encouraged them to do; and the second most common type were ballots ranking 3 candidates (18.2% of valid ballots); on average ballots ranked 7.3 candidates.

Independent France-wide professional approval-style polls were conducted by IPSOS, which indeed maintains a database of favorable:unfavorable ratings for French political figures across time. Meanwhile another pollster (BVA) on 22 March 2007 asked "would each of the following candidates be a good President of France?"

            IPSOS ratings April 2007
     Candidate   fav:unfav:don't know
     Bayrou      58:35:7 %
     Sarkozy     53:43:4
     Royal       48:46:6
     Besancenot  43:44:13
     Buffet      34:49:17 
     de Villiers 29:60:11
     J. Le Pen   22:74:4

         BVA 22 March
     Candidate   yes:no (%)
     Bayrou      60:36
     Sarkozy     59:38
     Royal       49:48
     J. Le Pen   12:84

These confirm (and remove all doubt) that Bayrou was the approval winner, with Sarkozy and Royal in 2nd and 3rd place respectively.

Baujard & Igersheim also asked their voters various questions about voting systems, and about 40% of them answered those questions. Of those respondents, 83.5% said "yes" to "does the principle of approval voting seem clear to you?" while 89.2% said yes for same question about range(0,1,2) voting. One particularly interesting question was #11:

In which official elections do you think the [0,1,2 rating] method could be used?

In all, among the voters who responded to the questionnaire, 83.52% considered range voting applicable to at least one election type, versus 72.43% for approval voting.

This is among the first clear evidence that score voting [more precisely, range(0,1,2)] actually is both more popular and comprehensible than approval voting!! It also is less-clear evidence that score and approval voting are "adoptible" and apparently would win by a massive margin if given as an option on a referendum. These conclusions unfortunately are attackable because only 40% of the pollees answered these optional questions and hence there may have been self-selection bias.

In contrast, an October 2010 nationwide professional telephone poll of 1202 random Australian adults found that they prefer plain-plurality voting versus the preferential (instant runoff) system they presently use to elect their House. If forced to choose one, they'd choose to abandon IRV – the poll's result was 57% to 37% (with 5% don't know/refuse). Also Britain conducted a binding referendum on 5 May 2011 asking voters to decide whether the UK should switch from plurality to IRV voting, resulting in a massive landslide victory (68% to 32% of the 19.3 million votes) for "stay with plurality." Another big referendum defeat for IRV was on 12 May 2009 in British Columbia, Canada, where "switch to IRV" got only 39.09% of the 1.65 million votes.

Britain: Turnout=42%; the total score was IRV=6152607(32.1%), plurality=13013123(67.9%), spoiled ballots=113292(0.59%).
British Columbia: Turnout=55%; the total score was IRV=623420(39.1%), plurality=971350(60.9%), spoiled ballots=56369(3.4%).
Australian poll question 9 wording: Currently, elections for the Federal House of Representatives, or lower house, use a preferential voting system. This is where voters indicate an order of preferences for all candidates, and these preferences are taken into account when deciding which candidate wins. (PAUSE). An alternative system would be "first past the post", where voters only vote for one candidate and the candidate with the most votes wins. Would you personally prefer...? READ OUT [PROG NOTE: - SINGLE RESPONSE - RANDOMISE 1-2, THEN 3 LAST]

1. A preferential system
2. A first past the post system
3. (DO NOT READ) Neither \ don't know

There also were France-wide pre-election polls by a number of polling agencies agreeing Bayrou would have defeated every rival in a head to head "Bayrou versus X" election (some already discussed).

Pairwise polls by Ifop: On 28 March and 19 April, the professional polling agency Ifop conducted France-wide polls about possible X-versus-Y pairwise runoffs. Ifop's results confirm Farvaque et al's.

Pairwise table from Balinski & Laraki's Majority-Judgment study:

IRV paradoxes that occurred in this election:

1. Royal was a "spoiler." That is, with IRV as the voting method the winner was Sarkozy, but if Royal had dropped out of the race, then the IRV winner would have been Bayrou.
2. Was this election "nonmonotonic"? Almost certainly. According to Farvaque, Jayet & Ragot's table 3, the final three candidates in IRV were Bayrou, Royal, and Sarkozy. Bayrou was eliminated with 27.4% of the (still active) ballots versus Royal's 32% and Sarkozy's 40.5%. Then, in the final IRV round, Sarkozy beat Royal 54.15% to 45.85%. Note that Royal got a 13.85% boost (from 32% to 45.85%) due to vote-transfers from Bayrou, whereas Sarkozy received a slightly smaller transfer of 13.65%. This small advantage in transfers was not enough for Royal to win.
       Assume ≥4.61% of the total number of voters ranked the "big three" in the order Royal>Sarkozy>Bayrou. (Because 4.61 is a small percentage, this assumption seems extremely likely to be true; but I cannot be certain of it without access to Farvaque, Jayet & Ragot's raw data.) Now consider the effect of such a 4.61% switching their vote from Royal to Sarkozy. In that case, Royal would have been eliminated. The final round would then have been Bayrou versus Sarkozy. According to the head-to-head pairwise data table, Bayrou would have beaten Sarkozy in this final round 52.0-48.0 (and those promotions of Sarkozy to top in those 4.61% would not have changed this at all). That's non-monotonicity: if about 1.8 million "Royal>Sarkozy>Bayrou" voters had switched their vote from Royal to the winner Sarkozy, that would have made Sarkozy lose!
3. Another paradox IRV suffered in this election was a no-show paradox. Suppose that 1.8 million Royal>Bayrou>Sarkozy voters – who, we reiterate, ranked Sarkozy dead last – had simply "not shown up" i.e. had chosen not to thus-vote against Sarkozy. That would have made Bayrou win. We repeat: the act of voting Sarkozy dead last, made him win, entirely against the expressed interests of these voters.
       Well, not quite. Actually, this paradox does not quite work as stated, but it does work in combination with the above non-monotonicity paradox, e.g. erase 1.4 million Royal>Bayrou>Sarkozy voters ("no show") plus make 0.4 million Royal>Sarkozy>Bayrou voters switch to Sarkozy>Royal>Bayrou ("non-monotonicity") – that combination paradoxically causes Sarkozy to lose.
   

   Again, this conclusion is attackable since our argument only works if ≥1.4 million Royal>Bayrou>Sarkozy voters actually existed. They probably did, but I reiterate am not 100% sure they did since I do not have access to Farvaque, Jayet & Ragot's raw data (which they did not provide when I asked for it).

In this IRV election, approximately 4% of the IRV voters with unspoiled ballots failed to affect the final Sarkozy vs. Royal round (because, e.g, they had failed to rank either) – plus 7.0% of the ballots were spoiled, so in all about 11% of the IRV voters failed to have any impact on the final Sarkozy vs Royal round. That considerably exceeds the Sarkozy-vs-Royal victory margin. In the official election (2-round plurality voting) however, the spoilage rates were well below the S-vs-R victory margin. This means that the official 2-round noninstant runoff, in this case, had a greater claim to legitimacy than an instant runoff would have had (even though they both delivered the same winner, Sarkozy).

Strategic voting: A TNS-Sofres-Unilog Groupe Logica CMG poll conducted on election day (22 April 2007) asked voters what most determined their votes. One of the seven possible answers was a deliberate strategic vote for somebody who was not their favorite ("le vote utile"): this answer was given by 22% of those (who said they voted) for Bayrou, 10% of those for Le Pen, 31% of those for Royal, and 25% of those for Sarkozy.

Correction for geographic bias in Balinski & Laraki's "Majority Judgment" median-based score-voting method: although their full 1733-voter sample elected Bayrou, B&L believed their sample was geographically biased since region they polled (Orsay) preferred Royal and Bayrou more, and Sarkozy less, than the rest of France:

                               Official vote-percentages
                   Sark Royl Bay LePen Bes Vil Voy Bov Buf Lag Nih Sch
        Orsay12    32.0 26.6 20.2 10.0 2.7 2.5 2.3 1.3 1.2 0.8 0.2 0.0 
        Nationwide 31.2 25.9 18.6 10.4 4.1 2.2 1.9 1.6 1.3 1.3 1.2 0.3

                               Official vote-percentages
                   Sark Royl Bay LePen Bes Vil Voy Bov Buf Lag Nih Sch
        The 501    30.7 25.9 18.7  9.3 2.5 (difference less than 0.6)
        Nationwide 31.2 25.9 18.6 10.4 4.1 2.2 1.9 1.6 1.3 1.3 1.2 0.3

They then extracted a uniform-random subsample of 201 ballots from their 501-ballot France-representative master set, and recorded who won within those 201 ballots according to each voting method. Then they repeated that experiment 10000 times in all. The result was this very interesting table of win-counts (this is from page 50 of Dolez et al):

     VOTE METHOD..................Royal......Bayrou.....Sarkozy.....Tie....Sum
     Plain Plurality...............977..........0.........9022........5....10004
     2-round(official) system.....1146.........98.........8197......559....10000
     Approval(≥"very good")........467........658.........7947......928....10000
     Majority Judgment.............606.......4326.........5065........3....10000
     Condorcet*....................142.......8329..........974......441.....9986
     Approval(≥"good")..............23.......9465...........40......472....10010
     Point Summing(i.e. Range).....139.......9463..........239......159....10000
     Borda Count....................12.......9976............0.......12....10000

*Also there were 114 Condorcet cycles, which B&L regard as a separate category from "tie."

Note the win-counts in each row of the table should sum to 10000, except for the "Condorcet" row which sums to 9986=10000-114. Unfortunately, this sum=10000 condition is (slightly) violated in two cases, indicating B&L's computer program had minor "bugs"! (Also, it would have been better to use subsample size≥251, not 201.) Note that even within B&L's corrected-to-unbias 501-sample, there still remains slight pro-Bayrou and anti-Sarkozy bias. Nevertheless in this experiment B&L found Sarkozy had a greater win-probability (using MJ) than Bayrou. So, MJ France-wide should really be regarded as "electing either Sarkozy or Bayrou, with greater chance that it was Sarkozy" or perhaps even just "Sarkozy, full stop" in this election. That "judgment" agrees with the official result but differs from approval, average-based range voting, Condorcet, and Borda.

Conclusions

These studies are very exciting. They look like they show the following:

1. Score voting is adoptable and will win if put to a referendum.
2. Score{0,1,2} voting is more clear and more popular than both approval and instant runoff voting. (More evidence re conclusions 1 & 2.)
3. The studies said approval, score{0,1,2}, median-based score{0,1,2,3,4,5} (pre-correction), average-based score{0,1,2,3,4,5}, Coombs, pseudo-Borda, and Condorcet – and pre-election head-to-head polls also indicated – that Bayrou was the candidate France wanted, and would have defeated every rival head-to-head. But the instant-runoff, non-instant top-2-runoff, and plain plurality election systems (and probably post-correction median score) would have elected (and did elect) Sarkozy. This was a failure of democracy which France in retrospect (based on later polls and elections) appears to have regretted.
4. Rank-order ballots exhibit the largest (7.0%) spoilage rates (which would be an enormous 60% if used for Borda). Approval-style ballots exhibit the smallest (0.8%) spoilage rates. Score voting ballots have 1.1% spoilage rate, while the official spoilage rates using plurality-style ballots were 1.44 and 4.20%.
5. Balinski & Laraki's "Majority Judgment" median-based score-voting method clearly worked less well in this election than the simpler "greatest average wins" score voting method (because MJ probably elected Sarkozy after correction for geographic biases, as B&L themselves showed; while average elected Bayrou).

Journal/Book references

We thank Lucas Nussbaum for pointing out the Le Figaro poll showing Bayrou was the Condorcet winner: Presidentielle 2007: Barometre presidentiel – vague 15, 18 March 2007.

Antoinette Baujard & Herrade Igersheim: Framed-field experiment on Approval voting and Evaluation voting. Some teachings to reform the French Presidential Electoral System, in B. Grofman, A. Laurent and B. Dolez (eds), In situ and Laboratory Experiments on Electoral Law Reform: French Presidential Elections, Heidelberg: Springer, 2010.

Antoinette Baujard & Herrade Igersheim: Framed field experiments on approval voting, lessons from the 2002 and 2007 French presidential elections, ch.15 (pp.357-395) in Handbook on Approval Voting (ed. J.Laslier & M.Remzi Sanver) Springer 2010.

Michel Balinski & Rida Laraki: A theory of measuring, electing, and ranking, Proc. Nat'l Acad. Sci. USA 104,21 (22 May 2007) 8720-8725.

Etienne Farvaque, Hubert Jayet, Lionel Ragot: Quel mode de scrutin pour quel vainqueur? Une experience sur le vote preferentiel transferable, Revue d'Economie Politique 119,2 (March-April 2009) 221-246.

Bernard Dolez, Bernard Grofman, Annie Laurent: Studies In Public Choice: In Situ and Laboratory Experiments On Electoral Law Reform: French Presidential Elections, Springer 2011 (Google eBook excerpts) contains a paper by M.Balinski & R.Laraki about MJ voting on pages 13-54.

Etienne Farvaque, Hubert Jayet, Lionel Ragot: A 'winner' under any voting rule? An experiment on the single transferable vote, English-language pdf, Sorbonne "Documents de travail du Centre d'Economie" #09067.

Messed-up Elections thru history

Range & other voting systems: elections over last ≈50 years

France 2000+

Return to main page