The abstract of Professor Jack H. Nagel's paper The Burr Dilemma in Approval Voting, Journal of Politics 69,1 (February 2007) 43-58] said
Quote from Nagel's abstract: It has not previously been recognized that the first four presidential elections (1788-1800) were conducted using a variant of approval voting. That experiment ended disastrously in 1800 with the infamous Electoral College tie between Jefferson and Burr. [After that, the 12th amendment then changed the presidential election system.]
But the early USA was not precisely using "approval voting" because there was a 2-vote limit (genuine approval voting has no limit on the number of approved candidates). This 2-vote limit would have made almost no difference if almost all electors would not have wanted to approve more than 2 candidates anyway. In such cases it was effectively the same thing as approval voting. That was probably true in 1788, 1792, and 1800 – i.e. these three elections were effectively approval voting – but is more dubious in 1796 because in that one there were 13 candidates so quite possibly a decent number of electors would have wanted to approve more than two if permitted.
Also, in another (unwise) twist, the rules said the 2nd place finisher became Vice President as a "consolation prize," which may have slightly altered some voters' calculations. (Note: this was not the same thing as some fairly common elections we have now where voters can vote for N candidates and the top N win the N available equal seats, because the two seats here were highly unequal.)
Anyway – what did happen in the first four (1788-1800) US presidential elections?
1788-9: Washington first with 69, Adams second with 34; 12 candidates but all the others got ≤9 and plainly had no chance.
1792: 5 candidates, Washington wins with 132, Adams 2nd with 77, Clinton 3rd with 50, rest ≤4 each [popular vote used in only 1 state – VA]
1796: 13 candidates, Adams wins with 71, Jefferson 2nd with 68, Pinckney 3rd with 59, Burr 4th with 30, ... 6th was Ellsworth with 11, all others 7 or below . Adams & Jefferson hated each other but became President & VP. The fact that the "runner-up becomes VP" system permitted P & VP to be two people who hated one another, was a major reason this system was discarded. But we do not feel that approval voting was a mistake. We feel the 2nd-place=VP rule was the mistake. The right fix would have been to use approval voting to vote for pre-agreed running-mate teams.
The worse fix that they actually adopted with the 12th amendment, was to have the electors use plurality voting to vote for the president, and a separate plurality election to vote for the vice president (with the additional twist that electors are sometimes not permitted to vote for somebody if it was a co-habitant of his own state!?!). Actually it is not "plurality" it is "majority" in the sense that if a majority does not happen then the congress steps in to "break the tie" (which actually need not be a "tie," merely a non-majority win). The election then is first sent to the House, which must choose one of the top three. That voting is complex, it is by state, not by representative, each state gets one vote (at most), and the consent of an absolute majority of states is required to complete. This incidentally has the disadvantage that it might never happen. The Vice-Presidential election requires an absolute majority of electors similarly, and if it fails, it goes to the Senate. An absolute majority of the whole number of senators is, again, a requirement.
Note that all this still permits the P & VP to hate one another, although it became less likely. The constitution never said (including all amendments up to the present day) anything about either "political parties" or "running mates" (acted as though these two concepts did not exist) although as time went by political parties did develop, eventually dominating the scene, and eventually they always ran two "running mates" for P & VP, although occasionally running different 2-mate pairs in different states or in different parties in cases where one man was running for president with the backing of more than one party. [That might create difficulties, but so far has not. For example in 1896 W.J.Bryan ran with running mate Arthur Sewell (D) and also with running mate Thomas E. Watson (P). If Bryan had won the presidency instead of coming in 2nd behind McKinley, then it would not have been obvious whom the vice president would then be – Sewell, Watson, or Hobart? – and indeed quite probably then would have been decided by congress not the electoral college.] Also we ended up with two kinds of voting – popular vote within states for president and VP, versus the Electoral Collegians voting for president and VP. The latter was originally far more important but now is almost of no importance. All this happened because of tradition and state laws, not the US constitution, which actually by itself in the 12th amendment is quite atrocious on this topic; if the 12th amendment, and it alone, were the law of the land that would be a disaster.
1800: 5 candidates, Jefferson wins with 73 after tiebreak with Burr (also 73). Burr actually encouraged the congress in tiebreak to vote Jefferson saying he did not want the presidency, although apparently if he had wanted it, he probably would have had more support than Jefferson and would have won! This election also featured (which was an outrage) Jefferson as the one counting congress's votes and thus electing himself!