Hey, catching up on AGI Friday after a little while away. I just wanted to respond to this point:
> Now there’s no such thing as humans getting anywhere near AI at any board-game or card-game style game.
In fact, the state of the art of AI in the card game bridge is hot garbage. Robots are truly bad at the game. Bridge has three main aspects of play: bidding, defense, and declaring. The robots are bad at all three, but worst at bidding.
Ooh, yes, my robot friends tell me you are correct. (Also my human friend, Alex Strehl, is a serious bridge [oops, I meant backgammon] player and confirms something like this [false, there are problems with AI backgammon, I guess, but losing to humans doesn't seem to be one of them].)
Presumably Bridge would fall to a concerted effort from a frontier AI lab, but in the meantime it's another great leading indicator for AGI.
Do you have any predictions on how far we are from a human-level AI Bridge player?
In 2022, Nukkai had its AI bridge player compete against 8 human world champions in a restricted contest (only playing in 3NT contracts reached by an uncontested auction 1NT-2NT-3NT, against robot defenders). Their player outplayed the human champions under these conditions of contest (https://challenge.nukk.ai/). However, this result came from essentially learning how to exploit the robot defenders, not from particularly strong play (analysis by another human champion at https://bridgewinners.com/article/view/conclusions/).
There are a number of challenges. First, a brief description of the game:
Each player (of four) is dealt 13 cards from a standard deck. The players are given the designations North, South, East, and West. North and South are partners, as are East and West.
In the auction period, each player in turn makes a call of Pass, Double, Redouble, or $X $STRAIN, where $X is a number between 1 and 7 and $STRAIN is one of clubs, diamonds, hearts, spades, or no trump (in increasing order of priority). Pass is always a legal call; double is legal only if the last non-pass call was $X $STRAIN by an opponent; redouble is legal only if the last non-pass call was Double by an opponent, and $X $STRAIN is legal only if the last call of this form had a smaller $X, or if it had the same $X and $STRAIN was earlier in the priority list.
The auction period comes to an end if all players pass at their first turn, or after three consecutive passes after a non-pass call. If all four players passed initially, the hand is finished with no score awarded. Otherwise, the last bid of the form $X $STRAIN is the final contract, with the partnership who made that bid as the declaring side. The declarer is the member of the declaring side who first mentioned the strain. That side must take $X + 6 tricks in order to satisfy their contract.
Play starts when the player to the left chooses a card to lead to the opening trick. At that point, the partner of the declarer, called dummy, puts all their cards face up. The declarer chooses which cards to play from both hands. Otherwise, the play is a standard trick taking game: when it is your turn to play, you must follow suit if you can, highest card in the led suit wins unless a trump has been played, in which case the highest trump wins. The hand that won the previous trick leads to the next trick.
So let's talk about some of the challenges. The main challenges all relate to the bidding. Even in the playing phase, competent human players must incorporate information gleaned during the auction phase. Different pairs have different bidding agreements, but they must make their agreements public and truthfully answer questions about those agreements when asked during the play. Even standard bidding methods taught to beginners are complicated; because of the many sequences (10^47 possible sequences, though they are not nearly equally likely), it is not possible to assign a meaning to every sequence. However, there are rules of thumb, etc, that human players use to reason about what a sequence must mean. This seems like a problem that LLMs would be well-suited to attack, but this has not yet been done successfully. It would also be difficult to integrate this with the sorts of tools that are well-suited to playing problems. Note that there are also card play agreements. For instance, some pairs, when making an opening lead from a long suit, lead the 4th highest card, while others lead the third highest from an even number and the lowest from an odd number. You might also signal to partner when following suit with a low card. You might give attitude (when you lead a high card in a suit, I'll play my lowest card to indicate that I also have a high card in the suit, or a high spot card like a 6 or 8 to indicate that I don't have a high card like a Queen), or count (if I started with an even number of cards in the suit, I'll play my lowest spot card), or suit preference (you've led a diamond when spades are trump; I'll follow with my smallest diamond to indicate that I prefer clubs to hearts, or a higher spot card to indicate that I prefer hearts to clubs).
Another issue is training data. To my knowledge there are very few hands that were (a) played by human experts (b) with the entire auction recorded (c) with each card played recorded and crucially (d) annotated with the players' agreements about the bidding and card play. So this will generally require either self-play or play against other robot players. This can lead (as in the Nukkai Challenge) to learning how to exploit the robot players rather than learning how to play well. It's also a real problem when trying to learn bidding. As mentioned above, players must be able to explain their methods to their opponents. They also must be able to deal with novel bidding agreements deployed against them by their opponents.
If forced to make a prediction about this, my 5-year prediction is less than 25% chance of a robot bridge team winning an event like the http://en.wikipedia.org/wiki/Bermuda_Bowl in which large numbers of hands are played against top human players.
PS: I was confusing Bridge and Backgammon in my previous comment. And also probably misremembering what my *backgammon*-playing friend said about the state of AI backgammon, which does seem to be superhuman.
Yes, backgammon was an early success in AI gameplay. TD-Gammon was changing expert opinion on certain aspects of gameplay in the early 90s https://en.wikipedia.org/wiki/TD-Gammon
Hey, catching up on AGI Friday after a little while away. I just wanted to respond to this point:
> Now there’s no such thing as humans getting anywhere near AI at any board-game or card-game style game.
In fact, the state of the art of AI in the card game bridge is hot garbage. Robots are truly bad at the game. Bridge has three main aspects of play: bidding, defense, and declaring. The robots are bad at all three, but worst at bidding.
Ooh, yes, my robot friends tell me you are correct. (Also my human friend, Alex Strehl, is a serious bridge [oops, I meant backgammon] player and confirms something like this [false, there are problems with AI backgammon, I guess, but losing to humans doesn't seem to be one of them].)
Presumably Bridge would fall to a concerted effort from a frontier AI lab, but in the meantime it's another great leading indicator for AGI.
Do you have any predictions on how far we are from a human-level AI Bridge player?
It's an interesting question!
In 2022, Nukkai had its AI bridge player compete against 8 human world champions in a restricted contest (only playing in 3NT contracts reached by an uncontested auction 1NT-2NT-3NT, against robot defenders). Their player outplayed the human champions under these conditions of contest (https://challenge.nukk.ai/). However, this result came from essentially learning how to exploit the robot defenders, not from particularly strong play (analysis by another human champion at https://bridgewinners.com/article/view/conclusions/).
There are a number of challenges. First, a brief description of the game:
Each player (of four) is dealt 13 cards from a standard deck. The players are given the designations North, South, East, and West. North and South are partners, as are East and West.
In the auction period, each player in turn makes a call of Pass, Double, Redouble, or $X $STRAIN, where $X is a number between 1 and 7 and $STRAIN is one of clubs, diamonds, hearts, spades, or no trump (in increasing order of priority). Pass is always a legal call; double is legal only if the last non-pass call was $X $STRAIN by an opponent; redouble is legal only if the last non-pass call was Double by an opponent, and $X $STRAIN is legal only if the last call of this form had a smaller $X, or if it had the same $X and $STRAIN was earlier in the priority list.
The auction period comes to an end if all players pass at their first turn, or after three consecutive passes after a non-pass call. If all four players passed initially, the hand is finished with no score awarded. Otherwise, the last bid of the form $X $STRAIN is the final contract, with the partnership who made that bid as the declaring side. The declarer is the member of the declaring side who first mentioned the strain. That side must take $X + 6 tricks in order to satisfy their contract.
Play starts when the player to the left chooses a card to lead to the opening trick. At that point, the partner of the declarer, called dummy, puts all their cards face up. The declarer chooses which cards to play from both hands. Otherwise, the play is a standard trick taking game: when it is your turn to play, you must follow suit if you can, highest card in the led suit wins unless a trump has been played, in which case the highest trump wins. The hand that won the previous trick leads to the next trick.
So let's talk about some of the challenges. The main challenges all relate to the bidding. Even in the playing phase, competent human players must incorporate information gleaned during the auction phase. Different pairs have different bidding agreements, but they must make their agreements public and truthfully answer questions about those agreements when asked during the play. Even standard bidding methods taught to beginners are complicated; because of the many sequences (10^47 possible sequences, though they are not nearly equally likely), it is not possible to assign a meaning to every sequence. However, there are rules of thumb, etc, that human players use to reason about what a sequence must mean. This seems like a problem that LLMs would be well-suited to attack, but this has not yet been done successfully. It would also be difficult to integrate this with the sorts of tools that are well-suited to playing problems. Note that there are also card play agreements. For instance, some pairs, when making an opening lead from a long suit, lead the 4th highest card, while others lead the third highest from an even number and the lowest from an odd number. You might also signal to partner when following suit with a low card. You might give attitude (when you lead a high card in a suit, I'll play my lowest card to indicate that I also have a high card in the suit, or a high spot card like a 6 or 8 to indicate that I don't have a high card like a Queen), or count (if I started with an even number of cards in the suit, I'll play my lowest spot card), or suit preference (you've led a diamond when spades are trump; I'll follow with my smallest diamond to indicate that I prefer clubs to hearts, or a higher spot card to indicate that I prefer hearts to clubs).
Another issue is training data. To my knowledge there are very few hands that were (a) played by human experts (b) with the entire auction recorded (c) with each card played recorded and crucially (d) annotated with the players' agreements about the bidding and card play. So this will generally require either self-play or play against other robot players. This can lead (as in the Nukkai Challenge) to learning how to exploit the robot players rather than learning how to play well. It's also a real problem when trying to learn bidding. As mentioned above, players must be able to explain their methods to their opponents. They also must be able to deal with novel bidding agreements deployed against them by their opponents.
If forced to make a prediction about this, my 5-year prediction is less than 25% chance of a robot bridge team winning an event like the http://en.wikipedia.org/wiki/Bermuda_Bowl in which large numbers of hands are played against top human players.
PS: I was confusing Bridge and Backgammon in my previous comment. And also probably misremembering what my *backgammon*-playing friend said about the state of AI backgammon, which does seem to be superhuman.
Yes, backgammon was an early success in AI gameplay. TD-Gammon was changing expert opinion on certain aspects of gameplay in the early 90s https://en.wikipedia.org/wiki/TD-Gammon