H. G. Muller wrote on Wed, Apr 1, 2015 07:40 AM UTC:
I had done little testing with the Rookies army, as it was added to Fairy-Max only recently. (Originally it could not do limited-range sliders, like the Rookies' R4.) In fact it was not even supplied with Rookies vs Nutters game definition, only with Rookies-FIDE and Rookies-Clobberers. I corrected that now in my own version by adding the required definitions, so that all 12 combinations of unequal armies can be selected.
I used that to do some more accurate Rookies measurements, by playing the Rookies against all other armies at 40 move/min with Fairy-Max 4.8U, both with black and with white. Each of the color/army combinations played 400 games. The table below summarizes the reults; +13% in the table means that the white army scored 13% better than equality, i.e. 63%, etc. Draw percentages were only 20-25%, i.e. significantly lower than the 32% seen in FIDE vs FIDE. For completeness I am redoing matches between the other armies as well. I am playing 4 matches simultaneously, on my 4-core (8 hyper threads) i7 CPU; this way each of the matches will use its own physical core, so that they won't affect each other. (Fairy-Max also uses only 1 core; using more cores in these kinds of tests is quite counter-productive anyway, as a lot of computational effort is wasted when multiple cores have to cooperate in finding the best move.)
black
w Rook Nutt Clob FIDE total
h Rookies #### +3% +13% +15% +62%
i Nutters -3% #### +3% +12% +19%
t Clobberers -13% +0% #### +8% -11%
e FIDE -16% -10% -10% #### -71%
(All results based on 400 games.)
A Pawn is in general worth 15-18% (I did not test that in these particular setups). So the strength differences seem to run upto about 1 Pawn. Indeed it seems that the Rookies are the strongest of all, in agreement with your ChessV and Zillions tests. I find them only marginally stronger than the Nutters, however, while the Clobberers seem to be aboult half-way in strength between them and FIDE. Statistical error in 400 games is 2%, so that means that the 3% of the Rookies-Nutters result can make use decide with >90% confidence that the Rookies indeed have the better chances here.
As to the explanation why FIDE is the weakest, I think you are correct: humans fail to get the most out of an unfamiliar army. This explanation is a bit suspect though (although I do not have any beter one): it is not like the FIDE side is playing normal Chess, and failing to grasp what the opponent can do should also put him to unpleasant surprises, as well as making his opponent miss opportunities. These computer self-tests are really highly insensitive to poor handling of the pieces of one side. If I intentionally mess up the play for one of the armies by putting in wrong piece values (e.g. give a piece that is stronger than a Rook a value lower than that of the Rook) it hardly changes things. The side handling the piece will now try to (inappropriately) trade it for a Rook, but the Rook side will now try to avoid that trading (equally inappropriately), so that in the end very little trading takes place, and both pieces will remain on the board to see the effect of which one is more effective in, say, gobbling up Pawns or delivering checkmates. As long as both sides share the same misconception, (and it is not too hilarious, like swapping Pawn and Queen value) the results are practically the same. It is not clear to me why this would be different for humans.
I used that to do some more accurate Rookies measurements, by playing the Rookies against all other armies at 40 move/min with Fairy-Max 4.8U, both with black and with white. Each of the color/army combinations played 400 games. The table below summarizes the reults; +13% in the table means that the white army scored 13% better than equality, i.e. 63%, etc. Draw percentages were only 20-25%, i.e. significantly lower than the 32% seen in FIDE vs FIDE. For completeness I am redoing matches between the other armies as well. I am playing 4 matches simultaneously, on my 4-core (8 hyper threads) i7 CPU; this way each of the matches will use its own physical core, so that they won't affect each other. (Fairy-Max also uses only 1 core; using more cores in these kinds of tests is quite counter-productive anyway, as a lot of computational effort is wasted when multiple cores have to cooperate in finding the best move.)
black w Rook Nutt Clob FIDE total h Rookies #### +3% +13% +15% +62% i Nutters -3% #### +3% +12% +19% t Clobberers -13% +0% #### +8% -11% e FIDE -16% -10% -10% #### -71%(All results based on 400 games.)A Pawn is in general worth 15-18% (I did not test that in these particular setups). So the strength differences seem to run upto about 1 Pawn. Indeed it seems that the Rookies are the strongest of all, in agreement with your ChessV and Zillions tests. I find them only marginally stronger than the Nutters, however, while the Clobberers seem to be aboult half-way in strength between them and FIDE. Statistical error in 400 games is 2%, so that means that the 3% of the Rookies-Nutters result can make use decide with >90% confidence that the Rookies indeed have the better chances here.
As to the explanation why FIDE is the weakest, I think you are correct: humans fail to get the most out of an unfamiliar army. This explanation is a bit suspect though (although I do not have any beter one): it is not like the FIDE side is playing normal Chess, and failing to grasp what the opponent can do should also put him to unpleasant surprises, as well as making his opponent miss opportunities. These computer self-tests are really highly insensitive to poor handling of the pieces of one side. If I intentionally mess up the play for one of the armies by putting in wrong piece values (e.g. give a piece that is stronger than a Rook a value lower than that of the Rook) it hardly changes things. The side handling the piece will now try to (inappropriately) trade it for a Rook, but the Rook side will now try to avoid that trading (equally inappropriately), so that in the end very little trading takes place, and both pieces will remain on the board to see the effect of which one is more effective in, say, gobbling up Pawns or delivering checkmates. As long as both sides share the same misconception, (and it is not too hilarious, like swapping Pawn and Queen value) the results are practically the same. It is not clear to me why this would be different for humans.