David Pritchard's criteria used in the 41-square contest are interesting:
playability, originality, simplicity and elegance. We also generally used
these criteria in judging the 42-square contest, with an added touch of
subjectivity--which should not be ignored. See:
http://www.chessvariants.com/41.dir/report.html