Question
The title says it all. How does the Enable Challenge Mode setting affect match finding or multiplayer game play?
Answer
I've since found an article (link) with an explanation of the 'Enable Challenge Mode' check box. Here is the relevant content:
... A normal Ranked 1v1 game has a 1X multiplier, meaning it's worth 1 game when calculating your final rating. ... In "Challenge Mode," you're increasing the multiplier so the result will count the same as if you played multiple matches against the same opponent. ... A "Challenge" is issued from a button in the scoreboard, which is accessed by pressing F2 during the match. ... At this point, that [challenged] player's options are to either "Accept" the newly raised stakes or "Forfeit" the match with the previously agreed-to multiplier.
Check more discussion of this question.