Skip Navigation
28 comments

  • jwz gave the game away, so i'll reveal:

    the One Weird Trick for this week is that the bots pretend to be an old version of Chrome. So you can block on useragent

    so I blocked old Chrome from hitting the expensive mediawiki call on rationalwiki and took our load average from 35 (unusable) to 0.8 (schweeet)

    caution! this also blocks the archive sites, which pretend to be old chrome. I refined it to only block the expensive query on mediawiki, vary as appropriate.

    nginx code:

     
        
        # block some bot UAs for complex requests
        # nginx doesn't do nested if, so we set a test variable
        # if $BOT is both Complex and Old, block as bot
        set $BOT "";
        if ($uri ~* (/w/index.php)) {
            set $BOT "C"; }

            if ($http_user_agent ~* (Chrome/[2-9])) {
                set $BOT "${BOT}O";}
            if ($http_user_agent ~* (Chrome/1[012])) {
                set $BOT "${BOT}O";}
            if ($http_user_agent ~* (Firefox/3)) {
                set $BOT "${BOT}O";}
            if ($http_user_agent ~* (MSIE)) {
                set $BOT "${BOT}O";}

            if ($BOT = "CO") {
                return 503;}

    you always return "503" not "403", because 403 says "fuck off" but the scrapers are used to seeing 503 from servers they've flattened.

    I give this trick at least another week.

28 comments