In the context of web design, it refers to design methods which discourage listing of sites or content in search engines by inhibiting "spiders" used by search sites.
See Steven Pemberton and original linux-elitists post.

See also : Arachnophobia (movie)