Google and robots.txt exclusions
Friday, May 6th, 2005It seems that Google does a very poor job of handling files that have been excluded by robots.txt. It seems that Google loves to display URLS only, for any file that they’ve visited from another file, even if that file is excluded explicity via robots.txt.
That behavior seems strange. Why have a wildcard to […]