Google may expand list of unsupported Robots.txt rules

Google may expand the list of unsupported robots.txt rules in its documentation based on analysis of real-world robots.txt data collected via HTTP Archive. Gary Illyes and Martin Splitt described the project in the last episode of Search Off the Record.…











