Class | WebRobots |
In: |
lib/webrobots/robotstxt.rb
lib/webrobots.rb |
Parent: | Object |
user_agent | [R] | Returns the robot name initially given. |
Creates a WebRobots object for a robot named user_agent, with optional options.
Tests if the robot is allowed to access a resource at url. If a malformed URI string is given, URI::InvalidURIError is raised. If a relative URI or a non-HTTP/HTTPS URI is given, ArgumentError is raised.
Returns an array of Sitemap URLs. See allowed?() for a list of errors that may be raised.