A quick and dirty C# port of Google's robots.txt parser. The methods not used by the example program and tests are not currently implemented.
Please see the Notes section of the original.
Like Google's robots.txt parser and matcher C++ library, this port is licensed under the terms of the Apache license. See LICENSE for more information.