From 56e73e017d87a6ee0d2404a426e2416d568fee8a Mon Sep 17 00:00:00 2001 From: Creston Bunch Date: Thu, 8 Apr 2021 17:40:24 -0500 Subject: [PATCH] Update readme --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index ad063da..7bdde62 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,7 @@ the following directives (notably `Site-map` is missing): - `User-agent` - `Allow` - `Disallow` +- `Crawl-Delay` In addition, Cylon supports `*` as a wildcard character to match any length substring of 0 or more characters, as well @@ -18,7 +19,7 @@ as the `$` character to match the end of a path. ## Usage -Using Cylon is very simple. Simply create a new complier +Using Cylon is very simple. Simply create a new compiler for your user agent, then compile the robots.txt file. ```rust