rei-forks
/
cylon
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Update readme

This commit is contained in:
Creston Bunch 2021-04-08 17:40:24 -05:00
parent cb01fad92a
commit 56e73e017d
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
README.md
View File

@ -11,6 +11,7 @@ the following directives (notably `Site-map` is missing):
- `User-agent` - `User-agent`
- `Allow` - `Allow`
- `Disallow` - `Disallow`
- `Crawl-Delay`
In addition, Cylon supports `*` as a wildcard character to In addition, Cylon supports `*` as a wildcard character to
match any length substring of 0 or more characters, as well match any length substring of 0 or more characters, as well
@ -18,7 +19,7 @@ as the `$` character to match the end of a path.
## Usage ## Usage
Using Cylon is very simple. Simply create a new complier Using Cylon is very simple. Simply create a new compiler
for your user agent, then compile the robots.txt file. for your user agent, then compile the robots.txt file.
```rust ```rust