rei-forks
/
cylon
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Update readme

This commit is contained in:
Creston Bunch 2021-04-08 17:40:24 -05:00
parent cb01fad92a
commit 56e73e017d
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
README.md
View File

@ -11,6 +11,7 @@ the following directives (notably `Site-map` is missing):
- `User-agent`
- `Allow`
- `Disallow`
- `Crawl-Delay`
In addition, Cylon supports `*` as a wildcard character to
match any length substring of 0 or more characters, as well
@ -18,7 +19,7 @@ as the `$` character to match the end of a path.
## Usage
Using Cylon is very simple. Simply create a new complier
Using Cylon is very simple. Simply create a new compiler
for your user agent, then compile the robots.txt file.
```rust