reivilibre
/
quickpeep
1
0
Fork 0
Code Issues 11 Pull Requests Projects Releases 11 Wiki Activity
quickpeep/README.md

4.5 KiB
Raw Permalink Blame History

QuickPeep

Small-scale 'artisanal web' search engine project, favouring quality over completeness.

Motivation

Modern web search can be rubbish. It feels like I'm getting the same websites time and time again, and that I rarely manage to come across small, independent websites instead of content/SEO mills.

Typical modern websites are rubbish. They bother you with adverts, annoying nagging pop-ups (often loaded with dark patterns), privacy-disregarding trackers and slow-loading content that relies on needless amounts of JavaScript to display plain rich text that has been possible with pure HTML for decades.

QuickPeep aims to index good-quality websites, favouring small personal sites (such as blogs), whilst cutting out the rubbish. Websites that don't care about the reader's experience are not welcome; QuickPeep will detect pop-up nags, adverts and privacy issues in order to keep them away.

As a separate issue, many websites decide to use CloudFlare to intercept their traffic. This is not in the spirit of the decentralised web; CloudFlare becomes a single party that, if compromised or dishonest, could exert a disproportional amount of power over users. One notable example of this is that they bombard privacy-seeking users on VPNs or Tor with needless CAPTCHAs, giving them a worse experience on a lot of the web. These websites will be detected so that they can be filtered out at will.

QuickPeep will follow the trade-off of preferring not to provide any results rather than to provide results laced with rubbish. If you need to fall back to a conventional search engine, this will eventually be possible to do right within QuickPeep...

Features

Crossed-out things are aspirational and not yet implemented.

  • Shareable 'rakepacks', so that anyone can run their own search instance without needing to rake (crawl) themselves
    • Dense encoding to minimise disk space usage; compressed with Zstd.
  • Raking (crawling) support for
    • HTML (including redirecting to Canonical URLs)
      • Language detection for when the metadata is absent.
    • Redirects
    • Gemtext over Gemini
    • RSS, Atom and JSON feeds
    • XML Sitemaps
  • Detection of anti-features, with ability to block or downrate as desired:
    • CloudFlare
    • Adverts
    • Nagging pop-ups
    • Trackers
  • Article content extraction, to provide more weight to words found within the article content (based on a Rust version of Mozilla's Readability engine)
  • (Misc)
    • Use of the Public Suffix List
    • Tagging URL patterns; e.g. to mark documentation as 'old'.
  • Page duplicate content detection (e.g. to detect / and /index.html, or non-HTTPS and HTTPS, or non-www and www...)

Limitations

  • Only supports English (and English dialects) for now.
    • Should hopefully be customisable later on, but even though I can speak some foreign languages, I don't know any communities to start with to seed non-English search.
  • Websites have to be manually allowed and tagged.
    • It's otherwise difficult to know how to detect good-quality sites automatically, or to tag what they are.
    • There may be ways to improve this; e.g. with machine learning techniques or crowdsourcing of data.
  • The search index needs to remain small enough to be usable on modest hardware, so there's no way we can hope to index everything.

Architecture

Not written yet.

The stages of the QuickPeep pipeline are briefly described in an introductory blog post.

Development and Running

Not written yet.

Some hints may be obtained from the introductory blog post mentioned in the 'Architecture' section, but it's probably quite difficult to follow right now.

Helper scripts

scripts contains some helper scripts, which you probably need to run before operating QuickPeep:

  • get_cf_ips.sh: fetches IP addresses ued for CloudFlare detection.
  • get_adblock_filters.sh: fetches adblock filters used to detect nags/adverts/trackers.

Licence

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 licence, shall be dual licensed as above, without any additional terms or conditions.