You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
Olivier 'reivilibre' 9dc49f12bd Add some 'About' info 3 days ago
.ci Use amd64 image for mdbook 5 days ago
docs STASH notes about seeds 3 months ago
nixos_modules Start moving Nix Flake into the root 4 weeks ago
quickpeep Add some 'About' info 3 days ago
quickpeep_densedoc Add MIT/Apache2 licence 3 months ago
quickpeep_html_charset_detection Remove forgotten println statements 2 weeks ago
quickpeep_index Bump fancy_mdbx to v0.1.1 to fix size limit 5 days ago
quickpeep_indexer Add context for unreadable rakepacks 4 days ago
quickpeep_moz_readability Add MIT/Apache2 licence 3 months ago
quickpeep_raker Use the sniffed encoding in page extraction 3 weeks ago
quickpeep_seed_parser Add MIT/Apache2 licence 3 months ago
quickpeep_static Add proof-of-concept search UI page 3 months ago
quickpeep_structs Add MIT/Apache2 licence 3 months ago
quickpeep_utils Add MIT/Apache2 licence 3 months ago
scripts Clean-ups and support pulling out references 4 months ago
test_vm Start moving Nix Flake into the root 4 weeks ago
.env Initial working version of the seed collection service 4 months ago
.gitignore Add backoff reinstatement function to store 3 weeks ago
Cargo.lock Bump fancy_mdbx to v0.1.1 to fix size limit 5 days ago
Cargo.toml Create a crate for HTML charset detection 3 weeks ago
FLAKE.md Start moving Nix Flake into the root 4 weeks ago
LICENCE.Apache2 Add MIT/Apache2 licence 3 months ago
LICENCE.MIT Add MIT/Apache2 licence 3 months ago
README.md Add MIT/Apache2 licence 3 months ago
book.toml Add CI config 4 months ago
deny.toml Add MIT/Apache2 licence 3 months ago
flake.lock Update lock 3 weeks ago
flake.nix Simplify Flake (remove 'src' input that causes problem on 'old' Nixes) 4 weeks ago
grafana.json Add a grafana dashboard 4 weeks ago
quickpeep.sample.ron Use the unified config in the indexer 3 months ago

README.md

QuickPeep

Small-scale 'artisanal web' search engine project, favouring quality over completeness.

Motivation

Modern web search can be rubbish. It feels like I'm getting the same websites time and time again, and that I rarely manage to come across small, independent websites instead of content/SEO mills.

Typical modern websites are rubbish. They bother you with adverts, annoying nagging pop-ups (often loaded with dark patterns), privacy-disregarding trackers and slow-loading content that relies on needless amounts of JavaScript to display plain rich text that has been possible with pure HTML for decades.

QuickPeep aims to index good-quality websites, favouring small personal sites (such as blogs), whilst cutting out the rubbish. Websites that don't care about the reader's experience are not welcome; QuickPeep will detect pop-up nags, adverts and privacy issues in order to keep them away.

As a separate issue, many websites decide to use CloudFlare to intercept their traffic. This is not in the spirit of the decentralised web; CloudFlare becomes a single party that, if compromised or dishonest, could exert a disproportional amount of power over users. One notable example of this is that they bombard privacy-seeking users on VPNs or Tor with needless CAPTCHAs, giving them a worse experience on a lot of the web. These websites will be detected so that they can be filtered out at will.

QuickPeep will follow the trade-off of preferring not to provide any results rather than to provide results laced with rubbish. If you need to fall back to a conventional search engine, this will eventually be possible to do right within QuickPeep...

Features

Crossed-out things are aspirational and not yet implemented.

  • Shareable 'rakepacks', so that anyone can run their own search instance without needing to rake (crawl) themselves
    • Dense encoding to minimise disk space usage; compressed with Zstd?
  • Raking (crawling) support for
    • HTML (including redirecting to Canonical URLs)
      • Language detection
    • Redirects
    • Gemtext over Gemini
    • RSS, Atom and JSON feeds
    • XML Sitemaps
  • Detection of anti-features, with ability to block or downrate as desired:
    • CloudFlare
    • Adverts
    • Nagging pop-ups
    • Trackers
  • Article content extraction, to provide more weight to words found within the article content (based on a Rust version of Mozilla's Readability engine)
  • (Misc)
    • Use of the Public Suffix List
    • Tagging URL patterns; e.g. to mark documentation as 'old'.
  • Page duplicate content detection (e.g. to detect / and /index.html, or non-HTTPS and HTTPS, or non-www and www...)
  • Language detection for pages that don't have that metadata available.

Limitations

  • Only supports English (and English dialects) for now.
    • Should hopefully be customisable later on, but even though I can speak some foreign languages, I don't know any communities to start with to seed non-English search.
  • Websites have to be manually allowed and tagged.
    • It's otherwise difficult to know how to detect good-quality sites automatically, or to tag what they are.
    • There may be ways to improve this; e.g. with machine learning techniques or crowdsourcing of data.
  • The search index needs to remain small enough to be usable on modest hardware, so there's no way we can hope to index everything.

Architecture

Not written yet.

Development and Running

Not written yet.

Helper scripts

scripts contains some helper scripts, which you probably need to run before operating QuickPeep:

  • get_cf_ips.sh: fetches IP addresses ued for CloudFlare detection.
  • get_adblock_filters.sh: fetches adblock filters used to detect nags/adverts/trackers.

Licence

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 licence, shall be dual licensed as above, without any additional terms or conditions.