How bad are search results? Let's compare Google, Bing, Marginalia, Kagi, Mwmbl, and ChatGPT

Search Quality in 2023: What Naive Queries Actually Return

The claim that Google search results have degraded over the past decade is easy to make but harder to verify. To test it, we ran straightforward, naive queries across five search engines and ChatGPT — Google, Bing, Marginalia, Kagi, and Mwmbl — deliberately disabling ad blockers to simulate the non-expert browsing experience. The queries fell into three categories: representative questions from a non-technical user setting up a new computer, simple factual questions that are hard to answer without the right resources, and a local weather query.

The test queries were:

• download youtube videos
• ad blocker
• download firefox
• Why do wider tires have better grip?
• Why do they keep making cpu transistors smaller?
• vancouver snow forecast winter 2023

Results were ranked on a scale from Terrible to Great, with particular weight given to the presence of scams or misleading content. The grading is subjective — it's not objectively clear whether moderately relevant results with no scams are better than highly relevant results interspersed with attempts to install badware or extract credit card details. For this analysis, scams are considered especially harmful, so results containing them were ranked lower.

The Hallucination Problem Isn't Unique to LLMs

ChatGPT's performance was inconsistent: it outperformed traditional search engines on one query, handled another adequately, refused to answer another, and hallucinated nonsense for several. This is the well-known failure mode of LLMs. However, attributing hallucination solely to ChatGPT misses the point. Traditional search engines returned their own versions of fabricated results on these queries.

In the snow forecast query, search engines delivered deliberately fabricated content: one fake forecast site designed to generate ad revenue through shady ads, and another designed to convince the user that a cold, snowy winter was coming — the opposite of the actual forecast — apparently to sell unnecessary snow removal services. Other fabricated results included a fake review site funneling users to a scammy ad blocker with a nearly-uncancellable subscription, and a fake "organic" blog post designed to install a Chrome extension that exposes shopping data. There were also numerous run-of-the-mill SEO garbage sites designed to farm ad clicks, plus human hallucinations on platforms like Quora, Reddit, and Stack Exchange, where users confidently state incorrect answers.

On these queries, the major search engines returned more hallucinated or hallucination-adjacent results than ChatGPT. Concerns about LLM hallucinations are legitimate, but the traditional ecosystem has an underlying problem: the system is incentivized to put whatever is most profitable in front of the user, which is often not the best result.

Ads and Incentives: The PageRank Warning, 25 Years On

Sergey Brin and Larry Page's original PageRank paper anticipated this exact issue:

Currently, the predominant business model for commercial search engines is advertising. The goals of the advertising business model do not always correspond to providing quality search to users. ... We expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the Consumers.

That prediction has proven accurate. If the best result for "ad blocker" is a free, high-quality tool like ublock origin, the top ad slot is worth far more to a company making an ad blocker designed to trick users into paying for a lower-quality product. That scam ad blocker will outbid the free one, and will also have more resources for SEO and marketing efforts. Unless search engines mount an effective countermeasure, the top results will favor paid ad blockers — even when they're significantly worse for users.

Many ranking signals are derived from clicks and engagement, which only drives users to the best results when users are sophisticated enough to know what those are. Human raters evaluate page quality, but they face the same problem. Trillion-dollar companies have demonstrably chosen not to invest enough in combating SEO spam and scams, since such ads are still common. Some employees have argued that ads inform users about options they wouldn't otherwise know about, but anyone browsing without an ad blocker will see misleading ads, ads that pretend to be windows, and ads for "GAMES THAT PAY REAL CASH" at the top of results for gambling addiction searches.

When Results Mimic Ads — And Vice Versa

Google and Bing have both made ads visually indistinguishable from organic results. Most users won't know they're clicking on ads. This is exactly the insidiousness that Brin and Page called out: users conflating real results with paid placements. To account for this in our ranking, results are ordered by page position — if four ads appear above the first organic result, they occupy ranks 1–4.

Re-running a query from the original paper — "cellular phone" — illustrates the shift. The top result was a Google Store link to buy Google's own Pixel 7. The remaining top results were Android phones sold on Amazon, followed by the Wikipedia page for Mobile Phone, then seven more commercial results, then an ad-laden SEO blogspam page defining "cell phone" with phone ads, then more commercial results. The result Brin and Page suggested as exemplary — "The Effect of Cellular Phone Use Upon Driver Attention" — is nowhere to be found. Google also buys ads that trick users, such as paying for installers that try to get users to choose Chrome over Firefox.

Independent Engines, Mixed Results

Marginalia, built by a single person, performed relatively well: decent but not great answers where it succeeded, obvious irrelevance where it couldn't answer, and the lowest rate of scams among the search engines tested. Mwmbl's results were unedited, despite the engine allowing users to directly modify rankings; editing your own benchmark is too easy to be meaningful. Just browsing Mwmbl reveals its moderation problem: the most recent ranking edit added blogspam as the top entry for "RSS," with no easily findable way to report the change or user.

Can a small engine displace Google for mainstream users? Searching for any obscure page in 2016, Google would return the correct result; Marginalia would have been a curiosity. Today, Google often fails to return pages even when you search for an exact string from the page. Non-popular pages are either unindexed or effectively unindexed in some slow corner of the index. Finding such a page requires remembering text from a page that links to it — often several hops removed — and then manually traversing the link graph, sometimes via archive.org when links are dead. This basically never happened in 2005 and rarely in 2015, but now happens a large fraction of the time. Google doesn't return most tweets I want to find, even with an exact string and site:twitter.com. The public web appears less indexed than at any point since the early days of web search.

Economies of Scale vs. Actual Results

The argument that a small search engine can't compete with Google's economies of scale is common — and contradicted by the actual results. Google's problems have made the requirements for a useful engine lower: real-time indexing and fancy NLP aren't necessary when the main search engine returns SEO garbage for many naive queries and can't find pages that smaller engines index. Marginalia's smaller index is no longer a disqualifier.

As for the idea that a collection of small engines could beat Google — that's already true for many queries. A modern Metacrawler-style aggregator of multiple search engines would arguably be technically strong today, though it would face licensing hurdles. The original Metacrawler became irrelevant because Google returned better results than aggregation; that's no longer obviously the case.

The verdict on the debate between "search results are terrible" and "the lamentations are overblown" is clear: the former is correct. Google's original advantage was that you didn't need qualifiers or site restrictions to get good results. We've come full circle — and then some, since the contortions now required exceed those of the AltaVista era.

Search engines that didn't make the cut

A few other engines were evaluated but left out of the main comparison, mostly for relevance or redundancy reasons.

• DuckDuckGo: Historically, DDG results with an ad blocker looked nearly identical to Bing's. That's less true now, but the overlap is still high enough that listing DDG separately felt redundant. DDG remains my default engine; like Google, it works well for experienced queriers but stumbles on the naive queries used here.
• wiby.me: Another engine aimed at obscure results, similar in spirit to Marginalia. I ran four of the queries from this post and got genuinely novel results — but none of them were relevant to what was asked.
• searchmysite.net: Delivered somewhat relevant results for some queries, though not as good as Marginalia's. Notably, many fewer scam and ad-laden pages surfaced than through Google, Bing, or Kagi.
• indieweb-search.jamesg.blog: Appeared to be down. Every query returned a server error.
• Teclis: The search box remains, but any query now responds that Teclis is closed due to bot abuse, with results still available via Kagi's “Non-commercial Web” lens and API. The front page notes that 99.9% of traffic was bots, an “insane amount of bot traffic.”

Chasing good results: the tire-width question

Most programmers can probably get decent results on every query above, with the tire-width-vs-grip question being the notable exception. Here's the path that eventually led somewhere useful.

A YouTube search was the first stop, since much of the best car content now lives there. A video title claimed to answer the question (the video itself didn't), but a comment directed readers to Carroll Smith's Tune to Win. The comment claimed chapter 1 explains why wider tires grip better, though no such explanation is in the book. Chapter 1 does note that race cars run wider tires than passenger cars, and that passenger cars are trending wider, along with some slip-angle commentary giving an intuitive sketch of cornering benefits. The braking difference goes unexplained. The book also suggests the primary reason for a wider contact patch is indirectly reducing heat buildup, allowing tire designs that operate over a narrower temperature range and thus use softer rubber — plausible, but far from accounting for the observed behavior.

Following references from Tune to Win led to Kummer's The Unified Theory of Tire and Rubber Friction and Browne's The Physics of Tire Traction (Smith misspells it as “Brooke”). Neither explained the phenomenon either. Related-title searches surfaced Milliken and Milliken's Race Car Vehicle Dynamics, which approached an explanation without delivering one, and then Guiggiani's The Science of Vehicle Dynamics, which finally addressed how to think about and model the relevant factors. Guiggiani's final chapter mentions the “brush model” of tires; searching “brush model tire width” turned up Pacejka's Tire and Vehicle Dynamics, which starts to explain wider-tire grip advantages and the modeling needed to capture easily observed tire behavior.

Different people have different tricks for getting good search results, and if you have a better route to an answer on this one, I'd like to hear it. But a recurring pattern: whenever I note something doesn't work, the most common suggestion is a fix that doesn't work, even when the original post explicitly says so. My post on filesystem correctness gets comments recommending the rename trick, despite explaining why it fails and citing a paper on that failure. An expanded talk on the subject drew the same suggestion, and it still doesn't hold up — auto_da_alloc on ext4 doesn't make the trick safe, since it requires checking filesystem compatibility and effectively auto-replacing incorrect code with correct code, at which point writing correct code directly is simpler. If you want to suggest an explanation for the tire-width phenomenon, please make sure it is not among the standard incorrect ones noted in this post, and that it can account for all the behavior a real explanation must cover.

Query-by-query breakdown

All queries were run in a fresh incognito window with cleared cookies, except Kagi, which requires a logged-in session. Kagi queries used a brand-new account with no personalization or filters, though the queries were run sequentially, so some basic click-based ranking could have influenced later results. Searches were executed from Vancouver, BC, which may have introduced a localized ranking component for some engines.

The queries below were mostly run in November 2023, with a few in mid-December. As a general habit I use keyword queries rather than natural language ones, but for the "Tire" and "Snow" queries I deliberately used natural language to reflect how many users actually search. The ChatGPT queries used the free ChatGPT 3.5 version. A full matrix of queries across keyword and natural language variations would have been ideal, but at 17,000 words already, this post is long enough without quadrupling the workload for marginal additional coverage. Instead, a representative handful of queries plus judgment based on broader experience will have to do.

For each query, results are ordered best to worst by my subjective assessment of the answer quality they returned.

download youtube videos

• Ok results
  • Marginalia: Top link is for youtube-dl. Most links aren't relevant, but far fewer scams than the big search engines.
• Bad results
  • Mwmbl: Some bad sites and scams, but fewer than the big engines. Top 10 includes one indirect link to youtube-dl and one GUI wrapper for it.
  • Kagi: Mostly scammy sites, but a couple pages down there's an archived 2010 link to youtube-dl via web.archive.org.
• Very bad results
  • ChatGPT: Basically refuses to answer. A prompt engineer could probably coax an answer out, but not with a naive question.
• Terrible results
  • Google: Mostly scam sites charging for worse versions of free software, plus ad-laden listicles with no useful suggestions. Zero good results, and some YouTube results equivalent to blogspam.
  • Bing: Same pattern of scams and listicles. Arguably zero good results; result #10 might be ok but appears to be malware.

The ideal top hit here is yt-dlp or a thin GUI wrapper around it. youtube-dl or other maintained forks would be acceptable. No engine delivered great or good results by that bar.

ad blocker

• Great results
  • ChatGPT: First suggestion is ublock origin.
• Ok results
  • Marginalia: ublock origin appears at positions 3, 4, and 8. Nothing directly scams you; only one SEO ad farm link, which is far better than the major engines.
• Bad results
  • Mwmbl: Many irrelevant links plus some ghostery results. One scam link, so fewer scams than the commercial engines.
• Very bad results
  • Google: Heavy on "Acceptable Ads" style blockers that let ads through in exchange for publisher payments, which makes them not just ineffective but misaligned with user interests. Some apparent scams. Zero ublock origin links and YouTube blogspam.
  • Kagi: Similar to Google, more scams, but still fewer than Bing.
• Terrible results
  • Bing: Same acceptable-ads-heavy pattern as Google but more scams and no YouTube spam.

An ideal result would put ublock origin at the top, or at least somewhere in the page without scam links. None of the engines managed a "good" result under that definition.

download Firefox

• Great results
  • Bing: Links to download Firefox.
  • Mwmbl: Links to download Firefox.
  • Kagi: Links to download Firefox.
• Good results
  • ChatGPT: Technically incorrect instructions, but a human could easily decode them and get to the download.
• Ok results
  • Marginalia: Indirect instructions that lead to a Firefox download.
• Bad results
  • Google: Top links are legitimate, but result #7 is a scam pushing badware and #10 is an ad trying to collect credit card info.

This query is simple enough that most engines get it right, but Google managing to slip a scam and a suspicious ad into the top ten is still notable.

Why do wider tires have better grip?

• Bad results
  • Mwmbl: One obviously incorrect result, nothing else.
  • Marginalia: Two obviously incorrect results, nothing else.
• Very bad results
  • ChatGPT: A classic ChatGPT-style "hallucination" — sounds plausible and resembles common but incorrect internet commentary on the topic, just better written.
• Terrible results
  • Google / Bing / Kagi: Incorrect ad-laden results with the usual rate of scammy ads.

The ideal answer would clearly explain the underlying physics without the common oversimplifications. No engine delivered even a "good" or "ok" result by that standard.

Why do they keep making cpu transistors smaller?

• Ok results
  • Bing: Top set includes a partial answer that could lead to correct explanations with further searching, mixed in with irrelevant results and SEO'd garbage.
• Bad results
  • Marginalia: No answers.
  • Mwmbl: One obviously irrelevant answer.
  • Google: The 5th link has the right keywords to maybe lead somewhere useful. Most others have misleading or incorrect partial answers, lots of Quora links that don't answer the question, and other poor SEO content.
  • Kagi: The 10th link offers a fairly direct path to the correct answer, but only if you scroll far enough on that page. The rest isn't good.
• Very bad results
  • ChatGPT: Doesn't really answer the question, and further prompting causes additional hallucinated incorrect reasons.

The clearest explanations of transistor scaling live in VLSI textbooks, lecture notes, and slides. That should be findable, but it isn't.

vancouver snow forecast winter 2023

• Bad results
  • Marginalia: No results.
  • ChatGPT: Incorrect results. Accidentally prepending "User\n" to the question made it return a link to the right website, but in a way that would make navigating to a decent answer difficult. A slightly different prompt might pseudo-randomly produce an ok result.
  • Mwmbl: A bunch of obviously irrelevant results.
• Terrible results
  • Bing: Mostly irrelevant. The top seemingly relevant result is 5th and appears to be a scam site fabricating weather forecasts to serve ads on a heavily SEO'd page.
  • Kagi: Top 4 results are all from the same scam forecast site that Bing had at 5th.
  • Google: Mostly irrelevant, and the #1 result is a fake answer from a local snow removal company projecting significant snow and cold to sell unnecessary service. The rest are SEO'd garbage full of ads.

A good answer here would point to Environment Canada's winter 2023 multi-month forecast predicting below-normal snow and above-normal temperatures. No engine got close; the commercial ones actively pushed fabricated predictions.

Appendix: a closer look at each query

“Download youtube videos”

We searched “download youtube videos” (Ycombinator’s Xe suggested “youtube downloader”, which returns similar results). The ideal link is yt-dlp, a fork of the now-defunct youtube-dlc, which itself forked from the sparsely updated youtube-dl. Older but functional forks are acceptable.

Google

1. Downloader site with “Norton SafeWeb” assurances. Interacting prompts an extension install and notification request. Download attempts trigger a full-screen pop-over pushing “CyberShield”. There is no dismissal path. Searching “cybershield chrome extension” returns a knowledge card describing it as an ad-injecting browser extension that opens scam tabs.
2. Downloader site. Pop-up prompts extension install. Entering a URL triggers a scam pop-up but also downloads the video.
3. PC Magazine listicle. Suggests paying, VLC (noted as not working), $15/yr and $26/yr software, and FlixGrab, with a warning against downloader sites. More than one ad per suggestion.
4. Downloader site with shady pop-overs before page interaction.
5. Downloader site with scam-ad pop-ups.
6. Downloader site with ads like “Samantha 24, vancouver | I want sex, write to WhatsApp | Close / Continue”. Any click pushes an “Adblock Ultimate” install.
7. ZDNet listicle whose top pick, ClipGrab, reportedly bundles adware/junkware. Page is ad-heavy with autoplay video.
8. [YouTube] 2+ minutes of ads, then instructions to buy YouTube Premium.
9. [YouTube] Video funneling users to paid download software.
10. [YouTube] PC Magazine video: users don't “have to” download videos; read their article instead.
11. Downloader site with scam ads. Interaction pushes “Adblock Ultimate”.
12. Downloader site with scam-ad pop-ups.
13. Downloader site with scam ads.

Nine of ten organic results try to install badware or link to ad scams. The one exception suggests paid tools, not the free, correct option. The three YouTube videos are video SEO spam. Google itself showed few ads, despite past tests showing the opposite.

Bing

1. Downloader site, same as Google’s #2.
2. [EXPLORE FURTHER] Downloader site with pulsing ads and “50% off” Christmas branding in mid-November. Downloads show a fake progress bar with a “too slow?” link, then a download link that installs the “oWebster Search extension”, which hijacks browsers to show ads. Removal guides are themselves scams. Declining the install loops back to the pop-over; no actual download is provided.
3. [EXPLORE FURTHER] Downloader site with fake chat ads. The download button pushes a copycat ad blocker that injects pop-over ads. Video downloads do work.
4. [EXPLORE FURTHER] Same as #3.
5. [EXPLORE FURTHER] Same as Google’s #1.
6. [EXPLORE FURTHER] MP4 conversion site without YouTube download claims.
7. Google’s #1 again.
8. [EXPLORE FURTHER] YouTube’s homepage.
9. [EXPLORE FURTHER] Downloader site with an ad that looks like a download button. A valid URL errors with “no video”.
10. Large card with a fake download button leading to the site, which claims to be adware-free. Antivirus vendors tag it as malware. The embedded YouTube video has been removed for ToS violations. Mac/Linux “downloads” are install instructions for youtube-dl; only Windows gets an executable.
11. PC Magazine listicle.
12. Ad for downloader software claiming “345,764,132 downloads today”. Reddit indicates malware.
13. Ad for paid downloader software.

First page ends there. No useful result, and several scams or paid wrappers around free open-source projects.

Marginalia

1. 12-year-old answer pointing to youtube-dl, but the URL now shows a German court blockage notice.
2. Typical SEO content.
3. Leawo YouTube Downloader. Not immediately suspicious.
4. SEO listicle.
5. Bug report for unrelated software.
6. Blogger recommending “4K Video Downloader”, which is legitimate but paywalled — worse than yt-dlp.
7. Blog post on installing and using yt-dlp, noting it has updated from youtube-dl.
8. More paid software with crack links on Reddit.
9. Outdated listicle recommending RealPlayer.
10. Keyboard Maestro script for YouTube downloads — useful only if already using that software.

Best set by a wide margin. First link is dead, but youtube-dl is discoverable from it. The yt-dlp recommendation with instructions is the most helpful, non-scam result any engine gave.

Kagi

• 1. Same NortonSafeWeb/CyberShield site as Google’s top result.
• 2. Same site again, incorrectly dated “Dec 20, 2003”.
• 3. Downloader with scam ads on download.
• 4. Downloader with fake “Ad removed” overlay that is itself an ad.
• 5. Duplicate of #4.
• 6–7. “Interesting Finds”: a YouTube transcription repo and a Google Takeout photo backup tool.
• 8. Downloader site.
• 9–13. “Blast from the Past”: one youtube-dl GitHub archive link from 2010 plus irrelevant entries.
• 14. SEO blogspam with a “Greasemonkey script” link that leads to scam ads.
• 15. $5/mo YouTube downloader service.

Mwmbl

1. Downloader site not seen elsewhere. Displays “503 NA - Service Deprecating” ads; download links spawn disappearing ad panes.
2. $20 YouTube downloader.
3. 2016 youtube-dl install blog post with two low-quality but non-venal ads.
4. Downloader site claiming it is “the only YouTube Downloader that is 100% ad-free”. Download link works.
5. YouTube video about youtube-dlg, a GUI wrapper for youtube-dl, on Linux (query run on Mac).
6. 2007 blog post now auto-forwarding to a 2020 SEO spam listicle with autoplay videos. Archive.org confirms the original was reasonable for its era.
7. Sponsored post for a downloader that Reddit users call a waste of money. The page's ads include a fake coupon extension funnel.
8. NortonSafeWeb downloader site, with a broken link to its Instagram-video variant.
9. Google help about downloading one's own videos.
10. SEO blogspam with newsletter and subscribe pop-overs, recommending Windows software reviewers call a PUP.

Essentially Google or Bing quality.

ChatGPT

Prompt: “How can I download youtube videos?”

First attempt returned “Our systems are a bit busy.” Second attempt produced a lecture about YouTube Premium and third-party risk. A follow-up (“What are the best third-party apps and websites?”) returned another warning:

I don't endorse or provide information on specific third-party apps or websites for downloading YouTube videos. It's essential to use caution and adhere to legal and ethical guidelines when it comes to online content.

“ad blocker”

The ideal result is ublock origin. Any blocker that fails to block ads by default is a letdown, and the bar for “not a scam that injects ads” is low. Independent benchmarks usually put ublock origin first, and it blocks ads for free.

Google

1. “AdBlock — best ad blocker”. Acceptable Ads means it doesn’t block everything.
2. Adblock Plus, likewise allowing Acceptable Ads by default.
3. Another AdBlock page, same policy.
4. Adblock Plus duplicate.
5. AdGuard, which pushes paid “AdGuard for Mac”; free use reportedly injects more ads if you don’t subscribe.
6. “AdBlock Pro” on Safari with in-app purchases for features like video blocking.
7. [YouTube] “How youtube is handling the adblock backlash” — 30 seconds of content after a 15-second ad.
8. [YouTube] “My thoughts on the youtube adblocker drama”.
9. [YouTube] “How to Block Ads online in Google Chrome for FREE [2023]”. Top comment notes it doesn't explain how to stop YouTube ads. Video shows installation of the #1 result, followed by a $30 payment screen.
10. “AdBlock for Mobile” App Store listing with a 3.2-star rating and reviews saying it fails.
11. MalwareBytes ad blocker, which doesn’t block all ads.
12. AdGuard’s Chrome page, same as #5.
13. [ad] NordVPN.
14. [ad] TOTAL Ad Block, timestamped with a fake year. Reddit indicates unsubscribing is nearly impossible.
15. [ad] Avast browser (not an ad blocker) promoted as one. Has history of security and data-collection issues.

No ublock origin. Some scams, though fewer and less egregious versus “youtube downloader”. Mostly default-on-selective blockers.

Bing

• 1. [ad] Something called a “100% Free Ad Blocker” that instills the entire Avast browser.
• 2. [ad] TOTAL Ad Block, with a nag overlay and a cancellation path that requires a phone call and chargeback threats.
• 3. [ad] A fake review site listing TOTAL as #1 with 5 stars and ublock origin #5 with 3.5. The ublock “visit” link routes to a lookalike fake ublock site.
• 4. [ad] AVG Free Antivirus.
• 5–6. [EXPLORE] Adblock Plus blog and features page.
• 7. “Adblock Plus”.
• 8–13. More Adblock Plus subpages.

The ad label is noticeably subtler than usual. Given how often users mistake ads for organic content, everyone likely believes the first page suggests scam blockers and a fake review site as objective results.

Marginalia

1. “Is ad-blocker software permissible?” (judaism.stackexchange.com).
2. Ghostery blogspam. Ghostery shows its own ads unless paid; it also sells sponsored-link removal.
3. https://shouldiblockads.com/, which recommends ublock origin first.
4. Firefox Add-ons Blog with similar first choice.
5. Personal account of installing an ad blocker.
6. Opera browser.
7. Anti-anti-adblocker polemic.
8. ublock origin.
9. Fairphone forum discussion.
10. SEO backlink spam.

Arguably the best result set: third and fourth links name the right tool, and nothing is a blatant scam.

Mwmbl

1. Bitly “thinkpiece” from a VC.
2. A cryptojacking test site that redirects to Opera.
3. Ghostery.
4. Ghostery again.
5. 1Blocker. Paid with complaints about tough cancellations, though functional.
6. AdGuard blogspam with banner ad.
7. An ad-laden one-page site that ranks by containing the phrase “ad blocker detected”.
8. Another “ad blocker detected” fake content site.
9. MalwareBytes ad blocker.
10. Hacker News comments about YouTube’s adblocker crackdown; the 41st comment recommends ublock origin.

After the test, suggestion access was attempted. Gmail flagged the signup as spam. Post-edit, “ad blocker” returns ublock origin as #1. The scoring below reflects the pre-edit state.

Kagi

• 1–11. Adblock Plus and subpages.
• 12. AdBlock.
• 13. Adblock Plus iOS.
• 14. A blog post about YouTube’s blocker crackdown.
• 15–18. “Interesting Finds”: more adblock-crackdown pieces, one with an unresolvable TOTAL Adblock pop-over, another with a giant ad above and below the fold.
• 19–20. AdBlock install pages.
• 21–25. “Blast from the Past”: Optimum ad blocker, a subvert-adblock Medium post, a Mozilla blog rebutting “Why Ad Blocking is devastating to the sites you love”, a network-wide blocker piece, and a link explaining how Ghostery feeds data to the ad industry.

Comparable to Google and Bing; middle of the pack on scams.

ChatGPT

Prompt: “How do I install the best ad blocker?”

First suggestion: ublock origin. Second: Adblock Plus. Overall best result — with a large head start.

“download firefox”

Google

• 1–6: Firefox links.
• 7: Blogspam ad with badware install push.
• 8–9: Firefox links.
• 10 [ad]: Site that asks for personal details and a credit card for Firefox downloads, with referral links including gambling sites.

Two scam results inside the top ten. In 2017 we observed a paid ad placed above Firefox’s own site for this query; no repeat is seen. Yet for an organic brand query, two scams is Bad — I would only call this acceptable if Google chose to trust the top of its list completely.

Bing

• 1–12: Firefox links.
• 13 [ad]: Avast browser.

No obvious scams; good first page.

Marginalia

• 1: StackExchange question on Firefox packages.
• 2–9: Firefox-adjacent links.
• 10: “Internet Download Accelerator online help”.

None explicitly for downloading Firefox. Ranking above Google assumes scam-avoidance is more valuable than direct answer accuracy, which this post does.

Mwmbl

• 1–7: Firefox download links.
• 8: An unrelated Tumblr blog titled “Love yourself, download firefox”.
• 9: Firefox Nightly page.
• 10: Suspicious “Firefox” binary that forces an Opera download install prompt. Neither binary was executed for verification.

Kagi

• 1–3: Firefox downloads.
• 4–5: “Interesting Finds” — a dead tweet about Edge’s OS-level push, plus an ad-heavy article.
• 6: Firefox download.
• 7–10: Ancient Firefox point releases and a Mozilla/eBay collaboration story.
• 11: Mozilla homepage.
• 12: Firefox download.

No scams, but poor relevance for macOS — nearly everything pointed at Windows binaries.

ChatGPT

Prompt: “How do I download firefox?”

Response is wrong on technical details, but starts with the correct official site, where users will likely self-correct. Roughly on par with Marginalia, but with a better end-state for an average user.

“Why do wider tires have better grip?”

A correct answer must be explain consistent empirical observations:

• A moderately wide tire is the baseline.
• Going wider within the wheel well helps late braking and lap times.
• Wider wet setups stop faster (with setup caveats), but aquaplane earlier.
• Wider wheels with the same tire help lap times.
• Wider tires on the same wheel do not help lap times.
• Smaller slip angles produce higher lateral force as tire pressure rises.
• Pressure cuts pile drive brake and grip effects non-linearly.

A shortlist of standard wrong or partial answers:

• “More surface area equals more friction” — wider tires don’t substantially increase patch area at sane pressures.
• “Friction = μN, area cancels; therefore width does nothing” — contradicted by braking and lap data.
• “It’s purely the softer compound” — samely-tired-but-wider-wheels already changes grip.
• “Contact patch shape works via [unspecified term]” — tautology without a mechanism, notably failing to explain braking gains.

Google

• 1. Knowledge card: “Bigger tires provide a wider contact area that optimizes performance,” then links to blogspam with demonstrably wrong winter-tire claims.
• 2–4. Question dropdowns with the correct broad-stroke yes/no but no reasoning in dropdown or destination page.
• 5–6. Answers sidestepping or bungling the “why” mechanics.
• 7: Says static friction is area-independent and that wider = no change — page then 502s, which may be for the best.
• 8. Answers about 20-inch wheels.
• 9. “race cars” answer that racers cover slippery spots by being wide.
• 10–49. A recursion: clicking a question dropdown yields more of them. Wrong or mismatched answers throughout.
• 50–54. r/cars thread; top vetted answer is scientifically off. One chain quotes a Brach Engineering PDF that is good but won’t answer the question.
• 55. SEO blogspam linking a video that doesn’t answer.
• 56. Ad farm with plagiarized incorrect text.

Bing

• 1. Knowledge card: “Larger contact patch” — factually wrong for typical inflation.
• 2–4. Carousel with repeats of non-answers from Google.
• 5. “This is why wider tires equals more grip” video, irrelevant.
• 6–10 [EXPLORE]. Same dead or unrelated links; several duplicate items 3–4.
• 11. Duplicate of 8.
• 12 [EXPLORE]. Ad farm funnel including an intercepting pop-over designed to emulate a real click.
• 13 [EXPLORE]. The source website for one of the videos; also no real answer.

Surface skim: mostly duplicates or wrong answers. None even approach the physical root cause.

Marginalia

The plain query returns zero results. Stripping out the “?” returns a single dead link — the same link as Bing’s #3.

Mwmbl

One result: an NYT story on differential interest rates paid by women. Removing the question mark surfaces a bike-tire winter article.

Kagi

1. Knowledge card repeating “greater contact patch” fallacy.
2. Same r/cars thread Google returned.
3. Unrelated to why, not to if.
4. YouTube “Do wider tyres give you more grip?” — 1:30 of ads, but the video focuses narrowly on tire-size effects and doesn’t mechanistically explain why wider works.
5. ChatGPT-style content with timestamp that allegedly dates from current navigation.
6. Approximately the ad farm blogspam returned by Google #56 or last Bing item.
7. Literal “last updated” server-side would calculate refresh. The wrong-nonse data saying wider does nothing.

ChatGPT

Generated ‘hallucinated reasons’. The response is grammatically fine, technically hollow. It faulters equally when prompted for more depth — similar to noted failures in foundational physics.

Existing engines, including the paid ones, produce no usable explanation, so plain wrongness balances out vs scam-free. Marginalia avoids the worst of it due to merely dying via zero results.

“Why do they keep making cpu transistors smaller?”

Naive phrasing rarely explains substrate physics. The academically helpful explanation — which requires voltage signal driving, near-transistor capacitance, wire delays — is very scantly present. In new engines’ favor of a thought exercise, one VLSI textbook would beat all engines. Test method’s only real interest here is how truly naive search terms surface background knowledge via indexing.

Google

• 1. Knowledge card on thermal efficiency — not an answer.
• 2. Pop-over-spam follows; prose doesn’t explain why.
• 3: Stack-level: discrete answer on physical capacitance — useful and correct, but only the tip.
• 4. SEO spam with no insight.
• 5. Best result avoids the easy hand-wave about cost/yield, giving a partial answer; third answer on same page gives capacitance basics.
• 6–10. Quora masked by ad walls and worth nothing.

Bing

• Mixes subsections. One late knowledge sub-reply resolves into a partial StackExchange explanation.
• More “people also ask” drifting toward adjacent subjects.

Kagi

• Wrong but well-answered question links.
• Some legit Stack/MetaFilter pieces surface MOSFET scaling generally without precision.

Marginalia and Mwmbl

Either zero results or comically irrelevant only-result (a Vox Holiday albums question).

ChatGPT

Issues vague “performance” boilerplate. Expanding does not fall into genuine device-physics reasoning, instead leaning on reduced-wire and duration fallacies previously applied with dubious confidence.

“vancouver snow forecast winter 2023”

Optimal answer, Environment Canada’s forecast — predicting below-normal snow with warm anomalies — is effectively absent from every engine.

Google

1. Knowledge card sourced from a snow-removal vendor that promisingly predicts ample snow. Farther down the article lies hidden fee-for-service copy to market its salting crews.

Bing

• 1–4. News carousels full of commercial content that exceed the text on the actual page.
• 5–8. SEO “forecast” cites statistically absurd 50% daily snowstorm probabilities under the chosen North Vancouver geometry; visually validates zero review is bad data.
• 9–14. Farmers Almanac and related pages with a BC map never refined to Vancouver level, all monetised.

Kagi

• 1–4. Same wrong “site”, plus not-even-BC duplicates.
• 5–7. Bad news-links.
• 8–13. Clean up from a content mill at heavy level.
• 14: A same scam forecast spotted by Google (#1).
• 15. Date-tag hallucination.

Marginalia, Mwmbl

Zero hits on Marginalia. Mwmbl returns Pennsylvania ice, old Vancouver outages from a previous year, of Oregon snowfront weather in Feb 2023; the last’s ads try to install a chrome extension with a false “savings blog” ploy… given how bad the state of local seasonal climate forecasting is, nothing to rate. No search engine delivers a direct Environment Canada winter outlook.

ChatGPT

Declines the question; recommends five forecast sources with no data. One query variant returns Environment Canada as a repository, along with links to irrelevant commercial outlets, closing the coverage but never with usable answer predictions.

What the knowledge card knows

Google's knowledge cards have long been a source of dubious information. Famously, "when was running invented" once returned a fabricated Quora answer crediting a nonexistent "Thomas Running" with the invention in 1748. To quantify the problem, I began collecting every knowledge card I encountered, intending to tabulate the fraction that were wrong. That spreadsheet never became a post, but a sampling of queries shows the pattern:

• "oc2 gemini length" (a canoe model) returned 20″ — the length of a baby mentioned in an article that also listed the boat's actual 24'7" length.
• "busy beaver number" returned a phone number: (604) 375-2754.
• "Feedly revenue" returned $5.2M/yr, sourced from a site that fabricates financial estimates for private companies.
• "What airlines fly direct from JFK to BLI?" returned plausible-sounding claims about Alaska and Delta flights that were demonstrably false, scraped from a page engineered to generate knowledge cards.
• "Air Canada Vancouver Newark" once showed a card for a flight (AC 7082) that hadn't existed for months.
• "TYR Hurricane Category 5 neoprene thickness" returned 1.5mm, which is incorrect.
• "Intel number of engineers" returned a phone number, (604) 742-3501.
• "boston up118s dimensions" (a piano) returned dimensions in the millions of inches — over 90 miles long.
• "number of competitive checkers players" returned 2.
• "fraser river current speed" returned an implausible 60–80 mph.
• "futura c-4 surfski weight" returned 39 pounds — the weight of a different model mentioned on the same page.

When I was actively tracking these, the majority of knowledge cards I saw were incorrect.

Common responses, and why they miss

A frequent reply to any critique of search quality is that the analysis isn't a peer-reviewed study. My median blog post is more rigorous than my median paper — including one that won a best paper award at a top conference in information retrieval. On a blog, there's no co-author pressure to include flattering but false material; I can report results that are accurate even when they're less interesting. I pushed hard to make that award-winning paper more rigorous too, but lost enough of those fights that it contains issues I wouldn't tolerate in a blog post.

A second recurring objection — usually from Kagi users, specifically — is that my results table is wrong because the same queries returned good results when they tried them. No one has yet reported good results for the tire, transistor, or snow queries, nor do the other queries fare well without an ad blocker. In one telling case, a user insisted all results were good, but had pinned GitHub results and had only run queries where GitHub was the right answer. That's a worse outcome than Google or Bing with well-formed queries, since you'll get noise when GitHub isn't the right place to search. Much like enthusiasts of certain consumer tech brands, some Kagi users respond to criticism with nonsensical defenses — a sign the company has tapped into the same loyalty vein.

Notes on expert blind spots

Experts often underestimate how much tacit knowledge goes into "good" search habits. A common suggestion is to append site:reddit.com to queries, but that makes many searches worse — you need a mental model of when it helps and when it hurts. One observer noted that Russian émigrés in Cape Town used Instagram to find products and services by searching for something, clicking around, and letting the algorithm refine recommendations over days. Naive users don't have the expert's intuition for which random clicks might be productive; they tend to click the wrong ones — including ads that install malware, a problem prevalent enough that many users are on devices compromised by clicking a malicious search result or ad.

The quirks run deep. Accidentally prepending User\n to a ChatGPT query turned bad results into good ones, much as ChatGPT once believed Colin Percival was dead when asked to "write about" him but alive when asked to "Write about" him. Twitter search, meanwhile, post-acquisition often fails to return tweets even on exact substring matches, forcing resorts to convoluted searches that surface a link to the desired tweet.

---

Footnotes: Claims that low-quality results stem from LLM-generated garbage don't hold up — most failed results are from pages pre-dating LLMs, and the problem isn't confined to tech topics. One Portland musician searching "[band name] members" got a knowledge card describing a entirely different UK band. Expert search tricks also don't transfer well: searching HN for a YouTube downloader returns reasonable results, but non-experts rarely have such strategies in their toolkit.