Show HN: A blocklist to remove spam and bad websites from search results
57 comments
·January 14, 2025Ringz
Installed! This should not be a function of the search engine nor a plugin. This should be integrated in the browser.
Another great function (not for this plugin) should be the option to "bundle" all search results from the same domain. Stuff them under one collapsible entry. I hate going through lists and pages of apple/google/synology/sonos/crab urls when I already know that I have to search somewhere else.
gtfiorentino
Hi @popcar2 — how are you sourcing the domains for the blocklist? We'd like to evaluate those domains and consider whether they should be removed from DuckDuckGo as spam. You can also report a site directly in the search results by clicking the three-dot menu next to the link and selecting "Share Feedback about this Site".
popcar2
Hi! I'm mostly going through them manually. Not all of the domains in the list are literally spam - most of the list also includes misleading sites like corporate blogs that trick people into downloading their software.
You might be interested in the AI spam/low effort section though, one that tops DDG often are these AI generated tech articles: https://github.com/popcar2/BadWebsiteBlocklist/issues/1
They're the same site under different domains, you can tell it's AI by its writing style, how much they churn out per day, how little info there is about who's writing it, how similarly the about pages are written, and how the same article is suspiciously also in similar-sounding sites.
Another one I just caught today that was on top of page 1: https://github.com/popcar2/BadWebsiteBlocklist/issues/84
I'll be sure to report these sites as I'm adding to the list, thanks.
gtfiorentino
Got it, that's good to know, thanks! I've added the domains in the AI spam / low effort section to our list of user reports for review.
james-bcn
With the Kagi search engine is a way in the settings to bulk-upload lists of domains to block (or upvote) them. Has anyone uploaded a list like this to it?
I may do that.
freedomben
That was my thought as well. Their UI is great for one-at-time operations, but an API endpoint I could curl and sync with a local file I keep in git would be killer.
Although, using this via the extension would make it cross-platform so the block affects kagi and google, which could be nice.
Although, that would require manual syncing between devices, which would not be nice.
Although, uploading it to kagi through API doesn't mean I have to not use the extension, so having the cake and eating it too may be possible.
thoughtpalette
Was thinking of that as I was browsing this doc! I just did the ole' reddit.com -> old.reddit.com redirect via kagi yesterday.
cormorant
I'm fed up too. Spammy, AI-looking sites are showing up more and more. For some reason, many of them use the same Wordpress theme with a light gray table of contents - they look like this: https://imgur.com/a/totally-not-ai-generated-efsumgZ
The problem seems worse on "alternative" search engines, e.g. DuckDuckGo and Kagi, which both use Bing. It's been driving me back to Google.
A blocklist seems like a losing proposition, unless, like adblock filter lists, it balloons to tens of thousands of entries and gets updated constantly.
Unfortunately, this kind of blocklist is highly subjective. This list blocks MSN.com! That's hardly what I would have chosen.
popcar2
Even Google is plagued by spam, I've tried all sorts of search techniques and alternative engines but I feel like the only solution seems to be doing things manually. I was already starting to block things by myself but I thought it'd be more productive to make the list public and try crowdsourcing. Even now, searching "how to partition a hard disk" would often drive you to low-effort sites telling you to use their software.
> Unfortunately, this kind of blocklist is highly subjective. This list blocks MSN.com! That's hardly what I would have chosen.
It's definitely a bit opinionated, but it's open to discussion - you can create an unblock request issue (if you care enough to do so, of course!). The reason I blocked MSN is that it just re-hosts articles from other websites, so I'd rather see the official source than be tricked into Microsoft's site which is very annoying, like how it opens another article if you scroll too fast down.
rendaw
I get tons when looking up recipes and cooking related information. Things that will say "X can be refrigerated for up to two weeks" then in the next paragraph "X is fine to refrigerate and eat for 2-3 days" or similar.
I'd block them but there seem to be infinite. They're probably buying 10+ character domains using random words/names/phrases in bulk.
radicality
Afaik DDG is just Bing, whereas Kagi is using Google, Bing, (Yandex?) among others - https://help.kagi.com/kagi/search-details/search-sources.htm...
As a Kagi user I actually haven’t encountered much search result spam, surprised you’re seeing enough there to drive you back to Google!
nosioptar
You can use ublacklist without a list and just block shit sites as you see them.
I'm loving being able to search for something without getting results from garbage sites like howtogeek, stackoverflow, MSN, Pinterest, etc.
shortformblog
The problem with a list like this is that a “bad website” is in the eye of the beholder. I’m not saying that there’s anything wrong with you personally not liking the Shopify or the Semrush blog. But I think that everyone else has their own calculus.
It’s the same reason why social media blocklists can be problematic—everyone’s calculus is different.
My suggestion is that you promote it as a starter and suggest that users fork it for their own needs.
swayvil
Some kind of democratic process. Where membership and blacklist are both something arrived at democratically.
It could be simple.
Good?
shortformblog
Seems like a kit that can be personalized across broad categories might be a better bet. By putting the onus on one list you don’t solve the main problem, which is that the list might block things you’re fine with.
edm0nd
I recently started a crypto scam/phishing blocklist if you wanna roll these into your list as well.
also works well with Pi-hole and other platforms.
https://github.com/spmedia/Crypto-Scam-and-Crypto-Phishing-T...
MortyWaves
Who’s going to be the first to make the PR for Medium and “dev.to”?
Night_Thastus
I've been using GoogleHitHider, which also works on other search engines like DDG. Worked well for many years. It's a list I curated myself though for personal use, I definitely wouldn't mind seeing what other people had.
mrweasel
I love that it just includes all of msn.com.
mrbluecoat
I guess the author's "boiling hatred to bad tech support articles" leads to some overreach
popcar2
This doesn't block you from visiting MSN, but it does stop their articles from appearing in search. The reason is that MSN just re-hosts articles from other sites rather than provide anything of value. MSN posts often outrank their original source because Microsoft is pushing it hard on Windows/Bing/Edge.
For example: https://www.msn.com/en-us/movies/news/jodie-foster-heckled-a... is just a re-hosted version https://www.independent.co.uk/arts-entertainment/tv/news/jod...
My hope in hiding MSN is to allow the original sources to rise back up to the top.
roskelld
I'm going to have a look at this. I currently run a script that adds `-site:msn.com` to all of my DDG searches. It's kinda ugly.
qingcharles
My small rebuttal to that is that msn.com occasionally has articles they've sucked in that are paywalled on the original sites.
But I have archive.is for the most part to get around that issue.
qingcharles
msn.com is actually useful from time-to-time as they have syndicated articles which are otherwise stuck behind paywalls on other sites.
nayuki
Related: Freya Holmér - "Generative AI is a Parasitic Cancer" https://www.youtube.com/watch?v=-opBifFfsMY (1h19m54s) [2025-01-02].
She talks at length about how pages of AI-generated nonsense text are cluttering search results on Google and all other search engines.
the_snooze
This is one of those features a proper search engine (i.e., not a thinly-veiled advertising network) should have. If users can customize their search results and share their sorting/filtering methods, then that presents a large number of constantly-moving targets that greatly drives up the cost of SEO. There's no "making the Google algorithm happy." Instead, it becomes more "making the users happy."
bityard
Google used to do this years ago but clawed it back around the time they started removing _all_ customizations under the premise of, "we know how to customize your results better than you do."
DuckDuckGo has site blocking. The problem is that there are so many SEO-optimized blogspam, referral link, and other "garbage" sites that you could spend a lifetime blocking each one individually before you get any actual work done. And it's only getting worse now that LLMs can generate a whole web site for you in a matter of minutes. I imagine a dedicated individual could provision several thousand websites/blogs per day, just chock full of ads and referral links.
noleary
This is cool! Not entirely sure whether I think it's a good idea, but I wonder if it'd be useful to come up with a way to tranche websites.
Some sites are complete garbage and should be blocked, for course. Others (e.g., in my experience, Quora) are sometimes quite good and sometimes quite bad. Wouldn't be my first choice, but I've found them useful at times.
For a given search, maybe you try with the most aggressive blocking / filtering. If you fail to find what you're looking for, maybe soften the restriction a bit.
Maybe this is overwrought...
Hi HN!
I've been fed up with search results so much that I decided to make a giant blocklist to remove garbage links by using uBlacklist.
I browsed other blocklists and wasn't very satisfied from what exists now; the goal of this one is to be super organized and transparent, explaining why each site was blocked via issues. Contributions welcome!
Even though around 100 domains are blocked so far, I already noticed a big improvement in casual searches. You'd be surprised how some AI generated websites can dominate the #1 page on DuckDuckGo.