Discussions

Ask a Question
Back to all

Using GitHub Tools to Sync Blocklists Automatically? Key Considerations & Caveats

In this discussion, the QA team often recommends a GitHub tool for automatically syncing blocklists or allowlists. The reason is that for some reason they don’t natively support certain lists many customers want, such as Hagezi Most Abused TLD or badware host.

I personally would recommend using ControlD Sync with that tool. The blocklist from badware.host can be quite useful, but if you automatically sync Hagezi's "Most Abused TLDs" list, there are a few things worth considering. If you also sync the allow list for "Most Abused TLDs" some domains like porn and URL shorteners are whitelisted and certain entries still redirect to well-known phishing or scam websites. Even though these domains appear in filters specifically meant to block malware scams or URL shorteners they will still always stay allowed no matter what.

I filtered out several domains and excluded them. I created a similar private project that only syncs newly added entries or removes ones that are no longer present. It also have automated backups and handles a few extra tasks.
Here is the list of domains I excluded. These are mostly porn sites, URL shorteners, file-sharing services or they have a bad reputation:

anonimayzer[.]site
anonymurl[.]biz
autoads[.]asia
cpolar[.]top
datapump[.]cc
debloat[.]win
discord[.]gift
edgeone[.]site
fap[.]bar
filepress[.]ink
fireshinegames[.]online
fmhy[.]lol
geekchic[.]shop
gimmeporn[.]biz
host[.]church
ishortn[.]ink
jpg6[.]su
libstc[.]cc
lurl[.]cc
mrprice[.]online
myppt[.]cc
nest[.]rip
nexdrive[.]lol
pahe[.]plus
peerflix[.]mov
qwertycxz[.]top
rdl[.]ink
reddit-doesnt-like-this[.]site
save[.]moe
sci-hub[.]red
shrinkme[.]ink
simple[.]ink
strem[.]bar
strims[.]top
tmdb[.]club
transfer[.]zip
trdiziizle[.]vip
tux[.]pizza
ultracloud[.]cc
vlo[.]name
work[.]ink