Show HN: 4B+ DNS Records Dataset
merklemap.comHi HN,
I've been working on building a pipeline to create a DNS records database lately. The goal is to enable research as well as competitive landscape analysis on the internet.
The dataset for now spans around 4 billion records and covers all the common DNS record types:
A
AAAA
ANAME
CAA
CNAME
HINFO
HTTPS
MX
NAPTR
NS
PTR
SOA
SRV
SSHFP
SVCB
TLSA
TXT
Each line in the CSV file represents a single DNS record in the following format:
www.example.com,A,93.184.215.14Let me know if you have any questions or feedback!
Neat! How is this different than domaintools/farsight [1]?
Passive DNS [2] has been in my toolbox for 15+ years, and is invaluable for security research / threat intelligence. Knowing historical resolutions to something are so helpful in investigations.
For anyone interested, they should check out the talk by one of the DomainTools people [3] on how it can be utilized for investigation.
Are you passively collecting this data, or actively querying for these records?
[1] - https://www.domaintools.com/products/threat-intelligence-fee...
[2] - https://www.circl.lu/services/passive-dns/
[3] - https://www.youtube.com/watch?v=oXmapqLkZd0
From what I understand [1] is just tlds, not subdomains?
That would be incorrect, they get subdomains for passive dns feeds.
Ok, it'd be interesting to know how big is their datasets compared to mine and how much they overlap.
Any possibility of adding (first seen, last seen) time stamps? There is basically no good way to reconstruct the state of e.g. SPF at a point in time from existing DNS data sets
I could in future releases, yes.
How often is it updated?
Does it include expired domains?
> How often is it updated?
I plan to do 2 releases a month for now, goal is one a day.
> Does it include expired domains?
Yes.