Question

我正在撰写一些分析注册域名并寻找趋势的软件。我正在尝试一些机器学习，以帮助预测将来根据注册的域名类型购买哪些域名。

我一直在寻找一种方法来下载存在的“所有”注册域名，但我还没有找到办法。

我很容易使用whois命令行工具查询单个域名，例如：

$ whois google.com
   Domain Name: GOOGLE.COM
   Registry Domain ID: 2138514_DOMAIN_COM-VRSN
   Registrar WHOIS Server: whois.markmonitor.com
   Registrar URL: http://www.markmonitor.com
   Updated Date: 2018-02-21T18:36:40Z
   Creation Date: 1997-09-15T04:00:00Z
   Registry Expiry Date: 2020-09-14T04:00:00Z
   Registrar: MarkMonitor Inc.
   Registrar IANA ID: 292
   Registrar Abuse Contact Email: abusecomplaints@markmonitor.com
   Registrar Abuse Contact Phone: +1.2083895740
   Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited
   Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
   Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited
   Domain Status: serverDeleteProhibited https://icann.org/epp#serverDeleteProhibited
   Domain Status: serverTransferProhibited https://icann.org/epp#serverTransferProhibited
   Domain Status: serverUpdateProhibited https://icann.org/epp#serverUpdateProhibited
   Name Server: NS1.GOOGLE.COM
   Name Server: NS2.GOOGLE.COM
   Name Server: NS3.GOOGLE.COM
   Name Server: NS4.GOOGLE.COM
   DNSSEC: unsigned
   URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/
>>> Last update of whois database: 2018-03-20T03:16:59Z <<<

For more information on Whois status codes, please visit https://icann.org/epp

NOTICE: The expiration date displayed in this record is the date the
registrar's sponsorship of the domain name registration in the registry is
currently set to expire. This date does not necessarily reflect the expiration
date of the domain name registrant's agreement with the sponsoring
registrar.  Users may consult the sponsoring registrar's Whois database to
view the registrar's reported date of expiration for this registration.

TERMS OF USE: You are not authorized to access or query our Whois
database through the use of electronic processes that are high-volume and
automated except as reasonably necessary to register domain names or
modify existing registrations; the Data in VeriSign Global Registry
Services' ("VeriSign") Whois database is provided by VeriSign for
information purposes only, and to assist persons in obtaining information
about or related to a domain name registration record. VeriSign does not
guarantee its accuracy. By submitting a Whois query, you agree to abide
by the following terms of use: You agree that you may use this Data only
for lawful purposes and that under no circumstances will you use this Data
to: (1) allow, enable, or otherwise support the transmission of mass
unsolicited, commercial advertising or solicitations via e-mail, telephone,
or facsimile; or (2) enable high volume, automated, electronic processes
that apply to VeriSign (or its computer systems). The compilation,
repackaging, dissemination or other use of this Data is expressly
prohibited without the prior written consent of VeriSign. You agree not to
use electronic processes that are automated and high-volume to access or
query the Whois database except as reasonably necessary to register
domain names or modify existing registrations. VeriSign reserves the right
to restrict your access to the Whois database in its sole discretion to ensure
operational stability.  VeriSign may restrict or terminate your access to the
Whois database for failure to abide by these terms of use. VeriSign
reserves the right to modify these terms at any time.

The Registry database contains ONLY .COM, .NET, .EDU domains and
Registrars.
Domain Name: google.com
Registry Domain ID: 2138514_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.markmonitor.com
Registrar URL: http://www.markmonitor.com
Updated Date: 2018-02-21T10:45:07-0800
Creation Date: 1997-09-15T00:00:00-0700
Registrar Registration Expiration Date: 2020-09-13T21:00:00-0700
Registrar: MarkMonitor, Inc.
Registrar IANA ID: 292
Registrar Abuse Contact Email: abusecomplaints@markmonitor.com
Registrar Abuse Contact Phone: +1.2083895740
Domain Status: clientUpdateProhibited (https://www.icann.org/epp#clientUpdateProhibited)
Domain Status: clientTransferProhibited (https://www.icann.org/epp#clientTransferProhibited)
Domain Status: clientDeleteProhibited (https://www.icann.org/epp#clientDeleteProhibited)
Domain Status: serverUpdateProhibited (https://www.icann.org/epp#serverUpdateProhibited)
Domain Status: serverTransferProhibited (https://www.icann.org/epp#serverTransferProhibited)
Domain Status: serverDeleteProhibited (https://www.icann.org/epp#serverDeleteProhibited)
Registry Registrant ID: 
Registrant Name: Domain Administrator
Registrant Organization: Google LLC
Registrant Street: 1600 Amphitheatre Parkway, 
Registrant City: Mountain View
Registrant State/Province: CA
Registrant Postal Code: 94043
Registrant Country: US
Registrant Phone: +1.6502530000
Registrant Phone Ext: 
Registrant Fax: +1.6502530001
Registrant Fax Ext: 
Registrant Email: dns-admin@google.com
Registry Admin ID: 
Admin Name: Domain Administrator
Admin Organization: Google LLC
Admin Street: 1600 Amphitheatre Parkway, 
Admin City: Mountain View
Admin State/Province: CA
Admin Postal Code: 94043
Admin Country: US
Admin Phone: +1.6502530000
Admin Phone Ext: 
Admin Fax: +1.6502530001
Admin Fax Ext: 
Admin Email: dns-admin@google.com
Registry Tech ID: 
Tech Name: Domain Administrator
Tech Organization: Google LLC
Tech Street: 1600 Amphitheatre Parkway, 
Tech City: Mountain View
Tech State/Province: CA
Tech Postal Code: 94043
Tech Country: US
Tech Phone: +1.6502530000
Tech Phone Ext: 
Tech Fax: +1.6502530001
Tech Fax Ext: 
Tech Email: dns-admin@google.com
Name Server: ns1.google.com
Name Server: ns4.google.com
Name Server: ns2.google.com
Name Server: ns3.google.com
DNSSEC: unsigned
URL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/
>>> Last update of WHOIS database: 2018-03-19T20:13:36-0700 <<<

The Data in MarkMonitor.com's WHOIS database is provided by MarkMonitor.com for
information purposes, and to assist persons in obtaining information about or
related to a domain name registration record.  MarkMonitor.com does not guarantee
its accuracy.  By submitting a WHOIS query, you agree that you will use this Data
only for lawful purposes and that, under no circumstances will you use this Data to:
 (1) allow, enable, or otherwise support the transmission of mass unsolicited,
     commercial advertising or solicitations via e-mail (spam); or
 (2) enable high volume, automated, electronic processes that apply to
     MarkMonitor.com (or its systems).
MarkMonitor.com reserves the right to modify these terms at any time.
By submitting this query, you agree to abide by this policy.

MarkMonitor is the Global Leader in Online Brand Protection.

MarkMonitor Domain Management(TM)
MarkMonitor Brand Protection(TM)
MarkMonitor AntiPiracy(TM)
MarkMonitor AntiFraud(TM)
Professional and Managed Services

Visit MarkMonitor at http://www.markmonitor.com
Contact us at +1.8007459229
In Europe, at +44.02032062220

For more information on Whois status codes, please visit
 https://www.icann.org/resources/pages/epp-status-codes-2014-06-16-en
--

WHOIS数据包含我需要的所有内容，但我找不到下载所有当前注册域名的WHOIS数据的方法。

我有什么方法可以获得这些数据吗？我觉得必须在某处公开提供，因为whois CLI工具可以轻松查询信息。

我在这里缺少什么？

Answer 1

TL; DR：你不能（下载所有“whois”数据）。

（初步说明：“whois数据”，虽然经常使用有点不正确。你使用whois协议与whois客户端查询注册表中的whois服务器，更具体地说，这里是一个域名注册表，存储有关其赞助的域名的联系数据。出于同样的原因，没有“whois数据库”。）

现在是一个漫长的悲伤故事：

由于许多明显的技术和非技术原因，这是不可能的。如果您认为whois CLI命令很简单，那么您深错误（请参阅我的其他答案：https://unix.stackexchange.com/a/407030/211833了解有关该点的详细信息）

首先，您的问题对所有TLD都没有意义。您至少要将ccTLD与gTLD分开。

1） ccTLD

ccTLD通常对个人数据的隐私有更严格的规定，对于正在进行的欧洲法规（如GDPR），这应该更为严格。基本上他们中的一些人已经禁止访问没有个人数据的完整域名列表（通常称为“区域文件”），因此您无法访问所有内容和个人数据。您可以尝试接近一些并询问是否有任何可能的研究研究，但我怀疑您是否会成功，您将需要单独处理每个ccTLD注册管理机构，因为它们各自处理自己的内容（域中的所有数据）他们管理的顶级域名中的名称）

2） gTLD

对于他们来说，情况完全不同。

首先，由于事情在默认情况下更加自由（不保护个人数据），您会看到许多注册商/公司提供代理/隐私服务，这意味着即使在whois查询输出中您也不会看到太多有用的数据。

但仍然由于GDPR和同化，事情正在发生变化。例如，在godaddy.com上做一个whois，并查看所有这些明星的联系人姓名和电子邮件，因此需要访问网站。

但是，注册商和注册管理机构与ICANN签订了合同。这意味着他们都有一些要求，而且他们是统一的。

首先，所有注册管理机构都被授权提供对其区域文件的访问权限。它通常通过CZDA完成，您可以在其中找到有关ICANN网站的详细信息。请注意，它实际上是所有域名的列表，而不是所有域名的列表，因为您可以注册域名而不是在DNS上显示域名。

至于联系人数据，即whois中可见的其他信息，还有其他一些未知的点。请参阅https://www.icann.org/resources/pages/approved-with-specs-2013-09-17-en上的注册商协议，特别是第3.3.6节，其中提供了对注册商“whois”数据的批量访问。注意它是如何与一些钱（10 000美元）挂钩，并且对你可以用它做什么有各种限制。请记住，您需要每个注册商，因此在gTLD世界中，超过1000个。

注册管理机构协议中没有关于公共批量访问的等效条款（请参阅https://newgtlds.icann.org/sites/default/files/agreements/agreement-approved-31jul17-en.html）。

事情变得复杂，因为在今天和几个月之前，.COM/.NET仍然是一个薄的注册表，没有在注册表级别存储的联系人数据，仅在注册商处。

由于新规定以及新协议RDAP计划在某一时刻取代whois，所有上述内容将在未来几个月/几年内发生变化。 RDAP将允许更高级别的访问权限和返回的数据量。

当然，在上述所有情况下，从技术上讲，没有人禁止任何人定期进行whois查询并在本地存储结果。正如您在whois输出中看到的那样，您对数据的使用受到各种限制的约束，并且批量查询whois服务器始终会使您面临被列入黑名单或至少严重限制的风险。请注意，对于输入（查询whois服务器的名称），很容易从区域文件开始，甚至跨TLD（如果site.example存在，即使您没有site.test也可以尝试.test {{1}} zonefile），或搜索引擎查询或词典等

多家公司这样做并提供搜索数据的工具，例如进行反向查询等。也许有些人可以为您提供批量结果，但肯定不是免费的。

Answer 2

您可以使用 WhoisFreaks 数据库获取数百万个注册域（活动和非活动）的 Whois 记录。 https://whoisfreaks.com/。它提供解析良好的 Whois 域信息，例如域注册详细信息、域注册商详细信息、注册人详细信息、管理联系人、技术联系人、服务器名称、域状态、注册数据等。

WhoisFreaks 数据库是拥有 whois 的最大的 whois 数据库之一 1000 多个 TLD（gTLD 和 ccTLD）、4.22 亿多个跟踪域的信息，超过 5.55 亿个 whois 记录。
经过良好解析和规范化的 whois 数据记录，可在 CSV 文件格式，易于阅读并集成到任何业务系统中。
准确且更新，WhoisFreaks 数据库在每天一次。
WhoisFreaks 数据库提供 1986 年的 whois 记录。
自定义 whois 域数据也可基于注册人特定域、国家特定域和 TLD 特定域。
每月注册的域名超过 450 万个。
WhoisFreaks 数据库只包含唯一的 Whois 记录域。

WhoisFreaks JSON 响应如下所示：

{
    "status": true,
    "domain_name": "google.com",
    "query_time": "2021-03-12 08:03:53",
    "whois_server": "whois.markmonitor.com",
    "domain_registered": "yes",
    "create_date": "1997-09-15",
    "update_date": "2019-09-09",
    "expiry_date": "2028-09-13",
    "domain_registrar": {
        "iana_id": "292",
        "registrar_name": "MarkMonitor, Inc.",
        "whois_server": "whois.markmonitor.com",
        "website_url": "http://www.markmonitor.com",
        "email_address": "abusecomplaints@markmonitor.com",
        "phone_number": "+1.2083895770"
    },
    "registrant_contact": {
        "company": "Google LLC",
        "state": "CA",
        "country_name": "United States",
        "country_code": "US",
        "email_address": "Select Request Email Form at https://domains.markmonitor.com/whois/google.com"
    },
    "administrative_contact": {
        "company": "Google LLC",
        "state": "CA",
        "country_name": "United States",
        "country_code": "US",
        "email_address": "Select Request Email Form at https://domains.markmonitor.com/whois/google.com"
    },
    "technical_contact": {
        "company": "Google LLC",
        "state": "CA",
        "country_name": "United States",
        "country_code": "US",
        "email_address": "Select Request Email Form at https://domains.markmonitor.com/whois/google.com"
    },
    "name_servers": [
        "ns1.google.com",
        "ns2.google.com",
        "ns3.google.com",
        "ns4.google.com"
    ],
    "domain_status": [
        "clientTransferProhibited",
        "clientDeleteProhibited",
        "serverDeleteProhibited",
        "serverTransferProhibited",
        "serverUpdateProhibited",
        "clientUpdateProhibited"
    ],
    "whois_raw_domain": "\nDomain Name: google.com\nRegistry Domain ID: 2138514_DOMAIN_COM-VRSN\nRegistrar WHOIS Server: whois.markmonitor.com\nRegistrar URL: http://www.markmonitor.com\nUpdated Date: 2019-09-09T08:39:04-0700\nCreation Date: 1997-09-15T00:00:00-0700\nRegistrar Registration Expiration Date: 2028-09-13T00:00:00-0700\nRegistrar: MarkMonitor, Inc.\nRegistrar IANA ID: 292\nRegistrar Abuse Contact Email: abusecomplaints@markmonitor.com\nRegistrar Abuse Contact Phone: +1.2083895770\nDomain Status: clientUpdateProhibited (https://www.icann.org/epp#clientUpdateProhibited)\nDomain Status: clientTransferProhibited (https://www.icann.org/epp#clientTransferProhibited)\nDomain Status: clientDeleteProhibited (https://www.icann.org/epp#clientDeleteProhibited)\nDomain Status: serverUpdateProhibited (https://www.icann.org/epp#serverUpdateProhibited)\nDomain Status: serverTransferProhibited (https://www.icann.org/epp#serverTransferProhibited)\nDomain Status: serverDeleteProhibited (https://www.icann.org/epp#serverDeleteProhibited)\nRegistrant Organization: Google LLC\nRegistrant State/Province: CA\nRegistrant Country: US\nRegistrant Email: Select Request Email Form at https://domains.markmonitor.com/whois/google.com\nAdmin Organization: Google LLC\nAdmin State/Province: CA\nAdmin Country: US\nAdmin Email: Select Request Email Form at https://domains.markmonitor.com/whois/google.com\nTech Organization: Google LLC\nTech State/Province: CA\nTech Country: US\nTech Email: Select Request Email Form at https://domains.markmonitor.com/whois/google.com\nName Server: ns3.google.com\nName Server: ns2.google.com\nName Server: ns1.google.com\nName Server: ns4.google.com\nDNSSEC: unsigned\nURL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/\n>>> Last update of WHOIS database: 2021-03-11T22:57:36-0800 <<<\n\nFor more information on WHOIS status codes, please visit:\n  https://www.icann.org/resources/pages/epp-status-codes\n\nIf you wish to contact this domainâs Registrant, Administrative, or Technical\ncontact, and such email address is not visible above, you may do so via our web\nform, pursuant to ICANNâs Temporary Specification. To verify that you are not a\nrobot, please enter your email address to receive a link to a page that\nfacilitates email communication with the relevant contact(s).\n\nWeb-based WHOIS:\n  https://domains.markmonitor.com/whois\n\nIf you have a legitimate interest in viewing the non-public WHOIS details, send\nyour request and the reasons for your request to whoisrequest@markmonitor.com\nand specify the domain name in the subject line. We will review that request and\nmay ask for supporting documentation and explanation.\n\nThe data in MarkMonitorâs WHOIS database is provided for information purposes,\nand to assist persons in obtaining information about or related to a domain\nnameâs registration record. While MarkMonitor believes the data to be accurate,\nthe data is provided \"as is\" with no guarantee or warranties regarding its\naccuracy.\n\nBy submitting a WHOIS query, you agree that you will use this data only for\nlawful purposes and that, under no circumstances will you use this data to:\n  (1) allow, enable, or otherwise support the transmission by email, telephone,\nor facsimile of mass, unsolicited, commercial advertising, or spam; or\n  (2) enable high volume, automated, or electronic processes that send queries,\ndata, or email to MarkMonitor (or its systems) or the domain name contacts (or\nits systems).\n\nMarkMonitor reserves the right to modify these terms at any time.\n\nBy submitting this query, you agree to abide by this policy.\n\nMarkMonitor Domain Management(TM)\nProtecting companies and consumers in a digital world.\n\nVisit MarkMonitor at https://www.markmonitor.com\nContact us at +1.8007459229\nIn Europe, at +44.02032062220\n--",
    "registry_data": {
        "domain_name": "GOOGLE.COM",
        "query_time": "2021-03-12 08:03:52",
        "whois_server": "whois.verisign-grs.com",
        "domain_registered": "yes",
        "create_date": "1997-09-15",
        "update_date": "2019-09-09",
        "expiry_date": "2028-09-14",
        "domain_registrar": {
            "iana_id": "292",
            "registrar_name": "MarkMonitor Inc.",
            "whois_server": "whois.markmonitor.com",
            "website_url": "http://www.markmonitor.com",
            "email_address": "abusecomplaints@markmonitor.com",
            "phone_number": "+1.2083895740"
        },
        "name_servers": [
            "NS2.GOOGLE.COM",
            "NS1.GOOGLE.COM",
            "NS4.GOOGLE.COM",
            "NS3.GOOGLE.COM"
        ],
        "domain_status": [
            "clientDeleteProhibited",
            "clientTransferProhibited",
            "serverDeleteProhibited",
            "serverTransferProhibited",
            "serverUpdateProhibited",
            "clientUpdateProhibited"
        ],
        "whois_raw_registery": "\n   Domain Name: GOOGLE.COM\n   Registry Domain ID: 2138514_DOMAIN_COM-VRSN\n   Registrar WHOIS Server: whois.markmonitor.com\n   Registrar URL: http://www.markmonitor.com\n   Updated Date: 2019-09-09T15:39:04Z\n   Creation Date: 1997-09-15T04:00:00Z\n   Registry Expiry Date: 2028-09-14T04:00:00Z\n   Registrar: MarkMonitor Inc.\n   Registrar IANA ID: 292\n   Registrar Abuse Contact Email: abusecomplaints@markmonitor.com\n   Registrar Abuse Contact Phone: +1.2083895740\n   Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited\n   Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited\n   Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited\n   Domain Status: serverDeleteProhibited https://icann.org/epp#serverDeleteProhibited\n   Domain Status: serverTransferProhibited https://icann.org/epp#serverTransferProhibited\n   Domain Status: serverUpdateProhibited https://icann.org/epp#serverUpdateProhibited\n   Name Server: NS1.GOOGLE.COM\n   Name Server: NS2.GOOGLE.COM\n   Name Server: NS3.GOOGLE.COM\n   Name Server: NS4.GOOGLE.COM\n   DNSSEC: unsigned\n   URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/\n>>> Last update of whois database: 2021-03-12T07:03:38Z <<<\n\nFor more information on Whois status codes, please visit https://icann.org/epp\n\nNOTICE: The expiration date displayed in this record is the date the\nregistrar's sponsorship of the domain name registration in the registry is\ncurrently set to expire. This date does not necessarily reflect the expiration\ndate of the domain name registrant's agreement with the sponsoring\nregistrar.  Users may consult the sponsoring registrar's Whois database to\nview the registrar's reported date of expiration for this registration.\n\nTERMS OF USE: You are not authorized to access or query our Whois\ndatabase through the use of electronic processes that are high-volume and\nautomated except as reasonably necessary to register domain names or\nmodify existing registrations; the Data in VeriSign Global Registry\nServices' (\"VeriSign\") Whois database is provided by VeriSign for\ninformation purposes only, and to assist persons in obtaining information\nabout or related to a domain name registration record. VeriSign does not\nguarantee its accuracy. By submitting a Whois query, you agree to abide\nby the following terms of use: You agree that you may use this Data only\nfor lawful purposes and that under no circumstances will you use this Data\nto: (1) allow, enable, or otherwise support the transmission of mass\nunsolicited, commercial advertising or solicitations via e-mail, telephone,\nor facsimile; or (2) enable high volume, automated, electronic processes\nthat apply to VeriSign (or its computer systems). The compilation,\nrepackaging, dissemination or other use of this Data is expressly\nprohibited without the prior written consent of VeriSign. You agree not to\nuse electronic processes that are automated and high-volume to access or\nquery the Whois database except as reasonably necessary to register\ndomain names or modify existing registrations. VeriSign reserves the right\nto restrict your access to the Whois database in its sole discretion to ensure\noperational stability.  VeriSign may restrict or terminate your access to the\nWhois database for failure to abide by these terms of use. VeriSign\nreserves the right to modify these terms at any time.\n\nThe Registry database contains ONLY .COM, .NET, .EDU domains and\nRegistrars."
    }
}

我希望这将是帮助您购买域名的最佳解决方案。

Answer 3

此网站允许下载注册域名列表：https://networksdb.io。

有些是免费的，有些是付费的。它还允许访问IP块的WHOIS数据汇总，以查找哪些公司拥有哪些地址块，反之亦然。

如何下载所有域WHOIS数据？

3 个答案: