There are many reasons you could possibly will need to uncover all the URLs on a website, but your exact goal will figure out Everything you’re trying to find. As an example, you may want to:
Determine each indexed URL to analyze issues like cannibalization or index bloat
Collect present-day and historic URLs Google has found, especially for web page migrations
Come across all 404 URLs to Get well from submit-migration errors
In each state of affairs, an individual Instrument won’t Provide you all the things you require. However, Google Search Console isn’t exhaustive, in addition to a “web-site:instance.com” search is proscribed and tough to extract information from.
During this article, I’ll walk you thru some instruments to construct your URL record and ahead of deduplicating the data employing a spreadsheet or Jupyter Notebook, dependant upon your website’s sizing.
Previous sitemaps and crawl exports
If you’re in search of URLs that disappeared from your live web-site not too long ago, there’s an opportunity someone on your own team could have saved a sitemap file or a crawl export prior to the variations had been built. In the event you haven’t by now, look for these information; they will often give what you would like. But, in the event you’re looking at this, you almost certainly did not get so lucky.
Archive.org
Archive.org
Archive.org is an invaluable Resource for Search engine optimization duties, funded by donations. In the event you search for a site and select the “URLs” option, you could entry around ten,000 shown URLs.
Even so, There are some limits:
URL limit: You could only retrieve as many as web designer kuala lumpur 10,000 URLs, and that is insufficient for much larger web sites.
High-quality: A lot of URLs might be malformed or reference source files (e.g., photographs or scripts).
No export choice: There isn’t a designed-in approach to export the listing.
To bypass The dearth of the export button, utilize a browser scraping plugin like Dataminer.io. On the other hand, these limits imply Archive.org may well not provide an entire Alternative for bigger internet sites. Also, Archive.org doesn’t indicate no matter if Google indexed a URL—however, if Archive.org observed it, there’s a very good chance Google did, too.
Moz Professional
Though you could usually make use of a url index to uncover exterior web sites linking to you, these resources also learn URLs on your internet site in the method.
Tips on how to utilize it:
Export your inbound links in Moz Pro to get a fast and simple list of goal URLs out of your web site. In case you’re working with a large Web-site, think about using the Moz API to export details beyond what’s workable in Excel or Google Sheets.
It’s important to Be aware that Moz Pro doesn’t ensure if URLs are indexed or found out by Google. Having said that, since most sites apply the same robots.txt rules to Moz’s bots as they do to Google’s, this method usually works perfectly to be a proxy for Googlebot’s discoverability.
Google Search Console
Google Look for Console delivers various precious resources for setting up your listing of URLs.
Inbound links stories:
Just like Moz Pro, the Inbound links area gives exportable lists of goal URLs. Sad to say, these exports are capped at one,000 URLs Each individual. You could utilize filters for precise pages, but considering the fact that filters don’t utilize on the export, you may must rely on browser scraping resources—restricted to five hundred filtered URLs at any given time. Not ideal.
Efficiency → Search engine results:
This export gives you an index of pages getting research impressions. Whilst the export is restricted, you can use Google Lookup Console API for larger datasets. You can also find cost-free Google Sheets plugins that simplify pulling more comprehensive data.
Indexing → Webpages report:
This portion gives exports filtered by difficulty sort, while they're also minimal in scope.
Google Analytics
Google Analytics
The Engagement → Pages and Screens default report in GA4 is a superb supply for amassing URLs, with a generous limit of a hundred,000 URLs.
Even better, it is possible to utilize filters to create distinct URL lists, proficiently surpassing the 100k Restrict. For instance, if you'd like to export only web site URLs, stick to these techniques:
Move 1: Include a segment towards the report
Phase two: Click “Produce a new section.”
Move three: Determine the phase which has a narrower URL pattern, like URLs containing /website/
Observe: URLs located in Google Analytics may not be discoverable by Googlebot or indexed by Google, but they supply beneficial insights.
Server log data files
Server or CDN log files are Maybe the ultimate Instrument at your disposal. These logs capture an exhaustive list of every URL route queried by buyers, Googlebot, or other bots during the recorded period.
Considerations:
Data size: Log data files is usually significant, a great number of web pages only keep the last two months of knowledge.
Complexity: Analyzing log files can be complicated, but several instruments can be found to simplify the process.
Incorporate, and fantastic luck
When you finally’ve collected URLs from every one of these sources, it’s time to mix them. If your website is sufficiently small, use Excel or, for much larger datasets, equipment like Google Sheets or Jupyter Notebook. Make sure all URLs are regularly formatted, then deduplicate the record.
And voilà—you now have a comprehensive listing of latest, previous, and archived URLs. Good luck!