How to Extract URLs from a Sitemap for Better SEO Audits

Why Sitemaps Matter for Technical SEO

If you have spent any time working on website optimization, you know that the XML sitemap is one of the first things you check during an SEO audit. It is the blueprint that tells Googlebot which pages on your site matter and how they relate to each other. Without a clean, well-structured sitemap, you risk leaving important pages out of the index or, worse, sending crawlers down dead ends. That is why knowing how to extract URLs from a sitemap is a skill worth having in your toolkit.

I have been on both sides of the fence. I have built sites where the sitemap was auto-generated by Yoast SEO or Rank Math and seemed fine, and I have inherited sites where the sitemap was a mess of outdated URLs, broken links, and orphaned pages. In every case, the first step was to pull out all the URLs and start sorting. You cannot fix what you cannot see, and you cannot see everything by just browsing the site manually.

The Basics: What a Sitemap Contains

An XML sitemap is a file that lists URLs from your site along with optional metadata like last modified date, change frequency, and priority. A single sitemap can hold up to 50,000 URLs, but many sites use a sitemap index file that points to multiple child sitemaps. That is common on large ecommerce sites or news publishers. If you are using Google Search Console, you can see which sitemaps have been submitted and whether any errors popped up during crawling. But the console only shows high-level issues. To do a proper audit, you need the raw list.

Manual Methods: When You Just Need a Quick List

Sometimes you only need to extract URLs from a sitemap once, maybe for a small site. In that case, you can open the sitemap URL in a browser, view the page source, and copy the tags. It is tedious but it works. I have done this for sites with fewer than 200 URLs and it took maybe ten minutes. But if you are dealing with thousands of URLs, manual copying is not practical. You also risk missing entries if the sitemap uses a sitemap index file that nests multiple sitemaps.

Another quick approach is using cURL in the terminal. A command like curl -s https://example.com/sitemap.xml | grep -oP '(?<=)[^<]+' will give you a clean list of URLs. That works on Unix systems and on Windows if you have a terminal emulator. It is fast and does not require any special tools. But again, if the sitemap uses an index file, you need to loop through each child sitemap. That is where a dedicated tool starts to pay off.

Using SEO Tools to Extract URLs from Sitemap

For regular work, I rely on tools like Screaming Frog, Ahrefs, and SEMrush. Each of them can extract URLs from a sitemap and give you a downloadable list. Screaming Frog, for example, lets you enter a sitemap URL directly and it will crawl every URL listed, reporting back on status codes, titles, meta descriptions, and more. That is useful when you are doing a full technical SEO audit and need to check for broken links, 301 redirects, or canonical URLs that point to the wrong place.

Ahrefs and SEMrush both have site audit features that start by pulling the sitemap. They compare the URLs in the sitemap against what they actually find during crawling. If there are pages in the sitemap that return 404 errors or redirect chains, those tools flag them. That is how I once discovered that a client had a sitemap full of product pages that had been deleted months earlier but never removed from the sitemap. Googlebot was hitting those dead pages every time it crawled, wasting crawl budget on nothing.

When you need to extract URLs from a sitemap programmatically, Python with Beautiful Soup is a solid choice. You can write a small script that fetches the sitemap, parses the XML, and writes the URLs to a CSV file. If the sitemap uses an index file, the script can follow the references recursively. That gives you full control and you can add logic to filter by last modified date or priority. I have used this approach for sites with complex sitemap structures that no single tool handled well.

What to Do With the Extracted URLs

Once you have the list, the real work begins. Compare the sitemap URLs against your actual site structure. Look for pages that should be indexed but are missing. Check for pages that are in the sitemap but blocked by robots.txt or noindex tags. Those are common issues that hurt content optimization and technical SEO. I once audited a site where the sitemap included a whole section of staging pages that were supposed to be blocked. The developer had forgotten to add the noindex directive, and those pages were being indexed instead of the live versions.

Also watch for broken links within the sitemap. A broken link in a sitemap is a signal to Googlebot that your site is not well maintained. It can lower your overall crawl priority. Fix those with 301 redirects or remove the URLs entirely if the pages are gone. And always check that canonical URLs match the sitemap URLs. If you have a page at /product in the sitemap but the canonical points to /product?sort=popular, you are telling search engines to ignore the URL you want indexed. That is a common mistake with parameter handling.

extract URLs from sitemap

When Sitemaps Are Not Enough

Sitemaps are a starting point, not a guarantee. Googlebot may still discover pages through internal links or external backlinks that are not in the sitemap. That is fine. But if you rely only on the sitemap for your audit, you might miss orphaned pages that have no internal links. Those pages exist in isolation and are hard for crawlers to find. A thorough SEO audit should combine sitemap extraction with a full crawl of the site. Tools like Screaming Frog can crawl from the homepage and compare the discovered URLs against the sitemap. Any page in the sitemap that the crawl did not find is a red flag that your internal linking is weak.

I also recommend checking the sitemap submission status in Google Search Console regularly. If the sitemap has errors, you will see them there. But do not stop at the console. Pull the sitemap yourself and verify it. I have seen cases where the console showed no errors, but the sitemap contained URLs with trailing slashes that redirected to non-slash versions. Those redirects were unnecessary and added latency for users and crawlers. Fixing them improved page speed slightly, and over thousands of pages, that matters.

Practical Workflow for a Sitemap Audit

Here is a workflow I have used on multiple projects. It keeps things organized and catches most issues.

Download the sitemap and all child sitemaps if it is a sitemap index file. Use a tool or script to extract URLs from sitemap and save them in a spreadsheet.
Run a full crawl of the site using a tool like Screaming Frog or a Python script. Export the list of all discovered URLs.
Compare the two lists. Mark any URL that appears in the sitemap but not in the crawl as a potential orphan or redirect. Mark any URL in the crawl but not in the sitemap as a missing entry.
Check every URL in the sitemap for status code, canonical tag, and noindex directive. Flag anything that deviates from the expected state.
Review the sitemap for broken links, outdated URLs, and parameter variations. Clean up the sitemap and resubmit it.

This process takes a few hours on a large site, but it pays off in better indexing and fewer crawl errors. I have seen sites recover from a crawl budget crisis just by cleaning up their sitemap and removing the dead weight.

Tools I Keep in My Arsenal

Beyond the big names, I keep a few lightweight options handy. For quick checks, I use a browser extension that parses sitemaps and shows the URLs in a table. It is not as thorough as a full tool, but it is fast. For deeper work, I rely on Python with Beautiful Soup and the requests library. That combination handles any sitemap format I have encountered, including those with namespaces or custom tags. I also use cURL for one-off tasks on servers where I cannot install software.

The key is to have a method that you can repeat consistently. If you extract URLs from a sitemap the same way every time, you can compare results across audits and spot trends. For example, if the number of URLs in your sitemap drops suddenly, it might mean a plugin like Yoast SEO or Rank Math stopped generating certain sections. Or if the number spikes, it might mean duplicate URLs are creeping in. Both are worth investigating.

Final Thoughts

Knowing how to extract URLs from a sitemap is not just a technical trick. It is a fundamental part of maintaining a healthy site that search engines can trust. Whether you use a GUI tool, a command line, or a script, the goal is the same: get the data, verify it, and act on it. That is where real content optimization and technical SEO improvements come from. Not from guessing, but from seeing exactly what the crawlers see and fixing the gaps.