Python urlparse: How to Parse URLs

In the last lesson, we had Python read URLs from a file instead of typing them into the code by hand.

So now we can take a file with thousands of URLs and do:

https://example.com/
https://example.com/blog/python-seo/
https://example.com/product/laptop/

But so far, Python treats the URL as text only. To Python, it's just a string.

As SEOs, though, we see more than that.

We see https, we see example.com, and we see /blog/python-seo/.

And if the URL were https://example.com/shop/?category=shoes&page=2, we'd also notice ?category=shoes&page=2.

This matters a lot, because in real work you might need to know:

• Is the URL HTTP or HTTPS?
• Is the link internal or external?
• Which domain does the URL belong to?
• What's the page's path?
• Does the URL have parameters?
• Do I have thousands of different URLs because of parameters?
• Is the URL a blog post, a product or a category?

So today the goal isn't to learn URLs as theory. The goal is to get Python to understand a URL's parts, instead of seeing it as one long piece of text.

First, let's understand the URL itself

Take the URL https://example.com/blog/python-seo/. We can split it simply into three parts:

The https part sets the protocol, or scheme. The example.com part is the domain. And the /blog/python-seo/ part is the path.

A bigger example:

Here we have:

• Scheme: https
• Domain: example.com
• Path: /shop/
• Query: category=shoes&page=2

The query is the part after ?, and you'll see it a lot in SEO. For example ?page=2, ?sort=price or ?filter=color.

On some sites you'll find thousands of URLs generated by these parameters, which can later lead to problems like duplicate URLs or crawl waste.

But first, we need Python to be able to separate these parts.

Can we treat the URL as a string?

Yes. And that's the first thing to understand.

https://example.com/blog/python-seo/

Since url is a string, we can use things we've already learned on it. For example:

Blog URL

We already used that in the last lesson. But we can do more.

For example, we want to know whether the URL uses HTTPS. We could write:

HTTPS URL

The code works. But there's a small problem in the thinking.

We don't want to know whether https:// appears anywhere in the text. We want to know:

And there's a string method that fits better.

Using startswith()

startswith() asks: does the text start with this value? Let's try:

True

Why? Because the URL really does start with https://. Let's try another one:

False

Because this URL starts with http://, not https://. We can use it with if:

Not HTTPS

And that's the first really useful thing we can do on a set of URLs.

Checking a set of URLs

https://example.com/ HTTPS
http://example.com/old-page/ Not HTTPS
https://example.com/blog/seo/ HTTPS

We used something very simple, but we've actually started checking URLs.

What if we want to know how the URL ends?

There's another method called endswith(). Just as startswith() checks the start of the text, endswith() checks its end. For example:

True

False

You can use this, for example, if you have a set of URLs and want to find the files that end in .pdf or .xml or some other extension. An example:

Normal URL: https://example.com/
XML: https://example.com/sitemap.xml
PDF: https://example.com/report.pdf

But we still have a bigger problem. If we want the domain from the URL, how do we get it?

Do we cut the URL up by hand?

You might think of using things like split() or slicing. For example, you might try to break up https://example.com/blog/python/.

But URLs don't all look the same. You might have:

So instead of trying to take a URL apart by hand, Python has a ready-made tool built for understanding URLs. It's called urllib.parse.

And before we use it, we need to understand one small thing: import.

What does import mean?

Python has basic tools we use directly, like print() and len(). But it also has many tools that aren't loaded into the program from the start.

So if I need a particular tool, I can tell Python: I need this tool in the program. We do that with import.

We need a tool called urlparse, which lives inside urllib.parse. So we write:

You don't need to memorize all of Python's libraries right now, or go into the details of modules and packages. What matters for now is understanding that this line lets us use urlparse() in the program.

A first try with urlparse()

ParseResult(scheme='https', netloc='example.com', path='/blog/python-seo/', params='', query='', fragment='')

The output looks a bit big, but it actually did something very useful.

It took https://example.com/blog/python-seo/ and split it into parts. Notice scheme='https', then netloc='example.com', then path='/blog/python-seo/'. So Python has started to really understand the URL.

How do we get just one part?

We kept the result in parsed_url. Now we can ask for the part we need. For example:

https

And if we write:

example.com

And if we write:

/blog/python-seo/

A new shape showed up here: parsed_url.scheme or parsed_url.path.

The idea is simply that urlparse() gave us back a result with several pieces of information in it. With the dot ., we ask for one piece. So parsed_url.scheme means get the scheme, and parsed_url.path means get the path.

Let's try a bigger URL

Scheme: https
Domain: example.com
Path: /shop/
Query: category=shoes&page=2

Now it's getting clearer.

The URL that was https://example.com/shop/?category=shoes&page=2 is now seen by Python like this:

• Scheme = https
• Domain = example.com
• Path = /shop/
• Query = category=shoes&page=2

And that opens up some very important uses later on.

Using hostname

In most of our work we don't need netloc itself; we need the host name, or the domain. We can use parsed_url.hostname. An example:

www.example.com

We'll use hostname a lot, especially once we start telling internal links from external ones.

A real SEO problem: is the link internal or external?

Say our site is example.com, and we have these URLs:

We want Python to tell us whether each URL is internal or external. First, we need the hostname.

example.com

So we can compare: if parsed_url.hostname == "example.com":. If the two are equal, the URL belongs to the site. Let's try:

https://example.com/blog/ Internal
https://example.com/services/ Internal
https://google.com/ External
https://python.org/ External

And that's the first real URL analysis we can use in SEO.

But we have a problem with www

The site might run as example.com, and in some data you'll find www.example.com.

False

That's expected, because the two texts aren't equal. But to us, both may belong to the same site.

So in our simple example, we can accept both. For example:

https://example.com/blog/ Internal
https://www.example.com/services/ Internal
https://google.com/ External

Here we used or, which we learned before.

At more advanced stages we'll normalize URLs better, but for now this way is clear and fits what we're learning.

Finding the URLs with parameters

This is one of the most important things in SEO.

We want to know which URLs have query parameters. We now know that parsed_url.query gets the part after ?.

Let's try it on a URL with no query:

The output is empty. But:

page=2

So we can use if.

No Parameters: https://example.com/shop/
Has Parameters: https://example.com/shop/?page=2
Has Parameters: https://example.com/shop/?sort=price
No Parameters: https://example.com/blog/

You may have stopped at if parsed_url.query: and thought: where's the comparison? We didn't write if parsed_url.query == something. That's because Python can use the text itself as a condition.

If parsed_url.query has text in it, the condition counts as True. If it's empty, "", it counts as False. Let's make it simpler, away from URLs.

There is a name

Empty

Same idea here. If there's a query, the condition runs.

Putting the URLs with parameters in a list

Instead of just printing them:

['https://example.com/shop/?page=2', 'https://example.com/shop/?sort=price']

Here we've started doing something that can later become a real audit.

Picture the file not with 4 URLs but with 50,000, and you can pull out every URL with parameters in seconds so you can start reviewing them.

Telling the page type from the path

Take the URL https://example.com/blog/python-seo/. We don't need to search the whole URL for /blog/. Now we can get the path itself:

Blog URL

Why is this better than searching the whole URL? Because we've set exactly which part we want to check. The word blog might appear in a query parameter, for example, but we care about where the page sits in the URL.

Back to the www problem: making URLs look the same

Remember when we said we'd normalize URLs better later? Let's start on it now in its simplest form, because it makes a big difference in any analysis.

Look at these two URLs:

False

To Python, these are two different URLs. One has Blog with a capital letter, and the other has no / at the end.

If you have a file of URLs from several sources, you'll see these cases a lot. And if you count the pages without making them look the same, the number comes out bigger than the real one.

The fix is to make the URL's shape the same before comparing. That's called normalization.

First: capital and small letters

We have lower(), which turns the whole text into small letters:

https://example.com/blog/python-seo/

Second: the / at the end

Remember strip() from the files lesson? It removed spaces and line breaks from the start and end of the text. We have rstrip(), which is the same idea but from the end only, and we can tell it exactly what to remove:

https://example.com/blog

Meaning: remove any / at the end of the text.

Third: www

We have replace(), which looks for some text and swaps it for other text:

https://example.com/blog/

We told Python: if you find ://www., put :// in its place. We wrote it that way, not www. on its own, so we don't delete www if it shows up anywhere else in the URL.

Putting them in a function

Since we'll do these three steps on every URL, this is a perfect place for a function:

https://example.com/blog
https://example.com/blog
True

The two URLs that were different at first now have the same shape.

Without running it, predict the output. And remember sets from lesson 2:

We have 3 URLs written differently, but after normalizing they're all one URL. If you had counted them without normalization, you'd think you had 3 pages.

Splitting the URL into parts: split()

We said earlier that we wouldn't cut up the URL by hand to get the domain, and that's still true: urlparse() is the right tool there. But there's another very useful use for split(): finding out which section of the site a page belongs to.

split() splits text at a given mark and gives back a list:

['https:', '', 'example.com', 'blog', 'python-seo', '']

We told Python: split the URL at every /.

The empty text '' might surprise you. It's there because between the two / in https:// there's nothing at all. Same at the end, after the last /.

So if you want the section the page belongs to, take parts[3]:

blog

Now let's apply it to a set of URLs:

blog https://example.com/blog/python-seo/
product https://example.com/product/laptop/
blog https://example.com/blog/technical-seo/
category https://example.com/category/phones/

In seconds, we know which section each page belongs to. When we get to Pandas, we'll use the same idea to find out how many clicks each section of the site gets.

Connecting lesson 7 with lesson 6

Now, instead of a list typed by hand, we go back to urls.txt. Say the file contains:

We want to read the file and pull out: the internal URLs, the external URLs, the HTTP URLs, and the URLs with parameters.

This is where we really start putting the lessons together. We write:

Internal URLs:
https://example.com/
https://example.com/blog/python-seo/
http://example.com/old-page/
https://example.com/shop/?page=2
https://example.com/product/laptop/
https://example.com/shop/?sort=price

External URLs:
https://google.com/

HTTP URLs:
http://example.com/old-page/

URLs with Parameters:
https://example.com/shop/?page=2
https://example.com/shop/?sort=price

Let's understand an important point here. We used more than one if.

Then if parsed_url.scheme == "http":, then if parsed_url.query:

Why didn't we use elif? Because one URL can be in more than one group.

For example, http://example.com/shop/?page=2 can be internal, HTTP, and have parameters, all at the same time.

If we used elif, the first condition that's met could stop the ones after it from running. But we want each test to happen on its own. That's a very important point in how to think.

Saving the results to files

Since we learned this in the last lesson, we can save the results. For example, the parameter URLs:

And the HTTP URLs:

Now Python doesn't just find problems or different types of URLs; it also gives you files you can review afterwards. And that's exactly the direction we're heading.

A complete program

You have a urls.txt file containing:

We'll write a program that reads the file and analyzes every URL.

Internal URLs:
https://example.com/
https://example.com/blog/seo/
http://example.com/blog/old-seo/
https://example.com/product/laptop/
https://example.com/shop/?page=2
https://example.com/shop/?sort=price

External URLs:
https://google.com/search/
https://python.org/

Blog URLs:
https://example.com/blog/seo/
http://example.com/blog/old-seo/

Product URLs:
https://example.com/product/laptop/

HTTP URLs:
http://example.com/blog/old-seo/

URLs with Parameters:
https://example.com/shop/?page=2
https://example.com/shop/?sort=price

If you look at this program and compare it with lesson 1, you'll notice we've started doing things much closer to real SEO work.

Python can now:

And we haven't used any crawler or visited the site itself yet. We're only working with a list of URLs we already have.

One last important thing: the fragment

You might see a URL like https://example.com/guide/#pricing. The #pricing part is called a fragment.

pricing

So the main parts we met today: scheme, hostname, path, query and fragment.

You don't have to memorize the names as if you were sitting a theory exam. What matters is that when you see https://example.com/shop/?page=2#products, you understand the URL isn't one piece. Python can take it apart, and you can pick the part that serves the problem you're trying to solve.

What you should understand after this lesson

If you have url = "https://example.com/blog/python-seo/", that's a string.

You can check how it starts with url.startswith("https://") and how it ends with url.endswith(".xml").

And if you need to understand the URL's parts properly, you use from urllib.parse import urlparse, then parsed_url = urlparse(url).

From there you can get parsed_url.scheme, parsed_url.hostname, parsed_url.path, parsed_url.query or parsed_url.fragment.

If you want URLs to look the same before you compare them, you have lower(), rstrip("/") and replace(). And to know which section a page belongs to, you have url.split("/")[3].

But more important than the commands themselves is understanding why we do this. We don't learn urlparse() just because there's a function called urlparse(). We learn it because, to an SEO, a URL holds information, and each part of it may need checking differently.

The practical test

Create a file called urls.txt and write in it:

Write a program that does the following:

1. Reads the URLs from the file.
2. Uses strip() to clean each URL.
3. Uses urlparse() to analyze the URL.
4. Creates a list called internal_urls and puts the example.com and www.example.com URLs in it.
5. Creates a list called external_urls and puts the rest in it.
6. Creates a list called http_urls and puts the URLs that use http instead of https in it.
7. Creates a list called blog_urls and puts any URL whose path contains /blog/ in it.
8. Creates a list called parameter_urls and puts the URLs with query parameters in it.
9. Prints each group clearly.

Once you've done the first part, add one last step.

If you managed this test yourself, you're no longer just reading URLs with Python. You've started analyzing the structure of the URL itself and pulling out information you can use in SEO.