How to Think in Python to Solve an SEO Problem
About 45 minutes
- Variable
- String
- Integer
- Float
- Boolean
- List
- Dictionary
- type
- key
- index
- append
- len
- if
- else
- elif
- Comparison Operators
- and
- or
- not
- for
- while
- break
- continue
- Function
- def
- Parameter
- Argument
- return
- open
- with
- read
- write
- strip
- startswith
- endswith
- import
- urlparse
- csv
- reader
- DictReader
- writer
- DictWriter
- fieldnames
- pandas
- DataFrame
- read_csv
- head
- columns
- shape
- value_counts
- isna
- notna
- sort_values
- contains
- to_csv
- set
- lower
- rstrip
- replace
- split
- duplicated
- drop_duplicates
- rename
- merge
- groupby
- Input/Processing/Output
- Pseudocode
- Debugging
- Sanity Check
- try
- except
The counter at the top counts this lesson's exercises (12 of them). Reading and running the examples don't count.
We've reached a point that's a bit different from the lessons before.
So far, every lesson taught us something new:
And we took examples of each one and applied them to SEO.
But if I gave you a crawl file with 50,000 URLs right now and said:
Should the first thing you do be writing code?
No. And that's exactly the idea we'll work on in this lesson.
The problem isn't whether or not you know a Python command. The more important problem is knowing:
And then:
Today we won't learn a new library, and I won't try to hand you twenty new commands. We'll use what we've learned, but focus on how to think before and while writing code.
And that's a very important lesson before the final project.
Don't start from the code; start from the problem
Let's take this example:
A beginner usually sees the problem and starts searching:
And then copies any code they find. The code may work. But if one column name changes, or the file's shape is different, it will probably break, and you won't understand why.
We want a different way. Before any code, ask yourself:
Where is the data?
I have a file, crawl.csv.
What information do I need?
I need URL, Status and Title.
What's the condition?
There are two: Status = 200 and Title missing.
What do I want to do with the result?
For example: print it, review it, save it in a new CSV.
So the problem that looked big has started to turn into small pieces.
- 1Read the file.
- 2Check the columns.
- 3Filter status 200.
- 4Filter missing titles.
- 5Show the result.
- 6Save it.
Only now do we start thinking about code.
Turn the problem into simple sentences first
Before you write:
you need to be able to say to yourself, in plain English:
If the sentence itself isn't clear in your head, the code will probably be harder.
Pay attention to this point. Python doesn't define the problem for you. You define the logic first, then turn it into Python.
So Status = 200 becomes df["Status"] == 200.
And Title missing becomes df["Title"].isna().
And "the first condition and the second condition" becomes & in Pandas. So:
You didn't memorize a line here. You built it. And that's a very big difference.
Don't solve the big problem all at once
Say you have this task:
If you try to write the whole program at once, you'll very likely make a mistake and not know where.
It's better to break up the task. First:
If the result looks right, we carry on. Then:
If the column names are what we expected, we carry on. Then we try the first condition:
Did it work? We carry on. Then:
Did it work? We carry on. Then we save:
You built the program piece by piece. And if something goes wrong, you'll know roughly which step it happened in.
This is a very important rule for learning
Don't write 50 lines of code and then run them for the first time.
- 1Write a small piece.
- 2Run it.
- 3Look at the output.
- 4Understand it.
- 5Then carry on.
That's why, since the start of the course, we've been printing results. Not because print() is the final answer, but because it lets you see what Python is doing.
Trick one: inspect the data before you analyze it
Say someone sent you a crawl.csv file and you went straight to:
But the column is actually called Status Code. The code will throw an error.
Or the status might be written as OK and Not Found, not numbers.
Or the file might not even be the file you think it is.
That's why, as soon as you read a new data set, do a quick check. For example:
The results might be:
dtype='str' instead of dtype='object' for text columns, a detail explained in the last lesson. The example here shows the general idea, not a particular version.Those four things answer important questions.
head() answers: what does the data look like?
print(df.head()) gives you the first five rows. From them you can see:
- Was the data read correctly?
- Do the values make sense?
- Do the columns hold the data you expect?
- Is anything odd?
columns answers: what do I actually have?
print(df.columns) matters a lot before you write df["Title"], because the column might be called Title 1 or Page Title. You're not supposed to guess. Look at the file first.
shape answers: how big is the data?
print(df.shape): if the output is (50000, 20), you have 50,000 rows and 20 columns. That's important information.
Picture expecting 50,000 URLs and getting (180, 20). Do you carry on with the analysis?
Better to ask yourself first:
Maybe you exported part of the crawl, not all of it. So this check isn't just Python. It's part of making the SEO analysis itself sound.
dtypes answers: how does Python see the data?
print(df.dtypes) shows you, for example, Status int64, Words int64 and Title object.
That helps you know whether the comparison you're about to make makes sense.
If the status is a number: df["Status"] == 404. But if it's text: df["Status"] == "404". You may need to handle it differently. Don't assume the data type. Check it.
Trick two: test the filter before you rely on it
Say you wrote:
The code ran without errors. Does that mean the result is right?
Not necessarily. Code can run and give a wrong result because the condition itself is wrong.
So print a sample:
And look for yourself. Do the pages really have status 200? Is the title really missing?
If the result is 0 rows:
and you get (0, 5), don't assume straight away that the site has no missing titles.
The reason might be, for example, that the empty cells say Missing instead of being NaN. Then df["Title"].isna() won't catch them. So you need to look at the data itself.
The code running doesn't mean the analysis is right
This is one of the most important points in the lesson. Python can run the code without any error, and you can still have asked the wrong question.
An example:
The code is correct. But if your goal is "get the 200 pages", the condition is wrong. Why? Because Status != 404 means: get anything that isn't 404.
So it might bring back 200, 301, 302, 500 and 503. What you needed was:
Both are correct Python. But only one answers the question you want. And that's why the SEO mindset matters more than just writing correct syntax.
Ask yourself: what am I filtering on?
Let's take an example. You have:
| URL | Status | Indexability |
|---|---|---|
| /page-1/ | 200 | Indexable |
| /page-2/ | 200 | Non-Indexable |
| /page-3/ | 404 | Non-Indexable |
| /page-4/ | 301 | Non-Indexable |
If the task is "get the non-indexable pages", you could do:
URL Status Indexability 1 /page-2/ 200 Non-Indexable 2 /page-3/ 404 Non-Indexable 3 /page-4/ 301 Non-Indexable
Right? Yes. But if the real question is "get the 200 pages that are non-indexable", you need to add:
So:
URL Status Indexability 1 /page-2/ 200 Non-Indexable
The difference here isn't Python. The difference is that you defined the question precisely.
Trick three: start with a small sample
If you have a file with 1,000,000 rows and you're still building the code, not every try has to run on a million rows.
You can make a small copy of the data and work on it first. For example:
Now you have the first 100 rows. Work on them. Try the conditions. Review the results. And once you're sure the logic is right, apply it to the whole DataFrame.
That's not just faster. It also makes mistakes easier to spot.
Trick four: print the count before and after filtering
Say you have df with 50,000 rows, and you want the 404 pages.
Instead of just saving the file, do:
And after filtering:
The output might be:
Now you understand what happened.
What does len() mean here?
We've used len() before. With a list, for example:
3
With a DataFrame, print(len(df)) gives the number of rows. So if you have 50,000 rows, the output is 50000. That's very useful when reviewing.
Do a sanity check
An important term in how to work: Sanity Check. The idea is simply:
Say a site has 20,000 URLs. You ran an analysis and got:
Could that be right? It could. But the number is very large, and it's worth stopping to review. Print:
And look. You might find you used the wrong column name. Or that every value turned into NaN while reading. Or that the export itself has a problem.
A sanity check isn't a Python command. It's a way of thinking.
Trick five: be clear with variable names
You could write:
The code will run. But 50 lines later:
you'll ask yourself: what was x? Better:
And instead of:
write:
And instead of:
write:
You don't pay by the letter. Write a name that helps you understand the code, especially while you're still learning.
Trick six: don't shorten the code before you understand it
You could write:
The line may be correct. But if you're still learning, what do you gain?
Let's take the same operation apart:
The result is more or less the same. But you can walk through the second version step by step. You can print print(pages_200), then print(low_word_pages). And if an error happens, you know where.
Once you understand the code well, you can shorten it if that really makes it better. But don't shorten it just because the shorter line looks smarter.
Trick seven: the difference between a loop and Pandas
This is a very important question. We learned loops, then Pandas. Does that mean we use Pandas for any SEO data?
No. Your choice depends on the problem.
When does a loop make sense?
If you have a simple list:
And you want to print each URL:
https://example.com/ https://example.com/blog/ https://example.com/product/
That's perfectly natural. Or if each URL needs its own separate operation. For example, later on, when we start sending a request to each page:
A loop will be a very natural part of the solution here.
When is Pandas better?
If you have a table:
| URL | Status | Title | Words |
|---|---|---|---|
| /page-1/ | 200 | Page 1 | 500 |
| /page-2/ | 404 | Page 2 | 0 |
| /page-3/ | 200 | — | 80 |
And you want to filter, sort, count, pick columns and find missing values, Pandas fits very well. For example:
That's easier than building a loop by hand.
Don't say loops are bad and Pandas is good
Both are tools. If you have a nail, a hammer helps. If you have a screw, a screwdriver helps. No tool is better in every case. The question is:
When do I write a function?
We learned functions, but the important question is: when do I actually need one?
Say you wrote:
Once. Do you need a function? No.
But say you have 5 files, and every time you want to run the same check. Then you might do:
Rows: 4
Columns: 5
URL Status Title Indexability Words
0 /page-1/ 200 Page 1 Indexable 500
1 /page-2/ 404 Page 2 Non-Indexable 0
2 /page-3/ 200 NaN Indexable 80
3 /page-4/ 301 Page 4 Non-Indexable 20And then inspect_data(df). If you have another DataFrame: inspect_data(search_console_df). Here the function makes sense, because you have a repeated operation.
A simple rule for functions
If you find yourself copying the same code more than once, ask yourself:
That doesn't mean every 3 lines must become a function. We don't write functions to make the code look professional. We write them when they help us organize the work or avoid repetition.
When do I use a list, and when a DataFrame?
Take this example:
I only have a set of URLs. A list fits perfectly.
But if each URL comes with Status, Title, Words and Indexability, now we have a table. A DataFrame will usually be better here.
So: a simple list of values → list. Rows + columns → DataFrame.
It's not a fixed law across all of Python, but it's a very good way to think at this stage.
Don't use a big tool for a small problem
You want to know whether one URL contains /blog/. Do you need Pandas? No.
Blog
That's enough.
You have 50,000 URLs in a CSV with 20 columns, and you want the blog pages and their statuses analyzed? Here Pandas makes sense.
Read the error instead of closing the code
When you start learning Python, it's completely normal to see errors. The problem is that the first reaction is usually: the code doesn't work.
But the error usually tells you something important. An example:
But the column that exists is called Status. You might see KeyError: 'Status Code'.
Instead of saying "Pandas is broken", read KeyError, then 'Status Code'. The message says the key, or column, you asked for doesn't exist in that form.
The first thing to do:
And see the real name.
Another error example
You wrote:
But the file isn't in the current location. You might see FileNotFoundError.
What's the first question?
- Is the file name right?
- Is the file in the same folder?
- Did you write the path correctly?
You don't need to memorize every error name. We want you to start treating errors as information, not as a wall in front of you.
What if I want the program to keep going even when an error happens?
So far we've read the error and fixed the code. But sometimes the error isn't a mistake in the code; the mistake is in the data itself.
For example, you have a word count column from a tool, and most values are numbers, but some pages say N/A or -. And we still want to turn them into numbers with int():
You'll see ValueError: invalid literal for int() with base 10: 'N/A'
And the bigger problem is that the program stopped at the second value. It never got to the values after it. Picture this happening with 50,000 rows, and the program stopping at row 12.
This is where we use try and except:
Not a number: N/A Not a number: - [850, 1200, 300]
Read it like this:
ValueError happens, don't stop: print the value and move on to the next one.The line inside try is the one that might cause a problem. And if the problem does happen, Python jumps straight to what's inside except, and then the loop carries on as normal.
Why did we write ValueError specifically?
You could write except: on its own, without the error type, and the code would run. But then you're telling Python: whatever problem happens, keep quiet about it.
And if there's a completely different mistake in the code, like a misspelled variable name, you'll never find out. The program will produce a wrong result, and you'll think it's right.
That's the opposite of everything we've talked about in this lesson. We want to know what happened, not hide it.
So name the error you expect, and when it happens, print it or log it. Like we did: we printed the values we couldn't convert, so you know there are two pages with missing data.
The same idea with files
File not found: crawl.csv
Instead of the program stopping with a long message, it tells you clearly what the problem is.
Without running it, predict the output:
What will be printed?
Why does this matter right now?
In the next stage we'll start sending requests to real pages. A page might time out, and a server might not respond. Without try and except, one broken page will stop the whole program while you're still in the first 100 pages out of 5,000.
When an error happens, don't change five things at once
This is an important point. Say the code doesn't work. Don't do all of this at once:
- Change the file name.
- Change the column name.
- Change the condition.
- Change how you read the file.
- Change the library.
And then try. If it works, you won't even know what was wrong.
Change one thing. Try. Look at the result.
It's the same way of thinking we use in technical SEO. If you have a problem and change 10 things at once, even if the problem goes away, you don't know what fixed it.
Separate input, processing and output
This is one of the best ways to understand any program we build from here on. Almost any task can be split into:
- 1Input
- 2Processing
- 3Output
Input
Where does the data come from? For example crawl.csv or urls.txt, and later a website, an API or Search Console.
In our example:
That's the input part.
Processing
What will we do with the data? For example:
That's processing. We took the data and applied logic to it.
Output
Where do you want the result? Maybe:
Or:
That's output.
Why does this split matter?
Because as the program grows, you can ask: is the problem in the input? Was the file read wrong?
Or is the problem in the processing? Is the condition wrong?
Or is the problem in the output? The result is right, but the file wasn't saved?
Instead of looking at the whole program as one block.
A full example of the way of thinking
The task:
Don't write code first. Let's think.
Where does the data come from?
crawl.csv
Which columns do we need?
URL, Status and Title
How many conditions are there?
Three: URL contains /blog/, Status = 200, and Title missing.
What result do we want?
A new CSV: blog-missing-titles.csv. Now we write the code.
URL Status Title Indexability Words 3 https://example.com/blog/python/ 200 NaN Indexable 1500 Total URLs: 8 Blog missing titles: 1
See the difference? The big line came at the end. First, the logic itself was clear.
But before trusting the result, we check a sample and look at the first ten results, which is what we actually did in the code above by printing the count before and after filtering.
Thinking before code can be pen and paper
Not all the thinking has to happen inside Python. If the task is big, you can first write:
That isn't Python. And that's not a problem. On the contrary: you've separated solving the problem from writing the code. Then you turn each step into Python.
And that leads us to an idea called pseudocode
We'll use this word a lot from here on. It isn't real Python code. It's a way of writing a program's logic in simple language before you build it. For example:
That's pseudocode. If you put it into Python like that, it won't run. But it helps you build the program.
Then we translate: Read the crawl file becomes df = pd.read_csv("crawl.csv"). And Get the status 200 pages becomes pages_200 = df[df["Status"] == 200]. And so on.
Don't think about syntax while you don't know the solution yet
This is a well-known mistake. Someone has a problem, and the first thing they think about is: what's the function called?
But they don't even know yet what they need a function to do. The better order:
- 1The problem
- 2The logic
- 3The steps
- 4The tool
- 5Syntax
Not: syntax first, then see what to do with it.
An example: 100,000 URLs and a possible crawl trap
We won't get into diagnosing the crawl trap itself now, but let's think about the data.
You have a CSV with URL and Status, and you noticed lots of URLs like:
Your first job: find out how many URLs have parameters. Let's think.
The data: CSV. The column: URL. The condition: the URL contains ?. So we could use:
But an odd shape appeared here: "\?". And I don't want to take you into regular expressions in this lesson.
So instead we could go back to something you already understand. Add a logical column with apply()? That's something new too, and we don't need it.
So what's the right decision for now?
We can use a simple loop over the URLs if that's clearer at the current level. For example:
2
The point here matters a lot. We picked one of the tools we already understand. The shortest solution doesn't have to be the best one for learning or for the task at hand.
If you can solve a problem with something you've learned, and there's no strong reason to bring in a new concept, use what you understand.
regex=False from the Pandas lesson? That's also a simple solution with a tool you understand: df[df["URL"].str.contains("?", regex=False)]In the final project, we'll try to work with the tools we've learned. We won't show off every Python library in one project.
An important trick: don't edit the original file at the start
You have crawl.csv, the original export from the tool. Don't start by writing the result over the same file. For example, don't do this from the start:
if you're experimenting. Better:
Or 404-pages.csv and missing-titles.csv.
Why? So the original file stays. If you make a mistake in the filtering, you can go back to the original data. That's a very important habit in data analysis.
An important trick: keep the columns that help with review
Say you're pulling out missing titles. Do you save the URL only? You could.
But during review it can be very useful to keep URL, Status, Title and Indexability instead of just the URL.
Why? So when you open the report, you have context that helps you confirm the result. For example:
Then:
Think about the output from the point of view of the person who'll review it later. Not from "I managed to produce a list, and that's it".
An important trick: don't treat everything as an issue just because a tool said so
Let's take an important example. You pulled out 200 non-indexable pages. Is every one of them an SEO problem?
No. It might be a page that's supposed to be noindex in the first place.
So Python is telling you: these are the pages that match the conditions. Not necessarily: these are problems that must be fixed.
This is a fundamental point. Python analyzes the data according to the logic you wrote. It's not a replacement for the SEO decision. For example:
That means: get the pages with status 200 that aren't indexable. It doesn't mean: all these pages are errors. That's where your role as an SEO comes in.
Python gives you candidates, not decisions
Make this a rule. For example, pages under 100 words, which you might pull out with:
Is every page under 100 words thin content? No. It might be:
- A contact page.
- A login page.
- A utility page.
- A product page that's naturally short.
- A page that's meant to be brief.
So Python says: I have 45 pages under 100 words. But you decide whether that's an issue.
That's one of the most important differences between automation and SEO judgment.
Don't let automation switch off your brain
Our goals with Python: save time, cut manual work, spot patterns, analyze big data. Not to stop thinking.
If you write a script that says "any page under 300 words = bad SEO", you've just built a weak rule that runs faster.
So automation won't fix wrong thinking. It'll make it faster.
Another example: a 404 doesn't automatically mean a redirect
We pulled out:
Does that mean we redirect every 404? No. A 404 might:
- Need a redirect.
- Stay a normal 404.
- Be a wrong internal URL.
- Come from an old backlink.
- Need a 410 in a particular case.
- Just need an internal link fixed.
Python pulled out the group. Deciding the fix needs context. That's the way of thinking we want to keep throughout the course.
Separate detection from decision
We can split SEO automation work into two stages:
- 1Detection
- 2Decision
So Python helps you with detection: get the 404s, get the missing titles, get the redirects, get the parameters.
And then comes the decision, a stage that needs an understanding of SEO. For example:
- Is this 404 a problem?
- Does it need a redirect?
- Should the page be indexable?
- Is the parameter URL really a problem?
Not every detection automatically becomes a fix.
Now let's take a problem from start to finish
The task: you have a crawl from Screaming Frog, and you want a quick report on the pages that need review.
The file: crawl.csv. The columns: URL, Status, Title, Indexability and Words.
Before we write code, we define what we want. We want:
- 404 pages.
- 301 pages.
- 200 pages with no title.
- 200 pages that aren't indexable.
- 200 pages under 100 words.
- The count for each status code.
Now we build the program step by step.
Step one: read the data
We don't run any analysis yet.
Step two: inspect the file
If everything makes sense, we carry on.
Step three: understand the file's size
The output might be:
Now we have a baseline.
Step four: status codes
The output might be:
Before you even make any reports, you've got a very quick picture of the crawl.
Step five: 404 pages
The output:
Then look at a sample:
Step six: redirects
Step seven: missing titles
Review a sample:
Step eight: non-indexable 200 pages
Step nine: pages with few words
But remember: this result is pages to review, not Pages that must be fixed.
Step ten: save the reports
Now the program is done.
Do we turn it into a function?
We could. But before we do, ask yourself:
If the answer is yes, we can start organizing it later. For example:
URL Status Title Indexability Words 0 /page-1/ 200 Page 1 Indexable 500 1 /page-2/ 404 Page 2 Non-Indexable 0 2 /page-3/ 200 NaN Indexable 80 3 /page-4/ 301 Page 4 Non-Indexable 20
But if the whole project reads only one file, you don't have to turn every step into a function. Let the organization serve the problem, not the problem serve the shape of the code.
An important mindset: know the input and output of every step
Take:
Before the line, the input is df. After the line, the output is broken_pages.
What's the difference? df has all the data. broken_pages has only the rows whose status is 404.
If you can't explain that line in words, go back and understand it before you carry on.
Try to predict the result before running
This is one of the best ways to learn. We have:
If we wrote a loop, try to predict the result before you press Run.
Same idea with SEO. If the file has:
| URL | Status |
|---|---|
| /page-1/ | 200 |
| /page-2/ | 404 |
| /page-3/ | 404 |
| /page-4/ | 301 |
And we did:
Before running it, you know the result should be two pages. If the code gives 4, there's a problem. If it gives 0, there's a problem.
Predicting keeps your brain switched on alongside the code.
A trick: make a small data set by hand to test the logic
This is a very useful move. Before trying on a real crawl, you can make a small file, test.csv, containing:
You know exactly what the result should be.
Missing titles: /page-3/. 404: /page-2/. 301: /page-4/. Low words: /page-3/.
URL Status Title Indexability Words 2 /page-3/ 200 NaN Indexable 80
Once the code works on the small data whose result you know, then try it on the real file.
The difference between debugging and guessing
If the code doesn't work, you have two ways.
The first way: change things at random until it works. That's guessing.
The second way:
- 1Find the last part that worked.
- 2Find the first part that went wrong.
- 3Check the values.
- 4Change the thing that has to do with the problem.
We want to get into the habit of the second way.
A simple debugging example
You have:
Empty DataFrame Columns: [URL, Status] Index: []
And the result: an empty DataFrame. You're sure the file has 404s. Don't change the condition straight away. Try:
0 200 OK 1 200 OK 2 404 Not Found Name: Status, dtype: str
You might find the values aren't 404 but 404 Not Found. You've found the problem from the data.
Another example
You wrote:
And got KeyError: 'Title'. Do:
You might find Title 1. So the problem is the column name, not Pandas.
Watch out for spaces
A header might be Title with a space at the end. You see Title with your own eyes. But Python sees "Title ", which is a different value from "Title".
We won't start advanced data cleaning now. But I want you to know that problems with the shape of the data itself exist.
And that's one more reason print(df.columns) matters.
Practical tricks before any SEO analysis
When you get a new data set, you can start with:
Then, if you have a status column:
If you have indexability:
That isn't a full analysis script. It's just a first exploration of the data. And it's an excellent habit.
Don't write a condition on a column before you know the values in it
For example, you have an Indexability column. Don't assume the value is Non-Indexable. It might be Non Indexable or Noindex or something different, depending on the source.
Do:
And see the actual values. Then write the condition.
Same idea with the status
You might find 200, 301, 302, 404, 403 and 500. And based on that, you start asking better questions.
Python can help you ask questions, not just answer them
This is an important point. Sometimes you go into a file without knowing the problem. You can use Python to explore. For example:
And you find 8,000 URLs returning 302. That's a number worth noticing. Or print(df["Indexability"].value_counts()), and you find half the site is non-indexable.
Or you sort:
And you start seeing a pattern.
So Python isn't only "I have a question, give me the answer". It can be "help me understand the data so I know the right question to ask".
Don't turn every audit into rigid rules
If you wrote:
that's a filter, not an SEO law.
If you wrote:
those are pages that went past a threshold you set. But you need to know why you chose that threshold, and whether it really fits the task.
The code carries out your decision. It doesn't prove the decision is right.
So how do you build any simple SEO script?
Use these questions, in order.
1. What's the problem?
For example: I want to find the 404 pages. Not: I want to use Pandas. Start with the problem, not the tool.
2. Where is the data?
For example crawl.csv or urls.txt.
3. Which columns or values do I need?
For example URL and Status.
4. What's the condition or logic?
For example Status = 404.
5. Which tool will I use?
- I have a table:
Pandas. - I have a simple list and want to go through the items:
Loop. - I have a repeated operation:
Function. - I have a URL and want to take it apart:
urlparse.
6. How will I make sure the result is right?
7. Where does the result go?
For example print, CSV or Excel.
Let's apply these questions
The problem: I want to find the blog pages with fewer than 100 words.
The problem?
Blog pages under 100 words
The data?
crawl.csv
The columns?
URL, Status and Words
The conditions?
URL contains /blog/, Status = 200 and Words < 100
The tool?
A table, so Pandas fits.
The code
Empty DataFrame Columns: [URL, Status, Title, Indexability, Words] Index: []
100 words (the fewest is 1500), so the result is empty. But the goal is to see how the code itself gets built from the seven questions, not this particular number.Checking
Saving
See how the code came almost at the end of the thinking? That's the point of this lesson.
Before the final project
I want you to go into the next project with a different mindset. When you see a big task, don't say:
Say:
Do it. Then: what's the next step? Do it.
In the end, a big program is a set of small steps.
The lesson quiz
This time the quiz isn't about knowing a function's name. It's about whether you've started thinking the right way.
You have a file, seo-audit.csv, with:
Before you write any code, write down for yourself the steps needed to solve each question.
Question 1
You want to know: how many URLs are in the file? Define the input, processing and output, then write the code.
Write the input, processing and output steps for counting the URLs in seo-audit.csv, then write the code itself.
Write your answer first. The value is in trying before you look.
Question 2
You want to know: how many pages do I have for each status code? Think first: do you need a loop by hand? Or is there a Pandas tool we learned that does this?
Write the tool you'd use to count the pages for each status code, then write the code.
Write your answer first. The value is in trying before you look.
Question 3
You want to pull out: the 200 pages whose title is missing. Before the code, write the condition in plain English. The result should contain only https://example.com/blog/python/.
Write the condition in plain English first (give me the rows where...), then turn it into Pandas code.
Write your answer first. The value is in trying before you look.
Question 4
You want to pull out: the blog pages whose status is 200. Careful, not just any blog URL. So https://example.com/blog/old/ must not appear, because it's a 404.
Write the two conditions you need (on the URL, and on the status), then write the code.
Write your answer first. The value is in trying before you look.
Question 5
You want to pull out: the 200 pages with fewer than 100 words. You should get https://example.com/blog/python/ and https://example.com/category/phones/.
Write the code that solves this question.
Write your answer first. The value is in trying before you look.
Question 6
When we pulled out the pages with fewer than 100 words, can we say: these two pages have thin content and must be fixed?
Answer: is Python making a thin-content decision here, or only producing candidates? Why?
Write your answer first. The value is in trying before you look.
Question 7
Say you wrote:
But the task was: the pages with status 200. Is the code correct Python? And is the logic right for the task? Explain the difference.
Explain the difference between df[df['Status'] != 404] and df[df['Status'] == 200] for the task 'get the 200 pages'.
Write your answer first. The value is in trying before you look.
Question 8
You wrote:
And got KeyError: 'Page Title'. What's the first step you take before you start changing the code?
What's the first practical step you take when a KeyError appears on a column name?
Write your answer first. The value is in trying before you look.
Question 9
You have a file with 100,000 URLs and you're still trying out new logic. What's better at the start? Trying it on all 100,000 URLs each time? Or starting with a small sample? Explain why.
What's better: trying on the full file, or on a small sample (df.head(100), for example)? Explain why.
Write your answer first. The value is in trying before you look.
Question 10
You have this code:
The code works. But rewrite it with clearer variable names that will help you understand the program a week from now.
Rewrite the code with clearer variable names than x and y.
Write your answer first. The value is in trying before you look.
Final mindset challenge
You have a big crawl file. The task:
Don't write the code straight away. First write:
- 1Input
- 2The columns needed
- 3The conditions
- 4How to check
- 5Output
Then turn each step into Python.
And pay attention to something very important in how the problem is worded. We have:
And within those pages, we want the page if:
or:
So the logic isn't all AND. Think this part through before you write the code.
Build a review report from start to finish
Read seo-audit.csv and pull out the pages with Status = 200 and Indexability = Indexable that also have a missing title or fewer than 100 words (an or, not an and). Keep only URL, Title and Words in the result, and save it in pages-to-review.csv without the index.
If you reached the solution yourself, you're ready for the final project. Because the most important thing before the project isn't memorizing read_csv() or isna(). It's knowing how to take an SEO problem, break it down, define the data and the conditions, pick the tool, test the result, and only then rely on the output.
And that's the difference between someone who memorizes Python code and someone who has really started using Python as a tool in SEO work.
You still have exercises left
0/12Exercises left: 12 of 12. The lesson counts as complete once you solve them all, and you can come back to them any time.
Well done, you reached the end of the lesson.