Server Log Analysis


Video Transcription for Server Log Analysis

Hi, guys. A quick video on server log analysis. Now, I’m not sure how many of you would know where to get your server log’s first. So I’m gonna show you where to get in if you’ve got your cPanel account or whatever. What you need to do before you do any server log analysis is go to metrics and go to row access, what you’ll find is you’ve archived your logs and you’ve got a bunch of different files here based on your own website. Obviously, I’ve got November’s logs here. So I can click here, download these logs from our website, and there’s quite a lot of them there.

Now, server log analysis was quite fiddly and not something many SEO’s used to do. One of the old ways to do it was use Screaming Frog as your log final analyzer. You can download that, which was fairly low cost. I believe there was a free version, AP version. So there are free and paid versions of Screaming Frog. You can upload your log files there, analyze them, see what bots are happening on your pages and whatnot. So that all works well, and that’s pretty much what a server log analysis guy would do for you. There are various other tools, and some people get there own bespoke stuff, and various other bits and bobs.

But, one of the main all in one SEO tools out there is SEMrush. They now have a log file analyzer. It’s in beta mode just now, so it’s still being developed and they are still messing about with it. If you just go down to the second bottom option on here that plays and you are able to then play about with your server logs and upload them to SEMrush, which then gives you them in a nice, clean, easy-to-understand platform. So what you can do is, obviously, download them from the row access there. You obviously want to uncompress them, so you just click on the compressed version and it will open up, and then you drop your files in. So that was my files there, so I click open and drop the files in there.

The tool does take 5 or 10 minutes to analyze your logs, and then it spits out a report that looks a lot like this. This is obviously my server log analysis from November. You can obviously see all of the pages here, and you can filter all of this stuff down to successful pages, file types, and various other bits and bobs. Obviously, the path … You’ll obviously see the path here, so you can obviously see the bots I had in my plug-ins, and stuff like that is where you can see all this data so you can break it all down. The file paths are all here. The bot hacks are there, share of the hits percentages there. Obviously, what we are … When I say we, I mean SEMrush, I invited 40 people over to  Helsinki in Finland and we worked with a product developer and this was one of the tools that we worked on. What people and the majority of people wanted to see the crawl frequency and the last crawl. That’s what was asked for, and the last status as well. So you can see here, crawl frequency every 20 minutes. It was last crawled at this time here. You can obviously see the last status called on them as well, obviously, 200 is a good thing. But, yeah, you obviously want to work your way through and analyze your server logs.

You can do all server logs, Google desktop bot, or Google smartphone, so you can check those out. You can’t obviously check other bots at present. That is something that may or may not come out in the future, you’d have to ask SEMrush that part. But, I think for the vast majority of people out there, the server logs is the one that’s most important. It’s the one that we all want to rank well on, we all want to make sure that the Googlebot likes our website and can get on about a page, and it can frequently crawl those pages, and stuff like that, and what you see what’s getting on there. You can obviously filter all of this information down by file type. If it was HTML files you were wanting to check out, then you can check all them out there. Obviously, the HTML files are all making a blog post and whatnot on your website. We can see that some of those are redirected and some of them are just pages that are working well. But, you can obviously filter all that down for yourself, and you can obviously filter it by path. But, that is entirely up to yourself and how deep you want to analyze your server logs. It’s great to find out how the bots are hitting your website.

I think,for me, who is not by any stretch of the imagination, a server log analysis expert. It’s not something that I obsess over, but there are certainly guys out there. One, for example, is Andy Halliday who is a geek on this type of stuff and is obsessed with it and is an expert on server logs. Dawn Anderson’s another one. These guys are right into the server logs, check that stuff out on a regular ongoing basis, because obviously, you can identify problems with Googlebot hacking your pages and stuff like that, and obviously make adjustments to your website as a result of that. But, what I find really helpful is having it in a nice, clean, easy-to-understand dashboard, which is what SEMrush has brought into play. Obviously, very easy for someone who’s not as technical, as some of the other ones that I’ve mentioned, to be able to understand and implement changes to the website as a result of these server logs.

So server logs is not a big thing, amazing that SEMrush has brought this out. If you’re not a SEMrush subscriber, you can do it the old-fashioned way through Screaming Frog, and if you Google some other stuff, there are a lot of places you can upload your server logs to that will allow you to do server log analysis. But, for me, SEMrush is a tool that many agencies, or freelancers, or whatever you may be, will be using, and it’s just another great feature that SEMrush have added on top of all the other stuff you can see here that’s all been added.

That is how you do server log analysis. Obviously, I can’t show you individual websites and stuff like that, but obviously, you want the Googlebot to be able to freely crawl your website and not find broken pages and all that kind of stuff. So that’s the end goal, so the green line is where you want to be at. Obviously, you can see it’s up and down, up and down, up and down for one reason or another. But, that’s obviously something that we always look to strive to better our own websites. Hopefully, that gives you a bit more of an update on server log analysis, how you can analyze your files, and also, where you can find your server logs, because a lot of people don’t even know where they can find that. This is where you can find your server logs within your control panel.

seo profile image

Craig Campbell

I am a Glasgow based SEO expert who has been doing SEO for 17 years.

  • social media icon
  • social media icon
  • social media icon

Online Courses