Finding And Reclaiming Lost Links Caused By Site Problems
Today, I’m going to take you through one of my approaches to reclaiming links that have been lost due to site issues rather than the site owner deciding that they don’t like you anymore.
For the sake of this post, let’s say you have an absolutely enormous site. Your team recently rebuilt it in a completely different CMS and, as a consequence of that, pretty much all your URLs had to change, meaning an awful lot of redirects had to be put in place.
Let’s assume it wasn’t the smoothest transition ever. This is where the link reclamation process can be really valuable, especially since these errors won’t show up during the course of a normal site crawl.
What You’ll Need
These are the tools I use for this process. You might have preferences, but I find these ones work best:
- Ahrefs and/ or MajesticSEO and/ or Buzzstream
- SEO Tools for Excel
- Excel
- Notepad
- Screaming Frog
Finding Your Lost Links
Ahrefs has an awesome Lost & New Links tab, and it even lets you segment them by your own specific timeframes with an export to CSV option for each one.
Once you’ve got the data from your specified time-period, you’re ready to start mining for data, but depending on the number of links, you might want to do the same process for new links in case Ahrefs missed them due to a crawler error, a bit of downtime on the linker’s site or taking time to find the redirected URL. I like to put the two columns side-by-side and run a clear duplicates function in Excel to eliminate these.
If you don’t use Ahrefs, Buzzstream‘s (aff) weekly reports also send you a list of lost links for your sites. Personally, I prefer using the Ahrefs function so I don’t have to dig through emails, but this is a good alternative.
If you’re a MajesticSEO fan or, like me, use both, that data’s still available and you can actually get it quicker through SEO Tools for Excel. This plugin really is the gift that keeps on giving.
There’s nothing wrong with exporting the data from all these tools and clearing the duplicates out in Excel. In fact, doing it this way will probably help you find more links as they’ve all got different indexes. Don’t be afraid to pull Majestic’s Historic Index too – you might find some good links that you lost in a previous rebuild that are worth reclaiming.
Now we’ve got our definitive list of lost links, we can start looking into why they went away. It’s time to unleash the spreadsheets.
Identifying the Faults
There are loads of ways that we can use this data to find out how many of the links were lost due to site issues, but I find SEO Tools for Excel to be one of the most efficient, although running Screaming Frog in list mode is quicker.
Firstly, set up a column for the status code of the linking page – you might have lost the link because the site or page disappeared. SEO Tools has a status code checker, so you can run them straight from your sheet.
Quick Tip: All functions from this plugin are run from formulae and, as is Excel’s way, formulae keep running, meaning it’s constantly checking status codes and anything else you tell it to. If you want to keep your spreadsheet running quickly and want to share it with people who don’t have the plugin, run your functions in batches of thirty or so, copy the results into Notepad and paste them back. This will put your results into plain text and stop the formula, meaning no productivity gets lost by you sitting there shouting at your computer telling it to hurry the hell up.
This should tell you straight away if any links have been lost due to the site or page going away or any bad redirects being put in place on their part.
Now you need to do the same thing for the page on your site that they’re actually linking to. Here’s where you can find out if there are technical issues in place that are causing the links to go away. In this case, due to the sheer volume of 301 redirects, we’ll needed to run a double-layered status code test, checking the status of the URL the old page was redirecting to.
If you find any 404’s, further redirects or a redirect loop, take it to the development team and then you’ve got those links back.
Digging Deeper
You’re going to look pretty silly if you approach the site owners that have either removed or redirected the linking page only to find that they’ve put your link somewhere else or that your tools didn’t find the link on that page last time it crawled for whatever reason. Tools aren’t perfect, after all.
This is where Screaming Frog can help: if you’ve got a good site that seems to have stopped linking and isn’t showing up in your new links tab, run a crawl with your brand name or site URL in the custom section. You might find that they’re still linking, saving you the time of contacting them.
That’s Not Everything
Depending on how far you want to dive, that might be it. You’ve probably found quite a few site issues here, and you’ve probably found a few sites in your link profile that have gotten broken. You can get in touch with them and help them fix their problems to reclaim those links and you should definitely have a word with your development team to fix your own site’s issues.
Due to the size and complexity of the site I first did this for, I added a couple of extra tabs to drill down into it a little bit further, segmenting it by links lost due to removal, ones that were lost due to site issues on their part and ones that went away because of mis-configured redirects or other site problems.
Try It Yourself
There’s so much else that can be done with this process that I won’t cover it all here – this post’s long enough as it is. Give it a go yourself and if you’ve got any other ideas as to how this can work, please let others know in the comments or drop me a line through the contact form.
Have fun.
