How To Find Hidden Links On Web Pages

Tabular array of Contents

Why y'all need to notice all the pages on your site

How your content actually gets to be seen

What is crawling and indexing?

Links

Sitemaps

CMS

What is indexing?

Using robots.txt

Using 'noindex'

What are orphan pages?

How do orphan pages come near?

How about dead-finish pages?

Where do dead-end pages come up from?

What are hidden pages?

Should all hidden pages be done away with?

Newsletter sign ups

Pages containing user information

How to notice subconscious pages

Using robots.txt

Manually finding them

How to observe all the pages on your site

Using your sitemap file

Using your CMS

Using a log

Using Google Analytics

Manually typing into Google's search query

What then practice you do with your URL list?

Manual comparison with log information

Using site crawling tools

SEOptimers SEO crawl tool

In decision

Retrieve near it. Why practise you create a website? For your potential customers or audience to easily find y'all and for yous to stand up out amidst the competition, right? How does your content really get to be seen? Is all the content on your site ever seen?

Why you need to discover all the pages on your website

Information technology is possible that pages containing valuable information that actually needs to exist seen, do not get to exist seen at all. If this is the case for your website, so you are probably losing out on significant traffic, or even potential customers.

At that place could also exist pages that are rarely seen, and when they are, users/visitors/potential customers hit a expressionless-end, as they cannot access other pages. They can only go out. This is as simply as bad as those pages that are never seen. Google will begin to note the high bounce rates and question your site's credibility. This will run across your spider web pages rank lower and lower.

How your content really gets to be seen

For users, visitors or potential customers to see your content, crawling and indexing needs to be washed and done often. What is crawling and indexing?

What is crawling and indexing?

For Google to show your content to users/visitors/potential customers, information technology needs to know showtime that content exists. How this happens is via crawling. This is when search engines search for new content and add it to its database of already existing content.

What makes crawling possible?

• Links
• Sitemaps
• Content Management Systems (CMS – Wix, Blogger)

Links:

When you add a link from an existing page to another new page, for case via anchor text, search engine bots or spiders are able to follow the new page and add it to Google'southward 'database' for future reference.

Sitemaps:

These are also known as XML Sitemaps. Here, the site owner submits a list of all their pages to the search engine. The webmaster tin as well include details like the last date of modification. The pages are then crawled and added to the 'database'. This is however non real time. Your new pages or content will not be crawled as before long every bit yous submit your sitemap. Crawling may happen later on days or weeks.

Most sites using a Content Direction Organisation (CMS) machine-generate these, and so it's a scrap of a shortcut. The but time a site might non have the sitemap generated is if you created a website from scratch.

CMS:

If your website is powered by a CMS like Blogger or Wix, the hosting provider (in this example the CMS) is able to 'tell search engines to crawl whatever new pages or content on your website.'

Here's some information to help you with the process:

Calculation a sitemap to WordPress

Viewing the sitemap

Where is sitemap for Wix?

Sitemap for Shopify

What is indexing?

Indexing in uncomplicated terms is the adding of the crawled pages and content into Google'southward 'database', which is actually referred to every bit Google's index.

Before the content and pages are added to the index, the search engine bots strive to empathise the page and the content therein. They fifty-fifty get alee to catalog files like images and videos.

This is why as a webmaster, on-page SEO comes in handy (page titles, headings, and apply of alt text, amidst others). When your page or pages have these aspects, it becomes easier for Google to 'understand' your content, catalog it appropriately and index it correctly.

Using robots.txt

Sometimes, you may non want some pages indexed, or parts of a website. You demand to give directives to search engine bots. Using such directives likewise makes crawling and indexing easier, as in that location are fewer pages being crawled. Learn more about robots.txt here.

Using 'noindex'

You can also this other directive if at that place are pages that yous practise not want to appear in the search results. Larn more about the noindex.

Earlier yous start adding noindex, you'll want to identify all of your pages so you can clean up your site and get in easier for crawlers to clamber and alphabetize your site properly.

What are some reasons why yous demand to find all your pages?

What are orphan pages?

An orphan folio can be defined as one that has no links from other pages on your site. This makes it near impossible for these pages to exist plant by search engine bots, and in improver by users. If the bots cannot find the folio, then they will not show information technology on search results, which further reduces the chances of users finding information technology.

How do orphan pages come nearly?

Orphan pages may result from an attempt to continue content individual, syntax errors, typos, indistinguishable content or expired content that was non linked. Here are more ways:

• Exam pages that were used for A/B testing and that were never deactivated
• Landing pages that were based on a flavor, for case, Christmas, Thanksgiving or Easter
• 'Forgotten' pages as a result of site migration

How nigh dead-end pages?

Unlike orphan pages, dead-terminate pages have links from other pages on the website only do non link to other external sites. Dead-finish pages examples include give thanks you pages, services pages with no phone call to actions, and "nothing found" pages when users search for something via the search option.

When you have dead-end pages, people who visit them only accept ii options: to leave the site or go back to the previous folio. That means that you are losing significant traffic, especially if these pages happen to be 'principal pages' on your website. Worse still, users are left either frustrated, confused or wondering, 'what's next'?

If users leave your site feeling frustrated, dislocated or with whatsoever negative emotions, they are never likely to come dorsum, just like unhappy customers are never likely to buy from a brand again.

Where do expressionless-finish pages come up from?

Dead end-pages are a consequence of pages with no calls to activity. An example hither would be an near page that alludes to the services that your company offers but has no link to those services. Once the reader understands what drives your company, the values you uphold, how the company was founded and the services you lot offer and is already excited, y'all need to tell them what to do adjacent.

A simple telephone call to activeness button 'view our services' will do the job. Brand sure that the push button when clicked really opens upwardly to the services page. You practise not want the user to be served with a 404, which will leave him/her frustrated besides.

What are hidden pages?

Hidden pages are those that are not attainable via a menu or navigation. Though a visitor may exist able to view them, specially through anchor text or inbound links, they tin be difficult to detect.

Pages that autumn into the category section are probable to exist hidden pages too, as they are located in the admin console. The search engine may never be able to access them, as they practice not access data stored in databases.

Subconscious pages tin can also result from pages that were never added to the site's sitemap but be on the server.

Should all hidden pages be done away with?

Not really. In that location are subconscious pages that are absolutely necessary, and should never be accessible from your navigations. Let'south expect at examples:

Newsletter sign ups

You lot tin have a folio that breaks downwardly the benefits of signing up to the newsletter, how frequently users should expect to receive it, or a graphic showing the newsletter (or previous newsletter). Remember to include the sign up link likewise.

Pages containing user data

Pages that require users to share their information should definitely be hidden. Users need to create accounts earlier they can access them. Newsletter sign ups can besides exist categorized here.

How to find hidden pages

Like we mentioned, you can observe hidden pages using all the methods that are used to find orphan or dead cease pages. Allow'due south explore a few more.

Using robots.txt

Hidden pages are highly likely to be subconscious from search engines via robots.txt. To access a site's robots.txt, blazon [domain proper name]/robots.txt into a browser and enter. Supplant 'domain name' with your site's domain proper name. Await out for entries commencement with 'disallow' or 'nofollow'.

Manually finding them

If you sell products via your website for example, and suspect that one of your product categories may be hidden, yous can manually expect for information technology. To do this, copy and paste some other products URL and edit it accordingly. If yous don't observe it, and so you were right!.

What if yous take no idea of what the hidden pages could be? If you organize your website in directories, y'all can add your domainname/folder-name to a site'south browser and navigate through the pages and sub-directories.

Once you take establish your subconscious pages (and they do not need to stay hidden equally discussed above), you need to add it to your sitemap and submit a crawl asking.

How to observe all the pages on your site

You need to detect all your web pages in order to know which ones are dead-stop or orphan. Let's explore the unlike ways to achieve this:

Using your sitemap file

Nosotros have already looked at sitemaps. Your sitemap would come up in handy when analyzing all of your web pages. If y'all exercise non take a sitemap, you tin can use a sitemap generator to generate one for y'all. All you demand to do is enter your domain name and the sitemap will exist generated for you.

Using your CMS

If your site is powered by a content management organisation(CMS) similar WordPress, and your sitemap does not comprise all the links, it is possible to generate the listing of all your web pages from the CMS. To do this, use a plugin like Export All URLs.

Using a log

A log of all the pages served to visitors besides comes in handy. To access the log, log in to your cPanel, then detect 'raw log files'. Alternatively, request your hosting provider to share it. This manner you lot go to see the most frequently visited pages, the never visited pages and those with the highest driblet off rates. Pages with loftier bounce rates or no visitors could be dead-cease or orphan pages.

Using Google Analytics

Hither are the steps to follow:

Pace 1: Log in to your Analytics page.

Step 2: Go to 'behavior' and then 'site content'

Footstep 3: Go to 'all pages'

Step iv: Roll to the bottom and on the right choose 'evidence rows'

Step 5: Select 500 or 1000 depending on how many pages you would judge your site to accept

Footstep 6: Coil upwardly and on the top correct cull 'export'

Pace vii: Choose 'consign as .xlsx' (excel)

Step 8: Once the excel is exported choose 'dataset ane'

Step 9: Sort by 'unique page views'.

Footstep x: Delete all other rows and columns apart from the one with your URLs

Step xi: Utilize this formula on the second column:

=CONCATENATE("http://domain.com,A1)

Step 12: Replace the domain with your site'southward domain. Elevate the formula so that it is applied to the other cells as well.

You now have all your URLs.

If you lot want to convert them to hyperlinks in order to easily click and access them when looking something upwardly, proceed to step xiii.

Pace 13: Use this formula on the third row:

=HYPERLINK(B1)

Drag the formula so that information technology is applied to the other cells as well.

Manually typing into Google'southward search query

Yous can also type this site: world wide web.abc.com into Google's search query. Supplant 'abc' with your domain proper name. You lot will go search results with all the URLs that Google has crawled and indexed, including images, links to mentions on other sites, and even hashtags your brand can be linked to.

You can then manually copy each and paste them onto an excel spreadsheet.

What then do you do with your URL list?

At this point, you may be wondering what y'all demand to practise with your URL list. Allow'due south wait at the available options:

Manual comparing with log data

1 of the options would be to manually compare your URL list with the CMS log and identify the pages that seem to have no traffic at all, or that seem to have the highest bounciness rates. You can then utilise a tool like ours to check for inbound and outbound links for each of the pages that yous doubtable to exist orphan or dead end.

Another approach is to download all your URLs equally a .xlsx file (excel) and your log too. Compare them adjacent (in two columns for example) so use the 'remove duplicates selection' in excel. Follow the step past step instructions. By the end of the process, you lot will accept merely orphan and expressionless-stop pages left.

The 3rd comparison approach is copying two data sets – your log and URL listing on to Google Sheets. This allows yous to utilise this formula: =VLOOKUP(A1, A: B,2,) to look up URLs that are nowadays in your URL listing, but not on your log. The missing pages (rendered as N/A) should be interpreted as orphan pages. Ensure that the log data is on the commencement or left column.

Using site crawling tools

The other option would exist to load your URL listing onto tools that tin perform site crawls, look for them to clamber the site and then you copy and paste your URLs onto a spreadsheet earlier analyzing them ane by one, and trying to figure out which ones are orphan or dead end.

These two options can be fourth dimension-consuming, peculiarly if you accept many pages on your site, right?

Well, how virtually a tool that not only finds you all your URLs merely also allows you to filter them and shows their status (so that you lot know which ones are dead finish or orphan?).  In other words, if you lot want a shortcut to finding all of your site'due south pages  SEOptimer's SEO Crawl Tool.

SEOptimer's SEO Clamber Tool

This tool allows you to access all your pages of your site. You lot can start by going to "Website Crawls" and enter your website url. Hit "Crawl"

Once the crawl is finished you can click on "View Report":

Our crawl tool will discover all the pages of your website and list them in the "Page Found" section of the crawl.

You can identify "404 Error" problems on our "Issues Plant" but beneath the "Pages Found" section:

Our crawlers can identify other problems like finding pages with missing Title, Meta Descriptions, etc. Once yous notice all of your pages, you can kickoff filtering and work on the issues at hand.

In conclusion

In this article we take looked at how to detect all the pages on your site and why it is of import. We have also explored concepts like orphan and dead end pages, as well as hidden pages. We have differentiated each one, how to identify each among your URls. There is no better time to find out whether you are losing out due to hidden, orphan or expressionless-finish pages.

How To Find Hidden Links On Web Pages,

Source: https://www.seoptimer.com/blog/find-all-pages-on-a-website/

Posted by: amundsonswayse.blogspot.com

Share this post