AngularJS: Index and Supercharge Your SEO

 

 

AngularJS is a good framework for building websites and apps because it makes them much faster and richer for users.

But there’s one problem every developer faces when pushing their AngularJS product: Search Engines Optimization, or SEO.

Quality SEO means getting found amidst all the noise online. When a site or app is optimized for search it is more likely to be found by prospective users. If it is not optimized, then a developer might as well be screaming at the wind – no exposure and no users are almost guarantees.

Right now the most common techniques for mitigating this problem is to either build separate versions of the app or website for search engines, or to pre-render pages on the server. The problem with these solutions, however, is that you have to maintain two separate systems: one for your users and another for Google and the other search engines.

While Google has attempted to help developers by improving its capability to index JavaScript and CSS but even if Google’s efforts to index JavaScript since 2014 have advanced, it doesn’t mean that your app will be indexed properly.  Indeed, Google still recommends creating snapshots to make an AJAX application crawlable.

But how exactly are these snapshots created? And how can a developer be sure that their AngularJS app or website is correctly and completely indexed by Google?

In this post we present a free and self hosted solution to generate snapshots and to make sure that your AngularJS website or application crawlable by, indexed by, and optimized for Google.

Want to download these instructions and all the code in a PDF?

Enter your email and get it for free!

AngularJS and SEO: The Problem

Search engines crawlers were originally designed to index the HTML content of web pages.

Today, however, JavaScript and other frameworks like AngularJS and BackboneJS  are playing a leading role in web development and the creation of content and application online.

Unfortunately, the crawlers and the other indexing mechanisms behind search engines remain decidedly unfriendly to JavaScript powered sites.

angularjs-js-code

AngularJS and SEO: The Solution

Overcoming the indexing problem is not difficult when developers embrace what are called ‘snapshots’.

Snapshots is a term used to refer to content generated for the search engine crawlers on the website’s backend. The idea behind snapshots is that the developer does the work for the crawler that it cannot or doesn’t want to do on it’s own. Optimizing and caching snapshots not only help you get indexed, but also improves significantly the speed of indexation.

An important note: JavaScript indexation currently only applies to Google’s crawler. Other search crawlers (such as those from Microsoft’s Bing search engine) do not support crawling JavaScript applications yet. As well, despite web content being increasingly shared to social networks like Facebook and Twitter, most social network crawlers don’t handle JavaScript either.

So how do you generate snapshots, and how do you work with them to make sure you are indexed?

Read on for the step-by-step guide.

Step One: Generate Snapshots

The first step is to generate the snapshots themselves.

To do this we need access to a snapshot server based on a headless browser such as PhantomJS or ZombieJS. In this example we will use the open source middleware Prerender that already packages PhantomJS and is ready to handle our special crawler requests and serve HTML snapshots.

In order to reduce the time, it takes to generate snapshots a cache can be employed. Snapshots are cached on a Redis Server the first time they are requested, and then re-cached once a day (note: this can be manually configured to suit your needs) to make sure the content stays up-to-date.  As a result, a static snapshot is always and instantly available to be served to the crawler.

angularjs-map

Step 2: Server Installation

In this example we will use an Apache server run on Ubuntu 14.04.2 LTS.

There are five sub-steps to work through here.

1 – Install NPM and NodeJS

sudo apt-get update
sudo apt-get install nodejs npm
ln -s /usr/bin/nodejs /usr/bin/node

2 – Install Forever

npm install forever -g

3 – Install  and Start Prerender.io

git clone https://github.com/prerender/prerender.git
cd prerender
npm install

Make sure the server starts on 4001 and that PhantomJS is on 4002.

You can edit this file if you want to change the port:

/lib/index.js

Return to the Prerender folder and start the server using forever – this will help to start the server continuously in the background.

forever start server.js

4 – Install Redis server

Add the Dotdeb repositories to your APT sources. To do this, create a new list file in /etc/apt/sources.list.d/ and fill it with the following content:

# /etc/apt/sources.list.d/dotdeb.org.list
deb http://packages.dotdeb.org squeeze all
deb-src http://packages.dotdeb.org squeeze all

Then you need to authenticate these repositories using their public key:

wget -q -O - http://www.dotdeb.org/dotdeb.gpg | sudo apt-key add -

Next, install Redis using apt-get:

sudo apt-get update
sudo apt-get install redis-server

Then enable the Redis service to start on boot:

sudo service redis_6379 start
sudo service redis_6379 stop

You should then check the Redis status:

$ redis-cli ping

You will get “PONG” if everything is ok.

5 – Make Prerender use the Redis server to cache snapshots

Prerender has an open source module, Prerender-Redis-Cache, that makes it easy to perform this task.

In your local prerender project ( prerender/server.js) run:

$ npm install prerender-redis-cache --save

Then add these two lines in prerender/server.js :

process.env.PAGE_TTL = 3600 * 24 * 5; // change to 0 if you want all time cache
server.use(require('prerender-redis-cache'));

Restart Prerender by:

forever stopall
forever start server.js

And if you want to clean all REDIS cache you can use:

redis-cli -p 6379 flushall

Step 3: Server Configuration

Now we will redirect crawlers to the local Prerender server using a simple .htaccess file.

This htaccess file have contain all the redirect configurations. Note that the .htaccess file needs to be in same directory with your main AngularJS index.html file.

<IfModule mod_rewrite.c>

RewriteEngine On

Options +FollowSymLinks

# redirect non www to www

RewriteCond %{HTTP_HOST} !^www\.

RewriteRule ^(.*)$ http://www.%{HTTP_HOST}/$1 [R=301,L]

# Don't rewrite files or directories

RewriteCond %{REQUEST_FILENAME} -f [OR]

RewriteCond %{REQUEST_FILENAME} -d

RewriteRule ^ - [L]

# List of all crawlers and bot you can add more

RewriteCond %{HTTP_USER_AGENT} Googlebot|bingbot|Googlebot-Mobile|Baiduspider|Yahoo|YahooSeeker|DoCoMo|Twitterbot|TweetmemeBot|Twikle|Netseer|Daumoa|SeznamBot|Ezooms|MSNBot|Exabot|MJ12bot|sogou\sspider|YandexBot|bitlybot|ia_archiver|proximic|spbot|ChangeDetection|NaverBot|MetaJobBot|magpie-crawler|Genieo\sWeb\sfilter|Qualidator.com\sBot|Woko|Vagabondo|360Spider|ExB\sLanguage\sCrawler|AddThis.com|aiHitBot|Spinn3r|BingPreview|GrapeshotCrawler|CareerBot|ZumBot|ShopWiki|bixocrawler|uMBot|sistrix|linkdexbot|AhrefsBot|archive.org_bot|SeoCheckBot|TurnitinBot|VoilaBot|SearchmetricsBot|Butterfly|Yahoo!|Plukkie|yacybot|trendictionbot|UASlinkChecker|Blekkobot|Wotbox|YioopBot|meanpathbot|TinEye|LuminateBot|FyberSpider|Infohelfer|linkdex.com|Curious\sGeorge|Fetch-Guess|ichiro|MojeekBot|SBSearch|WebThumbnail|socialbm_bot|SemrushBot|Vedma|alexa\ssite\saudit|SEOkicks-Robot|Browsershots|BLEXBot|woriobot|AMZNKAssocBot|Speedy|oBot|HostTracker|OpenWebSpider|WBSearchBot|FacebookExternalHit|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR]

RewriteCond %{QUERY_STRING} _escaped_fragment_

# Only proxy the request to Prerender if it's a request for HTML

RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent))(.*) http://www.mydomain.com:4001/http://www.mydomain.com/$2 [P,L]

# Rewrite everything else to index.html to allow html5 state links

RewriteRule ^ index.html [L]

</IfModule>

You have now finished all server side installation tasks, so it’s now time to configure the AngularJS App.

Step 4: App Configuration

First open you Angularjs index.html file and:

  1. make sure you have <base href=”/”> before </head>
  2. add <meta name=”fragment” content=”!”> between <head></head> (by adding this tag into the page www.example.com, the crawler will temporarily link this URL to www.example.com?_escaped_fragment_= and will request this from your server)

Second, activate HTML5 mode.

In your config.js file add:

$locationProvider.html5Mode(true);

This will tell your application to use HTML5 URL format.

URLs typically look like http://www.example.com/directory/page. By default AngularJS elements will have URLs like this: http://www.example.com/#!/directory/page

var app = angular.module('app')

.config(['$httpProvider', '$locationProvider', function ($httpProvider, $locationProvider) {

// use the HTML5 History API

$locationProvider.html5Mode(true);

…

Third, you need to manage the meta tags.

To improve the SEO of your app or website you need to have a unique title and description for each page. An AngularJS module called AngularJS-View-Head already exists to fix this problem. This module will help us to change the HTML title and head elements on a per-view basis.

How do you work this in practice?

Start by installing this module using bower.

Next,  declare the module as a dependency of your application:

var app = angular.module('myApp', ['ng', 'viewhead']);

This makes available the directives described in your HTML template.

Finally add the meta tags inside your template.

<view-title>{{ artist.title }}</view-title>

<meta view-head name="description" content="{{ 'SEO.ARTIST.DESCRIPTION' | translate:{ artist: artist.name, artist_ar: artist.name_ar } }}"/>

<!-- Open Graph data --->
<meta view-head property="og:title" content="{{artist.name}}" />
<meta view-head property="og:type" content="article" />
<meta view-head property="og:url" content="http://www.example.com/artist/{{artist.slug}}" />
<meta view-head property="og:image" content="{{artist.photo}}" />
<meta view-head property="og:description" content="{{ artist.description }}" />

Step 5: Test Prerender Server

If you’ve followed all of the steps things should be working well. However, better safe than sorry, it’s time to test.

Compare the source of one of your pages with and without _escaped_fragment_ in the URL.

You can check specific routes in your browser and compare :

http://www.example.com/directory/page
http://www.example.com/directory/page?_escaped_fragment_=

Step 6: Add a Sitemap

The final step in your AngularJS SEO strategy is to develop a sitemap.

To help search engines crawl your app or website and make sure pages are indexed quickly you must create a sitemap for all of your routes. This sounds difficult to do, however, with proper build processes this can be automated using a tool like Grunt.

(Heads up: we’ll be publishing another post soon explaining just how to automate a sitemap using Grunt!)

Conclusion

Making sure search engines and – through searches – users can find your app or website is essential. The strategy presented in this post is quick to implement, scalable and easy to maintain and, where employed, should help you make the connections you need and win the users you want.

Do you like what you read?

Get more content like this one, every Monday in your inbox! We manually curate the best Marketing content and send it to you. No spam ever.

35 thoughts on “AngularJS: Index and Supercharge Your SEO
  1. Jeff

    That article is great! Thanks for doing that. I like how you added the section about using the redis cache.

    Reply
  2. Jason

    Hi, probably a very confused question coming up right now, but when installed locally, how does it serve snapshots on the site/in production? Thanks for a great article, sorry about the ignorance.

    Reply
  3. Faouzi EL YAGOUBI

    Hi,
    Yes it will serve snapshots on the production server for search engines. Normal visitors will see normal app but search engines engines will see prerendred version. The pages will be the same in the content and design so it will respect search engine roles.

    Regards

    Reply
  4. Faouzi EL YAGOUBI

    You can optimize your app by following this steps :
    * Setup Grunt tasks to
    – Compile AngularJS templates from multiples files to one file
    – Minify and crompress JS, CSS, HTML
    – Combine all JS files to one file
    * Clean unused javascript and css code to reduce size of your files
    * Use prerender solution described in my article
    * Use redis Cache to make rendering faster
    * Use CDN and good hosting to make your app faster

    Reply
  5. Junior

    Your tutorial seems helpfull, although I struggled a hell of a lot to follow it. This guide would help someone who has experience and knowledge about how ubuntu works and where everything is located.

    For example. I had no clue what APT was or how to create a list in that folder, had to get external resources to find out.
    Also for many of the steps, you dont specify from what folder on the terminal they should be run….”sudo service redis_6379 start” did not work at all – from any folder.
    Where was I supposed to run the “install redis server” command? in the prerender directory? (That is where the previous step left off).

    These are just examples as on quite a few places I simply had to guess the step in between.

    I am not calling you out for being useless, im sure this is helpfull to many people, but for users who dont really have much experience developing on linux – this is extremely difficult to follow.
    If you want to make a step-by-step tutorial then you need to take into account just that, you cant simply ommit every second step and expect the end user to know that they need to now go back to the root directory or what APT is for that matter. Step-by-step tutorials are meant to dumb things down to the simplest level possible. This is more of a summarised guide or walkthrough, but not a step by step guide.

    I struggled for some hours now with this and have gotten nowhere.
    Either way, thanks for the effort, but it could have been a bit more in depth and aybe, just maybe I would have been able to get to the end.

    Reply
    1. Faouzi EL YAGOUBI

      Sorry about that but this article require some knowledge about server management.
      – Advanced Packaging Tool (APT) perform such functions as installation of new software packages, upgrade of existing software packages, updating of the package list index, and even upgrading the entire Ubuntu system. APT command it is accessible by default in Ubuntu and you don’t need to install it.
      – ”sudo service redis_6379 start” can be run in any folder if you have install correctly redis. Please follow this tutorial to install redis https://www.linode.com/docs/databases/redis/redis-on-ubuntu-12-04-precise-pangolin

      Reply
  6. Yemi A.

    Great post, Faouzi, I discovered this just in time. Thanks for sharing. I was wondering, have you published the post on using grunt to re/generate sitemap.xml yet? Otherwise, could you simply point me in the right direction to look? Thanks.

    Reply
    1. Faouzi EL YAGOUBI

      Thanks Yemi, I don’t get time to finish the article about sitemap.xml but it will be published soon. I will let you know.

      Reply
  7. tirta

    I’m newbie to angularjs, and I’m using angularjs and yii2 framework and try to follow your guidance step by step, on step 3 htaccess configuration. we have only html files in template folder. Main file : /frontend/web/index.php, in which folder should I put this htaccess and is it need modify htccess configuration ? Thanks

    Reply
    1. Faouzi EL YAGOUBI

      You need to create .htaccess file in the same folder of index.php (file : /frontend/web/ )

      Reply
  8. Leo

    Can you please post the .htaccess but for nginx? Or this .htaccess works for nginx too?

    Thank you.

    Reply
  9. test

    To me my markup looks exactly the same. I wonder if I’m doing something obviously wrong.

    Reply
  10. Ali Adravi

    I tried it but it is not working on different CARDS like Twitter Card, Facebook and Google plus.
    When we type a link on these websites, they crawl the page and show the preview but with Angularjs it is showing {{page.title}}, {{page.description}} etc.

    Reply
  11. faouzi

    You need to add the Google and Twitter social bot in the list
    RewriteCond %{HTTP_USER_AGENT} Googlebot|bingbot|Googlebot-Mobile|Baiduspider|Yahoo|YahooSeeker|DoCoMo|Twitterbot|TweetmemeBot|Twikle|Netseer|Daumoa|SeznamBot|Ezooms|MSNBot|Exabot|MJ12bot|sogou\sspider|YandexBot|bitlybot|ia_archiver|proximic|spbot|ChangeDetection|NaverBot|MetaJobBot|magpie-crawler|Genieo\sWeb\sfilter|Qualidator.com\sBot|Woko|Vagabondo|360Spider|ExB\sLanguage\sCrawler|AddThis.com|aiHitBot|Spinn3r|BingPreview|GrapeshotCrawler|CareerBot|ZumBot|ShopWiki|bixocrawler|uMBot|sistrix|linkdexbot|AhrefsBot|archive.org_bot|SeoCheckBot|TurnitinBot|VoilaBot|SearchmetricsBot|Butterfly|Yahoo!|Plukkie|yacybot|trendictionbot|UASlinkChecker|Blekkobot|Wotbox|YioopBot|meanpathbot|TinEye|LuminateBot|FyberSpider|Infohelfer|linkdex.com|Curious\sGeorge|Fetch-Guess|ichiro|MojeekBot|SBSearch|WebThumbnail|socialbm_bot|SemrushBot|Vedma|alexa\ssite\saudit|SEOkicks-Robot|Browsershots|BLEXBot|woriobot|AMZNKAssocBot|Speedy|oBot|HostTracker|OpenWebSpider|WBSearchBot|FacebookExternalHit|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR]

    Reply
  12. Eugen

    Hey guys,

    Recently found it your article and indeed it’s very good and well documented. But still having some questions. First of all, redirecting all bots is not something that could be considered as sneaky redirects or cloaking technique?
    How you deal with this?

    Second, how much reliable is this solution in terms of cost per implementation / benefits in indexing webpages and ranking improvements.

    Thanks.

    Reply
    1. Faouzi EL YAGOUBI

      Thanks for your comment but Google will not consider this solution as cloaking technique because the 2 pages are exactly the same. You prerendred page need to have exact same content!

      Generating pages on your server will not coast a lot if you use REDIS cache like i explain in this article.

      You will get exact same racking compared to normal website. Google will not see the difference.

      I test it for multiple projects and it works fine.

      Reply
      1. Eugen

        Hi Faouzi,

        Thank you for your answer. Yes, probably the right question would be if using screenshots technique is not an infringement of Google guidelines in general and cloaking in particular.
        I think this is more a pre-rendering page technique, but if the page is delivered identical for Googlebot as for humans it makes sense.

        Tell me pls, did you noticed a real benefit using this technique? Do you have any kinda of statistical data, in terms of no. of number of indexed pages before and after? Or ranking improvement for specified keywords with details about the visitors landing page on a page where you used this technique?

        Btw, it looks like after my comment you fixed the layout of this page. 🙂 Good job guys!

        Cheers.

        Reply
        1. Faouzi EL YAGOUBI

          Ther is no real benefit using this technique is exactly same like if you have normal pages indexed. except if you generate fast and prerender the pages before google try to index them like that you improve the speed of indexation. Google like when the server is fast. Prerendering page will help to solve this problem.

          This technique is good for Angular 1.X but using Angular 2.X is no need of this technique. Angular 2 is isomorphic and will have prerendering natively.

          Sorry but I don’t have stats you ask.

          Yes we have some problem on the layout and @konrad fix it 😉 Thanks for your message

          Reply
          1. Eugen

            Thank you @Faouzi for your answer.

            Are the prerender technique working also with phantom.js or is just applicable to Angular.js
            I recently read some documentation from BromBone guys and they suggest that html5 “pushState” urls are the new standard in using js. frameworks.
            Still necessary using a prerender server if use a “pushState” and changing the urls without a page refresh?

            I have identified some sites that are using this “pushState” urls, but doing a quick research I see that Google are still not indexing that pages.

            Do you have any good example of site, with good rankings, that use a prerender server and snapshots?

            Thanks

          2. Faouzi EL YAGOUBI

            Prerender use phantomJs.

            Angular rooting is using pushState when you are in html5 mode. If you are using angular 1.x you need to render the pages to google. Google can sometime prerender pages but others search engines can’t.
            My website http://www.ournia.co is using prerender 😉

  13. ajeet

    Great article but I would like to know the nginx config for the same as I want to setup this with nginx.

    Reply
  14. santhosh

    When i adding this .htaccess file to my blog(resultweek.in), site is not loading & it’s showing Internal server error and google is not crawlers my blog.
    Here I have use the json file but google is not rendering please resolve this issue

    Reply
    1. Faouzi EL YAGOUBI

      If you have error on your .htaccess Google can’t find the good pages. Please fix your htaccess first

      Reply
  15. Eugen

    Faouzi. I had a look over your site and I saw a loot of pages indexed. Looking over some individual pages I don’t localize the title tag within that url is indexed by Google.
    Let’s look over this one: http://www.ournia.co/artist/amar-aldek
    The title tag over the code is only your brand name (different than indexed title: Amar Aldek – Ournia). Second you don’t have a meta description into your code and the sentence that Google uses I can’t find at anywhere into code or front-end.

    The question is how to create a proper on-page optimization (title, meta description) plus content on page using phantom.JS prerender.
    Is there using phantom.JS any limitation for web analytics tools (e.g. Google Analytics, GTM)

    Cheers

    Reply
  16. Eugen

    One more thing Faouzi. I tried to crawl your site now using Screaming frog and only one page has been crawled (root).
    I crawled other site that uses “pushState” and Screaming frog crawled smoothly all pages. Moreover has been indicated both urls: address & “ugly url” using the ?_escaped_fragment_=

    Cheers.

    Reply
  17. amberhina

    am a freelancer most of the time my template got rejected just because of this, and each time they say am missing the tags i did not know about all those tags and importance of those tags thanks admin for the post ….

    Reply
  18. Prateek Gupta

    I have an angualrjs website with prerendering. I am not able to get my site crawled by alexa. Anyone else has same issue.. please advise

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Try DOZ Marketing Platform for Free

Start now

Thinking about your next Marketing move?

Get an instant quote!
Marketing Tools we use!

SEMrush

Web CEO SEO Tools
Marketing Books we read!

Receive DOZ News & Latest Posts

Good stuff from the web!

Get started now! It's free