AngularJS is a good framework for building websites and apps because it makes them much faster and richer for users.

    But there’s one problem every developer faces when pushing their AngularJS product: Search Engines Optimization, or SEO.

    Quality SEO means getting found amidst all the noise online. When a site or app is optimized for search it is more likely to be found by prospective users. If it is not optimized, then a developer might as well be screaming at the wind – no exposure and no users are almost guarantees.

    Right now the most common techniques for mitigating this problem is to either build separate versions of the app or website for search engines, or to pre-render pages on the server. The problem with these solutions, however, is that you have to maintain two separate systems: one for your users and another for Google and the other search engines.

    While Google has attempted to help developers by improving its capability to index JavaScript and CSS but even if Google’s efforts to index JavaScript since 2014 have advanced, it doesn’t mean that your app will be indexed properly.  Indeed, Google still recommends creating snapshots to make an AJAX application crawlable.

    But how exactly are these snapshots created? And how can a developer be sure that their AngularJS app or website is correctly and completely indexed by Google?

    In this post we present a free and self hosted solution to generate snapshots and to make sure that your AngularJS website or application crawlable by, indexed by, and optimized for Google.

    [freebiesub title=”these instructions and all the code in a PDF” download=”https://www.doz.com/wp-content/uploads/2016/03/DOZ-Angular-JS.pdf”]

    AngularJS and SEO: The Problem

    Search engines crawlers were originally designed to index the HTML content of web pages.

    Today, however, JavaScript and other frameworks like AngularJS and BackboneJS  are playing a leading role in web development and the creation of content and application online.

    Unfortunately, the crawlers and the other indexing mechanisms behind search engines remain decidedly unfriendly to JavaScript powered sites.

    angularjs-js-code

    AngularJS and SEO: The Solution

    Overcoming the indexing problem is not difficult when developers embrace what are called ‘snapshots’.

    Snapshots is a term used to refer to content generated for the search engine crawlers on the website’s backend. The idea behind snapshots is that the developer does the work for the crawler that it cannot or doesn’t want to do on it’s own. Optimizing and caching snapshots not only help you get indexed, but also improves significantly the speed of indexation.

    An important note: JavaScript indexation currently only applies to Google’s crawler. Other search crawlers (such as those from Microsoft’s Bing search engine) do not support crawling JavaScript applications yet. As well, despite web content being increasingly shared to social networks like Facebook and Twitter, most social network crawlers don’t handle JavaScript either.

    So how do you generate snapshots, and how do you work with them to make sure you are indexed?

    Read on for the step-by-step guide.

    Step One: Generate Snapshots

    The first step is to generate the snapshots themselves.

    To do this we need access to a snapshot server based on a headless browser such as PhantomJS or ZombieJS. In this example we will use the open source middleware Prerender that already packages PhantomJS and is ready to handle our special crawler requests and serve HTML snapshots.

    In order to reduce the time, it takes to generate snapshots a cache can be employed. Snapshots are cached on a Redis Server the first time they are requested, and then re-cached once a day (note: this can be manually configured to suit your needs) to make sure the content stays up-to-date.  As a result, a static snapshot is always and instantly available to be served to the crawler.

    angularjs-map

    Step 2: Server Installation

    In this example we will use an Apache server run on Ubuntu 14.04.2 LTS.

    There are five sub-steps to work through here.

    1 – Install NPM and NodeJS

    [snippet id=”727″]

    2 – Install Forever

    [snippet id=”728″]

    3 – Install  and Start Prerender.io

    [snippet id=”729″]

    Make sure the server starts on 4001 and that PhantomJS is on 4002.

    You can edit this file if you want to change the port:

    [snippet id=”730″]

    Return to the Prerender folder and start the server using forever – this will help to start the server continuously in the background.

    [snippet id=”731″]

    4 – Install Redis server

    Add the Dotdeb repositories to your APT sources. To do this, create a new list file in /etc/apt/sources.list.d/ and fill it with the following content:

    [snippet id=”732″]

    Then you need to authenticate these repositories using their public key:

    [snippet id=”733″]

    Next, install Redis using apt-get:

    [snippet id=”734″]

    Then enable the Redis service to start on boot:

    [snippet id=”735″]

    You should then check the Redis status:

    [snippet id=”736″]

    You will get “PONG” if everything is ok.

    5 – Make Prerender use the Redis server to cache snapshots

    Prerender has an open source module, Prerender-Redis-Cache, that makes it easy to perform this task.

    In your local prerender project ( prerender/server.js) run:

    [snippet id=”737″]

    Then add these two lines in prerender/server.js :

    [snippet id=”738″]

    Restart Prerender by:

    [snippet id=”739″]

    And if you want to clean all REDIS cache you can use:

    [snippet id=”740″]

    Step 3: Server Configuration

    Now we will redirect crawlers to the local Prerender server using a simple .htaccess file.

    This htaccess file have contain all the redirect configurations. Note that the .htaccess file needs to be in same directory with your main AngularJS index.html file.

    [snippet id=”741″]

    You have now finished all server side installation tasks, so it’s now time to configure the AngularJS App.

    Step 4: App Configuration

    First open you Angularjs index.html file and:

    1. make sure you have <base href=”/”> before </head>
    2. add <meta name=”fragment” content=”!”> between <head></head> (by adding this tag into the page www.example.com, the crawler will temporarily link this URL to www.example.com?_escaped_fragment_= and will request this from your server)

    Second, activate HTML5 mode.

    In your config.js file add:

    [snippet id=”742″]

    This will tell your application to use HTML5 URL format.

    URLs typically look like http://www.example.com/directory/page. By default AngularJS elements will have URLs like this: http://www.example.com/#!/directory/page

    [snippet id=”743″]

    Third, you need to manage the meta tags.

    To improve the SEO of your app or website you need to have a unique title and description for each page. An AngularJS module called AngularJS-View-Head already exists to fix this problem. This module will help us to change the HTML title and head elements on a per-view basis.

    How do you work this in practice?

    Start by installing this module using bower.

    Next,  declare the module as a dependency of your application:

    [snippet id=”744″]

    This makes available the directives described in your HTML template.

    Finally add the meta tags inside your template.

    [snippet id=”745″]

    Step 5: Test Prerender Server

    If you’ve followed all of the steps things should be working well. However, better safe than sorry, it’s time to test.

    Compare the source of one of your pages with and without _escaped_fragment_ in the URL.

    You can check specific routes in your browser and compare :

    http://www.example.com/directory/page
    http://www.example.com/directory/page?_escaped_fragment_=

    Step 6: Add a Sitemap

    The final step in your AngularJS SEO strategy is to develop a sitemap.

    To help search engines crawl your app or website and make sure pages are indexed quickly you must create a sitemap for all of your routes. This sounds difficult to do, however, with proper build processes this can be automated using a tool like Grunt.

    (Heads up: we’ll be publishing another post soon explaining just how to automate a sitemap using Grunt!)

    Conclusion

    Making sure search engines and – through searches – users can find your app or website is essential. The strategy presented in this post is quick to implement, scalable and easy to maintain and, where employed, should help you make the connections you need and win the users you want.

    Share.

    34 Comments

    1. Hi, probably a very confused question coming up right now, but when installed locally, how does it serve snapshots on the site/in production? Thanks for a great article, sorry about the ignorance.

    2. Faouzi EL YAGOUBI on

      Hi,
      Yes it will serve snapshots on the production server for search engines. Normal visitors will see normal app but search engines engines will see prerendred version. The pages will be the same in the content and design so it will respect search engine roles.

      Regards

    3. Faouzi EL YAGOUBI on

      You can optimize your app by following this steps :
      * Setup Grunt tasks to
      – Compile AngularJS templates from multiples files to one file
      – Minify and crompress JS, CSS, HTML
      – Combine all JS files to one file
      * Clean unused javascript and css code to reduce size of your files
      * Use prerender solution described in my article
      * Use redis Cache to make rendering faster
      * Use CDN and good hosting to make your app faster

    4. Your tutorial seems helpfull, although I struggled a hell of a lot to follow it. This guide would help someone who has experience and knowledge about how ubuntu works and where everything is located.

      For example. I had no clue what APT was or how to create a list in that folder, had to get external resources to find out.
      Also for many of the steps, you dont specify from what folder on the terminal they should be run….”sudo service redis_6379 start” did not work at all – from any folder.
      Where was I supposed to run the “install redis server” command? in the prerender directory? (That is where the previous step left off).

      These are just examples as on quite a few places I simply had to guess the step in between.

      I am not calling you out for being useless, im sure this is helpfull to many people, but for users who dont really have much experience developing on linux – this is extremely difficult to follow.
      If you want to make a step-by-step tutorial then you need to take into account just that, you cant simply ommit every second step and expect the end user to know that they need to now go back to the root directory or what APT is for that matter. Step-by-step tutorials are meant to dumb things down to the simplest level possible. This is more of a summarised guide or walkthrough, but not a step by step guide.

      I struggled for some hours now with this and have gotten nowhere.
      Either way, thanks for the effort, but it could have been a bit more in depth and aybe, just maybe I would have been able to get to the end.

      • Faouzi EL YAGOUBI on

        Sorry about that but this article require some knowledge about server management.
        – Advanced Packaging Tool (APT) perform such functions as installation of new software packages, upgrade of existing software packages, updating of the package list index, and even upgrading the entire Ubuntu system. APT command it is accessible by default in Ubuntu and you don’t need to install it.
        – ”sudo service redis_6379 start” can be run in any folder if you have install correctly redis. Please follow this tutorial to install redis https://www.linode.com/docs/databases/redis/redis-on-ubuntu-12-04-precise-pangolin

    5. Great post, Faouzi, I discovered this just in time. Thanks for sharing. I was wondering, have you published the post on using grunt to re/generate sitemap.xml yet? Otherwise, could you simply point me in the right direction to look? Thanks.

      • Faouzi EL YAGOUBI on

        Thanks Yemi, I don’t get time to finish the article about sitemap.xml but it will be published soon. I will let you know.

    6. I’m newbie to angularjs, and I’m using angularjs and yii2 framework and try to follow your guidance step by step, on step 3 htaccess configuration. we have only html files in template folder. Main file : /frontend/web/index.php, in which folder should I put this htaccess and is it need modify htccess configuration ? Thanks

      • Faouzi EL YAGOUBI on

        You need to create .htaccess file in the same folder of index.php (file : /frontend/web/ )

    7. I tried it but it is not working on different CARDS like Twitter Card, Facebook and Google plus.
      When we type a link on these websites, they crawl the page and show the preview but with Angularjs it is showing {{page.title}}, {{page.description}} etc.

    8. You need to add the Google and Twitter social bot in the list
      RewriteCond %{HTTP_USER_AGENT} Googlebot|bingbot|Googlebot-Mobile|Baiduspider|Yahoo|YahooSeeker|DoCoMo|Twitterbot|TweetmemeBot|Twikle|Netseer|Daumoa|SeznamBot|Ezooms|MSNBot|Exabot|MJ12bot|sogou\sspider|YandexBot|bitlybot|ia_archiver|proximic|spbot|ChangeDetection|NaverBot|MetaJobBot|magpie-crawler|Genieo\sWeb\sfilter|Qualidator.com\sBot|Woko|Vagabondo|360Spider|ExB\sLanguage\sCrawler|AddThis.com|aiHitBot|Spinn3r|BingPreview|GrapeshotCrawler|CareerBot|ZumBot|ShopWiki|bixocrawler|uMBot|sistrix|linkdexbot|AhrefsBot|archive.org_bot|SeoCheckBot|TurnitinBot|VoilaBot|SearchmetricsBot|Butterfly|Yahoo!|Plukkie|yacybot|trendictionbot|UASlinkChecker|Blekkobot|Wotbox|YioopBot|meanpathbot|TinEye|LuminateBot|FyberSpider|Infohelfer|linkdex.com|Curious\sGeorge|Fetch-Guess|ichiro|MojeekBot|SBSearch|WebThumbnail|socialbm_bot|SemrushBot|Vedma|alexa\ssite\saudit|SEOkicks-Robot|Browsershots|BLEXBot|woriobot|AMZNKAssocBot|Speedy|oBot|HostTracker|OpenWebSpider|WBSearchBot|FacebookExternalHit|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR]

    9. Hey guys,

      Recently found it your article and indeed it’s very good and well documented. But still having some questions. First of all, redirecting all bots is not something that could be considered as sneaky redirects or cloaking technique?
      How you deal with this?

      Second, how much reliable is this solution in terms of cost per implementation / benefits in indexing webpages and ranking improvements.

      Thanks.

      • Faouzi EL YAGOUBI on

        Thanks for your comment but Google will not consider this solution as cloaking technique because the 2 pages are exactly the same. You prerendred page need to have exact same content!

        Generating pages on your server will not coast a lot if you use REDIS cache like i explain in this article.

        You will get exact same racking compared to normal website. Google will not see the difference.

        I test it for multiple projects and it works fine.

        • Hi Faouzi,

          Thank you for your answer. Yes, probably the right question would be if using screenshots technique is not an infringement of Google guidelines in general and cloaking in particular.
          I think this is more a pre-rendering page technique, but if the page is delivered identical for Googlebot as for humans it makes sense.

          Tell me pls, did you noticed a real benefit using this technique? Do you have any kinda of statistical data, in terms of no. of number of indexed pages before and after? Or ranking improvement for specified keywords with details about the visitors landing page on a page where you used this technique?

          Btw, it looks like after my comment you fixed the layout of this page. 🙂 Good job guys!

          Cheers.

          • Faouzi EL YAGOUBI on

            Ther is no real benefit using this technique is exactly same like if you have normal pages indexed. except if you generate fast and prerender the pages before google try to index them like that you improve the speed of indexation. Google like when the server is fast. Prerendering page will help to solve this problem.

            This technique is good for Angular 1.X but using Angular 2.X is no need of this technique. Angular 2 is isomorphic and will have prerendering natively.

            Sorry but I don’t have stats you ask.

            Yes we have some problem on the layout and @konrad fix it 😉 Thanks for your message

            • Thank you @Faouzi for your answer.

              Are the prerender technique working also with phantom.js or is just applicable to Angular.js
              I recently read some documentation from BromBone guys and they suggest that html5 “pushState” urls are the new standard in using js. frameworks.
              Still necessary using a prerender server if use a “pushState” and changing the urls without a page refresh?

              I have identified some sites that are using this “pushState” urls, but doing a quick research I see that Google are still not indexing that pages.

              Do you have any good example of site, with good rankings, that use a prerender server and snapshots?

              Thanks

            • Faouzi EL YAGOUBI on

              Prerender use phantomJs.

              Angular rooting is using pushState when you are in html5 mode. If you are using angular 1.x you need to render the pages to google. Google can sometime prerender pages but others search engines can’t.
              My website http://www.ournia.co is using prerender 😉

    10. When i adding this .htaccess file to my blog(resultweek.in), site is not loading & it’s showing Internal server error and google is not crawlers my blog.
      Here I have use the json file but google is not rendering please resolve this issue

      • Faouzi EL YAGOUBI on

        If you have error on your .htaccess Google can’t find the good pages. Please fix your htaccess first

    11. Faouzi. I had a look over your site and I saw a loot of pages indexed. Looking over some individual pages I don’t localize the title tag within that url is indexed by Google.
      Let’s look over this one: http://www.ournia.co/artist/amar-aldek
      The title tag over the code is only your brand name (different than indexed title: Amar Aldek – Ournia). Second you don’t have a meta description into your code and the sentence that Google uses I can’t find at anywhere into code or front-end.

      The question is how to create a proper on-page optimization (title, meta description) plus content on page using phantom.JS prerender.
      Is there using phantom.JS any limitation for web analytics tools (e.g. Google Analytics, GTM)

      Cheers

    12. One more thing Faouzi. I tried to crawl your site now using Screaming frog and only one page has been crawled (root).
      I crawled other site that uses “pushState” and Screaming frog crawled smoothly all pages. Moreover has been indicated both urls: address & “ugly url” using the ?_escaped_fragment_=

      Cheers.

    13. am a freelancer most of the time my template got rejected just because of this, and each time they say am missing the tags i did not know about all those tags and importance of those tags thanks admin for the post ….

    14. Prateek Gupta on

      I have an angualrjs website with prerendering. I am not able to get my site crawled by alexa. Anyone else has same issue.. please advise

    Leave A Reply