How much impact can SEO have on the website traffic?

Does SEO really justify the time investement? Yes

This post describes what was done and how much impact SEO had in a corporate website. We went from 9K visits per month to 30K in a 14 months period:

1. Context
2. What was done (SEO steps)

I was hired by Kaizen Institue Consulting Group with the main task of improving their web apps which include their consulting website www.kaizen.com .

There were several tricky parts, first of the company is present in more than 30 countries and each one has their own country page, furthermore the website platform should allow each country to control the content.

When I joined, kaizen was was using typo3, which in most cases is not a good choice, I previously defended that for very static and template fixed pages it would be ok but with what I know today I can't recommend it for any scenario, with the main reason being  typoscript which is just a pain.

So we actually decided to create our own in-house CMS, which should be able to:

• Have a standard content common to all countries which would be localized providing a cohesive experience to users
• Allow online event booking. Kaizen Institute is a consulting group which offers training events worldwide and users should be able to buy and register on those events through the website.
• Everything should be translatable, if we had a registration form then it should be possible to translate the different input fields as well as adding new ones country specific.
• There should be a success stories database where each country could share their most valuable case studies, filterable by industry type or method used.
• Finally a blog system should also be in place to create long articles not suitable for the main website.

On top of all of these features the website should be search engine optimized and I thought:

"Ok, let's keep SEO in mind while developing the platform but considering that typo3 is being used and that it's a rather well known CMS we probably won't see a considerable improvement". 

But we did, in fact a year after we migrated to the new in-house system we are seeing more thean triple the monthly traffic that we had, from 9 000 monthly visits to 30 000 and we actually decreased the adwords budget.

So what changes were done.

To make this as brief as possible I'm going to enumerate them without go into details but if you would like to know more please feel free to leave a comment:

• Cleaner code. This was a simple side effect of dropping typo3 which adds a lot of garbage to the HTML.
• Multi language signals to Google. To prevent each website to "fight" against each other for search engine index we used the x-lang href element to indicate that there was another language version of the given page and obviously we set the target country in Google webmasters tools.
• Schema.org usage: considering event booking is a big part of the website it made sense to use the schema.org Event element. Using this data structure allows google to show events descriptions like the ones below directly in the SERs (search engine results):
  
• Flexible URL management including complex redirects: We actually don't use apache redirect rules instead everything is catched by php and then based on "starts with" and key map behavior the redirect is done. This is great when you are doing 301 or 302 and don't underestimate their relevance. For instance every time we change a page on the sitemap we always do 301 from the old URL to the new one, preventing shared links to be 404. The URL structure is also easy to parse by Search Engines and directly change by users, actually we almost never use query parameters or pages ids, the URLs are always relevant keywords .
• Keep it fresh: content freshness is good for SEO because it's a good signal of up to date pages and relevant material. The content management system role in this point is removing any technical obstacle and make it as easy as possible to add new content. Actually the tricky part on this one is getting quality content on a regular bases and to solve this issue we setup a reward system. Because Kaizen Institute is a consulting group we knew the best content would come directly from the consultant who are in the Gemba (working place) based on how they helped organizations achieving continuous improvement and what were the lessons learned. So, in order to share these amazing experiences through our online presence (website + social media) a points system was set in place in order to reward the best article writers. Consultants would work hard to have quality and on time articles which had a double positive effect, good examples to showcase to existing clients and great material to increase web awareness and attract new ones.
• Social Media: well this one is quite self explanatory, use social media to spread the word and even have direct interaction with existing or possible clients. To speed and improve our social media presence we use "buffer app" which allows  us to submit  to several social media websites (Kaizen Facebook page, g+, twitter, linkedin, etc) with a single click and at scheduled times (increasing visibility).

This was our one year rode to achieve triple the visits, all comes to good and regular content along with SEO. Our next step will be extending our blog responsive layout to the main corporate website.

How to run any command on php as root

Running sudo root commands in php

edit the sudoers file (vi /etc/sudoers) and then under "#User priviligie specification" (where it should have root    ALL=(ALL:ALL) ALL) add:

www-data ALL=NOPASSWD: path_to_command

example for chown:

www-data ALL=NOPASSWD: /bin/chown

then inside php run using:

exec("sudo -u root command);

example for chown:

exec("sudo -u root chown -R www-data /directory");

This is particular usefull if you are trying to edit/delete a file/folder that is owned by root through php. Meaning solution for:

unlink Permission denied

How to prevent spam from website contact form

A simple approach without negative usability impact

Website "Contact Us" forms are pretty common and quite useful to provide users with a way to quickly get in touch with the other side. The issue is that they also might be a source of unwelcome spam emails.

During the development of the new website for Kaizen Institute Consulting Group I had to find a way of stopping the spam messages we were getting due to the contact form used at the time. They weren't few because the website was a 11K+ per month hits (now is more like 15K).

So the first thing that crossed my mind was CAPTCHA but this always has a negative impact in usability as you force the user to make an extra effort to get in touch with you and that's never a good idea. So I wanted something that was completely transparent for the user. Thus, I did two small things:

• Add an input field with a class that had {display:none;} on the css file (not as an element style). If that field had content on the POST parameters the server would simply drop the request.
• Forced the from submission to be done through AJAX (when doing the submission by AJAX I would add a POST parameter indicating it was an AJAX request and checked that on the server side)

The first is a very well known way to do it but I've actually seen people saying to add the {display:none;} to the element style attribute which isn't very smart. We actually had this issue on our website. We tested directly on the element and on the css file and in the first scenario we got spam but never on the other.

The second has a theoretical downside: "What if the user doesn't have the JavaScript enabled ?", I'll be blunt on this:

• Firstly the "You always need to fallback to pure HTML if the user doesn't have JS activated" requirement makes very little sense nowadays, unless we are talking about SEO. Designing the website in a way where some pages are only accessible to users with JS enabled is not a cleaver move because search engines might have issues with javascript. Not doing AJAX because the user might turned off JS is stupid.
• Secondly, the type of user that visits our website (we are a consulting company) isn't probably worried with JS being activated or not.

So to sum up, before adding a CAPTCHA or doing something more complex just try to add a class to the element, hide it on the css file and make the form submission through AJAX.

SEO when changing CMS or page structure

What SEO steps should be followed to prevent negative impact when changing CMS and/or site structure ?

This subject will be divided on the following sections:

1. My scenario
2. SEO steps when designing the new website
   
   • SEO and social media integration
   • Ajax and SEO
3. The results
4. How We did it
• SEO: Changing CMS
• SEO: Changing site structure
5. Putting the website online

My scenario

This section introduces the reason for changing the website structure and the CMS and isn't required to understand the technical steps taken.

I was hired by a consulting company called Kaizen Institute Consulting Group and my main goal was to turn an outdated website in a clean, well designed and easy to manage system. At the time, all countries were using a TYPO3 instance as the CMS of choice and because I was fresh out from college and didn't know much about the system I started studying it.

After several experiments, reading a lot of documentation and talking with typo3 users I understood the following:

If your goal is to have a responsive, flexible, feature rich and customized website then forget TYPO3. The documentation is awful, it's based on one of the most stupid scripting languages (typo script, which basically forces you to memorize non-logical stuff) and has a terrible customization architecture based on extensions.

The only scenario I see typo3 to be an option is for multi-language websites (based on a sub domain approach), where most of the pages are static and the main goal is to display non-Ajax content.

But for our website we had ambitious requirements:

• Rich online event booking system that would allow each country to specify different forms for their participants with deep Ajax filtering options and connection with our existing booking management system
• Online payment system
• A complex success stories data base
• Complete control of the crawlable HTML code
• Effective spam controlling

When we start understanding the hours required to study and implement this using TYPO3 extension system, Kaizen Institute IT Leader had the courage to say "We need to drop typo3 and go with something different. Let's build our own CMS". This was at a time when I actually had a redesigned and improved version of the website running on an updated TYPO3 instance which took approx a month of development and deadlines were around the corner. So, without a moment to lose we started developing a new CMS tailored to our needs but flexible enough that would be easy to add new features.

After one month and a half, we had a custom backed CMS and a fronted. It's worth mentioning that we are a two man team and while I ended up implementing most of the system, due to improvements that needed to be made on our internal project managing system of which my colleague was considerable savvier than me, my time wasn't dedicated only to development as I had to provide support to the 25 counties Kaizen Institute has offices in, which amounts to 400+ employees.

Because we were aiming for global standard content that would be localized by each country, it was time to align the different visions from each country regarding the website content. When we finished implementing the improvements it was time to submit to the multiple search engines and hope we had done a solid SEO job.

SEO steps when designing the new website

SEO steps have flooded the internet and most of them give you pretty straight forward directives but don't seem to be supported by actual facts, being, in my opinion, just someone's else opinion. So I've put together a SEO list based on facts.

Then I follow a SEO checklist in order to create search engine optimized content, which is based in my opinion, what I've read and common sense.

SEO and social media integration

Although social media linking isn't as powerful as a reference from a reputable web site, it helps "spread the word" and improve branding. As such, and because we would have a couple of douzen countries creating content for the website, we wanted to harvest all that content and share it with others through social media, more precisely twitter, Facebook and LinkedIn.

Optimizing social media posting through scheduling

Until then I was the main responsible for handling Kaizen twitter and Facebook accounts, every time we had some relevant news like events promotions or global benchmark tours I'd posted it. There was one issue though, because I wasn't on the same time region as many of our potential clients, the posts weren't' being shared at the optimal time (target group was sleeping).

To fix this issue I started using the social media scheduling app called "bufferap" which allowed me to set when my posts should actually be posted on the social media networks. I set my posting schedule based on this study.

I wanted to give the same option to our CMS users, this way they could add a new Kaizen Institute success story to the website at 2 in the morning but instead of being lost among other posts during the night it would "freshly" popup at 8 am during a potential client breakfast twitter reading.

So we added this feature to the CMS. Whenever an user adds a news to the website it only takes a click on the option "share" to send it to Facebook, twitter or linked in (a preview is displayed to the user and the twitter version is shorten). During the content preview the user has the option to share it immediately or saving it on the "buffer" to be posted later. This feature was implemented using bufferapp's straight forward api.

Ajax and SEO

We added a rather complex but intuitive lean events calendar which is filtered and then update by Ajax.

Ajax is great for fast and real time behavior but has the downside of not being as SEO friendly as static content. So we wanted to achieve two things when using Ajax:

• Give a faster and more app-like experience
• Keep history behavior. 

Ajax browser history mean that three things needed to happen when a user would, for example, change the Lean Events filter type to only display tours:

1. The URL would change to the same state as if the user wasn't requesting by Ajax
2. The content was loaded dynamicly using Ajax
3. If the user pressed the browser back button it should take him to the state before applying the refered filter

To achieve this we used jQuery Address and it was applied in our success stories list and our events calendar (try changing the filters or browsing the success stories and then hitting the browser back button).

SEO Results

Our main goal was to redesign the website by giving it a more professional, global, clean and fresh look. So we came from this:

Kaizen Institute Website Old Layout

to this:

But we wanted to keep and if possible even improve the website visits.

When making a drastic change like ours and considering a lot of our traffic comes from organic search, it's normal for the hits to decrease but it in our case they actually increased and considerably (we made the transition on the morning of September 18th).

The visits:

Our visits increased about 57% and added to that the bounce rate also improved:

The bonce rate (lower is better):

Our bonce rate went from 73.24% to 47.8% meaning people would visit more pages inside our website instead of leaving it.

Pages / Visit

Our pages per visit went from 1.73 to 5.74 in the day of moving.

In my opinion this means we took the necessary SEO steps to at least allow the website improvements to overcome any negative search index impact.

How we did it?

SEO and changing CMS

Changing CMS, if done properly, shouldn’t have a negative impact in search ranking. The following guidelines are what to expect for a Search Engine friendly CMS:

Customizable page title, page description and menu names. The user should have full control over the page title and description meta tags

URLs Customization. Having keyword rich URLs is a good SEO practice and as such the CMS should allow to define a page’s url. It’s better to have www.kaizen.com/about-us/kaizen-institute.html then www.kaizen.com/index.php?id=22.

The HTML generated by the CMS should be clean and compliant to the W3C standard (valid html).

A considerable plus would be allowing to highlight data throught the schema.org rich snipets (more info). In our case we used it to mark our Lean Events

SEO and changing Website Structure

Ok this is the tricky part. First things first, changing website structure will have search index impact and you should do it if you are sure it’s a needed improvement and that in the long run it's a better fit for your business. I'm not saying don't do it, I'm saying this is not a decision to take lightly (SEO wise) and as such it should be done with commitment in mind, not something that will be changed every other month.

The goal of these steps is to prevent a simple but critical event, the 404 error. When you change the website structure Google might show a search result with an URL to a page that either doesn't exist anymore or has a new URL. If the user follows that link a page giving the “Not found” error is displayed and Google eventually will also get the same error when refreshing its index. When that happens the page is removed from Google’s index and all the page rank owned by that now extinct page will be lost, resulting in a decreased search index position.

The steps we took were the following:

1. Write down the fundamental structure of the new website
2. Check in Google analytics the top visited pages in the present website
3. From those see if it makes sense to include them in the new structure. If some of the old pages really attract a lot of hits then consider at least keeping them at the beginning and then phase them out.
4. Check with Google analytics or another tool the most famous back links (referrals) and take note of the ones that were removed and shouldn't be part of the new structure.
5. If possible change the referral URLs. All of the back links pointing to your website should be changed to the new URL if possible (which usually isn’t).
6. Do 301 permanent redirects. The previous steps were in preparation for this one, all of the URLs that aren't part of the new structure should return a HTTP 301 header with the new URL. This tells search engine that the content on that page was moved permanently to a new URL, so instead of getting a 404 error Google thinks “the content isn't gone it just changed pages”. Based in Matt Cuts videos, 301 shouldn't have a negative impact in page rank compared to normal links.
   I.E: if you access www.kaizen.com/kaizen-events.html the browser will be redirected to www.kaizen.com/events.html due to a 301 response.
7. Parse the website for link errors. Use a tool like Xenu which gives a report about links that are pointing to un-existing pages.

At the this point the new structure should go live. If the new website is on a new server then change the DNS and lower the TTL (time to leave).

It's now time to ask Google to crawl the website and check if it's accessing the new structure. To do that login into webmaster tools and chose the option "Cawl - Fetch as Google"and take a look into the resulting HTML. Take the opportunity to submit to Google index.

To speed up Google knowledge about the website consider submitting a sitemap, the recommended schema is to have it under www.domain.com/sitemap.xml.

Do the same for Microsoft Bing and that's it.

So to summarize the crucial steps of changing a website structure are:

• Consider the possibly of keeping the most "famous" links
• 301 extinct or moved links, at least the crucial ones
• Check for broken links
• Submit to webmaster tools and check regularly for errors

If you have any questions please let me know through the comments section.