High availability in Magento farm: the session problem

Big Magento websites require sometimes to be able to respond to a high traffic: there are many websites with high load, and sometimes, we must propose a magento farm behind a load balancer to be able to respond to all visits. But in this case, we have a choice to do about the load balancing: how is dispatched the load between Magento webservers?

And this choice that we have to make will have an impact about how sessions are managed

Depending on how are configured load balancers, we need perhaps share session between Magento front-webservers

Impact of load balancers configuration on the session storage management

Load balancers are in front of our webservers to be able to manage our webserver load

There are two ways to share the load between each front in load balancer when a customer is visiting a website:

  • he is always sent to the same webserver
  • According to the load, he is sent to the less loaded webserver

This load balancer configuration choice has an impact on the session management:

in the first case, we are always redirect to the same webserver.. But in this case, if the load of this server change we’ll have an end-user usage not as well as expected. In the worst case, if this websever become unavailable we wouldn’t be able to access the website anymore and the consequence is, and so loose some sales…

In the other case, we have not these disagrements because load is detected on load balancers and we are redirect to the most available server; but in this case we have the problem to share our sessions between each front to be able to retrieve our personals informations.

To ensure that we have the highest availability for our website, we’ll choose the second option and will see what solutions we have to share sessions between front web-servers

Reminder on PHP sessions storages

By default, PHP uses a session storage based on files: every time you start a session a file will be written on the webserver with the session content. it’s nice, but it has the disavantage to depend on which webserver you requested the information. If we are redirected to another webserver, the session file does not exist, and so, you’ve lost our cart…. How can we solve that?

PHP allows to redefine our PHP session storage model and define how are they managed. Nice, we’ll see how it can help us in our webserver farm management and our session sharing problem

The possible ways to share sessions in a Magento farm

Latest Magento release (1.6.2 CE, and related enterprise one) embeds many sessions storages management:

  • Files, the PHP default’s one
  • Database
  • Memcached
  • Othes choices?

Which advantage(s) have each one?

Sharing Magento sessions in a Magento farm with database(s)

In this case, we remplace the native PHP session storage management by a database: all sessions are stored in our database(s) server(s).

This solution is quite nice because each of our front webservers must be able to read and write in database; so there is no problem to share session: visitor will be redirected initially to a front webserver, init it session on it and save it’s customer session in database. Next page he request, load balancer will perhaps redirect him to another front-webserver, but with it session id, he’ll retrieve session data from the database and so, keep the same informations. Nice

But the problem in this situation is that we add another load on database server. Each visitor will make read and write instructions on our database server, and so, according to our traffic, it could be a very huge load. Furthermore, even if we add a very good cache policy, if our database webserver go down, our website won’t be available anymore.

Magento database table where are stored the sessions data is based upon file system. So If our server go down, we’ll be able to restore our session data. If Magento database availability is critical, there is some solutions like MySQL cluster to ensure that MySQL is always available. So we are able to provide a maximum service and a model tolerant to failure. But this model has the big problem to provide a important load on the database and it could be interesting to check the others availables solutionss

Sharing Magento sessions in a Magento farm with Memcached

Memcached is a memory server.

It can be use to:

  • Share cache data
  • Share sessions

One of the main advantage of memcached is that it’s a memory server, and so, read and write instructions will be made faster than with a file system

But we have two disavantages to use a memcached to store sessions:

  • Data integrity: memory does not check write as it’s done in a file system: writing in memory is not as secure as writing in a file system. Our data can be compromised, and so, loose part of our data.
  • We have the same problem as in the database: memcached is not tolerant to failure and so you’ll loose everything if it goes down; memcached can be clustered for cache storage, but in case of session sharing, we must provide only one server URI. So if server which manage session storage hang up, there is any session service available…

So for now, memcached has the only advantage to be able to share our sessions faster; it is not tolerant to failure and cannot be clustered so this is another risk in our architecture

Sharing sessions using a network shared folder and file system PHP storage

To avoid data integrity problem, we could continue to use a session file system storage, but share them between each front through a shared folder: in this case, all sessions storages location will be known by each server, and so, we can have the ability to use our session on each webserver. Data are also checked on write operations by the file system, so it’s nice

But this model has the disavantage to impose a network load every time you access a session. According to our website load, this network additionnal load could be critical

Sharing Magento sessions with Zend server

Based upon the PHP session storage file model, Zend Server embeds a session clustering between each front webserver zend server instance: Every starting session is copied on each Zend Server. So every time someone request a page from our webserver, it’ll be able to access our session file. Nice

Furthermore, when some data are written in session, Zend Server knows which is the main session storage file, and so is able to write in it.

If a front webserver go down, another session copy is used as main session file, and so you have no disagreement

For now, Zend Server session clustering is the only way to garantee that you won’t have any go down of our website and it provides data integrity using file system storage write control. But one of the disavantage is you need a Zend Server on each Magento front-office, and it increases our webhosting cost

Conclusion about the high availability storage management

Sales, television advertising or simply very basic traffic can lead you to have a dedicated magento farm to be able to answer all visitors requests. In this case, you should take care of always be able to answer all requests, not only for your orders but also for your branding

For sure, you can configure load balancing to always redirect a customer to the same server. This is the easyest way to configure load management, but is not the most reliable model and can provide some disagremments to our clients with possible no webservers response.

In the other case, the only way to ensure that we’ll have a high availability in a Magento farm is to ensure that the load is dispatched on front-webservers depending on their load. This brings the problem to share session between each webserver, but for now, the only secure way is to use Zend Server to clustering session.This model provide us a secure way to share session, ensure acces and write control, and does not add additionnal critical point in our architecture

Furthermore, it seems that choice of storage management has no performance impact. This is a first test, and we don’t know how it has been managed, but in the case mentionned, it seems that session_storage has no or few influence upon the performance

Some best practices to manage templates with the magento template loading API

I presented you a few days ago an introduction on how are built the Magento pages. Now we we’ll see how the templates files are loaded since Magento 1.4 with the template loading API and how we can use this loading API to reduce our work with Magento.

Reminder: how are defined the templates path in a layout file?

Templates are defined in the layout files in a couple block / template with a relative path like this:

<block type="module_identifier/class_suffix" template="relative/path/to/template/file/in/the/template/design/subfolder.phtml"/>

For example, here’s the definition of the block / template that defines the search form field available in your catalog pages:

So, how does Magento find the full path for the relative path defined in this structure?

Magento template loading API available since Magento community edition 1.4 and above (Enterprise 1.7 and above)

Since Magento 1.4.x, when a page is requested, Magento analyzes all couples block / template that compose it. Afterthat, Magento will look for the full path of the template file in the following folders:

  • 1. app/design/%context%/%design%/%theme%/template
  • 2. app/design/%context%/%design%/default/template
  • 3. app/design/%context%/base/default/template

Magento takes the first file found and use it.

What happens if the file is not found? By default, if template file is not found in these folders:

  • No output is renderering
  • A trace in log will be written if log is enabled

This loading API explains why we can define easily some temporary themes for your site like for christmas, summer, sales…: after setting up a new theme, you must only update some elements in your template in a new theme template folder

This is this loading API which allow Magento to drive some merchandising design on your website

We will now see how we can work more efficiantly whis this loading template API

Some best practices when building a new Magento design and theme

Copy paste the default theme when building your package?

When looking for the template file in the folders mentionned before, Magento runs a file_exists. Even if copy paste is the devil, when you build your package, duplicate content will avoid many file_exists when rendering your page, so it could be interesting to duplicate the base/default theme

But one of the drawback of duplicate base/default content will appear during magento upgrade: you won’t benefit of the new templates design for templates you have not overloaded: after upgrading the magento source code: do you really want to check if all template have changed? Or keep your own one and do not use the new functionnalities?

Today, I haven’t seen any limit of the file_exists loading. There are many others points to check before this performance option to make magento faster…

New functionnality in Magento: in base/default !!!!

Another common workaround in Magento project is that we can hear that it’s strictly forbidden to write in the in base/default template folder. I never understood why: if you run in multiple stores model and have specific packages defined for each store, this is the only way you have to share templates between each store. In case we do not want to write in base/default, the only way that we have to share template is to copy paste it in each package/default theme… wonderfull 🙂

Furthermore, if you want to share your module in Magento connect, this is the only way you have to avoid requiring a post configuration by copying the embedded template files in the current package folder (modules should be installed through only one click)

How organize your new templates file in your theme?

When we define a new couple block / template, we provide a relative path to define the related template to load. As a best practice, I cluster all my templates in a subfolder like %namespace%/%module%

This best practice has the following advantages:

  • When you develop your module, this is quicker to find it in the directory tree
  • If you share your module in Magento connect, this reduce the risk that other modules will have already existing templates in directory tree

Be lazy when you work with Magento templates!

I’ve made hosting for one of the biggest french bank and I liked this work: check everything is ok or be able to solve problem quickly and easily with the tools you made. Magento loading API is able to help you to provide you quick answer and easily manageable way to customize the appareance of a Magento website

Sometimes I hear that for performance issues, some people overload the Magento template loading API. It’s a shame to avoid using this model because your client won’t be able to drive the website marketing design: model is quite fine and allows you to do what you want easily without too much work. There are really many thing to do before updating this API for performance issues

Magento guide: the three concepts of a Magento website context

Magento embeds a native multistore model that allows to configure multiple e-commerce models on the same magento source. If you are a Magento beginner, perhaps you are quite a bit lost with these stores, websites and views context. Each one has a dedicated role, and you should respect them, otherwise you would have to modify deep Magento’s working layer which would be quite difficult.

Screenshot of the stores management administrative panel available through the Magento administrative panel System > Stores Management

The magento websites are used to define your way of working :

  • Define a sales catalog
  • Configure the price rules management
  • Configure how customer accounts are shared between all yours stores

Define a sales catalog in a Magento website

Your catalog is manageable through the administrative panel catalog > Manage products. You can define there all your products data: name, description, price, etc

But this is not because you have enabled a product in this administrative panel that it will be salable: it needs to be part of a sales catalog: what do you want to sell on your store? All your products? Only some elements of your work?

For example, take a look at the LVMH group. This group owns Dior fragrance, Louis Vuitton bags, etc… Imagine they want to build a Magento to sell all their catalog (and if you want, you can contact me :)), they would want to sell each dedicated brand products in a dedicated store. To do that, they need two things:

  • They will first describe all their products in Magento back-office: product’s names, reference in their informatic system, etc. Afterthat products are editable, but not salable

  • Secondly, they’ll have to publish them in their sales catalog to ensure that only fragrance will be published in the Dior dedicated store, and only bags will be sold through the Louis Vuitton online store

To do that, products need to be linked to a website. This is available through the product sheet website tab

Checking the website checkbox will publish your product in the related sales catalog

So you should have at least one website per sales catalog

Are your sure? I’m not LVMH group, have only one online store, and I never do that, and I’ve not the website tab …

When you are in a single website context, all is done in background without any action

Define the scope of your prices in Magento through website

At this point you should know how to define your sales catalog

Now imagine that your business model will propose two different prices for the same product: a discount store and a normal one for example. If your prices rules are not managed with conversion rates, you should configure the price rule scope as website instead of global

This configuration value is available though the administrative panel system > configuration > catalog > prices > Catalog price scope

Magento adminstrative panel which allow to configure the price scope in a multistore context

You can with this rule apply a specific price for an article in a dedicated sales catalog

Configure how customer accounts are shared between websites

Websites allow also to configure the visibility scope of your customers account.

Suppose now that Amazon would migrate to Magento. (Yes perhaps you need to do a benchmark for other solutions, but it’s another problem 🙂 ). Do they want the customer accounts to be shared between amazon.co.uk, amazon.com, and amazon.de? Sharing same shipping address, same customer email account, etc? If so if you update your customer account on amazon.com, data will be also updated on amazon.de.

Customer accounts scope is a configuration value available through the system > configuration > Customers > Account sharing scope.

Screenshot of the account sharing option available through the administrative panel System > Configuration > Customer configuration

This option value will define how your customer accounts will be shared between each website

Why is it so important to well define the store model on your Magento?

As you can see, website define deep main work of your client and for this reason website are probably the most important degree of organization in the Magento multistore model. Updating a store model during a Magento project life is not an easy task, and for this reason, you should well define it at the beginning of your project. So be sure to ask yourself the revelant questions