Zend_Cache component usage in Magento

Magento performances is a hot topic subject. One element of the Magento performance is the cache block policy which allows to store in cache part of the page content: You probably have all read the tutorial available through the Magento wiki, written by my old colleague Laurent BOURREL, about the cache block policy available in Magento.

All concepts of this policy are based upon Zend_Cache component. Let’s see how Magento use it

Reminder about Zend_Cache component

Zend cache module structure

Zend_Cache component is divided into two parts, frontends and backends models:

  • Frontend components define what is covered by cache policy:
    • Files
    • Output rendering
    • Method call
    • Class
  • Backend components define where will be stored cached data:
    • In file
    • In memcached
    • In apc
    • ….

This model allows to easily dissociate which data will be cached, and where they will be cached

Zend_Cache usage in Magento

Embedded Zend_Cache backend in Magento

If we check Zend framework version from CE 1.3.2.4 to CE 1.7.0.0 (and related Entreprise versions), we should be able to use the following backend within Magento:

CoreApcBlackholeFileLibmemcachedMemcachedSqliteStaticTestTwolevelsXcacheZend_PlateformZend_Server diskZend_Server shared memory
CE 1.3.2.41.7.2
CE 1.3.3.01.7.2
CE 1.4.0.11.9.6
CE 1.4.1.01.9.6
CE 1.4.1.11.9.6
CE 1.4.2.01.10.8
CE 1.5.0.11.11.1
CE 1.5.1.01.11.1
CE 1.6.0.11.11.1
CE 1.6.1.01.11.1
CE 1.6.2.01.11.1
CE 1.7.0.01.11.1
EE 1.6.0.01.9.3PL1
EE 1.7.0.01.9.6
EE 1.8.0.01.9.6
EE 1.9.0.01.10.5
EE 1.9.1.11.10.8
EE 1.10.0.11.11.1
EE 1.10.0.21.11.1
EE 1.11.1.11.11.1
Loaded from Zend FrameworkAdded or updated by Magento sources, but available

How does Magento use Zend_Cache?: the factory Mage_Core_Model_Cache

Before CE 1.4, Cache factory was made by the Mage_Core_Model_App model

Since Magento CE 1.4, all cache management is done through the Mage_Core_Model_Cache class; this class is a factory model that will load the Zend_Cache frontend and backends

Magento Zend_Cache frontend

In Mage_Core_Model_Cache, Frontend class is hard coded and restricted to Varien_Cache_Core. Varien_Cache_Core inherits from the frontend class Zend_Cache_Core.

So we are unable to use natively other frontends than Zend_Cache_Core

Magento Zend_Cache backend

Why is there some Backend available in Magento which does not exist in Zend_Cache component?

But this factory method also provides the ability to use some other backends storage that does not exist in Zend_Cache; see the following examples :

  • Database
  • Eaccelerator

These backends have been written by Varien core team

In conclusion, the Zend_Cache backend defined by Magento are the following ones
CoreApcBlackholeFileLibmemcachedMemcachedSqliteStaticTestTwolevelsXcacheZend_PlateformZend_Server diskZend_Server shared memoryEacceleratorDatabase
CE 1.3.2.41.7.2
CE 1.3.3.01.7.2
CE 1.4.0.11.9.6
CE 1.4.1.01.9.6
CE 1.4.1.11.9.6
CE 1.4.2.01.10.8
CE 1.5.0.11.11.1
CE 1.5.1.01.11.1
CE 1.6.0.11.11.1
CE 1.6.1.01.11.1
CE 1.6.2.01.11.1
CE 1.7.0.01.11.1
EE 1.6.0.01.9.3PL1
EE 1.7.0.01.9.6
EE 1.8.0.01.9.6
EE 1.9.0.01.10.5
EE 1.9.1.11.10.8
EE 1.10.0.11.11.1
EE 1.10.0.21.11.1
EE 1.11.1.11.11.1
Loaded from Zend FrameworkAdded or updated by Magento sources, but available

So we will update the available zend_cache backend list according to Magento versions:

And so, what If I want to use a backend that has not been thought of by Varien?

Perhaps you can want to use blackhole, test, or zend_plateform backend for exemple. Is it really difficult? no

Since Mage_Core_Model_Cache model exists (CE 1.4.0.1 and higher, related EE) the factory method has been also thought so that perhaps you want to use another backend :

The value defined in the local.xml node <config><global><cache><backend> can be a class name that implements the Zend_Cache_Backend_Interface.

So, and for my backend configuration? To specify some parameters to this new backend one, you can define a node <config><global><cache><backend_options> which will be passed as arguments when constructing the backend model

Other possibilities to use a custom backend model are the following ones:

  • Define your own backend class that will set up the expected configurations values
  • Overload Mage_Core_Model_Cache to be able to provide other configurations loading

Here’s an example of specific configuration you can have and which works

Conclusion about Zend_Cache usage in Magento

cache tags, cache lifetime and cache key are concepts brought by the zend_cache component

Even if there is no documentation about how to use some custom backend in Magento, it’s not really difficult: just specify the full backend class name in local.xml backend node, and specify arguments through the backend_options node

The only problem I can see with this method that is able to load any backend class is the method used to check if backend exists: it uses class_exists method, which, In Magento context, throws a fatal error if the class cannot be found. It could be interesting, if we provide a backend class which does not exist, that the default one (file) would be used instead of throwing a fatal exception

Updating zend_cache frontend class is quite difficult because it depends on how PHP code was written. So for now, even in Magento 4, 5, I’m not sure that we can use the buffer frontend

High availability in Magento farm: the session problem

Big Magento websites require sometimes to be able to respond to a high traffic: there are many websites with high load, and sometimes, we must propose a magento farm behind a load balancer to be able to respond to all visits. But in this case, we have a choice to do about the load balancing: how is dispatched the load between Magento webservers?

And this choice that we have to make will have an impact about how sessions are managed

Depending on how are configured load balancers, we need perhaps share session between Magento front-webservers

Impact of load balancers configuration on the session storage management

Load balancers are in front of our webservers to be able to manage our webserver load

There are two ways to share the load between each front in load balancer when a customer is visiting a website:

  • he is always sent to the same webserver
  • According to the load, he is sent to the less loaded webserver

This load balancer configuration choice has an impact on the session management:

in the first case, we are always redirect to the same webserver.. But in this case, if the load of this server change we’ll have an end-user usage not as well as expected. In the worst case, if this websever become unavailable we wouldn’t be able to access the website anymore and the consequence is, and so loose some sales…

In the other case, we have not these disagrements because load is detected on load balancers and we are redirect to the most available server; but in this case we have the problem to share our sessions between each front to be able to retrieve our personals informations.

To ensure that we have the highest availability for our website, we’ll choose the second option and will see what solutions we have to share sessions between front web-servers

Reminder on PHP sessions storages

By default, PHP uses a session storage based on files: every time you start a session a file will be written on the webserver with the session content. it’s nice, but it has the disavantage to depend on which webserver you requested the information. If we are redirected to another webserver, the session file does not exist, and so, you’ve lost our cart…. How can we solve that?

PHP allows to redefine our PHP session storage model and define how are they managed. Nice, we’ll see how it can help us in our webserver farm management and our session sharing problem

The possible ways to share sessions in a Magento farm

Latest Magento release (1.6.2 CE, and related enterprise one) embeds many sessions storages management:

  • Files, the PHP default’s one
  • Database
  • Memcached
  • Othes choices?

Which advantage(s) have each one?

Sharing Magento sessions in a Magento farm with database(s)

In this case, we remplace the native PHP session storage management by a database: all sessions are stored in our database(s) server(s).

This solution is quite nice because each of our front webservers must be able to read and write in database; so there is no problem to share session: visitor will be redirected initially to a front webserver, init it session on it and save it’s customer session in database. Next page he request, load balancer will perhaps redirect him to another front-webserver, but with it session id, he’ll retrieve session data from the database and so, keep the same informations. Nice

But the problem in this situation is that we add another load on database server. Each visitor will make read and write instructions on our database server, and so, according to our traffic, it could be a very huge load. Furthermore, even if we add a very good cache policy, if our database webserver go down, our website won’t be available anymore.

Magento database table where are stored the sessions data is based upon file system. So If our server go down, we’ll be able to restore our session data. If Magento database availability is critical, there is some solutions like MySQL cluster to ensure that MySQL is always available. So we are able to provide a maximum service and a model tolerant to failure. But this model has the big problem to provide a important load on the database and it could be interesting to check the others availables solutionss

Sharing Magento sessions in a Magento farm with Memcached

Memcached is a memory server.

It can be use to:

  • Share cache data
  • Share sessions

One of the main advantage of memcached is that it’s a memory server, and so, read and write instructions will be made faster than with a file system

But we have two disavantages to use a memcached to store sessions:

  • Data integrity: memory does not check write as it’s done in a file system: writing in memory is not as secure as writing in a file system. Our data can be compromised, and so, loose part of our data.
  • We have the same problem as in the database: memcached is not tolerant to failure and so you’ll loose everything if it goes down; memcached can be clustered for cache storage, but in case of session sharing, we must provide only one server URI. So if server which manage session storage hang up, there is any session service available…

So for now, memcached has the only advantage to be able to share our sessions faster; it is not tolerant to failure and cannot be clustered so this is another risk in our architecture

Sharing sessions using a network shared folder and file system PHP storage

To avoid data integrity problem, we could continue to use a session file system storage, but share them between each front through a shared folder: in this case, all sessions storages location will be known by each server, and so, we can have the ability to use our session on each webserver. Data are also checked on write operations by the file system, so it’s nice

But this model has the disavantage to impose a network load every time you access a session. According to our website load, this network additionnal load could be critical

Sharing Magento sessions with Zend server

Based upon the PHP session storage file model, Zend Server embeds a session clustering between each front webserver zend server instance: Every starting session is copied on each Zend Server. So every time someone request a page from our webserver, it’ll be able to access our session file. Nice

Furthermore, when some data are written in session, Zend Server knows which is the main session storage file, and so is able to write in it.

If a front webserver go down, another session copy is used as main session file, and so you have no disagreement

For now, Zend Server session clustering is the only way to garantee that you won’t have any go down of our website and it provides data integrity using file system storage write control. But one of the disavantage is you need a Zend Server on each Magento front-office, and it increases our webhosting cost

Conclusion about the high availability storage management

Sales, television advertising or simply very basic traffic can lead you to have a dedicated magento farm to be able to answer all visitors requests. In this case, you should take care of always be able to answer all requests, not only for your orders but also for your branding

For sure, you can configure load balancing to always redirect a customer to the same server. This is the easyest way to configure load management, but is not the most reliable model and can provide some disagremments to our clients with possible no webservers response.

In the other case, the only way to ensure that we’ll have a high availability in a Magento farm is to ensure that the load is dispatched on front-webservers depending on their load. This brings the problem to share session between each webserver, but for now, the only secure way is to use Zend Server to clustering session.This model provide us a secure way to share session, ensure acces and write control, and does not add additionnal critical point in our architecture

Furthermore, it seems that choice of storage management has no performance impact. This is a first test, and we don’t know how it has been managed, but in the case mentionned, it seems that session_storage has no or few influence upon the performance

Using libmemcached in Magento with versions before CE 1.6 / EE 1.11

My feedback is that it’s easily possible. Here’s the requirements

Summary of the cache usage in Magento

First remember how cache management in Magento works : Magento has the advantage to allow to choose your cache storage model. Some are better with only one server, others are better when you use many web servers. You can easily set up your cache management with your own requirements only by updating your local configuration file app/etc/local.xml: you have a template of configuration node you require to configure a specific cache storage management, located in the app/etc/local.xml.additional

Here’s a screenshot of the configuration you can set up to change your cache engine:

What is the interest of using libmemcached in Magento?

This extension is more up to date according with memcached server and so, provides enhanced functionnalities and fix than in php_memcache_extension

How to setup libmemcached as cache storage in Magento versions before 1.6?

All cache storage engine are those embedded in Zend Framework. But prior to Zend Framework 1.11 version, libmemcached backend was not embedded.

Do I need to upgrade Magento Zend Framework to enable libmemcached usage?

The first question we can ask is: “Ok, libmemcached is embedded in ZF 1.11 but not in my magento’s version. Do I need to upgrade all ZF to embedded libmemcached?

Answer is no: we do not need to upgrade all ZF in your Magento version, for multiples reasons:

  • ZF Cache API has not change its signature: so even if we add a new backend, we do not need to upgrade all ZF
  • Upgrading all ZF can have many other effects like in form build for example: upgrading ZF is not a simple operation which requires expertize

So ok, we do not need to upgrade all ZF. But how to?

How to install libmemcached support in Magento?

Well, for now, we know that Cache API has not change. So we need to do only the following things:

  • Add libmemcached support to our Magento
  • Allow to use libmemcached in our Magento

Add libmemcached support on our Magento

We just need to provide a libmemcached backend support. Nice, recent ZF versions have one. We can use it 🙂 But where do I put it? In lib/Zend/Cache??? no, please, use the overloading mechanism embedded in Magento and copy it in local folder

Ok, with that, our Magento has a backend able to communicate with libmemcached

Allow libmemcached support in Magento configuration

For the moment, if we update our local.xml to enable libmemcached support, we can’t use it: our new backend is not known by magento


We just have to allow libmemcached loading in this class (by a clean overload 🙂 ), and we will able to use libmemcached.

Conclusion

For now, even if libmemcached is in a beta state we can use it. My first tests are fine on Magento for cache storage, but I add exception for session storages.

The only thing we can regret is that automatic cleaning is not always available in libmemcached…