Support for cache tags for all cache engines in the Magento enterprise version 1.13

Did you follow the Magento news, you probably heard that Magento Enterprise version 1.13 completely moved the indexer’s logic to the database server.

This for sure leads to a significant increase in your performance.

But are the indexers the only explanation for those better performances? No: there is a few other things involved in this upgrade: one of these is the support of the cache tags for all cache backends.

Continue reading “Support for cache tags for all cache engines in the Magento enterprise version 1.13”

Optimize Eclipse and Zend studio refresh in Magento projects with the resources filters

If you’re using Zend Studio or eclipse and have kept the native structure of a Magento project (in its first version), refreshing a project could sometime be a long task.

This is partly due to the number of files that change and need to be refreshed: Magento needs to take their new content into account.

To avoid refreshing every file changing frequently, you can exclude them: they won’t appear in the project folder, and so, won’t be
refreshed

Continue reading “Optimize Eclipse and Zend studio refresh in Magento projects with the resources filters”

Looking for (Magento) developers

Develop : « enrich, growth a software »

Yes it’s my definition of the word « develop ». I checked the Internet to ensure that I wasn’t wrong and I agree with the correct definition.

Many years ago, on the first 1.0 Magento release, we made a comparison for a request between the commerce server and Magento plateforms. At this time, the clients chose Magento in part because of its functionalities : Yes Magento is rich, and cover many e-merchandiser requirements.

And in case it’s not enough, there is perhaps a module allowing you to customize it according to your specificities. You know it’s quite easily to reach your customer requirement with this solution.

But I’m always despite when I want to use a functionality on my client’s plateform and see it does not work anymore… Why ? How could that happen ? This function was working, and now, it does not exist anymore….

Yes, web developers have broken it, one more time.

Continue reading “Looking for (Magento) developers”

Magento tips: Display a Magento category path in SQL

Yes, it’s been quite a long time I’ve not been sharing something with you: many projects to handle and no time to share something. And when we are busy, it’s a bit disappointing to see how many time we may lost to do some repetitive tasks

One of these tasks, when you study a Magento sales catalog, is to find in which categories is published a product Continue reading “Magento tips: Display a Magento category path in SQL”

Magento layouts: the limit of reference syntax

If you ever tried to update the Magento page content, you probably used the reference node: this instruction tells that every node under the reference node is applied to the block having the name used in the reference. Here’s an example:

<reference name="head">
<action method="addJs"><script>prototype/prototype.js</script></action>
</reference>

The example above will perform the addJs action on the block with the attribute name head.

This syntax is very useful when, in your custom layouts files, you want to refer to another page content block without updating the layout file which contains the referenced block. This instruction is very useful when you create your own modules and want to only customize Magento page

But sometimes, it does not work.

Why the Magento reference node’s content is sometimes not interpreted?

To understand when this case happens, take a look at the following two modules

Test_FirstModule and Test_SecondModule; Each of these modules embeds its own layout update. The layout of the second module defines a block in the content node called test_secondmodule_content

<?xml version="1.0" encoding="UTF-8">
<layout version="1.0">
    <default>
        <reference name="content">
            <block type="test_secondmodule/content" name="test_secondmodule_content" template="test/secondmodule/content.phtml" />
        </reference>
    </default>
</layout>

The first module refers to this block to provide a child block

<?xml version="1.0" encoding="UTF-8">
<layout version="1.0">
    <default>
        <reference name="test_secondmodule_content">
            <block type="test_firstmodule/content_child" name="test_secondmodule_content_child" template="test/firstmodule/content.phtml" />
        </reference>
    </default>
</layout>

In this case block “test_secondmodule_content_child” will never be available: when Magento builds the blocks list composing the page content, it iterates all defined nodes in the appropriate handles within the Mage_Core_Model_Layout::generateBlock method. But merged layout are parsed in the order they have been added in the configuration, so by default by their name. So in our case, firstmodule.xml will be handled before secondmodule.xml. And this is the problem: Reference expects that referenced block already exists. In our example, it won’t be the case because it will be processed after; so the child block won’t make any render

If you call a reference prior the block, all the references content won’t be processed

The only way to use reference is when they have already be defined

Possible solutions to bypass the reference layout limitation

Play with names of your modules and your database layout updates

You can try to manage order by updating the name of your modules and name of the layout files, but this solution depends on the installed modules: there’s still a risk that your layout will be handled before the expected one.

Use the Magento database layout updates

The database layout updates are always managed after the file’s ones: so if you define your references in database layout updates, they should always refer to a valid defined block. But it’s a more complicated way to develop your module.

Remove reference instructions from the layout

This is a cleaner way: before generating blocks, you’ll move all the references nodes content into the appropriate block node: no more references in your layout, no more risk that there is something wrong because they have been loaded in the wrong order.

The Kyp_LayoutFix module

I’ve created a module, Kyp_LayoutFix, that does that very thing: before generating block content, it removes all references instructions from the used layout

This module has been tested on all Community Editions from 1.4.1 to 1.7.0.2 and on the related Enterprises versions and is a sure way to avoid this references problem

This module is provided under the Open Software License

Download the Kyp_LayoutFix module

This module can be downloaded from my pear channel:

pear channel-discover channel.kyp.fr
pear install kyp/kyp_layoutfix

Or you can download it in a tar.gz file here

Let me know if you have any issue with this module.

High traffic websites under Magento: the Teleshopping example

Téléshopping, one of the most older Magento project in France. With 41Meuros, website represents a tier of the turnover. Let’s see which technical choices has been made to provide a high performance website

Hello Gilles, maybe everyone doesn’t know you yet, could you please introduce yourself?

I’m in charge of the production center at Teleshopping. I am in charge of the operation, administration and system architectures for ERP and websites and for its subsidiaries (Euroshopping, Place des tendances and partly Direct Optic).

Two of your websites, Teleshopping and Euroshopping, are Magento-developed projects. Could you please tell us more about them?

Sure, Teleshopping’s website is among the first french Magento-developed websites, deployed in April 2009, and regularly updated since. There are approximately 1M unique visitors per month.

It allows its users to order products they have seen during the TF1 TV show. Ordering products from the catalog is done through another website.

Euroshopping’s website was deployed a little later, and there are about 100 000 unique visitors per month. It allows people to order products they have seen during TV shows on different TNT channels or on the cable.

Could you describe the general architecture within these two websites?

For Teleshopping, we have 10 front-end servers available for visitors, and one more for the back-office access

Proxy cache varnish is ahead each one of the servers. This may not be the best solution, but it allows us to regulate the number of front-ends without changing the configuration or having to ask ourselves too many questions.

Concerning the application, the source code is installed on every front-end server, there’s no replication and deliveries are hand-made.

But we share the media and skin directories on an NFS resource.

We use a memcached server with 1 port to put data in cache, and another port to put sessions in cache; this way we can flush data without any impact on the user’s sessions.

Concerning the database, we use a MySQL server which is replicated on a secondary server. All accesses are done on the main server.

Last element, the CDN. Medias are served by a CDN, which allows us to relieve at the outside our front-ends, even during the business campaigns.

Source images are put on an NFS resource available on every front-end.

For Euroshopping it’s at the most the same, except there’s no varnish, the 4 servers available are more than enough to manage the load.

How often do you deploy new versions?

It’s an average, but we do about 1 delivery in the production environment per month. Sometimes we may have more deliveries than that.

Could you tell us about your last deployment?

Every developer has its own development environment on a virtualized server, which is strictly the same as the production one.

A testing environment is available to validate the package before it’s installed on a pre-production environment. This delivery is done by my team.

Once the project manager has tested and approuved this package, it is delivered in the production environment, my team still doing it.

You are sharing the front-ends server’s contents on a NAS. What content do you exactly share?

We share the media and skin directories on our NAS, a filer NetApp. This sharing system is almost as performing as the local disk access, and its vantage is that we do not have to duplicate media files.

What made you choose this setup?

This sharing process is almost as performing as the local disk access, and its vantage is that we do not have to duplicate media files.

NFS and PHP tuning allows to manage access to the shared files with different cache levels.

We can manage the stock volume without any outage.

Could you tell us about the main problems you encountered while you were setting up this model?

We didn’t have any particular problem, only some tuning was needed for the NFS:

  • We mount the NFS shared folder with the following options tcp, rw, bg, hard, suid, noatime, nodiratime, nointr, timeo=600, retrans=2, rsize=32768, wsize=32768, actimeo=30
  • We have increased the PHP realpath cache to have a size of 1M and a TTL of 1 day

What benefits do you get from the source sharing between the different front-ends on a NAS?

Source sharing is useful to deliver the source code only once, it is then available for all of the front-ends; we do not need a “master” server from who to replicate the files.

Presently, the websites’s sources don’t allow us to share the whole source code, partly because of the /var/reports and /var/logs directories. For the next versions if source code allows it, we intend to share the whole source code for every front-end server.

How to schedule Magento sales rule correctly?

Magento provides a sales rule engine which allows you to offer discount to your clients: free shipping, buy x products and get y products free, providing a discount from a voucher are samples usage of prices rules

Two important concepts exist in this model:

  • Can we cluster sale rules: can rules be combined or if a rule is applied, are the others automatically excluded ?
  • Sorting rules importance within the priority concept

How does Magento apply prices rules on cart ?

Each time you update your cart content or each time you display the cart page, Magento loads all the active sales rules for the current date, current website, current user group and test for-each rule if the conditions match the cart content. If yes, Magento applies specified discounts to items in cart.

If a rule has been flagged as “stop rule processing”, we stop iteration on the rule collection. If not, we continue and apply the others matching discount until we have tested all rules or we have applied a discount which stop iteration

The Magento sales rule priority

Priority is a numeric id defining the rule importance and how it’ll be tested: the less is the priority value, the more important is the rule

Priority is specified when the sales rule is defined from back-office

Priority is not a mandatory field: if you do not specify a value, the priority applied will be 0, and so the rule will be tested at first before all other sale rules

Technically, when the rules collection is loaded, it’ll be sorted by the priority value (sort_order field)

Priority is not unique: you can define several rules having the same priority

What happens if some Magento sale rules have the same priority?

There is no second filter to order rules: so if you have two rules which can be applied, this is the oldest one which will be used and applied firstly. Is it really what you were expecting? Perhaps not.

Furthermore, if you have many sale rules, you should ensure to know exactly each conditions criteria to understand exactly how are applied rules and in which order. Possibly very difficult in a long time or if you have many sale rules defined

How can you reduce the risk that some rules are not executed as you want?

Reviewing the rules to ensure that priority is unique is the only one solution to avoid risk that rules are not executed the way you are expecting them. But it could be difficult to maintain if you define later others sales rules.

If you want to reorder priority, you can define priority levels: the most important one have a priority defined with one digit, lesser priority with two digits, oldest one or more generic sale rules with a priority of three digits. In this case, you’ll ensure that you’ll know with an unique priority how they’ll be tested, and be able to add others rules easily

One other best practice is to defind strong application conditions criteria: the more generic they are, the more risk you have to apply a rule that should not be executed

Enhance Magento enterprise full page cache performance by updating its storage backend

Magento enterprise supports since the 1.7 version the full page cache module: instead of storing only each block content in cache, this is the full page content that’s stored in cache

Everybody agrees this is a really performance enhancement

Like the native cache, storing these datas in a file is not the fastest way and becomes a problem when you need to share it between each front. How can we customize it?

The full_page_cache configuration node in the Magento enterprise.xml file

History of the full page cache configuration in the Magento enterprise versions

Until 1.11.0.0, there was no node to configure the full page cache option in the default Magento enterprise configuration files.

Since 1.11.0.0, you have seen a new node appearing in the app/etc/enterprise.xml file:

<config>
    <global> 
        <full_page_cache>
            <backend_options>
                <cache_dir>full_page_cache&</cache_dir>
           </backend_options>
        </full_page_cache>
   </global> 
</config>

This new node allows to customize the cache directory folder where the cache content will be located

Customize your full page cache configuration

Everywhere on the net you can find ways to enhance performance of the Magento cache storage engine by customizing the following configuration node

<config>
<global>
<cache>
...
</cache>
</global>
</config>

As with the config > global > cache node, the full page cache node works exactly: all nodes you’ll specify under will configure how full page cache works. For example, The following node defines the usage of memcached as cache storage engine for the full page cache content

<config>
<global>
<full_page_cache>
            <backend>memcached</backend>
            <slow_backend>database</slow_backend>
            <memcached>
                <servers>
                    <server>
                        <host><![CDATA[127.0.0.1]]></host>
                        <port><![CDATA[11211]]></port>
                        <persistent><![CDATA[0]]></persistent>
                        <weight><![CDATA[2]]></weight>
                        <timeout><![CDATA[10]]></timeout>
                        <retry_interval><![CDATA[10]]></retry_interval>
                        <status><![CDATA[1]]></status>
                    </server>
                </servers>
            </memcached>
        </full_page_cache>
</global>
</config>

Can the full_page_cache node be used within Magento enterprises version prior 1.11?

Yes. Even if this node was initialized in 1.11, all prior versions look for full page cache configuration in this node. So you can also customize your full page cache in all Magento enterprise versions which embeds full page cache module

Don’t forget to customize the full page cache management

As you can see, the two nodes config > global > cache and config > global > full_page_cache are strictly independent: you can provide two different ways to manage cache between Zend cache api and full page cache: we often see that there is cache backend configuration, but do it also for the full page cache

Make a clean sweep of commons Magento cache_lifetime workarounds usage

As I mentioned before, Magento cache policy API is based upon the Zend Framework Zend_Cache component. In this cache policy, there are three elements :

  • Cache key that defines the unity of the data we want to cache
  • Cache tags that allow to cluster our cache data
  • Cache lifetime that defines the duration within we consider that our stored data are valid and do not need to be rebuilt

My current topic will be about this latest point: cache lifetime. Its usage is quite simple: we define a duration while data will be stored in cache; during this period, data is always considered as valid, and so, not rebuilt. After that, even if data exists in cache, it will be rebuilt because considered as expired. Simple, no?

But we can hear so many things about cache lifetime usage that it seems interesting to evoke some specific points about its usage in Magento

What happens when Magento cache lifetime is specified as null?

When Magento cache lifetime is considered as null?

Excluding case you have defined a getCacheLifetime method which returns null value, this case occurs when you do not define the cache lifetime on one object which extends Varien_Object:

  • When rendering block content, Magento will call the getCachelifetime block method
  • If you have not defined a getCacheLifetime method, Magento will use Varien_Object __call method to retrieve cache_lifetime from attribute _data
  • If this index does not exists in _data, __call will return null

How is managed cache lifetime null value by Magento for block data?

All blocks extends Mage_Core_Block_Abstract. This class provides the following cache management:

As you can see, if lifetime is null the Magento Cache API will never fetch something from cache, so rebuild each time the block content. But also nothing will be stored in the cache(_saveCache method check if also there is a non null lifetime before save)

If you do not set up a cache lifetime for a block content, it will never be cached

How is managed cache lifetime null value by Magento blocks for non block data?

The cache API can also be used for other things than block output. This is the case for configuration cache. In this case, the control made by Mage_Core_Block_Abstract does not exist, and data will be cached: we’ll see that if cachelifetime duration is null for something else than block, it will use the default lifetime value like id you set up false as a cache lifetime value: 7200sec (2 hours)

What happens when cache lifetime is set up to 0 ?

Sometimes, we can hear that setting 0 as cache_lifetime duration disable cache. No, and this is for the following reason: 0 is a valid cache_lifetime duration.

Magento will define the timestamp until you consider your block content as valid with the following rule: current timestamp of data generation + cache_lifetime; so calculated expiration time will be the generation time. For sure, next time you’ll check if data is in cache and valid, you’ll compare new timestamp with expired timestamp, and there is a great probability that your current timestamp will be later.

Setting 0 as cache lifetime duration render your cached block content always expired

This is one of the worst case, because, each time you build content, you save it in cache, and invalidate it next time you’ll check it

What happens when cache lifetime is set up to false ?

The Zend Framework lifetime management for false values

Every cached backend used in Magento inherits from the Zend_Cache_Backend class

When saving content, Zend_Cache_Backend class will calculate the expiration timestamp trough the Zend_Cache_Backend::getLifetime method. If specified lifetime === false, backend use a default duration stored in its attributes

If you check Zend_Cache_Backend class, you’ll check that this value is hard_coded as 3600 sec (one hour)

But our content is stored longer, why?

Magento forces a default lifetime in its frontend

When the Magento cache API initializes the cache frontend object in the Varien_Cache_Core model, it merges cache options with directives options

This is the case for the cache_lifetime duration: default cache lifetime duration specified on Varien_Cache_Core is merged with directive options

But Varien_Cache_Core class extends Zend_Cache_Core. So when backend will be set up to frontend, we’ll use the native Zend Framework behaviour and so, merge frontend options as a backend directives.

So for all of our backend usage, because Magento forces usage of a Varien_Cache_Core frontend, default lifetime will be set up as 7200s

If your lifetime is set up to false, by default your block will be considered as valid during 7200s

Init Magento cache block policy in _construct method… or not

We often see the Magento cache block policy initialization in the _construct method, like in this example:

class Namespace_Module_Block_Type extends Mage_Core_Block_Template
{
     protected function _construct() {
          $category = Mage::registry('current_category');

          $this->addData(array(
               'cache_lifetime' => 86400,
               'cache_tags'     => array(Mage_Catalog_Model_Category::CACHE_TAG."_".$category->getId()),
               'cache_key'      => $category->getId(),
          ));
     }
     ....

This is not a quit good example of cache block policy initialization, for the following reasons

Init cache block policy in _construct adds unnecessary treatment

The problem is due to the way of how Magento loads the page structure and how it will render blocks content loaded from cache:

When you load the layout, Magento will parse and build all defined elements in layouts files. This means instantiate all defined blocks. After, Magento will check if your block content is already in cache, and will use it if yes, or build it if not.

But instantiating one block will call the _construct method. This means even if your block content is in cache, your block will execute each time all instructions within _construct method.

Do you require to rebuild cache tags list each time?

The answer is no. The cache_tags list is required only when saving generated content into cache. So if you initialize the cache tags list in _construct, it would have an interest only if your block content doesn’t come from your cache

Init Magento cache block policy in _construct let you avoid to use layout properties

Another problem is due to accessible data in _construct method: when calling _construct, block initialization is not finished: Magento will populate data, set name in layout, reference current layout in block and so all these informations won’t yet be available.

so layout informations are not yet available, and you restrict cache block policy only to data managed in block

Possible solutions to avoid these problems?

Specifying a cache block policy is not an option for performances reasons. But defining it in _construct method can reduce performances benefits if you continue to load some data within. A possible solution to bypass these problems is to define on your block the methods getCacheLifetime, getCacheKey and getCacheTags: treatments within will be executed only when they’ll be called