Memcache is one key component that cannot be neglected from a web stack. Its almost certain that when you have some sort of caching before your database layer it is certainly gonna improve your performance. But setting up memcache in a production environment is slightly trickly espcially when you have a growing data set.

I almost hate to install memcache via apt-get since it creates config file and disabled by default. Instead I chose to install memcached from sources and always start it with -d flag to daemonize it. I reason I chose to install memcaches via sources is due to the extra debug tools it provides like memstat and memdump. They are extremely useful to know how well your cache layer is performing. I disable CAS to save 64 bytes for every cache entry. But if you are using CAS operations do not disable it.

Installing memcache from source

Since memcached has a bunch of dependencies lets install them first.

    tar xzf libevent-2.0.10-stable.tar.gz
    cd libevent-2.0.10-stable/
    ./configure --prefix=/usr/local
    sudo make install

    tar xzf memcached-1.4.5.tar.gz
    cd memcached-1.4.5
    ./configure --with-libevent=/usr/local/
    sudo make install

    tar xzf libmemcached-0.44.tar.gz
    cd libmemcached-0.44/
    sudo make install

Now you can start memcached and allocate a fixed amount of memory to it.

memcached -u nobody -d -m 512

One thing I learned the hard way is to always allocate the amount of memory memcached should use. If you dont specify the memory it defaults to 32MB which is pretty small. If you have a growing dataset and traffic I suggest you dedicate some boxes to memcache and start memcache with 90% of available memory.

Few tools to monitor memcached

The best way to get few stats from memcached is using by collecting internal metrics.

    $ echo stats | nc localhost 11211
     STAT total_items 0
     STAT expired_unfetched 0
     STAT evicted_unfetched 0
     STAT evictions 0

     $ echo stats items | nc localhost 11211
      // some item related stats
     $ echo stats slabs | nc localhost 11211
     STAT 1:chunk_size 96
     STAT 1:chunks_per_page 10922
     STAT 1:total_pages 1
     STAT 1:total_chunks 10922
     STAT 1:used_chunks 1
     STAT 1:free_chunks 10921
     STAT 1:free_chunks_end 0
     STAT 1:mem_requested 67
     STAT 1:get_hits 0
     STAT 1:cmd_set 1
     STAT 1:delete_hits 0
     STAT 1:incr_hits 0
     STAT 1:decr_hits 0
     STAT 1:cas_hits 0
     STAT 1:cas_badval 0
     STAT 1:touch_hits 0
     STAT active_slabs 1
     STAT total_malloced 1048512

Monitoring slab related memcache is one of the most important things. As you launch you product and keep developing you add more attributes to your cache entry where the average item size increases over time which causes an effect called CACHE CALCIFICATION which means that your average items are now being stored in a slab which has less memory allocated. The new version of memcached 1.4.15 claims to solve it by supporting operations like slabs reassign -1 15.

Now that your memcached is up and running. We can see all the keys your memcached holds with memdump

    memdump --servers=

Specify the list of servers for memdump to introsect. This is useful for development purposes.

Now on to my most favorite tool -> memstat. Memstat collects a bunch of various statistics about how your memcached is perfoming.

    memstat --servers=
    Server: (11211)
    	 pid: 10394
    	 time: 1305424990
    	 version: 1.4.5
    	 pointer_size: 64
    	 curr_connections: 42
    	 total_connections: 177741649
    	 connection_structures: 308
    	 cmd_get: 1633514457
    	 cmd_set: 439392889
    	 get_hits: 1493612276
    	 get_misses: 137902181
    	 delete_misses: 9951479
    	 delete_hits: 8166456
    	 bytes_read: 770672258573
    	 bytes_written: 2206367771075
    	 limit_maxbytes: 6878658560
    	 conn_yields: 11785
    	 bytes: 6163615697
    	 curr_items: 3736681
    	 total_items: 235136283
    	 evictions: 181308239
    	 reclaimed: 4226382

I have trimmed the output to only include the useful stats. The most important ones I lookup for is get_hits and get_misses. This gives you your memcache hit ratio. Another most important one to keep in mind is the evictions. As your evictions increase over time that means its time to scale up your memcache servers. Also you can see that memcache is holding a lot of connections, may be connection pooling them can also improve the performance.

Along with these tools there are a few handy tools to remove keys from a cluster, copy keys etc.