Memcache is one key component that cannot be neglected from a web stack. Its almost certain that when you have some sort of caching before your database layer it is certainly gonna improve your performance. But setting up memcache in a production environment is slightly trickly espcially when you have a growing data set.

I almost hate to install memcache via apt-get since it creates config file and disabled by default. Instead I chose to install memcached from sources and always start it with -d flag to daemonize it. I reason I chose to install memcaches via sources is due to the extra debug tools it provides like memstat and memdump. They are extremely useful to know how well your cache layer is performing. I disable CAS to save 64 bytes for every cache entry. But if you are using CAS operations do not disable it.

Installing memcache from source

Since memcached has a bunch of dependencies lets install them first.

wget http://www.monkey.org/~provos/libevent-2.0.10-stable.tar.gz
tar xzf libevent-2.0.10-stable.tar.gz 
cd libevent-2.0.10-stable/
./configure --prefix=/usr/local
make
sudo make install

wget http://memcached.googlecode.com/files/memcached-1.4.5.tar.gz
tar xzf memcached-1.4.5.tar.gz
cd memcached-1.4.5
./configure --with-libevent=/usr/local/
make
sudo make install

wget http://launchpad.net/libmemcached/1.0/0.44/+download/libmemcached-0.44.tar.gz
tar xzf libmemcached-0.44.tar.gz
cd libmemcached-0.44/
./configure
make
sudo make install

Now you can start memcached and allocate a fixed amount of memory to it.

memcached -u nobody -d -m 512

One thing I learned the hard way is to always allocate the amount of memory memcached should use. If you dont specify the memory it defaults to 32MB which is pretty small. If you have a growing dataset and traffic I suggest you dedicate some boxes to memcache and start memcache with 90% of available memory.

Few tools to monitor memcached

The best way to get few stats from memcached is using by collecting internal metrics.

$ echo stats | nc localhost 11211
 ...
 STAT total_items 0
 STAT expired_unfetched 0
 STAT evicted_unfetched 0
 STAT evictions 0
 ...

 $ echo stats items | nc localhost 11211
  // some item related stats
 $ echo stats slabs | nc localhost 11211
 STAT 1:chunk_size 96
 STAT 1:chunks_per_page 10922
 STAT 1:total_pages 1
 STAT 1:total_chunks 10922
 STAT 1:used_chunks 1
 STAT 1:free_chunks 10921
 STAT 1:free_chunks_end 0
 STAT 1:mem_requested 67
 STAT 1:get_hits 0
 STAT 1:cmd_set 1
 STAT 1:delete_hits 0
 STAT 1:incr_hits 0
 STAT 1:decr_hits 0
 STAT 1:cas_hits 0
 STAT 1:cas_badval 0
 STAT 1:touch_hits 0
 STAT active_slabs 1
 STAT total_malloced 1048512

Monitoring slab related memcache is one of the most important things. As you launch you product and keep developing you add more attributes to your cache entry where the average item size increases over time which causes an effect called CACHE CALCIFICATION which means that your average items are now being stored in a slab which has less memory allocated. The new version of memcached 1.4.15 claims to solve it by supporting operations like slabs reassign -1 15.

Now that your memcached is up and running. We can see all the keys your memcached holds with memdump

memdump --servers=127.0.0.1

Specify the list of servers for memdump to introsect. This is useful for development purposes.

Now on to my most favorite tool -> memstat. Memstat collects a bunch of various statistics about how your memcached is perfoming.

memstat --servers=127.0.0.1
Server: 127.0.0.1 (11211)
	 pid: 10394
	 time: 1305424990
	 version: 1.4.5
	 pointer_size: 64
	 curr_connections: 42
	 total_connections: 177741649
	 connection_structures: 308
	 cmd_get: 1633514457
	 cmd_set: 439392889
	 get_hits: 1493612276
	 get_misses: 137902181
	 delete_misses: 9951479
	 delete_hits: 8166456
	 bytes_read: 770672258573
	 bytes_written: 2206367771075
	 limit_maxbytes: 6878658560
	 conn_yields: 11785
	 bytes: 6163615697
	 curr_items: 3736681
	 total_items: 235136283
	 evictions: 181308239
	 reclaimed: 4226382

I have trimmed the output to only include the useful stats. The most important ones I lookup for is get_hits and get_misses. This gives you your memcache hit ratio. Another most important one to keep in mind is the evictions. As your evictions increase over time that means its time to scale up your memcache servers. Also you can see that memcache is holding a lot of connections, may be connection pooling them can also improve the performance.

Along with these tools there are a few handy tools to remove keys from a cluster, copy keys etc.