Memcache is one key component that cannot be neglected from a web stack. Its almost certain that when you have some sort of caching before your database layer it is certainly gonna improve your performance. But setting up memcache in a production environment is slightly trickly espcially when you have a growing data set.
I almost hate to install memcache via apt-get since it creates config file and disabled by default. Instead I chose to install memcached from sources and always start it with -d flag to daemonize it. I reason I chose to install memcaches via sources is due to the extra debug tools it provides like memstat and memdump. They are extremely useful to know how well your cache layer is performing. I disable CAS to save 64 bytes for every cache entry. But if you are using CAS operations do not disable it.
Installing memcache from source
Since memcached has a bunch of dependencies lets install them first.
wget http://www.monkey.org/~provos/libevent-2.0.10-stable.tar.gz tar xzf libevent-2.0.10-stable.tar.gz cd libevent-2.0.10-stable/ ./configure --prefix=/usr/local make sudo make install wget http://memcached.googlecode.com/files/memcached-1.4.5.tar.gz tar xzf memcached-1.4.5.tar.gz cd memcached-1.4.5 ./configure --with-libevent=/usr/local/ make sudo make install wget http://launchpad.net/libmemcached/1.0/0.44/+download/libmemcached-0.44.tar.gz tar xzf libmemcached-0.44.tar.gz cd libmemcached-0.44/ ./configure make sudo make install
Now you can start memcached and allocate a fixed amount of memory to it.
memcached -u nobody -d -m 512
One thing I learned the hard way is to always allocate the amount of memory memcached should use. If you dont specify the memory it defaults to 32MB which is pretty small. If you have a growing dataset and traffic I suggest you dedicate some boxes to memcache and start memcache with 90% of available memory.
Few tools to monitor memcached
The best way to get few stats from memcached is using by collecting internal metrics.
$ echo stats | nc localhost 11211 ... STAT total_items 0 STAT expired_unfetched 0 STAT evicted_unfetched 0 STAT evictions 0 ... $ echo stats items | nc localhost 11211 // some item related stats $ echo stats slabs | nc localhost 11211 STAT 1:chunk_size 96 STAT 1:chunks_per_page 10922 STAT 1:total_pages 1 STAT 1:total_chunks 10922 STAT 1:used_chunks 1 STAT 1:free_chunks 10921 STAT 1:free_chunks_end 0 STAT 1:mem_requested 67 STAT 1:get_hits 0 STAT 1:cmd_set 1 STAT 1:delete_hits 0 STAT 1:incr_hits 0 STAT 1:decr_hits 0 STAT 1:cas_hits 0 STAT 1:cas_badval 0 STAT 1:touch_hits 0 STAT active_slabs 1 STAT total_malloced 1048512
Monitoring slab related memcache is one of the most important things. As you launch you product and keep developing you add more attributes to your cache entry where the average item size increases over time which causes an effect called CACHE CALCIFICATION which means that your average items are now being stored in a slab which has less memory allocated. The new version of memcached 1.4.15 claims to solve it by supporting operations like slabs reassign -1 15.
Now that your memcached is up and running. We can see all the keys your memcached holds with memdump
Specify the list of servers for memdump to introsect. This is useful for development purposes.
Now on to my most favorite tool -> memstat. Memstat collects a bunch of various statistics about how your memcached is perfoming.
memstat --servers=127.0.0.1 Server: 127.0.0.1 (11211) pid: 10394 time: 1305424990 version: 1.4.5 pointer_size: 64 curr_connections: 42 total_connections: 177741649 connection_structures: 308 cmd_get: 1633514457 cmd_set: 439392889 get_hits: 1493612276 get_misses: 137902181 delete_misses: 9951479 delete_hits: 8166456 bytes_read: 770672258573 bytes_written: 2206367771075 limit_maxbytes: 6878658560 conn_yields: 11785 bytes: 6163615697 curr_items: 3736681 total_items: 235136283 evictions: 181308239 reclaimed: 4226382
I have trimmed the output to only include the useful stats. The most important ones I lookup for is get_hits and get_misses. This gives you your memcache hit ratio. Another most important one to keep in mind is the evictions. As your evictions increase over time that means its time to scale up your memcache servers. Also you can see that memcache is holding a lot of connections, may be connection pooling them can also improve the performance.
Along with these tools there are a few handy tools to remove keys from a cluster, copy keys etc.