You can add hooks to the datastore, as I already showed in a previous (profiling) post. However, in order to return a cached entity instead of one retrieved from the datastore, we need more than a PreCall hook. From a PreCall hook will can not prevent the actual datastore call being made. Maybe we can “trick” MakeSyncCall by removing the keys from the argument list, but that is not a road I want to explore…

So, we need to replace the actual apiproxy.MakeSyncCall. Please be aware that there already may be references to the original MakeSyncCall, as described in this post. For our caching solution, this is not a real problem: the worst thing that could happen, is that there will be no caching. So, the app will be slower, but still functioning the same…

Replacing a registered datastore Stub

Replacing the registered datastore stub may seem like the way to go, if you want to wrap the datastore_v3 call. However, registering a stub is allowed only when there is no stub registered yet. And, you guessed it, in production there is already a stub present. This method therefore, is only useful for unit testing and not for production. We could replace the global MakeSyncCall, but then we add overhead for every remote call, which is something I do not prefer.

Replacing db.get

If we want to retrieve a model by using a key or key_name, all calls to the datastore are implemented using the global module function db.get. So, if we can wrap this call with a caching function, we’re done. Here’s the solution I came up with:

# define a global, weakly referenced CACHE...
_cache = weakref.WeakValueDictionary({})

def getCached(real_func, keys):
    real_keys, multiple = datastore.NormalizeAndTypeCheckKeys(keys)
    if multiple: # too difficult for now ;-)
        return real_func(keys)

    real_key = real_keys[0]
    instance = _cache.get(real_key, None)
    if instance:
        return instance
    model = real_func(keys)
    if model:
        _cache[real_key] = model
    return model

# Patch db.get
from functools import partial
db.get = partial(getCached, db.get)

A global function getCached is defined, which wraps the original db.get function. First, we check whether the function is called for multiple keys. If that is the case, we simply call the original function (implementing this would be too much code for this example). Otherwise, we check whether the key is already present in cache. If so we return that value, otherwise the real db.get is called and the result is added to the cache.

Query results

How about query results? Are they added to the cache as well? Unfortunately, they are not. When query results are fetched, the entire entity is retrieved at once, so there is no need to get the values by key. It would be nice if we could add the retrieved entities to the cache, so that we don’t have to retrieve them again for a ReferenceProperty value.

This turns out to be more difficult. We could subclass db.Query (and db.GqlQuery) so that we could implement a different iterator (a caching subclass of db._QueryIterator), but that seems a bit tedious. Besides, we would have to override more functions, since the model values are not always retrieved using the iterator. The fetch function, for example, uses a map function to run the entities over the Model.from_entity function.

Replacing Model.from_entity

Given the fact that entities are converted to Model instances by the Model.from_entity function, gives us a nice hook to implement a caching strategy, doesn’t it?

All we have to do, is subclass db.Model and re-implement the from_entity function:

class CachingModel(db.Model):

    @classmethod
    def from_entity(cls, entity):
        model = super(CachingModel,cls).from_entity(entity)
        if model:
            _cache[model.key()] = model
        return model

The resulting CachingModel class is the one we subclass to implement our business models, like this:

class Team(CachingModel):
    club     = db.ReferenceProperty(Club)
    naam     = db.StringProperty()
    poule    = db.ReferenceProperty(Poule)

If we add all model instances to the cache in the from_entity function, we can remove that from our global getCached function…

def getCached(real_func, keys):
    real_keys, multiple = datastore.NormalizeAndTypeCheckKeys(keys)
    if multiple: # too difficult for now ;-)
        return real_func(keys)

    real_key = real_keys[0]
    instance = _cache.get(real_key, None)
    if instance:
        return instance
    model = real_func(keys)
#   if model:
#       _cache[real_key] = model
    return model

We now have a complete, in-memory, caching solution, which can enhance our application performance significantly with a minimal effort!

Now this is quite different from the GAE datastore. As we saw in my previous post, every reference to an existing entity is

• fetched separately from the datastore, and
• leads to a new instance in memory.

Using memcache

Of course, we already have a caching mechanism in GAE: memcache. So, let’s use that and figure out if that will give the performance boost we need.

In order to prevent referenced entities to load more than once, we can subclass db.ReferenceProperty and use our own version:

class CachedReferenceProperty(db.ReferenceProperty):

  def __init__(self,
               reference_class=None,
               time=0,
               verbose_name=None,
               collection_name=None,
               **attrs):
    super(CachedReferenceProperty, self).__init__(reference_class, verbose_name, collection_name, **attrs)
    self.time = time

In order to intervene with the actual database access, we need to override the __get__ method. For convenience I have duplicated the original __get__ method in our class and patched it a bit:

  def __get__(self, model_instance, model_class):
    if model_instance is None:
      return self
    if hasattr(model_instance, self.__id_attr_name()):
      reference_id = getattr(model_instance, self.__id_attr_name())
    else:
      reference_id = None
    if reference_id is not None:
      resolved = getattr(model_instance, self.__resolved_attr_name())
      if resolved is not None:
        return resolved
      else:
        reference_str = str(reference_id)
        instance = memcache.get(reference_str)
        if instance is None:
          instance = db.get(reference_id)
          memcache.set(reference_str, instance, self.time)
        else:
          logging.info("CachedReferenceProperty - found object in cache: "+str(reference_id))
        if instance is None:
          raise Error('ReferenceProperty failed to be resolved')
        setattr(model_instance, self.__resolved_attr_name(), instance)
        return instance
    else:
      return None

Now, in order to get this code working, I had to duplicate the two methods below as well. Since I am a Python novice, I don’t understand why I had to do this, but it works… (if someone knows, please leave a comment )

  def __id_attr_name(self):
    return self._attr_name()

  def __resolved_attr_name(self):
    return "_RESOLVED"+self._attr_name()

And now the results of actually applying this caching method. I used the CachedReferenceProperty with a 5 second timeout, which should be good enough to cache the entity during the request lifetime.

    90565 function calls (88670 primitive calls) in 3.601 CPU seconds
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       61    1.858    0.030    1.921    0.031 {google3.apphosting.runtime._apphosting_runtime___python__apiproxy.Wait}
      523    0.674    0.001    0.674    0.001 {posix.stat}
       61    0.141    0.002    0.160    0.003 apiproxy.py:119(_MakeCallImpl)

    81628 function calls (80775 primitive calls) in 2.840 CPU seconds
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       39    1.169    0.030    1.222    0.031 {google3.apphosting.runtime._apphosting_runtime___python__apiproxy.Wait}
      973    1.130    0.001    1.130    0.001 {posix.stat}
       37    0.034    0.001    1.214    0.033 traceback.py:273(extract_stack)
       39    0.027    0.001    0.039    0.001 apiproxy.py:119(_MakeCallImpl)

The lesson to be learned here, is that memcache should be used on a more granular level, caching actions that involve a lot more than just getting an entity from the datastore by key…

Using a dictionary

Since we only want to cache entities during the duration of a request, the entities could just as well be stored locally in memory. So we change our CachedReferenceProperty class:

class CachedReferenceProperty(db.ReferenceProperty):

  _cache = weakref.WeakValueDictionary({})
    
  def __get__(self, model_instance, model_class):
    ...
        instance = CachedReferenceProperty._cache.get(reference_id, None)
        if instance is None:
          instance = db.get(reference_id)
          CachedReferenceProperty._cache[reference_id] = instance
        else:
          logging.info("CachedReferenceProperty - found object in cache: "+str(reference_id))
    ...

    66706 function calls (65996 primitive calls) in 1.578 CPU seconds
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      538    0.619    0.001    0.619    0.001 {posix.stat}
       23    0.543    0.024    0.581    0.025 {google3.apphosting.runtime._apphosting_runtime___python__apiproxy.Wait}
       23    0.028    0.001    0.036    0.002 apiproxy.py:119(_MakeCallImpl)

The number of apiproxy.Wait calls decreased from 39 to 23 and, better, the response time decreased from 2.8 to 1.6 seconds!

Profiling, first attempt

It’s time to find out what’s going on. So let’s turn profiling on! There’s a FAQ entry on how to do that. After adding the code and running it on the development server, I was a bit puzzled. The majority of the time is spent on eval() functions. Digging through the output a bit, made me realize profiling should not be executed on the development server.

    1873187 function calls (1871823 primitive calls) in 10.741 CPU seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    15174    0.763    0.000    2.506    0.000 datastore_types.py:175(from_path)
     7587    0.675    0.000    4.605    0.001 {eval}
    15174    0.610    0.000    1.386    0.000 datastore_types.py:422(__repr__)
     7609    0.530    0.000    1.308    0.000 datastore_types.py:442(__cmp__)

It looks like querying the datastore is done by creating expressions using repr() and subsequent calls to eval() to check the query conditions…

Profiling, 2nd attempt

Running the same code on Google App Engine reveals totally different usage stats:

    64751 function calls (63972 primitive calls) in 0.957 CPU seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       37    0.625    0.016    0.674    0.017 {google3.apphosting.runtime._apphosting_runtime___python__apiproxy.Wait}
        2    0.018    0.009    0.018    0.009 {method 'read' of 'file' objects}
       37    0.017    0.000    0.028    0.001 apiproxy.py:119(_MakeCallImpl)

As you can see above, time is mostly (65%) spent on a call to apiproxy.Wait. So, what is Google waiting for? I can guess, of course: datastore I/O. So, let’s find out if that is the case…

DB Profiling

For database profiling, I turned to an article: Using hooks in GAE. Using a “pre-call” hook, I log all datastore_v3 calls:

def patch_appengine():
    def hook(service, call, request, response):
        logging.info('%s %s - %s' % (service, call, str(request)))
        stack = traceback.format_stack()
        logging.debug('%s %s - %s' % (service, call, "n".join(stack)))
      
    apiproxy_stub_map.apiproxy.GetPreCallHooks().Append('db_log', hook, 'datastore_v3')

It turns out that 36 out of the 37 apiproxy.Wait are accounted for in the log. These are, as we expected, datastore calls. But there is an interesting thing going on, looking a part of the log:

3:39 30.425 Get - key .. {type: "Team" name: "Shot 1"}
3:39 30.506 Get - key .. {type: "Team" name: "Bijmaat 3"}
3:39 30.540 Get - key .. {type: "Team" name: "Almeerspin 2"}
3:39 30.580 Get - key .. {type: "Team" name: "Shot 1"}
3:39 30.614 Get - key .. {type: "Team" name: "Maarssen 1"}
3:39 30.651 Get - key .. {type: "Team" name: "Shot 1"}
3:39 30.678 Get - key .. {type: "Team" name: "Shot 1"}
3:39 30.702 Get - key .. {type: "Team" name: "SVO 2"}
3:39 30.732 Get - key .. {type: "Team" name: "Shot 1"}
3:39 30.755 Get - key .. {type: "Team" name: "Laren 1"}

As you can see above, my team is retrieved from the datastore for every single match, using an expensive (16 ms on average) remote procedure call!

Optimizing this is something I will work on in my next post.

Data

Besides this “official” data, I want users to be able to add there own content:

• reviews on tournaments they played / viewed,
• discussions on material (rubbers, frames, balls, tables etc.),
• general discussion forum.

This is a lot of data, which somehow needs to be organized on the screen.

Yahoo User Interface (YUI)

This is a research project, so I decided to try out a ready made Javascript and CSS library. I was drawn to the Yahoo CSS library, since it seems a solid solution for cross browser compatibility. I took on the javascript library as well: I also needed tabs, dialogs and data tables.

The initial experience with YUI was very good. A lot of documentation and examples are provided. However, as soon as I created something more complex, unexpected results would occur. Usually leading to: “Where did my tabs/datatable/dialog/etc. go?!?”. The required markup is quite strict. The javascript library is workable, but a bit tedious. Lot’s of components needed to throw together a workable component.

And to make things worse:  there will be a lot of differences between Firefox and IE. After working with it for 3-4 weeks, I gave up: Too much trouble and not enough fun for what I want to accomplish…

jQuery UI

At first I declined jQuery (UI) because there was no dataTable component…. I thought. I totally missed the plugins, since I assumed that would be solutions for some edge cases. Not the things you need when you first start using jQuery. I couldn’t be more wrong. The list of plugins is endless and most of them are maintained actively. There is a plugin for almost every problem you can think of…

And then I ran into the themable UI part. Since I eventually want the user interface to be customized by the end user, I was sold instantly. I can now offer the user a selection of over 10 themes to pick from, without any effort. And they all look very polished. Way to go!

It took me almost a week to migrate the application from YUI to jQuery, learning jQuery syntax on the fly. And that’s not hard at all. Very intuitive syntax, simple solutions and sensible defaults. That’s what I like! So, a big thank you to the jQuery team from me.

Programming in javascript just became fun!

Yesterday I ran into some strange problems. I use jQuery to issue ajax calls, to update parts of the screen, but under IE7 a DIV did not seem to be updated. After checking the development web server, I noticed that the Ajax call was not sent. Since I am pretty sure the Ajax call was issued, something else was going on.

Caching

The most probable cause was, that IE7 is caching the contents. Since I did not explicitly set caching options on the response, I checked this using Firebug. However, Firebug doesn’t show this. So I turned to “curl” using the verbose option. That clearly showed:

 Cache-Control: no-cache

And that’s exactly what I want! So, why does it not work? I did notice the development web server responded with HTTP/1.0. After “googling” around a bit, it turns out that the Cache-Control header was added in HTTP/1.1. IE7 was just very strict in applying the standards (that’s a change!).

Upgrading the development server

In trying to get this working on the Google App Engine development web server, I digged into the code of the DevAppServerRequestHandler class and added:

protocol_version = 'HTTP/1.1'

This turned out to be a bad idea. IE7 now got stuck when the page was loaded. According to the dev. web server the response was sent, but IE7 was still waiting for more. So, I reverted that change and tested it by deploying a new version to the Google App Engine, since this does support the HTTP/1.1 protocol. Now everything worked fine in IE7, the Ajax response was not cached and reloaded every single time…