client.db_query() behaves different with SQLite and MongoDB

[kaazoo]

The following code with IPython on Mac OSX and MongoDB

client.db_query({"header.session" : {"$ne" : ""}}, keys=["header.session"])

produces for example

[{'header': {'session': 'job029'},
  'msg_id': '817e5aa3-6a37-4211-b466-17249f141256'},
 {'header': {'session': 'job030'},
  'msg_id': 'dea85cde-455b-456b-96b7-2654f16fe814'},
 {'header': {'session': 'job031'},
  'msg_id': 'd8cec676-8c24-49ef-9103-2611b9888630'},
 {'header': {'session': 'job040'},
  'msg_id': 'e9c42ce9-0ba8-425e-80cd-2946a069a843'},
 {'header': {'session': 'job041'},
  'msg_id': '5a343982-8d79-47b9-972a-ccd545b7b53d'}]

Using the same code on the SQLite backend gives the following exception

In [29]: client.ip_client.db_query({"header.session" : {"$ne" : ""}}, keys=["header.session"])
---------------------------------------------------------------------------
RemoteError                               Traceback (most recent call last)
/home/kaazoo/DrQueueIPython-git/<ipython-input-29-8ad4d152ecc9> in <module>()
----> 1 client.ip_client.db_query({"header.session" : {"$ne" : ""}}, keys=["header.session"])

/home/kaazoo/DrQueueIPython-git/<string> in db_query(self, query, keys)

/usr/local/lib/python2.7/dist-packages/IPython/parallel/client/client.pyc in spin_first(f, self, *args, **kwargs)
     46     """Call spin() to sync state prior to calling the method."""
     47     self.spin()
---> 48     return f(self, *args, **kwargs)
     49 
     50 

/usr/local/lib/python2.7/dist-packages/IPython/parallel/client/client.pyc in db_query(self, query, keys)
   1330         content = msg['content']
   1331         if content['status'] != 'ok':
-> 1332             raise self._unwrap_exception(content)
   1333 
   1334         records = content['records']

RemoteError: KeyError("Bad record key(s): ['header.session']")
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/IPython/parallel/controller/hub.py", line 1254, in db_query
    records = self.db.find_records(query, keys)
  File "/usr/local/lib/python2.7/dist-packages/IPython/parallel/controller/sqlitedb.py", line 299, in find_records
    raise KeyError("Bad record key(s): %s"%bad_keys)
KeyError: "Bad record key(s): ['header.session']"

[minrk]

Yes, that's true. MongoDB supports introspection into sub-entries, but the other backends do not.

We implement a subset of the MongoDB query interface in the SQLite backend, but certainly not all of it.

[kaazoo]

However the following works with SQLite

client.ip_client.db_query({"header" : {"$ne" : ""}}, keys=["header"])

So the problem seems to be the access on dictionary members of the 'header' attribute of each DB entry.

[kaazoo]

OK, if it's not possible with SQLite, how can I query a list of all session names? Currently, I do the following:

def query_job_list(self):
    """Query a list of all jobs (IPython sessions)"""
    jobs = []
    query_data = self.ip_client.db_query({"header.session" : {"$ne" : ""}}, keys=["header.session"])
    for entry in query_data:
        jobs.append(entry['header']['session'])
    jobs = set(jobs)
    jobs = list(jobs)
    jobs.sort()
    return jobs

[minrk]

Simply doing s/header.session/header/ should do it:

def query_job_list(self):
    """Query a list of all jobs (IPython sessions)"""
    jobs = set()
    query_data = self.ip_client.db_query({"header" : {"$ne" : ""}}, keys=["header"])
    for entry in query_data:
        jobs.add(entry['header']['session'])
    jobs = sorted(jobs)
    return jobs

That will fetch the whole header rather than just the session ID, but that's not a lot of data.

If querying on the session ID is sufficiently common, I can promote the session to a top-level entry, like the timestamps. However, 99% of the time header.session == client_uuid. Though that change would make your use of the session ID as job ID work on all backends.

[kaazoo]

OK, thanks.