zql for zodb

this is just a dump of an email i sent out to the zope3-users list. i wanted to put it here too, so i can make edits on the ideas as i think about it.

I was thinking about this the other night, and thought i'd try to put
my thoughts into some kind of order, and then ask the questions I came
up with.

I was thinking about the fact that i don't know of a meaningful query
language to use with the zodb. i know hurry.query is excellent
against catalogs, i've used it in my own applications more than a few
times. but there isn't really a simple way to query the zodb outside
of python code, and outside of a zope3 application (that i know of).
the "that i know of" is a pretty big part of this... if this kind of
thing has already been addressed and i just missed it, then all i'd
ask is to be pointed in the right direction :D

i'm going to call a zodb query language zql for now, because i like it
and think it's catchy ;) ...i know there's a java query parser called
"zql" also, but i'm not proposing a formal name, so i'm going to use
it anyway. i'd think of zql as just an OQL specific to the zodb.

i think in some ways it would lower the bar for entry to zope3 and its
data storage methodology if the zodb had a query language associated
with it that allowed reports on stored content to be pulled, perhaps
allowed simple permission grants, and other data manipulation. it
would, to my mind, be relevant only to describe a DML not a DDL
feature set for a "zql" language spec. all object metadata would be
defined by the interface, so wildcard "show me all attributes on
object x" would determine what should be showed through the registered
interface.

the below are some of the things I think would be necessary:

a zmi per-site registerable zql listener utility, open on it's own
port, built with the ability to traverse all objects below the
installed context and register their interfaces in it's own indexes.

a zql shell, attaching to a zql listener with a connection string like
"zql admin@localhost/mysite/myzqllistener"

the zql language definition should/could support pythonic expressions,
and/or XPath expressions on data, as well as a smaller,
reasonable-given-these-are-objects subset of standard sql.

zql> select * from myobj;
zql> grant all on mycontainer.* to 'user'@mysite.pau;
zql> grant zope.View on myothercontainer.* to 'user2'@mysite.pau;
zql> commit;

commit just being transaction.commit()

other things... i think the zql listener utility should have the
ability to store it's own indexes on all objects providing a
particular interface. i think it should be a subscriber to
objectadded and objectdeleted events, and check each object against
its own set of interfaces. whenever an object at or below it's
current context is added, it indexes it for faster lookup. so i can
write zql like:

zql> select (first_name, last_name) as full_name, context from IMyObj
where last_name like "%mith";
full_name context
------------------ -----------------------------------------------
John Smith /mysite/container/child/johnsmith

or, returning as objects indicated by a decorator keyword pyobj:
zql> select (first_name, last_name) as pyobj(full_name),
pyobj(context) from IMyObj where last_name like "%mith";
[ u'John Smith', <mylib.MyObj object at 0xdeadbeef> ]

where IMyObj is just whatever interface for my objects that the zql
listener knows about.

i admit, i don't have a plan in place for implementing something like
this, but, at least in my head, it seems do-able. is there already a
project in place for anything like this that i've missed in my
googling, or has this come up and been abandoned in favor of other,
better ideas about accessing zodb data in a query-language-style?

any thoughts, comments, arguments, and discussion on things i'm not
thinking about, etc, please, feel free to let me know... this is
really just to spawn discussion because I was up too late last night
and started thinking about it ;)