zql for zodb

this is just a dump of an email i sent out to the zope3-users list. i wanted to put it here too, so i can make edits on the ideas as i think about it.

I was thinking about this the other night, and thought i'd try to put
my thoughts into some kind of order, and then ask the questions I came
up with.

I was thinking about the fact that i don't know of a meaningful query
language to use with the zodb. i know hurry.query is excellent
against catalogs, i've used it in my own applications more than a few
times. but there isn't really a simple way to query the zodb outside
of python code, and outside of a zope3 application (that i know of).
the "that i know of" is a pretty big part of this... if this kind of
thing has already been addressed and i just missed it, then all i'd
ask is to be pointed in the right direction :D

i'm going to call a zodb query language zql for now, because i like it
and think it's catchy ;) ...i know there's a java query parser called
"zql" also, but i'm not proposing a formal name, so i'm going to use
it anyway. i'd think of zql as just an OQL specific to the zodb.

i think in some ways it would lower the bar for entry to zope3 and its
data storage methodology if the zodb had a query language associated
with it that allowed reports on stored content to be pulled, perhaps
allowed simple permission grants, and other data manipulation. it
would, to my mind, be relevant only to describe a DML not a DDL
feature set for a "zql" language spec. all object metadata would be
defined by the interface, so wildcard "show me all attributes on
object x" would determine what should be showed through the registered
interface.

the below are some of the things I think would be necessary:

a zmi per-site registerable zql listener utility, open on it's own
port, built with the ability to traverse all objects below the
installed context and register their interfaces in it's own indexes.

a zql shell, attaching to a zql listener with a connection string like
"zql admin@localhost/mysite/myzqllistener"

the zql language definition should/could support pythonic expressions,
and/or XPath expressions on data, as well as a smaller,
reasonable-given-these-are-objects subset of standard sql.

zql> select * from myobj;
zql> grant all on mycontainer.* to 'user'@mysite.pau;
zql> grant zope.View on myothercontainer.* to 'user2'@mysite.pau;
zql> commit;

commit just being transaction.commit()

other things... i think the zql listener utility should have the
ability to store it's own indexes on all objects providing a
particular interface. i think it should be a subscriber to
objectadded and objectdeleted events, and check each object against
its own set of interfaces. whenever an object at or below it's
current context is added, it indexes it for faster lookup. so i can
write zql like:

zql> select (first_name, last_name) as full_name, context from IMyObj
where last_name like "%mith";
full_name context
------------------ -----------------------------------------------
John Smith /mysite/container/child/johnsmith

or, returning as objects indicated by a decorator keyword pyobj:
zql> select (first_name, last_name) as pyobj(full_name),
pyobj(context) from IMyObj where last_name like "%mith";
[ u'John Smith', <mylib.MyObj object at 0xdeadbeef> ]

where IMyObj is just whatever interface for my objects that the zql
listener knows about.

i admit, i don't have a plan in place for implementing something like
this, but, at least in my head, it seems do-able. is there already a
project in place for anything like this that i've missed in my
googling, or has this come up and been abandoned in favor of other,
better ideas about accessing zodb data in a query-language-style?

any thoughts, comments, arguments, and discussion on things i'm not
thinking about, etc, please, feel free to let me know... this is
really just to spawn discussion because I was up too late last night
and started thinking about it ;)

RoR and Zope3

i'm not really sure how to feel about this.

i REALLY like ruby on rails. this is difficult for me to come to terms with immediately. i love python. and, at least from my first taste of ruby, i'll probably always like python's syntax better. python taught me to program, not just script. but i guess we'll see as i use ruby more. the frustrating thing is that i love python, and i love zope3. theoretically.

i've spent a lot of time with zope3. at least 6 months or more, i don't really remember now. i admit, i was new to adapter/interface oriented design when i started with zope3, so a lot of the "why's" and "how's" of the framework took me some time to get into. then there's the learning curve of zcml and tales, and the magic of adapter registry and default views and all sorts of things. Stephan Richter and Phillip VW's books were/are both excellent, and got me through the initial trials and tribulations.

but i guess what i've been feeling lately is that i'm fighting with the framework as much as i'm developing in it. core things that i feel are indispensable from modern web frameworks, like simple ajax support and simple rdb integration, just aren't there. you can roll your own certainly, zope3 is nothing if not extensible, but i'm often left with the question of "what's the right way to do this? what's the right design pattern to use here?"

my trouble with zope3 is not how powerful it is. it's extremely powerful. my trouble with zope3 is getting at that power. a lot of things that i want to just be simple, aren't. i wanted to build my own login form with a few other parameters and i had to develop my own PAU just to do it the way i wanted to. and i understand my requirements were a bit picky on what needed to happen, but it just took so long to fight with it...

and i'm not talking about whether or not i'm too lazy to write my own way around these problems, i'm talking about whether or not i have time to roll my own, given the deadlines i'm working with. and whether i have time to go back and change an interface, a class, zcml, and potentially have to use generations code, every time the requirements change a little, or we want to capture a new bit of data, or move a piece of trackable data from one class to another.

ruby on rails' ability to be data-determined from the start in effect forces the framework to be adaptable to changes, since most of what it's displaying is data driven anyway.

and i think a lot of the blame is to be shifted back on to me. zope3 is, for better or worse, a CMS at it's core. and for the first few applications i needed to build in it, that was more than ok. i needed to implement a lot of functionality that would expose to a user the ability to create and manage content objects, and pass that content back and forth. i learned a lot, not just about zope3, but about programming in general. zope3 made me a better programmer.

but now i want it to be a web framework that can simply and efficiently do more than just content management, and i think i may be trying to use the wrong tool for the job.

i can see a lot of places where ruby on rails can't/won't be able to compete with zope3. but i can see a lot of places where it can both compete, and win.

i don't know. i'm scheduled for the may8-11th zope3 training with stephan, and i hope to be able to maybe see the pieces i'm missing with zope3. because right now i feel like zope3 is the best tool for the job for complex content management dealing with a large user bases. but i don't see it as the right tool for agile (both the methodology and in a more visceral sense) development and the ability to quickly adapt to refined/redesigned requirements.

i haven't come to a real conclusion in my thoughts on this. i still love zope3. but i'm disenchanted with it, the honeymoon is over. it has weak points. i know the hardcore dev guys can easily brush that off as my lack of skill or requisite knowledge. but i'm a smart guy, and i figure stuff out pretty quick, without much help usually, and there are things that are just downright difficult to do in zope3 that should be easier.

i guess though, my frustration may be that zope3 is fairly bleeding edge tech. the real frustration is that there just aren't plugins for a lot of things that i'd like. i'd like to be able to download "jsonform" or something that has the same render/update patterns as formlib, and just let it work. i'd like for zope3 to have an orm that works like sqlalchemy integrated at the transaction level. i know there's a project for this, and they have my impressed applause, but it just seems to me that ORM mapping should be so core to the framework as to be a utility, just like a PAU or something similar, where interfaces are registered to an rdb, and that's the last time the developer has to think about it.

i want there to be one standard way to create forms, no magic auto form generation that confuses the issue. the ruby strategy of simplicity based on convention rather than configuration is something that should be seriously considered. the long running thread of "zcml should do less" that went on a while ago highlights this.

i want modification of my interfaces and objects to be gracefully handled by the architechture, rather than having it croak if i change a TextLine to a PersistentList in some utility. if an object in the db no longer conforms to the interface, flag it and deprecate it out of being loaded, and warn to the log or console that the object no longer conforms and needs to be updated.

i've probably ranted too much now. i don't have the skill level to make any of that happen. but i want to keep using zope3 for more than just my CMS needs. i believe it is, at its core, a more extensible, powerful set of technology than ruby on rails. but ruby on rails is data driven, and agile, and most of all, simple to get going very fast with. i've spent a lot of time, energy, and money on zope3, and i feel like i can say without feeling out of line that it is not any of those things. like a cliff face it is imposing and powerful and unassailably potent. and learning how to make it do what you want is like climbing that same cliff face.

the choice for me, and i think a lot of developers, is not "what solves the problem most elegantly?" but "what solves the problem NOW, and lets me solve the evolving problems as they emerge, with the least downtime, the least overhead, and the least frustration?"

argh. i'm going to bed.

Zope3 PAU, redeux

I finally feel like I have a pretty good handle on Zope3's PAU. The pieces that had been confusing me have kinda fallen into place.

Admittedly, for things to "make sense" for me, I ended up writing my own PAU (not from scratch, but overriding the significant methods for PluggableAuthentication out of zope.app.authentication.authentication). I also overrode PrincipalFolder to authenticateCredentials in the way I wanted to, as well as created a credentials plugin to check both session and potential cookie data.

I had to build my own PAU because I needed my authenticated principals and principal info objects to return ConsumerPrincipal's and ConsumerPrincipalInfo objects, rather than PrincipalInfo and Principal's. pau, when pau.authenticate is called, actually re-runs AuthenticatedPrincipalFactory and adapts a Principal object out of the stored PrincipalInfo from the PrincipalFolder. because i built my own ConsumerPrincipals to store in my own ConsumerPrincipalFolder, i couldn't just drop them into the standard pau and get back principals that actually had all of the info i wanted from them. because my consumer principals were able to be adapted back down to the simpler level (Principal and PrincipalInfo) from which they were originally derived, this was, for some time, a transparent action. but when i needed to get principal.organization, which was one of my ConsumerPrincipalInfo extra fields, it wasn't available. on tracking it down, I realized what had happened.

I suppose that's the trouble with adapters, is that if you don't know they're performing an adaption operation behind the scenes, you may not always end up with the kind of object you think you should be getting, based on where it's adapting from.

I don't know enough about dynamic module loading/unloading from a particular memory space to really suggest it, nor do I fully understand the potential security issues (though I think if the dynamically loaded module is protected in the correct ways in zcml, it shouldn't be much of an issue), to really suggest a "how" for this...

but for the PAU, to make it more generic, and really "pluggable", since the way it stands now, it's only "pluggable" so long as you use a standard principal and principalfolder, i would be interested in seeing if the pau could inspect the kind of principalinfo obj being adapted, and return a principal obj of the intended type. might require a kind of registry between principalinfo, principal, and principalfolder though. right now though, if you add features to your own implementation, they don't translate through.

probably just smarter/simpler to leave the onus of the adapter building to the developer who needs other functionality. still, it would have been nice to know at the outset that pau.authenticate just wasn't going to give me the kind of principal i had actually created.

Oh well. Point is, I understand how it works now, and have created my own authentication implementation that overrides and provides all the necessary bits. works out just as well, since i was going (am going) to need to change the way "checkPassword" works anyway, since right now it forces a run of "encodePassword" for sha password managers. theoretically that's what you'd want, but i'd rather sha-hash the password the moment it's posted to the form, and then store the hash in the session, as well as in the cookie. i just feel slightly more secure that way.

Zope3 PAU

I've been working with zope3's PAU a bunch recently, and while I'm still hazy on a few points, a lot of things have crystalized for me that I didn't really "get" before. I originally worked with PluggableAuth from the 3.0 release, so moving to zope.app.authentication and the new PAU model was a little trying. not bad, but I just didn't really get how a lot of the bits worked together until after I'd spent rather a lot of time on it. And some pieces I'm still not clear on.

The first thing that finally "clicked" for me was the understanding of the "how and when" for the different PAU plugins. Each PAU has two primary sets of plugins, authenticator and credentials. Authenticator I understood more immediately, as it is the plugin that actually contains principals. Why you need principals and a principal container isn't really something that you can miss, so I didn't ;).

the credentials plugin though is something I didn't really understand before. Because the login and password are associated with the principal in the authenticator plugin, i didn't know where the credentials plugin really "fit", even though I'd read through the zope.app.authentication README.txt, principalfolder.txt, etc. I saw them in action, but didn't understand where in the overall "process" it fit.

The order of operations for principal login seems to be:
- form/challenge for username and password (or your own set of credentials)
- a credentials plugin extracts the credentials (with extractCredentials) from the request, and returns a dict of {'login' : login, 'password': password}.
- those credentials are then handed to the authenticator plugin, which runs authenticateCredentials(credentials) and returns a PrincipalInfo object based on the credentials passed in.
- the PAU itself then actually finds the associated principal, and returns it.

my current problem is "returns it where?"

principal = pau.authenticate(self.request) (where request has the necessary credentials in it) works, and, if you want, you can run self.request.setPrincipal(principal), which, for the current request, establishes that the authenticted principal should be used.

outside of setting that login information in cookies or a session though, i don't know how to keep a user authenticated by the PAU for the rest of the transaction/browser session.

I'll post more on it when I understand it better.

Zope3 Viewlets and Formlib

I recently came upon a situation where I needed to embed a number of forms inside viewlets to be displayed "dashboard" style from a primary application page in zope3. I've been trying to migrate away from 3.0/3.1's zope.app.form.browser's view libraries and toward zope.formlib.form's form builders. I figured I'd document what I put together to make it work, in case anyone else ends up trying the same thing. It didn't take long, but still... more documentation is better ;)

I started with a simple container class that I wanted to be able to be able to create a form for that would allow me to edit the values of a child object (with a form of it's own) in a viewlet. the class creation and zcml for it were simple enough, I won't relate them here, as they're pretty standard.

I then created a child object that, for our purposes, I'll call "Silly." The class and interface were as follows:

<code>
class ISilly(Interface):
    silly = Bool(title = _(u'This is Silly!'), description = _(u'I bet it is'), )
    someText = TextLine(title = _(u'Silly Text'),description = _(u'I am happy text'), )

class Silly(Persistent):
    """silly object"""
    implements(ISilly)
    silly = True
    someText = u''
</code>

next i created a viewlet manager, and the necessary zcml

<code>
class ISillyViewletManager(IViewletManager):
    """silly viewlet manager"""

from zope.viewlet.manager import ViewletManager
SillyViewletManager = ViewletManager(ISillyViewletManager)
</code>
<zcml>
<!--silly viewlet manager -->
<browser:viewletManager
     name=".silly.ISillyViewletManager"
     layer="petetest"
     permission="zope.Public"
     provides=".silly.ISillyViewletManager"
     class=".silly.SillyViewletManager"
     template="silly.zpt"
     />
</zcml>

next we actually build the form that we want to embed in the viewlet. as srichter mentioned to me, both formlib and viewlet use the same render/update patterns, so making a form inside a viewlet really flows very naturally.

the only thing i want to point out in the code below (explicitly) is the change in context. if you want to derive from editform, you need to change the context at the outset. if you want to derive from form.Form, you don't have to change it in __init__, but before you build any functionality that uses self.context, you have to make sure that it's actually pointing at the objects you want to affect. in this case, 'silly1' is an ISilly object contained in the current context.

<aside>
the reason for this is that setUpWidgets in EditForm pulls in the current zodb stored values to populate the fields with by running form.setUpEditWidgets. this is the behavior i wanted, but it does that at render. form.Form doesn't pull those values in, so you can theoretically wait to change self.context until later, if you override setUpWidgets out of form.Form.
</aside>

<code>
class SillyViewletForm(form.EditForm):

    implements(IViewlet)

    def __init__(self, context, request, view, manager):
        super(SillyViewletForm, self).__init__(context, request, view, manager)
        self.__parent__ = view
        self.context = context['silly1']
        self.request = request
        self.manager = manager

    prefix = u'silly'
    form_fields = form.Fields(ISilly, omit_readonly=True)
    template = ViewPageTemplateFile('sillyviewlet.zpt')
</code>

notice also that setting "prefix" explicitly ensures that you don't have field name overlap between viewlets and primary form fields (or other viewlets, even). one other piece that may look innocuous from a quick glance is the template line. while the viewlet zcml will accept a template parameter, it will not act on it in any meaningful way, and will instead assume that it should use a namedtemplate of 'default' since that's what all of formlib-derived forms use unless otherwise instructed. in our case, we set template to zope.app.pagetemplate's ViewPageTemplateFile, with a filename of the template we want to frame our viewlet in. a final bit to notice is that our SillyViewletForm implements IViewlet. necessary piece.

this is the zcml for this viewlet:

<zcml>
<browser:viewlet
    name="silly"
    for="*"
    class=".silly.SillyViewletForm"
    manager=".silly.ISillyViewletManager"
    permission="zope.Public"
    layer="petetest"
    />
</zcml>

the remaining configuration and use is simple, and is outlined in formlib more than adequately, but i'll include it here just for reference. to use a viewletmanager (you only use viewletmanagers, not viewlets themselves, at least as of 3.2) in tal markup on a page template, it's included like this:

<talmarkup>
<tal:block replace="structure provider:petetest.browser.silly.ISillyViewletManager" />
</talmarkup>

the actual page template markup to display the viewlets in a viewletmanager. in this example, this was silly.zpt:

<talmarkup>
<div class="box">
    <div class="entry"
      tal:repeat="viewlet options/viewlets"
      tal:content="structure viewlet/render" />
</div>
</talmarkup>

the final piece of the whole bit is what the form page template that makes up the viewlet form itself should look like. this is obviously up to the individual designer/developer, but for our example, i literally just took pageform.pt from zope.formlib and modified it slightly, such that it doesn't extend the view macro, nor define itself as main. if you don't extend or use view or page, you'll need to get rid of the fill-slot="body" div. i also redefined the define-macro defined at the <form level from "master" to "silly" so that the form itself wouldn't be in conflict with any other more generically derived combinations of tal on the page.

this is a pretty quick-and-dirty runthrough, but i think it has enough info that if i'd seen it when i started, i would have understood what was going on more quickly. if anyone has any comments, critiques, or suggestions, please let me know. baldtrol_at_gmail_dot_com. Thanks to Stephan Richter, Gary Poster, and Alen Stanisic for their help while I was figuring this out :)