[Community] ZCatalog spatial index

[Eric Bréhault]

Hello,

I use Rtree now but I still have some persistence issues:
as Rtree object cannot be pickled, my objective was to store the index in
the file system
so I have built a wrapper (named ZRtree) which has a rtree instance:
newrtree=Rtree("my_filename")
self.rtree=newrtree
using __getstate__ I make sure self.rtree is not pickled
and using __setstate__, I reload self.rtree from my_filename.idx and
my_filename.dat

unfortunately it doesn't work: the produced files are corrupted, it makes
Zope to crash violently when calling __setstate__, and if I try to load the
index manually from Python prompt, I get this error:
terminate called after throwing an instance of
'Tools::IllegalStateException'


the strange thing is:
- when running my unit test, it works fine, and I can load the exported
index files manually from Python prompt
- if I build a Rtree object but do not insert it in self (I mean if I remove
self.rtree=newrtree), the files are correct too

so it is definitely due to the ZODB and the persistence management
but I do not understand why it would impact my index files

Any idea ?

Eric

Here is the code of my wrapper:

from rtree import Rtree
import os
from random import random
from time import time
from md5 import md5

class ZRtree:
    """
    RTree wrapper
    The index content is saved in a file which will not be serialized in
ZODB
    """
    def __init__(self):
        filename = md5(str(time() *
1000L)+str(random()*100000000000000000L)).hexdigest()
        self.file = filename
        self.rtree = None

    def getRtree(self):
        if self.rtree is None:
            try:
                os.remove(file+'.dat')
                os.remove(file+'.idx')
            except:
                pass
            r=Rtree(self.file)
            self.rtree = r
        return self.rtree

    def add(self, i, coords):
        return self.getRtree().add(i, coords)

    def delete(self, i, coords):
        return self.getRtree().delete(i, coords)

    def intersection(self, coords):
        return self.getRtree().intersection(coords)

    def __getstate__(self):
        odict = self.__dict__.copy()
        del odict['rtree']
        return odict

    def __setstate__(self,dict):
        self.__dict__.update(dict)
        try:
            self.rtree = Rtree(self.file)
        except Exception(e):
            logger.error(e)


On 7/30/07, Kai Lautaportti <kai.lautaportti at hexagonit.fi> wrote:
>
> Hi Eric and Sean
>
> Eric Bréhault wrote:
> > Hello Sean,
> >
> > Yes, I found the persistence problem.
> > My solution was to let the Quadtree index volatile (index._v_quadtree)
> > so Zope does not try to persist it, and on first use, I rebuild it
> entirely.
>
> Using a volatile attribute to store the index may produce unexpected
> behaviour if you expect it to be built once on startup and be maintained
> indefinitely there after (until the next Zope restart of course).
>
> The ZODB does not (in theory) make any promises how long a volatile
> attribute is stored and you might end up needing to rebuild the index
> more often than you wish. Also the per connection locality of volatile
> attributes and the fact that the index depends on data from multiple
> objects may prove troublesome.
>
> More info at http://wiki.zope.org/ZODB/VolatileAttributes
>
> >
> > I will use ArrTree.
>
> I also think this will be the best option for us.
>
> cheers,
> Kai
>
> >
> > Thanks,
> >
> > Eric
> >
> > On 7/29/07, *Sean Gillies* <sgillies at frii.com
> > <mailto:sgillies at frii.com>> wrote:
> >
> >     Eric Bréhault wrote:
> >     > Hello,
> >     >
> >     > Did anybody already tried to implement a spatial index to handle
> >     spatial
> >     > operators directly in the ZCatalog ?
> >     > I want to try to do it, but if anybody has some feedback and/or
> >     idea about
> >     > it, it would help.
> >     >
> >     > Regards,
> >     >
> >     > Eric
> >
> >     Shaun Walbridge and I worked on
> >
> >        http://trac.gispython.org/projects/PrimaGIS/browser/SpatialIndex
> >     <http://trac.gispython.org/projects/PrimaGIS/browser/SpatialIndex>
> >
> >     at the 2006 Plone Conference sprint. The tests should be
> >     self-explanatory. The one obstacle for us at the time was that there
> >     wasn't a way to persist the index, which means that it's not really
> >     ready for production.
> >
> >     Now, there is an alternative index
> >
> >        http://trac.gispython.org/projects/PCL/wiki/ArrTree
> >
> >     which *can* be persisted (to a file on disk), and would be a better
> >     foundation for a Zope spatial index. It's been on my to-do list for
> a
> >     while. My Pleiades project has ~1000 features now, but will need a
> >     spatial index before it can grow much larger.
> >
> >     Cheers,
> >     Sean
> >
> >     _______________________________________________
> >     Community mailing list
> >     Community at lists.gispython.org <mailto:Community at lists.gispython.org>
> >     http://lists.gispython.org/mailman/listinfo/community
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Community mailing list
> > Community at lists.gispython.org
> > http://lists.gispython.org/mailman/listinfo/community
>
>
> --
> Kai Lautaportti               +358-50-558-7935
> Software engineer             www.hexagonit.fi
> Hexagon IT Oy                 kai.lautaportti at hexagonit.fi
> _______________________________________________
> Community mailing list
> Community at lists.gispython.org
> http://lists.gispython.org/mailman/listinfo/community
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gispython.org/pipermail/community/attachments/20070808/9673ba82/attachment.htm>