[Community] ZCatalog spatial index

[Howard Butler]

Eric,

I suspect that exception is coming from the index validation in  
_rtreemodule.cc.  I have commented this out in SVN.  I can you do an  
update and see if you still have the problem?

Howard


On Aug 8, 2007, at 10:49 AM, Eric Bréhault wrote:

> Hello,
>
> I use Rtree now but I still have some persistence issues:
> as Rtree object cannot be pickled, my objective was to store the  
> index in the file system
> so I have built a wrapper (named ZRtree) which has a rtree instance:
> newrtree=Rtree("my_filename")
> self.rtree=newrtree
> using __getstate__ I make sure self.rtree is not pickled
> and using __setstate__, I reload self.rtree from my_filename.idx  
> and my_filename.dat
>
> unfortunately it doesn't work: the produced files are corrupted, it  
> makes Zope to crash violently when calling __setstate__, and if I  
> try to load the index manually from Python prompt, I get this error:
> terminate called after throwing an instance of  
> 'Tools::IllegalStateException'
>
>
> the strange thing is:
> - when running my unit test, it works fine, and I can load the  
> exported index files manually from Python prompt
> - if I build a Rtree object but do not insert it in self (I mean if  
> I remove self.rtree=newrtree), the files are correct too
>
> so it is definitely due to the ZODB and the persistence management
> but I do not understand why it would impact my index files
>
> Any idea ?
>
> Eric
>
> Here is the code of my wrapper:
>
> from rtree import Rtree
> import os
> from random import random
> from time import time
> from md5 import md5
>
> class ZRtree:
>     """
>     RTree wrapper
>     The index content is saved in a file which will not be  
> serialized in ZODB
>     """
>     def __init__(self):
>         filename = md5(str(time() * 1000L)+str(random() 
> *100000000000000000L)).hexdigest()
>         self.file = filename
>         self.rtree = None
>
>     def getRtree(self):
>         if self.rtree is None:
>             try:
>                 os.remove(file+'.dat')
>                 os.remove(file+'.idx')
>             except:
>                 pass
>             r=Rtree(self.file)
>             self.rtree = r
>         return self.rtree
>
>     def add(self, i, coords):
>         return self.getRtree().add(i, coords)
>
>     def delete(self, i, coords):
>         return self.getRtree().delete(i, coords)
>
>     def intersection(self, coords):
>         return self.getRtree().intersection(coords)
>
>     def __getstate__(self):
>         odict = self.__dict__.copy()
>         del odict['rtree']
>         return odict
>
>     def __setstate__(self,dict):
>         self.__dict__.update(dict)
>         try:
>             self.rtree = Rtree(self.file)
>         except Exception(e):
>             logger.error(e)
>
>
> On 7/30/07, Kai Lautaportti <kai.lautaportti at hexagonit.fi> wrote:  
> Hi Eric and Sean
>
> Eric Bréhault wrote:
> > Hello Sean,
> >
> > Yes, I found the persistence problem.
> > My solution was to let the Quadtree index volatile  
> (index._v_quadtree)
> > so Zope does not try to persist it, and on first use, I rebuild  
> it entirely.
>
> Using a volatile attribute to store the index may produce unexpected
> behaviour if you expect it to be built once on startup and be  
> maintained
> indefinitely there after (until the next Zope restart of course).
>
> The ZODB does not (in theory) make any promises how long a volatile
> attribute is stored and you might end up needing to rebuild the index
> more often than you wish. Also the per connection locality of volatile
> attributes and the fact that the index depends on data from multiple
> objects may prove troublesome.
>
> More info at http://wiki.zope.org/ZODB/VolatileAttributes
>
> >
> > I will use ArrTree.
>
> I also think this will be the best option for us.
>
> cheers,
> Kai
>
> >
> > Thanks,
> >
> > Eric
> >
> > On 7/29/07, *Sean Gillies* < sgillies at frii.com
> > <mailto:sgillies at frii.com>> wrote:
> >
> >     Eric Bréhault wrote:
> >     > Hello,
> >     >
> >     > Did anybody already tried to implement a spatial index to  
> handle
> >     spatial
> >     > operators directly in the ZCatalog ?
> >     > I want to try to do it, but if anybody has some feedback  
> and/or
> >     idea about
> >     > it, it would help.
> >     >
> >     > Regards,
> >     >
> >     > Eric
> >
> >     Shaun Walbridge and I worked on
> >
> >        http://trac.gispython.org/projects/PrimaGIS/browser/ 
> SpatialIndex
> >     <http://trac.gispython.org/projects/PrimaGIS/browser/ 
> SpatialIndex>
> >
> >     at the 2006 Plone Conference sprint. The tests should be
> >     self-explanatory. The one obstacle for us at the time was  
> that there
> >     wasn't a way to persist the index, which means that it's not  
> really
> >     ready for production.
> >
> >     Now, there is an alternative index
> >
> >        http://trac.gispython.org/projects/PCL/wiki/ArrTree
> >
> >     which *can* be persisted (to a file on disk), and would be a  
> better
> >     foundation for a Zope spatial index. It's been on my to-do  
> list for a
> >     while. My Pleiades project has ~1000 features now, but will  
> need a
> >     spatial index before it can grow much larger.
> >
> >     Cheers,
> >     Sean
> >
> >     _______________________________________________
> >     Community mailing list
> >     Community at lists.gispython.org  
> <mailto:Community at lists.gispython.org>
> >     http://lists.gispython.org/mailman/listinfo/community
> >
> >
> >
> >  
> ---------------------------------------------------------------------- 
> --
> >
> > _______________________________________________
> > Community mailing list
> > Community at lists.gispython.org
> > http://lists.gispython.org/mailman/listinfo/community
>
>
> --
> Kai Lautaportti               +358-50-558-7935
> Software engineer             www.hexagonit.fi
> Hexagon IT Oy                 kai.lautaportti at hexagonit.fi
> _______________________________________________
> Community mailing list
> Community at lists.gispython.org
> http://lists.gispython.org/mailman/listinfo/community
>
> _______________________________________________
> Community mailing list
> Community at lists.gispython.org
> http://lists.gispython.org/mailman/listinfo/community