[Community] Shapely thread-safety?

[Allen Bierbaum]

Thanks.

I took a look at the fix attached to this ticket, and I think it will
work for my case.

Just a few hours I thought I was dead in the water and now there is a
fix and working code.  Gotta love open source. :)

Thanks,
Allen


On Tue, Apr 1, 2008 at 1:54 PM, Sean Gillies <sgillies at frii.com> wrote:
> Allen,
>
>  I've created a new ticket for this issue:
>  http://trac.gispython.org/projects/PCL/ticket/158#preview
>
>  I recommend that questions specifically about GEOS thread safety go to
>  the geos-devel list.
>
>  Sean
>
>
>
>  Allen Bierbaum wrote:
>  > I just tried the PyDLL option and that works.  I have no more seg
>  > faults but unfortunately this also means that any long running geos
>  > operation block all python threads.  This is probably better then
>  > crashing though. :)
>  >
>  > That said, I would really like to track down the source of this
>  > problem.  In our specific application case there are places in the
>  > code where we want to pass off geos objects to other C/C++ extensions
>  > (developed by us) and have them perform some long running operations
>  > on the geos object in background threads.  If geos is the source of
>  > the issues and is not thread safe then this could be a big problem for
>  > us.
>  >
>  > -Allen
>  >
>  >
>  > On Tue, Apr 1, 2008 at 1:26 PM, Allen Bierbaum <abierbaum at gmail.com> wrote:
>  >> I just tried the idea of using locks, and the outcome was not good.  I
>  >>  have attached the sample program.
>  >>
>  >>  Basically I added another method that simply accesses the x and y
>  >>  attribute of a global point.  When I use locks on the loads calls and
>  >>  don't put a lock around the access to the .x and .y, I get a seg
>  >>  fault.  When I put locks around everywhere I access shapely, then
>  >>  things run correctly.  Unfortunately, as you can guess, it would be
>  >>  very difficult to find everywhere that I may access shapely and put a
>  >>  call to a global lock around it.
>  >>
>  >>  Do you know if it is ctypes or geos that is causing the problem here?
>  >>  In other words, is geos also not thread-safe?
>  >>
>  >>  From looking at the ctypes link, the PyDLL option is interesting but I
>  >>  have not use ctypes before.  Do you have a suggestion for how to
>  >>  change this, or should I just do a search and replace for CLL with
>  >>  PyDLL?    As a side note, if this works it may be possible to add a
>  >>  runtime user flag to shapely that can tell it which of these methods
>  >>  to use.  This way the user could explicitly say "set it up for
>  >>  multi-threading" or "single threaded only please".  Something like:
>  >>
>  >>  shapely.make_thread_safe = True
>  >>
>  >>  or something like that.
>  >>
>  >>  -Allen
>  >>
>  >>
>  >>
>  >>
>  >>  On Tue, Apr 1, 2008 at 1:10 PM, Sean Gillies <sgillies at frii.com> wrote:
>  >>  >
>  >>  > Allen Bierbaum wrote:
>  >>  >  > Is shapely thread-safe?  I ask because my application is seg faulting
>  >>  >  > inside geos when I try to use shapely from multiple threads.  At first
>  >>  >  > it was hard to reproduce, but I was finally able to write a
>  >>  >  > reproducible test case that fails every time I run it (see attached).
>  >>  >  >
>  >>  >  > Can anyone shed some light on this?
>  >>  >  >
>  >>  >  > Thanks,
>  >>  >  > Allen
>  >>  >  >
>  >>  >
>  >>  >  Allen,
>  >>  >
>  >>  >  Shapely's WK* reading and writing uses GEOS C functions which use some
>  >>  >  global objects. I've assumed that ctypes function calls don't release
>  >>  >  the GIL, protecting us from GEOS threading problems, but maybe that has
>  >>  >  been a bad assumption.
>  >>  >
>  >>  >  http://starship.python.net/crew/theller/ctypes/reference.html
>  >>  >
>  >>  >  Grep for "GIL". Yes, it has been a bad assumption. I don't know how I
>  >>  >  overlooked that :( Are you interested in trying PyDLL instead of CDLL?
>  >>  >  Or locks around the WK* reading and writing?
>  >>  >
>  >>  >  Sean
>  >>  >
>  >>  >  _______________________________________________
>  >>  >  Community mailing list
>  >>  >  Community at lists.gispython.org
>  >>  >  http://lists.gispython.org/mailman/listinfo/community
>  >>  >
>  >>
>  > _______________________________________________
>  > Community mailing list
>  > Community at lists.gispython.org
>  > http://lists.gispython.org/mailman/listinfo/community
>  >
>
>  _______________________________________________
>  Community mailing list
>  Community at lists.gispython.org
>  http://lists.gispython.org/mailman/listinfo/community
>