Comments on: On Parallelism

By: Jed Wesley-Smith

Jed Wesley-Smith — Thu, 08 Oct 2009 00:38:51 +0000

I've been playing around with some parallel list comprehension strategies for a bit, and there are a bunch of reasons why it can be a difficult problem. The atomicity chosen for parallel execution is a fairly critical issue – too small and the overhead of the decomposition and scheduling heavily outweighs the benefits, too big and you don't get much benefit from the parallelism. Additionally, choosing data-structures and algorithms that are parallel friendly is important, Guy Steele's excellent talk on this gives some perspective on how far we need to go.

By: Zak

Zak — Wed, 07 Oct 2009 17:43:59 +0000

I don’t think threads are the big source of your overhead here. Using pmap sets up a sequence of futures and deferences them n at a time, where n is the number of processors plus two. I’m pretty sure creating the futures is what slows you down here, not spawning three threads.

According to its docstring, pmap is for a compute-intensive function and a relatively small sequence. I’ve used it on a dual-core machine for running filters on a folder full of images and got about a 90% speedup.

By: Ethan

Ethan — Wed, 07 Oct 2009 16:57:42 +0000

@Everyone,

Thanks for the comments!

@Jeff specifically,

Yes, I am using it “wrongly” — although I would prefer to look at it as inefficiently. I’m fairly certain that with some recoding, a parallel version of the framework could do much better. Naturally, that’s a bit harder than the naive “switch out map with pmap” approach… I simply wanted to see how well it would handle concurrency given minimal changes to the existing code structure.

By: Zach Dwiel

Zach Dwiel — Wed, 07 Oct 2009 15:43:44 +0000

I’d say you might try lowering the number of threads you are starting so that it is more similar to the number of cores you are running them on. Using to few or too many threads can both easily result in worse performance than a single thread. Your experiment shows that you can’t blindly use parallelism for gain, not that you can’t use parallelism for gain.

If you search “clojure pmap” you’ll find people in similar situations as yourself.

By: Tweets that mention On Parallelism | Ethan Fast -- Topsy.com

Tweets that mention On Parallelism | Ethan Fast -- Topsy.com — Wed, 07 Oct 2009 15:02:06 +0000

[...] This post was mentioned on Twitter by :=a name. :=a name said: Clojure, Parallelism, and Genetic Algorithms http://bit.ly/4Gs4RV [...]

By: Jan Rychter

Jan Rychter — Wed, 07 Oct 2009 14:59:44 +0000

I don’t know how much work the pick-one function really does — from my experience, it needs to be significant before pmap makes sense. Even then, your gains on two cores might be small: by default java will run your GC in a separate thread, so your Clojure program will get >100% CPU utilization (I’ve seen numbers like 137%). So there isn’t that much to be gained. That said, I have never seen a *loss* in performance yet.

I’m glad you are measuring wall time. Too many people forget that this is the real metric.

By: jeff

jeff — Wed, 07 Oct 2009 14:37:46 +0000

Isn’t this because you’re using pmap wrongly? From the documentation:

Like map, except f is applied in parallel. Semi-lazy in that the parallel computation stays ahead of the consumption, but doesn’t realize the entire result unless required. Only useful for computationally intensive functions where the time of f dominates the coordination overhead.

I used pmap for raytracing and used partition to group the map into larger work units that are sufficiently computationally intensive to make the effort of using a thread worthwhile. Is is possible to try that for your example?