Video

Main ideas

  • clojure has good support for parallelism
  • it helps to know how things work
  • there are tools

Future

(def my-future (future expression))
@my-future                          ; dereference the value
  • uses a Java solo executor under the hood

Limitations

You’ll have problems if

  • you expect exceptions to work normally
  • you want to control the number of concurrent threads
  • your tasks are small

pmap

  • runs roughly 2 + number of cpus items in futures
  • generates threads as needed
    • beware simultaneous pmaps
    • it’s wacky when there’s chunking
  • a slow task can stall it (due to the implementation)

Limitations

  • similar to future
  • you need to force evaluation of a pmap because it’s lazy

core.async

  • uses CSP channels and coroutines (coroutines are lighter weight than threads)
  • reads a lot like Go
  • read like one flow: avoid callback hell
  • uses cooperative multithreading
  • backed by a fixed-size thread pool (48 + number of cpus)
  • best for async programming (not so much parallelism)
    • you shouldn’t block many of its threads
    • easy to wait on other work
    • you might use it to interact with worker threads
  • parallelism using pipeline
  • warning: exceptions can kill coroutines

Claypoole

Disclaimer: Claypoole was written by the author.

  • use thread pools to control parallelism
  • can auto-manage thread pools (the JVM will not clean up your thread pools for you)
  • tries to get things done fast
    • it’s eager by default: not for infinite seqs!
    • output is “eagerly streaming sequence”
  • doesn’t stall on slow tasks in the way that pmap does
  • streaming sequences can be chained
  • cp/future, cp/pmap, cp/pfor, cp/pvalues, …
  • unordered functions available - cp/upmap, cp/upfor
  • lazy versions available - cp-lazy/pmap
    • can be better for chaining tasks (no risk of growing buffers between pmaps)
  • exceptions get rethrown correctly
  • eliminates chunking
  • priorities available - cp/priority-threadpool and :priority value

Reducers

  • r/map, r/fold
  • uses Java’s Fork/Join pool for flexibility
  • useful for cpu-bound operations

Tesser

  • t/map, t/fold
  • avoids Java’s Fork/Join pool for performance
  • useful for cpu-bound operations
  • distributable on Hadoop

Summary

  • Clojure has built-in parallelism
    • future: thread from pool
    • pmap: lazy future sequence
    • there are some drawbacks
  • There are tools
    • core.async: for asynchronous & CPU loading
    • claypoole: for thread pools, controlling degree of parallelism
    • reducers: for CPU loading, parallel reduce
    • tesser: for CPU loading, parellel reduce, and Hadoop