SBCL Threading 2011

SBCL improvements and a new portability library called MADEIRA
Thumbnail
Nikodemus Siivola
Email Verified
Technology
Helsinki
Finland
1 Team Member

EDIT:

The campaign is over, and all goals were reached!

This post on outlines the current status, and will be updated as work progresses.

If you'd still like to pitch in, there's a Donate button on my
homepage -- I will honor perks for donations made before Monday 5th of September 2011, should you request them.

Cheers, and Thank You Everyone,

 -- Nikodemus

Goal #7 reached, big compiler lock removal is go!

Goal #6 reached, Madeira is go!

Goal #5 reached, threading API refactoring is go!

Goal #4 reached, semaphore notification objects are go!

Goal #3 reached, read/write locks are go!

Goal #2 reached, timeouts across the threading API are go!

Goal #1 reached: locking improvements are go!

Hi!

I'm Nikodemus Siivola, an SBCL developer self-employed via Steel Bank Studio Ltd. I've been working on SBCL for about 8 years, and have around 1400 commits in the current tree.

My goal is to make SBCL the number one open source Common Lisp implementation.

Maybe 30% of my work on SBCL has been funded more-or-less-directly by my customers, and the rest I've essentially funded out of my own pocket. During the past couple of years -- since the mortgage crises, pretty much -- consulting gigs have been getting thinner on the ground, and I'm looking for a better way to fund future SBCL development.

This is my first experiment in crowdfunding: if it succeeds, there are other SBCL projects that I plan to fund in the same way -- hopefully involving other SBCL developers as well.


Everyone Loves or Hates Threads


SBCL has native threads on several POSIX platforms, but they aren't quite as robust as they should be outside Linux, and even on Linux the threading API is few pieces short of what I'd like to see there.


Individual platforms have issues of their own, but the major problem shared by most non-Linux platforms is our implementation of mutexes, and their misbehaviour when it comes to interrupt safety. 


The API is missing timeout arguments from several functions, read/write locks, and notification objects on semaphores... to start with.


Introducing Fair Spinlocks


I plan to improve things on non-Linux platforms by:

  • Replacing SBCL's mutexes on non-Linux platforms with a ones based on fair spinlocks. My proof of concept implementation shows that this is viable: in my tests fair spinlocks incorporating a backoff consistently outperform POSIX mutexes on Darwin.
  • Implementing a userspace solution for semaphores and condition variables for non-Linux platforms.

Futexes will remain in use on Linux: while it would be nice to use identical code on both Linux and elsewhere, futexes are simply too good for our purposes to not to use them.

The initial funding goal is enough to get this done. The results will be incorporated into SBCL and released into Public Domain as usual. (I have already developed the necessary extensions to ticket spinlock algorithm, allowing threads waiting for a spinlock to be interrupted without holding up the queue. What remains is integrating the work with SBCL, and dealing with semaphores and condition variables.)

The Missing Bits and Pieces

Even if you only use SBCL only on Linux, please consider participating. If I exceed the initial goal, I will also be able to work on:

  • At $4,000: adding timeout arguments to all blocking functions in the threading API.
  • At $5,000: implementing read/write locks.
  • At $6,000: implementing semaphore notification objects.
  • At $8,000: generally refactoring the threading API (while of course retaining backwards compatibility.)
  • At $12,000: new library called MADEIRA will be implemented, which is to be a compatibility library for Common Lisp implementations focused on threading and bits not covered by either CFFI or USOCKET. (Basic things like access to command-line arguments, environment, EXIT, etc. Bordeaux-Threads -like functionality, but better defined, with sharper edges. CAS. Remains to be seen how general I can make the support, though. Running external programs -- probably somewhere between TRIVIAL-SHELL and SBCL's RUN-PROGRAM in functionality.)
  • At $16,000: scope of the "big compiler lock" in SBCL will be curtailed, so that it will be possible to compile things in parallel. (What is the big compiler lock? SBCL grabs a lock when it compiles anything. The same lock is also used for some non-compiler activities like modifying the class graph. This lock prevents parallel compilation. Even if you don't care about parallel compilation as such, there are other unfortunate consequences: CLOS dispatch (before caches settle down) can involve compiling code. Most of the time BCL is a tiny annoyance, but sometimes it can be a real headache. Sometimes it is an insurmountable barrier. )

Spread the Word!


Tell people about this campaign!


Tell people about Steel Bank Studio Ltd.


Tell people about SBCL.


Tell people about Common Lisp.


Let Me Know What You Think!

Even should you choose not to chip in this time, don't hesitate to contact me at nikodemus@random-state.net: I would very much like to hear what you think of this, and what sort of SBCL projects you would be willing to participate in crowdfunding!

Thank you, and Happy Hacking!

 -- Nikodemus

PS. Due to high (and variable!) shipping costs I decided not to offer any material perks of note. If you're after t-shirts or mugs, check out my CafePress shop for some Lisp or SBCL swag -- 50% of the sale-prices there go towards supporting my work on SBCL.

$16,507USD
raised by 244 people in 22 days
550% funded
0 time left
$3,000 USD goal
Flexible Funding This campaign has ended and will receive all funds raised.
Campaign Closed
This campaign ended on August 31, 2011
Select a Perk
  • $5USD
    Warm and Fuzzy Feeling

    It's practically guaranteed, and many people rate it higher than a hot chocolate on a cold winter night.

    32 claimed
  • $20USD
    Acknowledgement

    I will acknowledge your support in a commit message to the SBCL Git repository.

    37 claimed
  • $50USD
    Blog By Demand & Surprise Gift

    Suggest a subject for a blog-post. I will collate all suggestions and you get to vote on them. I will write on at least top 5 subjects before the year is out. I will also send you a small personal surprise gift. Emphasis on the word "small". (Incl. previous perks.)

    90 claimed
  • $150USD
    Skype Call

    In addition to previous perks, I will call you on Skype, and we'll talk about whatever you like for the next 15 minutes at minimum. Consider this a golden opportunity to nag me about your favorite SBCL bug or feature request.

    3 claimed
  • $300USD
    Wish

    In addition to previous perks, you can make a wish, and I will do my best to fulfill it. Obviously, there are constraints. Most importantly, if it is clearly going to take longer than an hour, I reserve the right to veto it and ask for an easier wish. ...but you may be surprised by how much I can get done in an hour, as long as it's not Monday morning.

    9 claimed
Do you think this campaign contains prohibited content? Let us know.