Lately I write demos under thorn with a todo list using C++ under a BriarPig mu-babel license. The run and hex demos were done on 13apr2008; crc on 14apr2008; buf on 20apr2008; in on 27apr2008; ctype on 04may2008; out on 18may2008; slice on 25may2008; quote on 31may2008; escape on 31may2008; mutex on 14jun2008; rand on 16jun2008; stat on 17jun2008; primes on 19jun2008; list on 23jun2008; heap on 29jun2008; iter on 02jul2008 and 04jul2008; atomic on 06jul2008; node on 13jul2008; page on 23jul2008; hash on 27jul2008; book on 27jul2008; this menu links all demos:
mu: toy, peg, imm, tag, box, symbol, token, number, bigint, class, method, reader, writer, eval, env, vm, gc, world, pcode, compiler, asm, lathe, lisp, smalltalk, design, weight, jar, card, harp, debug, profile
thorn: todo, names, iovec, assert, log, run, hex, crc, buf, in, out, quote, escape, compare, file, deck, cow, arc, blob, tree, slice, rand, time, stat, heap, node, primes, page, book, pile, stack, atomic, lock, mutex, thread, map, list, iter, ctype
extended book demo ¶
I added sample code to the book demo, and a short dialog veering into comments on a near future pile demo.
book demo ¶
I just put up two demos, not just one. The book demo is very nearly complete; the right column is short — I might add a sample tomorrow showing what print code displays. The most interesting part of the new book demo is the section on garbage collection; I quickly sum up gc designs for languages I did years ago.
hash demo ¶
I spent far longer on the new hash demo, which I wrote in order to see whether FNV was a good hash compared to crc32(). Since there's rarely a conclusive answer to questions like this, I wrote most of the demo as a character dialog with Wil, Dex, and Stu seeking the truth while gently trashing each other.
I hope you find the unit test there useful and interesting, which you can easily adapt to study hash function effectiveness under new algorithms and key populations.
The dialog is slightly too edgy in terms of willingness to be sophomoric. Please forgive any stylistic excess. I put a lot of time in the code and rushed through the connecting narration.
page demo ¶
I finished the page demo, but I don't have time to write much else. Given how long that one took to finish, I guess the next piece will take more than a couple evenings. So most likely the book demo won't be up until this weekend.
cache invalidation ¶
At work I'm doing a little more cache invalidation in my piece, making the detail more precise to reduce false positives that chew up time to realize are not productive.
Sometimes I wonder if folks following my demos realize I only do that a couple hours in the evening each night, and not even every night since I take time off with my kids to do things and see movies. I manage to squeeze in a bit more writing time on weekends.
I seldom write about my day job for many reasons, not the least of which is — it's none of your business. However, sometimes I feel like describing the kind of work I do, since it's been consistent now for fifteen years. I do precise accounting type things in software, and I clean up messes.
There's a number of hard things in computing that include naming and cache invalidation, for example, as well as coming up with the right answers to precise problems. That's what I do: code the parts that scare you to think it might not be correct.
Some is model making, some is clever definition, and some of my work is just being careful with precise things while working out consequences of what will happen, then guiding results. I should talk more about all this later when the mood suits.
demo markup ¶
I underestimated how much more writing was needed in the next demo. My standards for good markup in a demo crept up a little higher in recent demos, making a bit more work, which continues to make sense through all the memory management code. I like links to method implementations.
demos due ¶
The next demo will most likely go up in a day. I've been doing several demos at once and one is almost done. I might as well post it tomorrow — no need to batch it with another.
I spent almost as much time each evening on demos this week as last week, which was mostly spent writing the node demo with suitable sample code. But most of this week was spent restructuring old code instead of writing prose. I put together another project and moved about ten thousand lines of code around in different files, eventually reconstructing tests from last summer. After that I decided which parts would go in which demo, then started re-formating source to fit in less than 64 columns instead of the 80 used before.
Making code source and comments shorter horizontally is a real pain, requiring lots of decisions for comments, all made very rapidly anyway. All slightly numbing, especially if you re-read code you wrote a year ago to decide what it does precisely as you version docs correctly.
Today I went on a long tangent writing new code to add a new format pooled by the page/book heap, adding support for 1K aligned buffers to go with 4K pages and 64K books.
(Some part of one demo or another will explain how these numbers are not carved in stone — they're just constants you change any time you want to rebuild with new constants. A lot of þ aims to be redone on demand as circumstances require, without hoping one size can fit all comers.)
Anyway, since I was going to document page/book heap sources soon, adding a new pooled format was an excuse to study existing code closely again, since part of one demo will aim to explain more formats can be added at need. Well, I thought I should show that. And I started with a new 1K kilo object because its use would actually improve the behavior of code in another demo. I should add another aligned buffer format named chapter with 16K aligned bytes, so sizes increase by a factor of four.
My past experience in software creating database node formats and serialization schemes taught me every new distinct physical layout format turns into a painful amount of detail work to satisfy every need when parts interact. That was why I had put off more formats besides pages and books, since two were enough to write correct buffering algorithms, if not necessarily the best ones in some situations needing better size matches.
However, once I knew I was going to say "you can just add more sizes as needed" in another heap demo, it seemed clear I should actually do one or two more as part of the demo, since the idea is showing beats saying. But then I lost a lot of time in coding today, studying all the code flows again looking for edge conditions in page allocation.
Page allocation has a lot of edge conditions because pages themselves are used to allocate some space needed to manage pages, and this causes a little self dependency in spots requiring care to satisfy preconditions without requiring a postcondition. New formats like kilo and chapter don't have this problem, because they use pages — not themselves — to alloc space used to help manage themselves.
Several of the demos I thought I'd list under the toy language section will appear with the other demos instead, because it seemed better to focus on runtime support apart from language stuff. So some of the demos will include material verging upon language mechanics when it includes garbage collection, stack roots, and gc object box formats. In other words, I felt it's best to keep material on physical memory representations together, even when the objective is language support in some cases.
node demo ¶
I finished the new node demo after working on it every day since the atomic demo. This new demo is the longest one by far — about 40% bigger than the old longest out demo. The new page is about 200K in size, which is eight times as big as the atomic demo posted a week ago. So it was like writing a demo's worth of content every day just to finish this one.
I worked on this demo so hard for one reason only: almost all the toy language memory management will be hard to understand unless this node refcounting api is clear.
Memory used for garbage collection is allocated in pages and books, both refcounted as nodes and collected in lists and maps using handle smart pointers. I started working on the next page and book demos this evening. In a few weeks I should be starting the language related material in earnest. Boxed memory formats used in garbage collected space will appear quite soon.
node dialog excerpt ¶
(Below appears half a dialog appearing late in the node demo. The first half is somewhat funnier since characters abuse each other a little more in the first half. The second half is listed here because it notes a few details on what I think I'm doing, which seems worthwhile to highlight.)
c++ filigree ¶
"I've hated C++ for years," Wil admitted. "But it's always been my most productive tool. It's the worst language, except for all the others. I definitely dislike C more, though."
"Why do you write all this C++ love poetry," Stu asked. "Am I reading you wrong? Is C++ just a tool?"
"Just a tool. I don't really hate it," Wil amended. "It's just painfully slow to get things done. I spoiled myself playing around with Lisp and Smalltalk toys I wrote in the 90's."
"I work by the hour," Dex joked. "Why should I care?"
"I'd like to work on projects of larger scope," Wil continued. "But it's hard to write big things in C++ by yourself."
"Just found startups and get folks to help you," Dex urged.
"Startups focus on income first," Wil countered. "Startups aren't for fun — I want more fun. I wish I liked fishing."
"Use tools provided by others," Dex suggested. "Save your energy for the fun high level parts. Don't re-invent wheels."
"Have you seen existing tools?" Wil scoffed. "Have you seen what passes as software these days? Ever notice bloat? Instability? Giant warrens of rats' nests of complexity?"
"Wil has a point," Stu prodded Dex.
"Pipe dreams," Dex dismissed. "There's no holy grail making software easier to write. Stop pining for mythical lost software edens where living is easy and good. Focus on making money. Enjoy scarcity created by horrors in making software. It's a good thing: keeps out the riff raff. I like being lazy and well paid."
"God, you suck," Stu blurted.
"La, la, la," Dex defended instantly.
"Well you're in luck," Wil granted. "Doesn't look like things will change any time soon."
"If you ever do make good software easy to write with high quality," Dex warned, "I'll have to hunt you down and kill you. Nothing personal, just business."
"I already keep that in mind," Wil nodded. "You'll never catch me trying to make things better, even if I lean that way. Right now I'm just pursuing a personal compromise."
"Practical versus non-practical?" guessed Stu.
"Close," Wil granted. "On the one hand, I want to write more dynamic language toys with some upgraded infrastruture making them viable in performance sensitive applications."
"Totally impractical," sneered Dex.
"On the other hand," Wil continued, "I know I'll keep coding in C++ for years — probably until I retire in twenty years — because folks will keep paying to have it done."
"Woohoo, money rules," Dex pumped a fist in the air.
"Chill out, Dex," Stu advised. "I get it."
"So a compromise is clearly catering to both," Wil noted. "For now I code language toys in C++, and refcounting nodes with handles is a way to manage memory with little chaos."
"I'm not convinced," Stu warned, "but okay."
"At the same time," Wil gestured, "I'd like tools I use in day jobs to improve a little, so tools I make for language toys aim to be agreeable in normal work contexts, too."
"Well they're not agreeable," Dex asserted. "So far."
"So I accept a bit of complex C++ style when it keeps a few more problems in check," Wil concluded. "I suppose I might chuck it later if the toy language stuff takes off, letting me generate a different base for bootstrapping."
"Don't count on it," Stu advised.
"It's just a theory," Wil agreed. "One possible path, but not the only one. My only short term goal is documented node refcounting so facets of toy language memory use are clear."
atomic demo ¶
The atomic demo is done on the topic of threadsafe integer increment and decrement, for use in refcounting and shared memory circular array element allocation.
I was about to start the node demo on refcounting, but it seemed prudent to do atomic integers first. I started re-testing the refcounted node code, though. I also reworked some code that will appear in a future deck demo because it tests node refcounting in an application. But I went on a long tangent updating a std::vector clone named yvt, because the last draft of deck code used std::vector, and that would be a little too different from where I'm going. So yesterday I'm, like, writing test sample code for template vectors — which was sorta fun — but eventually it seemed I should get back on track with writing about the page/book heap, requiring nodes first because pages and books are nodes.
See, holiday weekends are great for wasting your time on ideas that seem interesting, instead of what seems high priority long term. Actually, the idea of drafting another deck version with vectors seemed wrong — if vectors are defined as using contiguous memory (and std::vector is indeed defined this way).
Since memory-based streams in decks are an exercise in granularity using pages and other anti-fragmentation devices, I realized I should write a template based array instead named yat, where an array has an api like a vector but is permitted to be scatter/gather. But parts of yat would use pages and books, which are nodes. Thus I went back to nodes and started with the atomic demo, during which I realized I could not in fact ditch the distinction of shared and non-shared nodes if I wanted to keep the one-owner-only optimization in copy-on-write later.
Are you enjoying this day-in-the-life view of what a writer thinks about that's not reflected in what appears publically? Well I'm not. It's boring. Don't expect this level of tivia often; it's just gratuitous variety. Life has at least fifty channels worth writing about when there's only time for one at a time in writing.
more iter demo ¶
The iter demo is now complete as it will get until map iterators are added later. The non-template iters are done, and I added a mildly interesting dialog.
colors ¶
I tweaked the colors very slightly to improve text contrast. This is slightly hard to do casually by eye without altering general location in the spectrum.
I'm sure you noticed I used the thorny stem image as the key for a limited palette. When I was very young I made a decision not to pursue art as a career, though I occasionally dabbled in drawing as a hobby. Anyway, for a coder I know more than I should about graphic design.
character design ¶
I resist describing the way I designed the characters I use, because I know showing trumps telling, and revealing a formula can steal the amusement factor from anyone who wants to guess the rules. However, I can still mention some generalities without discussing the actual characters themselves. That way you won't have impressions spoiled for later.
You should bear in mind Wil is just a character. Wil is not me, even if he does share my middle name. I simply saddle him with things I'd otherwise have to attribute to myself, spoiling the clarity of second person exposition. But in fiction Wil can easily do things completely unrelated to what I would do. Try not to get cognitive dissonance when Wil crosses boundaries in stories involving other characters. All the characters are foils.
The interesting part of character design is molding organic internal motivation so all earlier behavior is still consistent and believable after you find out more about a character. You want to avoid any later revelation that a persona would never have done something at an earlier point in time, given some worldview you learn later. This is even harder when you design several characters at once, since you don't want inconsistent attitudes in how characters think of each other. And yet, you don't want what they think of each other to be boring.
The five main characters I have now took a lot of thinking to gel the basics of what they think about one another. Finch is the least well known, so she gets the most latitude from others, and she can behave most strangely without causing contradiction. Finch was actually designed first, which is ironic since she hasn't appeared in stories yet. Wil was the last added to my roster; I just needed someone kinda normal, modulo whatever amusing quicks he'd acquire, and I didn't have the slightest idea who he was other than what he looked like until late in the game. So it's ironic he's gotten the lion's share of air time so far.
When I was 17 years old I started studying books about musicals and plays to see how others designed characters that avoided being stiff and uninteresting. I was particularly interested in Pippin by Bob Fosse, which I saw in live production three different places when I was 18 — one each at two different Iowa colleges and one at a dinner theater in Minneapolis. I read a little about Bob Fosse's approach to casting interesting people in roles, and it fit what happened in Pippin — Fosse liked to cast actors against type so they didn't quite fit a role's expected type. For example, he cast a young and sexy actress in the role of Pippin's mother, and he cast black performer Ben Vereen (who you can also see in Fosse's All that Jazz) as the leading player of performers manipulating Pippin's plans and point of view. Both of these choices had the effect of putting audiences slightly on edge.
So part of my formula is making characters deliberately slightly contradictory, as real people often are, since many people espouse philosopical leanings at odds with their actual behavior — which looks hypocritical until you realize everyone tends to think about ideas they are struggling with themselves, resulting in theory that doesn't match all the practice, which varies. As a side effect, characters can exercise enormous latitude in choice along any dimension involving a center of their attentions, because they can advance or regress with respect to impulse vs control.
That's actually more of a clue than you need, since you'll probably read more into this than present in specific story situations. However, as long as I don't tell you on a case by case basis, you can entertain yourself by not knowing for sure.
I've been toying with the idea of having Zé, Wil, Ulf, Eli, and Finch participate in an exercise where each of them draws a caricature of the other in the setting of some story they imagine. This would have several very useful effects if I draw pictures and post them someday. First, none of them will have a canonical appearance, since it will be somewhere near the average of all of them. Second and more interestingly, the way each views the others will tell you something basic about themselves and their attitudes toward one another.
And in capsule form, this is what their project is about: writing stories about each other. Each character will have at least two different major versions: a normal underblown and a heroically overblown variation used in stories more like real life and more like exotic adventures. To be funny, sometimes they'll write stories about everday life when one or another heroic overblown version participates.
Some tension driving plots will arise organically when it turns out some of the bizarre premises they contrive for the sake of stories actually turn out to be true. Fictional problems will bleed over into the real world here and there. Their benefactor, angel investor Koi Flywheel, appears to know something about the larger context. They think he wants them to write stories about saving the world only because it would be amusing.
partial iter demo ¶
The iter demo is posted two thirds done, and I'll finish the list part shortly. It's marked half done now, and it will be three quarters done soon. The last quarter will be done someday when map iters are added to that page.
The template code is all horrifically verbose of course, which is one of the reasons I want to get it squared away on a page of its own, so I don't talk about templates any more.
The material done so far includes the use of the recently written heap demo class showing how much memory the heap allocated. This is useful to reveal the heap used by the template-based queue which makes heap copies of elements added, in a manner much like STL — notably different from the other queues that add elements by pointer instead of by value.
