Þ   briarpig  » thorn  » demos  » ctype

     When categorizing octets using character predicates resembling ctype.h in C's standard library, Wil likes to generalize apis with maps, including maps replacing ctype.h with dynamic usage (but still based on ctype.h) using yutm shown first below.

     Several þ classes use yutm and yum in their apis to include or exclude filtered octets by rule. Both map an octet key to an associated slot in an array of 256 values containing information about the octet. But the yutm map array is a singleton shared by all yutm instances; the yum map array is per instance and 256 bytes in size. A yum map is easily made from a yutm predicate.

     The value associated with an octet key in yum is just another octet. When used as a predicate like yutm, that value in yum is tested for zero or nonzero as a boolean. But yum can also be used to map octets to octets; for example, a map of opening delimitors in a language might contain closing delimitors as values.

     In contrast, the value associated with an octet key in yutm is a set of bits categorizing an octet key with ctype.h predicates. The first section immediately below defines predicate bitflags; later we show how a shared singleton map for yutm is constructed at static init time to map predicates in ctype.h by bitflag.

     Class name yutm means thorn unsigned (octet) type map where the u really means u8 because 8-bit is default when unspecified in þ. This class is just ctype.h repackaged in a new api letting this reified version of ctype.h be used like a value. (For example, you can make a yum map from a yutm instance describing isalpha which can then be edited.) The yutm class api depends on the following enum which names ctype.h predicates with bit values.

     This enum uses 12 of 16 bits in each u16 of the map shown below. When Wil has more predicates to test, he sometimes adds them to this map. But often using yum instead is just as easy, so trying to use all available bits in yutm has low priority.

     "Why did you define a bit for isalpha?" asked Stu. "Isn't that just the union of isupper and islower?"

     "Yes," replied Wil. "And representing isalpha in terms of other bits is traditional in ctype.h implementations."

     "But what?" prompted Stu. "You're not implementing ctype.h yourself? Spell it out for me."

     "Right," confirmed Wil. "Class yutm is just a map of whatever ctype.h says — anything it says — captured once only at static init time. So whatever C library's ctype.h returns for isalpha(), that's what yutm returns for ye_isalpha."

     "A subtle distinction," noted Stu. "Why should I care?"

     "This way yutm is a replacement for ctype.h," Wil explained, "because it actually is ctype.h in a different form factor. If you substitute yutm in your code, nothing should change."

     "Ah," Stu looked satisfied. "A no-change argument."

     "Yes," nodded Wil. "It comes from a refactoring mindset, where the game is avoiding change despite reorganization."

     "So if ctype.h has a bug," Stu considered, "you preserve it perfectly? Is that good?"

     "Yes, because unintentional change is worse," Wil explained. "The problem is doubt. Any time you make a change that might alter behavior, other folks wonder if you interfered with their code and they want you to settle their questions."

     "Time out the window?" Stu asked.

     "Yep," Wil chirped. "Okay, let's look at the class api."

     State members in yutm consist of a singleton map shared by every instance, and one u32 of predicate bitflags:

     Each yutm instance is only sizeof(u32) bytes in size because m_bits is just a logical OR of one or more yE_ctype bitflags (see last section). Private nested class Minit is used one time only to initialize singleton s_init, whose s_map member inside becomes the shared c_map containing yE_ctype bitflags for each possible octet value. The logical AND of per instance m_bits and shared c_map narrows the predicate to include only m_bits when any input byte is used to lookup a map entry:

     To exercise operator[] more easily in sample code below, Wil wrote the following new yv::vpick() resembling yv::vspn() in the run demo (cf «). A variant for yum instead of yutm looks exactly the same except for input param type.

     Then Wil used this new vpick() method to select octets matching sample yutm predicates from a C string converted into a yv run, containing a selection of ascii characters:

     And the output of this appears on stdout as follows:

     Note how ye_ispunct and ye_isdigit together create a predicate true for either one or the other, illustrating how you can efficiently combine multiple predicates at need.

     The next part of the class api shows how to alter a predicate after construction, by adding or subtracting a single bitflag, or all bitflags in another instance of yutm:

     Since those operators should be obvious, no usage sample is given. Instead let's wrap up the rest of the class api, so we can go on to initialization of the shared singleton map.

     This boilerplate for debug printing looks like that for any other þ class. Try the iovec demo's section on debug printing (cf «) or the quote demo for a guide to use of the quote() inline method convention. Let's wrap up this column with initialization.

     At static init time (before main() is called) the following constructor is called to initialize the singleton predicate map used by all yutm instances:

     Essentially all this does is sample the value of each ctype.h predicate with a bitflag assigned, for each of 256 possible octet values, so yutm_map contains all the same information but in a form easier to use efficiently with yutm's bitflags.

     The only thing left is debug print code showing the state of per instance m_bits bitflags and the resulting filtered view of the shared singleton predicate map. After that comes the class api for yum starting top of column right, which explicitly maps octets instead of mapping predicate bitflags.

     Below is actual code to debug print state of yutm, whose class api appears above. However, before the print code let's show an example of output since format of results is the purpose of the code. The code aims to list octet members clearly:

     Executing that prints the following on stdout:

     This illustrates four different aims of the print code:

• attribute bits=up:lo:bl:cn: reveals m_bits bitflags
• every non isgraph() octet is shown in two-glyph hex
• every isgraph() octet is shown one-glyph as itself
• line length is constrained

     If you choose octets to print as themselves with isprint() instead of isgraph(), blank (0x20) only appears as a gap. Here's the source code:

     And the mbits() method abbreviates bitflag names, in a format Wil thinks is "good enough" despite its brevity:

     The rest of this page describes similar class yum, so further yutm material only appears in interaction with yum.

     thorn: todo, names, fd, iovec, assert, log, run, hex, crc, buf, in, out, quote, escape, compare, file, deck, cow, arc, blob, tree, slice, rand, time, stat, hash, heap, node, primes, page, book, pile, stack, atomic, lock, mutex, thread, map, meter, list, iter, ctype « Þ

     (mu: toy, peg, imm, tag, box, symbol, token, number, bigint, class, method, reader, writer, eval, env, vm, gc, world, pcode, compiler, asm, lathe, lisp, smalltalk, design, weight, jar, card, harp, debug, profile)

     Some demos are stubs: todo is a demo guide. See toy for mu updates on language pages; names introduces naming schemes.

     Class name yum means thorn unsigned (octet) map where (just as with yutm) the u really means u8 because 8-bit is default when unspecified in þ. Each yum instance is mainly a 256 byte map of all possible u8 octet values. But yum also counts the nonzero values in the map (to track the number of "yes" members when such a map is used as a predicate resembling yutm).

     Code comments and material below both usually assume the only purpose of yum is to represent a predicate, where only the difference between zero and nonzero in the map is meaningful.

     But another use of yum is representing octet associations where a key octet maps to an associated value octet. (And since all state is public as a struct, you're explicitly free to use yum for any other useful purpose too.) One of the constructors takes two input sequences so a set of keys can map to a set of values.

     This example maps open delimitors to close delimitors:

     This sample code writes the following on stdout:

     Each key and value is separated by equal sign (=) and otherwise each octet prints the same way yutm does: using two-glyph hex for non-isgraph() octets, and as themselves otherwise.

     Note the yum example above can be used as a predicate meaning "is an open delimitor" for the keys involved.

     The following methods and operators edit yum map content by adding or subtracting entries one at a time or en masse. When unspecified, the value used on add is m_yes which defaults to + (plus).

     Obviously operators simply call madd() or msub() as appropriate, so no demo for operators is given below. The constructors — shown next — generally just call madd() after initializing with minit(). Let's add print api boilerplate with constructors to finish the class api:

     If you study all, you'll see nearly everything is done by minit() and madd(), plus debug printing by mdump(). A mixture of source code and examples is shown below.

     Code for minit() is tiny and also calls madd():

     So an empty yv run instance is needed to get a perfectly empty map to start. (But why start with an empty map?)

     Two non-inline constructors basically do an inline minit():

     Of several madd() methods, the interesting one is shown first, taking two arguments to specify explicit values for the keys passed:

     The other two madd() methods merely use m_yes as the value for each source octet. Each time a value changes from zero to nonzero, member m_len increments to show total nonzero values.

     The last madd() variant adds octets true in yutm, which creates a copy of the yutm predicate in yum when the map starts empty — an example below shows yum copying an input yutm:

     This code writes the following on stdout:

     Unlike yutm, this map can be edited octet by octet; for example, subtracting lower and uppercase b's and p's like this:

     Showing three fewer nonzero members afterward:

     Both msub() methods resemble madd() in reverse, subtracting instead of adding map members:

     The yum debug print code below closely resembles the way yutm methods print. Formats are only slightly different, mainly since here values must also be printed, and not just keys:

     Before a final dialog, let's look at another sample which gratuitously changes the value of m_yes between adding one set of members and another set of members:

     Output shows upper and lowercase use different values:

     The dialog below comments lightly on charsets.

     You might have little use for ctype style predicates when using charsets not compatible with Latin1 character predicates. In this case, don't use this api. Use something else instead geared for charsets and streams of codepoints in sizes your app needs.

     "Ahem," Dex cleared his throat self-importantly. "You shouldn't support ascii or Latin1 charsets at all because only Unicode matters, you western chauvinist pig, you."

     Before Wil could respond, Ira tapped on Dex's shoulder, making him flinch. "Got a wild hair problem?" Ira asked Dex.

     "No, no," Dex slowly got the squeak in his voice under control. "I just wondered why Wil didn't focus on Unicode exclusively."

     Wil said pleasantly, "I do still write apps needing no Unicode. For example, sometimes I write command line tools."

     "And that programming language of yours," Dex worried. "Are you going to have it parse Latin1 characters?"

     "Is he bothering you?" Ira asked Wil.

     "It's okay," Wil replied. Then Wil addressed Dex. "Yes, only ascii at first since I've no reason to bother with Unicode."

     "But what if I want to use Unicode?" Dex whined. "How can I use a language unless Unicode support is baked-in from the start? I just won't use your language then."

     "First," Wil ticked off on his fingers, "I couldn't care less if you use any programming language I publish. You keep acting like I give a crap about what you want, despite my insistence I don't."

     "I mean it," Dex warned. "I won't use your language."

     "Second," Wil continued without pause, "If I should happen to process some Unicode with my toy language — even when it has Latin1 based syntax — I'll have no problem since all my strings are in pointer plus length representation. Null bytes: no problem."

     "But," wailed Dex, "You won't display native code strings well in Unicode. Your user interface will suck."

     "Who cares," Wil dismissed. "I should be so lucky that I need a better UI for a language to get adoption. Right now all I care about is scratching one itch at a time."

     "But," puzzled Dex, "can you write servers using Unicode?"

     "You're not that stupid, are you?" wondered Wil. "Unicode content in data and whatever charsets I use in code development have nothing to do with each other. You knew that, right?"

     "Ha, ha," Dex dithered, "of course I knew that. But what if code really needs to grok Unicode. Won't you need library support?"

     "Yes," agreed Wil. "I just don't believe the silly idea everything can be done transparently without needing to deal with details."