Þ   briarpig(cf  » thorn  » demos  » in

     To unify treatment of input streams parsed for various purposes, Wil wants one api to read buffered i/o streams, and yi is the abstract api he uses for this in-stream purpose. Wil shows a few subclasses and types related to yi.

     Class yi0 shows a basic virtual api with all the virtual methods in yi. Wil's plan for yi0 is vague, subtle, and possibly useless, but can't do any harm. (To wit: objects managing to expose an yi0 api can support a yi subclass — an idea not currently pursued in any way.)

     Most yi0 methods are described below when code is shown. Instead let's focus on a problem caused by making seek a primitive base class operation: not all streams support seek, so what happens when you seek a stream which cannot? It fails. And in that case you are urged not to do that. Streams that cannot seek are unable to perform iseek(), iget(), or ipread(); the api doesn't have a way to ask a stream if seek is supported.

     Why did Wil do this? In earlier framework designs Wil tried to factor separable api features in a sensible way; it always led to a mess and Wil hated the result. It smelled of over-engineering, even when Wil added a virtual method to query capabilities. So the current design aims to avoid over-planning; it works with informal dynamic typing — you can avoid seeking streams with no seek support. If you get stuck, change the code (it's re-editable, not re-usable). Wil finds early fancy design often leads to evil; his new ethic: sketch in broad strokes first, handle fine detail later.

     You needn't override iseek() or ipread() unless used; by default they just complain. You also needn't override ileft() below unless you can do it faster than calling ilen() and ipos() separately to query remaining stream bytes.

     The last i2b() and iget() methods are just buffer friendly (see the buf demo) wrappers around virtual iread() and ipread(); the latter resemble read() and pread() system calls. But with a buffer in hand, i2b() and iget() might look easy to use.

     Note i2b() overwrites existing buffer content, versus iout(yb&) below (cf ») appending at current buffer length.

     Why no ireadv() method simulating a readv() system call? Because Wil hasn't needed it yet in this code rev, and didn't feel like adding it just for this demo. Consider it an obvious extension. If a subclass has an especially interesting ireadv() (say in the deck demo) Wil can show it later anyway.

     The method Wil actually wants to use most often in his work is preadv(), so that's more likely to appear first in the api. However, the current api uses slice subclass yiz (cf ») for some of the behavior of preadv().

     A main purpose of yi is to have the first triple (i_0, i_p, i_x) of pointers shown, describing a current buffer being read by an in-stream, so inlines like ic() can directly read octets without a virtual function call. As long as pointer i_p has not yet reached max i_x (one byte past the last in the buffer) then another byte can be read without need for refill via virtual method. This amortizes cost of virtual methods across many octets; cheap cost to read octets is a main design motivation. As long as an in-stream is buffered, the space buffer can be used in this way.

     An unbuffered stream can simply make the triple of pointers all equal to zero, indicating an empty buffer since i_p is then never less than i_x; a buffer is exhausted when i_p equals i_x.

     When yvi subclasses yi further below (cf ») to read from a yv run of octets (see the run demo) the following occurs:

     Which resembles the same organization of out triples. The base class inits all the triple pointers to zero, and subclasses set them to something else if a buffer can be used.

     Most methods above involve error handling or logging, and aim to minimize code for that in in-stream implementations. Any time igood() is false, it means the main invariant of yi is false: triple pointers are not ordered as required. So ibad() logs this and might assert as well since it's considered corruption equal to using nil as a valid memory address.

     Methods i2o() and i2oc() (c here means complement) resemble yv methods vspn() and vcspn() in the run demo (cf «) using octet predicate maps shown later in the ctype demo to write to an out stream only leading octets selected by a given map. Special case iword() skips leading whitespace then writes a whitespace terminated word on the out stream (more for illustration and unit testing than practical use).

     Method ic() is the reason yi exists: to read octets as cheaply as possible until a buffer is exhausted, after which protected _ic() refills a buffer (if possible) and tries again, returning negative on either eof or error. Negative one (-1) is the conventional value for eof, but another might be used as well, so yi clients must always interpret values less than zero as eof or error.

     Method iunc() is the moral equivalent of ungetc(), the same way ic() is the equivalent of getc(). It unreads an octet previously read, but only one octet capacity of pushback is guaranteed. (A few more bytes of pushback are available in ycg elsewhere on this page: cf ».) Method ipush() writes back an octet which might replace the original — not a good idea if the buffer was intended immutable space; it won't push back eof or an error.

     All these methods make yiz (cf ») in-stream slice instances selecting a byte span in this yi, typically to be read by ipread(), or to create a new yi for a subset. (Hmm, Wil uses yii for an in-stream based on another in-stream, but wasn't planning to show that in this demo; let's see how big this page gets first.)

     The api to debug print yi resembles that of other þ types (see the quote demo) to print, debug, or cite. Variants of iout() handle writing to destination streams. Here only yo and yb are main targets; Wil overloads iout() for other types at need. For an example, Wil added iout(yh32&) for crc support for this demo.

     Below Wil annotates base yi in-stream source code.

     For good form, buffer is empty after destruction.

     Better explanations and examples of debug print methods appear in the run, hex, and out demos. These methods print in-stream fields and hex dump spans already read (before i_p) and to be read later (after i_p). Just use icite() to avoid hex dumps.

     Methods logging messages are for convenience only.

     Method iword() is an informal word tokenizer for space delimited "words", using isspace() as the whitespace predicate. While seldom really useful, Wil likes iword() for a few unit tests, and to illustrate a general idea handled more flexibly by i2o() et al.

     Below is a bit of sample code showing what iword() returns from a static C string converted into a yv run (cf «) once represented as a yvi in-stream (shown below on this page). The destination in this sample is a ybo buf out stream (cf «) writing to the local temp buf in this example. (Here you can see how earlier demos begin to stack in terms of available low-effort unit test experiments.)

     Naturally this sample is a bit contrived, but it gets the job done. The following appears on stdout:

     The following methods filter leading in-stream bytes using octet map predicates explained in ctype demo, writing accepted octets to the yo out-stream destination. The effect resembles std C lib strspn() and strcspn(), also resembling comparable yv methods in the run demo (cf «). As with earlier iword(), these are still meant to illustrate a general idea you ought to tune to fit your app needs more specifically, rather than always using this api.

     Note how equivalent yum and yutm methods are identical but for input arg type; the yum and yutm usage api is the same.

     (See a discussion of the bug caused by not checking the return of owrite() above for error, cf ».)

     Sample code below calls iout() using operator<<() (cf «) but doesn't show an issue about the "best" way to write iout() efficiently. So let's discuss that first before sample code.

     Both iout() methods read all remaining input and append it to the destination — an out-stream or crc checksum. In both cases we use a local 4K stack buffer to hold content read. Why? Isn't 4K arbitrary? Wouldn't some other approach work better?

     Presumably a bigger stack buffer is more efficient for big writes. Maybe 8K or 16K would be better. However, this code officially doesn't care what size would be best. That's something to be tweaked after profiling. Wil tends to choose space with size "about one page" when picking arbitrary preliminary numbers.

     When writing to out-stream yo in particular, this local stack copy should be slower than using yo methods otake() and ogive() for direct writes into out-stream buffers. But this version of iout() has an advantage: simplicity. So it's better to debug in an app's first working version. A faster version passing the same unit tests can come later. (Wil tries several plausible options for i/o when tuning app performance for an empirical measure of "best.")

     The following example calls both versions of iout():

     And that code writes the following on stdout:

     This shows iout(yo&) wrote a perfert copy of original words inside buf, and shows iout(yh32&) generated the correct crc32 since it matches both words and buf.

     Class yvi is a yi subclass reading from yv with implementation as trivial as it gets. (Examples of use: above and below.)

     The final two vipos() and viseek() inlines are just fast, non-virtual versions of ipos() and iseek(), returning the same answers more quickly when you statically know the type is yvi.

     Method iseek() cannot seek after eof.

     The constructor and ilen() are trivial; the destructor is redundant with yi::~yi(). And _ic() next is also trivial in the sense it always returns negative for eof — it can only be called from public yi::ic() on empty buffer, so testing i_p < i_x is just good form.

     So the only methods that do anything of note are iread() and ipread() below. They need merely check for nil pointers and avoid reading more bytes than remain.

     Method ipread() does nearly the same thing as iread() but uses an explicit position in the stream, avoiding current read pointer i_p updates. Here's an example showing the difference:

     The output appears on stdout as below. In summary, the sample code does the following. A C string of 20 bytes s is converted to yv octet run v which is read by yvi in-stream i. After seeking offset 3 (skipping abc) six bytes are read, expecting defghi. Then five bytes (bcdef) are read from offset 1 using ipread() which doesn't move the read position. So the next read of six bytes (jklmno) with iread() continues where the previous ipread() left off.

     The final viseek() is done with a pointer address inside the buffer by adding an offset to the address of the string start. So this is different than passing an integer offset to iseek() in the yi api; the final citation shows the expected offset for i_p.

     thorn: todo, names, fd, iovec, assert, log, run, hex, crc, buf, in « Þ, out, quote, escape, compare, file, deck, cow, arc, blob, tree, slice, rand, time, stat, hash, heap, node, primes, page, book, pile, stack, atomic, lock, mutex, thread, map, meter, list, iter, ctype

     (mu: toy, peg, imm, tag, box, symbol, token, number, bigint, class, method, reader, writer, eval, env, vm, gc, world, pcode, compiler, asm, lathe, lisp, smalltalk, design, weight, jar, card, harp, debug, profile)

     Some demos are stubs: todo is a demo guide. See toy for mu updates on language pages; names introduces naming schemes.

     Because this page shows many classes, here's an index:

• yi0 (cf «) abstract base class with virtual methods
• yi (cf «) base class for all in-streams
• yvi (cf «) in-stream subclass reading from yv
• yiz (cf ») in-stream slice: subset of an in-stream
• yugi (cf ») in-stream reading pseudo-random generator
• ycg (cf ») char generator: position and line meter

     Obviously files have a yi subclass for reading, but yfi appears in the file demo because this page is filled with other types.

     A subset of a yi in-stream is slice yiz subclassing the yz slice base class. (See the slice demo for an intro to slices and the yz api.) Base class yz is just an offset and length inside a sequence, to name a subset.

     Subclass yiz adds a reference to yi, so you can think of yiz as triple (pos, len, ref) whose fields are named (z_p, z_n, z_i).

     A most important caveat when using yiz is that it requires a yi in-stream suporting random access with yi::ipread() implemented. A yiz slice reads content from yi without changing the stream's read position, so every read is positioned explicitly.

     For a change, let's start with the most interesting method, zread(), since it needs little explanation and because it's used inside other methods like zout() to actually get things done.

     Let's try an example to help show what this does: zread() does streamed reads over a described yi subset, using yz::zskip() to self-adjust after a read, making the slice itself smaller each time. When zread() must be used on const yiz instances (see zout() below) we need only make temp mutable copies that avoid altering an original slice.

     This next example creates a slice of a yvi in-stream shown column left (cf «). Successive reads consume more of the slice until eof is indicated when the slice is exhausted.

     This code names a ten byte slice of yvi in-stream i starting five bytes inside. Then it reads: six bytes once and fifteen bytes twice. The first two reads consume the slice in two parts; the third returns eof because the slice is empty. The output on stdout shows:

     The debug printing output show above was written like this:

     A normal approach to coding zdump() would also debug print the z_i in-stream inside as well, but making output more complex. So zdump() is just zcite() here for sake of brevity.

     The following two zout() methods look nearly identical to code in yi::iout() (cf «) and the same comments about iout() efficiency apply here (ie using give/take methods or size of reads).

     The version of yiz shown here is somewhat experimental. It's intended to rough out basics to be later adjusted for specific circumstances. Wil has written libraries in servers needing to read streams from subsets of a larger stream, and yiz is a gesture in this direction. However, a more useful representation of a yiz is a new yi subclass reading only from a subset slice. This demo doesn't show that, so you're being deprived of the most fun part.

     Wil finds it harder to explain compromises in such things; use your imagination, and note them as obvious when you see them.

     This section describes a new yi in-stream subclass reading from a pseudo random number generator. The name yugi means thorn unspecified bytes generator in-stream, which uses u in a recklessly casual way; but Wil tends to use u in class names with a lot of hand waving. The 32-bit pseudo random generator h32rand() used by yugi will appear later in the rand demo (cf »).

     Wil uses yugi in unit tests needing pseudo random content, when the normal way to acquire content is from streams.

     Note yugi does not override iseek() or ipread() so random access i/o is not supported. Wil sees no need for it.

     The g_body array of eight u32's is 32 bytes in size: the value of e_body_size. This is the buffer that triple (i_0, i_p, i_x) describes. Each time the buffer is filled, eight pseudo random 32-bit values are generated and buffered in g_body for subsequent demand. Perhaps a larger buffer would be faster, but Wil doesn't expect to use yugi in performance critical situations, and "about one cache line" is fine as buf size in this case. Filling the buffer looks like this:

     The constructor takes a pointer to an external random number seed, so the same seed can be shared by many unit tests all sharing the same stream of "random" values so behavior is deterministic (as long as threads are not involved with non-deterministic scheduling).

     The first call to _g_fill_rand() occurs in the constructor. Note how a nil pointer for seed address causes a global seed to be used.

     Neither length nor read position are very meaningful in ygui, so Wil defines both as the number of bytes returned so far from ygui, so ygui always appears positioned at eof. This works fine. Wil uses member variable g_pos to record the position of the first byte in the buffer, which is consistent with further g_pos bumps below when _g_fill_rand() fills the buffer each time.

     As always, protected _ic() is called from public yi::ic() only when the buffer is exhausted, so the test condition here is always true, refilling the buffer and resetting i_p to the origin.

     Method iread() is the source of pseudo random bytes read in bulk, when ic() is not used to read single bytes. Basically iread() counts down the sz number of requested bytes until this many have been written to destination dest. As the buffer becomes exhausted, new content is generated on demand using _g_fill_rand().

     As Wil thinks about writing a demo for yugi, it becomes apparent the absence of max stream size makes it possible to write an infinite loop by calling iout() (cf «) with yugi as a source. But this is realistic in situations where an in-stream might be unbounded as long as a system remains operational. So the problem might be in the original version of iout() — yes, that's it. The iout() api ought to allow a max transfer size. And with or without a max transfer size, the first draft of iout() has a bug when it doesn't check to see if writing makes progress. (The first time iout() hits a zero return from writing, it ought to stop transfer, instead of waiting for eof on read.) «

     Okay, here's a short demo of yugi emphasizing use of both ic() and iread() with debug printing to reveal change in buffer content when the buffer is exhausted.

     This code sample writes the following on stdout:

     On a final note, clearly yugi is badly named if you think the word "random" really ought to appear in the class name somewhere. However, Wil knows the content is pseudo random — and there's only one in-stream like this, so he doesn't get confused. So the class is only badly named if Wil prioritizes your need to be informed.

     But Wil doesn't; your needs don't rank. If Wil forgets the class name, he opens yi.cpp, searches for "rand", and the problem is solved. This point would be a good place for a dialog in which Dex gives Wil grief for not naming the class PseudoRandomInStream. But you could write that one easily. So the following short circuit exchange is a joke.

     "You know," Dex patronized, "normal people give meaningful names to classes. Would that trouble you too much?"

     "I don't mind if you burn," Wil replied.

     "That's not the question I asked," objected Dex.

     "Bite me," snapped Wil.

     "I'm lodging a complaint with the management," huffed Dex.

     "Good luck with that," wished Wil.

     The next class ycg uses an instance of yi and repackages its output, but is not itself a yi in-stream. You could make a yi based on ycg, but Wil doesn't need this (and Wil isn't driven to normative consistency without a reason). Class ycg is almost — but not quite — as badly named as yugi in the last section. It means thorn char generator where an eight bit codepoint is understood since unspecified.

     In the 90's Wil often used the name spigot for this class, and that's not any better, is it? It was where Wil metered out his input when parsing content that had to be annotated with line numbers.

     The code below mainly pursues a couple requirements Wil needs in language parsers using only a few bytes of token lookahead:

• several bytes of pushback are needed, and
• offset and line stay correct despite pushback.

     Before we start api and code for ycg, this is a good spot to mention the main problem Wil has with þ's runtime as it pertains to dynamic language implementations: i/o is sync with no continuations, so reading and parsing input is awkwardly monolithic and batch-oriented in a way that's strictly retarded in languages with continuations.

     In other words, you won't see much of Wil's approach to async i/o in this rev of þ, and that's why you might develop an itchy feeling when perusing code that reads and parses in-streams. (And the toy language's first version will be the same — one of the reasons it will just be a toy.) The first version of i/o for a toy language for processing source code must be used to bootstrap another async i/o system — so parsing and compiling must be written twice. The C++ version is a throw-away bootstrap layer. Cool, huh? As a side effect, a later more elaborate parsing engine built in a dynamic runtime can choose to support more upscale charsets and encodings. A first version can be dumber to bootstrap without limiting a later version.

     Note Wil's interesting decision to return a slice of the source yi in-stream when slicing ycg — this works because content appears at the same position in ycg that it does in the underlying yi.

     Almost all the api is inline, so all that remains is gc() to read a byte, gpush() to push back a byte, and debug printing. Let's start with printing boilerplate and conclude with good stuff.

     The gdump() output here only cites the cg_i in-stream, occupying only one line — which is nice for brevity, but lacks detail as you'll see in sample code below.

     Code for gpush() next reveals it only considers 0xA when counting line-endings, so pushing back 0xA has the effect of reducing the line count. Note you can push back a different byte than one that was originally read. This is considered a feature when useful for content rewriting — but dangerous when it might alter token positions so they no longer match source.

     Reading the next octet with gc() increments position when eof is not encountered, and increments line count when 0xA is returned. (Wil's portable version of this in the 90's was complex when working with Mac, Linux, and Windows line endings. This one is trivial.)

     The following code sample shows inner state of ycg before reading the first newline, after reading it, and after pushing it back again. The start and end positions of a token parsed before first newline is captured in a buffer, and then read again directly from ycg using slice notation. The slice is printed for comparison with what was parsed, invoking code in an earlier demo on this page for yiz::zout() (cf «) which is actually an expected practical application of this api: fetching the original source from offsets when presenting code commentary.

     The result of code above appears on stdout below. Since the in-stream inside is not dumped each time, you need to find integer values in more spots to verify expected behavior.

     Instead of a funny dialog here at the end, let's just wrap things up. Way past midnight on Sunday night is no time to exercise one's sense of humor — not with a full day ahead.

     All this code is available only under the BriarPig mu-babel license described fully on the rights page. You do not have permission to reprint this page in any way. No feeds or repackaging is allowed. You can link this page if you want folks to read it.