Þ   briarpig  » thorn  » demos  » buf

     When writing content of unknown size to a buffer space of fixed capacity, Wil uses a run subclass of yv named yb that adds a new member specifying max size. So a yb buffer is a yv run, but in addition has a max physical capacity b_x apart from logical length v_n.

     Most initial api below initializes inherited yv state (see run) and b_x from pointers, other yv runs, and iovecs (see iovec). The default copy contructor and assignment operator are fine to use too. With all public members, yb is a pod type (plain old data) of simple nature.

     The main purpose of yb is distinguishing total available space from currently used space in a contiguous memory-only format — quite narrow and straight forward. So a demo of yb alone is short; to make it longer, the right column shows a subclass of yo anticipating the out demo writing to a fixed sized yb buffer with a stream api.

     By it's nature yb aims to support writing to a yv run of contiguous octets with accuracy and safety from buffer overruns. The simplest method of writing a buf writes all the bytes with one value:

     However, before proceeding with more interesting ways to write yb, first let's dispense with slicing yb subsets because a buf slice is only slightly more complex than a run slice. Like other slice types, ybz is a subclass of yz (see slice) beginning with a z_p position and z_n length, where z_p is an offset from the start (negative: from the end).

     All this api except for the final zv() is devoted to debug printing the slice, or capturing initial state from the following yb inlines:

     And zv() shown immediately above is an inline following yb which simply generates the resulting yv corresponding to the slice of all the space in the buffer. (The idea is that most objects overriding a method to take ybz will explicitly call ybz::zv() when they want to resolve a buf slice as a yv run, because they might have something else they want to do instead. Thus ybz purposely does not define an operator yv() because this would fire too easily on methods accepting a yv input.)

     We're done with slices; now back to writing buffers. (The debug print methods of ybz are enough like those of yb that we'll just skip them. Look at yb print methods and use imagination.)

     Printing yb resembles printing yv (cf «) and several examples of usage printing yb appear in the iovec demo (cf «). Here's one more example where initial content is copied from another yv:

     The yv source vlo is written to this 1K buf using bappend() shown in the next section, then the final line generates a temp yb::Bq wrapper from quote() to invoke bdump() shown next. The result on stdout appears below — it's just like dumping yv except for x=1024:

     The yb api for printing resembles boilerplate for printing most þ objects (see the quote demo for quote() and ycite()):

     Code called here appears in demos: crc shows crc32 using yh32, and hex shows hexdump code using yv::vhexmax() (cf «). And of course out shows the yo out stream api.

     Because dumping buffers in hex is so simple and direct, yb makes a convenient demo mechanism to expose what's written by other i/o objects, even though in normal development there's little reason to use yb for that scaffolding purpose.

     You can write formatted content to yb using print() style format markup because under the covers bf() uses vsnprintf() with varargs to do hard parts — yb just provides space and records length.

     Clearly operator()() does the same thing as bf(): they both format printf() style input in available space in the buffer, while reserving one trailing octet to hold a zero byte for C string nul termination. A bit of preliminary error checking occurs so casual usage won't have minor errors punished without explanation. Some versions of vsnprintf() return number of bytes before the end nul, making the call to strlen() here unnecessary. But this way always works without change.

     The bf() method and operator()() aren't necessary, but as shortcuts they're convenient when otherwise one might jump through a few more hoops. And because both return yb ref, which is a subclass of yv, you can write it directly to any yo out stream, like this:

... calling operator()() which writes this line on stdout:

     Both bf() and operator()() write output at offset zero in the buffer. But why not let bf() target any position inside the buffer? That way you could overlay several writes to get what you want. But vsnprintf() make this problematic since it always writes an end nul after formatted output. So to avoid nul clobber something complex must be done; yb simply defers that sort of complexity to ybo — shown column right — allowing you to seek any yb point for formatted writes.

     By using two yb instances, then appending one to another, you can effect gradual accumulation in one buffer from multiple writes, without resorting to ybo in the right column. See bappend() below (cf »).

     As a convenience when using bappend() in the next section, the yb api includes means to seek the position where append next occurs. This is actually the length of content in a buffer because you only append at eof or end-of-buffer, which is the curent length.

     You can either call bseek() to set buffer's end, or assign an integer value to yb using operator=() for the same effect. This use of integer assignment is semantically marginal, but Wil thinks usage is clear, especially when a right hand side of assignment is usually zero (in Wil's code) to make a buffer empty before beginning anew.

     You might dislike assigning integers to yb to set current length, but Wil's going to do it anyway. The bappend() examples shown next include setting yb length just before appending new content.

     Each overloaded bappend() method adds a copy of source content to the buffer starting at current length v_n, continuing until either buf space or the source runs out. After N bytes have been added, the new length becomes v_n+N, never exceeding max capacity b_x.

     Operator << is also overloaded for each version of bappend() taking a single argument, so sources can be appended using buf << src syntax. Declarations and inlines are simple:

     Appending one contiguous yv run of src.v_n bytes is simplest, adding at most the min() of source size and remaining space:

     This next example appends several yv slices generated by yv::vz() (cf «) followed by use of yb::bseek() via operator=() from the last section to shorten length before appending a last yv run.

     Output from that appears on stdout as shown below. The first dump shows "abcde" appended to an empty buf, followed by "klmno" appended to that, before shortening length to seven and appending "vwxyz":

     Appending an iovec vector is only slightly more complex, involving the same calculation as above, but N times instead of once. You can see an example call to bappend() using operator<<() in the iovec demo (cf «) focusing on the api of yvdp, which is the source here.

     The following bappend() variant limits the max number of bytes to be accepted from source ydvp:

     The most complex bappend() (shown below) takes ydvpz as a source — a slice of a ydvp iovec vector. This requires generating a new, temp, stack-based ydvp instance from the slice, using the same sort of approach shown in the iovec demo (cf «), involving several helper methods to choose an iovec vector size (cf «), and involving several helper ydvp subclasses (cf «) with statically defined lengths of iovec vector.

     The top level bappend() for ydvpz checks the starting iovec vector length (because the final subset can't be longer than this) then calls one of the helper methods below to perform the actual append after making a new stack-based ydvp subclass whose constructor converts the input ydvpz slice into a simple ydvp of the desired size.

     Let's pick one of those ydvp subclasses above not shown previously and show it's definition in ydv.h. (This would be part of the iovec demo, but now it clarifies this buf demo.)

     The constructor used by b1K_ydvpz() is shown isolated by white space, calling ydvp::pcut() shown in the iovec demo as pcut().

     Method bdiff() below is used to find the difference between two contiguous yv runs. Wil really only uses bdiff() in unit tests expecting two yv runs to be identical: when this expectation is false Wil wants to find and debug print only what differs so cause can be analyzed. So the definition of bdiff() is driven by this purpose — it's a binary diff finding first and last differing bytes in two yv runs expected equal under yv::veq() (cf «). It also counts total differences.

     As output, yb becomes a description of the subset of arg one that covers the difference of both runs. So if yb started out referring to other space before bdiff(), it doesn't any more afterward. So this is one of those methods which does not operate on space described by current yb state. Instead bdiff() is really a kind of re-initialization of yb to describe an intersection of two partial differences.

     Unequal length also makes yv::veq() fail to return equal; so even if no differing bytes are seen inside, bdiff() counts unequal length as a difference, using b_x to show the position in one where the shorter run was exhausted. (Thus bdiff() is a yb not yv method.) In some unit tests that fail, the only difference might be length.

     For a real use case applying bdiff(), see the compare demo. In practice when any difference is seen, Wil also slices the second run in the same way and prints that too to support a full manual comparison. But that isn't illustrated below. The next short example just forces a difference in the first run, after first generating a zero sized difference (since that's a correct result from unit tests expecting perfect equality).

     The code above writes the following on stdout. As expected, the count of differing bytes is two — since only two were changed.

     Debug printing the result of bdiff() shows the byte range in the first yv run argument that covers the range where differing bytes were found. In this case, since we made changes in the first buffer, we see the span just covers both uppercased bytes and all between.

     "Gee, that's a lot to take in at one time," sighed Stu.

     "It was the give and take methods in the left column, wasn't it?" prompted Wil. "Did I get carried away?"

     "And the stream class too," observed Stu. "Why not just stick to buffers alone here?"

     "Well," Wil scratched his nose, "the out demo would have been too long with ybo there too. It has to go somewhere; at least here it clarifies yb a little."

     "Do you actually like writing this stuff?" asked Stu.

     "Actually, no," admitted Wil. "I'd rather not write any of these demos. But as I explained elsewhere, if I don't, my code won't make sense to anyone. Wouldn't you agree?"

     "Mmmm, yeah I suppose," agreed Stu. "Now you just need to lighten up the license. Why not BSD? Stingy."

     thorn: todo, names, fd, iovec, assert, log, run, hex, crc, buf « Þ, in, out, quote, escape, compare, file, deck, cow, arc, blob, tree, slice, rand, time, stat, hash, heap, node, primes, page, book, pile, stack, atomic, lock, mutex, thread, map, meter, list, iter, ctype

     (mu: toy, peg, imm, tag, box, symbol, token, number, bigint, class, method, reader, writer, eval, env, vm, gc, world, pcode, compiler, asm, lathe, lisp, smalltalk, design, weight, jar, card, harp, debug, profile)

     Some demos are stubs: todo is a demo guide. See toy for mu updates on language pages; names introduces naming schemes.

     This right column is devoted to out stream ybo subclassing yo from the near future out demo, writing into a fixed sized space described by a yb buffer instance. (Actually ybo also writes into yv and/or iovec just as easily by synthesizing an internal yb instance to describe them.)

     You might better read the out demo first, coming after this one. But material below is presented as if the base yo api doesn't matter. So the way in which ybo resembles other yo subclasses won't be emphasized.

     However, parts of yo and yb are semantically very similar because they both focus on managing writes to a single current buffer at a time. So as a result, the ybo class relating the two ought to introduce the idea of yo out streams without much explanation of yo itself. But let's start with yo state.

     Subclasses of yo all manage member vars of yo any way they like; the members shown below are declared in yo only to make them standard. In particular, the first triple o_0, o_p, and o_x are used in yo inlines, so they must be present in all yo subclasses. As shown in a diagram below, this triple corresponds very closely to the triple v_p, v_n, and b_x in yb (except the yo triple is all pointers).

     (Note pure virtual base class yo0 only declares pure virtual methods, so it won't otherwise be noted on this page; ignore it.)

     To understand ybo code below, you only need to know all this state is zeroed by yo::yo(), and yo::~yo() zeroes the main pointer triple. It's up to subclasses to give these member vars other meaningful values. Triple (o_0, o_p, o_x) is safe defaulting to all zero because this means "empty buffer" with nothing inside. (Note o_p is never dereferenced when o_p >= o_x; and when o_x is zero, o_p cannot be less: passive but effective magic.)

     Subclass ybo makes the following relation true:

     This diagram shows a buffer of only eight bytes holding specific values so you can imagine this as concretely as possible in a simple case. So in fact, you can make the diagram above manifest in code by doing the following.

     Though this is easy to do, we'll return to the original simple diagram again later since it summarizes with less detail. (This code sample maximizes detail in a concrete version so you can ground your model, but it's not clear.) The code just above writes this on stdout:

     This output aims to show you the earlier ascii diagram was achieved, so that 1) the entire v9 run still contains all nine bytes of "abcdefghi", but 2) buf b8 currently sees only five as content of eight at most, and 3) out stream o8 sees the same memory, position, and capacity as buf b8.

     As long as we've gone this far, instead of starting right away with constructing ybo to reach this state, let's see what happens when we write a few more bytes to o8 as shown above, since it's purpose is clearly to prevent writes beyond the end of space in b8. This next fragment writes six more bytes (three too many) then flushes to bring length in b8 up-to-date, before debug printing all those objects one more time. Method boplus() returns a count of extra bytes written that couldn't be stored due to limited space.

     Without the flush, length v_n in b8 would not be brought up-to-date with state in the o8 out stream. The code above writes this on stdout:

     Here you see run v9 and buf b8 both see three new bytes "zyx" starting after the first five bytes. And v9 shows the first byte after buf b8 still contains the orginal i put there originally. So o8 successfully stops an overwrite past the end of b8. (This was the only reason v9 has nine bytes.)

     Before code for ybo, let's show the declaration first:

     Additional state members are minimal: bo_b points to the buf updated by ybo, and bo_plus counts written bytes that didn't fit. Buf bo_b0 is only used when a pointer to yb is not passed to the constructor — in this case bo_b0 is used as the yb buffer udpated as writes and flushes occur. This supports writing to yv and iovec state, or a copy of an existing immutable yb.

     Note inlines bov() and bob() return the actual run or buf involved, whether or not ybo refers to or contains the target yb.

     Note the absence of debug printing methods because ybo inherits printing behavior of yo in the out demo (so look there).

     Let's show the diagram one more time:

     Constructor code below prepares state similar to the diagram shown above. But o_p won't be in position until after a seek. So the diagram above only occurs after construction, once the current write pointer o_p has been moved by writing or seeking.

     The starting v_n length of an input buffer has an important effect that's hard to see in the source code because the effect is actually one that's absent: the v_n length is never set to smaller value than it currently has. So ybo will never make a buffer have less content in it than it had at the start — it can only make the content longer. This allows you to use ybo to update existing buffer content without changing current size if you wish.

     This is the only ybo constructor that aliases (refers to) another ybo outside itself instead of having bo_b point at internal bo_b0. By passing a pointer to this constructor, you're saying you want ybo to write to this particular yb buffer, so flushing updates length v_n of that buffer.

     Though never touched again after the constructor, bo_b0 is initialized as a copy of input yb — excepting length v_n — to no purpose except consistency with other constructors. But bo_b doesn't point at it, so bo_b0 doesn't matter.

     Except the empty constructor, all ybo constructors call bo_init(). The main reason to do this — even when unnecessary — is to show similar effect. In every case o_p is set equal to o_0; if a buffer is already partially full, it's ignored by ybo under a theory you might intend starting anew. You can always call oseek() afterward to set a write position different from the buffer's origin.

     Every ybo constructor (given space to write) makes o_0 point at the first byte and o_x point one beyond the last buffer byte, by calling bo_init().

     The empty constructor isn't currently useful. (A later rev might re-add methods to re-init after construction.) The other three constructors take a reference to writable space, making a description in buffer bo_b0 and tracks future writing progress in that. You must use bob() to get the end result. None of these constructors modify immutable input args. Nor does ybo ever change input args later. (Only the constructor taking yb* updates the input.)

     However, note the space pointed at by the input args is not immutable, even though the input objects are immutable. As ybo writes, it modifies any memory given. So never mistake your obligation to pass into ybo constructors actual circumscribed space you intend to be written by ybo.

     Method boclear() reverts to state matching that right after construction, except current buffer length v_n is still not modified if the bo_b address is outside (not the address of bo_b0). You must decide how much previous content to keep yourself if an external yb instance is being written.

     The destructor is a more permanent kind of clear which zeroes most state including the bo_b address, making ybo unusable.

     Flushing ybo updates the v_n length of the buffer being written, but only if pointer o_p now occupies a distance above origin o_0 higher than any previous length known before. Flush only increases a length highwater mark.

     It should be okay to seek a new position lower than the current highwater mark for length without interfering with memory of largest length seen.

     Note getting current length of ybo with olen() also does a flush since the highwater length is only known as a result of flushing. Knowing this, other ybo methods might use olen() as a synonym for oflush().

     The only accurate internal calculation of length is the one performed by olen(), so oseek() below uses olen() to find eof.

     Not all yo out streams can support seeking. (How would you try to seek a socket?) But ybo is an out stream that seeks new write positions:

     Note we permit seeks to any position inside the buffer, up to max capacity at end of buffer, even if it passes a previous high length. Seeking a point above previous length just promotes the new bytes found in the buffer to actual content.

     The main purpose of out streams is writing, so we finally reached interesting methods. But since ybo supports only fixed sized space, there's nothing to do when space is exhausted except count bytes that did not fit inside.

     Protected method ybo::_oc() is only called by public yo::oc() for writing a buffered byte, and only when the buffer is exhausted, which is true when the condition tested here fails: o_p >= o_x. So when _oc() is called it will always increments bo_plus. But Wil always writes _oc() methods to recheck the test condition, since it documents meaning succinctly.

     The owrite() above is a main workhorse write method — almost all yo methods eventually funnel content to this one. All it does is copy bytes that fit to a current write pointer o_p, then advance o_p by bytes written. Any bytes that could not fit due to insufficient space are counted in bo_plus.

     The following write to a given offset is slightly trickier because it doesn't update o_p, which is the only way in other methods we tracked amount of content in a buffer before flushing. So opwrite() itself must perform the highwater content check done by oflush() since no one else can.

     The base version of owritev() defined by yo (in the out demo) calls the first owrite() above, so there's nothing to see here for that.

     This last section shows ybo versions of otake() and ogive() methods used by yo to optimize stream to stream copies. The code here was quickly put together for this demo, and actually hasn't been tested. So sample code really should be shown, demonstrating this version does approximately the right thing.

     Another reason this code should provide samples is to illustrate what these two methods means, since Wil hasn't seen a similar api in other stream classes before. (A similar api might exist but Wil hasn't looked for one since the one he invented works fine for his purposes.) But it's late and Wil's tired.

     This is part of Wil's zero copy support in yo. It aims to avoid an extra copy when reading from an in-stream and then writing to an out-stream. This seldom occurs; but when it does, efficiency becomes a concern.

     A naive approach to the problem would read from a source in-stream into a temp buffer, and then write that to a destination out-stream. This is inefficient when the out-stream itself is buffered, since one can instead write directly into the out-stream's buffer space. This is what otake() and ogive() are for.

     Method otake() describes all remaining space in an out-stream's current output buffer. It saves a copy of this description in member variable o_take so error checking can be performed in ogive(). The caller can then use this buffer space and read directly into it from an in-stream. The actual number of bytes should be put into the v_n of the buffer's description. Then the udpated buffer — with only v_n changed — should be returned in a call to ogive().

     This is a kind of transaction where only one outstanding take/give operation can be outstanding at one time. So it's an error if o_take looks like it already contains a copy of a buffer returned from otake(). Once otake() is called, you can't call it again until ogive() concludes the transation by saying how many bytes were actually used in the last buffer.

     Method ogive() must establish nothing has changed in ybo since the earlier otake() call. (You can't write directly to the buffer and indirectly through other method calls at the same time — writes would clobber each other that way.) Actually, ogive() should assert the current o_p value is exactly the same as it was earlier in otake().

     Besides error checking, ogive() should just advance o_p by the value of v_n in the buffer argument, in order to simply acknowledge space was consumed as the caller says. The simplest version of ogive() would do that after asserting o_p had not changed, and after asserting the buffer arg matched the saved o_take in all but v_n length. This version tries to make sense of poor usage.

     Okay, finally here's a demo of otake() using ydvp in a way resembling an example in the iovec demo (cf «).

     Thus far we prepared a buffer b36 which is half full, an out stream o36 with write position at current buf length, and an iovec iter pmix with at least a dozen more bytes inside than space remaining in o36. So now let's iterate over iovecs and use otake() to write each iovec into the buffer representing space left in o36:

     The code above writes output below on stdout. The extra comment printed each time an iovec is appended shows you how many times it happened, and what granularity of content was involved, which you can verify by checking offsets in the final hex dump.

     Note how o36.boplus() returned zero even though the last iovec written had length five. Why don't we see plus=2 in the output? Because the two bytes were truncated in the append to buf dest, not when written to o36 which knows nothing about it.