An interface for streams of TToken, which can be consumed by Parser<TToken, T>s.

A hint to the parser indicating a default number of tokens to request when calling Read(Span<TToken>).

ITokenStream<TToken> implementations may override this property if there's an optimal amount of data to pull from the stream in a single chunk. For example, if your token stream has an internal buffer, then you might want to override ChunkSizeHint to return buffer.Length.

The default is 1024.

Read up to buffer.Length tokens into buffer. Return the actual number of tokens read, which may be fewer than the size of the buffer if the stream has reached the end.

Push some un-consumed tokens back into the stream. Parser<TToken, T>s call this method when they are finished parsing.

ITokenStream<TToken> implementations may override this method if they want to implement resumable parsing. (See ResumableTokenStream<TToken>.) The default implementation does nothing and discards the leftovers.