Pidgin

Interface ITokenStream<TToken>

An interface for streams of TToken, which can be consumed by Parser<TToken, T>s.

Declaration
public interface ITokenStream<TToken>
Type Parameters
Name Description

TToken

The type of tokens the stream produces.

Properties

ChunkSizeHint

A hint to the parser indicating a default number of tokens to request when calling Read(Span<TToken>).

ITokenStream<TToken> implementations may override this property if there's an optimal amount of data to pull from the stream in a single chunk. For example, if your token stream has an internal buffer, then you might want to override ChunkSizeHint to return buffer.Length.

The default is 1024.

Declaration
int ChunkSizeHint { get; }
Property Value
Type Description

int

The default number of tokens to request when calling Read(Span<TToken>).

Methods

Read(Span<TToken>)

Read up to buffer.Length tokens into buffer. Return the actual number of tokens read, which may be fewer than the size of the buffer if the stream has reached the end.

Declaration
int Read(Span<TToken> buffer)
Parameters
Type Name Description

Span<TToken>

buffer

The buffer to read tokens into.

Returns
Type Description

int

The actual number of tokens read.

Return(ReadOnlySpan<TToken>)

Push some un-consumed tokens back into the stream. Parser<TToken, T>s call this method when they are finished parsing.

ITokenStream<TToken> implementations may override this method if they want to implement resumable parsing. (See ResumableTokenStream<TToken>.) The default implementation does nothing and discards the leftovers.

Declaration
void Return(ReadOnlySpan<TToken> leftovers)
Parameters
Type Name Description

ReadOnlySpan<TToken>

leftovers

The leftovers to push back into the stream.