API Documentation

Dataset(path)

Return a Dataset instance targeting path as an Onda dataset, without loading any content from path.

load(path)

Return a Dataset instance that contains all metadata necessary to read and write to the Onda dataset stored at path. Note that this constuctor loads all the Recording objects contained in path/recordings.msgpack.zst.

load(dataset::Dataset, uuid::UUID, signal_name::Symbol[, span::AbstractTimeSpan])

Load, decode, and return the Samples object corresponding to the signal named signal_name in the recording specified by uuid.

If span is provided, this function returns the equivalent of load(dataset, uuid, signal_name)[:, span], but potentially avoids loading the entire signal's worth of sample data if the underlying signal file format and target storage layer both support partial access/random seeks.

load(dataset::Dataset, uuid::UUID[, span::AbstractTimeSpan])

Return load(dataset, uuid, names[, span]) where names is a list of all signal names in the recording specified by uuid.

load(dataset::Dataset, uuid::UUID, signal_names[, span::AbstractTimeSpan])

Return Dict(signal_name => load(dataset, uuid, signal_name[, span]) for signal_name in signal_names).

See also: read_samples, deserialize_lpcm

load_encoded(args...)

Supports exactly the same methods as load, but doesn't automatically call decode on the returned Samples.

save(dataset::Dataset)

Save all metadata content necessary to read/write dataset to dataset.path.

Note that in-memory mutations to dataset will not persist unless followed by a save call. Furthermore, new sample data written to dataset via store! will not be readable from freshly loaded copies of dataset (e.g. load(dataset.path)) until save is called.

create_recording!(dataset::Dataset, uuid::UUID=uuid4())

Create uuid::UUID => recording::Recording, add the pair to dataset.recordings, and return the pair.

store!(dataset::Dataset, uuid::UUID, signal_name::Symbol, samples::Samples;
       overwrite::Bool=true)

Add signal_name => samples.signal to dataset.recordings[uuid].signals and serialize samples.data to the proper file path within dataset.path.

If overwrite is false, an error is thrown if a signal with signal_name already exists in dataset.recordings[uuid]. Otherwise, existing entries matching samples.signal will be deleted and replaced with samples.

delete!(dataset::Dataset, uuid::UUID)

Delete the recording whose UUID matches uuid from dataset. This function removes the matching Recording object from dataset.recordings, as well as deletes the corresponding subdirectory in the dataset's samples directory.

delete!(dataset::Dataset, uuid::UUID, signal_name::Symbol)

Delete the signal whose signalname matches `signalnamefrom the recording whose UUID matchesuuidindataset. This function removes the matchingSignalobject fromdataset.recordings[uuid], as well as deletes the corresponding sample data in thedataset'ssamples` directory.

Onda.validate_on_construction()

If this function returns true, Onda objects will be validated upon construction for compliance with the Onda specification.

If this function returns false, no such validation will be performed upon construction.

Users may interactively redefine this method in order to attempt to read malformed Onda datasets.

Returns true by default.

See also: validate_signal, validate_samples

Signal

A type representing an individual Onda signal object. Instances contain the following fields, following the Onda specification for signal objects:

• channel_names::Vector{Symbol}
• start_nanosecond::Nanosecond
• stop_nanosecond::Nanosecond
• sample_unit::Symbol
• sample_resolution_in_unit::Float64
• sample_offset_in_unit::Float64
• sample_type::DataType
• sample_rate::Float64
• file_extension::Symbol
• file_options::Union{Nothing,Dict{Symbol,Any}}

If validate_on_construction returns true, validate_signal is called on all new Signal instances upon construction.

Similarly to the TimeSpan constructor, this constructor will add a single Nanosecond to stop_nanosecond if start_nanosecond == stop_nanosecond.

validate_signal(signal::Signal)

Returns nothing, checking that the given signal is valid w.r.t. the Onda specification. If a violation is found, an ArgumentError is thrown.

Properties that are validated by this function include:

• sample_type is a valid Onda sample type
• sample_unit name is lowercase, snakecase, and alphanumeric
• start_nanosecond/stop_nanosecond form a valid time span
• channel names are lowercase, snakecase, and alphanumeric

signal_from_template(signal::Signal;
                     channel_names=signal.channel_names,
                     start_nanosecond=signal.start_nanosecond,
                     stop_nanosecond=signal.stop_nanosecond,
                     sample_unit=signal.sample_unit,
                     sample_resolution_in_unit=signal.sample_resolution_in_unit,
                     sample_offset_in_unit=signal.sample_offset_in_unit,
                     sample_type=signal.sample_type,
                     sample_rate=signal.sample_rate,
                     file_extension=signal.file_extension,
                     file_options=signal.file_options,
                     validate=Onda.validate_on_construction())

Return a Signal where each field is mapped to the corresponding keyword argument.

span(signal::Signal)

Return TimeSpan(signal.start_nanosecond, signal.stop_nanosecond).

sizeof_samples(signal::Signal)

Returns the expected size (in bytes) of the encoded Samples object corresponding to the entirety of signal:

sample_count(signal) * channel_count(signal) * sizeof(signal.sample_type)

Annotation <: AbstractTimeSpan

A type representing an individual Onda annotation object. Instances contain the following fields, following the Onda specification for annotation objects:

• value::String
• start_nanosecond::Nanosecond
• stop_nanosecond::Nanosecond

Similarly to the TimeSpan constructor, this constructor will add a single Nanosecond to stop_nanosecond if start_nanosecond == stop_nanosecond.

Recording

A type representing an individual Onda recording object. Instances contain the following fields, following the Onda specification for recording objects:

• signals::Dict{Symbol,Signal}
• annotations::Set{Annotation}

set_span!(recording::Recording, name::Symbol, span::AbstractTimeSpan)

Replace recording.signals[name] with a copy that has the start_nanosecond and start_nanosecond fields set to match the provided span. Returns the newly constructed Signal instance.

set_span!(recording::Recording, span::TimeSpan)

Return Dict(name => set_span!(recording, name, span) for name in keys(recording.signals))

annotate!(recording::Recording, annotation::Annotation)

Returns push!(recording.annotations, annotation).

Samples(signal::Signal, encoded::Bool, data::AbstractMatrix,
        validate::Bool=Onda.validate_on_construction())

Return a Samples instance with the following fields:

• signal::Signal: The Signal object that describes the Samples instance.

• encoded::Bool: If true, the values in data are LPCM-encoded as prescribed by the Samples instance's signal. If false, the values in data have been decoded into the signal's canonical units.

• data::AbstractMatrix: A matrix of sample data. The i th row of the matrix corresponds to the ith channel in signal.channel_names, while the jth column corresponds to the jth multichannel sample.

• validate::Bool: If true, validate_samples is called on the constructed Samples instance before it is returned.

Note that getindex and view are defined on Samples to accept normal integer indices, but also accept channel names for row indices and TimeSpan values for column indices; see Onda/examples/tour.jl for a comprehensive set of indexing examples.

See also: encode, encode!, decode, decode!

==(a::Samples, b::Samples)

Returns a.encoded == b.encoded && a.signal == b.signal && a.data == b.data.

validate_samples(samples::Samples)

Returns nothing, checking that the given samples are valid w.r.t. the underlying samples.signal and the Onda specification's canonical LPCM representation. If a violation is found, an ArgumentError is thrown.

Properties that are validated by this function include:

• encoded element type matches samples.signal.sample_type
• the number of rows of samples.data matches the number of channels in samples.signal

channel(signal::Signal, name::Symbol)

Return i where signal.channel_names[i] == name.

channel(signal::Signal, i::Integer)

Return signal.channel_names[i].

channel(samples::Samples, name::Symbol)

Return channel(samples.signal, name).

This function is useful for indexing rows of samples.data by channel names.

channel(samples::Samples, i::Integer)

Return channel(samples.signal, i).

channel_count(signal::Signal)

Return length(signal.channel_names).

channel_count(samples::Samples)

Return channel_count(samples.signal).

sample_count(signal::Signal)

Return the number of multichannel samples that fit within duration(signal) given signal.sample_rate.

sample_count(samples::Samples)

Return the number of multichannel samples in samples (i.e. size(samples.data, 2))

encode(sample_type::DataType, sample_resolution_in_unit, sample_offset_in_unit,
       samples, dither_storage=nothing)

Return a copy of samples quantized according to sample_type, sample_resolution_in_unit, and sample_offset_in_unit. sample_type must be a concrete subtype of Onda.VALID_SAMPLE_TYPE_UNION. Quantization of an individual sample s is performed via:

round(S, (s - sample_offset_in_unit) / sample_resolution_in_unit)

with additional special casing to clip values exceeding the encoding's dynamic range.

If dither_storage isa Nothing, no dithering is applied before quantization.

If dither_storage isa Missing, dither storage is allocated automatically and triangular dithering is applied to the signal prior to quantization.

Otherwise, dither_storage must be a container of similar shape and type to samples. This container is then used to store the random noise needed for the triangular dithering process, which is applied to the signal prior to quantization.

encode(samples::Samples, dither_storage=nothing)

If samples.encoded is false, return a Samples instance that wraps:

encode(samples.signal.sample_type,
       samples.signal.sample_resolution_in_unit,
       samples.signal.sample_offset_in_unit,
       samples.data, dither_storage)

If samples.encoded is true, this function is the identity.

encode!(result_storage, sample_type::DataType, sample_resolution_in_unit,
        sample_offset_in_unit, samples, dither_storage=nothing)
encode!(result_storage, sample_resolution_in_unit, sample_offset_in_unit,
        samples, dither_storage=nothing)

Similar to encode(sample_type, sample_resolution_in_unit, sample_offset_in_unit, samples, dither_storage), but write encoded values to result_storage rather than allocating new storage.

sample_type defaults to eltype(result_storage) if it is not provided.

encode!(result_storage, samples::Samples, dither_storage=nothing)

If samples.encoded is false, return a Samples instance that wraps:

encode!(result_storage,
        samples.signal.sample_type,
        samples.signal.sample_resolution_in_unit,
        samples.signal.sample_offset_in_unit,
        samples.data, dither_storage)`.

If samples.encoded is true, return a Samples instance that wraps copyto!(result_storage, samples.data).

decode(sample_resolution_in_unit, sample_offset_in_unit, samples)

Return sample_resolution_in_unit .* samples .+ sample_offset_in_unit

decode(samples::Samples)

If samples.encoded is true, return a Samples instance that wraps decode(samples.signal.sample_resolution_in_unit, samples.signal.sample_offset_in_unit, samples.data).

If samples.encoded is false, this function is the identity.

decode!(result_storage, sample_resolution_in_unit, sample_offset_in_unit, samples)

Similar to decode(sample_resolution_in_unit, sample_offset_in_unit, samples), but write decoded values to result_storage rather than allocating new storage.

decode!(result_storage, samples::Samples)

If samples.encoded is true, return a Samples instance that wraps decode!(result_storage, samples.signal.sample_resolution_in_unit, samples.signal.sample_offset_in_unit, samples.data).

If samples.encoded is false, return a Samples instance that wraps copyto!(result_storage, samples.data).

AbstractTimeSpan

A type repesenting a continuous, inclusive span between two points in time.

All subtypes of AbstractTimeSpan must implement:

• first(::AbstractTimeSpan)::Nanosecond: return the first nanosecond contained in span
• last(::AbstractTimeSpan)::Nanosecond: return the last nanosecond contained in span

For convenience, many Onda functions that accept AbstractTimeSpan values also accept Dates.Period values.

See also: TimeSpan

TimeSpan(first, last)

Return TimeSpan(Nanosecond(first), Nanosecond(last))::AbstractTimeSpan.

If first == last, a single Nanosecond is added to last since last is an exclusive upper bound and Onda only supports up to nanosecond precision anyway. This behavior also avoids most practical forms of potential breakage w.r.t to legacy versions of Onda that accidentally allowed the construction of TimeSpans where first == last.

See also: AbstractTimeSpan

contains(a::AbstractTimeSpan, b::AbstractTimeSpan)

Return true if the timespan b lies entirely within the timespan a, return false otherwise.

overlaps(a, b)

Return true if the timespan a and the timespan b overlap, return false otherwise.

shortest_timespan_containing(spans)

Return the shortest possible TimeSpan containing all timespans in spans.

spans is assumed to be an iterable of timespans.

duration(span)

Return the duration of span as a Period.

For span::AbstractTimeSpan, this is equivalent to last(span) - first(span).

For span::Period, this function is the identity.

duration(signal::Signal)

Return duration(span(signal)).

duration(recording::Recording)

Returns maximum(s -> s.stop_nanosecond, values(recording.signals)); throws an ArgumentError if recording.signals is empty.

duration(samples::Samples)

Returns the Nanosecond value for which samples[TimeSpan(0, duration(samples))] == samples.data.

time_from_index(sample_rate, sample_index)

Given sample_rate in Hz and assuming sample_index > 0, return the earliest Nanosecond containing sample_index.

Examples:

julia> time_from_index(1, 1)
0 nanoseconds

julia> time_from_index(1, 2)
1000000000 nanoseconds

julia> time_from_index(100, 100)
990000000 nanoseconds

julia> time_from_index(100, 101)
1000000000 nanoseconds

time_from_index(sample_rate, sample_range::AbstractUnitRange)

Return the TimeSpan corresponding to sample_range given sample_rate in Hz:

julia> time_from_index(100, 1:100)
TimeSpan(0 nanoseconds, 1000000000 nanoseconds)

julia> time_from_index(100, 101:101)
TimeSpan(1000000000 nanoseconds, 1000000001 nanoseconds)

julia> time_from_index(100, 301:600)
TimeSpan(3000000000 nanoseconds, 6000000000 nanoseconds)

index_from_time(sample_rate, sample_time)

Given sample_rate in Hz, return the integer index of the most recent sample taken at sample_time. Note that sample_time must be non-negative and support convert(Nanosecond, sample_time).

Examples:

julia> index_from_time(1, Second(0))
1

julia> index_from_time(1, Second(1))
2

julia> index_from_time(100, Millisecond(999))
100

julia> index_from_time(100, Millisecond(1000))
101

index_from_time(sample_rate, span::AbstractTimeSpan)

Return the UnitRange of indices corresponding to span given sample_rate in Hz:

julia> index_from_time(100, TimeSpan(Second(0), Second(1)))
1:100

julia> index_from_time(100, TimeSpan(Second(1)))
101:101

julia> index_from_time(100, TimeSpan(Second(3), Second(6)))
301:600

read_recordings_file(path)

Return deserialize_recordings_msgpack_zst(read(path)).

write_recordings_file(path, header::Header, recordings::Dict{UUID,Recording})

Write serialize_recordings_msgpack_zst(header, recordings) to path.

samples_path(dataset_path, uuid::UUID)

Return the path to the samples subdirectory within dataset_path corresponding to the recording specified by uuid.

samples_path(dataset_path, uuid::UUID, signal_name, file_extension)

Return the path to the sample data within dataset_path corresponding to the given signal information and the recording specified by uuid.

samples_path(dataset::Dataset, uuid::UUID)

Return samples_path(dataset.path, uuid).

samples_path(dataset::Dataset, uuid::UUID, signal_name::Symbol)

Return samples_path(dataset.path, uuid, signal_name, extension) where extension is defined as dataset.recordings[uuid].signals[signal_name].file_extension.

read_samples(path, signal::Signal)

Return the Samples object described by signal and stored at path.

read_samples(path, signal::Signal, span::AbstractTimeSpan)

Return read_samples(path, signal)[:, span], but attempt to avoid reading unreturned intermediate sample data. Note that the effectiveness of this method depends on the types of both path and format(signal).

write_samples(path, samples::Samples)

Serialize and write encode(samples) to path.

read_byte_range(path, byte_offset, byte_count)

Return the equivalent read(path)[(byte_offset + 1):(byte_offset + byte_count)], but try to avoid reading unreturned intermediate bytes. Note that the effectiveness of this method depends on the type of path.

deserialize_recordings_msgpack_zst(bytes::Vector{UInt8})

Return the (header::Header, recordings::Dict{UUID,Recording}) yielded from deserializing bytes, which is assumed to be in zstd-compressed MsgPack format and comply with the Onda format's specification of the contents of recordings.msgpack.zst.

serialize_recordings_msgpack_zst(header::Header, recordings::Dict{UUID,Recording})

Return the Vector{UInt8} that results from serializing (header::Header, recordings::Dict{UUID,Recording}) to zstd-compressed MsgPack format.

AbstractLPCMFormat

A type whose subtypes represents byte/stream formats that can be (de)serialized to/from Onda's standard interleaved LPCM representation.

All subtypes of the form F<:AbstractLPCMFormat must support a constructor of the form F(::Signal) and overload Onda.format_constructor_for_file_extension with the appropriate file extension.

See also:

• format
• deserialize_lpcm
• deserialize_lpcm_callback
• serialize_lpcm
• LPCM
• LPCMZst
• AbstractLPCMStream

AbstractLPCMStream

A type that represents an LPCM (de)serialization stream.

See also:

• deserializing_lpcm_stream
• serializing_lpcm_stream
• finalize_lpcm_stream

deserializing_lpcm_stream(format::AbstractLPCMFormat, io)

Return a stream::AbstractLPCMStream that wraps io to enable direct LPCM deserialization from io via deserialize_lpcm.

Note that stream must be finalized after usage via finalize_lpcm_stream. Until stream is finalized, io should be considered to be part of the internal state of stream and should not be directly interacted with by other processes.

serializing_lpcm_stream(format::AbstractLPCMFormat, io)

Return a stream::AbstractLPCMStream that wraps io to enable direct LPCM serialization to io via serialize_lpcm.

Note that stream must be finalized after usage via finalize_lpcm_stream. Until stream is finalized, io should be considered to be part of the internal state of stream and should not be directly interacted with by other processes.

finalize_lpcm_stream(stream::AbstractLPCMStream)::Bool

Finalize stream, returning true if the underlying I/O object used to construct stream is still open and usable. Otherwise, return false to indicate that underlying I/O object was closed as result of finalization.

Onda.format_constructor_for_file_extension(::Val{:extension_symbol})

Return a constructor of the form F(::Signal)::AbstractLPCMFormat corresponding to the provided extension.

This function should be overloaded for new AbstractLPCMFormat subtypes.

format(signal::Signal; kwargs...)

Return F(signal; kwargs...) where F is the AbstractLPCMFormat that corresponds to signal.file_extension (as determined by the format author via format_constructor_for_file_extension).

See also: deserialize_lpcm, serialize_lpcm

deserialize_lpcm(format::AbstractLPCMFormat, bytes,
                 samples_offset::Integer=0,
                 samples_count::Integer=typemax(Int))
deserialize_lpcm(stream::AbstractLPCMStream,
                 samples_offset::Integer=0,
                 samples_count::Integer=typemax(Int))

Return a channels-by-timesteps AbstractMatrix of interleaved LPCM-encoded sample data by deserializing the provided bytes in the given format, or from the given stream constructed by deserializing_lpcm_stream.

Note that this operation may be performed in a zero-copy manner such that the returned sample matrix directly aliases bytes.

The returned segment is at most sample_offset samples offset from the start of stream/bytes and contains at most sample_count samples. This ensures that overrun behavior is generally similar to the behavior of Base.skip(io, n) and Base.read(io, n).

This function is the inverse of the corresponding serialize_lpcm method, i.e.:

serialize_lpcm(format, deserialize_lpcm(format, bytes)) == bytes

deserialize_lpcm_callback(format::AbstractLPCMFormat, samples_offset, samples_count)

Return (callback, required_byte_offset, required_byte_count) where callback accepts the byte block specified by required_byte_offset and required_byte_count and returns the samples specified by samples_offset and samples_count.

As a fallback, this function returns (callback, missing, missing), where callback requires all available bytes. AbstractLPCMFormat subtypes that support partial/block-based deserialization (e.g. the basic LPCM format) can overload this function to only request exactly the byte range that is required for the sample range requested by the caller.

This allows callers to handle the byte block retrieval themselves while keeping Onda's LPCM Serialization API agnostic to the caller's storage layer of choice.

serialize_lpcm(format::AbstractLPCMFormat, samples::AbstractMatrix)
serialize_lpcm(stream::AbstractLPCMStream, samples::AbstractMatrix)

Return the AbstractVector{UInt8} of bytes that results from serializing samples to the given format (or serialize those bytes directly to stream) where samples is a channels-by-timesteps matrix of interleaved LPCM-encoded sample data.

Note that this operation may be performed in a zero-copy manner such that the returned AbstractVector{UInt8} directly aliases samples.

This function is the inverse of the corresponding deserialize_lpcm method, i.e.:

deserialize_lpcm(format, serialize_lpcm(format, samples)) == samples

LPCM{S}(channel_count)
LPCM(signal::Signal)

Return a LPCM<:AbstractLPCMFormat instance corresponding to Onda's default interleaved LPCM format assumed for sample data files with the "lpcm" extension.

S corresponds to signal.sample_type, while channel_count corresponds to length(signal.channel_names).

Note that bytes (de)serialized to/from this format are little-endian (per the Onda specification).

LPCMZst(lpcm::LPCM; level=3)
LPCMZst(signal::Signal; level=3)

Return a LPCMZst<:AbstractLPCMFormat instance that corresponds to Onda's default interleaved LPCM format compressed by zstd. This format is assumed for sample data files with the "lpcm.zst" extension.

The level keyword argument sets the same compression level parameter as the corresponding flag documented by the zstd command line utility.

See https://facebook.github.io/zstd/ for details about zstd.

Onda.upgrade_onda_format_from_v0_2_to_v0_3!(path, combine_annotation_key_value)

Upgrade the Onda v0.2 dataset at path to a Onda v0.3 dataset, returning the upgraded Dataset. This upgrade process overwrites path/recordings.msgpack.zst with a v0.3-compliant version of this file; for safety's sake, the old v0.2 file is preserved at path/old.recordings.msgpack.zst.backup.

A couple of the Onda v0.2 -> v0.3 changes require some special handling:

• The custom field was removed from recording objects. This function thus writes out a file at path/recordings_custom.msgpack.zst that contains a map of UUIDs to corresponding recordings' custom values before deleting the custom field. This file can be deserialized via MsgPack.unpack(Onda.zstd_decompress(read("recordings_custom.msgpack.zst"))).

• Annotations no longer have a key field. Thus, each annotation's existing key and value fields are combined into the single new value field via the provided callback combine_annotation_key_value(annotation_key, annotation_value).