Oxidef

Oxidef is an interface definition language and serialisation scheme intended for the dense, flexible and safe specification of, and encoding of, strongly-typed payloads.

The language is intended to feel familiar to Rust developers. It is strongly-typed and expressive.

Oxidef can be used for Remote Procedure Calls/APIs, as a persistence format, or more generally for describing payloads in custom protocols.

Concepts map closely to Rust and TypeScript, making it suitable as an interoperability format for those.

Links:

• Documentation: available in repository, will be published online, can be built as mdBook

Features

safety features:

• (TODO for verification) Safe against maliciously-crafted payloads
  • All variable-size collections have size limits defined, to ensure that a decoder has a bounded memory footprint.

language features:

• Structures, both final and extensible
  • extensible structures can have new fields added in later versions of the schema
• Unions, both final and extensible
  • extensible unions can have new variants added in later versions of the schema
• Type aliases
• (TODO) Customisable encoding rules
• (EXPERIMENTAL) Service definitions
  • Services are very experimental and a little bit unhygienic, so should be used with care and observing the caveats in the documentation.
  • Potentially-breaking changes are expected in the future.
• Validation rules (for those invariants that you can't describe just with the type system)

encodings:

• Dense binary encoding (compact1)
• Debuggable text encoding (text1)

integration with programming languages:

• Rust
• TypeScript

tooling:

• (TODO) mitmproxy integration
• (TODO) schema 'diff & check' tool
• (TODO) continuous integration rules
• CLI to transcode payloads between different encoding formats

Non-goals

• Absolute maximum speed — alternatives might be faster and that's fine. Oxidef aims for 'fast enough' for real world use cases.
• Stateful validation rules — all validation rules must be freestanding based purely on the payload
• Partial decoding — payloads will be encoded entirely all at once.
• Zero-copy decoding — decoding without copying is a very noble goal, but it's out of scope right now.
• Extra large payloads — lack of partial decoding and zero-copy decoding will make dealing with extra large payloads uncomfortable. Collections are also generally limited to 512M elements (e.g. strings are limited to 512 MB). It might be wise to consider other options if dealing with payloads > 16 MB is realistic. Oxidef might, however, be useful for describing and encoding small individual elements within a larger stream.
• Self-describing binary encodings — Oxidef is aimed at dense binary encoding, where the decoder has a copy of the schema available.

UNDER CONSTRUCTION! What's missing?

Features missing from the language

necessary for me:

• service definitions for RPC-style calls

can survive without, but would be nice to have:

• annotations for codec opinions
• overriding union variant discriminators
• override struct field indices
• newtypes or branded types
• type parameters on user-defined types (somewhat hard)

can wait until quite a bit later:

• cyclic references in types (currently not allowed)
• a formal specification for the language
• a formal specification for the Text1 and Compact1 encodings

Features missing from the TypeScript Backend

necessary for me:

• validation rules
• avoid emitting identifiers that conflict with TypeScript keywords

can survive without, but would be nice to have:

• emit documentation comments on generated items
• code cleanup: ValibotMetadata
• vu29 and vu64 types
• i54 and u53 types
• support for non-scalar map keys. Will need to add a wrapper around Map that serialises and base64-encodes keys. (Note Uint8Array.toBase64 will soon be available in mainstream browsers.)

can wait until quite a bit later:

Features missing from the Rust backend

necessary for me:

can survive without, but would be nice to have:

• emit documentation comments on generated items
• cosmetic: use more compact form for opinionless codecs
• i54 and u53 types
• vu29 and vu64 types

can wait until quite a bit later:

Features missing from the dynamic Rust library

necessary for me:

• support for fields in the extension of a struct
• fixes for some correctness issues when given invalid data to encode

can survive without, but would be nice to have:

can wait until quite a bit later:

Missing developer friendliness features

necessary for me:

can survive without, but would be nice to have:

• mitmproxy/mitmweb integration, or other 'intercept and debug' ability
• Support for custom types with language-native implementation like DateTimes and Ulids
• diff checking for schema evolution (and relevant CI integration)

can wait until quite a bit later:

Documentation

necessary for me:

• a reference manual

can survive without, but would be nice to have:

can wait until quite a bit later:

• a full-on tutorial
• publishing the documentation online

Testing

necessary for me:

• Many areas are untested or undertested
• CI

can survive without, but would be nice to have:

• Fuzzing

can wait until quite a bit later:

What does it look like?

TODO

Inspiration and Similar Projects

• Cap'n Proto
• Protobuf

Licence

TODO