Skip to contents

Changelog

Source: NEWS.md

typedjson 0.1.0

First release, carrying the format and the two round-trip contracts described in vignette("design").

  • Writing and reading through json_write(), json_read(), json_write_str() and json_read_str().

  • Ordinary JSON for attribute-free vectors and lists, with the integer-versus-double distinction carried by the number lexeme.

  • Prefix-tagged strings for typed NA, Inf, -Inf and NaN, with any ordinary string starting with the prefix escaped by doubling it.

  • Text carried as UTF-8 whatever the locale, so a value writes the same document on every machine. A string R has declared as UTF-8 or latin1 is converted from what it declares, and one it has not declared is taken as the bytes it holds rather than translated through the locale, which used to turn the two bytes of é into the eight characters <c3><a9> under a C locale and read them back as that text. Undeclared bytes that are not valid UTF-8 have no reading to fall back on, so they are refused where they sit and name the path, as is a string declared with the "bytes" encoding.

  • A tagged ~a / ~v object for anything carrying attributes, which covers Date, POSIXct, factors, matrices, data frames and classed lists through one rule, with a ~t naming the type only where the payload cannot. Attributes come back in the order the document records them rather than in a fixed rank with class last, which used to make a value carrying class beside a second attribute — a data frame among them — write a document that read and wrote back as a different one.

  • Objects from all four systems: S3 and S4 through the attribute rule, S7 through a recorded class, and R6 through its class generator. A generator is recorded by the class it names rather than by its contents, in both systems, which closes the gap where an S7 generator round-tripped and an R6 one was refused as an environment. On the R6 side the lookup that finds it again runs on the way out as well, so a class no reader could find again is refused where it is written, naming the path it stopped at, rather than producing a document that fails on the read; the S7 side answers the same question by carrying the definition, which the bullet below sets out. An S4 or S7 object is rebuilt from the document rather than constructed, which reaches past the point where the class decides whether to produce one at all, so the check the class does supply is run where it is rebuilt: methods::validObject() on the S4 side and S7::validate() on the S7 side. That is what stands between a hand-edited document and a value the class itself rejects, and the S7 call checks property types as well, so a property a document omits or spells as something else is caught with it. Neither disturbs a good round trip, and a class this session does not hold leaves nothing to check against, the lookup being allowed to find a class rather than to fetch one. What neither call replaces is the construction it reached past: an initialize method for S4 and a custom constructor for S7 do not run, so a slot or property one of them would have derived comes back as the document spells it.

  • An S7 class no name finds again carries its definition rather than a reference. S7 sets package for a class defined in a package and leaves it NULL for every class defined outside one, which is exactly the question of whether a name finds the class again in another session, so that attribute decides between the two forms: a package-scoped class is recorded as the class and package it names, and a class with no package carries its parent, properties, constructor and validator into the document. That retires the reading of NULL as the global environment, which used to make a class defined at top level look referenceable, write cleanly, and then fail on the read in every session but the one that wrote it. Carrying the definition is what base::serialize() already does with the same object, the class being an attribute and attribute values traveling by value. Each class the definition names is recorded by what identifies it rather than walked into — one S7 itself binds by the name it holds in that namespace, an S3 class by its class vector, a union by its members, and a class of your own by the two forms above — since every refusal a walk into one hits is inside S7’s own machinery rather than in the class an author wrote. An embedded definition comes back equivalent rather than identical wherever a part of it closes over an environment recorded by its contents, which S7 builds for the constructor of every class that has a parent, and that is a third entry beside environments and undeclared strings in the exceptions vignette("design") states. The class vector an object records is checked against the class it resolves to on the way back, so a document where the two disagree is refused rather than dispatching on the one and taking its properties from the other.

  • An R6 instance has no method of its own and is refused, naming the class and the method that would settle it. Every other type in scope has a value its own type system supplies — an S4 object is its slots as an S7 object is its properties — where an R6 object is defined by what its methods guarantee, and its private fields are private precisely because they are not part of that. Recording one as the bindings it happens to hold is an assertion about the class that only the class can make. See vignette("r6") for deciding what a class’s state is, and for writing the pair that records it.

  • A reference class instance is refused on the same grounds, naming the class and the method that would settle it, where it used to fail on a promise several environments inside the methods package’s own class definitions. Only half the R6 argument applies, since a reference class declares its fields and so already answers what the representation is; the other half applies unchanged, the object being a reference whose initialize may establish an invariant and whose fields hold whatever a private binding would. A method registered on the concrete class settles it, as does one on any class between that and envRefClass, which is the rung the refusal itself takes. The generator that makes one is refused as well rather than recorded by the class it names, the way an R6 or S7 generator is, since a walk into one reaches those same internals. What is not refused is an environment you have classed yourself: it claims nothing about its instances, so the environment rule applies and its bindings reach the document, which is that rule holding rather than a hole in it.

  • The r6_state() and r6_restore() pair, for a class whose bindings genuinely are its state, which is the mechanism the default used to apply unasked. It records the class’s package alongside the public and private bindings, skipping methods and active bindings, and rebuilds by finding the generator again, allocating without running initialize and writing the bindings back in. The generator settles the instance’s shape as well as its lock, so recorded state a class no longer declares is dropped with a warning naming it rather than bound into an object no constructor could produce. What that reaches past is initialize, so the pair is documented with what it does not guarantee and left to the author to opt into.

  • Language objects, which are values rather than handles and so round-trip exactly. A call, an expression and a pairlist are written as their elements under a ~t naming the type, keeping the argument names R stores as tags, and a symbol is written as the string ~:name. Nothing is deparsed, because a call may carry an arbitrary R object as a constant in its tree and str2lang(deparse(x)) turns a double vector there into a call to c(). This is what makes an object carrying a recorded call writable, a caught condition among them.

  • Closures, which compare by their parts rather than by reference — identical(function(x) x + 1, function(x) x + 1) is TRUE — and so round-trip exactly wherever the environment they close over is recorded by name, and up to the same equivalence an environment does wherever it is recorded by contents. Formals, body and environment go under a ~t of closure, all three through machinery the two entries above already carry, so nothing here is new beyond deciding that a closure is a value. A primitive closes over nothing and is recorded by the name that finds it again, which is what base R does with one; note what base R does not do, which is record a package export as stats::median, since the environment rung already does the work a name lookup would. No source reference is recorded: the srcref a parser attaches to a function definition, to a { block and to what parse() returns is dropped, which costs nothing against the first contract because identical() ignores one by default, and saves dragging the source text of a whole file into the document through the srcfile environment a srcref carries. A byte-compiled closure is written from body(), which is the source tree it was compiled from, and left to the compiler. Two shapes are still refused, and both are the environment rules holding rather than a limit on closures: an argument the function was called with stays a promise in its frame whether or not it has been forced, and a callback a method stored back into its own object reaches that object again through the frame it closes over, which is a cycle.

  • Environments round-trip up to equivalence, the way base R’s serialize() does, since a rebuilt environment cannot be the one that went in. The global, base and empty environments, a namespace, a package environment and the imports environment of a namespace are recorded as the name that finds them again — in the spelling R itself prints, so namespace:stats sits beside package:stats and imports:stats — and come back as the object they were written from. No version is recorded with a namespace: base R records one and never reads it, and a value the reader ignores would break the second contract the moment a document is reopened under a different R. Anything else is recorded by what it binds, with the parent written by the same rule and the locked bit and locked bindings alongside, so what comes back binds the same names to the same values under an equivalent parent. A recorded name that is not available on the read is replaced by the global environment with a warning, which is what base R already does. The name is the one environmentName() gives, so the rung set is base R’s own; what is added is that a rung counts as one only where the name resolves back to the same object, which keeps an environment of your own carrying a package: name attribute from being mistaken for the package it claims to be. Bindings are ordered by their bytes rather than through ls(), whose collation is a locale setting. Two bindings are refused rather than recorded, because reading either one runs code: a promise, which the writer will not force on its own initiative, and an active binding, which it will not call. Together with the language entry above this is what closes the formula gap: a formula carries its .Environment, so y ~ x, the terms built from one and an lm() fit built on that all round-trip exactly.

  • An extension protocol, json_state() and json_revive(), for classes the default rule does not fit. A method has no privileges: whatever it returns is written under the same rules as any other value, so it cannot smuggle out a handle by wrapping it in a list. A method is looked for on every class the value dispatches on, so one registered on the superclass of an S4 class, or on a reference class, is reached where the concrete class has no method of its own, and the chain it was found through is the class vector the document records for json_revive() to dispatch back through. A hook is reached through a call the writer or reader builds, and R evaluates a call or a symbol it finds at argument position, so a document that recorded one there used to run it on the way in, before anything had checked the payload. What gets spliced is now the value inside quote(), which is identity for everything else, so a recorded call reaches the hook intact and draws the same refusal any other malformed state does. The quote object itself is spliced rather than its name, since the call is evaluated in the global environment and a binding of that name there would otherwise be what decides whether this holds.

  • Reference identity across a document, which covers bare environments and R6 instances alike. A reference the walk reaches more than once is numbered with a ~id where it is first written, and each later position carries {"~ref": n} rather than a second copy, so two positions holding one environment on the way in hold one on the way back and a stateful pair of closures over one frame still moves together. The number is minted where the repeat happens, so a document carrying no sharing carries no marker either. A cycle rides the same numbering, since the reader creates an environment and numbers it before reading what it binds: an environment whose parent frame binds it back, which is what function() {e <- new.env(); e} returns, writes and comes back bound the same way. What stays refused, as an error naming both ends of it, is a cycle closing through an object the extension protocol builds in one call — an opted-in R6 instance among them — since a constructor cannot be handed an object that already exists.

  • Plain JSON for a consumer that already defines the shape it expects, through a typed flag on both writers. There R’s types are noise, since the document has to satisfy an external schema rather than describe the value it came from, and the output was already nearly right: taking a JSON Schema fragment as the example, the two forms agree everywhere except at length-one vectors, where {"required":"x"} is not the {"required":["x"]} the format asks for. Unboxing cannot be a global policy, because shape requirements run both ways inside one document — that same schema wants a scalar at additionalProperties — and no encoder can choose between them by inspecting a value, since at length one a scalar and a one-element array are the same R object. The distinction already lives in the R value, where I("x") differs from "x", so plain mode renders it rather than importing a policy for it: a length-one vector is a scalar unless it is AsIs, in which case it keeps its brackets, and that is the whole of the configuration. Everything else is this format with the annotations left out. The S4 bit is dropped and nearly every attribute with it, so a Date writes its number and a factor its codes; what survives is the two attributes JSON cannot be written without an answer to, the names of a list deciding object against array and the AsIs marker deciding scalar against array, so a name is a key only where it already was one, which is a list rather than a named vector, and a matrix flattens because dim decides neither. Two consequences of that go the other way from the rest and are documented rather than changed: a handle reachable only through an attribute is dropped with the attribute, so a connection writes its slot integer where the default refuses it, and a json_state() method is not consulted, plain mode being a shape for a schema rather than persistence. A missing value becomes null, which is what JSON spells absence with, and everything the annotations were the only way to write — complex and raw values, the non-finite doubles, symbols, calls, closures and environments — is refused where it sits, naming the path, rather than written as null, which is the answer TypedJSON.jl gives a function and the silent corruption this package exists to prevent. Nothing is escaped either, since the consumer asked for the name it asked for, so a document carrying a leading ~ of its own is the one this reader will not take back — the same answer it already gives any foreign document spelling a tag it does not know. The reader is otherwise untouched: a plain document is a foreign document and reads under the grammar already there.

  • Files written straight from the document, which keeps it out of R altogether. The json_write() function hands the document to a FILE* through yyjson’s own writer, where it used to render into a std::string, hand that to R as a character vector and write it through a connection. Both of those copies are proportional to the document and a file is the case documents are large for, so peak memory is now the DOM and the single buffer yyjson renders into: writing list(xs = strrep("a", 92e6)) peaks 88 MB above the value it came from where it used to peak 176 MB above it. The bytes are unchanged, trailing newline included, since the yyjson writers end on the last token and the byte writeLines() used to add is added explicitly. A refusal still leaves the file alone, by construction rather than by accident: the walk is where one fires and it runs to completion before the path is opened, so a document already sitting at that path outlives a write that is refused. What this is not is streaming, which vignette("design") records under Deferred.

  • Building on R 4.3 and R 4.4, where the ANY_ATTRIB() accessor the writer reaches for to skip an attribute-free value does not exist yet, having arrived in R 4.5.0, and the head of the attribute pairlist answers the same question. The floor is stated as Depends: R (>= 4.3) rather than left to the compiler, so an older R is refused while the install is being resolved rather than part way through compiling read.cpp, and the check matrix reads that floor back out of DESCRIPTION to pick the R it tests against, so the job follows the declared minimum rather than sliding off it the next time R releases.

Format

Breaking changes to the document shape, settled before any documents exist in the wild.

  • A JSON object is now always a named list, and its values are written bare rather than wrapped in a one-element array. A record therefore reads the way any other tool would write it: {"name":"config","retries":3} rather than {"name":["config"],"retries":[3]}.

  • A named atomic vector no longer takes the object form. It escalates through the ordinary attribute rule instead, since names are an attribute like any other, which removes a special case rather than adding one.

  • A complex vector records its parts as named re and im fields rather than one interleaved array of doubles, so 1 + 2i writes as {"~t":"complex","~v":{"re":1.0,"im":2.0}} — the spelling a Julia or Python consumer would reach for. Each part is an ordinary double payload, so the unboxing rule and the ~z tags apply to it independently and a value with only one missing part keeps both.

  • A length-one vector is written bare wherever an array cannot be mistaken for it: at the document root, as an object value, as an attribute value, and as the payload of a tagged object. Brackets survive only around an array element, where they are the one thing separating list(1, 2) from c(1, 2).

  • A key beginning with a single ~ is reserved throughout the document, and one the reader cannot use as a name is now an error rather than data. A name is a string and JSON keys are strings, so NA_character_ is the only name JSON cannot carry: ~zNA_character_ is the one string tag a key may hold, and the rest are refused there, which also settles {"~zInf":1} and {"~~zInf":1} having both rebuilt the name "~zInf". The escape rule already doubled the prefix on any name of your own, so nothing this package writes is affected, and a later tag can land without every reader built before it silently returning a wrong value.

  • The ~t tag is written only where the payload cannot state the type itself. The decimal points in [1.0,2.0] already make it a double vector, so as.Date("2026-01-01") writes as {"~a":{"class":"Date"},"~v":20454.0}. The key stays where reading the payload alone would escalate it in turn — an empty vector, a complex or raw value, an object with no data part — which makes its presence a property of the value’s type, so c(a = 1) carries none and c(a = 1)[0] carries one. A reader now recognizes the tagged form by any of ~t, ~a and ~v, and a ~t that repeats what its payload says is still honored on the way in and dropped on the way out. Across the test corpus of 467 values this drops 210 of the 433 tags; a board from blockr_ser() is untouched, since its class information is data rather than R attributes and it carries no ~t at all.

  • A string tag is now recognized by a reserved discriminator rather than by exact match, so one this reader does not know is an error rather than data. The reserved set is ~z, which spells the typed NA and non-finite tags, and ~:, which names a symbol; every other tilde-leading string stays a string, which is what keeps "~/data" a path. The key rule above shuts the same hole at key position, and the alphabet is what a later spelling has to fit inside, since a discriminator only earns a refusal from readers that already carry it.

  • The ~r6 record is gone. An instance is refused unless its class supplies a method, and a class that does supply one writes the ordinary ~x extension record, so the tag has no writer left. A reader meeting it in an old document reports `~r6` is not a tag this reader knows rather than guessing, and the spelling is retired rather than recycled, since reusing it for a different meaning later is what would turn that error into a wrong value.

  • A record naming a class carries the whole class vector rather than its first element, so {"class":["Derived","Base","R6"]} replaces {"class":"Derived"}. The reader rebuilds the same vector by walking the generator’s get_inherit() chain and errors when the two disagree, which turns a generator answering to the right name while declaring a different class into a report rather than a wrong object. A class prepended on an instance survives the round trip, since the recorded vector still names the class the generator declares.

Measured on a board produced by blockr.core::blockr_ser(), the unboxing rule unwraps all 55 wrapped scalars and shortens the document from 2284 to 2174 bytes. A payload rich in named atomic vectors moves the other way, since those now escalate.