Persisting a class that owns a handle

Where the default rule stops

Most classes need nothing from this page. An S3, S4 or S7 object is a base type plus attributes, so json_write() records it without help and the value that comes back is the value that went in. That rule stops at an external pointer, which names something that exists only inside the running session and does not survive being written down. An environment is recorded — by name where a name finds it again, by its contents otherwise — but only as far as what it binds, so one holding an external pointer is refused with it, as is one holding a promise or an active binding, since reading either would run code.

A connection is the case worth walking through, because it does not look like a handle. It prints as a small integer, and an integer is data.

library(typedjson)

path <- file.path(tempdir(), "events.log")
writeLines(sprintf("event %d", 1:6), path)

con <- file(path, open = "r")
str(unclass(con))
#>  int 4
#>  - attr(*, "conn_id")=<pointer: 0x105>

The integer is a slot in this process’s table of open connections, and the external pointer beside it is what makes using that slot safe. Neither means anything in the session that reads the document back, so the writer refuses.

json_write_str(con)
#> Error:
#> ! cannot write a value of type 'externalptr' at `x$conn_id`

Refusing is the whole point. Writing the integer would hand back a number indexing whatever connection occupies that slot next, and writing null would drop the handle quietly; either way the damage surfaces later and somewhere else. The error instead names the path it stopped at, so a handle buried deep in a large value is reported by where it sits.

Everything on this page is about the default, which is the mode that reads a value back. Plain mode — typed = FALSE, for a document whose shape a schema elsewhere already fixes — writes no annotation and reads an attribute only to settle object against array and scalar against array, so it never walks into the attribute the pointer above sits in, and writes the bare slot integer this section argues against. The argument is not weaker there, it is inapplicable: what makes the integer dangerous is a reader rebuilding a connection from it, and plain mode has no revival step to do that in. The plain output section of vignette("design") states the rule and the two places it lands.

A reader that owns a connection

Here is a small class of the shape that runs into this: a reader that walks a file in chunks, holding the connection open between calls.

chunk_reader <- function(path, size = 2L) {
  structure(
    list(path = path, size = size, con = file(path, open = "r")),
    class = "chunk_reader"
  )
}

read_next <- function(x) {
  readLines(x$con, n = x$size)
}

reader <- chunk_reader(path)

read_next(reader)
#> [1] "event 1" "event 2"
read_next(reader)
#> [1] "event 3" "event 4"

Two chunks in, the reader is a perfectly ordinary list of a string, an integer and a connection, and writing it fails on the third field.

json_write_str(reader)
#> Error:
#> ! cannot write a value of type 'externalptr' at `x$con$conn_id`

What to record

The extension protocol is a pair of generics modeled on the __getstate__ and __setstate__ methods of Python’s pickle. A json_state() method returns a plain list of what to persist, and is free to leave out anything that can be rebuilt.

The judgment it asks for is which parts of the object are the state and which are only the means. Here the connection is the means: what identifies this reader is the file it walks, the chunk size it walks in, and how far into the file it has got.

json_state.chunk_reader <- function(x) {
  list(path = x$path, size = x$size, offset = seek(x$con))
}

That is enough to write it.

cat(json_write_str(reader, pretty = TRUE))
#> {
#>   "~x": {
#>     "class": "chunk_reader",
#>     "state": {
#>       "path": "/tmp/Rtmp2g3HVD/events.log",
#>       "size": 2,
#>       "offset": 32.0
#>     }
#>   }
#> }

Nothing in the document is opaque. The ~x tag says an extension method produced it, class is what dispatches on the way back, and the rest is the state as the method returned it — read under the ordinary rules, so offset is a double because seek() returns one.

Getting it back

Reviving is the other half. Dispatch happens on the recorded class through an empty object carrying it, so the first argument is the class token rather than the object being rebuilt, which does not exist yet.

json_revive.chunk_reader <- function(class, state) {
  con <- file(state$path, open = "r")
  seek(con, state$offset)

  structure(
    list(path = state$path, size = state$size, con = con),
    class = "chunk_reader"
  )
}

The reader that comes back is a different connection on the same file, positioned where the old one stopped, and it carries on.

doc <- json_write_str(reader)

revived <- json_read_str(doc)
read_next(revived)
#> [1] "event 5" "event 6"

The document is the interface

The offset is a number in a text file, which means it can be edited. Winding this reader back to the start of the file is a matter of changing one number, and there is nothing else that has to be kept in step with it.

rewound <- json_read_str(sub('"offset":[0-9.]+', '"offset":0.0', doc))
read_next(rewound)
#> [1] "event 1" "event 2"

That is the argument for the format in one line. The persisted state of a handle-owning object is exactly the kind of thing worth reading in a diff and worth fixing by hand, and neither is possible once it is a blob only R can open.

When the resource is gone

Reviving runs your code, so what happens when the file has moved is your method’s decision rather than the format’s. The method above opens eagerly, so it reports the problem as soon as the document is read.

json_read_str(sub(basename(path), "no-such-file", doc, fixed = TRUE))
#> Warning in file(state$path, open = "r"): cannot open file
#> '/tmp/Rtmp2g3HVD/no-such-file': No such file or directory
#> Error in `file()`:
#> ! cannot open the connection

Opening lazily on the first read_next() call, or falling back to offset zero when the file is shorter than the recorded position, are equally valid; the protocol takes no view.

Methods have no privileges

Whatever a json_state() method returns is written under the same rules as any other value, which is what keeps the guarantee honest: a method cannot smuggle a handle out by wrapping it in a list.

The guarantee bounds what a method may write, and not when one is asked, so read it in that direction only. Plain mode does not ask at all: a method written to keep a field out of a document is not consulted under typed = FALSE, and the field it was holding back is written with the rest of the object.

json_state.leaky <- function(x) list(con = x$con)

leaky <- structure(list(con = file(path)), class = "leaky")
json_write_str(leaky)
#> Error:
#> ! cannot write a value of type 'externalptr' at `x$state$con$conn_id`

A class with no default to fall back on

An R6 class is the case where there is no default at all. What an R6 class guarantees is what its methods say, and its private fields are private precisely because they are not part of that, so recording an instance as the bindings it happens to hold is an assertion only the class can make. Writing one is refused, naming the class and the method that would settle it.

ChunkReader <- R6::R6Class("ChunkReader",
  public = list(
    path = NULL,
    size = NULL,
    con = NULL,
    initialize = function(path, size = 2L) {
      self$path <- path
      self$size <- size
      self$con <- file(path, open = "r")
    },
    read_next = function() {
      readLines(self$con, n = self$size)
    }
  )
)

reader <- ChunkReader$new(path)
json_write_str(reader)
#> Error:
#> ! an `R6` instance needs a `json_state()` method for class `ChunkReader` at `x`

The pair on this page is the answer, written exactly as it was for the S3 reader above. Ordinary dispatch settles which one is found: the class vector is c("ChunkReader", "R6"), so a method on ChunkReader is reached ahead of the one on R6 that does the refusing.

json_state.ChunkReader <- function(x) {
  list(path = x$path, size = x$size, offset = seek(x$con))
}

json_revive.ChunkReader <- function(class, state) {
  out <- ChunkReader$new(state$path, state$size)
  seek(out$con, state$offset)
  out
}

reader$read_next()
#> [1] "event 1" "event 2"

doc <- json_write_str(reader)
cat(doc)
#> {"~x":{"class":["ChunkReader","R6"],"state":{"path":"/tmp/Rtmp2g3HVD/events.log","size":2,"offset":16.0}}}

revived <- json_read_str(doc)
revived$read_next()
#> [1] "event 3" "event 4"

Going through $new() on the way back is the usual shape for a class whose constructor acquires the resource. The state list records what the constructor needs, and whatever the constructor cannot know — here the offset — is applied to the instance afterwards.

Rebuilding through the constructor is also what keeps a finalizer honest. Where initialize acquires a handle and finalize releases it, the two only line up if the revived object went through initialize as well; an object filled in from a document acquired nothing, and its finalizer runs against a handle it never held. That hazard is one of the reasons an R6 instance has no default in the first place, and vignette("r6") walks through deciding what a class’s state actually is.