There’s a 1GiB limit for a single Read call for an os.File entity (object? struct?) in Go, even though native read syscall can fill a 2GiB buffer (as tested in my arm macos and Intel Linux machine). I ran into this when looking at a pprof profile of a sample word count program I was writing, which showed the program was spending way too much time in the syscall module. That in this context can only mean one thing: way too many read syscalls were getting called. Something like this would show this behaviour:

f, err := os.Open("superlargefile.txt")
if err != nil {
    log.Fatal("error opening input file: ", err)
}
defer f.Close()

buf := make([]byte, 1024*1024*1024*2) // 2GiB buffer
fmt.Println("buffer size", len(buf))

for iter := 1; ; iter += 1 {
    n, err := f.Read(buf)

    if err != nil {
        if err == io.EOF {
            fmt.Println("done")
            break
        }

        log.Fatal("error reading input file: ", err)
    }

    fmt.Println("bytes read: ", n)
    fmt.Println("iter: ", iter)
}

That, on a 2.5G file would output something like:

buffer size 2147483648
bytes read:  1073741824
iter:  1
bytes read:  1073741824
iter:  2
bytes read:  490442752
iter:  3
done

Even though the initialised buffer size is 2GiB, only 1GiB is read into the buffer per iteration. Upon digging into the source code, it looks like this is a deliberate choice. The main change logs from the history point to the following:

1. https://codereview.appspot.com/89900044 as a fix for golang/go#7812. This had a fix for failing reads on file sizes greater than or equal to 2GiB on macos and freebsd by capping each read syscall to only read a 2GiB-1 bytes. For the rest of operating systems, at this point, there was no cap.
2. https://codereview.appspot.com/94070044 as a followup of 1, where the limit was decreased without any OS checks to 1GiB, with an explanation that at least it would allow for aligned reads from disk, as opposed to an odd number that might miss page caches (my understanding).

Note that a lot has changed since that changeset, and the current file reference for that _unix.go file in the changeset is src/internal/poll/fd_unix.go.

Aside: System limits

As per the linux read syscall documentation, the maximum bytes that can be transferred is 2GiB. And I tested this out with rudimentary scripts in Rust and C. The Rust program is taken verbatim from the example for read_to_end(). Running that under strace has the following output (truncated here):

read(3, ..., 6594816000) = 2147479552
read(3, ..., 4447336448) = 2147479552
read(3, ..., 2299856896) = 2147479552
read(3, ..., 152377344) = 152377344
read(3, "", 32)         = 0

And a similar, simple C program results in similar output, when using the read syscall in a loop until the file is read:

SSIZE_MAX: 9223372036854775807 # outputting the limits.h constant
bytes read: 2147479552
bytes read: 2147479552
bytes read: 2147479552
bytes read: 152377344

Although that’s neither here nor there, it’s still interesting that Go’s choice has been to pick 2GiB-1 and then 1GiB justifying the odd buffer size in the former.

The classnames library is a very handy tool to apply CSS classes conditionally in JavaScript components. Since the output of the function is just a string, this can be composed very well on multiple conditionals layered on on various parts of the code.

For example, consider the following:

import cx from 'classnames';

switch (type) {
  case: 'textarea':
    const textareaClassNames = cx('text-area', 'text-input', 'invalid': !this.state.valid);
    return <textarea className={textareaClassNames} />
  default:
    const inputClassNames = cx('text-input', 'invalid': !this.state.valid);
    return <input type={type} className={inputClassNames} />
}

The class-names are the same except for one extra item in the case of textarea type input field. Until today, I would’ve done something like the above example, since I never bothered to look at the actual output of the call. A quick glance at the source code of the library made it evident that the library would enable composition with output of another classname-generated value (which is a String). So that code can be simplified to:

import cx from 'classnames';
const className = cx('text-input', 'invalid': !this.state.valid);

switch (type) {
  case: 'textarea':
    return <textarea className={cx(className, 'text-area')} />
  default:
    return <input type={type} className={className} />
}

Much better.

node enum-to-const-object.mjs source.ts [...]
node enum-to-const-object.mjs -w source.ts [...]
node enum-to-const-object.mjs --write source.ts [...]

The first invocation would print out the result to standard out, whereas the latter two would update the source file in-place, exactly how prettier works. The standard library API is pretty neat for such a simple interface:

// file: enum-to-const-object.mjs
// Note the mjs extension, which is why I'm able to use import. Otherwise,
// you'll have to use require in place of the following line
import { parseArgs } from 'node:util';

const options = {
	write: {
		type: 'boolean',
		short: 'w',
		default: false
	}
};

const { values, positionals } = parseArgs({ options, allowPositionals: true });
// values is of the shape { write: <flag value> }
// positionals: [ source.ts, ... ]

The options object is the one used by the parser as the configuration of the flags. The keys of this object are the expected flags in long-hand. The short property for each of these long-hand flags helps with adding aliases.

In addition to the flag format and strings, there is one more additional option that I had to configure: allowPositionals. This returns rest of the arguments that are not flags, which in my case are the files I wanted to transform. Once parseArgs is called using the configuration, and (by default) on process.argv, the flag values as an key-value pair, and the rest of the arguments are returned―values which contains the flag values, and positionals which contains the file list.

Docs Link

Content-Security-Policy: default-src 'self' images.kgrz.io; report-uri: /report-block
Content-Security-Policy-Report-Only: default-src 'self'; report-uri: /report-only

This CSP setting ensures resources only from images.kgrz.io are successfully loaded onto the page. I had always had the implicit understanding that the image load will be blocked, and a report sent out to /report-block path. But this is not the case: there’ll be two reports sent-out: one to /report-block, and one to /report-only.

The sample application that demonstrates this example is hosted at /apps/csp. You may have to have a modern-ish website to use this since I’m using no build pipeline for the JS that’s used on the page. (Anything that [supports <script type="module">][module_can_i_use])

"/p

pastes the search pattern in the current buffer. This same register value is used for repeat searches (n in normal mode). I’m not sure when I’m ever going to use this, but at least I’ll start to understand some shortcut on vim.fandom.com or SO that has an odd / in the command.

So, here’s a small reminder for the future-me that if the container has to be in running state, the state should be set to started. present only ensures the container is there in the process list, and not in running state. i.e., ignoring all the other interactions that this parameter has with other options, state: present is effectively docker create, whereas state: started is analogous docker run.

Aside: Why use ansible to run docker?

This whole setup might seem convoluted, but there’s a reason why I chose this. Before I had an M1-based Macbook, I was using Vagrant for local testing of the ansible pipeline. Everytime I make code changes for the server configuration, I’ll run the entire Ansible playbook end to end to setup the local VM provisioned by Vagrant, followed by basic curl-based tests that check for status codes and cache headers against the server. Vagrant uses VirtualBox for running the virtual machines, which is not supported on M1 chips. The process of converting this exact setup to instead use Docker is not that straight forward in my experience so far. Running docker build and docker run directly won’t work in all cases since I use Ansible template that interpolates variables into the nginx configuration files, which have to be compiled before I actually build the docker image. So the short-term alternative I was trying out was to use another playbook that runs every Ansible task in it locally, which includes letting Ansible take care of building, running the server container and the tests.

The ubiquitous IEEE floating-point standard defines two numbers to represent zero, the positive and the negative zeros. You also have the positive and negative infinity. If you compute the inverse of the positive zero, you get the positive infinity. If you compute the inverse of the negative zero, you get the negative infinity.

I wanted to check this out for the most frequent languages I tend to use—Ruby and JavaScript.

First up, Ruby:

minus_zero = -0.0
plus_zero = 0.0

converted = "-0.0".to_f

puts 1.0 / minus_zero
puts 1.0 / plus_zero
puts 1.0 / converted

Output: -Infinity, Infinity, -Infinity

Next, JavaScript:

const minus_zero = -0.0
const plus_zero = 0.0

const converted = parseFloat("-0.0", 10)

console.log(1.0 / minus_zero)
console.log(1.0 / plus_zero)
console.log(1.0 / converted)

Output: -Infinity, Infinity, -Infinity

Both languages (Ruby v. 2.7.1, JavaScript(NodeJS) v. 12.14.x & Chrome 91.x) return the values as expected in the post.

A very simple way to circumvent this restriction is by using a webfont. The simplest option is to use an @import statement and load a webfont from, say, Google fonts or some other service (your own web server too!).

Example stylesheet:

@import url('https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:ital@0;1&display=swap');

pre, code {
	font-family: 'IBM Plex Mono', monospace;
}

And use the Preferences > Advanced > Style sheet, and load this file from disk. This trick, and a little bit of default zoom makes the IETF spec docs much much better to read.

Sample screenshot of one of the IETF spec docs I’m reading:

This is not perfect, but goes a long way without needing to install extra plugins ala Stylebot or Cascadea for the basic usecase.

I’ll use an example from GitHub’s /commits REST API, using PR: ruby/ruby#3365. I’ve saved the response in the repo where I’ve added full implementation of the example used in this post. The commits response from GitHub REST API is very verbose, depending on the PR size, and having depth greater than 1. In the hypothetical application that I’m writing, I need a list of “objects” that have the following information:

type MetaData struct {
	Author string
	Committer string
	SHA string
	Message string
}

That is, I want parse this response into a []MetaData slice. I do not want to traverse the structs in the format of the responses in my main “business logic”, as that makes it hard to follow the important bits. I don’t want to use interface{} as a placeholder. A better trade-off, in my opinion and use case, is to do as much as possible during the parse phase to massage the data into the structure you want¹. I’m positive that this is a common use case. I ended up learning one way to do this cleanly almost by accident. First, the components involved:

Use anonymous structs

Anonymous structs can be used to avoid defining a concrete type and skip giving it a name for one-off use-cases. It’s heavily used in parsing and marshalling code paths, and testing. In our case, this technique can be used to define a “dirty” struct inside the UnmarshalJSON function on the fly, and use that for parsing the JSON.

Implementing Unmarshaler interface

Any type that has a UnmarshalJSON function on it implements the Unmarshaler interface. This type then can be used as the target for parsing a JSON sub tree or the entire JSON itself!

Implementation

First step is to mock out the main function:

func main() {
	// This variable contains the raw json bytes that resulted from the
	// API call. I'm not adding the code for the actual network fetch
	// for now, but in the example repository, I read the commits
	// response from a file
	var jsonb []byte
	jsonb = JSONFromSomewhere()

	var metadatas []MetaData
	if err := json.Unmarshal(jsonb, &metadatas); err != nil {
		log.Fatalln("error parsing JSON", err)
	}

	fmt.Println(metadatas)
}

The JSON response of /commits endpoint is a list of commit objects, and I’m using a list of MetaData types to match that interface. For each commit item from the JSON array, the raw bytes get passed as the argument to the UnmarshalJSON function on MetaData.

Next step is to implement the UnmarshalJSON function using an anonymous struct to parse out the raw commit object JSON string into it:

func (m *MetaData) UnmarshalJSON(buf []byte) error {
	var commit struct {
		SHA    string `json:"sha"`
		Commit struct {
			Author struct {
				Name string `json:"name"`
			} `json:"author"`
			Committer struct {
				Name string `json:"name"`
			} `json:"committer"`
			Message string `json:"message"`
		} `json:"commit"`
	}

	if err := json.Unmarshal(buf, &commit); err != nil {
		return errors.Wrap(err, "parsing into MetaData failed")
	}

	// continued
}

Final step is to process the commit struct, and set the appropriate fields on MetaData struct:

func (m *MetaData) UnmarshalJSON(buf []byte) error {
	// same as above

	m.AuthorName = commit.Commit.Author.Name
	m.CommitterName = commit.Commit.Committer.Name
	m.SHA = commit.SHA
	m.Message = commit.Commit.Message

	return nil
}

That’s it! An additional advantage to this type of narrow types is it’s easier to test.

Bonus: Filtering the slice further

For bonus points, I want to skip certain []MetaData elements based on a condition. A way to do this, keeping the same principles as above in mind, is to define a type that covers []MetaData, which implements the Unmarshaler interface:

type MetaDatas []MetaData

func (ms *MetaDatas) UnmarshalJSON(buf []byte) error {
	// []MetaData is not the same as MetaDatas, and this difference is
	// important!
	var metadatas []MetaData

	if err := json.Unmarshal(buf, &metadatas); err != nil {
		log.Fatalln("error parsing JSON", err)
	}

	// filtering without allocations
	// https://github.com/golang/go/wiki/SliceTricks#filtering-without-allocating
	cleanedms := metadatas[:0]
	for _, metadata := range metadatas {
		if !strings.HasPrefix(metadata.Message, "WIP") {
			cleanedms = append(cleanedms, metadata)
		}
	}
	*ms = cleanedms

	return nil
}

Like before, I’m using a temporary type of the kind that matches our main type, and using that to parse into. Then I’m clean out slice based on a condition—I want to skip all the commits that start with WIP. Note that the metadatas variable defined inside the UnmarshalJSON function is defined as []MetaData and not as MetaDatas, since doing that would result in a parse-loop. By design, var metadatas Metadatas and var metadatas []MetaData are not the same type.

Finally, the filtered slice gets assigned to the underlying object that the JSON is getting parsed into.

A note about performance

In these examples, the parse flow will create the entire []MetaData slice, even though we filter out many of the elements. To my knowledge, this seems like a necessary hit to take. I’m not aware if there’s a way to avoid allocations by pre-pre-processing the incoming bytes to avoid the allocation in the first place. My thought process here is that if we didn’t filter, or cleanup the JSON data, it will anyway allocate all the objects, so this may not be a huge difference in allocations per se, but that’s just my opinion at this point.

I’m using /data directory to signify the data storage directory. This depends on the chosen configuration, however.

• Backups
  • SQL Dump/Load
  • Copy /data directory
    • frozen snapshots
    • rsync, tar, et al.
  • Continuous Archiving
• Restore
• Replication

Backups

Three main types of backup strategies:

• SQL dump/load (stop-the-world)
• Backup /data directory (stop-the-world)
• Continuous Archiving

The first two typically need lots of extra space on the database server to store the backup before you can upload it to some off-site storage.

“stop-the-world” in this context is not an official nomenclature. In these strategies, most likely, the database needs to be shut down at some point.

Continous archive-based backups are used for a leader-follower setup, or even delta backups—where there’s a base backup, and subsequent data as deltas that can be used to restore the entire database. Application-supported remote backups are quite simple, and so this is the best strategy if the database servers are space-constrained.

SQL Dump/Load

This is the easiest strategy. pg_dump takes a backup, while pg_restore command consumes the output of that backup. This strategy is the simplest to cron-ify a backup, without external dependencies: take a backup, upload the files to remote storage, test the backup on a different machine, and do this every night.

• pg_dump saves the database into a .sql statement. Requires large enough space to hold both the database and the backup script.

• File sizes might be limited by kernel/OS, so that’s something to look ahead while deciding to use this.

• Restore from the pg_dump output might also need extra configuration tweaking around connection times: too less, and the database might close the connection before the entire script has run.

Copy /data directory

PostgreSQL’s directory layout is straight-forward—once you get to know it. Most of the data is put in one directory, and this includes the two main components needed for any future restores: the data files, and the temporary append-only log files. If the database is shut down, you’re free to copy over the data directory to another machine, and start off from it. Configuration files typically aren’t placed in the data directory, so they might need to be copied as well.

Any strategies that have to rely on the file-system layout of PostgreSQL, or features provided by the file system itself.

Two routes here: frozen snapshots of the file system, or using tools like rsync, tar etc.

frozen snapshots

• If the underlying file system supports atomic volume snapshots (btrfs, zfs, Apple’s APFS for example), one can snapshot the entire data directory. Lots of caveats around how good the snapshot mechanism is implemented exist.

• The backup can be taken without stopping the server. During restore, this strategy would require replaying the logs as there might be some commits that weren’t turned to data files from the append only log.

rsync, tar, et al.

• rsync, gzip, tar the data directory. These utilities don’t take consistent snapshots of the disk, so it’s best to shutdown the server. Shutting down the server forces a full flush of the data to disk.

• An example two step process with rsync:

run rsync
shutdown the server
rsync --checksum

What’s interesting is that this two-step process is similar to the one used in online backups section. This is like a one-step delta backup process if we stretch it enough: the first backup is a base backup that contains the data committed till that point, then the second rsync takes the delta and copies that over.

Continuous Archiving

This system can be used to setup a replicated system, consisting of a leader and potentially multiple followers. The data from the leader is pushed, and each of the followers might pull the data. Where this data is stored is customisable. There are many ways to setup replication in PostgreSQL, and the documentation for it is exhaustive. The archival part deals with the first part: taking the backup and pushing it somewhere.

This strategy piggy-backs on the fact that a WAL log may be used to replay and restore a database. There are many caveats and configuration tweaks to how long the WAL log files are retained, the size of those log files and the naming of the files. It’s best to ship the log files as and when they are created to an external storage service. Rather than do this manually via rsync et. al., PostgreSQL provides a way: archive_command setting in the configuration, which takes a script.

• WAL logs should be secured while transmission and remote storage, because these contain the actual data. (that goes for the main database too, fwiw)

• archive_command should exit with 0 code. Otherwise, the command gets retried. The pg_wal directory may potentially get filled, and cause the server to crash!

• archive_command should be designed to ensure it doesn’t override existing files on the remote system.

• Missing WAL logs from the archive might hamper future restore, so regular base backups will help keep the error surface area a little small.

• old base backup + too many WAL logs to restore increase the restore time. It’s important to determine the maths behind this to figure out how much downtime you might need and tweak the base backup frequency, and WAL file size accordingly.

General mechanism:

• One base backup as a starting point
• Continuous deltas in the form of the append-only log (WAL) files

The base backup marks the point where the backup would start (checkpoint).

base backup

Two ways to take a base backup:

• Use the pg_basebackup command from an external machine (or the same machine, with a different data directory setting), providing the connection info to connect to the leader.

  • Multiple commands can be run from multiple machines, but might depend on replication slots configuration on the leader.

  • Might use one/two connections depending on the variant of backup used: copy WAL logs at the end (1) or stream WAL logs parallelly (2).

  • Does not run if /data directory is not empty.

• Two-step process via rsync. PostgreSQL provides two SQL statements for signalling the server that the user is taking a backup, and that a checkpoint has to be created: pg_start_backup, pg_stop_backup.

SELECT pg_start_backup('some_label')
rsync /data
SELECT * from pg_stop_backup();

Restore

stop-the-world restores

Data dumps taken with pg_dump or the file system strategy mentioned above can be restored by pg_restore or just starting the server. Needless to say, this strategy causes either data loss or needs downtime, depending on the operations chosen.

• If a system has a simple pg_dump cron job that ships the archive to remote storage, when the leader crashes or dies, the time to detection, copying the archive to the follower, pg_restore completion times is the amount of downtime that’s required.

• The cron job, if configured at a certain time in the day, differs from the time the crash happens, the delta in the data until that time on the leader is a potential loss in data.

• When the leader crashes/dies, but you still have access to the physical data disks, recovery using file system snapshot is possible, and that may potentially recover all the data up till the point of the last commit. Because this recovery would also have the WAL files handy, the replay will make sure as much data as possible is recovered.

Continuous Archive restores

If the system is setup with continuous archiving, it may be possible to recover all the data. Restore times depend on how fast the base backup archive, WAL logs can be copied over to the new server, and the WAL log replay.

Replication

There are many ways to do this, too, depending on the underlying infra: shared disk (two machines accessing the same disk), file system replication (a write to one drive is mirrored to a different machine atomically), side-car middlewares that execute a given statement on multiple machines simultaneously, or even application-level middlewares that do this. Streaming/Point-in-time replication is one preferred approach that can piggy back on the continuous archive backup strategy.

Streaming/Point-in-time replication strategy uses wal logs shipped to a remote server using archive_command from the leader to be used to replay the logs on a follower continously.

• Note that in streaming replication is possible without using archive_command, provided the data ingestion throughput never exceeds the rate of the follower streaming the logs directly from the leader, and applying them locally (also depends on the network latency).

  If the follower is not able to keep up with the logs, the logs on leader might get recycled, and the follower will keep waiting for the now-non-existent WAL file. Force-starting the follower in case of failure will result in data loss.