https://docs.datadoghq.com/account_management/billing/custom_metrics/?tab=distribution

Here is a list of the available regions as of October 2025 and where they run:

UniSuper’s production Google Cloud VMware Engine (GCVE) private cloud was automatically deleted one year after it’s creation due to a misconfiguration in how it was created. When it was created, there was a bug in the creation script which passed a null value. This caused the private cloud to be created with a one year subscription, rather than a perpetual one. After one year passed, Google Cloud dutifully deleted the private cloud.

The rest of this post is still relevant for more context, but it is now clear that this was indeed a Google Cloud bug. I remain puzzled why the communication over this incident was so bad. By their silence, Google Cloud let millions of people think that they had deleted a company’s entire Google Cloud environment. I’m also confused why this information came out via a non-employee on Twitter rather than through official channels.

Update 2: Google Cloud has published a blog post explaining exactly what happened. The details do bear out that this was entirely a Google Cloud issue from start to finish, with no fault on the part of UniSuper.

Over the past two weeks, UniSuper, an Australian superannuation fund, faced every cloud user’s nightmare: their cloud provider deleted their data. From May 2nd to 13th, UniSuper experienced a major outage and in several updates blamed Google Cloud for the incident.

The disruption of UniSuper services was caused by a combination of rare issues at Google Cloud that resulted in an inadvertent misconfiguration during the provisioning of UniSuper’s Private Cloud, which triggered a previously unknown software bug that impacted UniSuper’s systems. This was an unprecedented occurrence, and measures have been taken to ensure this issue does not happen again.

This culminated in a press release, jointly signed by Google Cloud CEO Thomas Kurian.

Google Cloud CEO, Thomas Kurian has confirmed that the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription.

This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again.

UniSuper had duplication in two geographies as a protection against outages and loss. However, when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies.

Restoring UniSuper’s Private Cloud instance has called for an incredible amount of focus, effort, and partnership between our teams to enable an extensive recovery of all the core systems. The dedication and collaboration between UniSuper and Google Cloud has led to an extensive recovery of our Private Cloud which includes hundreds of virtual machines, databases and applications.

The press release uses vague language, obscuring the technical details of what happened. Given the lack of precision, I assumed this was written solely by UniSuper. Gergely Orosz checked with Google Cloud, and they confirmed that this was an official joint statement.

Given the few details provided, I did more research to try to understand what exactly happened.

What was deleted?

Google Cloud doesn’t have a resource called a “Subscription”; the closest concept would probably be a “Billing Account.” While Google has a Virtual Private Cloud (VPC), you wouldn’t describe it as an instance, and deleting a VPC doesn’t seem like it would have the drastic effects described.

However, if you research UniSuper’s Google Cloud migration, you will see that they use Google Cloud VMWare Engine (GCVE). From June 2023:

The superannuation fund has shifted almost all non-production workloads out of its data centres and into Google Cloud so far.

UniSuper used the Google VMware Engine (GCVE) managed service and engaged partner Kasna to assist with the migration.

GCVE has a resource called a private cloud. A private cloud contains the hosts, management server, storage, and networking for a VMware stack.

Let’s look at the API method for deleting a private cloud:

A PrivateCloud resource scheduled for deletion has PrivateCloud.state set to DELETED and expireTime set to the time when deletion is final and can no longer be reversed. The delete operation is marked as done as soon as the PrivateCloud is successfully scheduled for deletion (this also applies when delayHours is set to zero), and the operation is not kept in pending state until PrivateCloud is purged. PrivateCloud can be restored using privateClouds.undelete method before the expireTime elapses. When expireTime is reached, deletion is final and all private cloud resources are irreversibly removed and billing stops.

All private cloud resources being irreversibly removed sounds a lot like the outage that UniSuper had. Some people have said that Google deleted their entire Google Cloud account. This seems less likely to me because Google has a number of safeguards and delays when deleting projects:

Warning: You can recover most resources if you restore a project within the 30-day period. Some services have delays in restoring and you might need to wait some time for services to be restored. Some resources, such as Cloud Storage or Pub/Sub resources, are deleted much sooner. These resources might not be fully recoverable even if you restore the project within the 30-day period.

Update: User smcwhtdtmc on HN pointed out the Terraform resource for vmwareengine_private_cloud hard-codes the delayHours parameter to 0, i.e. immediate deletion. All private cloud resources being immediately irreversibly deleted sounds a lot like the outage that UniSuper had.

How did everything get deleted?

UniSuper’s press release says:

UniSuper had duplication in two geographies as a protection against outages and loss. However, when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies.

Again, the language here is frustratingly vague and passive but implies that Google Cloud deleted multiple independent private clouds in separate regions. Google Cloud doesn’t have a “geography”; it has zones and regions. At first read, it sounds like they are describing a multi-region setup. Google Cloud has two Australian regions, Sydney and Melbourne, which would make sense.

Looking closer at the docs, though, GCVE offers two kinds of private clouds: a standard private cloud hosted in a single zone or a “stretched private cloud”. A stretched private cloud runs in a single region across two zones, with a third zone as a witness zone for failover. A close reading of the press release doesn’t rule out UniSuper having a single stretched private cloud running in a single region.

So we either have a single stretched private cloud being deleted or two separate private clouds being deleted. A single stretched private cloud seems more likely to me, but I can’t be definitive.

How was it deleted?

So we come to the big question: How were the private cloud(s) deleted? The press release makes heroic use of the passive voice to obscure the actors: “an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription.”

Update: 2024-05-21 Miles Ward has the details on what actually happened in this X thread. My guesses were close, but not quite right.

Based on my experiences with Google Cloud’s professional services team, they, and presumably their partners, recommend Terraform for defining infrastructure as code. This leads to several possible interpretations of this sentence:

1. UniSuper ran a terraform apply with Terraform code that was “misconfigured”. This triggered a bug in Google Cloud, and Google Cloud accidentally deleted the private cloud.

This is what UniSuper has implied or stated throughout the outage.

2. UniSuper ran a terraform apply with a bad configuration or perhaps a terraform destroy with the prod tfvar file. The Terraform plan showed “delete private cloud,” and the operator approved it.

Automation errors like this happen every day, although they aren’t usually this catastrophic. This seems more plausible to me than a rare one-in-a-million bug that only affected UniSuper.

3. UniSuper ran an automation script provided by Google Cloud’s professional services team with a bug. A misconfiguration caused the script to go off the rails. The operator was asked whether to delete the production private cloud, and they said yes.

I find this less plausible, but it is one way to interpret Google Cloud as being at fault for what sounds like a customer error in automation.

Maybe it was a Google Cloud bug?

There are a few holes in my theory that this was primarily UniSuper’s fault:

1. On Tuesday, 7 May, UniSuper attributed a statement to Google Cloud in which Google Cloud admitted fault for the outage:

The disruption of UniSuper services was caused by a combination of rare issues at Google Cloud that resulted in an inadvertent misconfiguration during the provisioning of UniSuper’s Private Cloud, which triggered a previously unknown software bug that impacted UniSuper’s systems. This was an unprecedented occurrence, and measures have been taken to ensure this issue does not happen again.

2. Thomas Kurian (apparently) signed off on a joint statement with UniSuper. This statement is less accusatory than UniSuper’s other statements but doesn’t clarify which party was at fault.

3. Google Cloud hasn’t released any pushback to the story, either directly or through proxies in the media.

Why would Google Cloud not respond to this story if it was UniSuper’s fault? It’s hard to say, but putting out a competing statement blaming or contradicting your customer is a bad look with that customer and with all future customers.

Conclusion

Given how little detail was communicated, it is difficult to make a conclusive statement about what happened, though I personally suspect UniSuper operator error was a large factor. Hopefully, APRA, Australia’s superannuation regulator, will investigate further and release a public report with more details.

create apple id “cannot find this person”

$ xcrun notarytool store-credentials "Whimsical Development" --apple-id "name@company.com" --team-id "2N6497CB83" --verbose
[18:23:07.326Z] Debug [MAIN] Running notarytool version: 1.0.0 (32.0.1), date: 2024-02-29T18:23:07Z, command: /Applications/Xcode-15.2.0.app/Contents/Developer/usr/bin/notarytool store-credentials Whimsical Development --apple-id name@company.com --team-id 2N6497CB83 --verbose

This process stores your credentials securely in the Keychain. You reference these credentials later using a profile name.

App-specific password for name@company.com:
Validating your credentials...
[18:23:21.756Z] Info [API] Initialized Notary API with base URL: https://appstoreconnect.apple.com/notary/v2/
[18:23:21.756Z] Info [API] Preparing GET request to URL: https://appstoreconnect.apple.com/notary/v2/test?, Parameters: [:], Custom Headers: private<Dictionary<String, String>>
[18:23:21.757Z] Debug [AUTHENTICATION] Delaying current request to refresh app-specific password token.
[18:23:21.757Z] Info [API] Preparing GET request to URL: https://appstoreconnect.apple.com/notary/v2/asp?, Parameters: [:], Custom Headers: private<Dictionary<String, String>>
[18:23:21.757Z] Debug [AUTHENTICATION] Authenticating request to '/notary/v2/asp' with Basic Auth. Username: name@company.com, Password: private<String>, Team ID: 2N6497CB83
[18:23:21.758Z] Debug [TASKMANAGER] Starting Task Manager loop to wait for asynchronous HTTP calls.
[18:23:22.825Z] Debug [API] Received response status code: 401, message: unauthorized, URL: https://appstoreconnect.apple.com/notary/v2/asp?, Correlation Key: U7UUHDJOSFM5N7DWNJZMTALNYE
[18:23:22.826Z] Error [TASKMANAGER] Completed Task with ID 2 has encountered an error.
[18:23:22.826Z] Debug [TASKMANAGER] Ending Task Manager loop.
Error: HTTP status code: 401. Unable to authenticate. The application is not allowed for primary authentication. Ensure that all authentication arguments are correct.

Turbo replication is a Cloud Storage feature designed to asynchronously replicate newly written Cloud Storage objects associated with any insert, rewrite, copy, or compose operation—regardless of object size—to a separate region within a target of 15 minutes. Available for dual-region buckets, turbo replication offers a shorter, more predictable recovery point objective (RPO), helping to reduce data loss exposure.

I was a bit puzzled by this at first. Google Cloud’s dual-regional buckets are already replicated across multiple regions, and can serve data from either region. Aren’t all objects replicated at write time?

After reading the docs a few times, I realised what this paragraph was saying:

Cloud Storage always understands the current state of the bucket and transparently serves objects from either region as required. As a result, dual-region buckets are designed to have a recovery time objective (RTO) of zero, and temporary regional failures are normally invisible to users; in the case of a regional outage, dual-region buckets automatically continue serving all data that has been replicated between the two regions.

1. Writes to a dual-region bucket are asynchronously replicated between the region pairs. This replication can be quite slow. Under standard replication, the target is for 99.9% of writes to be replicated within an hour.
2. If write replication is so slow, how can a multi-region bucket be useful when you need to read your writes immediately? Cloud Storage knows where each object has been replicated, and can route requests to the correct region if the object is only stored in one region.
3. If a Cloud Storage region goes down, a dual-region bucket can continue to serve objects from the other region. However, any unreplicated objects will be unavailable until the region comes back online.

Therefore, if you need stronger guarantees around the freshnesss of data during a regional outage, you can turn on Turbo Replication for a bucket. This has an SLA of replicating 100% of writes to the other region within 15 minutes. This gives an upper bound (as long as the SLA isn’t breached) of how much data might be unavailable during a regional outage. Note that once the outage is resolved, the data will be replicated to the other region.

Turbo Replication costs more than standard dual-region replication.

• For US and EU dual-region buckets, the cost goes from $0.02/GB to $0.04/GB
• For Asia dual-region buckets, the cost goes from $0.08/GB to $0.11/GB

[1208/221244.718990:ERROR:devtools_http_handler.cc(636)] Using unsafe HTTP verb GET to invoke /json/new. This action will stop supporting GET and POST verbs in future versions.

After some searching, I found that the solution is to change the request to use PUT instead of GET. It appears as if the change is to prevent XSRF attacks against the DevTools HTTP server.

I found this by using the Chromium code search tool and looking at the source of the warning. The blame led me to a discussion with Paul Irish on what should be used instead. The consensus was to use PUT. An example of this change can be seen in the Lighthouse project.

The maximum HTTP version to support on the distribution. Allowed values are http1.1, http2, http2and3 and http3. The default is http2.

I was confused about the difference between http2and3 and http3. I looked at the CloudFront API docs, which said pretty much the same thing as the Terraform docs.

I decided to look at the console to see what the options were:

After seeing this, it became a lot clearer what was going on. The documentation says “The maximum HTTP version”, but this is no longer accurate with the release of HTTP/3 support.

• http3 means means HTTP/3 and HTTP/1.1
• http2 means HTTP/2 and HTTP/1.1
• http2and3 means HTTP/3, HTTP/2, and HTTP/1.1.

Download all dependencies

Often in a CI environment, you want to download all of the dependencies for a project before building the project. In Leiningen, you can use lein deps to do this.

You can use clojure -P to Prepare the project with the Clojure CLI. This downloads libs and caches the classpath.

Note that you should avoid running clojure -P in parallel on new environments (like CI). When running clojure -P for the first time, it will set up files in ~/.clojure and clone gitlibs. If you run multiple processes doing this simultaneously, you can get race conditions that throw errors.

View the dependency tree

When debugging dependency issues, it’s often helpful to see which library pulled in a transitive dependency.

Leiningen can print the dependency showing the resolved transitive dependencies.

$ lein deps :tree
 [clansi "1.0.0"]
 [clj-kondo "2022.03.09" :scope "test"]
 [babashka/fs "0.1.2" :scope "test"]
 [cheshire "5.10.2" :scope "test"]
 [com.fasterxml.jackson.dataformat/jackson-dataformat-cbor "2.12.4" :scope "test" :exclusions [[com.fasterxml.jackson.core/jackson-databind]]]
 [com.fasterxml.jackson.dataformat/jackson-dataformat-smile "2.12.4" :scope "test" :exclusions [[com.fasterxml.jackson.core/jackson-databind]]]
 [tigris "0.1.2" :scope "test"]
 [com.cognitect/transit-clj "1.0.324" :scope "test"]
 # ...

clojure has a similar command, though with a slightly different output. In contrast to Leiningen, clojure prints the entire dependency tree. Dependencies clojure didn’t select get an X in front of them, and a short reason like :older-version or :excluded. These views can be helpful when debugging why you didn’t get the version you expected.

$ clojure -Stree
org.clojure/clojure 1.11.0
 . org.clojure/spec.alpha 0.3.218
 . org.clojure/core.specs.alpha 0.2.62
org.clojure/java.classpath 1.0.0
clansi/clansi 1.0.0
org.clojure/data.json 2.4.0
org.slf4j/slf4j-simple 2.0.0-alpha1
 . org.slf4j/slf4j-api 2.0.0-alpha1
org.owasp/dependency-check-core 7.0.2
 . us.springett/cpe-parser 2.0.2
 X org.slf4j/slf4j-api 1.7.30 :older-version
 . com.vdurmont/semver4j 3.1.0
 X org.slf4j/slf4j-api 1.7.36 :older-version

View the Classpath

While dependency trees can help track down dependency problems, sometimes you need to see the actual classpath.

Tip: I like to use | tr ':' '\n' | sort to get a sorted list of the classpath entries.

You can do this in Leiningen:

lein classpath
# or
lein classpath | tr ':' '\n' | sort

With clojure:

clojure -Spath

Both Leiningen and clojure have more commands for dependencies; you can read more about them at Dep Expansion, Deps and CLI, and lein deps.

In Maven you can specify a dependency on a version with <classifier> like this:

<dependency>
    <groupId>io.netty</groupId>
    <artifactId>netty-transport-native-epoll</artifactId>
    <version>4.1.74</version>
    <classifier>linux-x86_64</classifier>
</dependency>

To specify this dependency with Clojure’s deps.edn, you need to use the syntax <group>/<artifact>$<classifier>:

{:deps 
  {io.netty/netty-transport-native-epoll$linux-x86_64 
    {:mvn/version "4.1.74.Final"}}}

The trick is to use AWS Cost Explorer, and exclude the Credit and Tax Charge Types:

If you don’t see “Charge Type” in the Cost Explorer sidebar, you might need to expand Advanced Options.

You want to exclude taxes because these aren’t charged when you use credits. When you were paying for AWS services, you likely paid tax on that usage (depending on your location). Once you start paying for your AWS bill with credits, tax is not charged on the credit spend. Removing the tax Charge Type gives you a like-for-like comparison with previous months.

1. Prices never go up^{1 2}.
2. We will absolutely soak you on data transfer charges.

Last week Google Cloud published a post explaining how they were ignoring the first rule in favour of adopting the second.

The post was called “Unlock more choice with updates to Google Cloud’s infrastructure capabilities and pricing”, though the <title> was more straightforward: “Updates to Google Cloud’s Infrastructure pricing”. The main announcements in the post were:

• Prices were being raised on Cloud Storage (some decreases on Storage too), Persistent Disk Snapshots, Cloud Load Balancing, Network Intelligence Center, and Cloud Ops Monitoring (Weirdly, the Monitoring price increase was only mentioned in customer emails, not in the blog post).
• A new, cheaper archive snapshot option for persistent disks is coming later this year.
• Storage Transfer Service will be free for the rest of the year to let customers move files into different buckets.

The common theme with the price changes is the introduction of data transfer fees, an increase in fees for active storage tiers, and a decrease in archive storage cost.

So, today, we are announcing we will adjust our infrastructure product and pricing structure … [These changes] are also designed to better align with how other leading cloud providers charge for similar products, so customers can more easily compare services between leading cloud providers.

The main alignment here is Google adding data transfer charges to match AWS and breaking architectural promises they’ve made to their customers in the process. This is an incredibly short-sighted move and will damage customer trust in Google Cloud for many years.

Cloud Cost and Cloud Architecture

As Corey Quinn so eloquently put it, “all cloud cost is fundamentally about architecture”. When designing for the cloud, pricing is one of the most important signals to take into account. Pricing and quotas indicate how the Cloud provider has designed the product, how they want you to think about the product, and how you should use it.

The pricing changes Google is making strike at the heart of their customer’s applications and will force many customers to rearchitect their applications, or pay a much larger amount to keep their existing architecture.

One of GCP’s most differentiated features has been their multi-region and global services like Cloud Load Balancing, Cloud Storage, BigQuery, Cloud Spanner, and Cloud KMS. These let you operate on the same resources in multiple regions, giving you more resilience to an outage in a single region. As a bonus, Google’s multi-region services often come with strong consistency across regions so customers don’t have to deal with consistency at the application layer.

In contrast, most AWS services are region oriented, and when AWS provides multi-region services like DynamoDB Global Tables and S3 Cross-Region Replication, you need to handle eventual consistency yourself.

Google’s message was that multi-region services would give you higher availability. You too can run like Google. From the announcement of dual-region buckets:

With this new option, you write to a single dual-regional bucket without having to manually copy data between primary and secondary locations. No replication tool is needed to do this and there are no network charges associated with replicating the data, which means less overhead for you storage administrators out there. In the event of a region failure, we transparently handle the failover and ensure continuity for your users and applications accessing data in Cloud Storage.

People can argue about whether Google’s global/multi-regional control-plane provides more availability than AWS’s region-first approach, but it was a distinctive set of features that only Google had.

Storage changes

The deepest architectural changes Google is making are to storage:

• Increasing free internet egress from 1GB to 100GB per month. This matches AWS’ recent increase, but is still a welcome change.
• A mix of increases and decreases in multi-region pricing depending on the storage class and region (+50%, +25%, -40%, -25%). Standard storage pricing is unchanged.
• A mix of increases and decreases in dual-region pricing, depending on the storage class and region. nam4 (US) and eur4 (Europe) are +22% on standard, +10% on nearline, -2% on coldline, and -44% on archive. asia1 (Japan) is +10% for all storage classes except for -13% on archive storage.
• Storage operation pricing changes have a lot of fine details, but at a high level the cost for Insert/Update/Delete operations in multi-region and dual-region buckets is doubling (100%).
• Change in egress pricing. If I’m reading it correctly and the existing exclusions stay in place, it looks like a pretty large decrease in bucket egress for most uses. Previously 10TB of bucket egress (direct to the internet) would be charged at $0.11/GB to most of the world. Now that will be $0.02/GB from US buckets to the US, $0.05/GB to Europe, and $0.08/GB to Asia. This is a pretty large decrease, and I’m surprised Google didn’t make more of it in the announcement.
• Changes to data replication pricing from $0.00 GB to $0.02/GB in us, nam4, eu, and eur4, and to $0.08/GB in asia and asia1. (These are the dual-region and multi-region bucket locations).
• Reading data from a multi-region bucket in the same continent will no longer be free, it will now be charged at the same rate as data replication. This will make many data-heavy workloads unviable with multi-region buckets.

The last two changes in particular make dual-region and multi-region buckets a much less attractive offer and invalidate a lot of architectural assumptions developers will have made. Businesses that have built a high-availability architecture around multi-region buckets are now faced with the unappealing options of:

1. Pay a lot more for replication + reading from their buckets.
2. Rearchitect their application and migrate to a single region, or perhaps a dual-region if they can make the data-replication pricing work.

Load balancing

Google is also increasing prices on Cloud Load Balancing, adding a $0.008-$0.012/GB charge for outbound data processing.

Starting October 1, 2022, we’ll apply an outbound data processing charge of $0.008 - $0.012 per GB (based on region) to all Cloud Load Balancing products in order to maintain consistency and alignment with the variable costs of the services across our Cloud Load Balancing portfolio.

Depending on your egress rates and network tier, this will amount to a 5-10% increase on internet egress, and an added cost for most kinds of internal load balancing.

Persistent disks

Persistent disk snapshot pricing is also changing. I suspect this won’t be as big a deal for most customers as the previous changes.

• Regional snapshots are increasing from $0.026 to $0.050 (92% increase).
• Multi-region snapshots are increasing from $0.026 to $0.065 (150% increase).

The price changes mention a “us-central1 baseline” and “United States multi-region baseline”. This makes me think that prices in all other regions will be going up by an equivalent percentage.

Google also plans to introduce an archive snapshot tier later this year which will be priced at $0.019/GB/month, but with a minimum billing period of 90 days.

As with Cloud Storage, previously free data transfer for creation and restoration of multi-region snapshots will now be charged at inter-region rates. Restoring a 100GB multi-region snapshot will now cost $1 in the US and Europe, and more in other regions. Depending on your architecture, these charges could add up!

Why now?

On March 7th, 2022 The Information published Why AWS Makes Money and Google Cloud Doesn’t

[..] In a positive sign, Google Cloud CEO Thomas Kurian last month told colleagues during an internal all-hands meeting that he expects the cloud unit to be profitable later this year, according to a person who viewed the event.

Another issue is that Google Cloud has fewer high-margin services—such as cloud database and analytics software—to sell to customers than AWS does. […] Former employees say AWS relies on these offerings for a large portion of its operating profit.

The Information doesn’t mention it here, but the highest margin product AWS sells is bandwidth. Charging for data transfer is as close as you can get to pure profit in the cloud. As Cloudflare has previously demonstrated, AWS likely marks up egress bandwidth by up to 8,000%, and it wouldn’t surprise me if inter-AZ and inter-region charges were also in that vicinity.

If Google Cloud wants to get to profitability by the end of the year, then introducing data transfer pricing makes a lot of sense. The extra revenue from data transfer will be nearly 100% profit and go straight to Google Cloud’s bottom line. It will be difficult for existing customers to rearchitect their applications, and most customers will probably just pay the extra charges.

However, if Google Cloud is trying to build the number two cloud business behind AWS, this move is an unforced error that will damage their credibility for years.

Killed By Google

Google has developed a reputation for killing consumer products, and even Google Cloud has made several price increases which have changed architecture invariants, along with numerous deprecations:

• 2011 - Increasing App Engine pricing by 2x-10x
• 2018 - Increasing Google Maps API pricing by up to 1400%
• 2020 - Reintroducing management fees for Kubernetes clusters
• 2020 - Google Hire shut down

These changes have damaged Google’s credibility for many customers and undermined their commitment to building GCP to be an equal to AWS and Azure. Google has been trying to build customer trust with their Enterprise API stability promise, and signing many 10-year deals with large enterprises like Deutsche Bank, Mayo Clinic, and Sabre.

However the last few months have seen several shortsighted decisions:

1. Charging for personal Google Workspace accounts (Google Workspace comes under Google Cloud). Many of the people affected by this will be decision makers for Google Cloud purchasing decisions, and not people you want to remind about Google’s reputation for shutting down products.
2. Laying off some Cloud technical support staff and outsourcing support to third-party vendors.
3. Last week’s Cloud price increases.

Put together, it paints a picture of an organisation myopically focused on short-term profitability over long-term strategy. These changes show they don’t understand customer perceptions of Google Cloud, and the “rules” of cloud pricing.

These changes are hard to even understand as a business strategy. Raising prices on locked-in customers feels like a move you’d see from Oracle, not from the third-ranked competitor in a rapidly expanding market.

My best guess as to why Google is doing this now is that Google Cloud set (or had set for them) an OKR to reach profitability. All of the teams are trying to increase profits, regardless of the long-term costs.

A different future

Google Cloud’s strength has always been its technology. It has fewer products than AWS, but those products are more flexible and cohesive. Their IAM model is far simpler than AWS’, and as previously outlined, Google’s global infrastructure has offered many unique, differentiated products.

At a time when it’s hard to even keep track of Azure’s many multi-tenant security vulnerabilities, and multiple AWS outages still in recent memory, Google Cloud should have been pressing their advantages.

Instead of raising prices, I think a better strategy would have been to cut prices on data transfer to nearby regions and lean into multi-region services. Google owns massive amounts of dark-fibre and under-sea cables connecting their data centres and could afford to lower their prices. Once inter-region transfer is cheaper, many more applications become feasible to run in high-availability, multi-region configurations. This would be a strong offering to sell, and one that is hard for Amazon and Microsoft to replicate. Customers running compute in multiple regions and using higher-margin services like Spanner would help make back some of the revenue lost from lowering inter-region data transfer.

This would have been a compelling, differentiated offering, playing to Google’s strengths. Instead, Google has altered the deal. Their customers will now be praying that they don’t alter it further.

These are my thoughts after a few months use. I’ve split the review into DisplayLink and ALogic DX3 sections. Most of my complaints are about DisplayLink and I think would be the same no matter which dock/USB adapter you choose.

Alogic DX3

• The DX3 has 3 DisplayPort ports, but only two of them are DisplayLink compatible. The third one uses Alternate Mode (I think?) over USB-C to pass the DisplayPort signal directly to the monitor. This was good for me as I could make my center display DisplayPort and use DisplayLink on the peripheral display.
• The dock comes with a hefty external power supply almost the size and weight of the dock. The dock has a power button on the front which I thought was unnecessary at first but came in handy as I often need to restart the dock for the Mac to detect one or both of the displays. I’ve had this issue on a 2019 Intel Macbook Pro with directly connected DisplayPort cables so I suspect this may be more of a macOS problem. Having the power button on the front is much more convenient than unplugging the displays or turning the displays off and on.
• I haven’t had any issues with the USB ports. I haven’t pushed them very hard, but I also haven’t noticed anything “wonky” about them either. I have a 10-port USB hub that I plug into the dock and it seems to work fine including for an audio interface.

DisplayLink

• With a 4K display, macOS will let you choose a higher resolution to render at, which it will then downscale to the 4K pixels that are actually on the display. When I first bought the DX3, DisplayLink version 1.4 wouldn’t allow you to do this, the maximum resolution you could pick was 1920x1080 points at 2k (native 4K resolution). When writing this article, I went back to the feature request and found that they had released version 1.5 which was meant to fix this issue. It lets you choose 2560x1440 points but also renders at 2560x1440 pixels, instead of 5120x2880 pixels, downscaled to 4K. The good news is that it looks like they are continuing to work on the software and addressing limitations so hopefully this will be fixed in a future update.
• DisplayLink 1.4 was doing something which prevented you from using Apple Watch to unlock your computer. I discovered that it also stopped you from using your Watch as an admin password. However, the 1.5 update has a checkbox to “Use Apple Watch to unlock on login screen”. When you check that, the Watch unlock appears to work at the login screen and in everywhere else Watch unlock would work. I’m not sure why that wouldn’t just be a default setting?
• Note that you can’t evaluate DisplayLink CPU usage just by looking at the “DisplayLinkUserAgent” process. You also need to look at the WindowServer process, as DisplayLink causes work to be done there when rendering to the virtual display.
• The DisplayLink software could do with more diagnostics on what is happening. Currently you can’t see which displays are controlled by DisplayLink and which are directly connected.
• The DisplayLink software doesn’t seem to have any way to automatically update or even notify you of updates so you’ll need to check the website yourself.
• I haven’t noticed any latency from the compression/decompression signal transfer, but I’m only using the displays for writing code and web browsing so I wouldn’t expect many problems.

Other

• I’ve had a weird bug where sometimes the MacBook display won’t turn on for 10-60 seconds after opening the laptop, even when I’ve been using it only a few minutes prior. I’m not sure if this is related to DisplayLink software or just a macOS bug.

Summary

Overall, the ALogic DX3 dock serves it’s purpose well. If you had any other option, I wouldn’t recommend choosing DisplayLink technology, but with an M1 Mac this your best (and only) option. However, I’m encouraged that DisplayLink are continuing to improve the Mac user experience. When I upgrade to a new MacBook Pro, I’ll either use the DX3 without DisplayLink (if that will work), or sell it and purchase a standard Thunderbolt 4 dock.

I needed to buy a new Mac recently. I couldn’t bring myself to buy a legacy Intel Mac, but the new M1 Macs only support up to two displays. For the MacBook Air, MacBook Pro, and iMac that means you can connect one additional display along with the built-in display. For the Mac Mini, you can add two displays. I wanted a MacBook Air and already had two external displays which I wanted to keep using.

Update: I ended up purchasing the ALogic DX3 and wrote a follow-up post about my experiences with it.

DisplayLink

What are your options if you want to run more than two displays? Enter DisplayLink. DisplayLink (not to be confused with DisplayPort) is the name of a technology created by a company also named DisplayLink. It lets you send a video signal to a display over USB or Wi-Fi instead of via DisplayPort or HDMI. DisplayLink technology was first used in laptop docking stations but can now be found in other video-related products including adapter cables and monitors.

DisplayLink has two components - a software driver installed on your computer and a hardware chip in the dock or adapter. The software driver presents itself as one or more displays to the computer. The computer sends pixel data to the software driver, which then compresses the data and sends it over USB. The DisplayLink chip decompresses the data and sends the display signal to the (real) display.

I was pleasantly surprised to see that DisplayLink’s drivers have been updated for Apple Silicon and don’t require a kernel extension. From what I can tell, DisplayLink seems to be publishing regular updates to their drivers and is actively working on new features.

DisplayLink Limitations

Sending compressed video over USB is a pretty neat trick! However, DisplayLink has quite a few downsides to be aware of.

An uncompressed 4K60 video signal requires 12 Gbps of bandwidth. The latest DisplayLink chips run over USB 3.0 which has a bandwidth of 5 Gbps. Even a single 4K signal is too big, let alone multiple displays. This is why DisplayLink needs to compress the captured screen.

For desktop browsing, email, software development, and other general computer tasks where the display doesn’t change much from frame to frame, DisplayLink compression can be very efficient and is unlikely to be noticeable. DisplayLink claims “Pixel-perfect graphics”, and “Compression tuned for video content and high-quality graphics” on their newer chipsets. If you’re a video/photo/graphics professional, you’d probably want to check and see if this works for you.

It doesn’t sound like it will be as good for gaming though:

While DisplayLink technology is targeted primarily to productivity and video applications, it is suitable for the casual gamer. If you’re a “power gamer”, looking for every edge possible over your opponents, you might want to go another route.

Another downside to DisplayLink is that you need to install an optional program to show the Lock Screen on a DisplayLink monitor. Otherwise, the Lock Screen will only show up on the built-in display.

Unlock with Apple Watch doesn’t work when using DisplayLink. Apple disables Unlock with Apple Watch with the message “Unlocking with Apple Watch is not available while your screen is being shared”.

DisplayLink has a few other limitations on Macs: HDCP is not supported so you may not be able to watch Netflix, Hulu, iTunes Movies, etc.; only 4 external displays are supported; screen rotation is not supported; Clamshell mode is partially supported. Manufacturers publish blog posts announcing that new DisplayLink drivers are validated with their hardware, it’s probably good to check with your manufacturer before updating.

There are quite a few limitations to using DisplayLink to extend your displays. However, it’s also currently the only way to run more than two displays on a Mac, so if you need that then you’ll need to look at a DisplayLink powered Dock.

Choosing a Dock

If you already have a dock without DisplayLink, your cheapest option may be to purchase a DisplayLink USB adapter. You could connect your monitor to the adapter, and then the adapter to a free USB port. DisplayLink keeps a list of USB adapters you can purchase. Check compatibility as not all adapters list support for macOS, although sometimes the reviews will mention that they do still work.

DisplayLink sells chipsets to dock manufacturers. The chipset has their DisplayLink hardware, along with other ports like USB, Ethernet, Audio. This means that products from different manufacturers like Targus, Dell, Alogic, Plugable, Kensington can end up with very similar capabilities because they are all using the same electronics.

Unfortunately, all of the M1 compatible docks I saw had a maximum of one USB-C port. If you have a lot of USB-C devices you’ll either need to purchase USB-A->USB-C adapters or buy a non-DisplayLink dock and a DisplayLink adapter.

If you’re investigating docks, you may see reference to USB-C Alternate Mode. This is a way for a USB-C port to run non-USB signals over it like DisplayPort and Thunderbolt(!). Alternate Mode doesn’t let you run any more displays than you could otherwise do natively.

Dock Options

If you’re looking for a list of DisplayLink docks, DisplayLink has a list of DisplayLink docks. Here’s a list of docks that the manufacturers have said are supported on M1 Macs as a starting point:

Plugable has a few options:

• UD-3900PDZ, $160 - 3 HDMI ports, 1 supports 4K30, the other 2 only support 1080p/60Hz. 60W Power Delivery, 6 USB-A 3.0 ports, 3.5mm audio jack.
• UD-ULTC4k, $220 on Amazon - 2 DisplayPort ports that support 4K60 over DisplayLink, 1 HDMI port that supports 4K30. 60W Power Delivery, 4 USB-A 3.0 ports, 1 downstream USB Type-C port, 3.5mm audio jack.

Note that Plugable mentioned that their Ethernet ports are limited to 300Mbps on macOS due to driver issues. I’m unsure if that extends to all DisplayLink docks.

Alogic also has a few options:

• DX3, $250 on Amazon - 3 DisplayPort ports with DisplayLink, 100W Power Delivery, 3 USB-A ports, 1 USB-C port, Micro SD and SD card reader, Gigabit Ethernet port, 3.5mm audio jack.
• DX2 - 2X DisplayPort ports, 65W Power Delivery, 3 USB-A ports, 1 USB-C port, Micro SD and SD card reader, Gigabit Ethernet port, 3.5mm audio jack.
• Universal Twin HD

Targus has several docks with DisplayLink:

• DOCK190, $310 - 2 DisplayPort/HDMI ports (only 2 video ports can be used at the same time), 100W Power Delivery, 4 USB-A, 1 USB-C, Gigabit Ethernet, 3.5mm audio jack. Thunderbolt alt mode is supported.
• DOCK570, $415 - 4 DisplayPort/HDMI Ports, 4 USB-A, 1 USB-C, Gigabit Ethernet, 3.5mm audio jack. Thunderbolt alt mode is supported.

Kensington has four docks with DisplayLink:

• SD4700P, $250 - 1 DisplayPort, 1 HDMI port (both limited to 2K resolution), 60W Power Delivery, 5 USB-A, 1 USB-C, 1 Gigabit Ethernet, 3.5mm audio jack. Supports up to 5Gbps.
• SD4750P, $290 - 2 DisplayPort/HDMI ports up to 4K60, 85W Power Delivery, 5 USB-A, 1 USB-C, 1 Gigabit Ethernet, 3.5mm audio jack. Supports up to 5Gbps.
• SD4780P - 2 DisplayPort/HDMI ports up to 4K60, 100W Power Delivery, 5 USB-A, 1 USB-C, 1 Gigabit Ethernet, 3.5mm audio jack. Supports up to 10Gbps.
• SD4900P $320 - 3 DisplayPort/HDMI ports up to 4K60, 60W Power Delivery, 5 USB-A, 1 USB-C, 1 Gigabit Ethernet, 3.5mm audio jack. Supports up to 10Gbps.

StarTech has 4 docks with DisplayLink:

• DK30CH2DEP - 2 DisplayPort (4K60), 1 HDMI port (4K30), 100W Power Delivery, 4 USB-A, 1 USB-C, 1 Gigabit Ethernet, 3.5mm audio jack.
• DK30C2DPEP - 2 DisplayPort/HDMI (4K60), 100W Power Delivery, 3 USB-A, 1 USB-C, 1 Gigabit Ethernet, 3.5mm audio jack.
• DK30C2DPPD - 2 DisplayPort/HDMI (4K60), 60W Power Delivery, 3 USB-A, 1 USB-C, 1 Gigabit Ethernet, 3.5mm audio jack.
• DK30CHDDPPD - 1 DisplayPort (4K30), 1 HDMI (4K30), 60W Power Delivery, 4 USB-A, 1 USB-C, SD Card slot, 1 Gigabit Ethernet, 3.5mm audio jack.

Dell has some docks with DisplayLink technology, but they don’t list compatibility with Macs, so probably safest to avoid them unless you’re able to try them first.

Caldigit docks are highly regarded but don’t contain DisplayLink chips. This means you can’t run multiple displays on them. However, they seem to work if you only need a single external display on an M1 Mac, or purchase a USB display adapter. There is a video of someone using a Caldigit TS3 Thunderbolt dock with 4 DisplayLink USB adapters.

Disclaimer

I’m not an expert on any of this, I’ve just read a lot and compiled it into one place. Please let me know if I’ve missed something or if you have any suggestions.

Summary

There are lots of moving parts involved with picking a USB-C dock, let alone one also running DisplayLink. You need one which:

• Supports DisplayLink.
• Supports the number of displays you have, at the resolution and refresh rate you want.
• Has enough USB ports to support other peripherals, or is reliable enough to add a USB hub.
• Supports USB Power Delivery with your laptop, with enough wattage.
• Has enough bandwidth for all your devices.
• Doesn’t have audible fan noise.
• Works reliably and doesn’t cause other issues.

As you can tell from the number of limitations mentioned throughout this post, picking a dock that works with your setup is far from a simple task. I recommend checking the specs closely, checking out reviews, and ideally trying it out at home before committing.

You can also see the talk on Youtube.

Good morning. I was going to ask if everyone’s feeling awake, but I think after that karaoke, I think everyone’s pretty well awake by now.

This talk is called Building Stable Foundations, and it’s about building stable long lasting foundations for the Clojure community.

[00:00:30] Bozhidar has already introduced me slightly, but I’ll go back over it. My name’s Daniel Compton, I’m an open-source project maintainer. I am an administrator at Clojars. I write and record a Clojure podcast and newsletter at therepl.net, and I am working at a startup called Falcon.

Falcon is hiring for Clojure developers, remote Clojure developers. If you’re sort of in the U.S.-ish region, or you can work [00:01:00] on a similar kind of time zone, check out falconproj.com [Ed: Falcon shut down 😢] or come and talk to me, and I can tell you more about what we’re working on.

The other thing that you may know me from is an organization called Clojurists Together. With the help of our members, we fund and support open-source Clojure projects. The question is, well, why is that important? Firstly, who has heard of Clojurists Together?

Great. All right. Just about everyone, and who [00:01:30] here is a member or your company is a member of Clojurists Together. Great. The reason why Clojurists Together is important, or I think it’s important, is that Clojure is foundational, and open-source Clojure is foundational to what we work on. Of course, the Clojure language is open-source, but the vast majority of tools, and libraries, and services, that we all use as day-to-day Clojure programmers come from open-source software.

It’s important, not just for [00:02:00] us as a community, but open-source Clojure is really foundational for businesses too. Businesses are investing lots and lots of money into the Clojure ecosystem, and they’re relying on these investments lasting a long amount of time. I think it’s really important to sort of remember that context of the money being invested into Clojure and making sure that that is well-spent and it’s going to be [00:02:30] a long lasting investment.

Who here is lucky enough to get paid to work with Clojure? All right. Of those of you who had your hands up, who uses open-source Clojure code in your day job? It’s a bit of a silly question, isn’t it? That the idea that as a working Clojure programmer you wouldn’t be using a significant amount of open-source code as part of your work. Maybe there are some people. I suspect there are a couple, but for most of us a lot of the work we do is built on top of these really important open-source foundations. Open-source creates an incredible amount of value for everybody that gets to use it. It’s the shared resource that doesn’t run out when somebody uses it, that doesn’t mean that anyone else can use less of it. This is a really amazing property of software and open-source software.

But a few years ago, I started to notice [00:03:30] developers talking about burning out. In the Clojure community and also outside of the Clojure community, and this was kind of worrying to me, because a lot of these people were really important to the Clojure community. They were doing really important work.

I started to see these kinds of things being said. People saying things like, “Any tips for a post burnout return to open-source? I’m exhausted, but want to keep going.”

This [00:04:00] really broke my heart to see these people who I cared about, many friends and that they were not feeling supported. They were thinking of leaving, and I was really worried about this. We have this problem that the foundations of our community are perhaps not as strong as they could be, or they’re not being as supported as they could be.

Before we talk more about our open-source foundations, I think we can learn some things from building [00:04:30] foundations. This is what a building site looks like before you start building on it itself. The first thing you have to do when you’re building is to prepare these foundations, everything else goes on top of them. So you have to get this right first.

When you’re building foundations, there’s a few things you need to do. The first thing you need to do is survey the land. You need to find out, do you have the rights to work on this land that you think you own and is the structure that you’re going to [00:05:00] be building allowed to be built on this land. This is really important to get right, because if you get it wrong, you might invest all of this money into building something which you then need to take back down again.

Next thing you need to do is check under the ground. What is going on underneath the surface? You can see what’s on top of the ground, but it’s really important to find out what’s going on underneath there before you start building, so that you don’t end up being surprised by things later on. The last thing, and this is the most important thing and [00:05:30] what we most often think of when we’re thinking of building foundations is reinforcing the ground. A fully built house can weigh 50 to a hundred tons or more, and that’s a lot of weight to be putting into the ground, and so you really want to make sure that those foundations are going to be able to hold up to that kind of weight.

This is quite similar to what we do when we’re picking dependencies. The first thing we do is we’ve got to check the license. I hope we check the license and [00:06:00] we also will usually want to review the codebase. Is it well-written Clojure? Does it have tests? Is it actively maintained? These are all important things to be aware of before you pick a library, and the last thing, if this is going to be used in the critical path of your production software, you probably want to test it, find out how fast is it. Can it stand up to the full weight of production? Are there memory leaks, that sort of thing?

Foundations are really important [00:06:30] to building a house, but it turns out that they’re actually pretty cheap. In the U.S. on a $400,000 house build. They cost about $10,000, two and a half percent, so not really that much of the cost of the build. This is because it’s a very well understood process. It’s been done many times every day, and for the most part, it goes pretty smoothly.

Has anyone ever seen a house like this before? [00:07:00] What’s happening here is that the foundation on this house wasn’t built correctly the first time. It’s a little bit hard to tell, but this house has been lifted right up off the ground, quite a few meters up. Stacked up on these Jenga blocks so they could fit machinery underneath to repair the foundations, and then they’re going to slowly lower that house back down.

If you look closely, I’m not sure if it’s going to come through very well here, but there are some people standing underneath this house, [00:07:30] very brave people. Does anyone relate to this picture? Has anyone had to fix the foundational dependency in your codebase and feel like there’s a lot of things bearing down on top of you? As you can imagine, this is not a cheap process. It can cost $20,000 to $100,000 to do this kind of repair, which is two to 10 times more than the cost of building the foundations correctly in the first place

[00:08:00] Yesterday, Tiago talked to us about resilience, and I’m glad he really dug into this, because I don’t need to go too deeply, but we’ll remember resilience as the ability to recover from a stress or a change. This is what you’re going for when you’re building foundations, you want the foundations to be able to stand up to the elements and all of the things that a house is going to face over a hundred year lifetime. What we can see here is there’s two houses, on the left… If you look at them, superficially they kind of look the same. They’re [00:08:30] both held up by wooden poles. They’ve both got thatch kind of roofs. They’re both orangy. They’re both roughly the same size, but the one on the left is not very resilient at all, and the one on the right is incredibly resilient, even though it’s over the water.

This house on the right is, well, it’s a community. Firstly, it’s a community of houses. It’s not just a single house, and it’s also held up by many stilts, so that even if you lost one, or two or even a dozen of these stilts, [00:09:00] the whole structure would still be sound. Whereas on the left, you’ve got just two poles holding this house up. And if one of those two goes down, the whole thing is coming down. If you agree with me that open-source software is the foundation of the Clojure community, then we need to be asking this question, how resilient are our foundations? A few years ago, as I was looking around the Clojure community, it seemed to me that we were perhaps closer to the left-hand side than [00:09:30] the right-hand side.

We had a few people doing a tremendous amount of work for the Clojure community. People were depending on their work every day, they were using it. It was really important, but they were feeling unappreciated, perhaps burnt out, and there was this refrain of, “I’m exhausted, but want to keep going.” I thought they were at severe risk of burning out. If we lost these people, this would be somewhat equivalent to that house we saw, having to be lifted up on the blocks, [00:10:00] and there would be a huge investment. We would lose a huge amount if we lost some of these people, and it would cost us a lot of time and effort to regain that knowledge and hard won wisdom.

This led me and others to create this organization called Clojurists Together. We have a very simple mission. We want to fund and support open-source software, infrastructure, and documentation that’s important to the Clojure and Clojure script community.

We [00:10:30] fund three kinds of projects. The first kind of project is maintenance projects, and these are projects that are often not very interesting to fund. Maybe they’re very stable, used by lots of people, but over time, software needs maintenance, and if people aren’t being paid, or aren’t getting dedicated time to work on it these maintenance things can pile up over time. So this is a place where I think Clojurists Together is really well suited to help these projects, [00:11:00] because it’s work that sometimes wouldn’t get done otherwise, or it would take a lot longer. We really liked being able to fund these kinds of low-level boring kind of work, but these are really important improvements to make.

The next kind of work we like to fund is new development. This might be a project that is already well understood and used by many people, and they have an idea for a new feature, or a new release, or something that they would like to do, and they need some dedicated [00:11:30] time to work on it. We also take and fund projects who do this kind of work.

The last kind of project is new fledgling of projects, and we fund a few of these kinds of projects where someone comes to us with a seed of an idea, and perhaps they’ve already created proved out that this is an idea, but they would like some funding to go to the next level with it, and we’ve been able to fund some of these projects, and they have [00:12:00] really surpassed my expectations. You’ve done amazing work, so this is another kind of project we like to fund.

We launched in October 2017, and in 2018 and 2019, we’ve funded 11 projects.

• We’ve funded Bozhidar on CIDER.
• We funded Tim Pope to work on Fireplace, which is a Vim Clojure plugin.
• We funded Arne Brasseur, also here, to work on Kaocha, which is a next generation test runner for Clojure.
• We funded Mike Fikes to work [00:12:30] on ClojureScript and the ClojureScript compiler.
• We funded Nikita Prokotov to work on Datascript.
• We funded Thomas Heller to work on Shadow CLJS, which is a ClojureScript and JavaScript build tool.
• We funded Lee Hinman to work on clj-http. This is a HTTP client that’s used by many projects directly. I think it’s one of the most downloaded projects on Clojars, but it’s also used as the foundation for many of these other API clients. You may not even be aware that you’re using it, [00:13:00] but improvements to clj-http benefit a large part of the Clojure community.
• We also funded Aleph. This is a Netty based web server for Clojure.
• We funded Dragan Djuric to work on Neanderthal, and then he worked on documentation for Neanderthal and wrote 150 pages of documentation about deep learning and Clojure from scratch. It’s a really interesting read if you’ve been wanting to understand deep learning. [00:13:30] I’d really recommend checking that out. It’s really good.
• We also funded Martin Klepsch to work on cljdoc. Cljdoc automatically builds the documentation for you every time you push to Clojars. This is really nice, because it means that the maintainer of the project doesn’t need to remember to update the API documentation every time they do a deploy. It just happens automatically, and you can look back at older versions of the API documentation, and there’s all sorts of other good things that have come from [00:14:00] having a shared platform that we can all use to improve.
• The last project we funded was Bruce Hauman to work on Figwheel. Figwheel is a really key tool for the ClojureScript community. It’s had a lot of influence outside of the Clojure community as well. It’s been cited by many other environments. I’ve seen its influence, and the influence of things that came before it too, spreading across the entire community. Live reloading is one of ClojureScript’s [00:14:30] secret weapons, or perhaps not so secret anymore.

Who in the audience has used one of these projects? All right. What about three projects. Who’s used three or more of these projects? Wow. Okay. Who’s used five or more of these projects? Great. All right. Is there anybody here who has not [00:15:00] used a single one of these projects?

All right. Well, that’s really good to see. [Nobody put their hand up]. I’m glad that this has benefited people.

I’m really excited to be able to announce today, this Heart of Clojure conference has been timed really well to be able to announce on stage the next funding round. We’re going to be funding four projects, $9,000 over the next three months.

The first project we are funding [00:15:30] is Calva. Calva is a VS Code extension for Clojure. VS Code has been gaining in popularity hugely over the last few years. I think it’s one of the sort of surprise sleeper hits of the programming community, so we’re really excited to be able to fund Calva.

We’re also funding Thomas Heller to work on Shadow CLJS again. He’s been doing incredible work for the Clojure and ClojureScript community on Shadow CLJS. I hear lots of people talking about it and using [00:16:00] it, and it’s a really great tool, and those improvements are benefiting lots of people.

Another project we are funding is Meander. Meander is a really interesting Clojure, ClojureScript, data transformation library. I probably can’t do it justice here, but Michiel Borkent [Ed: this was Timothy Pratley, sorry Michiel!] wrote a really great blog post about this, so if you find him here at the conference, ask him to tell you more about Meander, because it’s a really cool project, transforming data and maps, which [00:16:30] we do every day.

The last project is CIDER. I would say CIDER needs no introduction as well, because many, many people use CIDER and many more people also use Bozhidar’s work on the Orchard and in REPL and all of these other foundational things that sit beneath CIDER that we as a community get to take advantage of. I’m really excited to be able to fund these four projects, thanks to the [00:17:00] support of our members.

The way Clojurists Together works is that we have members sign up throughout the year, and then every quarter we will go to them and say, “Hey, what do you think we should be working on? What’s interesting to you, what do you think is useful? What areas do you think need support?” Then we take that information and we create a call for proposals. We say to the wider Clojure community, “Hey, here’s what our members would like us to fund. Please [00:17:30] give us some proposals,” and every quarter we get an amazing amount of proposals, more than we could pick. The quality of these submissions is really high. Then the committee members vote on the projects that we want to fund, and then we’ll fund those projects for three months. At the end of those three months, we turn around and do it all again, so it’s pretty straightforward.

Clojurists Together has been getting noticed and has influenced beyond [00:18:00] just the Clojure community. This is my friend, Devon, who works at GitHub, and she’s talking at GitHub Satellite earlier this year about GitHub Sponsors. Clojurists Together was mentioned as an influence on GitHub Sponsors. GitHub Sponsors is a project for where you can fund directly the projects that you use on GitHub. I’m really excited to see this. I think GitHub has a really wide reach, clearly, in the development community, and they’re helping to [00:18:30] change attitudes around open-source funding for both, in the Clojure community, but also, in the wider community. This is something I’m really looking forward to.

Devon has this quote, “Our goal is for open-source to be a serious career path people can set up.”

This is kind of the, I think, the next frontier for the Clojure community too. That we have a bunch of projects that are really important, that people are using, and I would really like to see [00:19:00] people be able to be funded to work full-time or nearly full-time on these projects for the good of the community. Clojurists Together, can’t do it all. We’re set up to do a particular kind of funding, and so what I’d like to see is that the community as a whole starts to build this open-source middle class of single or small teams being funded in a meaningful way, and being appreciated by the community so that they can do this [00:19:30] work that benefits everybody.

Part of this is that we’ve just launched a page on our site called Beyond Clojurists Together, where we are collecting Clojure projects and Clojure programmers who are accepting money from Patreon, Open Collective, GitHub Sponsors, or whatever other tools they have, to be funded directly. If you’re interested in funding projects and you don’t really know where to start, this could be a really good place for you to go start looking and finding [00:20:00] some of the dependencies that maybe you or your company is using to fund them directly.

How do we get there? How do we get to this glorious future where all of our dependencies are supported and maintained? It needs two things, time and money. Sometimes people and companies have capabilities for one or the other. We have a lot of people investing, people in companies too, investing in open-source [00:20:30] projects, open-sourcing internal things or maintaining projects. Other times companies don’t really have the time or the capacity to be directly working on open-source projects, but they have some money available that they could put into maintaining them instead.

The other side of the equation is money, and companies spend money on lots of things. They spend it on our salaries, software licenses, cloud hosting, hopefully [00:21:00] green cloud hosting after that talk yesterday, and offices and snacks.

Companies are used to spending money on lots of things, but I think open-source projects often aren’t shaped very well for companies to give money to, and companies need these two things. They need value. They need to be able to say to the decision-makers, to the finance department, “We are getting something of value when we are giving money to this project.” That doesn’t mean [00:21:30] that you need to sell out to the man. There’s lots of different ways that you can provide value to projects, to companies that are both valuable to the companies, that fit with the skills and capacity of what you have, but most importantly, still meet the values that you hold as a person and the values of your project.

This could be things like consulting, feature development, long-term support, maintenance, [00:22:00] and these are things that oftentimes you may be doing anyway, but if you’re able to sell it, and sell this to companies in a way that they are able to understand, and sell onto their decision holders, this can be a really powerful thing.

The other thing that companies need is invoices, and not just invoices, but all of the other financial infrastructure that goes along with things. Companies are not used to using PayPal [00:22:30] donation links to pay for their office rent. They pay money to a bank account, and so this is something where open-source projects, again, aren’t often shaped very well to accept this money.

This is where I’m really excited to see platforms like Open Collective, Patreon, GitHub Sponsors who are providing the shared common infrastructure for projects to be able to work, to be able to provide the kinds of things that companies [00:23:00] need to be able to give money to.

The question now is, are we investing in stable foundations? Are we building stable, resilient foundations that are going to last us the test of time? To answer that question, we need to sort of answer, well, compared to what? The main input to open-source projects is labor, people spending their time on it. I thought perhaps a good comparison to [00:23:30] this will be, how much are we spending on Clojure developer salaries and businesses. To answer that we need two numbers. We need how many Clojures developers there are, and how much they’re getting paid.

Estimates vary from perhaps 20,000 to 50,000 working Clojure developers. We want to be conservative here, so we will just take 20,000, and then we need to figure out how much they’re getting paid. The Stack Overflow Survey for 2019, put the average Clojure developer salary, worldwide, [00:24:00] at 90,000 U.S. dollars. Clojure’s been at the top for the last three years, which is pretty good, but I’ve heard people quibble with this number, say that maybe it’s a bit inflated or unrepresentative in some way, so let’s just take it back a bit more to still be really conservative, and we would just say $80,000 a year.

If you multiply these two numbers together, you end up with $1.6 billion a year being spent on Clojure developers. This doesn’t even include all [00:24:30] of the other expenses that go around Clojure that you need to run a business, and it doesn’t even include the full cost of hiring an employee, but I think this is a really good number to compare to and to think about.

If we were to spend a relatively small percentage of what we spend on Clojure developers, on supporting the foundations, that could be a meaningful amount of money, say two and a half percent of 1.6 billion would be $ [00:25:00] 40 million a year. I think, I can’t even really imagine quite what that would look like. It’s just such a long way away from where we are currently that it would… Yeah, I can’t imagine quite that what that would look like, but what I’ve seen from funding projects, even at a very small level far below this, is that the return on investment there has been incredible.

I think we would see some really exciting things start to happen, even more than they are currently, [00:25:30] if we were to be able to even approach this level of funding, which itself is, I think, very small compared to the value that we as a community get from these open-source foundations.

We come back to this image of the houses on the water, and this image really appeals to me, because I see some similarities here with the Clojure community. These are not always the most pretty houses, some of [00:26:00] those stilts down below look a little bit rickety, but together as a whole, these come together to form a really solid stable foundation. This is what I’m hoping that we can build together for the Clojure community, and to be clear, this is already happening, well before Clojurists Together. This is still a really strong community before this, and I’m excited to see kind of where we can go in the future.

If this idea appeals to you, here’s some things that you could [00:26:30] do.

• First thing is support your dependencies. You can go have a look at that page on Clojurists Together, Beyond Clojurists Together, where you can find some projects that you are using, which you might not even be aware. We’re accepting donations.
• Another thing you can do is commit engineering time to maintaining the dependencies that you work on. This is a really key thing to do, and I know many companies already do this, so I really want to recognize that and thank them for their work they’re already doing here.
• Another interesting [00:27:00] thing you can do is ask when you are next looking for work, ask the companies that you’re talking to, do you support in some way, the open-source dependencies that you use? I think this is a really interesting question, because sort of below just your funding open-source, I think the primary barrier to that is changing mindset and expectations around what [00:27:30] it means to use open-source, and the potential benefits for the community as a whole to fund it. If you ask companies, when you’re looking for work, “Do you do this?” Companies are looking to stand out to you, and so I think we might start to see the answer more and more become, “Yes.”
• Another thing you can do is stand for Clojurists Together elections. We have a board of seven people and every year in November, we have elections, so if you would like to be directly involved in helping us plan where we [00:28:00] go into the future you can stand for these elections.
• The last thing you can do is join Clojurists Together. I see many people already in the room are Clojurists Together members, and I really appreciate that. You can sign up as a company, or a developer, as an individual. The choice is yours there.

I want to acknowledge many people here. Firstly, Heart of Clojure, Ana and Martin, they’ve put on a really great conference here, so I really appreciate them bringing me along to speak.

[00:28:30] I’d also like to thank Ruby Together, and André Arko in particular. Ruby Together is the organization that we modeled ourselves afterwards. They’ve done a lot of trailblazing work here and André Arko, in particular, has given us a lot of time so that we can learn from some of the things that they’ve done.

I’d also like to thank the Software Freedom Conservancy. I saw someone here had a [00:29:00] Software Freedom Conservancy shirt on yesterday. The Software Freedom Conservancy, if you’re not aware is our non-profit parent. They handle all of the accounting, and payments, and financial, and legal things that go on that companies need, and that we need to have happen, but that we can rely on them to provide for us. They’ve done an amazing amount of work for us, and so I really appreciate them.

Next group of people I’d like to think is [00:29:30] the projects that have applied. We have more projects apply every quarter than we could fund, and as I read over the applications, I’m really excited, because there’s so many good projects out there, and so I’d like to thank everyone who has applied.

I’d also like to thank the projects that we funded. Sometimes today in the talk, I might’ve slipped and said that we did this work and I wasn’t programming on CIDER, Bozhidar was, so I’d like to thank the projects that we funded, that have done this work. I know that often they’ve taken [00:30:00] time off paid work, or taken holiday time. While we’ve been able to fund them to a certain level, I know often that still means a pay cut for them to be able to do this work, so I really appreciate that.

Also, I’d like to thank the rest of the Clojurists Together team. We have these board members.

On the left we have Maria Geller, Daniel Solano Gómez, Larry Staton Jr., Nola Stowe, Fumiko Hanreich, and Laurens Van Houtven. These are all current board members. We also have [00:30:30] board members from the past, Bridget Hillyer, Toby Crawley, Devin Walters, and Rachel Magruder. [inaudible 00:30:36]is our admin assistant. If you’re a Clojurists Together member, you might’ve got some stickers from us recently, and if you’re wondering, why did this come from Spain? It’s because Rachel, our admin assistant lives in Spain.

I’d also like to thank our members. We’ve seen many of these names around the conference already. Pitch, Nubank, JUXT, Metosin, Adgoji, and Funding Circle. [00:31:00] These companies have all been a huge help to Clojurists Together, and have funded us in a really big way that have let us do all of the work that we’ve been able to do.

Also, I’d like to think these other company members, I don’t have time to go over all of their names, but many of these names are going to be familiar to you, and some of them are at the conference as well.

This is probably my favorite slide at the talk, because this is the names of the 200 members, developer members in the Clojure, [00:31:30] Clojurists Together that are members of Clojurists Together. Yeah, there’s far too many here to name them all. I tried to make the text bigger, so that you could even read the names, but then it just went on for slides, and slides, and slides.

I can’t name them all, but I’d really like to say thanks to the 204 developer members, the 34 company members, and also to the rest of the Clojure community.

The Clojure community is a [00:32:00] really special place. It’s warm. It’s very small, and I think we punch above our weight in terms of the impact that we can have, in terms of the tooling and libraries that we create. That’s in large part due to the contributions of all of the Clojure community coming together to build things for everybody. I think a good chunk of the Clojurists Together members are here at this conference, and so again, I especially want to say thanks to you for your support.

I’m joining the Social Coding team at GitHub as a product manager. For the last six years I’ve been writing Clojure professionally; product management is going to be quite different, but I’m looking forward to it. There are a few reasons why I’m looking forward to working at GitHub:

• Over the last few years working on Clojurists Together I’ve been looking at how to fund open source software that is in the community good. One of the best parts of the role has been talking to open source maintainers to help design funding programs that serve their needs, and the needs of the community. In my new role I’ll get to talk to even more open source maintainers, contributors, and users about their experiences with open source and GitHub.
• GitHub is a remote friendly company and the majority of their employees were remote pre-COVID. I’ve only worked in small remote teams and I’m looking forward to seeing how remote work scales to a large organisation.
• GitHub has been on a tear in the last few years, releasing many features and products. I’m excited to experience this from the inside.
• Im also looking forward to be working for Devon Zuegel. I met Devon 18 months ago when she was doing research for GitHub Sponsors, and I’ve been impressed with the work she and the rest of the GitHub Sponsors team has done since then.

GitHub is the worlds most important social network for developers, and I’m thankful for the opportunity to help build it.

The 31 March 2019 IIT return for {person} is due 8 July 2019.

and

The 31 March 2019 ITN return for {company} is due 8 July 2019.

I’d never heard of an IIT return or an ITN return. I searched around and couldn’t get any results on Google or IRD’s website search. However, my best guess at the acronyms is:

• IIT Return: IR3 - Individual Income Tax Return
• ITN Return: IR4 - can’t think of a good explanation for this acronym

I think IRD has renamed these in their recent system upgrades and I suspect they’ve renamed all of their returns. If you come across any other acronyms, let me know and I’ll add them here to help others out.

I studied as a musician, and accumulated several pieces of audio gear and knowledge about working with audio which have been very useful for working remotely. A friend asked me for advice about getting some audio gear so I sent him an email. That turned into a post on our company wiki, and now I’m posting it here publicly.

In my experience, for remote videoconferencing to work well, the following things are very beneficial:

• A fast, stable, and reliable internet connection
• A quiet space free of distractions, and without too much echo
• A space you’re able to comfortably talk in during work hours
• A good microphone close to your mouth and a set of headphones - Hearing yourself echo on someone else’s call makes it hard to have a conversation.

Improving audio quality when video conferencing

Here are some tips in rough order of importance for improving audio quality when videoconferencing. You can keep going down the list with diminishing returns of audio quality, stop whenever you and/or your team is happy with the quality you’re getting.

• Get really close to the mic. It should be no more than an inch or two from your lips. Having a mic far away from your mouth makes everything else more difficult. A stand on a desk is better if it goes up to your face, having the mic at desk level will pick up lots of other noises and echoes.
• Plug your computer in with Ethernet, or make sure you have really strong Wi-Fi. I always prefer to use a cable to remove a potential weak link.
• Find a quiet room to talk in. If you can’t find a quiet room, or you are on a call with more than 3 participants, I’d recommend muting yourself when you’re not talking.
• Find a room that doesn’t echo. Clapping your hands will help you spot echo quickly. You especially want to avoid sitting in any spot where you hear ringing after a clap. Blankets, couches, other soft surfaces help deaden echo.
• Get a decent mic. You should be able to get something fine for $70-$200 USD. Your first choice is whether to use an XLR mic + USB audio interface or a USB mic. An XLR mic will last a long time, especially in office use. My mic is about 10 years old now and still running fine. I’d personally steer clear of mics with a built in USB connection. Because they combine active electronics with the mic, the electronics may break or become obsolete even when the mic is still good. However, they don’t require an extra audio interface which is one less moving part and they will still sound fine. Getting a USB mic wouldn’t be a ‘wrong’ decision, this comes down to your comfort with audio gear and how much stuff you want on your desk. Marco Arment has a good overview of mics. Microphones have different polar patterns which control where they pick sound up from. If you want to be able to play meeting audio through speakers instead of wearing headphones, then look for a mic with a cardioid polar pattern to only pickup sound from the front of the mic. Dynamic microphones are often in a cardiod pattern. Condenser microphones are often bidirectional and pick up equally well front and back, which will not work well when combined with speakers.
• If you have an audio interface or USB mic, watch that you’ve set the input levels correctly. If the gain is set too high, you will get terrible sounding digital clipping. If your interface has lights, watch them while talking loudly and make sure they don’t turn red. You can also look at the input levels on your computer and make sure they aren’t hitting the very top of the input scale as this may also indicate clipping. When in doubt, set the gain levels a little lower; video conferencing software usually has some gain leveling built-in to match you with the other speakers.
• Get a pop filter, this takes the plosives out of your speech, a mic picks these up much more than a person does.
• If you don’t get a USB mic you’ll need a separate audio interface. https://marco.org/podcasting-microphones#interfaces has some suggestions. Anything over $80 USD will be fine. You only need a 1 input audio interface, but often 2 input interfaces will be about the same price. Look for one with a USB connection, you don’t need Thunderbolt for single channel recording. USB-C and/or USB 3.0 would be ideal for longevity, but they cost a little bit more, and USB 2.0 is just fine for a single recording channel.
• Once you’ve got echo in your room under control, you can look at treating it with acoustic panels to “deaden” the room further. You can make these yourself, or purchase them from lots of places online. My office has a wooden floor, and drywall roof and walls which created lots of unpleasant echoes. The person I bought my foam panels off suggested I target ceiling/floor reflections first, I mounted four panels on my roof, and this was enough for me. Acoustic treatment is a whole science in itself and is very room dependendent, so do some research for your space.
• If you’re not able to avoid echoey or noisy room environments, then you can look at processing your sound with noise removal software. I have used SoundSource (disclosure: a friend made it) and SPL De-Verb Plus to remove room noise on the other end of calls which turned them from painful to tolerable. I also tried Krisp but didn’t get great results from it. Your mileage may vary though.
• https://sixcolors.com/post/2016/11/a-podcast-studio-for-under-100/ has more general gear tips.

Others have also posted their thoughts on Heart of Clojure: Fork This Conference, The people’s conference, The hallway track conference, Community with lots of heart, and A courageous conference?.

Here were some things that I think made Heart of Clojure so special. If you’re running a conference, consider stealing some of these ideas.

• There was an site which listed activities around the event. Anyone could host an activity and others could join. Having it on a website meant everyone knew what was happening around the conference. I felt like this helped Heart of Clojure feel very welcoming to all of the conference goers, especially those who were less outgoing.
• One of the nights I joined an activity for an adventurous dinner. People got split into groups of 6-8 with a reservation at a restaraunt in Leuven. This broke up friend groups and introduced me to a bunch of new people. The restaurant we went to had their own brewery, and I got to try the flavours of a beer soup, beer stew, and beer crème brûlée!
• Yulia Startsev’s talk To loosen up, to put together set the tone for the conference and captured the spirit of mixing technology and the humanities. Many of the talks and conversations referenced her talk throughout the rest of the event.
• Leuven was a perfect town to have the conference. It was large enough to have a good nightlife, but small enough that you could walk between the conference venue, the bars, restauruants, and hotels in 10-15 minutes.
• In the months before the conference, Jan Stępień paired up new speakers with more experienced speakers. This was my first conference talk, and I got paired up with Tiago Luchini. Tiago was a very experienced public speaker and gave me great feedback on my talk. Having to show someone my presentation helped me get it into shape well before the conference. I think Jan’s work here contributed to the high quality of the talks at the conference.
• I’m not a vegeterian, but I really appreciated that all of the breakfast and lunches were vegeterian. I can imagine it would have made things very easy for vegeterians. The food was delicious and healthy tasting.
• There was ample time left around the talks so that you didn’t need to choose between the ‘hallway track’ and the conference track. On the Saturday there was a siesta break between 12pm and 4pm. You could do one of the activities, take a nap, or just chat with others. I was a bit skeptical before the conference that this was going to be too long, but it worked out really well.
• The lightning talks on Saturday broke up the afternoon and left space for serendipity to strike. Connie’s talk on waffles, Vim, and building her own language captured the audience.
• On the Saturday evening after the conference ended, lots of people stayed behind to help pack down the auditorium. This made the job much easier.
• There was a large contingent of people from Berlin, and they were all incredible. Berlin feels like it has a really vibrant community, one day I’d like to visit it.

Heart of Clojure was a very special event. I can’t imagine how much work it was for Arne Brasseur, Martin Klepsch, and all of the other helpers before, during, and after the event but it paid off. Heart of Clojure felt like a very polished event, not something being put on for the first time. Thanks to everyone who organised, helped out, spoke, sponsored, and attended the conference. I hope there is another Heart of Clojure in the future, if you get a chance to go, I highly recommend it.

If you see profiles like this in a project.clj, here is what is happening:

:dev  [:project/dev  :profiles/dev]
:test [:project/test :profiles/test]
:profiles/dev  {}
:profiles/test {}
:project/dev  { ... }
:project/test { ... }

Leiningen has a feature called Composite Profiles. On startup, if Leiningen sees a vector of keywords for a profile, it will lookup each keyword as a profile, and merge them together. For the :dev profile, Leiningen will merge the values of :project/dev and :profiles/dev.

If you want to add any custom settings or dependencies for your own use, you can place them into the :profiles/dev or :profiles/test in your ~/.lein/profiles.clj. If either of these are set in the user’s profiles.clj, they will override the empty :profiles/dev map specified in the project.clj. You need an empty map for :profiles/dev {} in the project.clj, because otherwise Leiningen will complain about a missing profile.

“Looks like you’re listening to music on another device.”

Eventually this error became too annoying and I looked at upgrading to a Family subscription. The Apple Music Family subscription allows up to six people that are part of the same Family Sharing group to play music at the same time.

It sounded like this was probably what I needed, but in all of the documentation I read, it wasn’t clear whether two devices that were logged into the same Apple ID would be able to play music at the same time. I talked with Apple’s support team and it still was unclear so I went ahead and upgraded to try it for myself.

After upgrading I tested playing on multiple devices, and it worked fine. If you have an Apple Music Family subscription, you’re able to play on multiple devices that are logged into the same Apple ID.

Update: More context from Benjamin Mayo on Reddit:

If you have the Apple Music family plan, multiple concurrent device streams increases to 6 rather than 1. HomePod and Apple TV usage do not count toward this limit.