Log in
Cameron’s World is a web-collage of text and images excavated from the buried neighbourhoods of archived GeoCities pages (1994–2009).
Holy shit this is amazing: Cameron’s World
There was recently a thread on r/slatestarcodex about “What life hacks are actually life changing”. One of the examples given was:
Buy audiobooks to read much more books, listen at 1.5-2x speed
This led to the following thread (later removed):
Aaa: A midwit in making
Bbb: Audio books help people consume many more books than they otherwise would, and listening to them in place of podcasts on commutes or during the types of workouts where listening is feasible is generally a good thing. Though there are certainly some fun podcasts.
Ccc: But you retain far less information. So it’s just so your mind is busy with something and doesn’t have to deal with existential crisis
I find this delightful.
You think you’re giving out helpful tips for internet friends? Haha, no, you’re inviting a review of your intelligence and philosophical outlook! It’s like a cute little bunny prancing through the forest until a piano suddenly falls from the sky.
Of course, “person on internet is mean” isn’t much of a story. (Ask me about my emails after I suggested that maybe Julian Assange isn’t the most admirable human being ever.) And anyway, everyone was fine other than Aaa.
No, what makes this funny is that there was a chorus of people agreeing with Ccc.
I too must admit Ccc has a point. Certainly, it’s human nature to do things that make us feel better. And for many of us, that includes doing things to prevent experiencing existential crises.
But try the following statements:
-
“Getting good sleep and regular exercise just means that your mind is full of happy chemicals and doesn’t have to deal with existential crisis.”
-
“Having kids just means that your mind is busy taking care of new life and experiencing ‘joy’ and doesn’t have to deal with existential crisis.”
-
“Hiking up to mountain vistas just means your mind is distracted by beauty and doesn’t have to deal with existential crisis.”
-
“Having deep relationships just satisfies ancient instincts designed to help you access resources and trigger a feeling of ‘meaning’ in your mind so it doesn’t have to deal with existential crisis.”
If angst is bad, then isn’t it doing things that reduce angst good?
I think most people believe that. As evidence, I’ll point to a weird norm our culture has that most people don’t seem to notice. You’re not supposed to talk about existential crises. Personally, I think my baseline instinctual angstiness would place me somewhere near the 90th percentile of the population. (I mean, I have a blog…) But don’t whine about it because I don’t want to be a lame edgelord.
It’s acceptable to refer to angst, but with remove. You can make dark jokes or create existential art. You can recite that “most men live lives of quiet desperation”. You can go on the internet and make clinical comments about other people’s angst. But you aren’t supposed to tell your friends that you’re currently having an existential crisis, please help. That crosses a line.
Of course, “edgelords are lame” is a modern norm. But I doubt proto-edgelording would have gotten you much further centuries in the past. People would probably have told you to go to church, or told you to take care of your family, or just avoided you.
I bring this up because I think the “edgelords are lame” norm is… good? Baseline angst notwithstanding, my life is quite good and would not be improved at all by more contact with people moaning about their feel-feels. (See that? See how I use humor and derision to demonstrate distance?)
In fact, why don’t we invert Ccc’s implied criticism? Why don’t we take, “prevent your mind from dealing with existential crisis” as a basic life strategy?
This would indicate lots of “healthy” stuff. (Travel! Friends! Romance! Hobbies! Meaningful work! Children!) But perhaps it would also indicate “wireheading”—shortcutting your brain’s reward architecture to feel good when you aren’t “supposed” to.
If you’re not familiar with wireheading, the classic thought experiment goes like this:
There’s a machine. If you go into the machine, electrodes will be connected to your brain, your memories of the machine will be erased, and you’ll live the rest of your life in a simulation of all your dreams and fantasies. Do you enter the machine?
(So, pretty much exactly the plot of Total Recall? Yes, Total Recall is a meditation on wireheading.)
Philosophically speaking, I find wireheading troubling. But practically, in the particular case we’re examining here, we can just consider a simpler question.
Is listening to books on tape at 1.5x-2x speed a form of wireheading?
And if so, what about music? What about jokes? What about comfortable furniture or attractive wallpaper?
People don’t have existential peace because they’ve figured out the meaning of life. They have existential peace because that’s their nature or because they’ve developed happy lives and healthy thought patterns that don’t lead to them spending their time moping around. Feeling like you understand the meaning of life is downstream of existential peace, not upstream.
So even though I think Ccc is right, in a deeper sense I think Ccc is totally wrong. Don’t put existential crises on a pedestal! Arguably they’re just a weird glitch in the human brain anyway. There’s no “correct” way to look at all this stuff. So if you like listening to audiobooks at 1.5x to 2x speed, listen away.
Like many people, I used to have trouble sleeping. I did all the standard things—I reduced alcohol, I kept the room cold and dark, I stopped drinking 17 cups of coffee every day, everything. And all that stuff helped. (If you have trouble sleeping, do the standard things.) But I wasn’t satisfied, so I tried various supplements. All of these were either ineffective, or left me groggy the next day. Finally, I discovered that what really worked was to skip all the supplements, and just listen to a podcast that was both very good and also very boring. The best podcast for this purpose turns out to be…
P.S. Today’s link is to RPGAdventures on Comedy Theory:
When you see a pattern (“A, B, C”), your brain automatically recognizes the pattern (“Alphabet”), and makes a prediction about what comes next.
Jokes have two parts, setup and punchline. Setup leads people to recognize a pattern, which their brains automatically use to make predictions and assumptions. Punchline forces their brain to switch to a different pattern, shattering their assumptions, making it clear that predictions they’ve made are invalid.
table of contents
How often do your simple prototypes or ad-hoc scripts turn into fully-fledged applications?
The simplicity of organic code growth has a flip side: it becomes too hard to maintain. The proliferation of dicts as primary data structures is a clear signal of tech debt in your code. Fortunately, modern Python provides many viable alternatives to plain dicts.
What’s wrong with dicts?
Dicts are opaque
Functions that accept dicts are a nightmare to extend and modify. Usually, to change the function that takes a dictionary, you must manually trace the calls back to the roots, where this dict was created. There is often more than one call path, and if a program grows without a plan, you’ll likely have discrepancies in the dict structures.
Dicts are mutable
Changing dict values to fit a specific workflow is tempting, and programmers often abuse this functionality. In-place mutations may have different names: pre-processing, populating, enriching, data massage, etc. The result is the same. This manipulation hinders the structure of your data and makes it dependent on the workflow of your application.
Not only do dicts allow you to change their data, but they also allow you to change the very structure of objects. You can add or delete fields or change their types at will. Resorting to this is the worst felony you can commit to your data.
Treat dicts as the wire format
A common source of dicts in the code is deserializing from JSON. For example, from a third-party API response.
>>> requests.get("https://api.github.com/repos/imankulov/empty").json()
{'id': 297081773,
'node_id': 'MDEwOlJlcG9zaXRvcnkyOTcwODE3NzM=',
'name': 'empty',
'full_name': 'imankulov/empty',
'private': False,
...
}
A dict, returned from the API.
Make a habit of treating dicts as a “wire format” and convert them immediately to data structures providing semantics.
The implementation is straightforward.
- Define your domain models. A domain model is simply a class in your application.
- Fetch and deserialize in the same step.
In Domain-Driven Design (DDD), this pattern is known as the anti-corruption layer. On top of semantic clarity, domain models provide a natural layer that decouples the exterior architecture from your application’s business logic.
Two implementations of a function retrieving repository info from GitHub:
import requests
def get_repo(repo_name: str):
"""Return repository info by its name."""
return requests.get(f"https://api.github.com/repos/{repo_name}").json()
The output of the function is opaque and needlessly verbose. The format is defined outside of your code.
>>> get_repo("imankulov/empty")
{'id': 297081773,
'node_id': 'MDEwOlJlcG9zaXRvcnkyOTcwODE3NzM=',
'name': 'empty',
'full_name': 'imankulov/empty',
'private': False,
# Dozens of lines with unnecessary attributes, URLs, etc.
# ...
}
class GitHubRepo:
"""GitHub repository."""
def __init__(self, owner: str, name: str, description: str):
self.owner = owner
self.name = name
self.description = description
def full_name(self) -> str:
"""Get the repository full name."""
return f"{self.owner}/{self.name}"
def get_repo(repo_name: str) -> GitHubRepo:
"""Return repository info by its name."""
data = requests.get(f"https://api.github.com/repos/{repo_name}").json()
return GitHubRepo(data["owner"]["login"], data["name"], data["description"])
>>> get_repo("imankulov/empty")
<GitHubRepo at 0x103023520>
While the example below has more code, this solution is better than the previous one if we maintain and extend the codebase.
Let’s see what the differences are.
- The data structure is clearly defined, and we can document it with as many details as necessary.
- The class also has a method
full_name()implementing some class-specific business logic. Unlike dicts, data models allow you to co-locate the code and data. - GitHub API’s dependency is isolated in the function
get_repo(). The GitHubRepo object doesn’t need to know anything about the external API and how objects are created. This way, you can modify the deserializer independently from the model or add new ways of creating objects: from pytest fixtures, the GraphQL API, the local cache, etc.
☝️ Ignore fields coming from the API if you don’t need them. Keep only those that you use.
In many cases, you can and should ignore most of the fields coming from the API, adding only the fields that the application uses. Not only duplicating the fields is a waste of time, but it also makes the class structure rigid, making it hard to adopt changes in the business logic or add support to the new version of the API. From the point of view of testing, fewer fields mean fewer headaches in instantiating the objects.
Streamline model creation
Wrapping dicts require creating a lot of classes. You can simplify your work by employing a library that makes “better classes” for you.
Create models with dataclasses
Starting from version 3.7, Python provides Data Classes. The dataclasses module of the standard library provides a decorator and functions for automatically adding generated special methods such as __init__() and __repr__() to your classes. Therefore, you write less boilerplate code.
I use dataclasses for small projects or scripts where I don’t want to introduce extra dependencies. That’s how the GitHubRepo model looks like with dataclasses.
from dataclasses import dataclass
@dataclass(frozen=True)
class GitHubRepo:
"""GitHub repository."""
owner: str
name: str
description: str
def full_name(self) -> str:
"""Get the repository full name."""
return f"{self.owner}/{self.name}"
When I create Data Classes, my Data Classes are almost always defined as frozen. Instead of modifying an object, I create a new instance with dataclasses.replace(). Read-only attributes bring peace of mind to a developer, reading and maintaining your code.
Alternatively, create models with Pydantic
Recently Pydantic, a third-party data-validation library became my go-to choice for model definition. Compared with dataclasses, they are much more powerful. I especially like their serializers and deserializers, automatic type conversions, and custom validators.
Serializers simplify storing records to external storage, for example, for caching. Type conversions are especially helpful when converting a complex hierarchical JSON to a hierarchy of objects. And validators are helpful for everything else.
With Pydantic, the same model can look like this.
from pydantic import BaseModel
class GitHubRepo(BaseModel):
"""GitHub repository."""
owner: str
name: str
description: str
class Config:
frozen = True
def full_name(self) -> str:
"""Get the repository full name."""
return f"{self.owner}/{self.name}"
Online service jsontopydantic.com saves my time creating Pydantic models from third-party APIs. I copy the response examples from their documentation to the service, and the service returns Pydantic models.
You can find some examples of me using Pydantic, in the post Time Series Caching with Python and Redis.
For legacy codebase, annotate dicts as TypeDict
Python 3.8 introduced so-called TypedDicts. In runtime, they behave like regular dicts but provide extra information about their structure for developers, type validators, and IDEs.
If you come across the dict-heavy legacy code and can’t refactor everything yet, at least you can annotate your dicts as typed ones.
from typing import TypedDict
class GitHubRepo(TypedDict):
"""GitHub repository."""
owner: str
name: str
description: str
repo: GitHubRepo = {
"owner": "imankulov",
"name": "empty",
"description": "An empty repository",
}
Below, I provide two screenshots from PyCharm to show how adding typing information can streamline your development experience with the IDE and protect you against errors.
For key-value stores, annotate dicts as mappings
A legitimate use-case of dict is a key-value store where all the values have the same type, and keys are used to look up the value by key.
colors = {
"red": "#FF0000",
"pink": "#FFC0CB",
"purple": "#800080",
}
A dict, used as a mapping.
When instantiating or passing such dict to a function, consider hiding implementation details by annotating the variable type as Mapping or MutableMapping. On the one hand, it may sound like overkill. Dict is default and by far the most common implementation of a MutableMapping. On the other hand, by annotating a variable with mapping, you can specify the types for keys and values. Besides, in the case of a Mapping type, you send a clear message that an object is supposed to be immutable.
Example
I defined a color mapping and annotated a function. Notice how the function uses the operation allowed for dicts but disallowed for Mapping instances.
# file: colors.py
from typing import Mapping
colors: Mapping[str, str] = {
"red": "#FF0000",
"pink": "#FFC0CB",
"purple": "#800080",
}
def add_yellow(colors: Mapping[str, str]):
colors["yellow"] = "#FFFF00"
if __name__ == "__main__":
add_yellow(colors)
print(colors)
Despite wrong types, no issues in runtime.
$ python colors.py
{'red': '#FF0000', 'pink': '#FFC0CB', 'purple': '#800080', 'yellow': '#FFFF00'}
To check the validity, I can use mypy, which raises an error.
$ mypy colors.py
colors.py:11: error: Unsupported target for indexed assignment ("Mapping[str, str]")
Found 1 error in 1 file (checked 1 source file)
Take dicts under control
Keep an eye on your dicts. Don’t let them take control of your application. As with every piece of technical debt, the further you postpone introducing proper data structures, the more complex the transition becomes.
