
_[Originally written for people onboarding a project I've been leading, to explain why services are structured the way they are, and to provide a set of reasoned guidelines.]_

## 1. Decouple service configuration, definition, creation, and startup

If your `index.ts` looks like this:

```ts
import express from 'express'

const app = express()

app.get('/', (req, res) => {
  /* ... */
})

app.listen(3000)
```

Then you'll have a hard time using the service in any environment other than a live one (such as testing), because definition, creation, and startup are all mixed with module loading.

Instead, define the service creation in `service.ts`, and **don't** start it as part of creation, but return a control interface so that the caller can decide when to start it:

```js
import express from 'express'

const createService = ({ port }: { port: number }) => {
  const app = express()

  app.get('/', (req, res) => {
    /* ... */
  })

  return {
    start () =>
      new Promise((resolve) => {
        app.listen(port, resolve)
      }),
  }
}
```

Then create, configure, and start the service in `index.ts`:

```ts
import { createService } from './service'

const main = async () => {
  const service = createService({ port: 3000 })
  await service.start()
}

main()
```

## 2. (Generalization of #1) With the exception of the entry point, use module scope only for definitions, never for side effects

Consider this:

```ts
import { computeSignature } from 'something-sdk'
```

And notice it's not:

```ts
importDefinitionWhileMakingSomethingElseHappen {
  computeSignature
} from 'acme-sdk'
```

Which is why we'd never expect `node_modules/acme-sdk/index.js` to be:

```ts
export const computeSignature = (data) => {
  /* ... */
}

startAcmeConnection(process.env.ACME_API_URL)
```

That would reduce the number of scenarios where `computeSignature` can be used (to live environments which have a connetion to `Acme`). Arguably, a module author's job is to _maximize_ that number.

A good rule of thumb is that **if a module contains side effects, it should probably not contain `export`s**.

## 3. Don't instantiate dependencies as part of service creation

Instead of passing `dbUri` to `createService` and expect it to instantiate a database connection, like so:

```ts
// service.ts

const createService = (config: { dbUri: string }) => {
  const app = express()
  const db = await createDatabaseConnection(config.dbUri)

  app.get('/items', async (req, res) => {
    const items = await db.getItems()
    /* ... */
  })

  /* ... */
}
```

Instantiate the database connection and pass it to the service factory as just one more configuration property:

```ts
// index.ts

import { createService } from './service'
import { createDbConnection } from './db'

const main = async () => {
  const db = createDbConnection('postgres://dbhost/devdb')
  const service = createService({ db })
  await service.start()
}
```

Mocking `db` when testing the service will be far easier that way.

## 4. Environment is broken until proven otherwise

If you cringed at hardcoded configuration values in the previous examples and wondered why not read them from environment variables, that's because the topic deserves its own section.

`process.env` is the silent killer of production services:

- Its value is set in places that can't validate it: `.env` files, Kubernetes manifests, the "Environment" tab on the CI web UI; to them, `DBURI` is just as good as `DB_URI`.
- It has all possible values in existence... at least according to its typings; you may have set `DB_URI` correctly in `.env`, but TypeScript will happily accept `process.env.DBURI`.
- Environment values are often optional, so we tend to provide default values, which compounds the previous point. What happens when you correctly set `DB_URI=posgress//dbhost/proddb` in `.env`, but your code has the following?

```ts
const dbUri = process.env.DBURI ?? 'postgres://dbhost/devdb'
```

The answer is that nothing happens _immediately_...

The solution is the same as for any external data: **validate**.

Instead of importing from `process.env`, import from a custom `env` module:

```ts
// index.ts

import { env } from './env'

const main = async () => {
  const service = createService({})
  await service.start()
}

main()
```

And define the `env` module as:

```ts
// env.ts
import { ensure, v } from 'suretype' // or your preferred validator

const envSchema = v.object({
  NODE_ENV: v.string().enum('development', 'production').required(),
  PORT: v.string(),
  DB_URI: v.string().required(),
})

export const env = ensure(envSchema, process.env)
```

That way, misconfigurations will cause evident crashes instead of hard to track, potentially catastrophic bugs.

Once you've banished `process.env`, to prevent it from sneaking in again (it's so ubiquitous that newcomers to the team will no doubt use it without giving it a second thought), consider adding this to `.eslintrc`:

```json
  "rules": {
    "no-process-env": "error",
  }
```


Four patterns for developer-friendly, production-grade NodeJS services


I recently wrote code similar to this:

```jsx
const App = (props) => {
  return (
    <ThemeProvider theme={theme}>
      <PageContent {...props} />
    </ThemeProvider>
  )
}

const PageContent = ({ Component, pageProps }) => {
  const getLayout = Component.getLayout ?? ((page) => page)
  return getLayout(<Component {...pageProps} />)
}
```

If it were my first time reading it, I'd think that the `PageContent` component is a pointless, even harmful addition: it's only used once; it doesn't express any remarkable idea; it forces the reader to store yet another name in an already crowded working memory.

I'd be strongly tempted to refactor it, and since it's a non-functional change, I might do it as part of a larger change, heeding the [boy scout rule](https://www.oreilly.com/library/view/97-things-every/9780596809515/ch08.html):

```jsx
const App = (props) => {
  const getLayout = Component.getLayout ?? ((page) => page)

  return (
    <ThemeProvider theme={theme}>
      {getLayout(<Component {...pageProps} />)}
    </ThemeProvider>
  )
}
```

I might even skip refreshing the page, because, again, it's just a change in form.

Or is it?

I'd later find out that the background no longer respects the dark mode setting, while every other component does; after trying for a long while to debug the larger functional change without getting anywhere, I would trace the problem to the code above.

If you're trying to figure out how the refactoring attemept could cause that, good luck. Although you might spot something if you're well versed in React _and_ you squint hard _and_ you've encountered the circumstance before, it's really hard, because the code you need to know the most to figure out what's going on is nowhere in sight, it's instead in the source for the page component itself, in a file far away:

```tsx
export const getLayout = (page) => (
  <Layout bgImage={useColorModeValue(backgroundLight, backgroundDark)}>
    {page}
  <Layout>
)
```

Even so, still not obvious...

Imagine, however, that while writing the original code, I had placed this comment inside the `PageContent` component:

```tsx
const App = (props) => (
  return (
    <ThemeProvider theme={theme}>
      <PageContent {...props} />
    </ThemeProvider>
  )
)

const PageContent = ({ Component, pageProps }) => {
  // Why not just have all this in <App>? Because getLayout uses the
  // color mode hook, which in turn must be called from under
  // <ThemeProvider> to work

  const getLayout = Component.getLayout || ((page) => page)
  return getLayout(<Component {...pageProps} />)
}
```

How likely is the reader now to jump into refactoring?

It's second nature for programmers, while they're writing code, to imagine it playing out at run time and continuously ask: "what's likely to go wrong here? what could cause this to break? how can I prevent or handle it?" Users may provide invalid input, network connections might go down, the environment might not support the cabability, ... Situations handled with schemas, timeouts, discovery, ...

It pays to adopt the same mentality and, while you're writing code, imagine it playing out at _read_ time and ask: "what's likely to go wrong here? what could cause this to be misunderstood? how can I prevent or handle it?" Commenting code that's not there — explaining the outside-of-source reasons why the code is the way it is, instead of a more "logical", "elegant", or "obvious" form — is a relatively cheap and absolutely effective way to prevent _read time bugs_.


Comment the code that is not there


[web3-mock](https://github.com/DePayFi/web3-mock) supports Cypress and JSDOM test environments out of the box. However, I prefer Playwright to those as it provides a compelling set of benefits: familiar Jest matchers, a fixture-based runner, and execution in multiple real browser environments. Getting web3-mock to work with Playwright requires setup that is minimal but not obvious, hence this quick note.

The test definitions configures the mock and then interact with the page:

```typescript
import { test } from '@playwright/test'

test('test wallet connection', async ({ page }) => {
  await mockWeb3(page, () => {
    // @ts-ignore
    Web3Mock.mock({
      blockchain: 'ethereum',
      accounts: { return: ['0xd73b04b0e696b0945283defa3eee453814758f1a'] },
    })
  })

  await page.goto('http://localhost:3000/')
  await page.locator('text=Sign in').click()
  await page.locator('text=Connected').waitFor()
})
```

The `mockWeb3()` call causes the `web3-mock` library plus the mock configuration to be loaded whenever a page is navigated to, but before the page's own scripts are run:

```typescript
import { Page } from '@playwright/test'
import { readFileSync } from 'fs'

const mockWeb3 = async (page: Page, fn: Function) => {
  await page.addInitScript({
    content:
      readFileSync(
        require.resolve('@depay/web3-mock/dist/umd/index.bundle.js'),
        'utf-8',
      ) +
      '\n' +
      `(${fn.toString()})();`,
  })
}
```

Why the ugly string concatenation, when we could call `addInitScript` twice, once for injecting the library, the next for injecting the mock configuration? Because the order of execution of scripts added with `addInitScript` is not defined, so the library might get loaded after (the attempt at) using it.


Mocking window.ethereum in Playwright for end-to-end dApp testing

Structure describes and prescribes; it tells about the system while nudging you into thinking of, and using it, in a certain way—hopefully, a way that was chosen deliberately, wisely, and points toward [the pit of success](https://blog.codinghorror.com/falling-into-the-pit-of-success/).

Discussion of React project structures usually centers on the descriptive ("what does folder/file X contain?") with an intent of easing navigation ("where do I find object/function Y?"). It's a start, though one with diminishing returns.

In this note I'd like to offer, rather than my preferred project structure, some ways of thinking about the project space, some ways of slicing it, and a discussion of how they nudge developers into thinking about important properties of the system. Even if those ways and properties aren't right or sufficient in your case, the process should help you come up with ones that are.

## Audience

But first: who does your project speak to? Many eyes will visit it over time and you can't optimize for all. Take a moment to think of which ones matter.

Developer seniority is less of a factor than you'd think: don't refrain from using words like _container_ or _machine_ in file names just because juniors might find them unfamiliar; if the concepts are central to the architecture, they need to be faced at some point, and it might well be through the filesystem.

Project seniority, on the other hand — the time developers spend on the project — **is** a factor: a project where people from other teams constantly jump in for just a few days, help with something, and jump out, needs loud signposts to macroscopic concepts that are inevitably forgotten between incursions and thus need constant refreshing; think `pages/`. A stabler team can afford to take the basics for granted and use the bandwidth for far finer points; think `pure/` (as in "pure functions").

## Navigation (...don't)

You step into a supermarket with sixty seconds to grab a microwaveable tandoori chicken. Do you use them to decipher the layout of the aisles, run to the first one that sounds related ("ready dishes"? "world kitchen"?), frantically browse through the shelves, ... or do you find a clerk and ask?

I like to think that something like that went through brains at Google's when faced with:

![Yahoo](https://i.insider.com/57977a3288e4a794048babc2?width=1000&format=jpeg&auto=webp)

To eventually lead to:

<p style="text-align:center;"><img src="https://www.versionmuseum.com/images/websites/google-search/webp/google-search%5E1999%5Esearch-homepage.webp" alt="Google"></p>

It's tempting to slice your project's space into a multitude of "aisles" and put everything on "its" shelf. We're just trying to ease navigation! (And, truth be told, we get a kick out of imparting order.)

```
~/projects/app$
index.jsx
components/
components/forms/
components/pages/
components/layouts/
components/widgets/
```

Do you look for `SubmitButton.jsx` under `forms/` or `widgets/`? More to the point: do you or anyone on your team actually "browses aisles" when looking for something?

It's worth asking around, as everyone's flow differs, but I almost always invoke the project-wide search and type a fragment of filename or actual content; or place cursor on a symbol, and jump to definition. Few keystrokes later, I'm where I need to be.

Browsing is alright when you don't know what you want and you want to see what's available. There _is_ some value to organizing for browsing; just not that much, and just not for long. You might still go for it for specific case, such as if people jump in and out of the project often.

## The questions, and _the_ question

The names we pick answer questions:

```text
components/Profile.test.jsx
^          ^       ^
|          |       +--- "What aspect does it cover?"
|          +----------- "What role does it play in the application?"
+---------------------- "What kind of construct is it?"
```

But we rarely pick our questions consciously. That's risky because our choice of questions is itself an answer to a more fundamental one: _What's worth focusing on?_ Focus on the inconsequential and, no matter how good the answers, you'll move slow, if not backward.

I encourage you to surface your own questions. Here are some that I use and a discussion of each:

1. What changes together?
2. What's used in one place, what in more?
3. What follows the same rules?
4. Does loading this produce side effects or just redefinition?
5. Does this depend on component state?

### 1. "What changes together?"

If you change a component test, chances are you'll change the corresponding component soon, and vice-versa. If you delete the component, you don't want the test lying around. If you look at the file list, you want to spot right away a component that lacks tests.

Strongly related code can, in principle, live in the same file. However, navigating source would require more work, and code would be processed unnecessarily in CI, where seconds matter, and interactive development, where milliseconds do.

Strongly related code shouldn't live in faraway folders (such as `src/`, `test/`, `stories/`). The small gain in visual tidiness is soon forgotten when an orphaned test has been waiting for months for someone to find the time and check if it can be removed.

**Sidecars** are a happy mean.

The example below is extreme, but don't be afraid of going one or two steps further than what you're used to:

```text
components/
components/Profile.jsx
components/Profile.test.jsx
components/Profile.stories.jsx
components/Profile.Layout.jsx
components/Profile.styles.jsx
components/Profile.reducer.ts
components/Profile.functions.ts
components/Profile.machine.ts
```

"But won't this make my `components` directory explode?"

Even without sidecars, a medium-sized React application has enough components files to sink way below the fold. At that point, already people aren't finding things by reading through the file list; two pages or twenty won't make a difference. But they _are_ looking at subsets of it when selecting files, though, so seeing related things together helps.

### 2. "What's used in one place, what in more?"

[TBD]

### 3. "What follows the same rules?"

When you associate rules to a space, "entering" it is enough to leverage all the thinking that went into creating the rule, without going again through it.

For example, importing stateful (or even just configured) objects in a React component file, like `api` below, makes testing hard, because you have to plug into the module import system:

```jsx
import React, { useState, useEffect } from 'react'
import { Profile } from 'src/components/Profile'
import api from 'src/api'

export const ProfilePage = () => {
  const [profile, setProfile] = useState(null)

  useEffect(() => {
    api.getProfile().then(setProfile)
  }, [])

  return profile ? <Profile profile={profile} /> : null
}
```

Thus the rule that a component file can import as many functions as it wants, but state should only "enter" via props or hooks.

That's workable, but becomes even quicker to handle when you state it as:

"Files under `components/` can only import from `hooks/`, and `components/` itself, and files under `hooks/` can only import from `hooks/` itself."

So what happens when I, as a React junior unaware of all that context, am writing `components/ProfilePage.jsx` and either my reviewer or my linter tells me I can't import `api` because it's not under `components/` or `hooks/`?

It's not a component, so I try to fit it under `hooks/`. Good, but that just shifts the problem: how do I get `api` from `hooks/` if I'm only allowed to import from `hooks/`? Some head-scratching later and questions later, I'm pointed to React Context. I read about it, and I end up creating one, and changing the top-level `index.jsx` to:

```jsx
import api from './api'

ReactDOM.render(
  <ApiProvider value={api}>
    <App />
  </ApiProvider>,
)
```

The same `ApiProvider` can be used for testing without acrobatics, and my reviewer is happy.

By constraining my options, the project structure guided me to a better implementation.

### 4. "Does reloading this produce side effects or just redefinition?"

[TBD]

### 5. "Does this depend on component state?"

[TBD]



## Summary

1. Project structure sets questions people have to answer in order to name, and place, things.
2. Pick questions that invite to think about important properties of the system ("what changes together?" "what architectural rule are all these things following?") and leave behind those that can better be addressed by tools ("where do I find X?").
3. Take into account the team's composition (over time!) and the team members' workflows.

[draft] Structuring projects to guide and inform (for React applications)


The "conventional commit" format advocates prefixing commit messages with keywords to give both humans and machines a quick idea of what's affected:

```text
41e7be2 feat: notify user on new messages
f3b33ab build: enable caching to decrease build times
7501fb2 fix(api): detect invalid query parameter
f5baa4d docs: typo
06e851a perf(ui): optimize loading time
```

The keyword mostly answers the question: "How does this modify existing code?" It **fix**es or **refactor**s it, enhances its **perf**ormance, or adds a new **feat**ure on top. Sometimes, however, it answers the question: "What area does this affect?" It affects the **build** process or the **docs**. Yet other times, it answers both questions by adding a parenthesized "scope": it's a **fix** to the **api**; it's a **perf**ormance enhancement to the **(ui)**.

And if you're fixing a doc? Improving the build's performance?

Having to stop and think in these loaded, non-orthogonal terms, to come up with only moderately descriptive descriptions, feels like too high of a cost for the benefit.

Fortunately, a few extra keystrokes are all that's need to lower the burden on the writer and boost the value for the reader.

## Meet the "-ilities"

A great question one can ask about a change is: "What quality of the system does this directly improve?" ^["Directly" because everything, eventually, aims at improving end-user functionality, business profitability, and similar bottom lines, so those aren't good descriptors at a local level.] A change to the front end might improve usability; one to the back end, observability; another to the docs, understandability.

The book _Building Evolutionary Architectures_ has a long list of these "-ilities":

- accessibility
- adaptability
- administrability
- auditability
- availability
- compatibility
- (...two more pages)

You can use them to better describe, and think about, changes.

## Value-focused, bi-dimensional commit messages

What qualities matter to your team and project? Pick a few, e.g.: **functionality**, **usability**, **devability** ^[Think of devability as of usability for developers: a mixture of code understandability, debuggability, and tool ergonomy.], **stability**, and **security**.

_What_ quality does a change influence? That's the first dimension:

|                                          | what?                              |          |
| ---------------------------------------- | ---------------------------------- | -------- |
| "notify user on new messages"            | makes the application more useful  | **func** |
| "enable caching to decrease build times" | makes the system easier to develop | **dev**  |
| "detect invalid query parameter"         | makes the system more stable       | **stab** |

_How_ does the change improve the quality? That's the second dimension:

|                                          | how?                                    |          |
| ---------------------------------------- | --------------------------------------- | -------- |
| "notify user on new messages"            | adds something that wasn't there before | **feat** |
| "enable caching to decrease build times" | makes something better                  | **enh**  |
| "detect invalid query parameter"         | makes something right                   | **fix**  |

And the original example becomes:

```diff
- 41e7be2 feat: notify user on new messages
+ 41e7be2 usab,feat: notify user on new messages
- f3b33ab build: enable caching to decrease build times
+ f3b33ab dev,enh: enable caching to decrease build times
- 7501fb2 fix(api): detect invalid query parameter
+ 7501fb2 stab,fix(api): detect invalid query parameter
- f5baa4d docs: typo
+ f5baa4d dev,fix: typo
- 06e851a perf(ui): optimize loading time
+ 06e851a usab,enh(ui): optimize loading time
```

This strategy is surprisingly quick to pick up, generates informative descriptions, and gently nudges one to focus on the fundamental question:

_What **value** does this change deliver?_

## References

- [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- [Building Evolutionary Architectures](https://www.thoughtworks.com/books/building-evolutionary-architectures) (O'Reilly, 2017)


Bi-dimensional commit messages


(The following is expanded from my comment at [#storybook/10987](https://github.com/storybookjs/storybook/issues/10987#issuecomment-814273313))

Storybook provides, among other things, a canvas to develop UI components interactively and in isolation.

In any kind of digital work, interactivity comes not only from the absence of manual steps between making a change and seeing the outcome, but also from a feedback loop tight enough to keep your mind on the task: a word processor that takes five seconds to turn a word to italic loses both the semblance and the benefits of interactivity.

Mostly due to the underlying toolchain, that's exactly the situation Storybook can find itself in: a freshly initialized Storybook in a freshly initialized React app can, depending on machine, take more than two seconds to refresh the example button after changing its size from "medium" to "large". I've seen that grow to 5+ seconds even for small (<5k LOC) projects.

These tweaks can bring those numbers down to more workable levels.

## Optimization #1: disable plugins irrelevant to interactive development

Some plugins either create value in areas other than interactive development (such as documentation) or replicate work already done in editors (linting, typechecking). That can take a lot of time. See for example the impact of the Docgen plugin (each timestamp is the time elapsed since the previous line):

```shell
~/.../src: yarn storybook | gnomon
...
   0.1254s   <s> [webpack.Progress] 70% finish module graph ESLintWebpackPlugin
             <s> [webpack.Progress] 70% finish module graph FlagDependencyExportsPlugin
...
   0.0096s   <s> [webpack.Progress] 70% sealing WarnCaseSensitiveModulesPlugin
   1.0012s   <s> [webpack.Progress] 70% sealing React Docgen Typescript Plugin
   1.5016s   <s> [webpack.Progress] 70% sealing React Docgen Typescript Plugin
   2.0022s   <s> [webpack.Progress] 70% sealing React Docgen Typescript Plugin
   2.3576s   <s> [webpack.Progress] 70% sealing React Docgen Typescript Plugin
             <s> [webpack.Progress] 72% basic dependencies optimization
   0.0002s   <s> [webpack.Progress] 72% basic dependencies optimization
```

Plugins can be disabled in `.storybook/main.js`:

```diff
  const { inspect } = require('util')

  module.exports = {
    stories: ['../src/**/*.stories.@(js|jsx|ts|tsx)'],
    addons: [
      '@storybook/addon-links',
      '@storybook/addon-essentials',
      '@storybook/preset-create-react-app',
    ],

+   typescript: {
+     check: false,
+     reactDocgen: false,
+   },
+
+   webpackFinal: async (config, { configType }) => {
+     config.plugins = config.plugins.filter(
+       (p) => !inspect(p).match(/^ESLintWebpackPlugin/),
+     )
+     return config
+   },
  }
```

If you're also publishing the Storybook output as documentation/style guide/showcase, you'll want to do the above only for `NODE_ENV=development`.

## Optimization #2: replace stages of the toolchain with faster ones

_**Note**: This one appears to no longer work with the current release of `esbuild-loader`._

Though primarily meant to reduce cold build times, the `storybook-addon-turbo-build` add-on does further shave time away from interactive refreshes. After installation, enable it with:

```diff
  module.exports = {
    stories: ['../src/**/*.stories.@(js|jsx|ts|tsx)'],
    addons: [
      '@storybook/addon-links',
      '@storybook/addon-essentials',
      '@storybook/preset-create-react-app',
+     {
+       name: "storybook-addon-turbo-build",
+       options: { optimizationLevel: 3 },
+     },
    ],
```

## Optimization #3: get (partially) rid of Webpack

Snowpack is an alternative to Webpack that, through a radically different approach to bundling, provides instant refresh. It can be used in development while leaving the production build to a tried-and-tested Webpack configuration. Unfortunately, Storybook doesn't run in Snowpack yet.

However, if the gains from optimization #1 and #2 are not enough (they weren't for me), it's worth to rethink the problem so that Snowpack can be brought in.

What is the core value of the Storybook canvas for interactive development? We already disabled documentation generation. Do you need widgets to update component props? Do you need add-ons to check the component on different backgrounds and with different locales? Much of that comes into play in a later, shorter "refinement" phase.

If you've adopted Snowpack for your dev environment, and can tolerate the trade-offs, you can create a `/canvas` route that only exists when `NODE_ENV=development`, where you develop components in semi-isolation, re-use Storybook stories, and benefit from instant refresh. The minimum viable version of it is:

`src/Components/App.tsx`:

```tsx
  <Router>
    <Switch>
      <Route ... />
      <Route ... />
      {process.env.NODE_ENV === 'development' && (
        <Route path="/canvas" component={Canvas} exact={true} />
      )}
```

`src/Components/Canvas.tsx`:

```tsx
import React from 'react'
import * as Button from './Button.stories'

export const Canvas: React.FC = () => {
  const StoryBeingWorkedOn = Button.Basic
  return (
    <div>
      <StoryBeingWorkedOn {...StoryBeingWorkedOn.args} />
    </div>
  )
}
```

(It's just as easy to import multiple stories and add a way to switch among them at runtime. You don't want to replicate the entire Storybook canvas, but a little convenience can go a long way.)

The only downsides of this strategy are 1) when you change component args in story files, the changes aren't picked up and you need a full page reload; and 2) if you use decorators, you need to write some more support code.

## References

- [Snowpack](https://www.snowpack.dev/)
- [#storybook/10987](https://github.com/storybookjs/storybook/issues/10987) Track this issue to know if/when Storybook will support Snowpack, superseding most the hacks presented here.
- [storybook-addon-turbo-build](https://storybook.js.org/addons/storybook-addon-turbo-build)


Faster interactive development in Storybook


_Stateful systems that get into inconsistent states are a frequent source of bugs. Making conceptually interdependent data also programmatically interdependent is one way of addressing the root of the problem._

When a bug breaks production, it's good to ask: "Why didn't a test catch that?".

An even better question is "Why did the code build at all?", but it isn't asked nearly as often. Why?

Perhaps because the bug was of the kind that could only be caught at runtime, so it _did_ need a test. Fair enough. Perhaps the bug _could_ be caught at compile time, we just didn't know how. That's easy to fix. Perhaps we adopted a hard and fast "test all the things" rule, eager to consider the matter of code reliability closed and move on to less taxing areas. That's harder to fix.

## Help the compiler to help you

Preventing bugs costs less than catching them, and one class of bugs amenable to prevention is inconsistent states, e.g. a `fetchError` and an `isLoading` variables ending up set at the same time, the rest of the code scrambling through the paradox.

It's a programmer bug before a programming one: give your coworkers a piece of code where that can happen, and you're giving them work (tracking data relations in their heads to make sure they stay consistent) that a machine is able to do. Giving to humans work suitable for the machine is anti-programming.

Such bugs can be often prevented by encoding constraints ("it's not possible for `fetchError` and `isLoading` to be set at the same time") in a form that development tools can understand.

Notably: **types**.

## A before-and-after demonstration

Below is a React hook that exposes the browser's Screen Wake Lock API. A wake lock tells the device not to blank the screen even if the user isn't interacting, e.g. on pages with long texts.

The API is deceptively simple: ask the browser for the lock, get the response. But:

1. the browser may or may not have the capability;
2. the browser doesn't respond synchronously;
3. you get a "sentinel" object that you can later used to release the lock; the releasing is also asynchronous, so you can't just set the sentinel to `null` to signify "lock is inactive": if you null it before release, other code can't know if the lock is really inactive or being released; if you null it after release, other code can't know if the lock is really active or being released.

We want to use the hook like this:

```typescript
const wakeLock = useWakeLock()
wakeLock.setActive(true)
```

Here's a traditional "flag fest" implementation:

```typescript
import { useState } from 'react'

export const useWakeLock = () => {
  const [sentinel, setSentinel] = useState<WakeLockSentinel | null>(null)
  const [isActive, setIsActive] = useState(false)
  const [error, setError] = useState<null | Error>(null)

  const enable = async () => {
    try {
      const sentinel = await navigator.wakeLock.request('screen')
      setSentinel(sentinel)
      setIsActive(false)
      setError(null)
    } catch (err) {
      setError(err)
    }
  }

  const disable = async () => {
    if (!isActive) return
    if (sentinel === null) return
    sentinel.release()
    setSentinel(null)
    setIsActive(false)
    setError(null)
  }

  const setActive = async (wantActive: boolean) => {
    if (!('wakeLock' in navigator)) return
    if (wantActive) {
      enable()
    } else {
      disable()
    }
  }

  return { setActive, error }
}
```

What happens when you inadvertently put the system in an inconsistent state?

```diff
        const sentinel = await navigator.wakeLock.request('screen')
        setSentinel(sentinel)
-       setIsActive(true)
+       setIsActive(false)
        setError(null)
```

The TypeScript compiler:

> _...Silence..._

Here's an implementation where you tell the compiler that interdependent data is, well, interdependent:

```typescript
import { useState } from 'react'

export const useWakeLock = () => {
  const [state, setState] = useState<
    | { type: 'inactive' }
    | { type: 'requesting' }
    | { type: 'active'; sentinel: WakeLockSentinel }
    | { type: 'error'; error: Error }
  >({ type: 'inactive' })

  const enable = async () => {
    if (state.type === 'inactive') {
      setState({ type: 'requesting' })
      try {
        const sentinel = await navigator.wakeLock.request('screen')
        setState({ type: 'active', sentinel })
      } catch (error) {
        setState({ type: 'error', error })
      }
    }
  }

  const disable = async () => {
    if (state.type === 'active') {
      state.sentinel.release()
      setState({ type: 'inactive' })
    }
  }

  const setActive = (wantActive: boolean) => {
    if (!('wakeLock' in navigator)) return
    if (wantActive) {
      enable()
    } else {
      disable()
    }
  }

  const error = state.type === 'error' ? state.error : null
  return { setActive, error }
}
```

What happens when the inconsistent state tries to sneak in?

```diff
          const sentinel = await navigator.wakeLock.request('screen')
-         setState({ type: 'active', sentinel })
+         setState({ type: 'inactive', sentinel })
```

The TypeScript compiler:

```shell
src/hooks/wake-lock.ts:16:20 - error TS2322: Type '"inactive"' is not assignable to type '"active"'.

16         setState({ type: 'inactive', sentinel })
                      ~~~~

[6:22:21 PM] Found 1 error. Watching for file changes.
```

The compiler detects that something is wrong: there is no legal scenario where the state is inactive and the sentinel is set.

(For simplicity, neither implementation handles the case where activation is requested while the lock is releasing, or deactivation is requested while the lock is activating, but it's easy to see which one is better equipped to deal with it without complexity exploding.)

This strategy has been called "making illegal states unrepresentable". I prefer to steal a motto from cooking, where combining seasonal ingredients into something delicious is equal parts description and prescription:

_What grows together, goes together._

## Further reading

- [Compilers and assistants](https://elm-lang.org/news/compilers-as-assistants). Elm's take on compilers is no less interesting than its influential UI architecture.
- [XState](https://xstate.js.org/). Explicit state machines are the next step to rein in complexity. In fact, the second example above is just a less implicit state machine than the first. XState is an excellent framework for state machines in JavaScript, though one with a steep learning curve.


Data that grows together, goes together

_For many tasks in a developer's day, a well-tuned compilation mode is more efficient than reaching for a shell (fewer keystrokes, less context switching) and lighter on the brain than specialized major modes (no new key bindings)._



Emacs's "compilation" mode is not limited to running compilers. You can run bundlers for front-end development, test runners for red-green-refactor TDD, project-wide linters, dev servers for back-end code, Makefile targets, and more.

These notes cover:

- handling commands that produce rich output, and/or clear the screen between multiple runs in a single long-lived session
- choosing efficient key bindings
- browsing command output without leaving the source window
- selecting commands from configurable, contextual lists using the `run-command` package

## Rich output and long-lived commands

Colors are often the only way to notice errors in a flurry of output. Also, commands with a long startup time sometimes offer a "watch" mode to do several runs for a single invocation, and clear the screen between runs.

Compilation mode by default handles neither the control characters that produce colors, nor those that clear the screen. A simple solution that covers many cases is to watch out for a few sequences in the command output using a "compilation filter", and apply them:

```emacs-lisp
(defun local/postprocess-compilation-buffer ()
  (goto-char compilation-filter-start)
  (when (looking-at "\033c")
    (delete-region (point-min) (match-end 0)))
  (ansi-color-apply-on-region (point) (point-max))))

(add-hook 'compilation-filter-hook 'local/postprocess-compilation-buffer)
```

Some commands produce even richer output (progress bars, simple animations) or clear the screen by other means (ANSI codes), requiring more complete terminal capabilities. The `run-command` package (described later) can run such commands in Emacs's `term-mode`, providing those capabilities, while maintaining `compilation-mode` functionality.

## Choosing efficient key bindings

The basic way to run a command in compilation mode is `M-x compile RET <command> RET`, e.g. `M-x compile RET npm start RET`. Next time you invoke `compile`, `<command>` defaults to the previous command, so recompiling effectively becomes `M-x compile RET RET`.

However, recompiling being the more common case, Emacs also provides `M-x recompile`, which reuses the last command right away, without prompting. (You can still edit the command with `C-u M-x recompile`.) So, `recompile` is the better candidate for quick access via a key binding, e.g.:

```emacs-lisp
(global-set-key (kbd "C-c c") 'recompile)
```

With that, `C-c c` will re-run the last command, and `C-u C-c c` will let you edit the command first.

There's an even shorter binding for when the cursor is already in the compilation window: `g`. This is useful when you need to alternate among several complex commands: leave their buffers open, then switch back to them and press `g` as needed.

To stop a command, move the cursor to the compilation buffer and press `C-c k`.

## Browsing command output without leaving the source window

Unless you need to copy a command's output, or re-run a previous command, there's little reason to move the cursor to the command output window; you can more efficiently browse it without leaving the window where you're writing using these key bindings:

- To scroll to the beginning of the output: `M-home`
- To scroll to the end of the output: `M-end`
- To scroll the output forward: `M-next` (where "next" means `PageDown`) or `C-M-v`
- To scroll the output backward: `M-prior` (where "prior" means `PageUp`) or `C-M-S-v`

These bindings are not specific to compilation mode, the work in every situation where there is an "other window". They may seem cumbersome, especially on laptop keyboards which require an additional `Fn` press, but are well worth the effort.

Another flow saver, provided that the command prints errors in a format that compilation mode understands, is `` C-x ` ``, which jumps to errors in the output, in succession. For example, given a TypeScript project with a file `src/App.tsx`:

```tsx
import React from 'react'
import ReactDOM from 'react-dom'

console.helloWorld()
```

Running `M-x compile RET npxj tsc -P . --noEmit --watch` will launch the type checker and print the following:

```shell
[8:13:36 PM] File change detected. Starting incremental compilation...

src/App.tsx:3:9 - error TS2339: Property 'helloWorld' does not exist on type 'Console'.

3 console.helloWorld()
          ~~~~~~~~~~

[8:13:36 PM] Found 1 error. Watching for file changes.
```

If you now hit `` C-x ` ``, the `App.tsx` buffer will appear, with the cursor on `helloWorld`.

## Selecting commands from configurable, contextual lists using the `run-command` package

If, in a given context, you tend to run the same small subset of commands over and over, (e.g. `npm test:watch` in a JavaScript project, `make clean` in a C project, `hugo server --buildDrafts` in a Hugo blog), that's a prime case for automation.

The `run-command` Emacs package (by the author) lets you write simple command lists, bring them up depending on the context, and select them with autocompletion:

![run-command demo](https://raw.githubusercontent.com/bard/emacs-run-command/master/demo.gif)

See the [run-command GitHub page](https://github.com/bard/emacs-run-command) for setup and usage instructions.

## Additional customization

- To have the window scroll as output comes in, set `compilation-scroll-output` to `t`.
  - The default scrolling mode often leads to a large amount of empty space at the bottom of the window; this is due to "recentering". To prevent that and display as much output as possible, set `scroll-conservatively` to `101` (see `M-x describe-variable scroll-conservatively RET` for why).
- To have it scroll, but stop scrolling when the first error is detected, set `compilation-scroll-output` to `error`.

Lightweight external command integration in Emacs via compilation mode

_Functions that ask for more than they need trick the developer into writing unnecessary code. At best, this slows down future development; at worst, it harms runtime qualities: of the present system._



Imagine that you inherited this code:

```ts
const post = await api.fetchPostById('abc123')
// => {
//  id: 'abc123',
//  title: 'The dangers of greedy functions',
//  authorName: 'Max'
// }

const postUrl = getPublishedUrl(post)
```

One day, the back-end team changes the API so that author data is exposed through a separate endpoint. Your IDE tells you that `getPublishedUrl` needs an argument of type `{ id: string, title: string, authorName: string }`, so you make the necessary changes:

```ts
const post = await api.fetchPostById('abc123')
const author = await api.fetchAuthorByPostId(post.id)

const postUrl = getPublishedUrl({
  ...post,
  authorName: author.name,
})
```

Tests succeed, production works, all is good.

Some time later, you look into a file called `post‑utils.ts`:

```ts
type Post = {
  id: string
  title: string
  authorName: string
}

function getPublishedUrl(post: Post): string {
  return '/posts/' + post.id
}
```

It turns out that `getPublishedUrl` didn't need author data at all.

And yet, it led you into adding complexity, and even causing some performance degradation (an extra network request).

## The problem

_Surely you could have peeked at `getPublishedUrl` implementation sooner?_

For every `getPublishedUrl` in a contrived example, dozens exist in a real code base. But impracticality aside: why establish contracts, if you're required to look at implementations? Function signatures and object interfaces are there to save you that work. If you still have to do it, they're not doing their job right.

In fact, the real problem (at least at the _program level_) is that `getPublishedUrl` is lying. This:

```ts
function getPublishedUrl(post: Post): string
```

Says: _My name is `getPublishedUrl` and, to do my job, I need a post (i.e. a map of id, title, and authorName)."_ A greedy lie.

The solution (again, at least at the _program level_) is for functions to only ask for what they really need:

```ts
function getPublishedUrl(postId: string): string
```

(Unless they really need 90% or so of a large, complex type, in which case there's little point in leaving out the 10%.)

It's harder to pinpoint the problem at the _programmer level_, i.e. what led someone to choose to write that code. Most likely it was a well-intentioned quest for simplicity and readability. Unfortunately, simplicity that isn't backed by truth looks the part, and quietly invites its opposite.

Hello, I’m Massimili...

Latest notes