
Something that frustrates even the most honest effort to enact an efficient testing strategy is misunderstanding this:

1. prepare **input**
2. run computation
3. assert **output**

For this:

1. prepare **arguments**
2. run function
3. assert **return** value

But arguments aren't the only way data gets into a function. This is input:

```ts
const activateUser = async (userId: string) => {
  // ...

  const blockedUserIds: string[] =
    process.env.BLOCKED_USERS?.split(",") ?? [];

  // ...
};
// ...
```

This, too, is input:

```ts
const activateUser = async (userId: string) => {
  // ...

  const currentTimeMs = Date.now() / 1000;

  // ...
};
```

Return values aren't the only way data gets out of a function, either. This is output:

```ts
const db = new DB(process.env.DB_URI);

const activateUser = async (userId: string) => {
  // ...

  await db.updateUser({ userId, active: true });

  // ...
};
```

Faced with testing something like `activateUser()`, where the task can't be fully determined through arguments and its outcome cannot be fully verified via the return value, people usually resort to one of:

- mocking modules, at the price of deep doubts ("...does this mean I'm testing implementation details...") and fragile acrobatics (should your system's testability really depend on how good the test framework can hack into the runtime's import mechanism?)
- presenting a heavy, as-realistic-as-possible environment to the test (network services, browser DOM, ...), at the price of slowness in writing the test and slowness in running it.[^1]

The real solution is simpler, more robust, faster, battle-tested, and almost smells like [pure functions](https://en.wikipedia.org/wiki/Pure_function); its only sin is hiding behind a name that gives no clue about its benefits: "dependency injection".

Which is why I like to say "make input/channels channels explicit" instead:

```ts
const io = { env, db, clock };
const userId = "user-123";
const result = await activateUser(userId, io);
```

The corresponding test is crystal-clear and runs in milliseconds:

```ts
test("activating a user sets the active flag in the corresponding database record", async () => {
  // prepare input

  const io = {
    env: {
      get: jest.fn(() => "user-456,user-789"),
    },
    db: {
      updateUser: jest.fn(() =>
        Promise.resolve({
          id: "user-123",
          email: "bob@example.com",
        }),
      ),
    },
    clock: {
      getTime: jest.fn(() => 1704067200000),
    },
  };
  const userId = "user-123";

  // run task

  const result = await activateUser(userId, io);

  // assert output

  expect(result.ok).toBe(true);
  expect(io.clock.getTime).toHaveBeenCalled();
  expect(io.env.get).toHaveBeenCalledWith("BLOCKED_USERS");
  expect(db.updateUser).toHaveBeenCalledWith({
    id: "user-123",
    active: true,
  });
});
```

**But how do I know that the value will actually be written to the database?**

You write a test for `db.updateUser`.

Remember, when you test `activateUser`, you test that it fulfills its contract. Its contract involves sending out a call to `db.updateUser` with the correct arguments. What happens afterwards it's `db`'s business.

## Summary

Make all input and output channels explicit (or, if you prefer the phrasing, inject dependencies ruthlessly) via a dedicated function argument, and enjoy tests that are simpler to understand and faster to run.

[^1]: It's tempting to write both kinds of slowness off as "it's a human problem, just work harder", but human resources (in the sense of attention, energy, etc.) are as finite as computing resources, and misusing them leads nowhere good. A system where tests take longer to write is necessarily less robust or less featureful, because the effort that could have gone into writing features has to go into overcoming the testing friction. A system where tests take longer to run is necessarily less robust or less featureful, because the attention that could have improved the system is eroded by frequent, forced ten-second (optimistically) breaks waiting for tests to pass.


Faster, simpler tests with explicit IO channels

When I open a file and see `process.env` used like this, I know I'm in trouble:

```tsx
function Cart() {
 useEffect(() => {
 fetch(`${process.env.REACT_APP_API_BASE_URL}/cart`).then(
 (res) => {
 // ...
```

Why? Well, suppose I offer you a strategy to make your app work locally, yet break in production, in a way that your users will notice but you won't, is hard to debug, and is guaranteed to happen at some point because the root cause is unsolvable.

Assuming you're not the villain of this story... do you take the offer?

If the answer is "no", then I hope you're not using `process.env` as in the example above, because it's got all the ingredients of trouble:

- if you forget to set `REACT_APP_API_BASE_URL` in CI, the app breaks only in production
- unless you manually test after each deployment, you won't find out that the cart is broken until users report it
- users will say "the cart doesn't load" and the console message they'll (hopefully) report is `Uncaught SyntaxError: Unexpected token '<', "<!doctype "... is not valid JSON`, both of which are a long shot away from "I wonder if somebody forgot to set an environment variable"
- you don't introduce variables often enough for the task of setting them in CI to become a habit, so it will always depend on conscious attention, which you'll inevitably lack at some point.

The issues with that code don't stop there (hardcoded strings are a fragile way of representing routes, there are much better alternatives to the combination of `useEffect` and `fetch`, ...) but, in my experience, `process.env` is the one that most consistently flies under the radar.

Below, I'll show a way of removing this class of bug from your process. But first, let's dig a little deeper, because the "fix" that most of us rush toward in such a situation is a worse cure than the disease.

## A sustainable definition of "fixed"

_It's late afternoon, the cart has been broken for most of the day, but you just found out. Business lost revenue, long-time users are frustrated, first-time users are gone, and you've canceled your evening plans. After much digging, you finally realize what's going on. You set that damn variable and redeploy..._

Well done! By all means go ahead with those evening plans. But don't declare anything "fixed" just yet.

Teams like to debate the "definition of done", but I've seldom been in a conversation about the "definition of fixed". Yet software keeps breaking in ways it already broke, and nobody connects the dots.

"Fixed" isn't putting back up the sign that was knocked down by a gust of wind, only to find it down again two weeks later. "Fixed" is anchoring the sign so that it stays up. "Fixed" is making the change that lets you say not only "it works now" but also "it will work tomorrow". Other definitions are unsustainable: code bases grow, time does not. If we let weak spots grow along with code, soon we spend all our time on dealing with breakage rather than adding value.

If setting that environment variable didn't really fix the problem, then, what does? And what problem are we really talking about anyway?

Forgetting to set an environment variable isn't a problem — _it's certainty_. And so is the failure of any process that relies on something inherently fallible like human attention. Not recognizing certain failure and not planning for it, that is the true problem.

It's ironic: we plan for the failure machines and networks, which work fine 99.99% of the time, yet when it comes to memory, which for most people fails many times a day, the response is "let's remember next time".

The fix isn't to try harder, it is to reframe the problem so it be can attacked with what machines are good at rather than what people are bad at.

## Centralize, parse, anticipate

We got into trouble because:

1. we were reading environment variables all over the place
2. didn't see something break until users told us
3. received obscure error messages

So let's define the goal as:

1. read variables all in one place
2. if any is invalid, fail the build so that the bug never gets deployed
3. produce errors that point at the exact problem

Start by creating a `config.ts` to centralize all configuration code so that the rest of the app doesn't need to worry about it. A React component has no business dealing with the environment anyway.

Treat thes variables as external data (because they are) and have them go through security before letting them on board:

```ts
// config.ts
import { createContext } from "react";
import { z } from "zod";

export interface Config {
 api: {
 baseUrl: string;
 };
}

export const parseConfig = (
 envVars: Record<string, string | undefined>,
): Config => {
 const envSchema = z.object({
 REACT_APP_API_BASE_URL: z.string().url(),
 });

 const env = envSchema.parse(envVars);

 return {
 api: { baseUrl: env.REACT_APP_API_BASE_URL },
 };
};

export const ConfigContext = createContext<Config | null>(
 null,
);
```

In the `index.tsx` entry point, read the configuration and make it available to the rest of the app via React Context. (Bonus: this will also save you from mocking `process.env` and other acrobatics in tests and Storybook.)

```diff
// index.tsx
 import reportWebVitals from "./reportWebVitals";
+ import { parseConfig, ConfigContext } from "./config";
+
+ const config = parseConfig(process.env);

 const root = ReactDOM.createRoot(
 document.getElementById("root") as HTMLElement,
 );
 root.render(
 <React.StrictMode>
- <App />
+ <ConfigContext.Provider value={config}>
+ <App />
+ </ConfigContext.Provider>
 </React.StrictMode>,
 );
```

Adapt `Cart.tsx`:

```diff
// Cart.tsx
- import { useState, useEffect } from "react";
+ import { useState, useEffect, useContext } from "react";
+ import { ConfigContext } from "./config";

 export const Cart: React.FC = () => {
+ const config = useContext(ConfigContext);
+ if (config === null) throw new Error("Config not set");
+
 const [cart, setCart] = useState(null);

 useEffect(() => {
- fetch(`${process.env.REACT_APP_API_BASE_URL}/carts/1`)
+ fetch(`${config.api.baseUrl}/carts/1`)
 .then((res) => res.json())
 .then(setCart)
 .catch(console.error);
- }, []);
+ }, [config]);
```

Finally, add a `package.json` script that will call `parseConfig` in CI. For example using [tsx](https://www.npmjs.com/package/tsx):

```diff
 "scripts": {
+ "validate-env": "tsx -e 'import { parseConfig } from \"./src/config\"; parseConfig(process.env)'",
 "start": "react-scripts start",
```

Now, when a build variable is unset or invalid, you get an error in CI, not in production, and it tells you exactly what's wrong:

```sh
$ REACT_APP_API_BASE_URL=foo npm run validate-env
...
ZodError: [
 {
 "validation": "url",
 "code": "invalid_string",
 "message": "Invalid url",
 "path": [
 "REACT_APP_API_BASE_URL"
 ]
 }
]
```

See the [full code example on Github](https://github.com/bard/build-time-frontend-var-validation-example/) (also covers the advanced bits below).

## Advanced

### _Really_ disconnecting from `process.env`

A `config.REACT_APP_API_BASE_URL` that's guaranteed to be valid is better than a fickle `process.env`, but it might not map to what's optimal for calling code. For example, we might want feature flags as an array of literals rather than a raw string that must be split every time:

```diff
 const SearchResults: React.FC = () => {
 const config = useContext(ConfigContext);
- if (config.REACT_APP_ENABLED_FEATURES.split(",").includes("infinite-scroll")) {
+ if (config.enabledFeatures.includes("infinite-scroll")) {
 // ...
```

To achieve that, validate environment variables like before, but instead of returning them verbatim, use them to fill a domain-specific `Config` object:

```ts
// config.ts

interface Config {
 enabledFeatures: Array<"infinite-scroll" | "dark-mode" | "share-button">
}

export const parseConfig = (
 envVars: Record<string, string | undefined>,
): Config => {
 const envSchema = z.object({
 REACT_APP_ENABLED_FEATURES: z.string().optional()
 });

 const env = envSchema.parse(envVars);

 const enabledFeatures = env.REACT_APP_ENABLED_FEATURES
 ? z
 .array(z.enum(["infinite-scroll", "dark-mode", "share-button"]))
 .parse(env.REACT_APP_ENABLED_FEATURES)
 : [];

 return {
 enabledFeatures
 }
})
```

### Default configuration values

To prevent `.env` fatigue, you can set default values:

```ts
export const parseConfig = (
 envVars: Record<string, string | undefined>,
): Config => {
 const envSchema = z.object({
 REACT_APP_API_BASE_URL: z
 .string()
 .url()
 .default("https://example.com/api"),
 });
```

### Dependent configuration values

Let's say that some environment variables only make sense together, and you want them all set or none at all. Here's how to do it:

```ts
interface Config {
 api: {
 baseUrl: string;
 };
 datadog?: {
 applicationId: string;
 site: string;
 };
}

export const parseConfig = (
 envVars: Record<string, string | undefined>,
): Config => {
 const apiEnvSchema = z.object({
 REACT_APP_API_BASE_URL: z.string().url(),
 });

 const dataDogEnvSchema = z
 .object({
 REACT_APP_DATADOG_APPLICATION_ID: z.string(),
 REACT_APP_DATADOG_SITE: z.string(),
 })
 .or(
 z.object({
 REACT_APP_DATADOG_APPLICATION_ID: z.undefined(),
 REACT_APP_DATADOG_SITE: z.undefined(),
 }),
 );

 const fullEnvSchema = apiEnvSchema
 .and(dataDogEnvSchema);

 const env = fullEnvSchema.parse(envVars);

 const datadog = env.REACT_APP_DATADOG_APPLICATION_ID
 ? {
 site: env.REACT_APP_DATADOG_SITE,
 applicationId: env.REACT_APP_DATADOG_APPLICATION_ID,
 }
 : undefined;

// ...
```

### "My teammates keep forgetting about this and end up using `process.env` in component code again"

Add this to `.eslintrc.json`:

```json
{
 "rules": {
 "no-process-env": "error"
 }
}
```

## Summary

- Code bugs can be just symptoms of process bugs.
- Breakage due to a missing env variable reveals that we're giving people tasks that are suited to machines (a process bug if there ever was one).
- Reframe such tasks and do give them to machines.

## Credits

_Thanks to [Atris](https://twitter.com/atris_eth) for many productive conversations on the topic, [Sebastién](https://thisweekinreact.com/) for his review and additional perspectives, and [Kyle](https://twitter.com/kylecureau) (as well as his excellent product [Crone](https://www.crone.ai/)) for the precious feedback on writing._

The process.env frontend time bomb (plus: a sustainable definition of “fixed”)

_NextJS's conventions are not TDD-friendly, but with some work and a little structure we can enjoy a smooth red-green ride._

Let's say we need a NextJS API route that gives us dictionary definitions. The route handler should validate the request, query a 3rd-party dictionary API, extract desired data from the response, and relay it to the front end.

We want a fast feedback loop and low chances of future regressions, so we decide to develop it test-first.

## 0/4 Prep

Install the [node-mocks-http](https://www.npmjs.com/package/node-mocks-http) package.

We'll assume a working `jest` setup. Start it in watch mode with:

```sh
npx jest --watch
# or
pnpm jest --watch
```

## 1/4 Basic test

Create the simplest possible test in `src/server/api/definitions.test.ts`:

```ts
import { NextApiRequest, NextApiResponse } from "next";
import {
 createMocks,
 createRequest,
 createResponse,
} from "node-mocks-http";
import { createHandler } from "./definitions";
type ApiRequest = NextApiRequest &
 ReturnType<typeof createRequest>;
type ApiResponse = NextApiResponse &
 ReturnType<typeof createResponse>;

describe("/api/definitions", () => {
 const getDeps = () => {
 return {
 fetch: jest.fn(),
 env: {
 DICTIONARY_API_KEY: "abcdef123456",
 },
 };
 };

 test("responds with error if no query was provided", async () => {
 const deps = getDeps();
 const { req, res } = createMocks<
 ApiRequest,
 ApiResponse
 >({
 method: "GET",
 url: "/api/definitions",
 });

 const handler = createHandler(deps);
 await handler(req, res);

 expect(res.statusCode).toBe(400);
 });
});
```

### Things to note

1. `node-mocks-http` allows to execute handlers in the test environment without having to launch and control a server as part of the test. Running a server would increase accuracy (we have no assurance that the mocking library 100% matches the server's request processing) but decrease simplicity (we'd need to tightly manage the server lifecycle, ports, and state to ensure tests stay self-contained), control (harder to test for error conditions), and speed (the overhead of launching servers might slow down large test suites). We especially want test to run fast so that our attention doesn't have a chance to drop, se we accept the trade off in favour of speed and use a mocking library.

2. Instead of expecting `./definitions` to export a handler _function_, we expect it to export a handler _factory_:

```ts
import { createHandler } from "./definitions";
```

We prefer a factory to a function because it allows configuration to be provided at the call site instead of being statically defined in the `./definitions` module. This lets us configure the handler depending on the environment (dev/test/prod) by just passing a parameter object instead instead of doing `jest.mock` acrobatics.

Also note that we begin to state expectations long before the first call to `test()`.

3. The configuration we pass to the handler factory consists of external dependencies:

```ts
const getDeps = () => {
 return {
 fetch: jest.fn(),
 env: {
 DICTIONARY_API_KEY: "abcdef123456",
 },
 };
};
```

_External_ can be thought of as "anything the function doesn't know by looking at the arguments".

To justify why we pass `env` and even `fetch` as external dependencies, let's look at the handler function signature:

```ts
export declare type NextApiHandler<T = any> = (
 req: NextApiRequest,
 res: NextApiResponse<T>,
) => unknown | Promise<unknown>;
```

- Can the handler know the content of `process.env` by just looking at `req` or `res`? No → pass `env` as part of the configuration.
- Can the handler know the content of 3rd-party APIs by just looking at `req` or `res`? No → pass a `fetch` implementation to the factory as configuration.

If the 3rd-party dictionary service provided an SDK (or if we were willing to write a minimal one ourselves), the above could become:

```ts
const getDeps = () => {
 return {
 dictionaryClient: { lookup: jest.fn() },
 };
};
```

4. We don't place files in the usual `src/pages/api` folder so we don't need to worry about Next misinterpreting test files as handlers, nor we need to modify `next.config.js` to prevent such misinterpretation.

(Eventually we'll place just a minimal file there to configure the handler for dev/prod and activate the route.)

## 2/4 Basic implementation

Write the simplest implementation of `src/server/api/definitions.ts` capable of making the test pass:

```ts
import crossFetch from "cross-fetch";
import { NextApiHandler } from "next";

export const createHandler =
 ({
 fetch,
 env,
 }: {
 fetch: typeof crossFetch;
 env: {
 DICTIONARY_API_KEY: string;
 };
 }): NextApiHandler =>
 async (req, res) => {
 if (!("q" in req.query)) {
 res.status(400).end();
 return;
 }
 };
```

## 3/4 Iterate

From here on it's the usual TDD flow:

1. write another test;
2. write the code that makes it pass;
3. occasionally stop and improve other qualities of the code (understandability, performance, ...) while using tests to confirm that nothing broke.

For example, work on the invocaiton of the 3rd-party API, we could start by writing the following test:

```ts
test("invokes the dictionary API", async () => {
 // Setup
 const deps = getDeps();
 deps.fetch.mockResolvedValue({
 async json() {
 return {
 word: "cat",
 definition: "A mammal popular in memes",
 usage: { queriesLeft: 100 },
 };
 },
 });
 const { req, res } = createMocks<
 ApiRequest,
 ApiResponse
 >({
 method: "GET",
 url: "/api/definitions",
 query: { q: "cat" },
 });
 const handler = createHandler(deps);

 // Execution
 await handler(req, res);

 // Assertion
 expect(res.statusCode).toBe(200);
 expect(res._getJSONData()).toEqual({
 definition: "A mammal popular in memes",
 });
 expect(deps.fetch).toHaveBeenCalledWith(
 "https://dict.example.com/api",
 {
 headers: { "x-api-key": "abcdef123456" },
 },
 );
});
```

And the following implementation:

```ts
import crossFetch from "cross-fetch";
import { NextApiHandler } from "next";

export const createHandler =
 ({
 fetch,
 env,
 }: {
 fetch: typeof crossFetch;
 env: {
 DICTIONARY_API_KEY: string;
 };
 }): NextApiHandler =>
 async (req, res) => {
 if (!("q" in req.query)) {
 res.status(400).end();
 return;
 }

 const dictRes = await fetch(
 "https://dict.example.com/api",
 {
 headers: {
 "x-api-key": env.DICTIONARY_API_KEY,
 },
 },
 );
 const data = await dictRes.json();

 res.json({ definition: data.definition });
 };
```

## 4/4 Make the handler available to Next

Create `src/pages/api/definitions.ts`:

```ts
import crossFetch from "cross-fetch";
import { createHandler } from "src/server/api/definitions";

const handler: NextApiHandler = async (req, res) => {
 const { DICTIONARY_API_KEY } = process.env;

 return createHandler({
 fetch: crossFetch,
 env: { DICTIONARY_API_KEY },
 })(req, res);
};

export default handler;
```

### Things to note

1. We configure the handler for production and development environments.
2. We could have written a shorthand `export default (req, res) => { return createHandler... }` but TypeScript doesn't (at least at the time of writing) allow typing default exports, so we go for the extra variable.
3. There's no real need to make the handler async here, but, in some cases, a dependency of the handler might have to be initialized asynchronously (e.g. `const db = async createDbConnection()`).

## 5/4 What next

If you'd like me to cover what's left for a production-grade API route handler (validation, authorization, error handling, ...), let me know via [mail](mailto:hyperstruct@gmail.com?subject=Hello) or [twitter](https://twitter.com/MaxIsBuilding).

If you'd like to read more about the why's of TDD, check out Kent Beck's classic [Aim, Fire](https://users.csc.calpoly.edu/~djanzen/setopics/testing/IEEESoftwareBeck.pdf) or [some of my own notes](/notes/cafe-tech-chat-tdd-and-vs-testing).

Test-driven development of NextJS API routes


_I recently started hosting AMA-style cafe chats with developers at various stages of their journey, about software engineering and anything else within my technical experience. Below are some notes from the chat on Nov 16th, 2022._

Imagine you feed this to your front-end i18n library:

```ts
const i18enStrings = {
  "home.title": "Life, universe, and everything",
  "home.subtitle": "A guide",
};

const i18itStrings = {
  "home.title": "La vita, l'universo, e tutto quanto",
  "home.subtitl": "Una guida",
};
```

## Challenge #1: can you guess what happens when the user switches from English to Italian?

On the user side, either of these is likely to happen:

1. the typo hides the second string from the i18n library and the site lacks the subtitle
2. the i18n library is smart enough to fall back on a default, and the subtitle stays in English

What's more interesting though is what happens — or rather **doesn't** happen
— on the developer side:

1. the dev environment doesn't crash;
2. the build doesn't break;
3. tests don't break;
4. the pre-production environment doesn't crash;
5. the production environment doesn't crash.

## Challenge #2: how do you fix this?

And _what_ do you fix?

The typo isn't the problem. The problem is that all the checkpoints on the way to prod are blind to the bug.

Ask yourself:

1. how can I, at the very least, make future instances of this problem crash prod?
2. better: how can I make them crash preprod?
3. better: how can I make them break tests?
4. better: how can I make them break the build?
5. better: how can I make them annoy the IDE to the point that it shouts at me even before I save?

In other words: picturing an imaginary line from dev to prod going from left to right, you want shift detection and prevention/correction as left as possible. [^1]

## Challenge #3: how do you shift this bug to the left?

If you can write a test to guard against a problem, good.

If you can write a type, great.

Here it's such a small diff that someone unaware of all the above could easily dismiss it as minor. Assuming English strings are the authoritative ones regarding keys:

```diff
  const i18enStrings = {
    "home.title": "Life, universe, and everything",
    "home.subtitle": "A guide"
  }

+ type I18Strings = typeof i18enStrings

- const i18itStrings = {
+ const i18itStrings: I18nStrings = {
    "home.title": "La vita, l'universo, e tutto quanto",
    "home.subtitl": "Una guida"
  }
```

Which results in:

```sh
error TS2322: Type '{ "home.title": string; "home.subtitl": string; }' is not assignable to type '{ "home.title": string; "home.subtitle": string; }'.
  Object literal may only specify known properties, and '"home.subtitl"' does not exist in type '{ "home.title": string; "home.subtitle": string; }'.

8   "home.subtitl": "Una guida",
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~
```

[^1]: [DevOps: Shift Left to Reduce Failure](https://devops.com/devops-shift-left-avoid-failure/)


How two lines of TypeScript can make a bug go from “undetected in production” to “impossible to introduce”


_I recently started hosting AMA-style cafe chats with developers at various stages of their journey, about software engineering and anything else within my technical experience. Below are some rough & rapid-fire notes from the chat on Oct 21st, 2022._

## “I never did any formal design, where can I start?”

**A dead-easy yet high-return strategy**: take a core functionality of the system (e.g. "purchasing an item") and jot down all facts that must be true for it to work as intended. Examples:

- "an item has `id: string`, `name: string`, and `stock: number` fields"
- "items are stored in a database table"
- "the list of items is exposed via the `/api/items` REST endpoint"
- "when an item is purchased, the `stock` field is decreased"

Don't worry about mixing abstraction levels; better have them all mixed than get sidetracked into the problem of how to arrange such a document and give up.

**Bonus #1**: rules can be reused as test descriptions for various kinds of tests [^1]. Examples:

- "when user orders item, logistics service is notified" → e2e
- "when button is clicked, API request is sent" → integration
- "item to cart data structure updates total figure" → unit

**Bonus #2**: in a cross-functional team, the list helps to keep alignment with product owners and business analysts; it's easier for them to look at higher-level rules and say "that's right" or "that's wrong" than clicking around in the live application until something feels off. [^2]

**Informal diagrams are plagued by ambiguity**. Want to draw boxes and arrows? Go for it, but keep asking: "In how many ways can this be interpreted?" If the answer is anything but "one", keep working on it. Example:

```
[Service A] ---(item)---> [Service B]
```

Does the arrow mean "sends to?" or "request from?" or "responds with after being requested?"

Mitigate by religiously labeling every edge with verbs:

```
[Service A] ---(requests item from)---> [Service B]
```

**When you're ready for formal diagrams, start with [sequence diagrams](https://en.wikipedia.org/wiki/Sequence_diagram).** They're likely to serve you well for a long time, being high on expressiveness (they capture the flow of time along with various modalities of communication) and low on ambiguity.

[^1]: And [that's not a coincidence](/notes/cafe-tech-chat-tdd-and-vs-testing).
[^2]: Good formalisms tend to do that. There's a great anecdote in [Harel's paper about the development of state charts](https://www.wisdom.weizmann.ac.il/~harel/papers/Statecharts.History.pdf): _in the midst of one of the sessions at the IAI the blackboard contained a rather complicated statechart that specified the intricate behavior of some portion of the Lavi avionics system. [...] There was a knock on the door and in came one of the air force pilots from the headquarters of the project. He was a member of the "customer" requirements team, so he knew all about the intended aircraft (and eventually he would probably be able to fly one pretty easily too…), was smart and intelligent, but he had never seen a state machine or a state diagram before, not to mention a statechart. He stared for a moment at this picture on the blackboard, with its complicated mess of blobs, blobs inside other blobs, colored arrows splitting and merging, etc., and asked "What's that?" One of the members of the team said "Oh, that's the behavior of the so-and-so part of the system, and, by the way, these rounded rectangles are states, and the arrows are transitions between states". And that was all that was said. The pilot stood there studying the blackboard for a minute or two, and then said, "I think you have a mistake down here, this arrow should go over here."_


“I never did any formal design, where can I start?”

## Install Eldev

Eldev is `make` (and then some) for Emacs Lisp development.

```sh
curl -fsSL https://raw.github.com/doublep/eldev/master/webinstall/eldev | sh
```

The `eldev` executable will be installed in `~/.local/bin`.

## Create a project directory

```sh
~/projects$ mkdir emacs-simple-calculator
~/projects$ cd emacs-simple-calculator
~/projects/emacs-simple-calculator$ git init
```

## Create the main project file

`simple-calculator.el`:

```lisp
;;; simple-calculator.el --- Simple Calculator -*- lexical-binding: t -*-

;; Copyright (C) 2022 Some One

;; Author: Some One <someone@example.com>
;; URL: https://github.com/someone/emacs-simple-calculator
;; Version: 0.1.0
;; Package-Requires: ((emacs "27.1"))
;; Keywords: examples

;; This file is not part of GNU Emacs

;; This file is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation; either version 3, or (at your option)
;; any later version.

;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
;; GNU General Public License for more details.

;; For a full copy of the GNU General Public License
;; see <https://www.gnu.org/licenses/>.

;;; Commentary:
;;
;; [This will contain a description of the package]

;;; Code:

(defun simple-calculator-add (a b)
 (+ a b))

(provide 'simple-calculator)

;;; simple-calculator.el ends here
```

## Initialize Eldev project

```sh
~/projects/emacs-simple-calculator$ eldev init --non-interactive
Created file ‘Eldev’ for this project
Modified file ‘.gitignore’
```

In the file `Eldev`, uncomment the `gnu-elpa` and `melpa` sources:

```lisp
; -*- mode: emacs-lisp; lexical-binding: t -*-

;; Uncomment some calls below as needed for your project.
(eldev-use-package-archive 'gnu-elpa)
;(eldev-use-package-archive 'nongnu-elpa)
(eldev-use-package-archive 'melpa)
```

## Add a test file

`simple-calculator-test.el`:

```lisp
(require 'simple-calculator)
(require 'buttercup)

(describe "A calculator"
 (it "adds two numbers"
 (expect (simple-calculator-add 1 2) :to-be 3)))
```

## Run tests

```sh
~/projects/emacs-simple-calculator$ eldev test
Loading /home/user/projects/emacs-simple-calculator/simple-calculator.el (source)...

[1/1] Installing package ‘buttercup’ (1.28) from ‘melpa-stable’...
Running 1 specs.

A calculator
 adds two numbers (0.05ms)

Ran 1 specs, 0 failed, in 0.10ms.
```

## Run tests in watch mode

Using [watchexec](https://github.com/watchexec/watchexec):

```sh
~/projects/emacs-simple-calculator$ watchexec eldev test
```

If you're using FlyCheck with ELisp, a temporary file gets written to the current folder whenever flycheck runs, which [cannot be moved to the system temporary directory](https://github.com/flycheck/flycheck/issues/92#issuecomment-14663824). Prevent unnecessary test runs with:

```sh
~/projects/emacs-simple-calculator$ watchexec --ignore 'flycheck_*' eldev test
```

Emacs package quickstart with Eldev

_I recently started hosting AMA-style cafe chats with developers at various stages of their journey, about software engineering and anything else within my technical experience. Below are some rough & rapid-fire notes from the chat on Oct 21st, 2022._

## Getting started

You can test at various levels, from an individual function that knows nothing about browsers to an entire application running in a real browser and hitting real network services.

Testing small or medium-sized interactions in a simulated browser environment is a good middle ground — approachable yet useful.

Pick a basic interaction and write it down:

> "when button 'About' is clicked, modal with information appears".

Stick to interactions where the UI is both the input and the output channel.

<details>
 <summary>About “channels”...</summary>

- A test's "input channel" and "output channel" are terms I made up; probably better ones exist; related: "test boundaries".
- When testing a pure function, input and output channels are obvious: arguments in, return value out.
- When testing a callback function, it's a little less obvious: arguments in, callback argument out.
- When testing a UI, it's even less obvious: user action in, UI change out; or: user action in, API request out; or: push notification in, UI change out; and so on.
</details>

[Install & set up](https://testing-library.com/docs/react-testing-library/setup) `jest`, `jest-environment-jsdom`, `@testing-library/react`, and `@testing-library/jest-dom`.

Pick a component high enough in your app's tree to support the chosen interaction. Don't be afraid to use `<App/>` itself.

Write a `<component-you-picked>.test.tsx` file containing a `test("when button 'About' is clicked'...)` where you `render()` the component, `user.click()` the button, and `expect()` that the intended change happened in the UI.

The UI change might happen on the next iteration of the JavaScript event loop; use `await waitFor(() => { expect(...) })` to account for it.

Use `screen.debug()` to inspect the state of the DOM at any point of the test.

Continue with interactions that involve the network and repeat.

<details>
 <summary>Choosing output channels when network interactions are involved...</summary>

- "When button is clicked, API request is sent" → the network is the output channel.
- "When button is clicked, table gets populated with server data" → the UI is the output channel.
- "When button is clicked, API request is sent and table gets populated with the returned data" → nothing bars you from looking at multiple output channels.
 </details>

Arrange something that handles the network request **within the test environment**.

- Since you're getting started, use [nock](https://github.com/nock/nock) — it's the least magical option and forces you to understand what's going on.
- `nock()` a request and its response, `render()` the component, `user.click()` the button that triggers the network request, `expect()` the UI changes and/or that the request was made.
 - **if you're using Node 18** and your frontend code calls the global `fetch` instead of one imported from [isomorphic-fetch](https://www.npmjs.com/package/cross-fetch) or [cross-fetch](https://www.npmjs.com/package/isomorphic-fetch)): Node 18 introduces a global `fetch`, so your tests might end up invoking that, and [nock can't mock it yet](https://github.com/nock/nock/issues/2336). Fix by adding `import "cross-fetch/polyfill"` in a jest `setupFile` or by importing `fetch` from `cross-fetch` in application code rather then relying on the global. Alternatively, you can use the mocking facilities of Node 18's new HTTP client [undici](https://undici.nodejs.org/en/), but those are unlikely to match `nock`'s for a while.

## Getting better

As you become more comfortable with the testing machinery, explore other ways of simulating parts of your system inside the test environment:

- Your API service is written in Node and lives next to the React app as part of a monorepo? Launch the actual server in `beforeAll()`, point your components to it by passing them an `apiBaseUrl` prop or context value, and let them interact with the real (albeit sandboxed) thing.
- If your API service is an external? Mock it with something like [MSW](https://github.com/mswjs/msw).
- `jest.mock()` and `spyOn()` things that your components affect in ways other than the network, e.g. third-party SDKs
- Zero in on non-UI parts of your applications and test them independently of React: vanilla JS components, reducers, ...
 - The more logic you manage to move away from React components and into functions (especially pure functions), the bigger benefits you'll reap in terms of architecture and maintainability of the code base, as well as of the quality of your flow (since tests are faster and easier to understand).

“I’ve got a React codebase, I want to introduce tests, where do I start?”

_I recently started hosting AMA-style cafe chats with developers at various stages of their journey, about software engineering and anything else within my technical experience. Below are some rough & rapid-fire notes from the chat on Oct 21st, 2022._

## "Why test?"

Tricky question because it puts you in the mindset of writing test code after application code (rarely you'd think IRL of testing something that's not there).

- Better question: _Why write down your goal before starting to work on it?_
- TDD used to be called "Test-First Design"; good but not great (the word _test_ is still there); attempts at rebranding as "Behaviour-Driven Development" sadly failed.
- Ideally someone with a magic want would replace `test(description, tester)` with `rule(definition, validator)` everywhere.

## "If you look at two typical hours of your coding, what's taking most of that time?"

For me, checking that a change I made does what I want, and checking that a change that I made doesn't break something else. In other words: validation.

- Not just a time cost — the frequent context switch has high attentional cost.
- Automating validation shifts a large chunk of the time cost to the machine. Well written tests (running in <5s) also save the attentional cost.

## "My application is still simple. Maybe I can postpone writing tests for a while longer still?"

- Has it happened yet that something that was working before suddenly stopped working and you don't know why? If so, guess what that means.
- Any form of automated validation shrinks the time between defect introduction and defect detection/correction. If you imagine your codebase's evolution as an arrow growing with time, automated validation helps increase the work you do near the pointy end of the line instead of repeatedly jumping back on the line to fix things that used to work.

## Critical thinking not optional

- Test all the things, then? Not so fast. A situation might not just call for automated validation (e.g. exploring a problem space) or not call for it _yet_ (e.g. integrating with pieces you have no control over that will drastically influence the choice of a testing strategy).
- Even when automated validation is called for, example-based tests aren't necessarily the best form; sometimes you may want snapshot-based tests (whether at the data or the visual level) or even [property-based](https://github.com/dubzzz/fast-check) tests.

TDD and (vs?) testing


_[Originally written for people onboarding a project I've been leading, to explain why services are structured the way they are, and to provide a set of reasoned guidelines.]_

## 1. Decouple service configuration, definition, creation, and startup

If your `index.ts` looks like this:

```ts
import express from "express";

const app = express();

app.get("/", (req, res) => {
  /* ... */
});

app.listen(3000);
```

Then you'll have a hard time using the service in any environment other than a live one (such as testing), because definition, creation, and startup are all mixed with module loading.

Instead, define the service creation in `service.ts`, and **don't** start it as part of creation, but return a control interface so that the caller can decide when to start it:

```js
import express from 'express'

const createService = ({ port }: { port: number }) => {
  const app = express()

  app.get('/', (req, res) => {
    /* ... */
  })

  return {
    start () =>
      new Promise((resolve) => {
        app.listen(port, resolve)
      }),
  }
}
```

Then create, configure, and start the service in `index.ts`:

```ts
import { createService } from "./service";

const main = async () => {
  const service = createService({ port: 3000 });
  await service.start();
};

main();
```

## 2. (Generalization of #1) With the exception of the entry point, use module scope only for definitions, never for side effects

Consider this:

```ts
import { computeSignature } from "something-sdk";
```

And notice it's not:

```ts
importDefinitionWhileMakingSomethingElseHappen {
  computeSignature
} from 'acme-sdk'
```

Which is why we'd never expect `node_modules/acme-sdk/index.js` to be:

```ts
export const computeSignature = (data) => {
  /* ... */
};

startAcmeConnection(process.env.ACME_API_URL);
```

That would reduce the number of scenarios where `computeSignature` can be used (to live environments which have a connetion to `Acme`). Arguably, a module author's job is to _maximize_ that number.

A good rule of thumb is that **if a module contains side effects, it should probably not contain `export`s**.

## 3. Don't instantiate dependencies as part of service creation

Instead of passing `dbUri` to `createService` and expect it to instantiate a database connection, like so:

```ts
// service.ts

const createService = (config: { dbUri: string }) => {
  const app = express();
  const db = await createDatabaseConnection(
    config.dbUri,
  );

  app.get("/items", async (req, res) => {
    const items = await db.getItems();
    /* ... */
  });

  /* ... */
};
```

Instantiate the database connection and pass it to the service factory as just one more configuration property:

```ts
// index.ts

import { createService } from "./service";
import { createDbConnection } from "./db";

const main = async () => {
  const db = createDbConnection(
    "postgres://dbhost/devdb",
  );
  const service = createService({ db });
  await service.start();
};
```

Mocking `db` when testing the service will be far easier that way.

## 4. Environment is broken until proven otherwise

If you cringed at hardcoded configuration values in the previous examples and wondered why not read them from environment variables, that's because the topic deserves its own section.

`process.env` is the silent killer of production services:

- Its value is set in places that can't validate it: `.env` files, Kubernetes manifests, the "Environment" tab on the CI web UI; to them, `DBURI` is just as good as `DB_URI`.
- It has all possible values in existence... at least according to its typings; you may have set `DB_URI` correctly in `.env`, but TypeScript will happily accept `process.env.DBURI`.
- Environment values are often optional, so we tend to provide default values, which compounds the previous point. What happens when you correctly set `DB_URI=posgress//dbhost/proddb` in `.env`, but your code has the following?

```ts
const dbUri =
  process.env.DBURI ?? "postgres://dbhost/devdb";
```

The answer is that nothing happens _immediately_...

The solution is the same as for any external data: **validate**.

Instead of importing from `process.env`, import from a custom `env` module:

```ts
// index.ts

import { env } from "./env";

const main = async () => {
  const service = createService({});
  await service.start();
};

main();
```

And define the `env` module as:

```ts
// env.ts
import { ensure, v } from "suretype"; // or your preferred validator

const envSchema = v.object({
  NODE_ENV: v
    .string()
    .enum("development", "production")
    .required(),
  PORT: v.string(),
  DB_URI: v.string().required(),
});

export const env = ensure(envSchema, process.env);
```

That way, misconfigurations will cause evident crashes instead of hard to track, potentially catastrophic bugs.

Once you've banished `process.env`, to prevent it from sneaking in again (it's so ubiquitous that newcomers to the team will no doubt use it without giving it a second thought), consider adding this to `.eslintrc`:

```json
  "rules": {
    "no-process-env": "error",
  }
```


Four patterns for developer-friendly, production-grade NodeJS services

I recently wrote code similar to this:

```jsx
const App = (props) => {
 return (
 <ThemeProvider theme={theme}>
 <PageContent {...props} />
 </ThemeProvider>
 )
}

const PageContent = ({ Component, pageProps }) => {
 const getLayout = Component.getLayout ?? ((page) => page)
 return getLayout(<Component {...pageProps} />)
}
```

If it were my first time reading it, I'd think that the `PageContent` component is a pointless, even harmful addition: it's only used once; it doesn't express any remarkable idea; it forces the reader to store yet another name in an already crowded working memory.

I'd be strongly tempted to refactor it, and since it's a non-functional change, I might do it as part of a larger change, heeding the [boy scout rule](https://www.oreilly.com/library/view/97-things-every/9780596809515/ch08.html):

```jsx
const App = (props) => {
 const getLayout = Component.getLayout ?? ((page) => page)

 return (
 <ThemeProvider theme={theme}>
 {getLayout(<Component {...pageProps} />)}
 </ThemeProvider>
 )
}
```

I might even skip refreshing the page, because, again, it's just a change in form.

Or is it?

I'd later find out that the background no longer respects the dark mode setting, while every other component does; after trying for a long while to debug the larger functional change without getting anywhere, I would trace the problem to the code above.

If you're trying to figure out how the refactoring attemept could cause that, good luck. Although you might spot something if you're well versed in React _and_ you squint hard _and_ you've encountered the circumstance before, it's really hard, because the code you need to know the most to figure out what's going on is nowhere in sight, it's instead in the source for the page component itself, in a file far away:

```tsx
export const getLayout = (page) => (
 <Layout bgImage={useColorModeValue(backgroundLight, backgroundDark)}>
 {page}
 <Layout>
)
```

Even so, still not obvious...

Imagine, however, that while writing the original code, I had placed this comment inside the `PageContent` component:

```tsx
const App = (props) => (
 return (
 <ThemeProvider theme={theme}>
 <PageContent {...props} />
 </ThemeProvider>
 )
)

const PageContent = ({ Component, pageProps }) => {
 // Why not just have all this in <App>? Because getLayout uses the
 // color mode hook, which in turn must be called from under
 // <ThemeProvider> to work

 const getLayout = Component.getLayout || ((page) => page)
 return getLayout(<Component {...pageProps} />)
}
```

How likely is the reader now to jump into refactoring?

It's second nature for programmers, while they're writing code, to imagine it playing out at run time and continuously ask: "what's likely to go wrong here? what could cause this to break? how can I prevent or handle it?" Users may provide invalid input, network connections might go down, the environment might not support the cabability, ... Situations handled with schemas, timeouts, discovery, ...

It pays to adopt the same mentality and, while you're writing code, imagine it playing out at _read_ time and ask: "what's likely to go wrong here? what could cause this to be misunderstood? how can I prevent or handle it?" Commenting code that's not there — explaining the outside-of-source reasons why the code is the way it is, instead of a more "logical", "elegant", or "obvious" form — is a relatively cheap and absolutely effective way to prevent _read time bugs_.

Comment the code that is not there


[web3-mock](https://github.com/DePayFi/web3-mock) supports Cypress and JSDOM test environments out of the box. However, I prefer Playwright to those as it provides a compelling set of benefits: familiar Jest matchers, a fixture-based runner, and execution in multiple real browser environments. Getting web3-mock to work with Playwright requires setup that is minimal but not obvious, hence this quick note.

The test definitions configures the mock and then interact with the page:

```typescript
import { test } from '@playwright/test'

test('test wallet connection', async ({ page }) => {
  await mockWeb3(page, () => {
    // @ts-ignore
    Web3Mock.mock({
      blockchain: 'ethereum',
      accounts: { return: ['0xd73b04b0e696b0945283defa3eee453814758f1a'] },
    })
  })

  await page.goto('http://localhost:3000/')
  await page.locator('text=Sign in').click()
  await page.locator('text=Connected').waitFor()
})
```

The `mockWeb3()` call causes the `web3-mock` library plus the mock configuration to be loaded whenever a page is navigated to, but before the page's own scripts are run:

```typescript
import { Page } from '@playwright/test'
import { readFileSync } from 'fs'

const mockWeb3 = async (page: Page, fn: Function) => {
  await page.addInitScript({
    content:
      readFileSync(
        require.resolve('@depay/web3-mock/dist/umd/index.bundle.js'),
        'utf-8',
      ) +
      '\n' +
      `(${fn.toString()})();`,
  })
}
```

Why the ugly string concatenation, when we could call `addInitScript` twice, once for injecting the library, the next for injecting the mock configuration? Because the order of execution of scripts added with `addInitScript` is not defined, so the library might get loaded after (the attempt at) using it.


Mocking window.ethereum in Playwright for end-to-end dApp testing

Structure describes and prescribes; it tells about the system while nudging you into thinking of, and using it, in a certain way—hopefully, a way that was chosen deliberately, wisely, and points toward [the pit of success](https://blog.codinghorror.com/falling-into-the-pit-of-success/).

Discussion of React project structures usually centers on the descriptive ("what does folder/file X contain?") with an intent of easing navigation ("where do I find object/function Y?"). It's a start, though one with diminishing returns.

In this note I'd like to offer, rather than my preferred project structure, some ways of thinking about the project space, some ways of slicing it, and a discussion of how they nudge developers into thinking about important properties of the system. Even if those ways and properties aren't right or sufficient in your case, the process should help you come up with ones that are.

## Audience

But first: who does your project speak to? Many eyes will visit it over time and you can't optimize for all. Take a moment to think of which ones matter.

Developer seniority is less of a factor than you'd think: don't refrain from using words like _container_ or _machine_ in file names just because juniors might find them unfamiliar; if the concepts are central to the architecture, they need to be faced at some point, and it might well be through the filesystem.

Project seniority, on the other hand — the time developers spend on the project — **is** a factor: a project where people from other teams constantly jump in for just a few days, help with something, and jump out, needs loud signposts to macroscopic concepts that are inevitably forgotten between incursions and thus need constant refreshing; think `pages/`. A stabler team can afford to take the basics for granted and use the bandwidth for far finer points; think `pure/` (as in "pure functions").

## Navigation (...don't)

You step into a supermarket with sixty seconds to grab a microwaveable tandoori chicken. Do you use them to decipher the layout of the aisles, run to the first one that sounds related ("ready dishes"? "world kitchen"?), frantically browse through the shelves, ... or do you find a clerk and ask?

I like to think that something like that went through brains at Google's when faced with:

![Yahoo](https://i.insider.com/57977a3288e4a794048babc2?width=1000&format=jpeg&auto=webp)

To eventually lead to:

<img src="https://www.versionmuseum.com/images/websites/google-search/webp/google-search%5E1999%5Esearch-homepage.webp" alt="Google">

It's tempting to slice your project's space into a multitude of "aisles" and put everything on "its" shelf. We're just trying to ease navigation! (And, truth be told, we get a kick out of imparting order.)

```
~/projects/app$
index.jsx
components/
components/forms/
components/pages/
components/layouts/
components/widgets/
```

Do you look for `SubmitButton.jsx` under `forms/` or `widgets/`? More to the point: do you or anyone on your team actually "browses aisles" when looking for something?

It's worth asking around, as everyone's flow differs, but I almost always invoke the project-wide search and type a fragment of filename or actual content; or place cursor on a symbol, and jump to definition. Few keystrokes later, I'm where I need to be.

Browsing is alright when you don't know what you want and you want to see what's available. There _is_ some value to organizing for browsing; just not that much, and just not for long. You might still go for it for specific case, such as if people jump in and out of the project often.

## The questions, and _the_ question

The names we pick answer questions:

```text
components/Profile.test.jsx
^ ^ ^
| | +--- "What aspect does it cover?"
| +----------- "What role does it play in the application?"
+---------------------- "What kind of construct is it?"
```

But we rarely pick our questions consciously. That's risky because our choice of questions is itself an answer to a more fundamental one: _What's worth focusing on?_ Focus on the inconsequential and, no matter how good the answers, you'll move slow, if not backward.

I encourage you to surface your own questions. Here are some that I use and a discussion of each:

1. What changes together?
2. What's used in one place, what in more?
3. What follows the same rules?
4. Does loading this produce side effects or just redefinition?
5. Does this depend on component state?

### 1. "What changes together?"

If you change a component test, chances are you'll change the corresponding component soon, and vice-versa. If you delete the component, you don't want the test lying around. If you look at the file list, you want to spot right away a component that lacks tests.

Strongly related code can, in principle, live in the same file. However, navigating source would require more work, and code would be processed unnecessarily in CI, where seconds matter, and interactive development, where milliseconds do.

Strongly related code shouldn't live in faraway folders (such as `src/`, `test/`, `stories/`). The small gain in visual tidiness is soon forgotten when an orphaned test has been waiting for months for someone to find the time and check if it can be removed.

**Sidecars** are a happy mean.

The example below is extreme, but don't be afraid of going one or two steps further than what you're used to:

```text
components/
components/Profile.jsx
components/Profile.test.jsx
components/Profile.stories.jsx
components/Profile.Layout.jsx
components/Profile.styles.jsx
components/Profile.reducer.ts
components/Profile.functions.ts
components/Profile.machine.ts
```

"But won't this make my `components` directory explode?"

Even without sidecars, a medium-sized React application has enough components files to sink way below the fold. At that point, already people aren't finding things by reading through the file list; two pages or twenty won't make a difference. But they _are_ looking at subsets of it when selecting files, though, so seeing related things together helps.

### 2. "What's used in one place, what in more?"

[TBD]

### 3. "What follows the same rules?"

When you associate rules to a space, "entering" it is enough to leverage all the thinking that went into creating the rule, without going again through it.

For example, importing stateful (or even just configured) objects in a React component file, like `api` below, makes testing hard, because you have to plug into the module import system:

```jsx
import React, { useState, useEffect } from 'react'
import { Profile } from 'src/components/Profile'
import api from 'src/api'

export const ProfilePage = () => {
 const [profile, setProfile] = useState(null)

 useEffect(() => {
 api.getProfile().then(setProfile)
 }, [])

 return profile ? <Profile profile={profile} /> : null
}
```

Thus the rule that a component file can import as many functions as it wants, but state should only "enter" via props or hooks.

That's workable, but becomes even quicker to handle when you state it as:

"Files under `components/` can only import from `hooks/`, and `components/` itself, and files under `hooks/` can only import from `hooks/` itself."

So what happens when I, as a React junior unaware of all that context, am writing `components/ProfilePage.jsx` and either my reviewer or my linter tells me I can't import `api` because it's not under `components/` or `hooks/`?

It's not a component, so I try to fit it under `hooks/`. Good, but that just shifts the problem: how do I get `api` from `hooks/` if I'm only allowed to import from `hooks/`? Some head-scratching later and questions later, I'm pointed to React Context. I read about it, and I end up creating one, and changing the top-level `index.jsx` to:

```jsx
import api from './api'

ReactDOM.render(
 <ApiProvider value={api}>
 <App />
 </ApiProvider>,
)
```

The same `ApiProvider` can be used for testing without acrobatics, and my reviewer is happy.

By constraining my options, the project structure guided me to a better implementation.

### 4. "Does reloading this produce side effects or just redefinition?"

[TBD]

### 5. "Does this depend on component state?"

[TBD]



## Summary

1. Project structure sets questions people have to answer in order to name, and place, things.
2. Pick questions that invite to think about important properties of the system ("what changes together?" "what architectural rule are all these things following?") and leave behind those that can better be addressed by tools ("where do I find X?").
3. Take into account the team's composition (over time!) and the team members' workflows.

[draft] Structuring projects to guide and inform (for React applications)


The "conventional commit" format advocates prefixing commit messages with keywords to give both humans and machines a quick idea of what's affected:

```text
41e7be2 feat: notify user on new messages
f3b33ab build: enable caching to decrease build times
7501fb2 fix(api): detect invalid query parameter
f5baa4d docs: typo
06e851a perf(ui): optimize loading time
```

The keyword mostly answers the question: "How does this modify existing code?" It **fix**es or **refactor**s it, enhances its **perf**ormance, or adds a new **feat**ure on top. Sometimes, however, it answers the question: "What area does this affect?" It affects the **build** process or the **docs**. Yet other times, it answers both questions by adding a parenthesized "scope": it's a **fix** to the **api**; it's a **perf**ormance enhancement to the **(ui)**.

And if you're fixing a doc? Improving the build's performance?

Having to stop and think in these loaded, non-orthogonal terms, to come up with only moderately descriptive descriptions, feels like too high of a cost for the benefit.

Fortunately, a few extra keystrokes are all that's need to lower the burden on the writer and boost the value for the reader.

## Meet the "-ilities"

A great question one can ask about a change is: "What quality of the system does this directly improve?" ^["Directly" because everything, eventually, aims at improving end-user functionality, business profitability, and similar bottom lines, so those aren't good descriptors at a local level.] A change to the front end might improve usability; one to the back end, observability; another to the docs, understandability.

The book _Building Evolutionary Architectures_ has a long list of these "-ilities":

- accessibility
- adaptability
- administrability
- auditability
- availability
- compatibility
- (...two more pages)

You can use them to better describe, and think about, changes.

## Value-focused, bi-dimensional commit messages

What qualities matter to your team and project? Pick a few, e.g.: **functionality**, **usability**, **devability** ^[Think of devability as of usability for developers: a mixture of code understandability, debuggability, and tool ergonomy.], **stability**, and **security**.

_What_ quality does a change influence? That's the first dimension:

|                                          | what?                              |          |
| ---------------------------------------- | ---------------------------------- | -------- |
| "notify user on new messages"            | makes the application more useful  | **func** |
| "enable caching to decrease build times" | makes the system easier to develop | **dev**  |
| "detect invalid query parameter"         | makes the system more stable       | **stab** |

_How_ does the change improve the quality? That's the second dimension:

|                                          | how?                                    |          |
| ---------------------------------------- | --------------------------------------- | -------- |
| "notify user on new messages"            | adds something that wasn't there before | **feat** |
| "enable caching to decrease build times" | makes something better                  | **enh**  |
| "detect invalid query parameter"         | makes something right                   | **fix**  |

And the original example becomes:

```diff
- 41e7be2 feat: notify user on new messages
+ 41e7be2 usab,feat: notify user on new messages
- f3b33ab build: enable caching to decrease build times
+ f3b33ab dev,enh: enable caching to decrease build times
- 7501fb2 fix(api): detect invalid query parameter
+ 7501fb2 stab,fix(api): detect invalid query parameter
- f5baa4d docs: typo
+ f5baa4d dev,fix: typo
- 06e851a perf(ui): optimize loading time
+ 06e851a usab,enh(ui): optimize loading time
```

This strategy is surprisingly quick to pick up, generates informative descriptions, and gently nudges one to focus on the fundamental question:

_What **value** does this change deliver?_

## References

- [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- [Building Evolutionary Architectures](https://www.thoughtworks.com/books/building-evolutionary-architectures) (O'Reilly, 2017)


Bi-dimensional commit messages

(The following is expanded from my comment at [#storybook/10987](https://github.com/storybookjs/storybook/issues/10987#issuecomment-814273313))

Storybook provides, among other things, a canvas to develop UI components interactively and in isolation.

In any kind of digital work, interactivity comes not only from the absence of manual steps between making a change and seeing the outcome, but also from a feedback loop tight enough to keep your mind on the task: a word processor that takes five seconds to turn a word to italic loses both the semblance and the benefits of interactivity.

Mostly due to the underlying toolchain, that's exactly the situation Storybook can find itself in: a freshly initialized Storybook in a freshly initialized React app can, depending on machine, take more than two seconds to refresh the example button after changing its size from "medium" to "large". I've seen that grow to 5+ seconds even for small (<5k LOC) projects.

These tweaks can bring those numbers down to more workable levels.

## Optimization #1: disable plugins irrelevant to interactive development

Some plugins either create value in areas other than interactive development (such as documentation) or replicate work already done in editors (linting, typechecking). That can take a lot of time. See for example the impact of the Docgen plugin (each timestamp is the time elapsed since the previous line):

```shell
~/.../src: yarn storybook | gnomon
...
 0.1254s <s> [webpack.Progress] 70% finish module graph ESLintWebpackPlugin
 <s> [webpack.Progress] 70% finish module graph FlagDependencyExportsPlugin
...
 0.0096s <s> [webpack.Progress] 70% sealing WarnCaseSensitiveModulesPlugin
 1.0012s <s> [webpack.Progress] 70% sealing React Docgen Typescript Plugin
 1.5016s <s> [webpack.Progress] 70% sealing React Docgen Typescript Plugin
 2.0022s <s> [webpack.Progress] 70% sealing React Docgen Typescript Plugin
 2.3576s <s> [webpack.Progress] 70% sealing React Docgen Typescript Plugin
 <s> [webpack.Progress] 72% basic dependencies optimization
 0.0002s <s> [webpack.Progress] 72% basic dependencies optimization
```

Plugins can be disabled in `.storybook/main.js`:

```diff
 const { inspect } = require('util')

 module.exports = {
 stories: ['../src/**/*.stories.@(js|jsx|ts|tsx)'],
 addons: [
 '@storybook/addon-links',
 '@storybook/addon-essentials',
 '@storybook/preset-create-react-app',
 ],

+ typescript: {
+ check: false,
+ reactDocgen: false,
+ },
+
+ webpackFinal: async (config, { configType }) => {
+ config.plugins = config.plugins.filter(
+ (p) => !inspect(p).match(/^ESLintWebpackPlugin/),
+ )
+ return config
+ },
 }
```

If you're also publishing the Storybook output as documentation/style guide/showcase, you'll want to do the above only for `NODE_ENV=development`.

## Optimization #2: replace stages of the toolchain with faster ones

_**Note**: This one appears to no longer work with the current release of `esbuild-loader`._

Though primarily meant to reduce cold build times, the `storybook-addon-turbo-build` add-on does further shave time away from interactive refreshes. After installation, enable it with:

```diff
 module.exports = {
 stories: ['../src/**/*.stories.@(js|jsx|ts|tsx)'],
 addons: [
 '@storybook/addon-links',
 '@storybook/addon-essentials',
 '@storybook/preset-create-react-app',
+ {
+ name: "storybook-addon-turbo-build",
+ options: { optimizationLevel: 3 },
+ },
 ],
```

## Optimization #3: get (partially) rid of Webpack

Snowpack is an alternative to Webpack that, through a radically different approach to bundling, provides instant refresh. It can be used in development while leaving the production build to a tried-and-tested Webpack configuration. Unfortunately, Storybook doesn't run in Snowpack yet.

However, if the gains from optimization #1 and #2 are not enough (they weren't for me), it's worth to rethink the problem so that Snowpack can be brought in.

What is the core value of the Storybook canvas for interactive development? We already disabled documentation generation. Do you need widgets to update component props? Do you need add-ons to check the component on different backgrounds and with different locales? Much of that comes into play in a later, shorter "refinement" phase.

If you've adopted Snowpack for your dev environment, and can tolerate the trade-offs, you can create a `/canvas` route that only exists when `NODE_ENV=development`, where you develop components in semi-isolation, re-use Storybook stories, and benefit from instant refresh. The minimum viable version of it is:

`src/Components/App.tsx`:

```tsx
 <Router>
 <Switch>
 <Route ... />
 <Route ... />
 {process.env.NODE_ENV === 'development' && (
 <Route path="/canvas" component={Canvas} exact={true} />
 )}
```

`src/Components/Canvas.tsx`:

```tsx
import React from 'react'
import * as Button from './Button.stories'

export const Canvas: React.FC = () => {
 const StoryBeingWorkedOn = Button.Basic
 return (
 <div>
 <StoryBeingWorkedOn {...StoryBeingWorkedOn.args} />
 </div>
 )
}
```

(It's just as easy to import multiple stories and add a way to switch among them at runtime. You don't want to replicate the entire Storybook canvas, but a little convenience can go a long way.)

The only downsides of this strategy are 1) when you change component args in story files, the changes aren't picked up and you need a full page reload; and 2) if you use decorators, you need to write some more support code.

## References

- [Snowpack](https://www.snowpack.dev/)
- [#storybook/10987](https://github.com/storybookjs/storybook/issues/10987) Track this issue to know if/when Storybook will support Snowpack, superseding most the hacks presented here.
- [storybook-addon-turbo-build](https://storybook.js.org/addons/storybook-addon-turbo-build)

Faster interactive development in Storybook

_Stateful systems that get into inconsistent states are a frequent source of bugs. Making conceptually interdependent data also programmatically interdependent is one way of addressing the root of the problem._

When a bug breaks production, it's good to ask: "Why didn't a test catch that?".

An even better question is "Why did the code build at all?", but it isn't asked nearly as often. Why?

Perhaps because the bug was of the kind that could only be caught at runtime, so it _did_ need a test. Fair enough. Perhaps the bug _could_ be caught at compile time, we just didn't know how. That's easy to fix. Perhaps we adopted a hard and fast "test all the things" rule, eager to consider the matter of code reliability closed and move on to less taxing areas. That's harder to fix.

## Help the compiler to help you

Preventing bugs costs less than catching them, and one class of bugs amenable to prevention is inconsistent states, e.g. a `fetchError` and an `isLoading` variables ending up set at the same time, the rest of the code scrambling through the paradox.

It's a programmer bug before a programming one: give your coworkers a piece of code where that can happen, and you're giving them work (tracking data relations in their heads to make sure they stay consistent) that a machine is able to do. Giving to humans work suitable for the machine is anti-programming.

Such bugs can be often prevented by encoding constraints ("it's not possible for `fetchError` and `isLoading` to be set at the same time") in a form that development tools can understand.

Notably: **types**.

## A before-and-after demonstration

Below is a React hook that exposes the browser's Screen Wake Lock API. A wake lock tells the device not to blank the screen even if the user isn't interacting, e.g. on pages with long texts.

The API is deceptively simple: ask the browser for the lock, get the response. But:

1. the browser may or may not have the capability;
2. the browser doesn't respond synchronously;
3. you get a "sentinel" object that you can later used to release the lock; the releasing is also asynchronous, so you can't just set the sentinel to `null` to signify "lock is inactive": if you null it before release, other code can't know if the lock is really inactive or being released; if you null it after release, other code can't know if the lock is really active or being released.

We want to use the hook like this:

```typescript
const wakeLock = useWakeLock()
wakeLock.setActive(true)
```

Here's a traditional "flag fest" implementation:

```typescript
import { useState } from 'react'

export const useWakeLock = () => {
 const [sentinel, setSentinel] = useState<WakeLockSentinel | null>(null)
 const [isActive, setIsActive] = useState(false)
 const [error, setError] = useState<null | Error>(null)

 const enable = async () => {
 try {
 const sentinel = await navigator.wakeLock.request('screen')
 setSentinel(sentinel)
 setIsActive(false)
 setError(null)
 } catch (err) {
 setError(err)
 }
 }

 const disable = async () => {
 if (!isActive) return
 if (sentinel === null) return
 sentinel.release()
 setSentinel(null)
 setIsActive(false)
 setError(null)
 }

 const setActive = async (wantActive: boolean) => {
 if (!('wakeLock' in navigator)) return
 if (wantActive) {
 enable()
 } else {
 disable()
 }
 }

 return { setActive, error }
}
```

What happens when you inadvertently put the system in an inconsistent state?

```diff
 const sentinel = await navigator.wakeLock.request('screen')
 setSentinel(sentinel)
- setIsActive(true)
+ setIsActive(false)
 setError(null)
```

The TypeScript compiler:

> _...Silence..._

Here's an implementation where you tell the compiler that interdependent data is, well, interdependent:

```typescript
import { useState } from 'react'

export const useWakeLock = () => {
 const [state, setState] = useState<
 | { type: 'inactive' }
 | { type: 'requesting' }
 | { type: 'active'; sentinel: WakeLockSentinel }
 | { type: 'error'; error: Error }
 >({ type: 'inactive' })

 const enable = async () => {
 if (state.type === 'inactive') {
 setState({ type: 'requesting' })
 try {
 const sentinel = await navigator.wakeLock.request('screen')
 setState({ type: 'active', sentinel })
 } catch (error) {
 setState({ type: 'error', error })
 }
 }
 }

 const disable = async () => {
 if (state.type === 'active') {
 state.sentinel.release()
 setState({ type: 'inactive' })
 }
 }

 const setActive = (wantActive: boolean) => {
 if (!('wakeLock' in navigator)) return
 if (wantActive) {
 enable()
 } else {
 disable()
 }
 }

 const error = state.type === 'error' ? state.error : null
 return { setActive, error }
}
```

What happens when the inconsistent state tries to sneak in?

```diff
 const sentinel = await navigator.wakeLock.request('screen')
- setState({ type: 'active', sentinel })
+ setState({ type: 'inactive', sentinel })
```

The TypeScript compiler:

```shell
src/hooks/wake-lock.ts:16:20 - error TS2322: Type '"inactive"' is not assignable to type '"active"'.

16 setState({ type: 'inactive', sentinel })
 ~~~~

[6:22:21 PM] Found 1 error. Watching for file changes.
```

The compiler detects that something is wrong: there is no legal scenario where the state is inactive and the sentinel is set.

(For simplicity, neither implementation handles the case where activation is requested while the lock is releasing, or deactivation is requested while the lock is activating, but it's easy to see which one is better equipped to deal with it without complexity exploding.)

This strategy has been called "making illegal states unrepresentable". I prefer to steal a motto from cooking, where combining seasonal ingredients into something delicious is equal parts description and prescription:

_What grows together, goes together._

## Further reading

- [Compilers and assistants](https://elm-lang.org/news/compilers-as-assistants). Elm's take on compilers is no less interesting than its influential UI architecture.
- [XState](https://xstate.js.org/). Explicit state machines are the next step to rein in complexity. In fact, the second example above is just a less implicit state machine than the first. XState is an excellent framework for state machines in JavaScript, though one with a steep learning curve.

Data that grows together, goes together

_For many tasks in a developer's day, a well-tuned compilation mode is more efficient than reaching for a shell (fewer keystrokes, less context switching) and lighter on the brain than specialized major modes (no new key bindings)._



Emacs's "compilation" mode is not limited to running compilers. You can run bundlers for front-end development, test runners for red-green-refactor TDD, project-wide linters, dev servers for back-end code, Makefile targets, and more.

These notes cover:

- handling commands that produce rich output, and/or clear the screen between multiple runs in a single long-lived session
- choosing efficient key bindings
- browsing command output without leaving the source window
- selecting commands from configurable, contextual lists using the `run-command` package

## Rich output and long-lived commands

Colors are often the only way to notice errors in a flurry of output. Also, commands with a long startup time sometimes offer a "watch" mode to do several runs for a single invocation, and clear the screen between runs.

Compilation mode by default handles neither the control characters that produce colors, nor those that clear the screen. A simple solution that covers many cases is to watch out for a few sequences in the command output using a "compilation filter", and apply them:

```emacs-lisp
(defun local/postprocess-compilation-buffer ()
 (goto-char compilation-filter-start)
 (when (looking-at "\033c")
 (delete-region (point-min) (match-end 0)))
 (ansi-color-apply-on-region (point) (point-max))))

(add-hook 'compilation-filter-hook 'local/postprocess-compilation-buffer)
```

Some commands produce even richer output (progress bars, simple animations) or clear the screen by other means (ANSI codes), requiring more complete terminal capabilities. The `run-command` package (described later) can run such commands in Emacs's `term-mode`, providing those capabilities, while maintaining `compilation-mode` functionality.

## Choosing efficient key bindings

The basic way to run a command in compilation mode is `M-x compile RET <command> RET`, e.g. `M-x compile RET npm start RET`. Next time you invoke `compile`, `<command>` defaults to the previous command, so recompiling effectively becomes `M-x compile RET RET`.

However, recompiling being the more common case, Emacs also provides `M-x recompile`, which reuses the last command right away, without prompting. (You can still edit the command with `C-u M-x recompile`.) So, `recompile` is the better candidate for quick access via a key binding, e.g.:

```emacs-lisp
(global-set-key (kbd "C-c c") 'recompile)
```

With that, `C-c c` will re-run the last command, and `C-u C-c c` will let you edit the command first.

There's an even shorter binding for when the cursor is already in the compilation window: `g`. This is useful when you need to alternate among several complex commands: leave their buffers open, then switch back to them and press `g` as needed.

To stop a command, move the cursor to the compilation buffer and press `C-c k`.

## Browsing command output without leaving the source window

Unless you need to copy a command's output, or re-run a previous command, there's little reason to move the cursor to the command output window; you can more efficiently browse it without leaving the window where you're writing using these key bindings:

- To scroll to the beginning of the output: `M-home`
- To scroll to the end of the output: `M-end`
- To scroll the output forward: `M-next` (where "next" means `PageDown`) or `C-M-v`
- To scroll the output backward: `M-prior` (where "prior" means `PageUp`) or `C-M-S-v`

These bindings are not specific to compilation mode, the work in every situation where there is an "other window". They may seem cumbersome, especially on laptop keyboards which require an additional `Fn` press, but are well worth the effort.

Another flow saver, provided that the command prints errors in a format that compilation mode understands, is `` C-x ` ``, which jumps to errors in the output, in succession. For example, given a TypeScript project with a file `src/App.tsx`:

```tsx
import React from 'react'
import ReactDOM from 'react-dom'

console.helloWorld()
```

Running `M-x compile RET npxj tsc -P . --noEmit --watch` will launch the type checker and print the following:

```shell
[8:13:36 PM] File change detected. Starting incremental compilation...

src/App.tsx:3:9 - error TS2339: Property 'helloWorld' does not exist on type 'Console'.

3 console.helloWorld()
 ~~~~~~~~~~

[8:13:36 PM] Found 1 error. Watching for file changes.
```

If you now hit `` C-x ` ``, the `App.tsx` buffer will appear, with the cursor on `helloWorld`.

## Selecting commands from configurable, contextual lists using the `run-command` package

If, in a given context, you tend to run the same small subset of commands over and over, (e.g. `npm test:watch` in a JavaScript project, `make clean` in a C project, `hugo server --buildDrafts` in a Hugo blog), that's a prime case for automation.

The `run-command` Emacs package (by the author) lets you write simple command lists, bring them up depending on the context, and select them with autocompletion:

![run-command demo](https://raw.githubusercontent.com/bard/emacs-run-command/master/demo.gif)

See the [run-command GitHub page](https://github.com/bard/emacs-run-command) for setup and usage instructions.

## Additional customization

- To have the window scroll as output comes in, set `compilation-scroll-output` to `t`.
 - The default scrolling mode often leads to a large amount of empty space at the bottom of the window; this is due to "recentering". To prevent that and display as much output as possible, set `scroll-conservatively` to `101` (see `M-x describe-variable scroll-conservatively RET` for why).
- To have it scroll, but stop scrolling when the first error is detected, set `compilation-scroll-output` to `error`.

Lightweight external command integration in Emacs via compilation mode

_Functions that ask for more than they need trick the developer into writing unnecessary code. At best, this slows down future development; at worst, it harms runtime qualities: of the present system._



Imagine that you inherited this code:

```ts
const post = await api.fetchPostById('abc123')
// => {
// id: 'abc123',
// title: 'The dangers of greedy functions',
// authorName: 'Max'
// }

const postUrl = getPublishedUrl(post)
```

One day, the back-end team changes the API so that author data is exposed through a separate endpoint. Your IDE tells you that `getPublishedUrl` needs an argument of type `{ id: string, title: string, authorName: string }`, so you make the necessary changes:

```ts
const post = await api.fetchPostById('abc123')
const author = await api.fetchAuthorByPostId(post.id)

const postUrl = getPublishedUrl({
 ...post,
 authorName: author.name,
})
```

Tests succeed, production works, all is good.

Some time later, you look into a file called `post‑utils.ts`:

```ts
type Post = {
 id: string
 title: string
 authorName: string
}

function getPublishedUrl(post: Post): string {
 return '/posts/' + post.id
}
```

It turns out that `getPublishedUrl` didn't need author data at all.

And yet, it led you into adding complexity, and even causing some performance degradation (an extra network request).

## The problem

_Surely you could have peeked at `getPublishedUrl` implementation sooner?_

For every `getPublishedUrl` in a contrived example, dozens exist in a real code base. But impracticality aside: why establish contracts, if you're required to look at implementations? Function signatures and object interfaces are there to save you that work. If you still have to do it, they're not doing their job right.

In fact, the real problem (at least at the _program level_) is that `getPublishedUrl` is lying. This:

```ts
function getPublishedUrl(post: Post): string
```

Says: _My name is `getPublishedUrl` and, to do my job, I need a post (i.e. a map of id, title, and authorName)."_ A greedy lie.

The solution (again, at least at the _program level_) is for functions to only ask for what they really need:

```ts
function getPublishedUrl(postId: string): string
```

(Unless they really need 90% or so of a large, complex type, in which case there's little point in leaving out the 10%.)

It's harder to pinpoint the problem at the _programmer level_, i.e. what led someone to choose to write that code. Most likely it was a well-intentioned quest for simplicity and readability. Unfortunately, simplicity that isn't backed by truth looks the part, and quietly invites its opposite.

Massimiliano Mirra

Notes

Tags