1 # Javascript {#language-javascript}
3 ## Introduction {#javascript-introduction}
5 This contains instructions on how to package javascript applications.
7 The various tools available will be listed in the [tools-overview](#javascript-tools-overview).
8 Some general principles for packaging will follow.
9 Finally some tool specific instructions will be given.
11 ## Getting unstuck / finding code examples {#javascript-finding-examples}
13 If you find you are lacking inspiration for packaging javascript applications, the links below might prove useful.
14 Searching online for prior art can be helpful if you are running into solved problems.
16 ### Github {#javascript-finding-examples-github}
18 - Searching Nix files for `mkYarnPackage`: <https://github.com/search?q=mkYarnPackage+language%3ANix&type=code>
19 - Searching just `flake.nix` files for `mkYarnPackage`: <https://github.com/search?q=mkYarnPackage+path%3A**%2Fflake.nix&type=code>
21 ### Gitlab {#javascript-finding-examples-gitlab}
23 - Searching Nix files for `mkYarnPackage`: <https://gitlab.com/search?scope=blobs&search=mkYarnPackage+extension%3Anix>
24 - Searching just `flake.nix` files for `mkYarnPackage`: <https://gitlab.com/search?scope=blobs&search=mkYarnPackage+filename%3Aflake.nix>
26 ## Tools overview {#javascript-tools-overview}
28 ## General principles {#javascript-general-principles}
30 The following principles are given in order of importance with potential exceptions.
32 ### Try to use the same node version used upstream {#javascript-upstream-node-version}
34 It is often not documented which node version is used upstream, but if it is, try to use the same version when packaging.
36 This can be a problem if upstream is using the latest and greatest and you are trying to use an earlier version of node.
37 Some cryptic errors regarding V8 may appear.
39 ### Try to respect the package manager originally used by upstream (and use the upstream lock file) {#javascript-upstream-package-manager}
41 A lock file (package-lock.json, yarn.lock...) is supposed to make reproducible installations of `node_modules` for each tool.
43 Guidelines of package managers, recommend to commit those lock files to the repos.
44 If a particular lock file is present, it is a strong indication of which package manager is used upstream.
46 It's better to try to use a Nix tool that understand the lock file.
47 Using a different tool might give you hard to understand error because different packages have been installed.
48 An example of problems that could arise can be found [here](https://github.com/NixOS/nixpkgs/pull/126629).
49 Upstream use npm, but this is an attempt to package it with `yarn2nix` (that uses yarn.lock).
51 Using a different tool forces to commit a lock file to the repository.
52 Those files are fairly large, so when packaging for nixpkgs, this approach does not scale well.
54 Exceptions to this rule are:
56 - When you encounter one of the bugs from a Nix tool. In each of the tool specific instructions, known problems will be detailed. If you have a problem with a particular tool, then it's best to try another tool, even if this means you will have to recreate a lock file and commit it to nixpkgs. In general `yarn2nix` has less known problems and so a simple search in nixpkgs will reveal many yarn.lock files committed.
57 - Some lock files contain particular version of a package that has been pulled off npm for some reason. In that case, you can recreate upstream lock (by removing the original and `npm install`, `yarn`, ...) and commit this to nixpkgs.
58 - The only tool that supports workspaces (a feature of npm that helps manage sub-directories with different package.json from a single top level package.json) is `yarn2nix`. If upstream has workspaces you should try `yarn2nix`.
60 ### Try to use upstream package.json {#javascript-upstream-package-json}
62 Exceptions to this rule are:
64 - Sometimes the upstream repo assumes some dependencies be installed globally. In that case you can add them manually to the upstream package.json (`yarn add xxx` or `npm install xxx`, ...). Dependencies that are installed locally can be executed with `npx` for CLI tools. (e.g. `npx postcss ...`, this is how you can call those dependencies in the phases).
65 - Sometimes there is a version conflict between some dependency requirements. In that case you can fix a version by removing the `^`.
66 - Sometimes the script defined in the package.json does not work as is. Some scripts for example use CLI tools that might not be available, or cd in directory with a different package.json (for workspaces notably). In that case, it's perfectly fine to look at what the particular script is doing and break this down in the phases. In the build script you can see `build:*` calling in turns several other build scripts like `build:ui` or `build:server`. If one of those fails, you can try to separate those into,
76 when you need to override a package.json. It's nice to use the one from the upstream source and do some explicit override. Here is an example:
80 patchedPackageJSON = final.runCommand "package.json" { } ''
81 ${jq}/bin/jq '.version = "0.4.0" |
82 .devDependencies."@jsdoc/cli" = "^0.2.5"
83 ${sonar-src}/package.json > $out
88 You will still need to commit the modified version of the lock files, but at least the overrides are explicit for everyone to see.
90 ### Using node_modules directly {#javascript-using-node_modules}
92 Each tool has an abstraction to just build the node_modules (dependencies) directory.
93 You can always use the `stdenv.mkDerivation` with the node_modules to build the package (symlink the node_modules directory and then use the package build command).
94 The node_modules abstraction can be also used to build some web framework frontends.
95 For an example of this see how [plausible](https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/web-apps/plausible/default.nix) is built. `mkYarnModules` to make the derivation containing node_modules.
96 Then when building the frontend you can just symlink the node_modules directory.
98 ## Javascript packages inside nixpkgs {#javascript-packages-nixpkgs}
100 The [pkgs/development/node-packages](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages) folder contains a generated collection of [npm packages](https://npmjs.com/) that can be installed with the Nix package manager.
102 As a rule of thumb, the package set should only provide _end user_ software packages, such as command-line utilities.
103 Libraries should only be added to the package set if there is a non-npm package that requires it.
105 When it is desired to use npm libraries in a development project, use the `node2nix` generator directly on the `package.json` configuration file of the project.
107 The package set provides support for the official stable Node.js versions.
108 The latest stable LTS release in `nodePackages`, as well as the latest stable current release in `nodePackages_latest`.
110 If your package uses native addons, you need to examine what kind of native build system it uses. Here are some examples:
116 After you have identified the correct system, you need to override your package expression while adding in build system as a build input.
117 For example, `dat` requires `node-gyp-build`, so we override its expression in [pkgs/development/node-packages/overrides.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/overrides.nix):
121 dat = prev.dat.override (oldAttrs: {
122 buildInputs = [ final.node-gyp-build pkgs.libtool pkgs.autoconf pkgs.automake ];
123 meta = oldAttrs.meta // { broken = since "12"; };
128 ### Adding and Updating Javascript packages in nixpkgs {#javascript-adding-or-updating-packages}
130 To add a package from npm to nixpkgs:
132 1. Modify [pkgs/development/node-packages/node-packages.json](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/node-packages.json) to add, update or remove package entries to have it included in `nodePackages` and `nodePackages_latest`.
136 ./pkgs/development/node-packages/generate.sh
139 3. Build your new package to test your changes:
142 nix-build -A nodePackages.<new-or-updated-package>
145 To build against the latest stable Current Node.js version (e.g. 18.x):
148 nix-build -A nodePackages_latest.<new-or-updated-package>
151 If the package doesn't build, you may need to add an override as explained above.
152 4. If the package's name doesn't match any of the executables it provides, add an entry in [pkgs/development/node-packages/main-programs.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/main-programs.nix). This will be the case for all scoped packages, e.g., `@angular/cli`.
153 5. Add and commit all modified and generated files.
155 For more information about the generation process, consult the [README.md](https://github.com/svanderburg/node2nix) file of the `node2nix` tool.
157 To update npm packages in nixpkgs, run the same `generate.sh` script:
160 ./pkgs/development/node-packages/generate.sh
163 #### Git protocol error {#javascript-git-error}
165 Some packages may have Git dependencies from GitHub specified with `git://`.
166 GitHub has [disabled unencrypted Git connections](https://github.blog/2021-09-01-improving-git-protocol-security-github/#no-more-unauthenticated-git), so you may see the following error when running the generate script:
169 The unauthenticated git protocol on port 9418 is no longer supported
172 Use the following Git configuration to resolve the issue:
175 git config --global url."https://github.com/".insteadOf git://github.com/
178 ## Tool specific instructions {#javascript-tool-specific}
180 ### buildNpmPackage {#javascript-buildNpmPackage}
182 `buildNpmPackage` allows you to package npm-based projects in Nixpkgs without the use of an auto-generated dependencies file (as used in [node2nix](#javascript-node2nix)).
183 It works by utilizing npm's cache functionality -- creating a reproducible cache that contains the dependencies of a project, and pointing npm to it.
188 { lib, buildNpmPackage, fetchFromGitHub }:
190 buildNpmPackage rec {
194 src = fetchFromGitHub {
198 hash = "sha256-BR+ZGkBBfd0dSQqAvujsbgsEPFYw/ThrylxUbOksYxM=";
201 npmDepsHash = "sha256-tuEfyePwlOy2/mOPdXbqJskO6IowvAP4DWg8xSZwbJw=";
203 # The prepack script runs the build script, which we'd rather do in the build phase.
204 npmPackFlags = [ "--ignore-scripts" ];
206 NODE_OPTIONS = "--openssl-legacy-provider";
209 description = "Modern web UI for various torrent clients with a Node.js backend and React frontend";
210 homepage = "https://flood.js.org";
211 license = lib.licenses.gpl3Only;
212 maintainers = with lib.maintainers; [ winter ];
217 In the default `installPhase` set by `buildNpmPackage`, it uses `npm pack --json --dry-run` to decide what files to install in `$out/lib/node_modules/$name/`, where `$name` is the `name` string defined in the package's `package.json`.
218 Additionally, the `bin` and `man` keys in the source's `package.json` are used to decide what binaries and manpages are supposed to be installed.
219 If these are not defined, `npm pack` may miss some files, and no binaries will be produced.
221 #### Arguments {#javascript-buildNpmPackage-arguments}
223 * `npmDepsHash`: The output hash of the dependencies for this project. Can be calculated in advance with [`prefetch-npm-deps`](#javascript-buildNpmPackage-prefetch-npm-deps).
224 * `makeCacheWritable`: Whether to make the cache writable prior to installing dependencies. Don't set this unless npm tries to write to the cache directory, as it can slow down the build.
225 * `npmBuildScript`: The script to run to build the project. Defaults to `"build"`.
226 * `npmWorkspace`: The workspace directory within the project to build and install.
227 * `dontNpmBuild`: Option to disable running the build script. Set to `true` if the package does not have a build script. Defaults to `false`. Alternatively, setting `buildPhase` explicitly also disables this.
228 * `dontNpmInstall`: Option to disable running `npm install`. Defaults to `false`. Alternatively, setting `installPhase` explicitly also disables this.
229 * `npmFlags`: Flags to pass to all npm commands.
230 * `npmInstallFlags`: Flags to pass to `npm ci`.
231 * `npmBuildFlags`: Flags to pass to `npm run ${npmBuildScript}`.
232 * `npmPackFlags`: Flags to pass to `npm pack`.
233 * `npmPruneFlags`: Flags to pass to `npm prune`. Defaults to the value of `npmInstallFlags`.
234 * `makeWrapperArgs`: Flags to pass to `makeWrapper`, added to executable calling the generated `.js` with `node` as an interpreter. These scripts are defined in `package.json`.
235 * `nodejs`: The `nodejs` package to build against, using the corresponding `npm` shipped with that version of `node`. Defaults to `pkgs.nodejs`.
236 * `npmDeps`: The dependencies used to build the npm package. Especially useful to not have to recompute workspace dependencies.
238 #### prefetch-npm-deps {#javascript-buildNpmPackage-prefetch-npm-deps}
240 `prefetch-npm-deps` is a Nixpkgs package that calculates the hash of the dependencies of an npm project ahead of time.
244 package.json package-lock.json index.js
245 $ prefetch-npm-deps package-lock.json
247 sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
250 #### fetchNpmDeps {#javascript-buildNpmPackage-fetchNpmDeps}
252 `fetchNpmDeps` is a Nix function that requires the following mandatory arguments:
254 - `src`: A directory / tarball with `package-lock.json` file
255 - `hash`: The output hash of the node dependencies defined in `package-lock.json`.
257 It returns a derivation with all `package-lock.json` dependencies downloaded into `$out/`, usable as an npm cache.
259 #### importNpmLock {#javascript-buildNpmPackage-importNpmLock}
261 `importNpmLock` is a Nix function that requires the following optional arguments:
263 - `npmRoot`: Path to package directory containing the source tree
264 - `package`: Parsed contents of `package.json`
265 - `packageLock`: Parsed contents of `package-lock.json`
266 - `pname`: Package name
267 - `version`: Package version
269 It returns a derivation with a patched `package.json` & `package-lock.json` with all dependencies resolved to Nix store paths.
271 This function is analogous to using `fetchNpmDeps`, but instead of specifying `hash` it uses metadata from `package.json` & `package-lock.json`.
273 Note that `npmHooks.npmConfigHook` cannot be used with `importNpmLock`. You will instead need to use `importNpmLock.npmConfigHook`:
276 { buildNpmPackage, importNpmLock }:
282 npmDeps = importNpmLock {
286 npmConfigHook = importNpmLock.npmConfigHook;
290 ### corepack {#javascript-corepack}
292 This package puts the corepack wrappers for pnpm and yarn in your PATH, and they will honor the `packageManager` setting in the `package.json`.
294 ### node2nix {#javascript-node2nix}
296 #### Preparation {#javascript-node2nix-preparation}
298 You will need to generate a Nix expression for the dependencies. Don't forget the `-l package-lock.json` if there is a lock file. Most probably you will need the `--development` to include the `devDependencies`
300 So the command will most likely be:
302 node2nix --development -l package-lock.json
305 See `node2nix` [docs](https://github.com/svanderburg/node2nix) for more info.
307 #### Pitfalls {#javascript-node2nix-pitfalls}
309 - If upstream package.json does not have a "version" attribute, `node2nix` will crash. You will need to add it like shown in [the package.json section](#javascript-upstream-package-json).
310 - `node2nix` has some [bugs](https://github.com/svanderburg/node2nix/issues/238) related to working with lock files from npm distributed with `nodejs_16`.
311 - `node2nix` does not like missing packages from npm. If you see something like `Cannot resolve version: vue-loader-v16@undefined` then you might want to try another tool. The package might have been pulled off of npm.
313 ### pnpm {#javascript-pnpm}
315 Pnpm is available as the top-level package `pnpm`. Additionally, there are variants pinned to certain major versions, like `pnpm_8` and `pnpm_9`, which support different sets of lock file versions.
317 When packaging an application that includes a `pnpm-lock.yaml`, you need to fetch the pnpm store for that project using a fixed-output-derivation. The functions `pnpm_8.fetchDeps` and `pnpm_9.fetchDeps` can create this pnpm store derivation. In conjunction, the setup hooks `pnpm_8.configHook` and `pnpm_9.configHook` will prepare the build environment to install the prefetched dependencies store. Here is an example for a package that contains a `package.json` and a `pnpm-lock.yaml` files using the above `pnpm_` attributes:
323 # This is pinned as { pnpm = pnpm_9; }
327 stdenv.mkDerivation (finalAttrs: {
329 version = "0-unstable-1980-01-01";
333 nativeBuildInputs = [
338 pnpmDeps = pnpm.fetchDeps {
339 inherit (finalAttrs) pname version src;
345 NOTE: It is highly recommended to use a pinned version of pnpm (i.e. `pnpm_8` or `pnpm_9`), to increase future reproducibility. It might also be required to use an older version, if the package needs support for a certain lock file version.
347 In case you are patching `package.json` or `pnpm-lock.yaml`, make sure to pass `finalAttrs.patches` to the function as well (i.e. `inherit (finalAttrs) patches`.
349 #### Dealing with `sourceRoot` {#javascript-pnpm-sourceRoot}
351 NOTE: Nixpkgs pnpm tooling doesn't support building projects with a `pnpm-workspace.yaml`, or building monorepos. It maybe possible to use `pnpm.fetchDeps` for these projects, but it may be hard or impossible to produce a binary from such projects ([an example attempt](https://github.com/NixOS/nixpkgs/pull/290715#issuecomment-2144543728)).
353 If the pnpm project is in a subdirectory, you can just define `sourceRoot` or `setSourceRoot` for `fetchDeps`. Note, that projects using `pnpm-workspace.yaml` are currently not supported, and will probably not work using this approach.
354 If `sourceRoot` is different between the parent derivation and `fetchDeps`, you will have to set `pnpmRoot` to effectively be the same location as it is in `fetchDeps`.
356 Assuming the following directory structure, we can define `sourceRoot` and `pnpmRoot` as follows:
369 pnpmDeps = pnpm.fetchDeps {
371 sourceRoot = "${finalAttrs.src.name}/frontend";
374 # by default the working directory is the extracted source
375 pnpmRoot = "frontend";
378 ### Yarn {#javascript-yarn}
380 Yarn based projects use a `yarn.lock` file instead of a `package-lock.json` to pin dependencies. Nixpkgs provides the Nix function `fetchYarnDeps` which fetches an offline cache suitable for running `yarn install` before building the project. In addition, Nixpkgs provides the hooks:
382 - `yarnConfigHook`: Fetches the dependencies from the offline cache and installs them into `node_modules`.
383 - `yarnBuildHook`: Runs `yarn build` or a specified `yarn` command that builds the project.
385 An example usage of the above attributes is:
399 stdenv.mkDerivation (finalAttrs: {
403 src = fetchFromGitHub {
406 rev = "v${finalAttrs.version}";
407 hash = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
410 yarnOfflineCache = fetchYarnDeps {
411 yarnLock = finalAttrs.src + "/yarn.lock";
412 hash = "sha256-mo8urQaWIHu33+r0Y7mL9mJ/aSe/5CihuIetTeDHEUQ=";
415 nativeBuildInputs = [
418 # Needed for executing package.json scripts
420 npmHooks.npmInstallHook
429 Note that there is no setup hook for installing yarn based packages - `npmHooks.npmInstallHook` should fit most cases, but sometimes you may need to override the `installPhase` completely.
431 #### `yarnConfigHook` arguments {#javascript-yarnconfighook}
433 By default, `yarnConfigHook` relies upon the attribute `${yarnOfflineCache}` (or `${offlineCache}` if the former is not set) to find the location of the offline cache produced by `fetchYarnDeps`. To disable this phase, you can set `dontYarnInstallDeps = true` or override the `configurePhase`.
435 #### `yarnBuildHook` arguments {#javascript-yarnbuildhook}
437 This script by default runs `yarn --offline build`, and it relies upon the project's dependencies installed at `node_modules`. Below is a list of additional `mkDerivation` arguments read by this hook:
439 - `yarnBuildScript`: Sets a different `yarn --offline` subcommand (defaults to `build`).
440 - `yarnBuildFlags`: Single string list of additional flags to pass the above command, or a Nix list of such additional flags.
442 ### yarn2nix {#javascript-yarn2nix}
444 WARNING: The `yarn2nix` functions have been deprecated in favor of the new `yarnConfigHook` and `yarnBuildHook`. Documentation for them still appears here for the sake of the packages that still use them. See also a tracking issue [#324246](https://github.com/NixOS/nixpkgs/issues/324246).
446 #### Preparation {#javascript-yarn2nix-preparation}
448 You will need at least a `yarn.lock` file. If upstream does not have one you need to generate it and reference it in your package definition.
450 If the downloaded files contain the `package.json` and `yarn.lock` files they can be used like this:
454 offlineCache = fetchYarnDeps {
455 yarnLock = src + "/yarn.lock";
461 #### mkYarnPackage {#javascript-yarn2nix-mkYarnPackage}
463 `mkYarnPackage` will by default try to generate a binary. For package only generating static assets (Svelte, Vue, React, WebPack, ...), you will need to explicitly override the build step with your instructions.
465 It's important to use the `--offline` flag. For example if you script is `"build": "something"` in `package.json` use:
470 export HOME=$(mktemp -d)
476 The `distPhase` is packing the package's dependencies in a tarball using `yarn pack`. You can disable it using:
484 The configure phase can sometimes fail because it makes many assumptions which may not always apply. One common override is:
489 ln -s $node_modules node_modules
494 or if you need a writeable node_modules directory:
499 cp -r $node_modules node_modules
500 chmod +w node_modules
505 #### mkYarnModules {#javascript-yarn2nix-mkYarnModules}
507 This will generate a derivation including the `node_modules` directory.
508 If you have to build a derivation for an integrated web framework (rails, phoenix..), this is probably the easiest way.
510 #### Overriding dependency behavior {#javascript-mkYarnPackage-overriding-dependencies}
512 In the `mkYarnPackage` record the property `pkgConfig` can be used to override packages when you encounter problems building.
514 For instance, say your package is throwing errors when trying to invoke node-sass:
517 ENOENT: no such file or directory, scandir '/build/source/node_modules/node-sass/vendor'
520 To fix this we will specify different versions of build inputs to use, as well as some post install steps to get the software built the way we want:
526 buildInputs = with final;[ python libsass pkg-config ];
528 LIBSASS_EXT=auto yarn --offline run build
536 #### Pitfalls {#javascript-yarn2nix-pitfalls}
538 - If version is missing from upstream package.json, yarn will silently install nothing. In that case, you will need to override package.json as shown in the [package.json section](#javascript-upstream-package-json)
539 - Having trouble with `node-gyp`? Try adding these lines to the `yarnPreBuild` steps:
544 mkdir -p $HOME/.node-gyp/${nodejs.version}
545 echo 9 > $HOME/.node-gyp/${nodejs.version}/installVersion
546 ln -sfv ${nodejs}/include $HOME/.node-gyp/${nodejs.version}
547 export npm_config_nodedir=${nodejs}
552 - The `echo 9` steps comes from this answer: <https://stackoverflow.com/a/49139496>
553 - Exporting the headers in `npm_config_nodedir` comes from this issue: <https://github.com/nodejs/node-gyp/issues/1191#issuecomment-301243919>
554 - `offlineCache` (described [above](#javascript-yarn2nix-preparation)) must be specified to avoid [Import From Derivation](#ssec-import-from-derivation) (IFD) when used inside Nixpkgs.
556 ## Outside Nixpkgs {#javascript-outside-nixpkgs}
558 There are some other tools available, which are written in the Nix language.
559 These that can't be used inside Nixpkgs because they require [Import From Derivation](#ssec-import-from-derivation), which is not allowed in Nixpkgs.
561 If you are packaging something outside Nixpkgs, consider the following:
563 ### npmlock2nix {#javascript-npmlock2nix}
565 [npmlock2nix](https://github.com/nix-community/npmlock2nix) aims at building `node_modules` without code generation. It hasn't reached v1 yet, the API might be subject to change.
567 #### Pitfalls {#javascript-npmlock2nix-pitfalls}
569 There are some [problems with npm v7](https://github.com/tweag/npmlock2nix/issues/45).
571 ### nix-npm-buildpackage {#javascript-nix-npm-buildpackage}
573 [nix-npm-buildpackage](https://github.com/serokell/nix-npm-buildpackage) aims at building `node_modules` without code generation. It hasn't reached v1 yet, the API might change. It supports both `package-lock.json` and yarn.lock.
575 #### Pitfalls {#javascript-nix-npm-buildpackage-pitfalls}
577 There are some [problems with npm v7](https://github.com/serokell/nix-npm-buildpackage/issues/33).