For Instance Admins

Setting Up the User Repository

An administrator's guide to adding analysis tools and preparing the repository that stores folders and steps.

The user repository stores the available analysis tools, the steps researchers create, and the folders that collect them. As the administrator, you set it up and maintain its templates. Researchers use those templates through the agent or the step forms; they do not need to edit the setup described here.

A template has two parts. A Pointy language declaration describes the program's inputs and output. A Nix file chooses how the inputs appear in the browser and how the program runs. The standard library combines them into the configuration Pointy reads.

Choose compatible inputs

The current frontend reads step-config version 4. This guide describes the main branch of pointy-stdlib, which produces that format. Its schema comes from pointy-lang. Older examples using compile.options and pointyLib.types describe a different API; selecting the stdlib's old branches is not a compatible substitute.

Declare pointy-stdlib and let its own lock file choose the language inputs. The stdlib's lock pins pointy-lang and pointy-lang-stdlib, so a repository that takes its language inputs from pointy-stdlib uses the compiler and language library the stdlib was tested with. The nixpkgs follow in the example below keeps the repository on the same nixpkgs as the compiler. You normally do not need to override the language inputs.

The two language repositories are private, and the lock file records them as git+ssh URLs. Every machine that locks, evaluates, or builds the repository — your workstation and the Pointy host — needs SSH read access to 421anon/pointy-lang and 421anon/pointy-lang-stdlib.

For local language development, declare pointy-lang yourself and override its fixtures and pointy-lang-stdlib inputs with absolute paths. Keep the language, stdlib, and language-library revisions together, and do not commit a lock file that depends on paths available only on your laptop.

Use Nix with nix-command, flakes, and pipe-operators enabled. This stdlib fixes the target system to x86_64-linux. Generating the argument schema uses import-from-derivation: evaluation may build the language compiler and schema, and must permit that operation.

Minimal repository structure

Start with:

flake.nix
flake.lock
pointy.nix
main.pointy
templates/hello.nix
steps/1.nix
projects/0.nix
projects/1.nix

The examples below form one small working repository.

  • pointy.nix is required. It supplies the package set and the entry program.
  • main.pointy must have that exact name at the repository root. It must be a single file: only this file is copied for evaluation, so it cannot import other paths.
  • Every entry in templates/ is loaded as a template, and the filename is the template key. Put helper Nix files in packages/, not here.
  • Every regular <id>.nix file in steps/ is a step. Do not put a placeholder such as .gitkeep there; it would be read as a step named ".gitkeep" and fail evaluation.
  • projects/ holds folders, including the root projects/0.nix ("Home").
  • srcFiles/ and presets/ are optional; they can be missing until you need them.

loadDir imports every entry in templates/, projects/, and presets/ as Nix, so do not put placeholder or unrelated files in those directories.

You author the flake, language declarations, and templates. Pointy writes steps, folders, and source files in response to UI actions. You can edit those files outside Pointy too, provided you preserve their structure and push your changes to the configured branch.

Minimal flake.nix and pointy.nix

{
  inputs = {
    pointy-stdlib.url = "github:421anon/pointy-stdlib";
    nixpkgs.follows = "pointy-stdlib/nixpkgs";
  };

  outputs = inputs@{ pointy-stdlib, ... }:
    pointy-stdlib.lib.mkFlake { inherit inputs; };
}
{
  inputs,
  ...
}:
{
  pkgs = inputs.nixpkgs.legacyPackages.x86_64-linux;
  source = ./main.pointy;
}

source points to the language file that declares the template inputs. pkgs supplies the Nix packages used to prepare those declarations and run the steps. Both are required. Pointy generates the forms from these declarations; you do not maintain a second set of field definitions in the frontend.

The stdlib copies flake.nix, pointy.nix, templates/, packages/, and main.pointy into one store path and reads them from there, so pointy.nix and your templates may reach those paths and nothing else. The steps, projects, and presets are read from the repository itself, one store path per step; a step definition holds everything it needs and cannot import a sibling file.

A repository that needs flake-parts configuration of its own passes it as modules, where the loaded pointy.nix result is available as pointyRepo.cfg:

outputs = inputs@{ pointy-stdlib, ... }:
  pointy-stdlib.lib.mkFlake {
    inherit inputs;
    modules = [
      ({ pointyRepo, ... }: {
        perSystem = { system, ... }: {
          _module.args.pkgs = pointyRepo.cfg.pkgs;
        };
      })
    ];
  };

Declare the arguments

Put this in main.pointy:

program example {
  struct Greeting
    (message : String = "Hello, research!")
    : Directory < /message.txt >

  struct DataSource : Directory

  struct Download (url : String) : Directory < /data >
}

Greeting has one text argument with a default and describes a directory containing message.txt. The other two declarations are used by the optional upload and download examples below. The language declaration sets parameter names, types, order, defaults, and which values are required. The Nix template chooses how to display them.

Evaluating the form document does not build or check anything. When a step runs, Pointy builds the step's certificate, which checks the output against the output type its program declares, and the step's status reports whether that check passed. See Reviewing results.

Example: derivation template

Put this in templates/hello.nix:

{
  displayName = "Greeting";
  description = "Write a message into a result file.";
  sortKey = 0;
  icon = "description";
  contract.interface = "Greeting";
  pointy.type.derivation = { };

  bindings.message = {
    displayName = "Message";
    description = "Text to put in message.txt.";
    widget.kind = "textarea";
  };

  compile = { lib, ... }: {
    build = { args, ... }: {
      mkDerivation = {
        installPhase = ''
          mkdir -p "$out"
          printf '%s\n' ${lib.escapeShellArg args.message} > "$out/message.txt"
        '';
        dontFixup = true;
      };
    };
  };
}

The filename gives the template its key, hello. contract.interface connects it to Greeting, and bindings.message changes the label and control without redefining the argument. The builder receives the resolved args. Shell quoting matters: a user's message is data here, not a shell fragment to execute.

compile receives { lib, pkgs, pointyLib } and returns a build function. That function receives { args, public }, where public is the step derivation. It returns mkDerivation attributes, optionally name, version, and env. The stdlib supplies the standard Nix build setup and Pointy metadata, including the step id, template type, arguments, and resource requirements. You do not need to repeat those in every template.

Add a step and a folder

Put this in steps/1.nix:

{
  type = "hello";
  name = "First greeting";
  args.message = "Hello, research!";
}

type selects the template, name is the label shown in Pointy, and args holds the parameter values keyed by their language names.

Put this in projects/1.nix:

{
  name = "Example analysis";
  templates = [ "hello" ];
  children = [
    { step = { id = 1; hidden = false; sortKey = 0; }; }
  ];
}

Put this in projects/0.nix:

{
  name = "Home";
  templates = [ ];
  children = [
    { project = { id = 1; hidden = false; sortKey = 0; }; }
  ];
}

Each projects/<id>.nix file is a plain attribute set:

  • name is the folder name shown in Pointy.
  • Exactly one of preset (a preset key) or templates (a list of template keys) is required, in projects/0.nix too; use templates = [ ]; for a folder that starts with no template selection.
  • children lists the folder's contents. Each entry is exactly one of { step = { ... }; } or { project = { ... }; }, with an integer id and the placement fields hidden and sortKey.
  • Entries with a sortKey come first, ordered by (sortKey, id); entries with sortKey = null follow, ordered by id. hidden and sortKey belong to the entry, not to the child: the same step or folder can sit in several folders with different placement.
  • projects/0.nix is the root folder, Home. It cannot become a child or be deleted, and folders created in the UI are linked under it; keep the file in the repository.
  • The children graph must stay acyclic; the backend refuses a change that would introduce a cycle. An id with no matching steps/<id>.nix or projects/<id>.nix is dropped and reported in validationErrors.

A step linked in no folder is unfiled; it stays reachable through search and the Link existing action. Organize actions rewrite these files: Pointy re-renders the file from its data, so comments are dropped, keys are sorted, and the parent's sort keys are renumbered. See Managing Projects for the UI actions.

Add the files to Git before the first flake command. Nix does not include untracked files automatically, and a Git-backed flake with an untracked flake.nix is rejected. From the repository root:

nix flake lock
nix eval --json .#pointy.stepConfig
nix eval --json .#pointy.projects
nix build .#pointy.steps.1
cat result/message.txt
nix build .#pointy.steps.1.certificate

The configuration should have version: 4, the projects output should show Home and Example analysis, the build should write Hello, research!, and the certificate build should succeed. Once the repository and its inputs are available to the backend, configure its URL and branch as described in the Instance Admin Guide.

Migrating to the folder format

A repository locked to an older stdlib stores its folders as a flat steps = [ ... ] list in projects/<id>.nix. That shape no longer evaluates: with the current stdlib, nix eval .#pointy.projects fails with error: attribute 'children' missing. The backend still reads revisions in the old format, so past commits remain viewable, but it refuses to organize such files:

projects/<id>.nix uses the legacy steps format. Run pointy-migrate-project-tree to convert it.

The migration tool is part of the Pointy backend package, not a flake app. It needs nix on PATH. On a clean checkout of the repository, on the configured branch, update the pointy-stdlib input to a revision that reads children (the main branch this guide describes), run the tool, and check the result:

nix flake update pointy-stdlib
nix build github:421anon/pointy#backend
result/bin/pointy-migrate-project-tree .
nix eval --json .#pointy.projects

The tool:

  • refuses to run when projects/0.nix already exists (... already exists; the projects are already in the children format.) or when the checkout has no projects/ directory;
  • rewrites each projects/<id>.nix: the old steps entries become tagged children, and the project-level hidden and sortKey are dropped;
  • writes projects/0.nix ("Home") listing every existing folder with its old placement;
  • does not commit or push. It re-renders the files from JSON, so comments are lost and keys are sorted.

Commit the lock bump and the migrated files together, then push.

Presets

A preset bundles templates under a useful name, such as "RNA-seq", so people do not have to reconstruct the same selection for each folder. Presets are read from presets/, one file per preset; the filename is the preset key. Add this to presets/example.nix:

{
  displayName = "Example analysis";
  description = "A small starting point.";
  sortKey = 0;
  templates = [ "hello" ];
}

templates is required and every entry must be a loaded template; description and sortKey default to null, and displayName must be a string. An unknown template fails evaluation with Preset `<name>` references unknown templates: <list>..

A folder must choose exactly one of preset = "example"; or templates = [ "hello" ];. The selection does not remove the folder's steps or filter its contents; it decides which templates appear directly in the folder's New menu, and the remaining templates stay available under Other types. When someone creates a folder, its initial preset is the one with the smallest sortKey (a preset without a sortKey sorts last); if the repository defines no presets, the folder starts with an empty custom selection.

Making extra packages available to templates

Extend pkgs in pointy.nix:

pkgs = inputs.nixpkgs.legacyPackages.x86_64-linux.extend (_: prev: {
  myLab.helper = prev.callPackage ./packages/helper { };
});

Templates can then use pkgs.myLab.helper. packages/ is copied into the same store path as pointy.nix and the templates, but Pointy never loads its files as templates: import them explicitly. A package edit is visible to step evaluation. A flake-parts module that needs the same package set reads it from the pointyRepo module argument, as pointyRepo.cfg.pkgs.

Example: file upload template

Add templates/dataSource.nix using the DataSource declaration from main.pointy:

{
  displayName = "Data source";
  description = "Upload files for an analysis.";
  sortKey = 1;
  contract.interface = "DataSource";
  pointy.type.fileUpload.allowedExtensions = [ ".csv" ".tsv" ];

  compile = { ... }: {
    build = { args, ... }: {
      mkDerivation = {
        installPhase = "ln -s ${args.uploaded} \"$out\"";
        dontFixup = true;
      };
    };
  };
}

allowedExtensions is required; use null to accept any extension. It controls the browser picker, not a server-side security boundary. An upload runs as an ingest job; when it succeeds, the backend records the payload hash in args.uploaded, and the stdlib resolves that value to the stored object for the builder; a missing store object fails the build. args.uploaded is not a language parameter or a user-entered form field. Keep uploaded store objects in your backup plan: the upload reference cannot reconstruct missing data from Git alone.

Example: download template

Add templates/download.nix using the Download declaration:

{
  displayName = "Download";
  description = "Fetch a remote file and retain its content hash.";
  sortKey = 2;
  contract.interface = "Download";
  pointy.type.download = { };
  bindings.url.displayName = "URL";

  compile = { ... }: {
    build = { args, ... }: {
      mkDerivation = {
        installPhase = ''
          mkdir -p "$out"
          ln -s ${args.downloaded} "$out/data"
        '';
        dontFixup = true;
      };
    };
  };
}

Unlike the generated downloaded value, url is a parameter in the language declaration. On save, the backend prefetches it and writes args.downloaded = { url; hash; downloadedAt; }. The stdlib resolves that value with pkgs.fetchurl; the form also gets a read-only download timestamp. An unchanged URL reuses the recorded hash, and the URL must use HTTP or HTTPS. See Download steps.

Resource requirements

A template may declare an attribute set or a function of resolved arguments:

requirements = args: {
  cpu = 2;
  ram = "4G";
};

Template requirements are merged over { cpu = 1; ram = "1G"; ior = "0"; iow = "0"; }. A step's own non-null requirements takes precedence as a complete override and must contain all four fields. The Slurm policy decides whether CPU and memory requests are enforced; see Backend configuration.

Optional step notices

Inside the builder's mkDerivation attributes, a template can add:

passthru.meta.pointy.notices = [
  {
    field = "message";
    severity = "info";
    message = "This text is written verbatim to the output file.";
  }
];

Notices are Markdown hints loaded with the parameter form. Use the exact field name and severity = "info"; other severities are not supported by the current frontend. A notice does not enforce a constraint or block a build.

The step id and metadata

The stdlib adds args.id from the numeric filename, as a string, and supplies meta.pointy.id, type, requirements, and args on the exported step. Do not declare a hidden id parameter or copy that metadata into every template. Put additional metadata, such as notices, under mkDerivation.passthru.meta.pointy.

A step definition may also carry note, requirements, refine, reviewedRevision, reviewedBy, and reviewComments. requirements overrides the template's resources and refine adds a type expression to the output the step must certify; neither reaches the build derivation. The review fields are managed through the normal save and review operations. A UI save rewrites the whole file from the form fields, so keep hand-added requirements, refine, and comments in Git. See Reviewing results.

Injecting srcFiles into a build

Enable repository-backed files at the template's top level:

pointy.type.derivation = {
  withSrcFiles = true;
};

The stdlib symlinks each top-level entry from srcFiles/<step-id>/ into the build working directory during unpacking. If the directory is absent, there is nothing to unpack. srcFiles/ is optional; create it when you first add a source file. The UI exposes the same files for browsing and editing. Treat them as read-only build inputs; write generated files elsewhere and copy results into $out.

Injecting typed column metadata via extras

For CSV/TSV output, add this at the template's top level:

extras = "csv";

The stdlib attaches a separate scanner derivation as meta.pointy.extras. It mirrors output directories with meta.json files describing column types and nullability. This lets the grid viewer sort numbers as numbers. extras = "fastq"; instead scans FASTQ files for read counts; a line count not divisible by four fails that scan.

Metadata is requested per folder. Pointy can build missing extras in the background, and a failed scan does not prevent ordinary file browsing. Each returned meta.json is limited to 10 MiB. CSV scans default to 2 CPUs and 2 GiB of RAM; FASTQ scans default to 1 CPU and 1 GiB.

For a custom extras builder, omit the top-level shorthand and attach your derivation under mkDerivation.passthru.meta.pointy.extras. It should expose requirements = { cpu; ram; ior; iow; } for scheduling. The public pointyLib.csvExtras and pointyLib.fastqExtras helpers accept { pkgs; baseDrv; requirements?; }, where baseDrv can be the builder's public argument.

A CSV folder's metadata looks like this, keyed by filename with columns in header order:

{
  "results.csv": {
    "delimiter": ",",
    "hasHeader": true,
    "columns": [
      { "type": "string", "nullable": false },
      { "type": "float", "nullable": true }
    ]
  }
}

Supported column types are int, float, and string. Unknown types and columns beyond the supplied schema fall back to strings.

After you commit template changes

Evaluate .#pointy.stepConfig and build a representative step before pushing a template change. Keep the language declaration and its Nix bindings in the same change, and commit the lock file when inputs change. Once the backend has fetched the revision, reload the UI to pick up the new form configuration.

A change to flake.nix, pointy.nix, templates/, packages/, main.pointy, or flake.lock moves the key of every step. A step file or its srcFiles/ directory moves that step's key and the keys of the steps that consume it. Pointy stores certificates per key and refreshes them after repository changes, so expect a rebuild after such a change.

The Type Reference covers bindings and widgets in detail. The CLI Reference gives commands for inspecting configurations, dependencies, and outputs without going through the browser.

Built withSourcey
GitHub