Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/explanation/custom-codecs.md
Original file line number Diff line number Diff line change
Expand Up @@ -342,4 +342,4 @@ Check for existing plugin codecs that may already solve your needs:
- **[dj-zarr-codecs](https://github.com/datajoint/dj-zarr-codecs)** — General numpy arrays with Zarr storage
- **[dj-photon-codecs](https://github.com/datajoint/dj-photon-codecs)** — Photon-limited movies with Anscombe transformation and compression

See the [Use Plugin Codecs](../how-to/use-plugin-codecs.md) guide for installation and usage of existing codec packages. Creating a custom codec is straightforward, but reusing existing ones saves time and ensures compatibility.
See the [Use Plugin Codecs](../how-to/use-plugin-codecs.md/) guide for installation and usage of existing codec packages. Creating a custom codec is straightforward, but reusing existing ones saves time and ensures compatibility.
10 changes: 5 additions & 5 deletions src/explanation/data-pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ A **scientific data pipeline** extends beyond a database with computations. It i

A DataJoint pipeline integrates three core components:

![DataJoint Platform Architecture](../images/dj-platform.png)
![DataJoint Platform Architecture](../images/dj-platform.png/)

| Component | Purpose |
|-----------|---------|
Expand All @@ -26,7 +26,7 @@ These components work together: code defines the schema and computations, the da

A DataJoint pipeline forms a **Directed Acyclic Graph (DAG)** at two levels:

![Pipeline DAG Structure](../images/pipeline-illustration.png)
![Pipeline DAG Structure](../images/pipeline-illustration.png/)

**Nodes** represent Python modules, which correspond to database schemas.

Expand Down Expand Up @@ -65,7 +65,7 @@ This model treats the database schema as an **executable workflow specification*

Each schema corresponds to a dedicated Python module. The module import structure mirrors the foreign key dependencies between schemas:

![Schema Structure](../images/schema-illustration.png)
![Schema Structure](../images/schema-illustration.png/)

```
my_pipeline/
Expand All @@ -78,7 +78,7 @@ my_pipeline/
│ └── analysis.py # analysis schema (depends on acquisition)
```

For practical guidance on organizing multi-schema pipelines, configuring repositories, and managing team access, see [Manage a Pipeline Project](../how-to/manage-pipeline-project.md).
For practical guidance on organizing multi-schema pipelines, configuring repositories, and managing team access, see [Manage a Pipeline Project](../how-to/manage-pipeline-project.md/).

## Object-Augmented Schemas

Expand Down Expand Up @@ -175,4 +175,4 @@ The schema remains central—defining data structures, dependencies, and computa
- [Relational Workflow Model](relational-workflow-model.md) — The conceptual foundation
- [Entity Integrity](entity-integrity.md) — Primary keys and dimensions
- [Type System](type-system.md) — Codec types and storage modes
- [Manage a Pipeline Project](../how-to/manage-pipeline-project.md) — Practical project organization
- [Manage a Pipeline Project](../how-to/manage-pipeline-project.md/) — Practical project organization
2 changes: 1 addition & 1 deletion src/explanation/entity-integrity.md
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,7 @@ Lineage enables **semantic matching**—DataJoint only joins attributes that
trace back to the same dimension. Two attributes named `id` from different
dimensions cannot be accidentally joined.

See [Semantic Matching](../reference/specs/semantic-matching.md) for details.
See [Semantic Matching](../reference/specs/semantic-matching.md/) for details.

### Recognizing Dimensions in Diagrams

Expand Down
12 changes: 6 additions & 6 deletions src/explanation/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ DataJoint provides a custom data definition language and [query algebra](query-a

### The Definition Language

DataJoint's [definition language](../reference/specs/table-declaration.md) is a standalone scripting language for declaring table schemas — not Python syntax embedded in strings. It is designed for uniform support across multiple host languages (Python, MATLAB, and potentially others). The same definition works identically regardless of which language you use.
DataJoint's [definition language](../reference/specs/table-declaration.md/) is a standalone scripting language for declaring table schemas — not Python syntax embedded in strings. It is designed for uniform support across multiple host languages (Python, MATLAB, and potentially others). The same definition works identically regardless of which language you use.

### Composite Primary Keys: A Clarity Comparison

Expand Down Expand Up @@ -74,13 +74,13 @@ The definition string **is** the specification — a declarative language that d

### Why Custom Query Algebra?

DataJoint's operators implement **[semantic matching](../reference/specs/semantic-matching.md)** — joins and restrictions match only on attributes connected through the foreign key graph, not arbitrary columns that happen to share a name. This prevents:
DataJoint's operators implement **[semantic matching](../reference/specs/semantic-matching.md/)** — joins and restrictions match only on attributes connected through the foreign key graph, not arbitrary columns that happen to share a name. This prevents:

- Accidental Cartesian products
- Joins on unrelated columns
- Silent incorrect results

Every query result has a defined **[entity type](entity-integrity.md)** with a specific [primary key](../reference/specs/primary-keys.md) (algebraic closure). SQL results are untyped bags of rows; DataJoint results are entity sets you can continue to query and compose.
Every query result has a defined **[entity type](entity-integrity.md)** with a specific [primary key](../reference/specs/primary-keys.md/) (algebraic closure). SQL results are untyped bags of rows; DataJoint results are entity sets you can continue to query and compose.

### Object-Augmented Schemas

Expand Down Expand Up @@ -191,7 +191,7 @@ No. DataJoint provides a Python API that abstracts SQL:
| `WHERE` | `table & condition` |
| `GROUP BY` | `.aggr()` |

Understanding relational concepts ([primary keys](entity-integrity.md), foreign keys, [normalization](normalization.md)) is helpful but not required to start. The [tutorials](../tutorials/index.md) teach these concepts progressively.
Understanding relational concepts ([primary keys](entity-integrity.md), foreign keys, [normalization](normalization.md)) is helpful but not required to start. The [tutorials](../tutorials/index.md/) teach these concepts progressively.

Since DataJoint uses standard database backends (MySQL, PostgreSQL), data remains accessible via SQL for users who prefer it or need integration with other tools.

Expand All @@ -208,7 +208,7 @@ The database maintains references to external objects, preserving:
- Query capability (filter by metadata, join across tables)
- Deduplication (identical content stored once)

See [Object Storage](../how-to/use-object-storage.md) for details.
See [Object Storage](../how-to/use-object-storage.md/) for details.

## Can Multiple Users Share a Pipeline?

Expand All @@ -225,4 +225,4 @@ Teams typically:
2. Connect to a shared database server
3. Run `populate()` from multiple machines simultaneously

See [Distributed Computing](../how-to/distributed-computing.md) for multi-process patterns.
See [Distributed Computing](../how-to/distributed-computing.md/) for multi-process patterns.
2 changes: 1 addition & 1 deletion src/explanation/query-algebra.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ DataJoint provides a powerful query algebra built on five core operators: restri

A fundamental property of DataJoint's query algebra is **algebraic closure**: every query result is itself a valid entity set with a well-defined **entity type** — you always know what kind of entity the result represents, identified by a specific primary key. Unlike SQL where query results are unstructured "bags of rows," DataJoint determines the entity type of each result based on the operator and the functional dependencies between operands.

This means operators can be chained indefinitely — the output of any operation is a valid input to any other operation. See [Primary Keys](../reference/specs/primary-keys.md) for the precise rules.
This means operators can be chained indefinitely — the output of any operation is a valid input to any other operation. See [Primary Keys](../reference/specs/primary-keys.md/) for the precise rules.

## Core Operators

Expand Down
20 changes: 10 additions & 10 deletions src/explanation/type-system.md
Original file line number Diff line number Diff line change
Expand Up @@ -369,16 +369,16 @@ class Network(dj.Computed):

**How-to Guides:**

- [Choose a Storage Type](../how-to/choose-storage-type.md) — Decision guide for selecting the right type
- [Configure Object Storage](../how-to/configure-storage.md) — Setting up stores for external data
- [Use Object Storage](../how-to/use-object-storage.md) — Working with `<blob@>`, `<attach@>`, `<object@>`
- [Use the npy Codec](../how-to/use-npy-codec.md) — Storing NumPy arrays as `.npy` files
- [Use Plugin Codecs](../how-to/use-plugin-codecs.md) — Installing and using third-party codecs
- [Create a Custom Codec](../how-to/create-custom-codec.md) — Building your own codec
- [Choose a Storage Type](../how-to/choose-storage-type.md/) — Decision guide for selecting the right type
- [Configure Object Storage](../how-to/configure-storage.md/) — Setting up stores for external data
- [Use Object Storage](../how-to/use-object-storage.md/) — Working with `<blob@>`, `<attach@>`, `<object@>`
- [Use the npy Codec](../how-to/use-npy-codec.md/) — Storing NumPy arrays as `.npy` files
- [Use Plugin Codecs](../how-to/use-plugin-codecs.md/) — Installing and using third-party codecs
- [Create a Custom Codec](../how-to/create-custom-codec.md/) — Building your own codec

**Reference:**

- [Type System Specification](../reference/specs/type-system.md) — Complete type reference
- [Codec API](../reference/specs/codec-api.md) — Codec class interface
- [npy Codec Specification](../reference/specs/npy-codec.md) — NpyRef and NpyCodec details
- [Object Store Configuration](../reference/specs/object-store-configuration.md) — Store settings reference
- [Type System Specification](../reference/specs/type-system.md/) — Complete type reference
- [Codec API](../reference/specs/codec-api.md/) — Codec class interface
- [npy Codec Specification](../reference/specs/npy-codec.md/) — NpyRef and NpyCodec details
- [Object Store Configuration](../reference/specs/object-store-configuration.md/) — Store settings reference
30 changes: 15 additions & 15 deletions src/explanation/whats-new-2.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ DataJoint 2.0 is a major release that establishes DataJoint as a mature framewor

> **📘 Upgrading from legacy DataJoint (pre-2.0)?**
>
> This page summarizes new features and concepts. For step-by-step migration instructions, see the **[Migration Guide](../how-to/migrate-to-v20.md)**.
> This page summarizes new features and concepts. For step-by-step migration instructions, see the **[Migration Guide](../how-to/migrate-to-v20.md/)**.

## Overview

Expand All @@ -27,13 +27,13 @@ If you're upgrading from legacy DataJoint, these changes require code updates:
| **Type syntax** | `longblob`, `int unsigned` | `<blob>`, `int64` |
| **Jobs** | `~jobs` table | Per-table `~~table_name` |

See the [Migration Guide](../how-to/migrate-to-v20.md) for complete upgrade steps.
See the [Migration Guide](../how-to/migrate-to-v20.md/) for complete upgrade steps.

## Object-Augmented Schema (OAS)

DataJoint 2.0 unifies relational tables with object storage into a single coherent system. The relational database stores metadata and references while large objects (arrays, files, Zarr datasets) are stored in object storage—with full referential integrity maintained across both layers.

→ [Type System Specification](../reference/specs/type-system.md)
→ [Type System Specification](../reference/specs/type-system.md/)

**Three storage sections:**

Expand All @@ -59,7 +59,7 @@ zarr_array : <object@store> # Path-addressed for Zarr/HDF5

**Breaking change**: DataJoint 2.0 makes all type conversions explicit through a three-tier architecture.

→ [Type System Specification](../reference/specs/type-system.md) · [Codec API Specification](../reference/specs/codec-api.md)
→ [Type System Specification](../reference/specs/type-system.md/) · [Codec API Specification](../reference/specs/codec-api.md/)

### What Changed

Expand Down Expand Up @@ -103,7 +103,7 @@ class GraphCodec(dj.Codec):

**Breaking change**: Redesigned job coordination with per-table job management.

→ [AutoPopulate Specification](../reference/specs/autopopulate.md) · [Job Metadata Specification](../reference/specs/job-metadata.md)
→ [AutoPopulate Specification](../reference/specs/autopopulate.md/) · [Job Metadata Specification](../reference/specs/job-metadata.md/)

### What Changed

Expand Down Expand Up @@ -139,7 +139,7 @@ Analysis.jobs.update({'session_id': 123}, priority=1) # High priority

**Breaking change**: Query operations now use **lineage-based matching** by default.

→ [Semantic Matching Specification](../reference/specs/semantic-matching.md)
→ [Semantic Matching Specification](../reference/specs/semantic-matching.md/)

### What Changed

Expand All @@ -163,7 +163,7 @@ result = TableA.join(TableB, semantic_check=False)

A cleaner configuration approach with separation of concerns.

→ [Configuration Reference](../reference/configuration.md)
→ [Configuration Reference](../reference/configuration.md/)

- **`datajoint.json`**: Non-sensitive settings (commit to version control)
- **`.secrets/`**: Credentials (never commit)
Expand Down Expand Up @@ -229,7 +229,7 @@ DataJoint 2.0 is licensed under the **Apache License 2.0** (previously LGPL-2.1)

## Migration Path

→ **[Complete Migration Guide](../how-to/migrate-to-v20.md)**
→ **[Complete Migration Guide](../how-to/migrate-to-v20.md/)**

Upgrading from DataJoint 0.x is a **phased process** designed to minimize risk:

Expand Down Expand Up @@ -273,19 +273,19 @@ Most users complete Phases 1-2 in a single session. Phases 3-4 only apply if you
## See Also

### Migration
- **[Migration Guide](../how-to/migrate-to-v20.md)** — Complete upgrade instructions
- [Configuration](../how-to/configure-database.md) — Setup new configuration system
- **[Migration Guide](../how-to/migrate-to-v20.md/)** — Complete upgrade instructions
- [Configuration](../how-to/configure-database.md/) — Setup new configuration system

### Core Concepts
- [Type System](type-system.md) — Understand the three-tier type architecture
- [Computation Model](computation-model.md) — Jobs 2.0 and AutoPopulate
- [Query Algebra](query-algebra.md) — Semantic matching and operators

### Getting Started
- [Installation](../how-to/installation.md) — Install DataJoint 2.0
- [Tutorials](../tutorials/index.md) — Learn by example
- [Installation](../how-to/installation.md/) — Install DataJoint 2.0
- [Tutorials](../tutorials/index.md/) — Learn by example

### Reference
- [Type System Specification](../reference/specs/type-system.md) — Complete type system details
- [Codec API](../reference/specs/codec-api.md) — Build custom codecs
- [AutoPopulate Specification](../reference/specs/autopopulate.md) — Jobs 2.0 reference
- [Type System Specification](../reference/specs/type-system.md/) — Complete type system details
- [Codec API](../reference/specs/codec-api.md/) — Build custom codecs
- [AutoPopulate Specification](../reference/specs/autopopulate.md/) — Jobs 2.0 reference
6 changes: 3 additions & 3 deletions src/how-to/choose-storage-type.md
Original file line number Diff line number Diff line change
Expand Up @@ -790,6 +790,6 @@ large_data : <object@>

- [Use Object Storage](use-object-storage.md) — How to use codecs in practice
- [Configure Object Storage](configure-storage.md) — Store configuration
- [Type System](../explanation/type-system.md) — Complete type system overview
- [Type System Specification](../reference/specs/type-system.md) — Technical details
- [NPY Codec Specification](../reference/specs/npy-codec.md) — NumPy array storage
- [Type System](../explanation/type-system.md/) — Complete type system overview
- [Type System Specification](../reference/specs/type-system.md/) — Technical details
- [NPY Codec Specification](../reference/specs/npy-codec.md/) — NumPy array storage
2 changes: 1 addition & 1 deletion src/how-to/deploy-production.md
Original file line number Diff line number Diff line change
Expand Up @@ -337,5 +337,5 @@ if __name__ == '__main__':
## See Also

- [Manage Pipeline Project](manage-pipeline-project.md) — Project organization
- [Configuration Reference](../reference/configuration.md) — All settings
- [Configuration Reference](../reference/configuration.md/) — All settings
- [Manage Secrets](manage-secrets.md) — Credential management
2 changes: 1 addition & 1 deletion src/how-to/fetch-results.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,4 +123,4 @@ All output methods accept:
## See Also

- [Query Data](query-data.md) — Building queries
- [Fetch API Specification](../reference/specs/fetch-api.md) — Complete reference
- [Fetch API Specification](../reference/specs/fetch-api.md/) — Complete reference
4 changes: 2 additions & 2 deletions src/how-to/manage-pipeline-project.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ A production DataJoint pipeline typically involves:
- **Shared infrastructure** — Database server, object storage, code repository
- **Coordination** — Between code, database, and storage permissions

This guide covers practical project organization. For conceptual background on pipeline architecture and the DAG structure, see [Data Pipelines](../explanation/data-pipelines.md).
This guide covers practical project organization. For conceptual background on pipeline architecture and the DAG structure, see [Data Pipelines](../explanation/data-pipelines.md/).

For a fully managed solution, [request a DataJoint Platform account](https://www.datajoint.com/sign-up).

Expand Down Expand Up @@ -369,7 +369,7 @@ These challenges grow with team size and pipeline complexity. The [DataJoint Pla
## See Also

- [Deploy to Production](deploy-production.md) — Production mode and environment configuration
- [Data Pipelines](../explanation/data-pipelines.md) — Conceptual overview and architecture
- [Data Pipelines](../explanation/data-pipelines.md/) — Conceptual overview and architecture
- [Configure Object Storage](configure-storage.md) — Storage setup
- [Distributed Computing](distributed-computing.md) — Multi-worker pipelines
- [Model Relationships](model-relationships.ipynb) — Foreign key patterns
2 changes: 1 addition & 1 deletion src/how-to/manage-secrets.md
Original file line number Diff line number Diff line change
Expand Up @@ -478,4 +478,4 @@ git log --all --full-history -- .secrets/datajoint.json

- [Configure Database Connection](configure-database.md) — Database-specific configuration
- [Configure Object Stores](configure-storage.md) — Storage-specific configuration
- [Configuration Reference](../reference/configuration.md) — Complete configuration options
- [Configuration Reference](../reference/configuration.md/) — Complete configuration options
Loading