feat(psl): set up declarative test suite

The main problem this addresses are:

- Diffs are noisy.
  Datamodel tests involve large strings literals, and very little
  significant information otherwise. In a single file, even for simple
  changes like reordering or renaming tests, or changing indentation,
  the diffs get unreadable quickly, as the large blocks of Prisma
  schemas and validation errors get mixed with the test function names
  and calls to test helpers. This state of things disincentivizes
  reorganizing tests. The new setup has one test per file, with the test
  name being the file name, including its full path from the validation
  tests root.

- Churn.
  We often need to subtly adjust many tests to account for trivial
  changes in the public APIs, when all we want to do is assert that a
  certain schema is valid, or invalid with a given error. This prevents
  us from evolving the datamodel API as much as we would like, and as a
  consequence of these changes being a lot of work, we have a lot of
  inconsistency in how tests are written and which test helpers are
  used. This has manifested recently in difficulties simply moving
  tests, without changing them, from `libs/datamodel/core` to `psl/psl`.

  With this test suite, we rely on a lot less context, setup and noise
  (variable names for expectations). We don't expect the shape of these
  tests to evolve much, churn should be minimal and if we need to change
  something, it will be possible to autofix through UPDATE_EXPECT=1.

- The test suite gets slower to compile as we add more tests, and that
  noticeably hurts our feedback loops.
  This setup is less rust code to compile: each test case translates to
  a single test function containing a single synchronous function call.
  We can transparently make test runs compile no rust code at all in the
  future if we need to.

Prior art:

- In prisma-engines: we already did that in introspection engine tests in that commit: ef84679
- In other compiler-like projects, this is a common approach. Here are a few examples:
  - go compiler (https://github.com/golang/go/blob/master/test/assign1.go)
  - rustc UI tests (https://github.com/rust-lang/rust/tree/master/src/test/ui)
  - zig compiler (ziglang/zig#11288)

next steps: same setup for reformatting tests

Author: tomhoule

Cargo.lock

@@ -20,10 +20,11 @@ regex = "1.3.7"
 serde = { version = "1.0.90", features = ["derive"] }
 serde_json = { version = "1.0", features = ["preserve_order", "float_roundtrip"] }
 enumflags2 = "0.7"
-either = "1.6"
 indoc = "1"
+either = "1.6"
 
 [dev-dependencies]
+dissimilar = "1.0.4"
 expect-test = "1.1.0"
 native-types = { path = "../../native-types" }
 pretty_assertions = "0.6.1"

libs/datamodel/core/Cargo.toml

@@ -10,73 +10,49 @@ Schema Language, and it is used by all Prisma engines in this repository.
 The query engine further processes Prisma schemas to produce the client API,
 and the DMMF JSON format.
 
-Here is a (slightly dated) overview diagram:
-
-![Architecture Overview](doc/images/overview.png?raw=true)
-
 ### Design goals
 
 - Strict parsing: a duplicate attribute, unknown attribute, unknown argument or extra argument is an error.
 - Expose a _convenient_ and _obvious_ / hard-to-misuse public API.
 - Expose the source information (AST node spans, etc) in the parsed schema.
 - Accumulate errors to present them at the end instead of returning early.
 
-### Data Formats
-
-**Sources** represents the different datasources declared in the schema.
-
-**DML** is a datamodel data structure which is used by other Rust components of Prisma.
-
-**AST** is the internal AST representation of a Prisma datamodel V2 file.
-
-**Datamodel V2 String** is the string representation of a Prisma datamodel V2 file.
-
-**DMMF** Internal JSON format for transferring datamodel and schema information
-between different components of Prisma. The DMMF is in parts in the `dmmf`
-crate in datamodel, but for the most part defined in the query engine.
-
-### Steps
-
-**Parse** parses a string to an AST and performs basic syntactic checks.
-
-**Load Sources** Loads all datasource and generator declarations. This injects
-source-specific attributes into the validation pipeline.
-
-**Validate** performs several checks to ensure the datamodel is valid. This
-includes, for example, checking invalid type references, or relations which are
-impossible to model on a database.
+## Usage
 
-**Lift** converts a validated schema to a DML struct. That step cannot fail, it
-does not perform any validation.
+Please see [`lib.rs`](src/lib.rs) and the [rustdoc documentation](https://prisma.github.io/prisma-engines/doc/datamodel/) for the public API.
 
-**Lower** generates an AST from a DML struct. This step will attempt to
-minimize the AST by removing all attributes and attribute arguments which are a
-default anyway.
+Main use-case, parsing a string to datamodel:
 
-**Render** renders a given AST to its string representation.
+```
+let file = std::fs::read_to_string(&args[1]).unwrap();
+let validated_schema = datamodel::parse_schema_parserdb(&file)?;
+```
 
-## Error handling
+## Tests
 
-The datamodel parser strives to provide good error diagnostics to schema
-authors. As such, it has to be capable of dealing with partially broken input
-and keep validating. The validation process can however not proceed to the
-validating models in presence of a broken datasource, for example, because that
-would lead to a cascade of misleading errors. Like other parsers, we introduce
-*cutoff points* where validation will stop in the presence of errors.
+###  Running the test suite
 
-These are:
+Run `cargo test` for this crate (`datamodel`). There should be no setup required beyond that.
 
-- AST parsing stage. Syntax errors.
-- Configuration validation
-- Datamodel validation
+### For test authors
 
-## Usage
+There are two entry points for PSL tests. The tests involving writing plain
+Rust tests, in `datamodel_tests.rs`, and the declarative validation tests, in
+`validation.rs`.
 
-Please see [`lib.rs`](src/lib.rs) and the [rustdoc documentation](https://prisma.github.io/prisma-engines/doc/datamodel/) for all convenience methods.
+For new tests, the validation test suite should be preferred: the tests are
+more straightforward to write, more declarative (no unnecessary function calls,
+test helpers, variable names, etc. that we always have to change at some
+point), and much faster to compile.
 
-Main use-case, parsing a string to datamodel:
+A validation test is a `.prisma` file in the `tests/validation` directory, or
+any of its subdirectories (transitively). It is a regular Prisma schema that
+will be passed through the PSL validation process, and is expected to be valid,
+or return errors. That expectation is defined by the comment (`//`) at the end
+of the file. If there is no comment at the end of the file, the schema is
+expected to be valid. If there is a comment at the end of the file, the
+rendered, user-visible validation warnings and errors will be expected to match
+the contents of that comment.
 
-```
-let file = std::fs::read_to_string(&args[1]).unwrap();
-let validated_schema = datamodel::parse_schema_parserdb(&file)?;
-```
+Like in `expect_test` tests, the last comment can be left out at first, and
+updated like a snapshot test using the `UPDATE_EXPECT=1` env var.

libs/datamodel/core/README.md

@@ -0,0 +1,46 @@
+use std::{env, fs, io::Write as _, path};
+
+const ROOT_DIR: &str = "tests/validation";
+
+fn main() {
+    println!("cargo:rerun-if-changed={ROOT_DIR}");
+
+    let mut all_schemas = Vec::new();
+    find_all_schemas("", &mut all_schemas);
+
+    let out_dir = env::var("OUT_DIR").unwrap();
+    let out_file_path = path::Path::new(&out_dir).join("validation_tests.rs");
+    let mut out_file = fs::File::create(out_file_path).unwrap();
+
+    for schema_path in &all_schemas {
+        let test_name = schema_path.trim_start_matches('/').trim_end_matches(".prisma");
+        let test_name = test_name.replace(&['/', '\\'], "_");
+        let file_path = schema_path.trim_start_matches('/');
+        writeln!(
+            out_file,
+            "#[test] fn {test_name}() {{ run_validation_test(\"{file_path}\"); }}"
+        )
+        .unwrap();
+    }
+}
+
+fn find_all_schemas(prefix: &str, all_schemas: &mut Vec<String>) {
+    let cargo_manifest_dir = env!("CARGO_MANIFEST_DIR");
+    for entry in fs::read_dir(format!("{cargo_manifest_dir}/{ROOT_DIR}/{prefix}")).unwrap() {
+        let entry = entry.unwrap();
+        let file_name = entry.file_name();
+        let file_name = file_name.to_str().unwrap();
+        let entry_path = format!("{}/{}", prefix, file_name);
+        let file_type = entry.file_type().unwrap();
+
+        if file_name == "." || file_name == ".." {
+            continue;
+        }
+
+        if file_type.is_file() {
+            all_schemas.push(entry_path);
+        } else if file_type.is_dir() {
+            find_all_schemas(&entry_path, all_schemas);
+        }
+    }
+}

libs/datamodel/core/build.rs

@@ -17,7 +17,6 @@ mod ignore_positive;
 mod index_clustering;
 mod index_negative;
 mod index_positive;
-mod map_negative;
 mod map_positive;
 mod postgres_indices;
 mod relations;

libs/datamodel/core/doc/images/overview.png

@@ -1,5 +1,4 @@
 mod cockroachdb;
-mod mongodb;
 mod mysql;
 mod postgres;
 mod sqlite;

libs/datamodel/core/tests/attributes/map_negative.rs

@@ -0,0 +1,16 @@
+model User {
+  id Int @id
+  fooId Int
+  relationField  Foo @relation(fields: [fooId], references: [id]) @map("custom_name")
+}
+
+model Foo {
+  id Int @id
+}
+
+// error: Error parsing attribute "@map": The attribute `@map` cannot be used on relation fields.
+//   -->  schema.prisma:4
+//    | 
+//  3 |   fooId Int
+//  4 |   relationField  Foo @relation(fields: [fooId], references: [id]) @map("custom_name")
+//    | 

libs/datamodel/core/tests/attributes/mod.rs

@@ -0,0 +1,16 @@
+datasource mydb {
+    provider = "mongodb"
+    url = env("TEST_DB_URL")
+}
+
+model Foo {
+  id    Int    @id @map("_id")
+  field String @map("field.schwield")
+}
+
+// error: Error parsing attribute "@map": The field name cannot contain a `.` character
+//   -->  schema.prisma:8
+//    | 
+//  7 |   id    Int    @id @map("_id")
+//  8 |   field String @map("field.schwield")
+//    | 

libs/datamodel/core/tests/capabilities/mod.rs

@@ -0,0 +1,16 @@
+datasource mydb {
+    provider = "mongodb"
+    url = env("TEST_DB_URL")
+}
+
+model Foo {
+  id    Int    @id @map("_id")
+  field String @map("$field")
+}
+
+// error: Error parsing attribute "@map": The field name cannot start with a `$` character
+//   -->  schema.prisma:8
+//    | 
+//  7 |   id    Int    @id @map("_id")
+//  8 |   field String @map("$field")
+//    | 

libs/datamodel/core/tests/capabilities/mongodb.rs

@@ -0,0 +1,16 @@
+datasource mydb {
+    provider = "mongodb"
+    url = env("TEST_DB_URL")
+}
+
+model User {
+  id Int @id @default(autoincrement()) @map("_id")
+}
+
+
+// error: Error parsing attribute "@default": The `autoincrement()` default value is used with a datasource that does not support it.
+//   -->  schema.prisma:7
+//    | 
+//  6 | model User {
+//  7 |   id Int @id @default(autoincrement()) @map("_id")
+//    | 

libs/datamodel/core/tests/validation/attributes/map/map_must_error_for_relation_fields.prisma

@@ -0,0 +1,9 @@
+datasource mydb {
+    provider = "mongodb"
+    url = env("TEST_DB_URL")
+}
+
+
+type Address {
+    street String
+}