^ ^                 ^ ^
=(o.o)=  ~byakuren  =(o.o)=
  m m                 m m

Quad Tests

Quad (Query Using Answer Description) is Flowlog’s conformance testing framework for ISO Prolog behavior. This document describes the quad test format, the test suites, and how to run them.

Code references (section markers in flowlog.c):

• Engine implementations: SECTIONS 22/23/24
• CLI entrypoints and REPL behavior used by the runners: SECTION 26

Overview

Quad tests are embedded REPL transcripts in .pl files that verify Prolog queries produce expected results. The format is simple and readable:

?- member(X, [1,2,3]).
   X = 1
;  X = 2
;  X = 3.

The framework parses queries (?- Goal.) and their answer descriptions, executes the queries, and verifies the results match.

Running Quad Tests

From the project root:

# Run all quad tests (Flowlog)
make quad

# Run all quad tests in other Flowlog engines
make quad-wam
make quad-wamvm

# Run all quad tests with tracing enabled (Flowlog; output on stderr)
make quad-trace
make quad-itrace

# Run against reference engines (Trealla, Scryer)
make quad-tpl
make quad-scryer

# Run a single quad file (two equivalent forms)
make quad tests/predicates/assertz_quad.pl
make quad QUAD_FILE=tests/predicates/assertz_quad.pl

These make targets invoke tests/quad.sh:

• make quad runs ./flowlog with --parallel-profile iso --engine tree (SUBSECTION 26.1)
• make quad-wam runs ./flowlog with --parallel-profile iso --engine wam (SUBSECTION 26.1)
• make quad-wamvm runs ./flowlog with --parallel-profile iso --engine wamvm (SUBSECTION 26.1)
• make quad-tpl runs tpl
• make quad-scryer runs scryer-prolog

You can also run the runner directly if you want different engine flags:

bash tests/quad.sh ./flowlog --parallel-profile iso --engine wamvm

Process Isolation

Quad tests are run by tests/quad.sh, which launches a fresh Prolog process per *_quad.pl file. This means a crash or hang in one test file does not prevent later test files from running.

Timeouts

If timeout(1) is available, each quad file is limited to QUAD_TIMEOUT seconds (default: 30). Timeouts are reported as a CRASH for that file.

Per-test Failure Logs

When a quad file FAILs or CRASHes, the runner writes a per-test log file containing that test’s output (with internal runner markers removed).

• Default location (Flowlog): results/<test_path>.log
• Default location (other engines): results/<prolog>/<test_path>.log
• Examples:
  • results/predicates/arg_quad.pl.log
  • results/tpl/ulrich/conformity_quad.pl.log
  • results/scryer-prolog/predicates/open_quad.pl.log
• Logs are only created for failing/crashing files; a fully successful run produces no log files.
• For Flowlog, this is designed so you can take a failing test path like tests/predicates/arg_quad.pl and replace the leading tests/ with results/ to find the corresponding log.
• Control the output directory with QUAD_RESULTS_DIR (default: results). Set it to an empty string to disable per-test logs entirely.

Test File Location

Quad test files are organized under tests/:

tests/
├── quad.sh                    # Shell runner used by `make quad*`
├── ulrich/                    # Ulrich Neumerkel's ISO conformity tests
│   ├── conformity_quad.pl     # Syntax conformity tests
│   ├── number_chars_quad.pl   # number_chars/2 tests
│   ├── variable_names_quad.pl # read_term variable_names option tests
│   ├── member_quad.pl         # member/2 prologue tests
│   ├── append_quad.pl         # append/3 prologue tests
│   ├── length_quad.pl         # length/2 prologue tests
│   ├── between_quad.pl        # between/3 prologue tests
│   ├── select_quad.pl         # select/3 prologue tests
│   ├── succ_quad.pl           # succ/2 prologue tests
│   ├── maplist_quad.pl        # maplist/2..4 prologue tests
│   ├── nth_quad.pl            # nth0/3, nth1/3 prologue tests
│   ├── foldl_quad.pl          # foldl/4..6 prologue tests
│   ├── call_nth_quad.pl       # call_nth/2 prologue tests
│   └── countall_quad.pl       # countall/2 prologue tests
└── predicates/                # ISO built-in predicate tests
    ├── abolish_quad.pl
    ├── arg_quad.pl
    ├── ... (140+ predicate test files)
    └── writeq_quad.pl

Answer Description Syntax

Basic Results

% Query succeeds once with no bindings
?- true.
   true.

% Query fails
?- fail.
   false.

% Query succeeds with bindings
?- X = 1.
   X = 1.

% Multiple solutions (disjunction)
?- member(X, [a,b]).
   X = a
;  X = b.

Discarding Bindings (\+ \+)

When a test has variables but expects just true (we only care that it succeeds, not what the bindings are), wrap the query with \+ \+:

% Without wrapper - framework captures bindings, test fails
?- T = (X, Y), write_term(T, [variable_names(['X'=X, 'Y'=Y])]).
   true.                           % FAILS: got T = (X,Y), X = X, Y = Y

% With wrapper - double negation discards bindings
?- \+ \+ (T = (X, Y), write_term(T, [variable_names(['X'=X, 'Y'=Y])])).
   true.                           % PASSES: goal succeeded, bindings discarded

How it works: \+ \+ Goal succeeds if and only if Goal succeeds at least once, but because \+ (negation as failure) doesn’t export bindings, the double negation effectively tests “does this succeed?” without capturing any variable bindings.

When to use: Use \+ \+ when: - The query contains variables (like T, X, N) - The expected answer is just true - You don’t care about the specific variable bindings

This pattern is commonly needed for write_term/2, writeq/1, and other I/O predicates where the test verifies the predicate succeeds but the variables in the query are incidental.

Error Cases

Errors use the ISO error(Formal, ImplDefined) format:

% Instantiation error
?- length(L, N).
   error(instantiation_error, _).

% Type error
?- atom_length(123, N).
   error(type_error(atom, 123), _).

% Domain error
?- length(L, -1).
   error(domain_error(not_less_than_zero, -1), _).

% Existence error
?- current_input(foo).
   error(existence_error(stream, foo), _).

% Permission error
?- open('/nonexistent', read, S).
   error(permission_error(open, source_sink, '/nonexistent'), _).

Shorthand error descriptions are also supported:

?- atom_length(X, N).
   instantiation_error.          % Expands to error(instantiation_error, _)

?- atom_length(123, N).
   type_error(atom, 123).        % Expands to error(type_error(atom,123), _)

Alternative Answers (|)

When multiple answers are acceptable (implementation-defined behavior):

?- catch(throw(ball), E, true).
   E = ball
|  error(system_error, _).       % Some systems may wrap in system_error

Infinite Solutions (..., ad_infinitum)

For queries with infinite solutions:

?- repeat.
   true
;  true
;  true
;  ..., ad_infinitum.            % Continues indefinitely

Non-Termination (loops)

For queries that don’t terminate:

?- loop.                         % Assuming loop :- loop.
   loops.

Partial Matching (...)

To match any additional solutions:

?- between(1, 100, X).
   X = 1
;  X = 2
;  X = 3
;  ....                          % More solutions follow

STO (Subject To Occurs-check)

The sto annotation marks tests whose behavior depends on occurs-check settings:

?- X = f(X).
   sto, X = f(X)                 % With occurs-check: fails or error
|  sto, loops.                   % Without occurs-check: may loop

Unexpected Answers

Mark answers that should NOT occur:

?- safe_operation(X).
   X = result
;  unexpected, error(_, _).      % Test fails if this matches

Test Suites

Ulrich Neumerkel’s ISO Conformity Tests

Based on https://www.complang.tuwien.ac.at/ulrich/iso-prolog/

These tests verify strict ISO Prolog conformance:

Ulrich Neumerkel’s ISO Prologue Tests

Based on https://www.complang.tuwien.ac.at/ulrich/iso-prolog/prologue

These test library predicates proposed for ISO standardization:

ISO Built-in Predicate Tests

The tests/predicates/ directory contains comprehensive tests for all ISO built-in predicates, organized one file per predicate family.

Writing New Quad Tests

File Structure

% Test file header
% ISO Prolog [Predicate] Tests
% Based on [source URL if applicable]

% Optional: predicate definitions for testing
helper(X) :- member(X, [1,2,3]).

% Optional: directives
:- set_prolog_flag(double_quotes, chars).

% Test queries
?- helper(X).
   X = 1
;  X = 2
;  X = 3.

?- helper(4).
   false.

Adding Tests to the Suite

Create a new *_quad.pl file anywhere under tests/. The runner discovers quad tests by filename, so no additional registration step is required.

Best Practices

1. Use proper error format: Always use error(Type, _) wrapper, not bare error types
2. Test edge cases: Empty lists, zero, negative numbers, unbound variables
3. Test error conditions: All applicable ISO error types
4. Document source: Reference the specification or source page
5. Use sto for occurs-check sensitive tests
6. Use | for implementation-defined behavior

Framework Internals

The quad framework is implemented in lib/quad.pl and provides:

• Parsing: Reads queries and answer descriptions from .pl files
• Execution: Runs queries with inference limits to detect non-termination
• Matching: Compares actual results against expected answer descriptions
• Reporting: Displays mismatches with expected vs actual values

Key configuration:

See Also

• CONFORMANCE.md - ISO conformance goals
• TESTS.md - Full test suite documentation
• ISO_CHECKLIST.md - ISO predicate implementation status
• Ulrich Neumerkel’s ISO Prolog pages