  ^ ^                 ^ ^
=(o.o)=  ~byakuren  =(o.o)=
  m m                 m m

Futhark-J Bridge

A bridge between the J programming language and Futhark, enabling high-performance parallel array computations from J.

Overview

This library allows you to write Futhark code directly in your J source files and execute it with full parallel performance. Futhark code is compiled to native shared libraries and called via J’s FFI, providing seamless interoperability between J’s expressive array notation and Futhark’s parallel execution model.

Features

Inline Futhark Code: Write Futhark directly in J source files as strings or multiline 0 : 0 blocks
Immediate Compilation: Code compiles when fut is called, not on first use
Persistent Context: Futhark context created once, reused for all calls
Content-Based Caching: Identical code shares the same compiled library
Parallel Execution: Multicore backend uses all available CPU cores
Type Support: Full support for f32, f64, i32, i64 and more
Scalar and Array Inputs: Functions can take scalar or array inputs
Return Types: Both scalar reductions and array transformations
Multi-dimensional Arrays: Dynamic support for arrays of any dimension (1D, 2D, 3D, etc.)
Dynamic Output Sizes: Output arrays can differ in size and shape from input
Dyadic Verbs: Two-input functions work naturally as x verb y
Clear Error Messages: Full Futhark compiler output on compilation failures

Requirements

J 9.x with FFI support (require 'dll')
Futhark compiler (https://futhark-lang.org/)
GCC for compiling generated C code
POSIX system: Linux, FreeBSD, or macOS

Installing Futhark

# Using Homebrew (macOS/Linux)
brew install futhark

# Using Stack (Haskell)
stack install futhark

# FreeBSD
pkg install hs-futhark

Quick Start

NB. Load the library
load 'path/to/futhark.ijs'

NB. Define a Futhark function
double =: 'entry f (xs: []f64) : []f64 = map (*2) xs' fut

NB. Use it like any J verb
double 1 2 3 4 5
   2 4 6 8 10

NB. Scalar reductions work too
total =: 'entry f (xs: []f64) : f64 = reduce (+) 0 xs' fut
total 1 2 3 4 5
   15

NB. Dyadic verbs (two inputs) work naturally
dot =: 'entry f (xs: []f64) (ys: []f64) : f64 = reduce (+) 0 (map2 (*) xs ys)' fut
(1 2 3) dot (4 5 6)
   32

NB. Scalar inputs work too (v1.6.0)
factors =: 0 : 0
  entry f (n: i64) : []i64 =
    let (_, _, result) =
      loop (x, i, acc) = (n, 2i64, ([] : []i64))
      while x > 1 do
        if x % i == 0 then (x / i, i, acc ++ [i])
        else (x, i + 1, acc)
    in result
) fut
factors 12
   2 2 3

Usage Guide

Basic Syntax

The fut adverb takes a Futhark source string and returns a J verb:

NB. Single-line syntax for simple functions
my_verb =: 'futhark_source_code' fut
result =: my_verb input_array

NB. Multiline syntax using J's 0 : 0 for complex functions
NB. Note: 0 : 0 creates a string, then apply fut separately
my_src =: 0 : 0
  entry f (xs: []f64) : []f64 =
    let result = map (*2) xs
    in result
)
my_verb =: my_src fut

Futhark Entry Point Format

Futhark functions must be declared as entry points:

entry <name> (<param>: <type>) : <return_type> = <body>

Examples:

-- Array transformation (returns []f64)
entry f (xs: []f64) : []f64 = map (*2) xs

-- Scalar reduction (returns f64)
entry f (xs: []f64) : f64 = reduce (+) 0 xs

-- Using lambda expressions
entry f (xs: []f64) : []f64 = map (\x -> x * x + 1) xs

Supported Operations

Map (Element-wise transformations)

square =: 'entry f (xs: []f64) : []f64 = map (\x -> x * x) xs' fut
negate =: 'entry f (xs: []f64) : []f64 = map (\x -> 0 - x) xs' fut

Reduce (Fold to scalar)

sum     =: 'entry f (xs: []f64) : f64 = reduce (+) 0 xs' fut
product =: 'entry f (xs: []f64) : f64 = reduce (*) 1 xs' fut
minimum =: 'entry f (xs: []f64) : f64 = reduce f64.min f64.highest xs' fut
maximum =: 'entry f (xs: []f64) : f64 = reduce f64.max f64.lowest xs' fut

Scan (Prefix operations)

cumsum =: 'entry f (xs: []f64) : []f64 = scan (+) 0 xs' fut
runmax =: 'entry f (xs: []f64) : []f64 = scan f64.max f64.lowest xs' fut

Complex Operations

NB. Mean
mean =: 'entry f (xs: []f64) : f64 = reduce (+) 0 xs / f64.i64 (length xs)' fut

NB. Variance
variance =: 'entry f (xs: []f64) : f64 = let mean = reduce (+) 0 xs / f64.i64 (length xs) in let sq_diffs = map (\x -> (x - mean) ** 2) xs in reduce (+) 0 sq_diffs / f64.i64 (length xs)' fut

NB. Standard deviation
stddev =: 'entry f (xs: []f64) : f64 = let mean = reduce (+) 0 xs / f64.i64 (length xs) in let sq_diffs = map (\x -> (x - mean) ** 2) xs in f64.sqrt (reduce (+) 0 sq_diffs / f64.i64 (length xs))' fut

NB. Min-max normalization
normalize =: 'entry f (xs: []f64) : []f64 = let min_val = reduce f64.min f64.highest xs in let max_val = reduce f64.max f64.lowest xs in map (\x -> (x - min_val) / (max_val - min_val)) xs' fut

NB. Softmax
softmax =: 'entry f (xs: []f64) : []f64 = let max_val = reduce f64.max f64.lowest xs in let exps = map (\x -> f64.exp (x - max_val)) xs in let sum_exps = reduce (+) 0 exps in map (\e -> e / sum_exps) exps' fut

Dyadic Operations (Two Inputs)

Futhark functions with two input parameters become dyadic J verbs:

NB. Dot product: x dot y
dot =: 'entry f (xs: []f64) (ys: []f64) : f64 = reduce (+) 0 (map2 (*) xs ys)' fut
(1 2 3 4 5) dot (6 7 8 9 10)   NB. Returns 130

NB. Element-wise operations
mul =: 'entry f (xs: []f64) (ys: []f64) : []f64 = map2 (*) xs ys' fut
add =: 'entry f (xs: []f64) (ys: []f64) : []f64 = map2 (+) xs ys' fut
sub =: 'entry f (xs: []f64) (ys: []f64) : []f64 = map2 (-) xs ys' fut

NB. Element-wise max/min
max2 =: 'entry f (xs: []f64) (ys: []f64) : []f64 = map2 f64.max xs ys' fut
min2 =: 'entry f (xs: []f64) (ys: []f64) : []f64 = map2 f64.min xs ys' fut

NB. Weighted sum
weighted_sum =: 'entry f (ws: []f64) (xs: []f64) : f64 = reduce (+) 0 (map2 (*) ws xs)' fut

2D Array Operations

NB. Transpose a matrix
transp =: 'entry f (xs: [][]f64) : [][]f64 = transpose xs' fut
mat =: 3 4 $ i. 12
transp mat    NB. Returns 4 3 matrix (matches J |: mat)

NB. Flatten 2D to 1D
flat =: 'entry f (xs: [][]f64) : []f64 = flatten xs' fut
flat mat      NB. Returns 1D array of 12 elements (matches J , mat)

3D Array Operations

NB. 3D arrays work seamlessly
arr3d =: 2 3 4 $ i. 24

NB. 3D identity
id3d =: 'entry f (xs: [][][]f64) : [][][]f64 = xs' fut
id3d arr3d   NB. Returns same 2x3x4 array

NB. 3D element-wise operations
map3d =: 'entry f (xs: [][][]f64) : [][][]f64 = map (map (map (*2))) xs' fut
map3d arr3d  NB. Returns 2x3x4 array with doubled values (matches J 2*arr3d)

NB. Flatten 3D to 1D
flat3d =: 'entry f (xs: [][][]f64) : []f64 = flatten (flatten xs)' fut
flat3d arr3d NB. Returns 24-element 1D array (matches J , arr3d)

NB. Sum over 3D array
sum3d =: 'entry f (xs: [][][]f64) : f64 = flatten (flatten xs) |> reduce (+) 0' fut
sum3d arr3d  NB. Returns scalar sum (matches J +/, arr3d)

Scalar Input Functions (v1.6.0)

Futhark functions can take scalar inputs instead of arrays:

NB. Prime factorization: scalar in, array out
factors =: 0 : 0
  entry f (n: i64) : []i64 =
    let (_, _, result) =
      loop (x, i, acc) = (n, 2i64, ([] : []i64))
      while x > 1 do
        if x % i == 0 then (x / i, i, acc ++ [i])
        else (x, i + 1, acc)
    in result
) fut
factors 12      NB. Returns 2 2 3
factors 100     NB. Returns 2 2 5 5

NB. Fibonacci: scalar in, scalar out
fib =: 'entry f (n: i64) : i64 = loop (a, b, i) = (0i64, 1i64, 0i64) while i < n do (b, a + b, i + 1) |> (.0)' fut
fib 10   NB. Returns 55

NB. Check if prime: scalar in, scalar out (bool as i64)
is_prime =: 'entry f (n: i64) : i64 = if n < 2 then 0 else if n == 2 then 1 else if n % 2 == 0 then 0 else loop (i, result) = (3i64, 1i64) while i * i <= n && result == 1 do (i + 2, if n % i == 0 then 0 else 1) |> (.1)' fut

Fixed-Size Array Dimensions (v1.7.0)

Futhark supports fixed-size inner dimensions using [n] syntax. These are fully supported:

NB. 1D input to 2D output with fixed inner dimension
triple =: 'entry f (ns: []i64) : [][3]i64 = map (\n -> [n, n*2, n*3]) ns' fut
triple 1 2 3 4 5
   1  2  3
   2  4  6
   3  6  9
   4  8 12
   5 10 15

NB. Batch factorization: factor many numbers in parallel
NB. Uses helper function 'def' - parser correctly finds entry point
batch_factor =: 0 : 0
  def factorize (n: i64) : [32]i64 =
    let (_, _, factors, _) =
      loop (x, i, acc, c) = (n, 2i64, replicate 32 0i64, 0i64)
      while x > 1 && c < 32 do
        if x % i == 0
          then (x / i, i, acc with [c] = i, c + 1)
          else (x, i + 1, acc, c)
    in factors

  entry f (ns: []i64) : [][32]i64 =
    map factorize ns
) fut

NB. Factor multiple numbers at once - runs in parallel!
batch_factor 12 100 30
   2 2 3 0 0 ...
   2 2 5 5 0 ...
   2 3 5 0 0 ...

Note: v1.7.0 fixes parsing so helper functions (def) don’t interfere with entry point detection.

Operations with Different Output Sizes

NB. Take first n elements
take5 =: 'entry f (xs: []f64) : []f64 = take 5 xs' fut
take5 i. 10   NB. Returns 0 1 2 3 4

NB. Drop first n elements
drop3 =: 'entry f (xs: []f64) : []f64 = drop 3 xs' fut
drop3 i. 10   NB. Returns 3 4 5 6 7 8 9

NB. Replicate each element
rep2 =: 'entry f (xs: []f64) : []f64 = flatten (map (\x -> replicate 2 x) xs)' fut
rep2 1 2 3    NB. Returns 1 1 2 2 3 3

NB. Concatenate array with itself
concat_self =: 'entry f (xs: []f64) : []f64 = concat xs xs' fut
concat_self 1 2 3   NB. Returns 1 2 3 1 2 3

Data Types

Futhark	J FFI	Description
`f64`	`d`	64-bit float (double)
`f32`	`f`	32-bit float
`i64`	`x`	64-bit signed integer
`i32`	`i`	32-bit signed integer
`i16`	`s`	16-bit signed integer
`i8`	`c`	8-bit signed integer
`u64`	`x`	64-bit unsigned integer
`u32`	`i`	32-bit unsigned integer

Configuration

Global Variables

NB. Cache directory (default: /tmp/futhark_j/)
FUTHARK_CACHE =: '/path/to/cache/'

NB. Backend: 'c' (sequential), 'multicore' (parallel), 'ispc'
FUTHARK_BACKEND =: 'multicore'

NB. Thread count for parallel backends
FUTHARK_THREADS =: 12

Utility Functions

NB. Clear compiled library cache
fut_clear_cache ''

NB. Change thread count
fut_set_threads 8

NB. Display version info
fut_version ''

Performance

When Futhark Excels

Large arrays (>100,000 elements)
Computationally intensive operations
Operations that parallelize well (map, reduce, scan)

When J May Be Faster

Small arrays (FFI overhead dominates)
Simple operations (J is highly optimized)

Benchmarks

Typical results for standard deviation on different array sizes:

Size	Futhark	J	Speedup
100K	1.7ms	0.4ms	0.2x
500K	3.2ms	2.8ms	0.9x
1M	11ms	14ms	1.3x
2M	40ms	31ms	0.8x

Results vary by operation complexity and system configuration.

J Verb Equivalence

Futhark operations are semantically equivalent to J verbs:

Futhark	J	Description
`reduce (+) 0 xs`	`+/ xs`	Sum
`reduce (*) 1 xs`	`*/ xs`	Product
`map (*2) xs`	`2 * xs`	Double
`map (\x -> x*x) xs`	`*: xs`	Square
`scan (+) 0 xs`	`+/\ xs`	Running sum
`transpose xs`	`\\|: xs`	Transpose
`flatten xs`	`, xs`	Ravel
`map2 (*) xs ys`	`xs * ys`	Element-wise multiply
`map2 (+) xs ys`	`xs + ys`	Element-wise add
`reduce (+) 0 (map2 (*) xs ys)`	`xs +/@:* ys`	Dot product

The test suite verifies all operations match their J equivalents.

Limitations

Two inputs max: Functions take one or two array inputs
No GPU: Only CPU backends (c, multicore, ispc)

Version History

Version	Changes
1.7.0	Fixed-size array dimensions (`[][32]i64`), correct parsing with `def` helper functions, 123 tests
1.6.0	Scalar input support, improved error reporting with full compiler output, 96 tests
1.5.0	Use J’s built-in SHA-256 (128!:6) for cache key hashing instead of polynomial hash
1.4.0	Immediate compilation at definition time, persistent Futhark context (no warmup needed)
1.3.0	Dyadic verb support (two inputs), 82 tests
1.2.0	Dynamic n-dimensional array support, unified executor, 73 tests
1.1.0	2D array support, output shape queries, 44 tests
1.0.0	Initial release, 1D arrays, scalar/array returns, 30 tests

Files

futhark.ijs            - Main library (load this)
futhark_test.ijs       - Test suite (123 tests)
dot_product_bench.ijs  - Dot product benchmark
stddev_bench.ijs       - Standard deviation benchmark
compute_bench.ijs      - Compute-intensive benchmarks (logistic map, Mandelbrot)
futhark_demo.ijs       - Interactive demonstration
README.md              - This documentation
DOCUMENTATION.md       - Technical deep-dive

License

MIT License