Skip to content

Instantly share code, notes, and snippets.

@alcides
Last active November 4, 2026 13:04
Show Gist options
  • Select an option

  • Save alcides/7b243e2cb954ce08c3f4f0ca2d8a05ea to your computer and use it in GitHub Desktop.

Select an option

Save alcides/7b243e2cb954ce08c3f4f0ca2d8a05ea to your computer and use it in GitHub Desktop.
Aeon 32-item knapsack with inductive stdlib List: 41 synthesizer IDs, hard 10-second timeout; single-step tactics excluded

Aeon: 32-item 0–1 knapsack with inductive stdlib lists

Run on 2026-11-04T13:04:08Z, using Aeon 41a88a70 and Python 3.13.15.

Attempted 36 canonical backend/model IDs plus 5 aliases (41 runs), sequentially, with a hard 10-second wall-clock limit per run. Excluded all 12 single-step tactics as requested. The multi-step tactics search is included.

No independently verified complete solution was reported within the 10-second deadline. This is a measured timeout/interface result, not a claim that these algorithms cannot solve knapsack.

There is no synthesized best_solution.ae for this run. reference_solution.ae is clearly labeled as an independently computed reference, not a synthesis result.

Timing limitation: compilation finished between 9.448 and 9.679 seconds after worker launch. Compilation is inside the deadline, so substantially less than 10 seconds remains for actual search. See compiled_s and search_started_s in results.json.

Instance and encoding

Capacity is 100. The 32 decisions are a stdlib inductive List of binary integers, in item order. chosen has the return type:

{xs:(List {b:Int | 0 <= b && b <= 1}) |
    List.size xs = 32 && weight xs <= 100}

@maximize_int(value (chosen unit)) supplies the objective. list32 is a regular function with exactly 32 binary arguments; its native body constructs actual stdlib List.cons / List.nil values. Its trusted return contract specifies both the exact length and the weighted sum. Weight and value are computed by an Aeon recursive dot product over inductive lists. The constants are also constructed with the stdlib constructors through native code, avoiding expensive checking of deeply nested list literals. There is no integer-mask representation of the decisions.

As with other native bindings, Aeon trusts the constructor's declared contract; it does not prove its Python body. Before timing, the runtime representation, order, weight and value were checked for all 32 single-item selections. A weight-100 selection typechecked; weight-101 and non-binary selections were rejected.

Item index Weight Value
0 12 24
1 7 13
2 11 23
3 8 15
4 9 16
5 15 28
6 6 12
7 13 25
8 4 9
9 10 19
10 5 11
11 14 26
12 3 7
13 16 29
14 2 5
15 17 31
16 8 18
17 6 14
18 9 20
19 12 22
20 7 17
21 11 21
22 5 10
23 13 27
24 4 8
25 10 20
26 15 30
27 3 6
28 14 28
29 6 13
30 9 19
31 8 16

Independent optimum: value 217, weight 100. A 0–1 dynamic program over exact weights computes the reference. A separate OR-Tools CP-SAT model confirmed optimality. This oracle is used only for reporting; no reference solution, examples or target value are supplied to synthesis.

One optimal selection is [1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0]. Its list32 expression was recompiled and executed by the same independent verifier used for synthesized candidates.

Results

Status counts: error: 24; timeout: 17.

A dash means no verified candidate. Alias runs are listed explicitly and are not additional algorithms. Timeout rows with a candidate preserve an incumbent reported before the deadline.

Backend Alias of Outcome Best value Weight First best (s) Compiled (s)
synquid — timeout — — — 9.526
tactics — timeout — — — 9.532
tdsyn_enumerative — timeout — — — 9.485
tdsyn_random — timeout — — — 9.525
bustle_baseline — error — — — 9.504
bustle — error — — — 9.523
decision_tree — timeout — — — 9.448
noisy — error — — — 9.511
structure_guided — error — — — 9.487
contata — error — — — 9.479
boolean_bounded — error — — — 9.510
ortools — error — — — 9.472
polynomial — error — — — 9.492
smt — timeout — — — 9.533
sygus — error — — — 9.540
afta — timeout — — — 9.507
cata — timeout — — — 9.589
fta — timeout — — — 9.491
arborist — error — — — 9.487
lta — timeout — — — 9.526
symetric — timeout — — — 9.471
amaze — error — — — 9.453
enumerative — timeout — — — 9.531
random_search — timeout — — — 9.551
regus — error — — — 9.577
rl_mcts — error — — — 9.558
dp_gp — error — — — 9.590
gp — timeout — — — 9.473
llm_codellama-13b — error — — — 9.514
llm_deepseek-coder-v2-16b — error — — — 9.593
llm_deepseek-coder-6.7b — error — — — 9.561
llm_openai — timeout — — — 9.679
llm_qwen2.5-coder-14b — error — — — 9.573
llm_qwen2.5-coder-32b — error — — — 9.598
llm_starcoder2-15b — error — — — 9.571
llm_type_constrained — error — — — 9.574
tdsyn tdsyn_enumerative timeout — — — 9.505
ortools_int ortools error — — — 9.555
cpsat ortools error — — — 9.592
xfta symetric timeout — — — 9.546
llm llm_qwen2.5-coder-32b error — — — 9.504

Reported errors

These are the actual errors from this source encoding and environment.

Backend Error
bustle_baseline Unsupported synthesis goal: Cannot sample opaque type Unit
bustle Unsupported synthesis goal: Cannot sample opaque type Unit
noisy Unsupported synthesis goal: noisy requires every example to be a scalar literal I/O equality
structure_guided Unsupported synthesis goal: structure_guided: needs unary constructor-literal @example I/O pairs
contata Unsupported synthesis goal: Cannot sample opaque type Unit
boolean_bounded Unsupported synthesis goal: Unsupported synthesis goal: every Boolean synthesis target needs @bounded_cnf(K)
ortools Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead.
polynomial Joint solver cannot translate type { xs⁵⁹⁶:List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) } | ((List_size⁷⁰²(xs⁵⁹⁶) == 32) && (weight⁵²²(xs⁵⁹⁶) <= 100)) }; expected Int, Bool or Float
sygus SygusSynthesizer: hole of type { xs⁵⁹⁶:List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) } | ((List_size⁷⁰²(xs⁵⁹⁶) == 32) && (weight⁵²²(xs⁵⁹⁶) <= 100)) } is outside the SyGuS-supported subset (needs an Int/Bool/Float type, optionally refined, over native operators).
arborist Unsupported synthesis goal: arborist requires literal scalar/constructor @example inputs and Int/Bool outputs
amaze Unsupported synthesis goal: supported sorts are monomorphic Int/Bool; abstract refinements and higher-order types are unsupported
regus Unsupported synthesis goal: regus expects a ControlProgram hole; use a structure-synthesis problem
rl_mcts rl_mcts solves refinement specifications; optimization goals are not supported
dp_gp Unsupported synthesis goal: dp_gp requires @dp_gp(reference, max_first_input, max_second_input)
llm_codellama-13b Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
llm_deepseek-coder-v2-16b Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
llm_deepseek-coder-6.7b Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
llm_qwen2.5-coder-14b Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
llm_qwen2.5-coder-32b Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
llm_starcoder2-15b Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
llm_type_constrained Unsupported synthesis goal: llm_type_constrained needs monomorphic Int/Bool/String/Float/Unit or unrestricted arrows; got List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }
ortools_int Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead.
cpsat Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead.
llm Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download

Excluded one-level tactics

backward_abs, backward_app, backward_close, tdsyn_backward, backward_if, backward_lit, forward_close, forward_let_abs, forward_let_app, forward_let_if, forward_let_tabs, forward_let_tapp.

Protocol and limitations

Each worker receives Aeon's budget of 10 seconds. A separate supervisor enforces a hard 10-second wall-clock deadline from process launch, including imports, compilation, backend setup and search. The entire worker process group is killed on expiry. wall_s includes a small scheduling/termination overhead, but only events timestamped at or before 10.000 seconds are eligible.

The runner uses the normal AeonDriver with a recording SynthesisUI; it does not change backend algorithms, the source specification or fitness. It retains reported improving complete candidates and returned expressions. Verification occurs outside the search budget, in fresh processes, with a 30-second limit per distinct expression: recompile/typecheck, evaluate chosen unit, require exactly 32 binary entries ending in stdlib List.nil, check capacity independently in Python, and compare Aeon's weight/value with the independent calculation.

Runs are sequential, with AEON_SEED=0, PYTHONHASHSEED=0, and common BLAS thread counts set to 1. Automatic LLM downloads are disabled. OpenAI API keys and AEON_TC_MODEL are not configured in this environment. Optional backend failures are retained as results, and their exact diagnostics are included. This is one seed on one fixed instance; it is not a general algorithm ranking.

Reproduce

Download the gist files into knapsack-gist, then run from its parent directory:

git clone https://github.com/alcides/aeon.git aeon-knapsack
cd aeon-knapsack
git checkout 41a88a70b29e626bcbea98b61de8ed5fd758426c
uv venv --python 3.13
uv pip install -r ../knapsack-gist/requirements.txt -e ".[dev,synthesis-ortools]"
.venv/bin/python ../knapsack-gist/run_benchmark.py --timeout 10 --verify-timeout 30

The runner discovers all registered IDs and aliases and automatically excludes the single-step tactics. To select specific backends, add --backends gp tdsyn_random.

Equivalent direct CLI command for one backend:

AEON_SEED=0 PYTHONHASHSEED=0 OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 \
  timeout --signal=KILL 10s .venv/bin/python -u -m aeon \
  --budget 10 -s gp --synthesis-format json ../knapsack-gist/knapsack.ae

The recording runner is needed to preserve incumbents when the watchdog fires before the CLI prints its final result. main returns the objective value; the Aeon CLI does not print that return value. The verifier prints the full selection and metrics:

.venv/bin/python ../knapsack-gist/run_benchmark.py --verify \
  --expression 'list32 1 0 1 0 0 0 0 0 1 0 1 0 1 0 1 0 1 1 1 0 1 0 1 1 0 0 0 0 0 1 1 0'

That command checks the independent reference, not a synthesized result.

Files and previous experiment

  • knapsack.ae: the exact 32-item inductive-list source used by every run.
  • run_benchmark.py: discovery, timeout supervisor, recorder, dynamic-programming oracle and independent verifier.
  • results.csv, results.json: all attempts, candidates, validation results and errors.
  • environment.json, requirements.txt: revision, source hash, configuration and dependencies.
  • events.jsonl, backend_logs.txt: timestamped events and captured diagnostics for each backend.
  • reference_solution.ae: an independently computed optimum, explicitly not a synthesis result.

The earlier five-item integer-mask experiment is preserved in the previous gist revision. Its scores describe a different instance and should not be compared directly with these 32-item results.

===== synquid =====
===== tactics =====
===== tdsyn_enumerative =====
===== tdsyn_random =====
===== bustle_baseline =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 365, in _synthesize_one
generated = derive_refinement_examples(
ctx, prog, fun_name, hole_name=hole_name, budget=min(2.0, max(0.0, budget / 5))
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/refinement_examples.py", line 246, in derive_refinement_examples
solver.add(_domain(value, ty, constructors, env))
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/refinement_examples.py", line 78, in _domain
raise UnsupportedSynthesisGoal(f"Cannot sample opaque type {base}")
aeon.synthesis.api.UnsupportedSynthesisGoal: Unsupported synthesis goal: Cannot sample opaque type Unit
===== bustle =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 365, in _synthesize_one
generated = derive_refinement_examples(
ctx, prog, fun_name, hole_name=hole_name, budget=min(2.0, max(0.0, budget / 5))
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/refinement_examples.py", line 246, in derive_refinement_examples
solver.add(_domain(value, ty, constructors, env))
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/refinement_examples.py", line 78, in _domain
raise UnsupportedSynthesisGoal(f"Cannot sample opaque type {base}")
aeon.synthesis.api.UnsupportedSynthesisGoal: Unsupported synthesis goal: Cannot sample opaque type Unit
===== decision_tree =====
===== noisy =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 350, in _synthesize_one
return synthesizer.synthesize_source(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
ctx, tyctx, ectx, prog, fun_name, hole_name, ty, metadata, validator, budget, ui
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/noisy/__init__.py", line 46, in synthesize_source
raise UnsupportedSynthesisGoal("noisy requires every example to be a scalar literal I/O equality")
aeon.synthesis.api.UnsupportedSynthesisGoal: Unsupported synthesis goal: noisy requires every example to be a scalar literal I/O equality
===== structure_guided =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 506, in _synthesize_one
raise e
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 491, in _synthesize_one
t = syn_impl.synthesize(
ctx=tyctx,
...<7 lines>...
output_value=output_evaluator,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/structure_guided/synthesizer.py", line 241, in synthesize
raise UnsupportedSynthesisGoal("structure_guided: needs unary constructor-literal @example I/O pairs")
aeon.synthesis.api.UnsupportedSynthesisGoal: Unsupported synthesis goal: structure_guided: needs unary constructor-literal @example I/O pairs
===== contata =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 365, in _synthesize_one
generated = derive_refinement_examples(
ctx, prog, fun_name, hole_name=hole_name, budget=min(2.0, max(0.0, budget / 5))
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/refinement_examples.py", line 246, in derive_refinement_examples
solver.add(_domain(value, ty, constructors, env))
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/refinement_examples.py", line 78, in _domain
raise UnsupportedSynthesisGoal(f"Cannot sample opaque type {base}")
aeon.synthesis.api.UnsupportedSynthesisGoal: Unsupported synthesis goal: Cannot sample opaque type Unit
===== boolean_bounded =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 794, in synthesize_holes
return _synthesize_joint(
ctx,
...<8 lines>...
constructor_names,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 537, in _synthesize_joint
mapping = synthesizer.run_bounded(
lambda remaining, worker_ui: _synthesize_joint(
...<12 lines>...
ui,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/boolean_bounded.py", line 185, in run_bounded
raise UnsupportedSynthesisGoal(message)
aeon.synthesis.api.UnsupportedSynthesisGoal: Unsupported synthesis goal: Unsupported synthesis goal: every Boolean synthesis target needs @bounded_cnf(K)
===== ortools =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/ortools_cpsat.py", line 268, in synthesize_program
self._declare_hole(hole, holes_info[hole][0])
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/ortools_cpsat.py", line 370, in _declare_hole
raise _Unsupported(f"type {_base(ty)}")
aeon.synthesis.modules.ortools_cpsat._Unsupported: type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 780, in synthesize_holes
return synthesizer.synthesize_program(ctx, ectx, term, targets, metadata, budget, ui)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/ortools_cpsat.py", line 270, in synthesize_program
raise UnsupportedSynthesisGoal(
...<2 lines>...
) from e
aeon.synthesis.api.UnsupportedSynthesisGoal: Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead.
===== polynomial =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 794, in synthesize_holes
return _synthesize_joint(
ctx,
...<8 lines>...
constructor_names,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 621, in _synthesize_joint
synthesis_metadata[joint_name]["solver_problem"] = build_solver_problem(
~~~~~~~~~~~~~~~~~~~~^
ctx, prog, holes, metadata, lexical_only=getattr(synthesizer, "lexical_inputs_only", False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/solver_problem.py", line 146, in build_solver_problem
base = base_type(ty)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/solver_problem.py", line 36, in base_type
raise SynthesisNotSuccessful(f"Joint solver cannot translate type {ty}; expected Int, Bool or Float")
aeon.synthesis.api.SynthesisNotSuccessful: Joint solver cannot translate type { xs⁵⁹⁶:List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) } | ((List_size⁷⁰²(xs⁵⁹⁶) == 32) && (weight⁵²²(xs⁵⁹⁶) <= 100)) }; expected Int, Bool or Float
===== smt =====
===== sygus =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 506, in _synthesize_one
raise e
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 491, in _synthesize_one
t = syn_impl.synthesize(
ctx=tyctx,
...<7 lines>...
output_value=output_evaluator,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/sygus/synthesizer.py", line 64, in synthesize
raise SynthesisNotSuccessful(
...<2 lines>...
)
aeon.synthesis.api.SynthesisNotSuccessful: SygusSynthesizer: hole of type { xs⁵⁹⁶:List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) } | ((List_size⁷⁰²(xs⁵⁹⁶) == 32) && (weight⁵²²(xs⁵⁹⁶) <= 100)) } is outside the SyGuS-supported subset (needs an Int/Bool/Float type, optionally refined, over native operators).
===== afta =====
===== cata =====
===== fta =====
===== arborist =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 780, in synthesize_holes
return synthesizer.synthesize_program(ctx, ectx, term, targets, metadata, budget, ui)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/arborist/synthesizer.py", line 80, in synthesize_program
raise UnsupportedSynthesisGoal(
"arborist requires literal scalar/constructor @example inputs and Int/Bool outputs"
)
aeon.synthesis.api.UnsupportedSynthesisGoal: Unsupported synthesis goal: arborist requires literal scalar/constructor @example inputs and Int/Bool outputs
===== lta =====
===== symetric =====
===== amaze =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/amaze/synthesizer.py", line 66, in synthesize
goal = sort_of(type, refinements=True)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/amaze/core.py", line 23, in sort_of
raise ValueError(
"supported sorts are monomorphic Int/Bool; abstract refinements and higher-order types are unsupported"
)
ValueError: supported sorts are monomorphic Int/Bool; abstract refinements and higher-order types are unsupported
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 506, in _synthesize_one
raise e
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 491, in _synthesize_one
t = syn_impl.synthesize(
ctx=tyctx,
...<7 lines>...
output_value=output_evaluator,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/amaze/synthesizer.py", line 68, in synthesize
raise UnsupportedSynthesisGoal(str(exc)) from exc
aeon.synthesis.api.UnsupportedSynthesisGoal: Unsupported synthesis goal: supported sorts are monomorphic Int/Bool; abstract refinements and higher-order types are unsupported
===== enumerative =====
===== random_search =====
===== regus =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 506, in _synthesize_one
raise e
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 491, in _synthesize_one
t = syn_impl.synthesize(
ctx=tyctx,
...<7 lines>...
output_value=output_evaluator,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/regus/synthesizer.py", line 63, in synthesize
raise UnsupportedSynthesisGoal("regus expects a ControlProgram hole; use a structure-synthesis problem")
aeon.synthesis.api.UnsupportedSynthesisGoal: Unsupported synthesis goal: regus expects a ControlProgram hole; use a structure-synthesis problem
===== rl_mcts =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 506, in _synthesize_one
raise e
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 491, in _synthesize_one
t = syn_impl.synthesize(
ctx=tyctx,
...<7 lines>...
output_value=output_evaluator,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/rl_mcts/synthesizer.py", line 74, in synthesize
raise SynthesisNotSuccessful(
"rl_mcts solves refinement specifications; optimization goals are not supported"
)
aeon.synthesis.api.SynthesisNotSuccessful: rl_mcts solves refinement specifications; optimization goals are not supported
===== dp_gp =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 780, in synthesize_holes
return synthesizer.synthesize_program(ctx, ectx, term, targets, metadata, budget, ui)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/dp_gp/synthesizer.py", line 43, in synthesize_program
raise UnsupportedSynthesisGoal("dp_gp requires @dp_gp(reference, max_first_input, max_second_input)")
aeon.synthesis.api.UnsupportedSynthesisGoal: Unsupported synthesis goal: dp_gp requires @dp_gp(reference, max_first_input, max_second_input)
===== gp =====
===== llm_codellama-13b =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 506, in _synthesize_one
raise e
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 491, in _synthesize_one
t = syn_impl.synthesize(
ctx=tyctx,
...<7 lines>...
output_value=output_evaluator,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/__init__.py", line 125, in synthesize
prepare_ollama_model(self.model)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 200, in prepare_ollama_model
pull_model(model)
~~~~~~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 139, in pull_model
if not is_model_installed(model):
~~~~~~~~~~~~~~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 84, in is_model_installed
return model in _installed_model_names()
~~~~~~~~~~~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 76, in _installed_model_names
for entry in _ollama().list().models:
~~~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 653, in list
return self._request(
~~~~~~~~~~~~~^
ListResponse,
^^^^^^^^^^^^^
'GET',
^^^^^^
'/api/tags',
^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 203, in _request
return cls(**self._request_raw(*args, **kwargs).json())
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 149, in _request_raw
raise ConnectionError(CONNECTION_ERROR_MESSAGE) from None
ConnectionError: Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
===== llm_deepseek-coder-v2-16b =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 506, in _synthesize_one
raise e
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 491, in _synthesize_one
t = syn_impl.synthesize(
ctx=tyctx,
...<7 lines>...
output_value=output_evaluator,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/__init__.py", line 125, in synthesize
prepare_ollama_model(self.model)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 200, in prepare_ollama_model
pull_model(model)
~~~~~~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 139, in pull_model
if not is_model_installed(model):
~~~~~~~~~~~~~~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 84, in is_model_installed
return model in _installed_model_names()
~~~~~~~~~~~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 76, in _installed_model_names
for entry in _ollama().list().models:
~~~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 653, in list
return self._request(
~~~~~~~~~~~~~^
ListResponse,
^^^^^^^^^^^^^
'GET',
^^^^^^
'/api/tags',
^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 203, in _request
return cls(**self._request_raw(*args, **kwargs).json())
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 149, in _request_raw
raise ConnectionError(CONNECTION_ERROR_MESSAGE) from None
ConnectionError: Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
===== llm_deepseek-coder-6.7b =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 506, in _synthesize_one
raise e
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 491, in _synthesize_one
t = syn_impl.synthesize(
ctx=tyctx,
...<7 lines>...
output_value=output_evaluator,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/__init__.py", line 125, in synthesize
prepare_ollama_model(self.model)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 200, in prepare_ollama_model
pull_model(model)
~~~~~~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 139, in pull_model
if not is_model_installed(model):
~~~~~~~~~~~~~~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 84, in is_model_installed
return model in _installed_model_names()
~~~~~~~~~~~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 76, in _installed_model_names
for entry in _ollama().list().models:
~~~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 653, in list
return self._request(
~~~~~~~~~~~~~^
ListResponse,
^^^^^^^^^^^^^
'GET',
^^^^^^
'/api/tags',
^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 203, in _request
return cls(**self._request_raw(*args, **kwargs).json())
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 149, in _request_raw
raise ConnectionError(CONNECTION_ERROR_MESSAGE) from None
ConnectionError: Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
===== llm_openai =====
===== llm_qwen2.5-coder-14b =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 506, in _synthesize_one
raise e
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 491, in _synthesize_one
t = syn_impl.synthesize(
ctx=tyctx,
...<7 lines>...
output_value=output_evaluator,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/__init__.py", line 125, in synthesize
prepare_ollama_model(self.model)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 200, in prepare_ollama_model
pull_model(model)
~~~~~~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 139, in pull_model
if not is_model_installed(model):
~~~~~~~~~~~~~~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 84, in is_model_installed
return model in _installed_model_names()
~~~~~~~~~~~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 76, in _installed_model_names
for entry in _ollama().list().models:
~~~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 653, in list
return self._request(
~~~~~~~~~~~~~^
ListResponse,
^^^^^^^^^^^^^
'GET',
^^^^^^
'/api/tags',
^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 203, in _request
return cls(**self._request_raw(*args, **kwargs).json())
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 149, in _request_raw
raise ConnectionError(CONNECTION_ERROR_MESSAGE) from None
ConnectionError: Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
===== llm_qwen2.5-coder-32b =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 506, in _synthesize_one
raise e
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 491, in _synthesize_one
t = syn_impl.synthesize(
ctx=tyctx,
...<7 lines>...
output_value=output_evaluator,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/__init__.py", line 125, in synthesize
prepare_ollama_model(self.model)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 200, in prepare_ollama_model
pull_model(model)
~~~~~~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 139, in pull_model
if not is_model_installed(model):
~~~~~~~~~~~~~~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 84, in is_model_installed
return model in _installed_model_names()
~~~~~~~~~~~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 76, in _installed_model_names
for entry in _ollama().list().models:
~~~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 653, in list
return self._request(
~~~~~~~~~~~~~^
ListResponse,
^^^^^^^^^^^^^
'GET',
^^^^^^
'/api/tags',
^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 203, in _request
return cls(**self._request_raw(*args, **kwargs).json())
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 149, in _request_raw
raise ConnectionError(CONNECTION_ERROR_MESSAGE) from None
ConnectionError: Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
===== llm_starcoder2-15b =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 506, in _synthesize_one
raise e
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 491, in _synthesize_one
t = syn_impl.synthesize(
ctx=tyctx,
...<7 lines>...
output_value=output_evaluator,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/__init__.py", line 125, in synthesize
prepare_ollama_model(self.model)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 200, in prepare_ollama_model
pull_model(model)
~~~~~~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 139, in pull_model
if not is_model_installed(model):
~~~~~~~~~~~~~~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 84, in is_model_installed
return model in _installed_model_names()
~~~~~~~~~~~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 76, in _installed_model_names
for entry in _ollama().list().models:
~~~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 653, in list
return self._request(
~~~~~~~~~~~~~^
ListResponse,
^^^^^^^^^^^^^
'GET',
^^^^^^
'/api/tags',
^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 203, in _request
return cls(**self._request_raw(*args, **kwargs).json())
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 149, in _request_raw
raise ConnectionError(CONNECTION_ERROR_MESSAGE) from None
ConnectionError: Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
===== llm_type_constrained =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 764, in synthesize_holes
check_program_scope(term, metadata)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/type_constrained/__init__.py", line 62, in check_program_scope
visit(term)
~~~~~^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/type_constrained/__init__.py", line 60, in visit
visit(child)
~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/type_constrained/__init__.py", line 60, in visit
visit(child)
~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/type_constrained/__init__.py", line 60, in visit
visit(child)
~~~~~^^^^^^^
[Previous line repeated 16 more times]
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/type_constrained/__init__.py", line 52, in visit
ordinary(node.var_type)
~~~~~~~~^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/type_constrained/types.py", line 58, in ordinary
return Mono("->", ordinary(ty.var_type, primitive=primitive), ordinary(ty.type, primitive=primitive))
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/type_constrained/types.py", line 52, in ordinary
return ordinary(ty.type, primitive=primitive)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/type_constrained/types.py", line 59, in ordinary
raise UnsupportedSynthesisGoal(
f"llm_type_constrained needs monomorphic Int/Bool/String/Float/Unit or unrestricted arrows; got {ty}"
)
aeon.synthesis.api.UnsupportedSynthesisGoal: Unsupported synthesis goal: llm_type_constrained needs monomorphic Int/Bool/String/Float/Unit or unrestricted arrows; got List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }
===== tdsyn =====
===== ortools_int =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/ortools_cpsat.py", line 268, in synthesize_program
self._declare_hole(hole, holes_info[hole][0])
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/ortools_cpsat.py", line 370, in _declare_hole
raise _Unsupported(f"type {_base(ty)}")
aeon.synthesis.modules.ortools_cpsat._Unsupported: type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 780, in synthesize_holes
return synthesizer.synthesize_program(ctx, ectx, term, targets, metadata, budget, ui)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/ortools_cpsat.py", line 270, in synthesize_program
raise UnsupportedSynthesisGoal(
...<2 lines>...
) from e
aeon.synthesis.api.UnsupportedSynthesisGoal: Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead.
===== cpsat =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/ortools_cpsat.py", line 268, in synthesize_program
self._declare_hole(hole, holes_info[hole][0])
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/ortools_cpsat.py", line 370, in _declare_hole
raise _Unsupported(f"type {_base(ty)}")
aeon.synthesis.modules.ortools_cpsat._Unsupported: type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 780, in synthesize_holes
return synthesizer.synthesize_program(ctx, ectx, term, targets, metadata, budget, ui)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/ortools_cpsat.py", line 270, in synthesize_program
raise UnsupportedSynthesisGoal(
...<2 lines>...
) from e
aeon.synthesis.api.UnsupportedSynthesisGoal: Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead.
===== xfta =====
===== llm =====
Traceback (most recent call last):
File "/tmp/aeon-knapsack-list-results/run_benchmark.py", line 129, in worker
driver.synth()
~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 284, in synth
sterm_mapping = self._run_synthesis(self.cfg.synthesis_ui)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 70, in wrapped
return method(self, *args, **kwargs)
File "/tmp/aeon-knapsack-benchmark/aeon/facade/driver.py", line 262, in _run_synthesis
mapping: dict[Name, Term] = synthesize_holes(
~~~~~~~~~~~~~~~~^
self.typing_ctx,
^^^^^^^^^^^^^^^^
...<7 lines>...
constructor_names=self.constructor_names,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 835, in synthesize_holes
mapping[hole_name] = _synthesize_one(
~~~~~~~~~~~~~~~^
ctx,
^^^^
...<12 lines>...
open_targets=open_targets,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 506, in _synthesize_one
raise e
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/entrypoint.py", line 491, in _synthesize_one
t = syn_impl.synthesize(
ctx=tyctx,
...<7 lines>...
output_value=output_evaluator,
)
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/__init__.py", line 125, in synthesize
prepare_ollama_model(self.model)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 200, in prepare_ollama_model
pull_model(model)
~~~~~~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 139, in pull_model
if not is_model_installed(model):
~~~~~~~~~~~~~~~~~~^^^^^^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 84, in is_model_installed
return model in _installed_model_names()
~~~~~~~~~~~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-benchmark/aeon/synthesis/modules/llm/ollama_manager.py", line 76, in _installed_model_names
for entry in _ollama().list().models:
~~~~~~~~~~~~~~^^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 653, in list
return self._request(
~~~~~~~~~~~~~^
ListResponse,
^^^^^^^^^^^^^
'GET',
^^^^^^
'/api/tags',
^^^^^^^^^^^^
)
^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 203, in _request
return cls(**self._request_raw(*args, **kwargs).json())
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/tmp/aeon-knapsack-venv/lib/python3.13/site-packages/ollama/_client.py", line 149, in _request_raw
raise ConnectionError(CONNECTION_ERROR_MESSAGE) from None
ConnectionError: Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
{
"commit": "41a88a70b29e626bcbea98b61de8ed5fd758426c",
"date_utc": "2026-11-04T13:04:08Z",
"python": "3.13.15 (main, Aug 7 2026, 02:25:50) [Clang 22.1.3 ]",
"platform": "Linux-7.0.0-34-generic-x86_64-with-glibc2.43",
"cpu_count": 48,
"timeout_s": 10,
"seed": 0,
"oracle": {
"picks": [
1,
0,
1,
0,
0,
0,
0,
0,
1,
0,
1,
0,
1,
0,
1,
0,
1,
1,
1,
0,
1,
0,
1,
1,
0,
0,
0,
0,
0,
1,
1,
0
],
"items": [
0,
2,
8,
10,
12,
14,
16,
17,
18,
20,
22,
23,
29,
30
],
"weight": 100,
"value": 217,
"feasible": true
},
"backend_ids": [
"synquid",
"tactics",
"tdsyn_enumerative",
"tdsyn_random",
"bustle_baseline",
"bustle",
"decision_tree",
"noisy",
"structure_guided",
"contata",
"boolean_bounded",
"ortools",
"polynomial",
"smt",
"sygus",
"afta",
"cata",
"fta",
"arborist",
"lta",
"symetric",
"amaze",
"enumerative",
"random_search",
"regus",
"rl_mcts",
"dp_gp",
"gp",
"llm_codellama-13b",
"llm_deepseek-coder-v2-16b",
"llm_deepseek-coder-6.7b",
"llm_openai",
"llm_qwen2.5-coder-14b",
"llm_qwen2.5-coder-32b",
"llm_starcoder2-15b",
"llm_type_constrained",
"tdsyn",
"ortools_int",
"cpsat",
"xfta",
"llm"
],
"excluded_ids": [
"backward_abs",
"backward_app",
"backward_close",
"tdsyn_backward",
"backward_if",
"backward_lit",
"forward_close",
"forward_let_abs",
"forward_let_app",
"forward_let_if",
"forward_let_tabs",
"forward_let_tapp"
],
"source_sha256": "bfbb8c7352ca3b6b419da34238179d31b15f95190eaa56d8d593252ecc6b1d45",
"weights": [
12,
7,
11,
8,
9,
15,
6,
13,
4,
10,
5,
14,
3,
16,
2,
17,
8,
6,
9,
12,
7,
11,
5,
13,
4,
10,
15,
3,
14,
6,
9,
8
],
"values": [
24,
13,
23,
15,
16,
28,
12,
25,
9,
19,
11,
26,
7,
29,
5,
31,
18,
14,
20,
22,
17,
21,
10,
27,
8,
20,
30,
6,
28,
13,
19,
16
],
"capacity": 100,
"verification_timeout_s": 30,
"timeout_includes": "process startup, imports, compilation and synthesis",
"packages": {
"AeonLang": "4.10.1",
"z3-solver": "5.3.3.0",
"cvc5": "1.4.2",
"ortools": "9.15.6755",
"scikit-learn": "1.9.1",
"numpy": "2.2.6"
}
}
{"backend": "synquid", "event": "compiled", "elapsed_s": 9.526167938951403}
{"backend": "synquid", "event": "search_started", "elapsed_s": 9.532957394607365, "target": "bag"}
{"backend": "tactics", "event": "compiled", "elapsed_s": 9.531790220178664}
{"backend": "tactics", "event": "search_started", "elapsed_s": 9.538575659040362, "target": "bag"}
{"backend": "tdsyn_enumerative", "event": "compiled", "elapsed_s": 9.484505955129862}
{"backend": "tdsyn_enumerative", "event": "search_started", "elapsed_s": 9.491630794946104, "target": "bag"}
{"backend": "tdsyn_random", "event": "compiled", "elapsed_s": 9.525378956925124}
{"backend": "tdsyn_random", "event": "search_started", "elapsed_s": 9.532175515778363, "target": "bag"}
{"backend": "bustle_baseline", "event": "compiled", "elapsed_s": 9.504113586153835}
{"backend": "bustle_baseline", "event": "search_started", "elapsed_s": 9.510973354801536, "target": "bag"}
{"backend": "bustle_baseline", "event": "error", "elapsed_s": 9.514894702006131, "error_type": "UnsupportedSynthesisGoal", "error": "Unsupported synthesis goal: Cannot sample opaque type Unit "}
{"backend": "bustle", "event": "compiled", "elapsed_s": 9.522843789309263}
{"backend": "bustle", "event": "search_started", "elapsed_s": 9.529754144139588, "target": "bag"}
{"backend": "bustle", "event": "error", "elapsed_s": 9.533693666104227, "error_type": "UnsupportedSynthesisGoal", "error": "Unsupported synthesis goal: Cannot sample opaque type Unit "}
{"backend": "decision_tree", "event": "compiled", "elapsed_s": 9.447508943267167}
{"backend": "noisy", "event": "compiled", "elapsed_s": 9.511439625173807}
{"backend": "noisy", "event": "search_started", "elapsed_s": 9.518334740772843, "target": "bag"}
{"backend": "noisy", "event": "error", "elapsed_s": 9.518543045036495, "error_type": "UnsupportedSynthesisGoal", "error": "Unsupported synthesis goal: noisy requires every example to be a scalar literal I/O equality"}
{"backend": "structure_guided", "event": "compiled", "elapsed_s": 9.48663290310651}
{"backend": "structure_guided", "event": "search_started", "elapsed_s": 9.493514162953943, "target": "bag"}
{"backend": "structure_guided", "event": "error", "elapsed_s": 9.510554443113506, "error_type": "UnsupportedSynthesisGoal", "error": "Unsupported synthesis goal: structure_guided: needs unary constructor-literal @example I/O pairs"}
{"backend": "contata", "event": "compiled", "elapsed_s": 9.478736926335841}
{"backend": "contata", "event": "search_started", "elapsed_s": 9.485526171978563, "target": "bag"}
{"backend": "contata", "event": "error", "elapsed_s": 9.489521659910679, "error_type": "UnsupportedSynthesisGoal", "error": "Unsupported synthesis goal: Cannot sample opaque type Unit "}
{"backend": "boolean_bounded", "event": "compiled", "elapsed_s": 9.510299362242222}
{"backend": "boolean_bounded", "event": "search_started", "elapsed_s": 9.515479627996683, "target": "bag"}
{"backend": "boolean_bounded", "event": "error", "elapsed_s": 9.534877332858741, "error_type": "UnsupportedSynthesisGoal", "error": "Unsupported synthesis goal: Unsupported synthesis goal: every Boolean synthesis target needs @bounded_cnf(K)"}
{"backend": "ortools", "event": "compiled", "elapsed_s": 9.471507670357823}
{"backend": "ortools", "event": "error", "elapsed_s": 9.623292559292167, "error_type": "UnsupportedSynthesisGoal", "error": "Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead."}
{"backend": "polynomial", "event": "compiled", "elapsed_s": 9.491737025324255}
{"backend": "polynomial", "event": "error", "elapsed_s": 9.49952502688393, "error_type": "SynthesisNotSuccessful", "error": "Joint solver cannot translate type { xs⁵⁹⁶:List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) } | ((List_size⁷⁰²(xs⁵⁹⁶) == 32) && (weight⁵²²(xs⁵⁹⁶) <= 100)) }; expected Int, Bool or Float"}
{"backend": "smt", "event": "compiled", "elapsed_s": 9.533018661197275}
{"backend": "smt", "event": "search_started", "elapsed_s": 9.539879912044853, "target": "bag"}
{"backend": "sygus", "event": "compiled", "elapsed_s": 9.540425994899124}
{"backend": "sygus", "event": "search_started", "elapsed_s": 9.547238103114069, "target": "bag"}
{"backend": "sygus", "event": "error", "elapsed_s": 9.565422826912254, "error_type": "SynthesisNotSuccessful", "error": "SygusSynthesizer: hole of type { xs⁵⁹⁶:List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) } | ((List_size⁷⁰²(xs⁵⁹⁶) == 32) && (weight⁵²²(xs⁵⁹⁶) <= 100)) } is outside the SyGuS-supported subset (needs an Int/Bool/Float type, optionally refined, over native operators)."}
{"backend": "afta", "event": "compiled", "elapsed_s": 9.50717320619151}
{"backend": "afta", "event": "search_started", "elapsed_s": 9.513942494057119, "target": "bag"}
{"backend": "cata", "event": "compiled", "elapsed_s": 9.589145892299712}
{"backend": "cata", "event": "search_started", "elapsed_s": 9.59602233208716, "target": "bag"}
{"backend": "fta", "event": "compiled", "elapsed_s": 9.491152150090784}
{"backend": "fta", "event": "search_started", "elapsed_s": 9.497986069880426, "target": "bag"}
{"backend": "arborist", "event": "compiled", "elapsed_s": 9.487093653995544}
{"backend": "arborist", "event": "error", "elapsed_s": 9.49238927802071, "error_type": "UnsupportedSynthesisGoal", "error": "Unsupported synthesis goal: arborist requires literal scalar/constructor @example inputs and Int/Bool outputs"}
{"backend": "lta", "event": "compiled", "elapsed_s": 9.525506439153105}
{"backend": "lta", "event": "search_started", "elapsed_s": 9.532276578713208, "target": "bag"}
{"backend": "symetric", "event": "compiled", "elapsed_s": 9.471338198985904}
{"backend": "symetric", "event": "search_started", "elapsed_s": 9.478143003769219, "target": "bag"}
{"backend": "amaze", "event": "compiled", "elapsed_s": 9.45294292923063}
{"backend": "amaze", "event": "search_started", "elapsed_s": 9.45979439187795, "target": "bag"}
{"backend": "amaze", "event": "error", "elapsed_s": 9.490009772125632, "error_type": "UnsupportedSynthesisGoal", "error": "Unsupported synthesis goal: supported sorts are monomorphic Int/Bool; abstract refinements and higher-order types are unsupported"}
{"backend": "enumerative", "event": "compiled", "elapsed_s": 9.53092859312892}
{"backend": "enumerative", "event": "search_started", "elapsed_s": 9.537709692958742, "target": "bag"}
{"backend": "random_search", "event": "compiled", "elapsed_s": 9.550534389913082}
{"backend": "random_search", "event": "search_started", "elapsed_s": 9.557411811780185, "target": "bag"}
{"backend": "regus", "event": "compiled", "elapsed_s": 9.577049711253494}
{"backend": "regus", "event": "search_started", "elapsed_s": 9.5837801550515, "target": "bag"}
{"backend": "regus", "event": "error", "elapsed_s": 9.601569401100278, "error_type": "UnsupportedSynthesisGoal", "error": "Unsupported synthesis goal: regus expects a ControlProgram hole; use a structure-synthesis problem"}
{"backend": "rl_mcts", "event": "compiled", "elapsed_s": 9.558441828005016}
{"backend": "rl_mcts", "event": "search_started", "elapsed_s": 9.565309822093695, "target": "bag"}
{"backend": "rl_mcts", "event": "error", "elapsed_s": 9.582942231092602, "error_type": "SynthesisNotSuccessful", "error": "rl_mcts solves refinement specifications; optimization goals are not supported"}
{"backend": "dp_gp", "event": "compiled", "elapsed_s": 9.590247923042625}
{"backend": "dp_gp", "event": "error", "elapsed_s": 9.595597960054874, "error_type": "UnsupportedSynthesisGoal", "error": "Unsupported synthesis goal: dp_gp requires @dp_gp(reference, max_first_input, max_second_input)"}
{"backend": "gp", "event": "compiled", "elapsed_s": 9.473391328006983}
{"backend": "gp", "event": "search_started", "elapsed_s": 9.480300891213119, "target": "bag"}
{"backend": "llm_codellama-13b", "event": "compiled", "elapsed_s": 9.513522299937904}
{"backend": "llm_codellama-13b", "event": "search_started", "elapsed_s": 9.520399220753461, "target": "bag"}
{"backend": "llm_codellama-13b", "event": "error", "elapsed_s": 9.646658768877387, "error_type": "ConnectionError", "error": "Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download"}
{"backend": "llm_deepseek-coder-v2-16b", "event": "compiled", "elapsed_s": 9.593429051805288}
{"backend": "llm_deepseek-coder-v2-16b", "event": "search_started", "elapsed_s": 9.600421371869743, "target": "bag"}
{"backend": "llm_deepseek-coder-v2-16b", "event": "error", "elapsed_s": 9.726786558981985, "error_type": "ConnectionError", "error": "Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download"}
{"backend": "llm_deepseek-coder-6.7b", "event": "compiled", "elapsed_s": 9.56073258491233}
{"backend": "llm_deepseek-coder-6.7b", "event": "search_started", "elapsed_s": 9.567893282976002, "target": "bag"}
{"backend": "llm_deepseek-coder-6.7b", "event": "error", "elapsed_s": 9.696212342008948, "error_type": "ConnectionError", "error": "Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download"}
{"backend": "llm_openai", "event": "compiled", "elapsed_s": 9.67854826292023}
{"backend": "llm_openai", "event": "search_started", "elapsed_s": 9.68555164290592, "target": "bag"}
{"backend": "llm_qwen2.5-coder-14b", "event": "compiled", "elapsed_s": 9.57292224187404}
{"backend": "llm_qwen2.5-coder-14b", "event": "search_started", "elapsed_s": 9.579849417787045, "target": "bag"}
{"backend": "llm_qwen2.5-coder-14b", "event": "error", "elapsed_s": 9.710454043932259, "error_type": "ConnectionError", "error": "Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download"}
{"backend": "llm_qwen2.5-coder-32b", "event": "compiled", "elapsed_s": 9.59823248675093}
{"backend": "llm_qwen2.5-coder-32b", "event": "search_started", "elapsed_s": 9.60515247983858, "target": "bag"}
{"backend": "llm_qwen2.5-coder-32b", "event": "error", "elapsed_s": 9.732117890845984, "error_type": "ConnectionError", "error": "Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download"}
{"backend": "llm_starcoder2-15b", "event": "compiled", "elapsed_s": 9.571298936847597}
{"backend": "llm_starcoder2-15b", "event": "search_started", "elapsed_s": 9.578201335854828, "target": "bag"}
{"backend": "llm_starcoder2-15b", "event": "error", "elapsed_s": 9.705881136935204, "error_type": "ConnectionError", "error": "Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download"}
{"backend": "llm_type_constrained", "event": "compiled", "elapsed_s": 9.573928404133767}
{"backend": "llm_type_constrained", "event": "error", "elapsed_s": 9.58227254729718, "error_type": "UnsupportedSynthesisGoal", "error": "Unsupported synthesis goal: llm_type_constrained needs monomorphic Int/Bool/String/Float/Unit or unrestricted arrows; got List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }"}
{"backend": "tdsyn", "event": "compiled", "elapsed_s": 9.505353247746825}
{"backend": "tdsyn", "event": "search_started", "elapsed_s": 9.512207637075335, "target": "bag"}
{"backend": "ortools_int", "event": "compiled", "elapsed_s": 9.555381095968187}
{"backend": "ortools_int", "event": "error", "elapsed_s": 9.705892297904938, "error_type": "UnsupportedSynthesisGoal", "error": "Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead."}
{"backend": "cpsat", "event": "compiled", "elapsed_s": 9.592233133967966}
{"backend": "cpsat", "event": "error", "elapsed_s": 9.741221589036286, "error_type": "UnsupportedSynthesisGoal", "error": "Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead."}
{"backend": "xfta", "event": "compiled", "elapsed_s": 9.545521688647568}
{"backend": "xfta", "event": "search_started", "elapsed_s": 9.55237323185429, "target": "bag"}
{"backend": "llm", "event": "compiled", "elapsed_s": 9.504071296192706}
{"backend": "llm", "event": "search_started", "elapsed_s": 9.510900628287345, "target": "bag"}
{"backend": "llm", "event": "error", "elapsed_s": 9.637133245356381, "error_type": "ConnectionError", "error": "Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download"}
import List;
# Native bodies build genuine stdlib List.cons/List.nil values.
# Weights and values for the same 32-item instance.
def weights : {xs:(List Int) | List.size xs = 32} :=
native "__import__('functools').reduce(lambda tail, x: List_cons(x)(tail), reversed([12, 7, 11, 8, 9, 15, 6, 13, 4, 10, 5, 14, 3, 16, 2, 17, 8, 6, 9, 12, 7, 11, 5, 13, 4, 10, 15, 3, 14, 6, 9, 8]), List_nil)";
def values : {xs:(List Int) | List.size xs = 32} :=
native "__import__('functools').reduce(lambda tail, x: List_cons(x)(tail), reversed([24, 13, 23, 15, 16, 28, 12, 25, 9, 19, 11, 26, 7, 29, 5, 31, 18, 14, 20, 22, 17, 21, 10, 27, 8, 20, 30, 6, 28, 13, 19, 16]), List_nil)";
# Traverse the inductive lists using their stdlib constructors.
def dot (xs:(List Int)) (ys:(List Int)) : Int :=
match xs with
| List.nil => 0
| List.cons x xt =>
match ys with
| List.nil => 0
| List.cons y yt => x * y + dot xt yt;
def weight (picks:(List Int)) : Int := dot weights picks;
def value (picks:(List Int)) : Int := dot values picks;
# Native constructor: exactly 32 binary decisions, in item order.
def list32
(x0:{b:Int | 0 <= b && b <= 1}) (x1:{b:Int | 0 <= b && b <= 1})
(x2:{b:Int | 0 <= b && b <= 1}) (x3:{b:Int | 0 <= b && b <= 1})
(x4:{b:Int | 0 <= b && b <= 1}) (x5:{b:Int | 0 <= b && b <= 1})
(x6:{b:Int | 0 <= b && b <= 1}) (x7:{b:Int | 0 <= b && b <= 1})
(x8:{b:Int | 0 <= b && b <= 1}) (x9:{b:Int | 0 <= b && b <= 1})
(x10:{b:Int | 0 <= b && b <= 1}) (x11:{b:Int | 0 <= b && b <= 1})
(x12:{b:Int | 0 <= b && b <= 1}) (x13:{b:Int | 0 <= b && b <= 1})
(x14:{b:Int | 0 <= b && b <= 1}) (x15:{b:Int | 0 <= b && b <= 1})
(x16:{b:Int | 0 <= b && b <= 1}) (x17:{b:Int | 0 <= b && b <= 1})
(x18:{b:Int | 0 <= b && b <= 1}) (x19:{b:Int | 0 <= b && b <= 1})
(x20:{b:Int | 0 <= b && b <= 1}) (x21:{b:Int | 0 <= b && b <= 1})
(x22:{b:Int | 0 <= b && b <= 1}) (x23:{b:Int | 0 <= b && b <= 1})
(x24:{b:Int | 0 <= b && b <= 1}) (x25:{b:Int | 0 <= b && b <= 1})
(x26:{b:Int | 0 <= b && b <= 1}) (x27:{b:Int | 0 <= b && b <= 1})
(x28:{b:Int | 0 <= b && b <= 1}) (x29:{b:Int | 0 <= b && b <= 1})
(x30:{b:Int | 0 <= b && b <= 1}) (x31:{b:Int | 0 <= b && b <= 1}) :
{a:(List {b:Int | 0 <= b && b <= 1}) |
List.size a = 32 &&
weight a = (12 * x0 + 7 * x1 + 11 * x2 + 8 * x3 + 9 * x4 + 15 * x5 + 6 * x6 + 13 * x7
+ 4 * x8 + 10 * x9 + 5 * x10 + 14 * x11 + 3 * x12 + 16 * x13 + 2 * x14 + 17 * x15
+ 8 * x16 + 6 * x17 + 9 * x18 + 12 * x19 + 7 * x20 + 11 * x21 + 5 * x22 + 13 * x23
+ 4 * x24 + 10 * x25 + 15 * x26 + 3 * x27 + 14 * x28 + 6 * x29 + 9 * x30 + 8 * x31)} :=
native "__import__('functools').reduce(lambda tail, x: List_cons(x)(tail), reversed([x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15, x16, x17, x18, x19, x20, x21, x22, x23, x24, x25, x26, x27, x28, x29, x30, x31]), List_nil)";
# The native constructor's contract gives the solver the exact weight.
# Capacity is a hard refinement; maximize value among feasible selections.
@maximize_int(value (chosen unit))
def chosen (_:Unit) :
{xs:(List {b:Int | 0 <= b && b <= 1}) |
List.size xs = 32 && weight xs <= 100} :=
?bag;
def main (_:Int) : Int := value (chosen unit);
# Independent dynamic-programming reference; NOT a synthesized benchmark result.
# Optimal value 217, weight 100; independently confirmed with CP-SAT.
import List;
# Native bodies build genuine stdlib List.cons/List.nil values.
# Weights and values for the same 32-item instance.
def weights : {xs:(List Int) | List.size xs = 32} :=
native "__import__('functools').reduce(lambda tail, x: List_cons(x)(tail), reversed([12, 7, 11, 8, 9, 15, 6, 13, 4, 10, 5, 14, 3, 16, 2, 17, 8, 6, 9, 12, 7, 11, 5, 13, 4, 10, 15, 3, 14, 6, 9, 8]), List_nil)";
def values : {xs:(List Int) | List.size xs = 32} :=
native "__import__('functools').reduce(lambda tail, x: List_cons(x)(tail), reversed([24, 13, 23, 15, 16, 28, 12, 25, 9, 19, 11, 26, 7, 29, 5, 31, 18, 14, 20, 22, 17, 21, 10, 27, 8, 20, 30, 6, 28, 13, 19, 16]), List_nil)";
# Traverse the inductive lists using their stdlib constructors.
def dot (xs:(List Int)) (ys:(List Int)) : Int :=
match xs with
| List.nil => 0
| List.cons x xt =>
match ys with
| List.nil => 0
| List.cons y yt => x * y + dot xt yt;
def weight (picks:(List Int)) : Int := dot weights picks;
def value (picks:(List Int)) : Int := dot values picks;
# Native constructor: exactly 32 binary decisions, in item order.
def list32
(x0:{b:Int | 0 <= b && b <= 1}) (x1:{b:Int | 0 <= b && b <= 1})
(x2:{b:Int | 0 <= b && b <= 1}) (x3:{b:Int | 0 <= b && b <= 1})
(x4:{b:Int | 0 <= b && b <= 1}) (x5:{b:Int | 0 <= b && b <= 1})
(x6:{b:Int | 0 <= b && b <= 1}) (x7:{b:Int | 0 <= b && b <= 1})
(x8:{b:Int | 0 <= b && b <= 1}) (x9:{b:Int | 0 <= b && b <= 1})
(x10:{b:Int | 0 <= b && b <= 1}) (x11:{b:Int | 0 <= b && b <= 1})
(x12:{b:Int | 0 <= b && b <= 1}) (x13:{b:Int | 0 <= b && b <= 1})
(x14:{b:Int | 0 <= b && b <= 1}) (x15:{b:Int | 0 <= b && b <= 1})
(x16:{b:Int | 0 <= b && b <= 1}) (x17:{b:Int | 0 <= b && b <= 1})
(x18:{b:Int | 0 <= b && b <= 1}) (x19:{b:Int | 0 <= b && b <= 1})
(x20:{b:Int | 0 <= b && b <= 1}) (x21:{b:Int | 0 <= b && b <= 1})
(x22:{b:Int | 0 <= b && b <= 1}) (x23:{b:Int | 0 <= b && b <= 1})
(x24:{b:Int | 0 <= b && b <= 1}) (x25:{b:Int | 0 <= b && b <= 1})
(x26:{b:Int | 0 <= b && b <= 1}) (x27:{b:Int | 0 <= b && b <= 1})
(x28:{b:Int | 0 <= b && b <= 1}) (x29:{b:Int | 0 <= b && b <= 1})
(x30:{b:Int | 0 <= b && b <= 1}) (x31:{b:Int | 0 <= b && b <= 1}) :
{a:(List {b:Int | 0 <= b && b <= 1}) |
List.size a = 32 &&
weight a = (12 * x0 + 7 * x1 + 11 * x2 + 8 * x3 + 9 * x4 + 15 * x5 + 6 * x6 + 13 * x7
+ 4 * x8 + 10 * x9 + 5 * x10 + 14 * x11 + 3 * x12 + 16 * x13 + 2 * x14 + 17 * x15
+ 8 * x16 + 6 * x17 + 9 * x18 + 12 * x19 + 7 * x20 + 11 * x21 + 5 * x22 + 13 * x23
+ 4 * x24 + 10 * x25 + 15 * x26 + 3 * x27 + 14 * x28 + 6 * x29 + 9 * x30 + 8 * x31)} :=
native "__import__('functools').reduce(lambda tail, x: List_cons(x)(tail), reversed([x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15, x16, x17, x18, x19, x20, x21, x22, x23, x24, x25, x26, x27, x28, x29, x30, x31]), List_nil)";
# The native constructor's contract gives the solver the exact weight.
# Capacity is a hard refinement; maximize value among feasible selections.
@maximize_int(value (chosen unit))
def chosen (_:Unit) :
{xs:(List {b:Int | 0 <= b && b <= 1}) |
List.size xs = 32 && weight xs <= 100} :=
list32 1 0 1 0 0 0 0 0 1 0 1 0 1 0 1 0 1 1 1 0 1 0 1 1 0 0 0 0 0 1 1 0;
def main (_:Int) : Int := value (chosen unit);
absl-py==2.5.1
annotated-types==0.8.0
anyio==4.16.0
attrs==26.1.0
beartype==0.23.1
cattrs==26.2.1
certifi==2026.10.23
charset-normalizer==3.5.2
cloudpickle==3.1.2
coverage==7.17.1
cvc5==1.4.2
dill==0.4.1
execnet==2.1.2
h11==0.16.0
httpcore==1.0.9
httpx==0.28.1
hypothesis==6.170.14
idna==3.20
imageio==2.38.1
immutabledict==4.3.1
iniconfig==2.3.1
joblib==1.6.0
lark==1.3.1
lazy-loader==0.6
llvmlite==0.50.0
loguru==0.7.3
lsprotocol==2025.0.0
mpmath==1.3.0
multiprocess==0.70.19
narwhals==2.28.0
networkx==3.7
numpy==2.2.6
ollama==0.6.3
ortools==9.15.6755
packaging==26.3
pandas==3.1.0
pillow==13.0.0
pluggy==1.7.0
protobuf==6.33.6
psb2==1.1.2
pydantic==2.14.2
pydantic-core==2.50.2
pygls==2.1.1
pygments==2.21.0
pytest==9.2.0
pytest-beartype==0.3.0
pytest-cov==7.1.0
pytest-xdist==3.8.0
python-dateutil==4.0.0
requests==2.34.2
scikit-image==0.26.0
scikit-learn==1.9.1
scipy==1.18.1
six==1.17.0
sortedcontainers==2.4.0
sympy==1.14.0
textdistance==4.6.3
threadpoolctl==3.7.0
tifffile==2026.10.15
types-requests==2.33.0.20261030
typing-extensions==4.16.0
typing-inspection==0.4.4
tzdata==2026.6
urllib3==2.8.0
z3-solver==5.3.3.0
zss==1.2.0
backend alias_of status picks weight value gap first_best_s wall_s expression error
synquid timeout 10.023037449922413
tactics timeout 10.015118711162359
tdsyn_enumerative timeout 10.023383263964206
tdsyn_random timeout 10.023177983704954
bustle_baseline error 9.681452578864992 Unsupported synthesis goal: Cannot sample opaque type Unit
bustle error 9.682414656039327 Unsupported synthesis goal: Cannot sample opaque type Unit
decision_tree timeout 10.040620406158268
noisy error 9.681949196849018 Unsupported synthesis goal: noisy requires every example to be a scalar literal I/O equality
structure_guided error 9.68156204605475 Unsupported synthesis goal: structure_guided: needs unary constructor-literal @example I/O pairs
contata error 9.631878016982228 Unsupported synthesis goal: Cannot sample opaque type Unit
boolean_bounded error 9.733009306248277 Unsupported synthesis goal: Unsupported synthesis goal: every Boolean synthesis target needs @bounded_cnf(K)
ortools error 9.832463395316154 Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead.
polynomial error 9.682061368133873 Joint solver cannot translate type { xs⁵⁹⁶:List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) } | ((List_size⁷⁰²(xs⁵⁹⁶) == 32) && (weight⁵²²(xs⁵⁹⁶) <= 100)) }; expected Int, Bool or Float
smt timeout 10.014682627283037
sygus error 9.733235322870314 SygusSynthesizer: hole of type { xs⁵⁹⁶:List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) } | ((List_size⁷⁰²(xs⁵⁹⁶) == 32) && (weight⁵²²(xs⁵⁹⁶) <= 100)) } is outside the SyGuS-supported subset (needs an Int/Bool/Float type, optionally refined, over native operators).
afta timeout 10.023864814080298
cata timeout 10.014625450130552
fta timeout 10.024041056167334
arborist error 9.631882174871862 Unsupported synthesis goal: arborist requires literal scalar/constructor @example inputs and Int/Bool outputs
lta timeout 10.024903275072575
symetric timeout 10.018933376763016
amaze error 9.682134106289595 Unsupported synthesis goal: supported sorts are monomorphic Int/Bool; abstract refinements and higher-order types are unsupported
enumerative timeout 10.02315284917131
random_search timeout 10.022784471977502
regus error 9.78195700282231 Unsupported synthesis goal: regus expects a ControlProgram hole; use a structure-synthesis problem
rl_mcts error 9.731873820070177 rl_mcts solves refinement specifications; optimization goals are not supported
dp_gp error 9.731884369626641 Unsupported synthesis goal: dp_gp requires @dp_gp(reference, max_first_input, max_second_input)
gp timeout 10.024025697261095
llm_codellama-13b error 9.831500104628503 Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
llm_deepseek-coder-v2-16b error 9.881423054728657 Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
llm_deepseek-coder-6.7b error 9.88222035067156 Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
llm_openai timeout 10.0210221237503
llm_qwen2.5-coder-14b error 9.882546776905656 Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
llm_qwen2.5-coder-32b error 9.93191818613559 Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
llm_starcoder2-15b error 9.88315461575985 Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
llm_type_constrained error 9.73150697723031 Unsupported synthesis goal: llm_type_constrained needs monomorphic Int/Bool/String/Float/Unit or unrestricted arrows; got List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }
tdsyn tdsyn_enumerative timeout 10.024059490766376
ortools_int ortools error 9.881848237942904 Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead.
cpsat ortools error 9.931967650074512 Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List⁶⁸⁰ { b⁵⁹⁵:Int | ((0 <= b⁵⁹⁵) && (b⁵⁹⁵ <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead.
xfta symetric timeout 10.008187244646251
llm llm_qwen2.5-coder-32b error 9.832285890355706 Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download
[
{
"backend": "synquid",
"alias_of": null,
"status": "timeout",
"wall_s": 10.023037449922413,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.526167938951403,
"search_started_s": 9.532957394607365,
"progress": null
},
{
"backend": "tactics",
"alias_of": null,
"status": "timeout",
"wall_s": 10.015118711162359,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.531790220178664,
"search_started_s": 9.538575659040362,
"progress": null
},
{
"backend": "tdsyn_enumerative",
"alias_of": null,
"status": "timeout",
"wall_s": 10.023383263964206,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.484505955129862,
"search_started_s": 9.491630794946104,
"progress": null
},
{
"backend": "tdsyn_random",
"alias_of": null,
"status": "timeout",
"wall_s": 10.023177983704954,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.525378956925124,
"search_started_s": 9.532175515778363,
"progress": null
},
{
"backend": "bustle_baseline",
"alias_of": null,
"status": "error",
"wall_s": 9.681452578864992,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.514894702006131,
"error_type": "UnsupportedSynthesisGoal",
"error": "Unsupported synthesis goal: Cannot sample opaque type Unit "
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.504113586153835,
"search_started_s": 9.510973354801536,
"progress": null
},
{
"backend": "bustle",
"alias_of": null,
"status": "error",
"wall_s": 9.682414656039327,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.533693666104227,
"error_type": "UnsupportedSynthesisGoal",
"error": "Unsupported synthesis goal: Cannot sample opaque type Unit "
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.522843789309263,
"search_started_s": 9.529754144139588,
"progress": null
},
{
"backend": "decision_tree",
"alias_of": null,
"status": "timeout",
"wall_s": 10.040620406158268,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.447508943267167,
"search_started_s": null,
"progress": null
},
{
"backend": "noisy",
"alias_of": null,
"status": "error",
"wall_s": 9.681949196849018,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.518543045036495,
"error_type": "UnsupportedSynthesisGoal",
"error": "Unsupported synthesis goal: noisy requires every example to be a scalar literal I/O equality"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.511439625173807,
"search_started_s": 9.518334740772843,
"progress": null
},
{
"backend": "structure_guided",
"alias_of": null,
"status": "error",
"wall_s": 9.68156204605475,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.510554443113506,
"error_type": "UnsupportedSynthesisGoal",
"error": "Unsupported synthesis goal: structure_guided: needs unary constructor-literal @example I/O pairs"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.48663290310651,
"search_started_s": 9.493514162953943,
"progress": null
},
{
"backend": "contata",
"alias_of": null,
"status": "error",
"wall_s": 9.631878016982228,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.489521659910679,
"error_type": "UnsupportedSynthesisGoal",
"error": "Unsupported synthesis goal: Cannot sample opaque type Unit "
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.478736926335841,
"search_started_s": 9.485526171978563,
"progress": null
},
{
"backend": "boolean_bounded",
"alias_of": null,
"status": "error",
"wall_s": 9.733009306248277,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.534877332858741,
"error_type": "UnsupportedSynthesisGoal",
"error": "Unsupported synthesis goal: Unsupported synthesis goal: every Boolean synthesis target needs @bounded_cnf(K)"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.510299362242222,
"search_started_s": 9.515479627996683,
"progress": null
},
{
"backend": "ortools",
"alias_of": null,
"status": "error",
"wall_s": 9.832463395316154,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.623292559292167,
"error_type": "UnsupportedSynthesisGoal",
"error": "Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List\u2076\u2078\u2070 { b\u2075\u2079\u2075:Int | ((0 <= b\u2075\u2079\u2075) && (b\u2075\u2079\u2075 <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead."
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.471507670357823,
"search_started_s": null,
"progress": null
},
{
"backend": "polynomial",
"alias_of": null,
"status": "error",
"wall_s": 9.682061368133873,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.49952502688393,
"error_type": "SynthesisNotSuccessful",
"error": "Joint solver cannot translate type { xs\u2075\u2079\u2076:List\u2076\u2078\u2070 { b\u2075\u2079\u2075:Int | ((0 <= b\u2075\u2079\u2075) && (b\u2075\u2079\u2075 <= 1)) } | ((List_size\u2077\u2070\u00b2(xs\u2075\u2079\u2076) == 32) && (weight\u2075\u00b2\u00b2(xs\u2075\u2079\u2076) <= 100)) }; expected Int, Bool or Float"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.491737025324255,
"search_started_s": null,
"progress": null
},
{
"backend": "smt",
"alias_of": null,
"status": "timeout",
"wall_s": 10.014682627283037,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.533018661197275,
"search_started_s": 9.539879912044853,
"progress": null
},
{
"backend": "sygus",
"alias_of": null,
"status": "error",
"wall_s": 9.733235322870314,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.565422826912254,
"error_type": "SynthesisNotSuccessful",
"error": "SygusSynthesizer: hole of type { xs\u2075\u2079\u2076:List\u2076\u2078\u2070 { b\u2075\u2079\u2075:Int | ((0 <= b\u2075\u2079\u2075) && (b\u2075\u2079\u2075 <= 1)) } | ((List_size\u2077\u2070\u00b2(xs\u2075\u2079\u2076) == 32) && (weight\u2075\u00b2\u00b2(xs\u2075\u2079\u2076) <= 100)) } is outside the SyGuS-supported subset (needs an Int/Bool/Float type, optionally refined, over native operators)."
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.540425994899124,
"search_started_s": 9.547238103114069,
"progress": null
},
{
"backend": "afta",
"alias_of": null,
"status": "timeout",
"wall_s": 10.023864814080298,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.50717320619151,
"search_started_s": 9.513942494057119,
"progress": null
},
{
"backend": "cata",
"alias_of": null,
"status": "timeout",
"wall_s": 10.014625450130552,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.589145892299712,
"search_started_s": 9.59602233208716,
"progress": null
},
{
"backend": "fta",
"alias_of": null,
"status": "timeout",
"wall_s": 10.024041056167334,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.491152150090784,
"search_started_s": 9.497986069880426,
"progress": null
},
{
"backend": "arborist",
"alias_of": null,
"status": "error",
"wall_s": 9.631882174871862,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.49238927802071,
"error_type": "UnsupportedSynthesisGoal",
"error": "Unsupported synthesis goal: arborist requires literal scalar/constructor @example inputs and Int/Bool outputs"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.487093653995544,
"search_started_s": null,
"progress": null
},
{
"backend": "lta",
"alias_of": null,
"status": "timeout",
"wall_s": 10.024903275072575,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.525506439153105,
"search_started_s": 9.532276578713208,
"progress": null
},
{
"backend": "symetric",
"alias_of": null,
"status": "timeout",
"wall_s": 10.018933376763016,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.471338198985904,
"search_started_s": 9.478143003769219,
"progress": null
},
{
"backend": "amaze",
"alias_of": null,
"status": "error",
"wall_s": 9.682134106289595,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.490009772125632,
"error_type": "UnsupportedSynthesisGoal",
"error": "Unsupported synthesis goal: supported sorts are monomorphic Int/Bool; abstract refinements and higher-order types are unsupported"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.45294292923063,
"search_started_s": 9.45979439187795,
"progress": null
},
{
"backend": "enumerative",
"alias_of": null,
"status": "timeout",
"wall_s": 10.02315284917131,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.53092859312892,
"search_started_s": 9.537709692958742,
"progress": null
},
{
"backend": "random_search",
"alias_of": null,
"status": "timeout",
"wall_s": 10.022784471977502,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.550534389913082,
"search_started_s": 9.557411811780185,
"progress": null
},
{
"backend": "regus",
"alias_of": null,
"status": "error",
"wall_s": 9.78195700282231,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.601569401100278,
"error_type": "UnsupportedSynthesisGoal",
"error": "Unsupported synthesis goal: regus expects a ControlProgram hole; use a structure-synthesis problem"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.577049711253494,
"search_started_s": 9.5837801550515,
"progress": null
},
{
"backend": "rl_mcts",
"alias_of": null,
"status": "error",
"wall_s": 9.731873820070177,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.582942231092602,
"error_type": "SynthesisNotSuccessful",
"error": "rl_mcts solves refinement specifications; optimization goals are not supported"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.558441828005016,
"search_started_s": 9.565309822093695,
"progress": null
},
{
"backend": "dp_gp",
"alias_of": null,
"status": "error",
"wall_s": 9.731884369626641,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.595597960054874,
"error_type": "UnsupportedSynthesisGoal",
"error": "Unsupported synthesis goal: dp_gp requires @dp_gp(reference, max_first_input, max_second_input)"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.590247923042625,
"search_started_s": null,
"progress": null
},
{
"backend": "gp",
"alias_of": null,
"status": "timeout",
"wall_s": 10.024025697261095,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.473391328006983,
"search_started_s": 9.480300891213119,
"progress": null
},
{
"backend": "llm_codellama-13b",
"alias_of": null,
"status": "error",
"wall_s": 9.831500104628503,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.646658768877387,
"error_type": "ConnectionError",
"error": "Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.513522299937904,
"search_started_s": 9.520399220753461,
"progress": null
},
{
"backend": "llm_deepseek-coder-v2-16b",
"alias_of": null,
"status": "error",
"wall_s": 9.881423054728657,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.726786558981985,
"error_type": "ConnectionError",
"error": "Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.593429051805288,
"search_started_s": 9.600421371869743,
"progress": null
},
{
"backend": "llm_deepseek-coder-6.7b",
"alias_of": null,
"status": "error",
"wall_s": 9.88222035067156,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.696212342008948,
"error_type": "ConnectionError",
"error": "Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.56073258491233,
"search_started_s": 9.567893282976002,
"progress": null
},
{
"backend": "llm_openai",
"alias_of": null,
"status": "timeout",
"wall_s": 10.0210221237503,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.67854826292023,
"search_started_s": 9.68555164290592,
"progress": null
},
{
"backend": "llm_qwen2.5-coder-14b",
"alias_of": null,
"status": "error",
"wall_s": 9.882546776905656,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.710454043932259,
"error_type": "ConnectionError",
"error": "Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.57292224187404,
"search_started_s": 9.579849417787045,
"progress": null
},
{
"backend": "llm_qwen2.5-coder-32b",
"alias_of": null,
"status": "error",
"wall_s": 9.93191818613559,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.732117890845984,
"error_type": "ConnectionError",
"error": "Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.59823248675093,
"search_started_s": 9.60515247983858,
"progress": null
},
{
"backend": "llm_starcoder2-15b",
"alias_of": null,
"status": "error",
"wall_s": 9.88315461575985,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.705881136935204,
"error_type": "ConnectionError",
"error": "Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.571298936847597,
"search_started_s": 9.578201335854828,
"progress": null
},
{
"backend": "llm_type_constrained",
"alias_of": null,
"status": "error",
"wall_s": 9.73150697723031,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.58227254729718,
"error_type": "UnsupportedSynthesisGoal",
"error": "Unsupported synthesis goal: llm_type_constrained needs monomorphic Int/Bool/String/Float/Unit or unrestricted arrows; got List\u2076\u2078\u2070 { b\u2075\u2079\u2075:Int | ((0 <= b\u2075\u2079\u2075) && (b\u2075\u2079\u2075 <= 1)) }"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.573928404133767,
"search_started_s": null,
"progress": null
},
{
"backend": "tdsyn",
"alias_of": "tdsyn_enumerative",
"status": "timeout",
"wall_s": 10.024059490766376,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.505353247746825,
"search_started_s": 9.512207637075335,
"progress": null
},
{
"backend": "ortools_int",
"alias_of": "ortools",
"status": "error",
"wall_s": 9.881848237942904,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.705892297904938,
"error_type": "UnsupportedSynthesisGoal",
"error": "Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List\u2076\u2078\u2070 { b\u2075\u2079\u2075:Int | ((0 <= b\u2075\u2079\u2075) && (b\u2075\u2079\u2075 <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead."
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.555381095968187,
"search_started_s": null,
"progress": null
},
{
"backend": "cpsat",
"alias_of": "ortools",
"status": "error",
"wall_s": 9.931967650074512,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.741221589036286,
"error_type": "UnsupportedSynthesisGoal",
"error": "Unsupported synthesis goal: CPSatHole cannot handle hole bag (type List\u2076\u2078\u2070 { b\u2075\u2079\u2075:Int | ((0 <= b\u2075\u2079\u2075) && (b\u2075\u2079\u2075 <= 1)) }). Use a grammar-based synthesizer (e.g. -s gp) instead."
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.592233133967966,
"search_started_s": null,
"progress": null
},
{
"backend": "xfta",
"alias_of": "symetric",
"status": "timeout",
"wall_s": 10.008187244646251,
"exit_code": -9,
"error": null,
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.545521688647568,
"search_started_s": 9.55237323185429,
"progress": null
},
{
"backend": "llm",
"alias_of": "llm_qwen2.5-coder-32b",
"status": "error",
"wall_s": 9.832285890355706,
"exit_code": 1,
"error": {
"event": "error",
"elapsed_s": 9.637133245356381,
"error_type": "ConnectionError",
"error": "Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download"
},
"best": null,
"candidates": [],
"optimum": 217,
"gap": null,
"compiled_s": 9.504071296192706,
"search_started_s": 9.510900628287345,
"progress": null
}
]
#!/usr/bin/env python3
"""Benchmark 32-item inductive-list knapsack, excluding single-step tactics.
Run from the pinned Aeon checkout using its virtualenv. Only the public driver
and synthesis UI are used; backend search code and objectives are unchanged.
The hard timeout includes Python startup, imports, compilation and synthesis.
Candidate verification and an independent dynamic-programming oracle are outside the timed search.
"""
import argparse
import csv
import importlib.metadata
import hashlib
import json
import os
from pathlib import Path
import platform
import signal
import subprocess
import sys
import time
import traceback
WEIGHTS = [12,7,11,8,9,15,6,13,4,10,5,14,3,16,2,17,8,6,9,12,7,11,5,13,4,10,15,3,14,6,9,8]
VALUES = [24,13,23,15,16,28,12,25,9,19,11,26,7,29,5,31,18,14,20,22,17,21,10,27,8,20,30,6,28,13,19,16]
CAPACITY = 100
def is_single_step(backend):
return backend == "tdsyn_backward" or backend.startswith(("backward_", "forward_"))
def decoded(picks):
if len(picks) != 32 or any(type(x) is not int or x not in (0, 1) for x in picks):
raise ValueError(f"Expected 32 binary entries, got {picks!r}")
weight = sum(w * p for w, p in zip(WEIGHTS, picks, strict=True))
value = sum(v * p for v, p in zip(VALUES, picks, strict=True))
return dict(picks=picks, items=[i for i, p in enumerate(picks) if p], weight=weight,
value=value, feasible=weight <= CAPACITY)
def oracle_solution():
# 0-1 dynamic programming over exact weights. Never supplied to a worker.
states = {0: (0, [])}
for i, (weight, value) in enumerate(zip(WEIGHTS, VALUES, strict=True)):
for used, (score, selected) in list(states.items()):
total = used + weight
if total <= CAPACITY and score + value > states.get(total, (-1, []))[0]:
states[total] = (score + value, selected + [i])
_, (_, selected) = max(states.items(), key=lambda pair: (pair[1][0], -pair[0]))
return decoded([int(i in selected) for i in range(32)])
def emit(args, event, **fields):
record = dict(event=event, elapsed_s=time.monotonic() - args.started, **fields)
with (args.run_dir / f"{args.backend}.jsonl").open("a") as stream:
stream.write(json.dumps(record, allow_nan=False) + "\n")
def worker(args):
from loguru import logger
logger.remove()
from aeon.facade.driver import AeonConfig, AeonDriver
from aeon.synthesis.identification import term_has_holes
from aeon.synthesis.joint import JointCandidate
from aeon.sugar.lifting import lift
from aeon.synthesis.uis.api import SynthesisUI
from aeon.utils.pprint import pretty_print_sterm
class CaptureUI(SynthesisUI):
def __init__(self):
self.best_score = float("-inf")
self.seen = set()
self.last_progress = 0.0
def start(self, typing_ctx, evaluation_ctx, target_name, target_type, budget):
emit(args, "search_started", target=str(target_name))
def progress(self, created, assessed, elapsed_time):
if time.monotonic() - self.last_progress >= 1:
self.last_progress = time.monotonic()
emit(args, "progress", created=created, assessed=assessed)
def record(self, term, quality, origin):
if isinstance(term, JointCandidate):
if len(term.terms) != 1:
return
term = term.terms[0]
if term is None or term_has_holes(term):
return
if isinstance(quality, str):
return
score = quality[0] if isinstance(quality, (tuple, list)) and len(quality) == 1 else None
if score is not None and score <= self.best_score:
return
expression = pretty_print_sterm(lift(term))
if expression in self.seen:
return
self.seen.add(expression)
if score is not None:
self.best_score = score
emit(args, "candidate", expression=expression, reported_fitness=score, origin=origin)
def register(self, solution, quality, elapsed_time, is_best):
if is_best:
self.record(solution, quality, "register")
def register_front(self, front, elapsed_time):
for quality, term in front:
self.record(term, quality, "pareto_front")
def end(self, solution, quality):
self.record(solution, quality, "end")
def display_results(self, program, terms, synthesis_format=None):
for name, term in terms.items():
emit(args, "returned", hole=name.name, expression=pretty_print_sterm(term))
try:
driver = AeonDriver(AeonConfig(args.backend, CaptureUI(), args.timeout))
errors = list(driver.parse(filename=str(args.source)))
if errors:
raise RuntimeError("Compilation failed: " + "\n".join(map(str, errors)))
if not driver.has_synth():
raise RuntimeError("Benchmark has no synthesis hole")
emit(args, "compiled")
driver.synth()
emit(args, "finished")
except Exception as exc:
emit(args, "error", error_type=type(exc).__name__, error=str(exc))
traceback.print_exc()
return 1
return 0
def verify(args):
from loguru import logger
logger.remove()
from aeon.backend.evaluator import eval as evaluate
from aeon.core.terms import Var
from aeon.facade.driver import AeonConfig, AeonDriver
from aeon.synthesis.fitness_eval import set_program_tail
from aeon.synthesis.identification import iterate_top_level
from aeon.synthesis.uis.api import SilentSynthesisUI
source = args.source.read_text().replace("?bag", f"({args.expression})")
driver = AeonDriver(AeonConfig("smt", SilentSynthesisUI(), 0))
errors = list(driver.parse(aeon_code=source))
if errors:
raise ValueError("Candidate failed typechecking: " + "\n".join(map(str, errors)))
if driver.has_synth():
raise ValueError("Candidate still has holes")
names = {binding.var_name.name: binding.var_name for binding in iterate_top_level(driver.core)}
with driver.session.activate():
functions = {key: evaluate(set_program_tail(driver.core, Var(names[key])), driver.evaluation_ctx)
for key in ("chosen", "weight", "value")}
selection = functions["chosen"](None)
picks, node = [], selection
while isinstance(node, tuple) and len(node) == 3 and node[0] == "List_cons":
picks.append(node[1])
if len(picks) > 32:
raise ValueError("Selection exceeds 32 elements")
node = node[2]
if node != ("List_nil",):
raise ValueError(f"Expected stdlib List.nil tail, got {node!r}")
measurements = {key: functions[key](selection) for key in ("weight", "value")}
result = decoded(picks)
if not result["feasible"]:
raise ValueError(f"Capacity violated: {result}")
if any(type(measurements[key]) is not int or measurements[key] != result[key] for key in measurements):
raise ValueError(f"Aeon metrics disagree with independent oracle: {measurements} vs {result}")
print(json.dumps(result))
return 0
def stop_group(proc):
try:
os.killpg(proc.pid, signal.SIGKILL)
except ProcessLookupError:
pass
proc.wait()
def environment():
env = dict(os.environ)
env.update(AEON_SEED="0", PYTHONHASHSEED="0", PYTHONUNBUFFERED="1", OMP_NUM_THREADS="1",
OPENBLAS_NUM_THREADS="1", MKL_NUM_THREADS="1", AEON_OLLAMA_AUTO_PULL="0",
AEON_OLLAMA_RELEASE_AFTER="0", HF_HUB_OFFLINE="1", AEON_LLM_TIMEOUT="10")
return env
def run(args):
from aeon.synthesis.modules.synthesizerfactory import synthesizer_menu_ids, SYNTHESIZER_ALIASES
available_ids = synthesizer_menu_ids() + list(SYNTHESIZER_ALIASES)
excluded_ids = [b for b in available_ids if is_single_step(b)]
backend_ids = [b for b in (args.backends or available_ids) if not is_single_step(b)]
unknown = set(backend_ids) - set(available_ids)
if unknown:
raise ValueError(f"Unknown backends: {sorted(unknown)}")
print(f"Running {len(backend_ids)} IDs; excluding {len(excluded_ids)} single-step tactics.", flush=True)
args.run_dir.mkdir(parents=True, exist_ok=True)
env = environment()
oracle = oracle_solution()
print(f"Independent reference: value {oracle['value']}, weight {oracle['weight']} (not supplied to synthesis).", flush=True)
records = []
verification_cache = {}
script = str(Path(__file__).resolve())
for backend in backend_ids:
args.backend = backend
events_path = args.run_dir / f"{backend}.jsonl"
events_path.write_text("")
args.started = time.monotonic()
command = [sys.executable, script, "--worker", "--backend", backend, "--source", str(args.source),
"--run-dir", str(args.run_dir), "--timeout", str(args.timeout), "--started", str(args.started)]
with (args.run_dir / f"{backend}.log").open("w") as log:
proc = subprocess.Popen(command, stdout=log, stderr=subprocess.STDOUT, env=env, start_new_session=True)
timed_out = False
try:
proc.wait(timeout=max(0.001, args.timeout - (time.monotonic() - args.started)))
except subprocess.TimeoutExpired:
timed_out = True
finally:
stop_group(proc)
elapsed = time.monotonic() - args.started
events = []
for line in events_path.read_text().splitlines():
try:
event = json.loads(line)
except json.JSONDecodeError:
continue # SIGKILL may interrupt the last write.
if event["elapsed_s"] <= args.timeout:
events.append(event)
candidates = []
for event in events:
if event["event"] not in {"candidate", "returned"}:
continue
expression = event["expression"]
if expression not in verification_cache:
check_cmd = [sys.executable, script, "--verify", "--source", str(args.source), "--expression", expression]
try:
checked = subprocess.run(check_cmd, capture_output=True, text=True, env=env, timeout=args.verify_timeout)
verification_cache[expression] = (json.loads(checked.stdout) if checked.returncode == 0
else {"validation_error": checked.stderr[-2000:]})
except Exception as exc:
verification_cache[expression] = {"validation_error": str(exc)}
verified = verification_cache[expression]
candidates.append(event | verified)
good = [c for c in candidates if c.get("feasible") is True]
best = max(good, key=lambda c: (c["value"], -c["elapsed_s"])) if good else None
error = next((e for e in events if e["event"] == "error"), None)
if timed_out:
status = "timeout_with_incumbent" if best else "timeout"
elif error:
status = "error_with_incumbent" if best else "error"
else:
status = "completed" if best else "no_valid_solution"
record = dict(backend=backend, alias_of=SYNTHESIZER_ALIASES.get(backend), status=status,
wall_s=elapsed, exit_code=proc.returncode, error=error, best=best, candidates=candidates,
optimum=oracle["value"], gap=oracle["value"]-best["value"] if best else None,
compiled_s=next((e["elapsed_s"] for e in events if e["event"] == "compiled"), None),
search_started_s=next((e["elapsed_s"] for e in events if e["event"] == "search_started"), None),
progress=next((e for e in reversed(events) if e["event"] == "progress"), None))
records.append(record)
(args.run_dir / "results.json").write_text(json.dumps(records, indent=2) + "\n")
print(f"{backend}: {status}; value={best['value'] if best else '-'}; wall={elapsed:.2f}s", flush=True)
with (args.run_dir / "results.csv").open("w", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=["backend", "alias_of", "status", "picks", "weight", "value", "gap", "first_best_s", "wall_s", "expression", "error"])
writer.writeheader()
for r in records:
best = r["best"] or {}
writer.writerow(dict(backend=r["backend"], alias_of=r["alias_of"], status=r["status"],
picks=best.get("picks"), weight=best.get("weight"), value=best.get("value"),
gap=r["gap"], first_best_s=best.get("elapsed_s"), wall_s=r["wall_s"],
expression=best.get("expression"), error=(r["error"] or {}).get("error")))
metadata = dict(commit=subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip(),
date_utc=time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), python=sys.version,
platform=platform.platform(), cpu_count=os.cpu_count(), timeout_s=args.timeout,
seed=0, oracle=oracle, backend_ids=backend_ids, excluded_ids=excluded_ids,
source_sha256=hashlib.sha256(args.source.read_bytes()).hexdigest(),
weights=WEIGHTS, values=VALUES, capacity=CAPACITY,
verification_timeout_s=args.verify_timeout,
timeout_includes="process startup, imports, compilation and synthesis",
packages={p:importlib.metadata.version(p) for p in ["AeonLang", "z3-solver", "cvc5", "ortools", "scikit-learn", "numpy"]})
(args.run_dir / "environment.json").write_text(json.dumps(metadata, indent=2) + "\n")
return 0
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--timeout", type=int, default=10)
parser.add_argument("--verify-timeout", type=int, default=30)
parser.add_argument("--source", type=Path, default=Path(__file__).with_name("knapsack.ae"))
parser.add_argument("--run-dir", type=Path, default=Path(__file__).with_name("runs"))
parser.add_argument("--backends", nargs="+")
parser.add_argument("--backend")
parser.add_argument("--started", type=float, default=0)
parser.add_argument("--worker", action="store_true")
parser.add_argument("--verify", action="store_true")
parser.add_argument("--expression")
args = parser.parse_args()
args.source = args.source.resolve()
args.run_dir = args.run_dir.resolve()
return worker(args) if args.worker else verify(args) if args.verify else run(args)
if __name__ == "__main__":
raise SystemExit(main())
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment