12 minute lessonReady to learn

Specify an experiment before running it

Separate instance generation, algorithm repetitions, and data used for tuning.

Course lessons · 3 of 5

Your goal

Count experimental units correctly and identify the records needed for a reproducible protocol.

Useful background: Choose baselines that answer the question · Make an optimization run repeatable

01 · Understand the idea

An experiment has nested units

The recovered MIP study uses four network sizes, three construction families, and five seeds per size-family combination. Those factors produce sixty network instances. Repeating an optimizer on one instance does not create a new network instance.

Some repetitions run independent searches. Others draw repeated readout batches from the same trained circuit. Record which quantity varies at each level; this determines which results share data or fitted parameters.

Prespecify choices that affect the answer

Write down the generator, seed policy, objective, stopping budget, failure rules, comparison tolerance, and metrics before inspecting final outcomes. When selecting settings on development data, keep a separate evaluation set if the claim concerns performance on unseen cases.

A source and results package should retain the generated instances, per-run outcomes, environment, and analysis procedure. Summary charts are useful, but without their underlying records it is harder to investigate ties, failures, or a changed conclusion.

02 · Work through an example

Count the study’s instance design

1Cross size and family

Four sizes multiplied by three construction families gives twelve size-family combinations.

2Apply the seed count

Five seeds for each combination gives sixty independently specified instances in the recorded design.

4\cdot3\cdot5=60

3Keep optimizer repetitions separate

Ten recorded runs or readout batches per instance would create six hundred outcome records, but still sixty instances. Describe what each method’s repetition represents.

03 · Change something

Make the idea respond

Classify the workshop’s complete-small-test scenario and independent-repeat scenario, then identify which inputs, procedure, and outputs would make each inspectable. These hypothetical cases exercise the protocol reasoning in this lesson. The workshop does not generate new networks, run optimizer repetitions, or add experimental records to the archive.

Change one input at a time. Predict what should happen, then compare the computed result with your prediction.

Teaching model · computed in your browser

Read a claim like a researcher

Practice separating a proposed design, an inspected implementation, and evidence from an actual experiment.

Each scenario is hypothetical. Classify the evidence actually provided, then inspect the explanation and the next evidence needed. A correct answer here is practice feedback, not a scientific or professional certification.

Try this: identify what each scenario directly establishes before deciding whether it supports a broader capability claim.

Practice controls

The default worked example is shown below.

Scenario1 / 6

Reviewed with explanation0 / 6

Correct on first check0Practice feedback, not a certification

A promising architecture

A design note says: “The next version will combine persistent state, a graph, and a planning loop.” No implementation or experiment is attached. What does this support?

Choose the strongest category directly supported by this scenario Proposed design Inspected source or code Reported runtime claim Measured or computed test result Independent replication

These are hypothetical teaching scenarios. Choose a category and check your reasoning to reveal a specific explanation. A title, model name, equation, or moving counter alone cannot establish a scientific capability.

Read all six scenarios and their explanations

A promising architecture

A design note says: “The next version will combine persistent state, a graph, and a planning loop.” No implementation or experiment is attached. What does this support?

Proposed design. The note specifies an intended design. It establishes what someone proposes to build; it does not establish that the system exists or works. Stronger evidence would link the design to inspectable implementation and a clearly defined test.

A function in a repository

You inspect a versioned function that implements s_next = a × s + b × u. You have not executed it. What is directly established?

Inspected source or code. The source documents the expression and implementation at that revision. Execution, integration, and performance remain separate questions. A reproducible test can verify behavior for stated inputs; integration tests can then establish how the function is used.

A changing dashboard

A dashboard shows a rising “intelligence” counter. Its backend, metric definition, and stored observations have not been inspected. How should the displayed capability be described?

Reported runtime claim. The display is observable, but its interpretation is a reported claim. A changing number alone does not validate the capability named beside it. Look for the metric definition, the data path, repeatable inputs, and the records from which the number was calculated.

A complete small test

An experiment publishes its six-node graph, objective, every bipartition, and a script. Re-running the script gives the stated minimum on that graph. What is supported?

Measured or computed test result. This is a reproducible measured result for the stated graph and objective. It does not establish performance on larger graphs or a different MIP definition. Generalization requires additional cases, a defined sampling procedure, appropriate baselines, and clearly stated limitations.

Another team repeats the work

A separate team implements a published algorithm from its specification, repeats the stated experiment, and reports matching results with its own code and environment. What additional evidence is present?

Independent replication. An independent implementation and repeat study reduce reliance on one team’s code and environment. The conclusion still applies to the tested claim and conditions. Inspect methodological differences, uncertainty, failures, and the scope of agreement before extending the conclusion.

A score of one

The unity lab computes 1 / (1 + |1 − 1|) = 1. A student says this proves consciousness. Which category describes what the computation actually establishes?

Measured or computed test result. The computation establishes the value of a defined teaching function at a defined input. It does not validate consciousness, AGI, or a real machine state. To assess a scientific interpretation, define the observable claim and identify evidence that could support or contradict it.

Open the full lab

04 · Check your understanding

Your turn.

Try an answer before opening a hint. Feedback will explain the result, and you can practice again.

Interactive practice is loading. The lesson, worked example, and printable questions remain available below.

View printable practice and solutions

Use these questions when studying without interactive controls. Opening a written solution does not earn progress.

How many instances result from 4 sizes, 3 families, and 5 seeds per combination?
Read the solution
The design contains 60 instances.
Ten outcome records are collected for each of 60 instances. How many outcome records are there?
Read the solution
There are 600 outcome records.
Which action helps prevent evaluation data from silently becoming tuning data?
- Choose settings on development data and preserve a separate final evaluation set.
- Tune repeatedly on final outcomes and call them untouched.
- Delete disappointing evaluation records.
- Choose a metric only after finding the largest gain.
Read the solution
A separate evaluation set reduces direct reuse of the same outcomes for selecting and assessing settings.
Which artifact most helps investigate one surprising benchmark result?
- Only a rounded aggregate percentage.
- Only a promotional diagram.
- Only a count of repository files.
- The instance, run configuration, and per-run saved output.
Read the solution
Detailed run artifacts let another reader inspect the specific outcome.

05 · Apply it to the archive

Follow the original record

Use the definitions you just practiced to read the project’s own notation, assumptions, and evidence.

Exact partitions and measured search performance How to read the evidence

Take it with you

The ideas to keep

Count instances and repetitions separately.
Specify metrics and failure rules in advance.
Retain per-run artifacts and data roles.

Reading completion is separate from your practice evidence.

Progress is kept in this browser. How learning works