Working tools for the calibration sections of Jeramie General Intelligence: how to know whether a prediction was good, whether a hypothesis survived, whether an action caused an outcome, and whether a finding replicates.
What this lab is. The paper gives exact formulas in only two places here: Value of Information is in Section 6 (not covered here) and the Brier score in Section 56, Brier = mean((p-y)^2). Every other tool in this lab is an illustrative implementation of the paper's stated rule, and the page says so where it applies. Nothing here is presented as a validated result.
What this lab is not. It does not prove calibration, causality, or replication. It records forecasts, outcomes, and replications so that calibration can be measured instead of asserted.
Every meaningful proposition becomes an epistemic object.
Schema: claim_id, claim, subject, predicate, object, classification, source, source_ref, observed_at, valid_from, valid_until, freshness, confidence, authority, contradictions[], supports[], supersedes[], verification_state.
Classification: PROVEN, PROBABLE, HYPOTHESIS, CONFLICTED, UNKNOWN, FALSE, OBSOLETE.
A model-generated inference begins as HYPOTHESIS. It may only become PROVEN through appropriate evidence.
JGI should know what it does not know.
For important decisions estimate uncertainty. Represent, when practical, P(claim=true). Instead of "this is probably correct" store: claim probability, source reliability, evidence independence, freshness, contradiction count.
The system should actively select investigations that reduce uncertainty where doing so changes action.
JGI should generate hypotheses, not merely answers.
For substantial ambiguous problems, generate H1, H2, H3, and so on. For each: prior probability, supporting evidence, contradictory evidence, cheapest discriminating test, expected value if true.
Select experiments that best differentiate hypotheses. Avoid anchoring on the first explanation.
Before high-value actions ask: What happens if we do nothing? What happens if we choose A? What happens if we choose B? What would we expect to observe if our causal model is correct?
Record predicted differences. After execution compare predicted vs actual. This enables causal learning instead of story-telling.
For sufficiently material decisions simulate: BASE, UPSIDE, DOWNSIDE, ADVERSARIAL, TAIL-RISK.
Estimate: probability, value, time, dependencies, failure modes. Do not demand fake numerical precision. Use ranges when uncertainty is large.
Before significant missions record: P(success), expected value, time, cost, founder minutes, risk, expected observation.
After: actual outcome. Compare.
Without pre-action forecasts, calibration becomes hindsight storytelling.
Track Brier score for binary predictions.
Brier = mean((p-y)^2)
Track: forecast error, time error, cost error, revenue error. Models and agents that are poorly calibrated lose decision weight.
An outcome occurring after an action does not prove the action caused it.
Estimate: counterfactual, alternative causes, temporal relationship, mechanism, supporting evidence. Assign: ATTRIBUTION_CONFIDENCE.
Only strong attribution should change high-level policy.
Learning requires: ACTION, then OUTCOME.
Not: ACTION, then MODEL THINKS IT WAS GOOD.
If independent models strongly disagree: increase uncertainty. Do not simply majority-vote. Investigate why.
Measure: false claims, corrected claims, stale-state incidents, contradictions caught, Sentinel rejection rate.
Increasing intelligence without epistemic quality is unacceptable.
For novel system improvements:
OBSERVATION, QUESTION, HYPOTHESIS, PREDICTION, EXPERIMENT, RESULT, ANALYSIS, REPLICATION, PROMOTION
This is the preferred self-improvement pattern.
Important improvements should survive replication. A single successful run may be luck. Repeat under varied conditions where practical.
No material external action may rely solely on an unsupported model-generated factual premise.
Gather evidence first.
All tools run in your browser. Nothing is sent anywhere. Sample data is labelled illustrative.
Section 55 and Section 56. Add forecasts before the outcome is known, then record the outcome. Brier score is the paper's formula: mean((p-y)^2). Buckets are 0 to 0.2, 0.2 to 0.4, 0.4 to 0.6, 0.6 to 0.8, 0.8 to 1.0. A perfectly calibrated forecaster sits on the diagonal: in the 0.8 to 1.0 bucket, about 90 percent of forecasts should come true.
Enter a forecast to begin. The ledger is empty.
Section 15. Add hypotheses with a prior probability, supporting and contradictory evidence, and the cheapest discriminating test. Record the outcome for each. The paper gives no scoring formula for hypotheses, so the tally below is a plain count, not a score.
No hypotheses yet.
Section 16. Record the choice taken and its actual outcome, plus the alternative choice and its estimated outcome. The difference is alternative minus taken. A positive difference means the alternative would have been better. Estimated outcomes for untaken paths are estimates, not measurements, and the page treats them as such.
Run the comparison to see the difference.
Section 57. Enter the outcome, then candidate actions. Each action gets an evidence strength from 0 to 10, plus whether a mechanism is known and whether the action happened before the outcome. The ranking formula is an illustrative implementation of the paper's concept, not a formula from the paper: score = evidence strength, times 1.5 if a mechanism is known, times 1.2 if the action preceded the outcome. Share is the action's score divided by the total of all scores.
Add candidate actions to rank them.
Section 131, with the epistemic rules of Sections 4 and 5. A claim may be treated as fact for a material external action only when every check passes. Bands are an illustrative implementation of the paper's rule: 6 of 6 is CLEAR, 4 to 5 is PROCEED WITH EVIDENCE, 0 to 3 is BLOCKED.
0 of 6 checks passed.
Sections 113 and 114. Track replications of a finding. A single successful run may be luck. Verdict bands are an illustrative implementation of the paper's rule: fewer than 3 attempts is insufficient, 80 percent or more is Replicated, 50 to 79 percent is Mixed, below 50 percent is Failed to replicate.
Built from Jeramie General Intelligence, JGI V6. Sections 4, 5, 15, 16, 17, 55, 56, 57, 58, 95, 111, 113, 114, 131 are reproduced above in full, in order. Where the paper gives no formula, the tool is labelled an illustrative implementation.