AI×ML Framework

Every machine-learning trading framework begins by lying to itself about how hard the problem is. The lie usually sounds reasonable — markets have patterns, neural networks find patterns, therefore neural networks find market patterns. The honest version is that markets are adaptive, non-stationary, and adversarial, and any model good enough to extract a real edge is good enough to be arbitraged away by the next model that finds it.

What makes BTC/USDT specifically interesting — and specifically difficult — is the collision of three regimes. It trades like a tech equity during risk-on rallies, like gold during currency stress, and like nothing in particular during liquidation cascades. A model trained on 2017–2018 learns one BTC. A model trained on 2020–2021 learns a different one. A model trained on 2022–2024 learns a third. The honest question is not what predicts BTC but what predicts BTC during the regime we are about to enter, which is unknowable.

The framework below treats prediction not as an oracle problem but as a decision problem under uncertainty. We are not trying to forecast tomorrow's price; we are trying to assemble a system that, on average, sizes correctly, exits early, and survives the runs where it is wrong. The model is one component. Validation discipline, position sizing, execution latency, drawdown control, and the willingness to retire a degraded edge are each at least as important as the model itself.

The Four Failure Modes

1. OverfittingFitting noise that will not repeat. The default outcome of any sufficiently flexible model on a finite history. Section IX is the cure.
2. Look-ahead biasUsing information at training time that would not have been available at decision time. Subtle: a "previous close" feature joined on the wrong calendar can leak the future.
3. Regime changeThe market that produced the training data no longer exists. Crypto experiences this at least every 18 months.
4. Cost denialA 0.04 % per-trade fee plus 1–5 bps of slippage eats most "high-frequency" edges before they leave the notebook.

The unfair advantage in quantitative finance does not live in the model. It lives in the data — in having more of it, cleaner versions of it, faster access to it, and the right kind of it. Spending three months tuning XGBoost on a dataset full of survivorship bias and unaligned timestamps is a more popular activity than spending three months getting the data right, but the second project produces strategies that work and the first does not.

Sources

Frequencies & Horizons

The choice of bar interval defines the strategy more than any single hyperparameter. A 1-minute model is a different beast — different features, different costs, different infrastructure — from a 4-hour model, even if the algorithm code is identical.

Information Bars (López de Prado)

Time bars are a calendar artefact, not a market property. Volume, dollar, and imbalance bars sample by activity rather than by clock and produce returns that are closer to i.i.d. — which matters because most ML assumes it. A dollar bar that closes every \$50M traded breathes with the market: it slows down on quiet Sundays and speeds up during liquidation events, which is exactly the inverse of what time bars do.

Quality Problems You Will Have

1. Timestamps misalignedExchange clocks drift. UTC versus exchange-local time confusions kill more strategies than bad models.
2. Survivorship in token universesIf you're modelling alts as well as BTC, your basket of "top 50 coins" must be the top 50 as of that date, not today's top 50.
3. Lookahead via revised dataOn-chain metrics get revised. The MVRV you query today for last Wednesday is not the MVRV that was available last Wednesday. Use point-in-time data or accept the bias.
4. Halt and outage gapsBinance went down for an hour during the LUNA collapse. Your model needs to know that — either fill, or flag and refuse to trade.
5. Wash tradingSmaller venues print volume that did not happen. Confirm OHLCV against multiple exchanges or stick to top-tier venues.

Storage

For anything beyond a notebook prototype: store raw data once, store engineered features in a parquet-based warehouse, and version both. pandas + parquet works to about a year of 1m bars. Beyond that, look at DuckDB, arctic, or ClickHouse. Pickled DataFrames are a footgun — they bind to library versions.

Feature engineering is where domain expertise enters the model. A textbook ML practitioner trying their hand at crypto will typically build 500 features, none of them informed by how markets actually move, then complain that random forests don't work. The features below are the ones that consistently survive the validation gauntlet in practitioner research — not because they are magic, but because they encode something real about market microstructure or human behaviour.

Returns & Volatility

Momentum & Mean Reversion

The classical technicals — RSI, MACD, Bollinger Z-score, donchian width — are not magic, but they are compressions of price action that often correlate with whatever real edge exists. Use them as inputs to a model, not as standalone signals. The model decides when each one is informative.

Microstructure & Order Flow

1. Cumulative Volume Delta (CVD)Signed volume — buys minus sells classified by trade side. Divergence from price often precedes reversals.
2. Trade size distributionWhale prints versus retail. The 95th percentile of trade notional is a different feature from the median.
3. Order book imbalance(Bid depth − Ask depth) / total at top N levels. Short-horizon directional signal.
4. Book slopeHow quickly liquidity thins moving away from mid. Predicts slippage and breakout fragility.
5. Funding ratePerpetual swap funding. Persistently positive funding = crowded longs = fade candidate at extremes.
6. Open interest deltaRising OI with rising price = new longs. Rising OI with falling price = new shorts. Different conviction profiles.

Cross-Asset & Regime

BTC does not trade in isolation. Useful inputs from outside the BTC tape:

Fractional Differentiation

The standard fix for non-stationary price series is to take first differences (returns). The cost is that all memory is destroyed — long-horizon levels become invisible. Fractional differentiation, popularised by López de Prado, takes a non-integer derivative that preserves as much memory as possible while passing a stationarity test. For BTC, fractional orders around 0.3–0.5 typically work.

Labelling — The Triple Barrier Method

Most beginners label as y = sign(return_t+k), which produces noisy targets dominated by drift. The triple-barrier method instead labels each observation by which of three events fires first: a profit-taking horizontal barrier, a stop-loss horizontal barrier, or a vertical time barrier. Labels become trading-decision-aligned, and the path matters, not just the endpoint.

# Triple-barrier sketch (Lopez de Prado, AFML ch.3)
def apply_triple_barrier(price, events, pt_mult, sl_mult, vol):
    out = events[['t1']].copy()
    for loc, t1 in events['t1'].items():
        df = price[loc:t1] / price[loc] - 1
        out.loc[loc, 'sl'] = df[df < -sl_mult * vol[loc]].index.min()
        out.loc[loc, 'pt'] = df[df >  pt_mult * vol[loc]].index.min()
    return out  # first of {pt, sl, t1} per row defines the label

Sample Weighting

Adjacent labels share overlapping information (the future-looking window of bar t contains bars t+1 … t+k). Training on them naively over-weights clustered observations. The fix is to weight each sample by the inverse of how many other samples its information overlaps with — again, López de Prado has the canonical recipe.

Before reaching for transformers, exhaust the classical toolkit. Tree ensembles and well-regularised linear models are the workhorses of practitioner quant finance for reasons that have nothing to do with fashion: they are fast to train, robust to noisy features, interpretable, and difficult to overfit catastrophically. A gradient-boosted decision tree on 80 well-engineered features will beat a poorly-tuned LSTM in a walk-forward eight times out of ten.

The Roster

The Honest Default

Start with LightGBM. Engineer 30–100 features. Predict either (a) the sign of next-bar return classified by triple barrier, or (b) the volatility-normalised return. Use purged time-series cross-validation. Inspect feature_importance with shap rather than gini — gini lies on correlated features. Retrain weekly or monthly. If LightGBM cannot find a stable edge, neither can a transformer.

import lightgbm as lgb
from sklearn.model_selection import TimeSeriesSplit

params = {
    'objective': 'binary',
    'metric': 'auc',
    'learning_rate': 0.02,
    'num_leaves': 31,
    'feature_fraction': 0.8,
    'bagging_fraction': 0.8,
    'min_data_in_leaf': 200,   # crucial for noisy targets
    'lambda_l2': 0.1,
}
# Walk-forward, NEVER plain k-fold. Purge + embargo around test windows.

Deep learning earns its place when the input has structure that hand-crafted features cannot easily express: long sequences, irregular spacing, multi-modal inputs, or non-linear interactions across hundreds of features. For BTC at hourly and above, a well-engineered LightGBM usually matches a deep model. For high-frequency order-book prediction, sub-minute regime classification, and multi-asset cross-sectional work, deep models pull ahead.

The Architectures That Matter

Inputs & Outputs

A deep model for BTC is rarely a single number in, single number out. The shape you will most often build:

# A multi-horizon, multi-input sequence model
input_features:  [batch, seq_len=240, n_features=48]   # 240 hours of 48 features
static_features: [batch, n_static=12]                  # asset metadata, regime flags

# Output options:
direction:   [batch, n_horizons=4]    # P(up) at +1h, +4h, +24h, +96h
quantiles:   [batch, n_horizons, 7]   # 7 quantile forecasts per horizon
return_mean: [batch, n_horizons]      # point forecast of returns
return_std:  [batch, n_horizons]      # predicted volatility

Training Discipline

1. Standardise features by training-window statistics onlyComputing the mean/std on the full series is the most common lookahead bug.
2. Early stop on a held-out walk-forward windowNot on a random validation split — that lies for time series.
3. Use dropout, but on the input as well as hidden layersInput dropout corrupts the feature mix the model sees, which approximates a regime-shift augmentation.
4. Mix-up across timeLinear interpolation of nearby training samples regularises sequence models more than people expect.
5. Loss matching the targetUse quantile loss for distributional forecasts; use focal loss when classes are imbalanced (which they are during ranging regimes).
6. Predict deltas, not levelsAlways. Levels look impressive on a plot and are useless to trade.

What a Modern Stack Looks Like

The current best practice for BTC deep-learning research, circa 2026:

Supervised learning predicts the future. Reinforcement learning chooses actions. The distinction matters because in trading, the question is rarely "what will price do" and almost always "what should I do next, knowing what I currently hold and what it will cost me to change that." RL frames the problem natively as a sequential decision process and learns a policy end-to-end — including position sizing, holding periods, and tactical exits.

The MDP

1. StateCurrent features (price history, indicators, regime), current position, current unrealised P&L, time in trade, recent reward.
2. ActionDiscrete: {-1, 0, +1} for short/flat/long. Or continuous: target leverage in [-L, +L]. Continuous spaces are richer but harder to learn.
3. RewardThe deeply contested choice. Naïve: P&L per step. Better: differential Sharpe, log-utility return, or risk-adjusted return with explicit cost penalty.
4. TransitionThe market — non-stationary, partially observed, and not yours to define.

Algorithm Choices

Reward Design Is Everything

The single largest determinant of whether an RL agent develops a sane trading policy is the reward function. A few patterns that consistently perform better than naïve P&L:

# Differential Sharpe Ratio (Moody & Saffell, 2001) - online Sharpe approximation
def differential_sharpe(r_t, A_prev, B_prev, eta=0.01):
    delta_A = r_t - A_prev
    delta_B = r_t**2 - B_prev
    A = A_prev + eta * delta_A          # EMA of returns
    B = B_prev + eta * delta_B          # EMA of squared returns
    denom = (B_prev - A_prev**2)**1.5
    dsr   = (B_prev * delta_A - 0.5*A_prev*delta_B) / denom
    return dsr, A, B
# Reward = dsr - cost_penalty * |action_change|

Environments

Off-the-shelf options worth knowing: FinRL (the AI4Finance lab), gym-anytrading, TensorTrade. None will give you production-ready alpha out of the box — they exist to remove the boilerplate of building an OpenAI-Gym-compatible market environment so you can focus on the parts that matter (features, reward, action design).

Equities have decades of fundamental data — earnings, balance sheets, analyst coverage. Crypto does not. What it has instead is an unusually transparent ledger and an unusually loud social layer. Both are exploitable — and both are the two feature families most often misused by retail-level practitioners, who treat raw on-chain charts as signals rather than as inputs to a model that has to decide when each one is informative.

On-Chain Signals That Matter

Derivatives Microstructure

1. Funding ratePersistently positive funding indicates a crowded long book. Extremes — say, > 0.05 % every 8 h sustained — historically precede long squeezes.
2. Open interestRising OI confirms a trend; falling OI suggests a move is unwinding.
3. Perp-spot basisThe premium of perpetual swap vs spot. Wide premiums = bullish positioning; deep discounts = capitulation or arbitrage opportunity.
4. Liquidation mapsCumulative leverage thresholds where forced selling/buying triggers. Price often hunts these zones.
5. Options skew & term structure25-delta put skew, vol surface curvature, calendar spreads. Hedging pressure leaves footprints.

Sentiment via LLMs (the 2024+ shift)

Before 2023, sentiment analysis in crypto meant FinBERT fine-tunes on tweet datasets and lexicon-based scoring. Both worked poorly: crypto Twitter speaks in irony, memes, and ticker abbreviations that drift weekly. The current state of the art is to use a frontier LLM (Claude, GPT-class, or a local Llama variant) as a feature extractor: feed it a window of posts and ask for structured outputs — bullish/bearish scores, conviction level, narrative tags, and influencer-weighted aggregates.

# Pseudo-pipeline. Run hourly. Cache aggressively.
for hour in recent_hours:
    posts   = fetch_posts(hour, sources=['twitter', 'reddit', 'telegram_pub'])
    posts   = filter_by_engagement(posts, min_views=500)
    chunks  = batch(posts, n=50)
    results = [llm.extract_sentiment(chunk) for chunk in chunks]
    features[hour] = aggregate(results, weights=influencer_score)

The catch: LLM inference is expensive at scale and rate-limited at the API. Most serious practitioners use a tiered system — a cheap classifier filters posts, an LLM scores the survivors, and aggregated scores are cached.

The Fear & Greed Index

The widely-cited Alternative.me index is a composite of volatility, momentum, social media, surveys, dominance, and Google trends. It is a useful indicator as a feature, not as a signal. Buying when fear < 20 and selling when greed > 80 has worked as a contrarian heuristic over long horizons but has substantial drawdowns and embeds severe lookahead if used naively in backtests.

Every model in this framework is wrong in a different way. Tree ensembles miss long-range dependencies. LSTMs hallucinate trends. RL agents overfit to their reward function. Sentiment models lag price. The practical response is not to find the right model — it does not exist — but to combine several so that their idiosyncratic errors partially cancel and the parts each one gets right are weighted appropriately.

Ensemble Hierarchy

1. BaggingSame algorithm, different data subsets. Random forest is bagged trees. Cheap, robust.
2. BoostingSequential — each model fixes errors of the previous. XGBoost, LightGBM. Already an ensemble.
3. StackingTrain K diverse models. Train a meta-learner on their out-of-fold predictions. The meta-learner decides who to trust when.
4. BlendingStacking, but with a held-out blend set rather than out-of-fold predictions. Simpler, slightly less data-efficient.
5. Bayesian Model AveragingWeight models by their posterior probability of being correct. Theoretically elegant; rarely beats stacking in practice.
6. Mixture of ExpertsLearn a gating network that routes each input to the best specialist. Strong in regime-changing markets.
7. Online ensemble selectionWeight models by their recent walk-forward performance. Drops degraded models; promotes recovering ones.

A Working Composition

The shape of a production-ready BTC ML stack often looks like:

The meta-learner gets the base predictions plus regime probabilities as inputs, so it learns to trust LightGBM during trends and the TFT during regime transitions. This is not magic — it is a small, well-validated linear model whose entire job is to apologise for each base learner's blind spots.

This is the section that decides whether a research project will lose money in live. Validation is to ML trading what controlled trials are to medicine: tedious, expensive, often unwelcome, and the only thing standing between a plausible-sounding hypothesis and a costly mistake. Every method below exists because a previous generation of quants discovered, painfully, that a simpler method lied to them.

Why Random K-Fold Lies

Shuffling time-series data and splitting into folds destroys the autocorrelation structure that you are trying to exploit, but it also leaks information from the future into the past. A model evaluated by random k-fold on price data routinely shows AUC of 0.85 when its real out-of-sample AUC is 0.52. The single most common cause of "my backtest looked amazing but live lost money" is random shuffling somewhere upstream.

The Validation Ladder

Beyond Single Numbers

A single Sharpe ratio from a single backtest is a number with almost no statistical content. The real questions are:

1. How many strategies did you test before this one?If you tried 50 variants and reported the best, the Deflated Sharpe Ratio penalises that and often reveals it was noise.
2. What is the probability this strategy is overfit?The Probability of Backtest Overfitting (PBO) measures, across CPCV paths, how often the in-sample-best strategy is actually one of the worst out-of-sample.
3. Does the strategy survive bootstrap resampling?Block-bootstrap the trade returns. If the Sharpe distribution straddles zero, your edge is statistical noise.
4. Does the strategy survive a Monte Carlo permutation test?Shuffle the returns within blocks; recompute the strategy. If the original Sharpe is not in the top 5% of the permutation distribution, you have nothing.
5. Reality Check / SPA testsWhite's Reality Check and Hansen's SPA correct for multiple-testing across competing strategies. Use when comparing model families.

Deflated Sharpe Ratio

The deflated Sharpe ratio (Bailey & López de Prado, 2014) adjusts the observed Sharpe for the number of trials you ran, the variance of returns, skewness, and kurtosis. It produces a probability that the observed Sharpe is genuinely above zero given everything you tried. For a researcher who tested 100 variants and reports a Sharpe of 1.8 on the best one, the DSR is often below 0.5 — which means the result is not statistically significant.

# DSR sketch — Bailey & Lopez de Prado, 2014
# Inputs: observed Sharpe SR_obs, n_trials, sample length T,
#         return skew gamma_3, kurtosis gamma_4
SR_expected_max = 0.0 + sigma_SR * (
    (1 - GAMMA) * inv_norm(1 - 1/n_trials)
    + GAMMA * inv_norm(1 - (1/n_trials)*exp(-1))
)
denom = sqrt(1 - gamma_3*SR_obs + (gamma_4 - 1)/4 * SR_obs**2)
DSR  = norm_cdf((SR_obs - SR_expected_max) * sqrt(T-1) / denom)
# Interpretation: probability that true SR > 0 given the trials you ran.

Walk-Forward in Practice

1. Trainon data from t₀ to t₀ + W
2. Embargoa period (t₀+W, t₀+W+E) — discard, prevents leakage
3. Teston (t₀+W+E, t₀+W+E+H) — out-of-sample window
4. Advancet₀ by H (or by some smaller step for finer-grained metrics) and repeat
5. Aggregateconcatenate all test-window predictions; compute Sharpe, max drawdown, hit rate once on the union

An average ML model with disciplined sizing makes money. A brilliant ML model with greedy sizing goes bust. The math is asymmetric: a 50% drawdown requires a 100% recovery, and a 90% drawdown requires a 900% recovery. Most quant blow-ups are not modelling failures — they are sizing failures.

Sizing Methods

The Kelly Math

# Continuous Kelly for a strategy with edge mu and variance sigma^2:
f_star = mu / sigma**2

# With proportional cost c per unit traded:
f_star_costed = (mu - c) / sigma**2

# In practice: half-Kelly (f_star / 2) cuts geometric growth by ~25%
# but reduces drawdown variance by ~75%. Almost always worth it.

Drawdown Control Mechanisms

1. Equity-curve filteringIf your strategy's recent rolling Sharpe drops below zero, reduce size or pause. The simplest "model" of model degradation.
2. Volatility regime cutsCap exposure when realised vol exceeds a threshold. Markets in crisis correlate to one — diversification fails when you need it.
3. Time stopsExit a position after N bars regardless of P&L. Prevents the slow-bleed scenario where a thesis is "still valid" while capital decays.
4. Maximum concurrent riskIf you run multiple sub-strategies, cap their summed gross exposure. Correlations spike in crashes.
5. Daily loss limitsHard stop at X% intraday loss. Resume tomorrow. Removes the worst variance from your monthly distribution.

A strategy that lives in a notebook is not a strategy — it is a research artefact. Moving from research to live is where most projects discover that 80% of the work is the part the academic papers do not cover: the infrastructure that pulls features in real time, serves predictions with millisecond consistency, monitors for drift, and rolls back gracefully when something goes wrong.

The Production Stack

The Latency Budget

For an hourly-bar strategy, latency is mostly irrelevant — you can wait two seconds for a model to score. For a minute-bar strategy, you have perhaps 200ms total budget from bar close to order on the exchange. For sub-second strategies, you need colocation, kernel-bypass networking, and pre-compiled prediction pipelines. Most retail-scale BTC strategies operate at the 1m–1h timeframe where this is not a binding constraint.

Drift Detection

Models degrade. The question is whether you detect it before the loss is large enough to matter.

1. Feature driftTest if today's feature distributions differ from training. Kolmogorov-Smirnov, PSI (Population Stability Index), Jensen-Shannon divergence.
2. Prediction driftDistribution of model outputs over time. A model that suddenly predicts "up" 90% of the time when it used to be balanced is broken.
3. Performance driftRolling Sharpe, rolling hit rate. The lagging but ultimate measure.
4. Concept driftThe relationship between features and target has changed. Detect via rolling correlation of predicted vs realised, or by retraining on recent windows and comparing.

Deployment Discipline

1. Paper-trade firstRun live for at least one full market cycle (a month minimum) before risking capital. Compare paper P&L to backtest expectation.
2. Ramp capital slowlyStart at 5% of intended size. Double weekly if metrics hold. Full size only after 30+ trades in line with expectation.
3. Kill switchesManual and automatic. An external dashboard that can flatten everything in one click. Daily loss limits that trigger automatic stand-down.
4. Audit every fillCompare expected vs realised slippage. If realised consistently exceeds expected, your cost model is wrong and your strategy is smaller than you thought.
5. Retrain cadenceDefined in advance, not panicked. Weekly, monthly, or trigger-based on drift alarms. Pre-commit to the schedule.

If you came here looking for confirmation that AI will predict BTC for you — this section won't. But it's the section a serious practitioner has internalised before they spend a year on the rest of the framework.

What This Means

None of the above means do not try. It means that the realistic ambition for an individual practitioner is not "build an oracle that prints money" but rather: build a system whose risk-adjusted returns marginally exceed a vol-targeted buy-and-hold of BTC, after honest costs, over a multi-year horizon, with drawdowns you can stomach. That is a real, hard, valuable engineering problem. It is not the problem most people say they are working on, but it is the one worth working on.

The reading order below moves from foundations to specialisation. None of them will make you money on its own. Together they build the worldview a serious practitioner needs.

Foundations

Finance ML

Microstructure & Execution

Probability & Risk

Papers — Start Here

Python Libraries

Data Vendors

Communities & Ongoing