下载此 Notebook

分层模型教程#

本教程演示如何使用 GluonTS 基于深度学习的分层模型 DeepVarHierarchical。我们首先解释分层/分组时间序列的数据准备，然后展示使用常见用例的模型训练、预测和评估。

简介#

使用分层模型的重要方面是分层时间序列数据的正确准备。构建分层或分组时间序列的最低要求是层次结构的底层时间序列和分层/分组聚合矩阵。GluonTS 提供了一种简单的方法，通过从 csv 文件读取底层时间序列和聚合矩阵来构建分层时间序列。在这里，我们首先描述分层时间序列的准备工作，然后再讨论分层模型的训练。

分层时间序列的准备#

底层时间序列假设以 csv 文件中的列形式提供。csv 文件还应包含一个索引列，列出每行的时间戳。请注意，聚合时间序列会自动构建，无需提供。这是一个底层时间序列的示例 csv。类似地，聚合矩阵也可以从 csv 文件中读取；这里有一个示例。我们使用求和矩阵的标准格式；例如，参见教科书Forecasting: Principles and Practice中的第 10.3 节。请注意，分组时间序列也可以用与分层时间序列相同的方式由聚合矩阵表示，因此此处介绍的内容也适用于分组时间序列。

[1]:

import pandas as pd
from gluonts.dataset.hierarchical import HierarchicalTimeSeries

# Load (only!) the time series at the bottom level of the hierarchy.
ts_at_bottom_level_csv = (
    "https://gist.githubusercontent.com/rshyamsundar/39e57075743537c4100a716a7b7ec047/"
    "raw/f02f5aeadad73e3f3e9cf54478606d3507826492/example_bottom_ts.csv"
)

# Make sure the dataframe has `PeriodIndex` by explicitly casting it to `PeriodIndex`.
ts_at_bottom_level = pd.read_csv(
    ts_at_bottom_level_csv,
    index_col=0,
    parse_dates=True,
).to_period()

# Load the aggregation matrix `S`.
S_csv = (
    "https://gist.githubusercontent.com/rshyamsundar/17084fd1f28021867bcf6f2d69d9b73a/raw/"
    "32780ca43f57a78f2d521a75e73b136b17f34a02/example_agg_mat.csv"
)
S = pd.read_csv(S_csv).values

hts = HierarchicalTimeSeries(
    ts_at_bottom_level=ts_at_bottom_level,
    S=S,
)

可以使用 ts_at_all_levels 属性访问层次结构中的所有时间序列。

[2]:

hts.ts_at_all_levels.head()

[2]:
0 1 2 3 4 5 6
2020-03-22 00:00 0.686671 0.156873 0.529798 0.056962 0.099911 0.039827 0.489971
2020-03-22 01:00 2.189128 0.669261 1.519866 0.246535 0.422727 0.763164 0.756702
2020-03-22 02:00 1.152853 0.582213 0.570640 0.314393 0.267820 0.169645 0.400996
2020-03-22 03:00 1.198889 0.653139 0.545750 0.609158 0.043981 0.235009 0.310741
2020-03-22 04:00 2.069197 0.678490 1.390707 0.380788 0.297702 0.898429 0.492278

模型训练和预测#

现在我们展示最简单的用例，即在整个可用数据集上训练模型并生成未来时间步长的预测。请注意，一旦根据用户特定的评估标准选择了最佳模型，这便是模型在实践中的使用方式；请参见下一节的模型评估。

我们假设类型为 HierarchicalTimeSeries 的分层时间序列 hts 已经按上述方式构建。第一步是将此分层时间序列转换为可以在其上运行 mini-batch 训练的 Dataset。在这里，我们将其转换为 gluonts.dataset.pandas.PandasDataset。

[3]:

dataset = hts.to_dataset()

数据集创建后，使用分层模型就很简单了。在这里，为了快速演示，我们固定预测长度并选择较少的 epoch 数。我们在整个数据集上进行训练，并将相同的数据作为输入提供给训练好的模型（称为预测器）以生成未来/未见时间步长的预测（或预测）。最终输出 forecasts 是 gluonts.model.forecast.SampleForecast 的一个实例，包含层次结构中所有时间序列的基于样本的预测。

[4]:

from gluonts.mx.model.deepvar_hierarchical import DeepVARHierarchicalEstimator
from gluonts.mx.trainer import Trainer

prediction_length = 24
estimator = DeepVARHierarchicalEstimator(
    freq=hts.freq,
    prediction_length=prediction_length,
    trainer=Trainer(epochs=2),
    S=S,
)
predictor = estimator.train(dataset)

forecast_it = predictor.predict(dataset)

# There is only one element in `forecast_it` containing forecasts for all the time series in the hierarchy.
forecasts = next(forecast_it)

100%|██████████| 50/50 [00:32<00:00,  1.53it/s, epoch=1/2, avg_epoch_loss=178]
100%|██████████| 50/50 [00:32<00:00,  1.53it/s, epoch=2/2, avg_epoch_loss=146]

使用外部动态特征#

默认情况下，分层模型 DeepVarHierarchical 内部为模型训练创建了几个基于时间/动态特征。这些是从目标时间序列的频率自动推导出的季节性特征。如果可用，也可以向模型提供外部动态特征。在这里，我们展示如何实现这一点；我们从已经创建了分层时间序列 hts 的位置重新开始。

我们首先从 csv 文件加载可用的外部特征。

[5]:

dynamic_features_csv = (
    "https://gist.githubusercontent.com/rshyamsundar/d8e63bad43397c95a4f5daaa17e122f8/"
    "raw/a50657cf89f86d48cee41122f02cf5b1fcafdd2f/example_dynamic_features.csv"
)

dynamic_features_df = pd.read_csv(
    dynamic_features_csv,
    index_col=0,
    parse_dates=True,
).to_period()

动态特征假设同时可用于“训练范围”（目标可用的时间点）和“预测范围”（需要预测的未来时间点）。因此，动态特征比目标时间序列长 prediction_length 个时间步长。

为了训练模型，我们只需要训练范围内的动态特征。

[6]:

dynamic_features_df_train = dynamic_features_df.iloc[:-prediction_length, :]

我们通过传递外部特征 dynamic_features_df_train 创建训练数据集，并在其上训练模型。

[7]:

dataset_train = hts.to_dataset(feat_dynamic_real=dynamic_features_df_train)
estimator = DeepVARHierarchicalEstimator(
    freq=hts.freq,
    prediction_length=prediction_length,
    trainer=Trainer(epochs=2),
    S=S,
)
predictor = estimator.train(dataset_train)

100%|██████████| 50/50 [00:33<00:00,  1.48it/s, epoch=1/2, avg_epoch_loss=192]
100%|██████████| 50/50 [00:33<00:00,  1.49it/s, epoch=2/2, avg_epoch_loss=147]

要生成未来/未见时间步长的预测，我们需要将过去的目标（即训练范围内的目标）以及完整的特征 dynamic_features_df（包括预测范围内的特征）都传递给训练好的模型。因此，我们需要创建一个与 dataset_train 分开的新数据集，这与前面的情况不同。

[8]:

predictor_input = hts.to_dataset(feat_dynamic_real=dynamic_features_df)
forecast_it = predictor.predict(predictor_input)

# There is only one element in `forecast_it` containing forecasts for all the time series in the hierarchy.
forecasts = next(forecast_it)

通过回测进行模型评估#

现在我们解释如何通过回测来评估分层模型。为了便于说明，我们描述了存在外部动态特征的情况下的模型评估。但是，在没有外部特征的情况下，修改代码也很简单；只需调用 to_dataset() 函数而无需任何参数即可。

我们假设底层时间序列 ts_at_bottom_level 和聚合矩阵 S 已按上述方式创建。我们通过保留最后 prediction_length 个时间点用于评估，其余时间点用于训练，沿时间轴创建训练集和测试集拆分。

[9]:

prediction_length = 24
hts_train = HierarchicalTimeSeries(
    ts_at_bottom_level=ts_at_bottom_level.iloc[:-prediction_length, :],
    S=S,
)
hts_test_label = HierarchicalTimeSeries(
    ts_at_bottom_level=ts_at_bottom_level.iloc[-prediction_length:, :],
    S=S,
)

我们加载外部特征并切分与训练范围相对应的特征。

[10]:

dynamic_features_csv = (
    "https://gist.githubusercontent.com/rshyamsundar/d8e63bad43397c95a4f5daaa17e122f8/"
    "raw/a50657cf89f86d48cee41122f02cf5b1fcafdd2f/example_dynamic_features.csv"
)

dynamic_features_df = pd.read_csv(
    dynamic_features_csv,
    index_col=0,
    parse_dates=True,
).to_period()

dynamic_features_df_train = dynamic_features_df.iloc[:-prediction_length, :]

我们将 hts_train 通过传递外部特征 dynamic_features_df_train 转换为 PandasDataset，并在其上训练分层模型。

[11]:

dataset_train = hts_train.to_dataset(feat_dynamic_real=dynamic_features_df_train)

estimator = DeepVARHierarchicalEstimator(
    freq=hts.freq,
    prediction_length=prediction_length,
    trainer=Trainer(epochs=2),
    S=S,
)

predictor = estimator.train(dataset_train)

100%|██████████| 50/50 [00:33<00:00,  1.48it/s, epoch=1/2, avg_epoch_loss=176]
100%|██████████| 50/50 [00:33<00:00,  1.49it/s, epoch=2/2, avg_epoch_loss=144]

要为保留观测值对应的时间点生成预测，我们需要将完整的特征以及训练范围内的目标都传递给训练好的模型。因此，我们相应地为预测器创建输入数据集并生成预测。

[12]:

predictor_input = hts_train.to_dataset(feat_dynamic_real=dynamic_features_df)
forecast_it = predictor.predict(predictor_input)

获得预测后，我们可以根据保留的观测值对其进行评估。GluonTS 评估器接收真实（保留）观测值和相应预测值的迭代器作为输入进行评估。我们的预测已经采用正确的格式，并且我们的保留观测值存储在 hts_test_label 中。

[13]:

from gluonts.evaluation import MultivariateEvaluator

evaluator = MultivariateEvaluator()
agg_metrics, item_metrics = evaluator(
    ts_iterator=[hts_test_label.ts_at_all_levels],
    fcst_iterator=forecast_it,
)

print(
    f"Mean (weighted) quantile loss over all time series: "
    f"{agg_metrics['mean_wQuantileLoss']}"
)

Running evaluation: 1it [00:00, 145.95it/s]
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
Running evaluation: 1it [00:00, 154.48it/s]
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
Running evaluation: 1it [00:00, 156.65it/s]
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
Running evaluation: 1it [00:00, 157.82it/s]
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
Running evaluation: 1it [00:00, 163.84it/s]
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
Running evaluation: 1it [00:00, 140.81it/s]
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
Running evaluation: 1it [00:00, 164.84it/s]

Mean (weighted) quantile loss over all time series: 0.27951188855488635

/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
  return arr.astype(dtype, copy=True)