6. Post-estimation diagnostics#

The class ModifiedCausalForest provides you with several diagnostic tools to analyse the estimated \(\text{IATE's}\). They cover

descriptive statistics
a correlation analysis
\(k\)-means clustering
a feature importance analysis

To conduct any post-estimation diagnostics, the parameter post_est_stats of the class ModifiedCausalForest needs to be set . Once you have estimated your \(\text{IATE's}\) using the predict() method, you can conduct the post-estimation diagnostics with the analyse() method:

from mcf.example_data import example_data
from mcf.mcf_main import ModifiedCausalForest
from mcf.reporting import McfOptPolReport

# Generate example data using the built-in function `example_data()`
training_df, prediction_df, name_dict = example_data()


my_mcf = ModifiedCausalForest(
        var_y_name="outcome",
        var_d_name="treat",
        var_x_name_ord=["x_cont0", "x_cont1", "x_ord1"],
        var_x_name_unord=["x_unord0"],
        # Enable post-estimation diagnostics
        post_est_stats=True
    )

my_mcf.train(training_df)
results = my_mcf.predict(prediction_df)

post_estimation_diagnostics = my_mcf.analyse(results)

The easiest way to inspect the results of the post-estimation diagnostics, is to read the PDF-report that can be generated using the class McfOptPolReport:

mcf_report = McfOptPolReport(
    mcf=my_mcf, outputpath=my_mcf.gen_cfg.outpath,
    outputfile='Modified-Causal-Forest_Report'
)
mcf_report.report()

post_relative_to_first_group_only (the default) limits the post-estimation analysis to IATEs relative to the treatment with the lowest value. For discrete treatments, set it to False to include all estimated treatment comparisons.

To use analyse(), set post_est_stats and p_iate, and pass the results returned by predict(). Output must be enabled with _int_with_output. This also enables _int_return_iate_sp, which provides the required IATE dataframe.

Set post_kmeans_yes to group observations with similar estimated IATEs using k-means clustering. In this case, analyse() returns a results dictionary with cluster membership added to iate_data_df. With post_kmeans_yes=False, it returns None, although other requested analyses are still performed.

Set post_random_forest_vi to examine which covariates predict variation in the estimated IATEs. These feature-importance statistics and the clusters are descriptive: they do not measure causal effects of the covariates.

p_ci_level sets the confidence level for pointwise intervals in effect plots, for example p_ci_level=0.95. These intervals do not account for selecting subgroups after inspecting the estimated effects. Formal inference for such subgroups requires an analysis that accounts for how they were selected.

6.1. Descriptive statistics#

With post_est_stats set, the distribution of the estimated \(\text{IATE's}\) will be presented. The produced plots are also available in the output folder that is produced by the mcf package. You can find the location of this folder by accessing the “outpath” entry of the gen_cfg attribute of your Modified Causal Forest:

my_mcf.gen_cfg.outpath

You can also specify this path through the gen_outpath parameter of the class ModifiedCausalForest. The output folder will contain the jpeg/pdf-files of the plots as well as csv-files of the underlying data in the subfolder ate_iate.

6.2. Correlation analysis#

The correlation analysis estimates the dependencies between the different \(\text{IATE's}\), between the \(\text{IATE's}\) and the potential outcomes, and between the \(\text{IATE's}\) and the features. You can activate the correlation analysis by setting the parameter post_bin_corr_yes . Note that the correlation coefficients are only displayed if their absolute values exceed the threshold specified by the parameter post_bin_corr_threshold.

6.3. \(k\)-means clustering#

To analyze heterogeneity in different groups (clusters), you can conduct \(k\)-means clustering by setting the parameter post_kmeans_yes to True. The mcf package uses the k-means++ algorithm from scikit-learn to build clusters based on the \(\text{IATE's}\).

from mcf.example_data import example_data
from mcf.mcf_main import ModifiedCausalForest
from mcf.reporting import McfOptPolReport

# Generate example data using the built-in function `example_data()`
training_df, prediction_df, name_dict = example_data()

my_mcf = ModifiedCausalForest(
        var_y_name="outcome",
        var_d_name="treat",
        var_x_name_ord=["x_cont0", "x_cont1", "x_ord1"],
        var_x_name_unord=["x_unord0"],
        post_est_stats=True,
        # Perform k-means clustering
        post_kmeans_yes=True
    )

my_mcf.train(training_df)
results = my_mcf.predict(prediction_df)

post_estimation_diagnostics = my_mcf.analyse(results)

The report obtained through the class McfOptPolReport will contain descriptive statistics of the \(\text{IATE's}\), the potential outcomes and the features for each cluster.

mcf_report = McfOptPolReport(
    mcf=my_mcf, outputpath=my_mcf.gen_cfg.outpath,
    outputfile='Modified-Causal-Forest_Report'
)
mcf_report.report()

If you wish to analyse the clusters yourself, you can access the cluster membership of each observation through the “iate_data_df” entry of the dictionary returned by the analyse() method. The cluster membership is stored in the column IATE_Cluster of the DataFrame.

if post_estimation_diagnostics is not None:
    cluster_df = post_estimation_diagnostics["iate_data_df"]

Cluster labels start at zero. _int_max_obs_kmeans may restrict the returned rows to a subsample; preserve their identifiers when merging.

You can define a range for the number of clusters through the parameter post_kmeans_no_of_groups. The final number of clusters is chosen via silhouette analysis. To guard against getting stuck at local extrema, the number of replications with different random start centers can be defined through the parameter post_kmeans_replications. The parameter post_kmeans_max_tries sets the maximum number of iterations in each replication to achieve convergence.

6.4. Feature importance#

If you are interested in learning which of your features have a lot of predictive power for the estimated \(\text{IATE's}\) you can activate the feature importance procedure by setting the parameter post_random_forest_vi . This procedure will build a predictive random forest to determine which features predict the \(\text{IATE's}\) most. The feature importance statistics are presented in percentage points of the coefficient of determination, \(R^2\), that is lost when the respective feature is randomly permuted. The \(R^2\) statistics are obtained through the RandomForestRegressor provided by scikit-learn.

6.5. Parameter overview#

Below is an overview of the above mentioned parameters related to post-estimation diagnostics in the class ModifiedCausalForest:

Parameter	Description
`post_est_stats`	If True, post-estimation diagnostics are conducted. Default: True.
`post_relative_to_first_group_only`	If True, post-estimation diagnostics will only be conducted for \(\text{IATE's}\) relative to the first treatment state. If False, the diagnostics cover the \(\text{IATE's}\) of all possible treatment combinations. Default: True.
`p_ci_level`	Pointwise confidence level for effect plots. Default: 0.95.
`post_bin_corr_yes`	If True, the binary correlation analysis is conducted. Default: True.
`post_bin_corr_threshold`	If `post_bin_corr_yes` is True, correlations are only displayed if their absolute value is at least `post_bin_corr_threshold`. Default: 0.1.
`post_kmeans_yes`	If True, \(k\)-means clustering is conducted to build clusters based on the \(\text{IATE's}\). Default: True.
`post_kmeans_no_of_groups`	Only relevant if `post_kmeans_yes` is True. Determines the number of clusters for \(k\)-means clustering. Should be specified as a list of values. Default: See the `API`.
`post_kmeans_max_tries`	Only relevant if `post_kmeans_yes` is True. Determines the maximum number of iterations to achieve convergence in each \(k\)-means clustering replication. Default: 1000.
`post_kmeans_replications`	Only relevant if `post_kmeans_yes` is True. Determines the number of replications for \(k\)-means clustering. Default: 10.
`post_kmeans_min_size_share`	Smallest share of cluster size allowed in %. Default (None) is 1.
`post_random_forest_vi`	If True, the feature importance analysis is conduced. Default: True.
`post_plots`	If True, post-estimation diagnostic plots are printed during runtime. Default: True.
`post_tree`	Regression trees (honest and standard) of Depth 2 to 5 are estimated to describe IATES(x). Default (or None) is True.

Please consult the API for more details.

6.6. Example#

from mcf.example_data import example_data
from mcf.mcf_main import ModifiedCausalForest
from mcf.reporting import McfOptPolReport

# Generate example data using the built-in function `example_data()`
training_df, prediction_df, name_dict = example_data()

my_mcf = ModifiedCausalForest(
        var_y_name="outcome",
        var_d_name="treat",
        var_x_name_ord=["x_cont0", "x_cont1", "x_ord1"],
        var_x_name_unord=["x_unord0"],
        p_ci_level=0.95,
        # Parameters for post-estimation diagnostics
        post_est_stats=True,
        post_relative_to_first_group_only=True,
        post_bin_corr_yes=True,
        post_bin_corr_threshold=0.1,
        post_kmeans_yes=True,
        post_kmeans_no_of_groups=[3, 4, 5, 6, 7],
        post_kmeans_max_tries=1000,
        post_kmeans_replications=10,
        post_random_forest_vi=True,
        post_plots=True,
        post_kmeans_min_size_share=1,
        post_tree=True
    )

my_mcf.train(training_df)
results = my_mcf.predict(prediction_df)

# Compute the post-estimation diagnostics
post_estimation_diagnostics = my_mcf.analyse(results)

# Access cluster memberships (column 'IATE_Cluster')
if post_estimation_diagnostics is not None:
    cluster_df = post_estimation_diagnostics["iate_data_df"]

# Produce a PDF-report with the results, including post-estimation diagnostics
mcf_report = McfOptPolReport(
    mcf=my_mcf, outputpath=my_mcf.gen_cfg.outpath,
    outputfile='Modified-Causal-Forest_Report'
)
mcf_report.report()