Last updated: 20 October 2026
Research
Papers, articles and preprints.
📰 Featured
FACE: Feasible and actionable counterfactual explanations
Explainability fact sheets: A framework for systematic assessment of explainable approaches
2026
Trust hinders bidirectional human–AI alignment
Trust calibration remains the dominant framing of AI-assisted decision making:
the goal is to close the gap between perceived and actual trustworthiness of AI systems so that users rely on them appropriately.
We argue that this approach, while useful for decision problems that are well-defined and clearly scoped, overly restricts what it means to make a decision with AI support.
To overcome this limitation, we develop three dimensions of AI-assisted decision making.
First, trusting AI is only one of several possible strategies for managing uncertainty.
Second, trust calibration is often misapplied when AI models large-world decision problems, i.e., ill-defined situations where goals and states are only partially known or unknowable.
Third, in such settings human–AI alignment requires addressing higher-order decision processes.
We conclude by reflecting on the implications of our observations on the design of process-oriented human–AI coupled systems that jointly identify and negotiate uncertainty throughout the decision-making task.
@article{benk2026trust,
title={Trust hinders bidirectional human--{AI} alignment},
author={Benk, Michaela and Sokol, Kacper},
journal={NeurIPS Workshop on Dynamic Alignment in
Human--AI Coupled System},
pages={1--7},
year={2026},
doi={10.48550/arXiv.2610.15136}
}Counterfactual explanations via locally-guided sequential algorithmic recourse
Counterfactuals operationalised through algorithmic recourse have become a powerful tool to make AI systems explainable.
Conceptually, given an individual classified as – the factual – we seek actions such that their prediction becomes the desired class – the counterfactual.
This process offers algorithmic recourse that is (1) easy to customise and interpret, and (2) directly aligned with the goals of each individual.
However, the properties of a ‘good’ counterfactual are still largely debated;
it remains an open challenge to locate an effective counterfactual along with its corresponding recourse.
Some strategies use gradient-driven methods, but these offer no guarantees on the feasibility of the recourse and are open to adversarial attacks on carefully created manifolds.
This can lead to unfairness and lack of robustness.
Other methods are data-driven, which mostly addresses the feasibility problem at the expense of privacy, security and secrecy as they require access to the entire training dataset.
Here, we introduce a model-agnostic technique that composes feasible and actionable counterfactual explanations using locally-acquired information at each step of the algorithmic recourse.
Our explainer preserves the privacy of users by only leveraging data that it specifically requires to construct actionable algorithmic recourse, and protects the model by offering transparency solely in the regions deemed necessary for the intervention.
@article{small2023counterfactual,
title={Counterfactual explanations via locally-guided sequential
algorithmic recourse},
author={Small, Edward A and Clark, Jeffrey N and
McWilliams, Christopher J and Ambler, Michael and
Sokol, Kacper and Chan, Jeffrey and Salim, Flora D and
Santos-Rodriguez, Raul},
journal={ACM Transactions on Intelligent Systems and Technology},
volume={17},
number={6},
pages={1--27},
year={2026},
publisher={ACM},
address={New York, NY, USA},
doi={10.1145/3770065}
}Explanation navigator: Rectifying out-of-scope human interpretations of leaky AI explanations through conversational guidance
As explanations of artificial intelligence systems proliferate, their recipients must grasp not only what they convey but also recognise what they cannot.
We conducted an interview study with nine participants to examine how explainees reason when their information needs exceed the scope of available explanations.
Participants often unwittingly confabulated explanatory insights when relevant information was missing from the explanations, not recognising the inherent limitations thereof.
We characterise such explanations as leaky explanations – simplifications that strive to hide complexity yet whose correct interpretation hinges on understanding of the concealed details.
To address out-of-scope interpretations we propose Explanation Navigator:
a conversational interaction framework that detects mismatches between users’ information needs and explanations’ content, elucidating pertinent yet implicit details and providing complementary explanations for unmet information needs.
@article{xuan2026explanation,
title={Explanation navigator: {Rectifying} out-of-scope human
interpretations of leaky {AI} explanations through
conversational guidance},
author={Xuan, Yueqing and Sokol, Kacper and Hettiachchi, Danula},
journal={arXiv Preprint},
pages={1--43},
eprint={2609.22707},
archivePrefix={arXiv},
year={2026},
doi={10.48550/arXiv.2609.22707}
}Hacking the "wicked problems" of search: A hackathon as a collaborative approach towards capturing query variability
Small differences in how people word search queries, shaped by their values, information needs and search literacy, can greatly change the results they get, with real consequences for social cohesion and online safety.
This paper reports on a two-day hackathon held in Melbourne in November 2025 by the ARC Centre of Excellence for Automated Decision-Making and Society.
Sixteen early-career researchers from computer science and the humanities, working in five teams, explored this “query variability”.
They used computational modelling, interviews and digital ethnography to build frameworks for assessing cross-platform search results on wicked problems such as online dating, disaster response, domestic violence and children’s exposure to sensitive content.
The authors observed that these problems are fractal and that commercial search systems handle their complexity poorly.
Participants responded with agile mixed-methods approaches while continually negotiating shared epistemologies, task scope and social impact.
The authors argue that embracing these tensions makes hackathons a useful participatory method for context-sensitive, interdisciplinary research on search, despite their known limitations.
@inproceedings{kasianenko2026hacking,
title={Hacking the ``wicked problems'' of search:
{A} hackathon as a collaborative approach towards
capturing query variability},
author={Kasianenko, Kateryna and Nagappa, Ashwin and
Zendel, Oleg and Storey, Sally and Bruns, Axel and
Angus, Daniel and Albury, Kath and Sanderson, Mark and
Obeid, Abdul and Lawrence, Amanda and Badola, Anand and
Ma, Chenglong and Tokareva, Ekaterina and Yang, Fan and
Scott, Fletcher and Abu Shaqra, Futoon M and Read, India and
Kuai, Joanne and Ran, Kun and Luong, Khanh and
Hegarty, Kieran and Khirbat, Madhurima and Abu Onq, Nuha and
El Masri, Rayane and Sharmin, Sadia and Allawati, Sara and
Sun, Shuoqi and Dewi, Udiana and Kusumawati, Utami Diah and
Wanyan, Xinye and Fang, Yanheng and Khan, Awais Hameed and
Mamaev, Bogdan and Sokol, Kacper and Herbertson, Kyle and
Trippas, Johanne and Spina, Damiano and
Hettiachchi, Danula},
booktitle={The Search Engines and Society Network Annual Conference},
pages={1--3},
publisher={Search Engines and Society Network},
year={2026},
doi={10.48441/4427.3716}
}Diversity-augmented negative sampling for implicit collaborative filtering
Recommenders built upon implicit collaborative filtering are typically trained to distinguish between users’ positive and negative preferences.
When direct observations of the latter are unavailable, negative training data are constructed with sampling techniques.
But since items often exhibit clustering in the latent space, existing methods tend to oversample negatives from dense regions, resulting in homogeneous training data and limited model expressiveness.
To address these shortcomings, we propose a novel negative sampler with diversity guarantees.
To achieve them, our approach first pairs each positive item of a user with one that they have not yet interacted with;
this instance, called hard negative, is chosen as the top-scoring item according to the model.
Instead of discarding the remaining highly informative items, we store them in a user-specific cache.
Next, our diversity-augmented sampler selects a representative subset of negatives from the cache, ensuring its dissimilarity from the corresponding user’s hard negatives.
Our generator then combines these items with the hard negatives, replacing them to produce more effective (synthetic) negative training data that are informative and diverse.
Experiments show that our method consistently leads to superior recommendation quality without sacrificing computational efficiency.
@inproceedings{xuan2025diverse,
title={Diversity-augmented negative sampling for
implicit collaborative filtering},
author={Xuan, Yueqing and Sokol, Kacper and Sanderson, Mark and
Chan, Jeffrey},
booktitle={Proceedings of the ACM Web Conference},
publisher={ACM},
address={New York, NY, USA},
pages={6288--6298},
year={2026},
doi={10.1145/3774904.3792346}
}2025
Evaluating and addressing fairness across user groups in negative sampling for recommender systems
Recommender systems trained on implicit feedback data rely on negative sampling to distinguish positive items from negative items for each user.
Since the majority of positive interactions come from a small group of active users, negative samplers are often impacted by data imbalance, leading them to choose more informative negatives for prominent users while providing less useful ones for users who are not so active.
This leads to inactive users being further marginalised in the training process, thus receiving inferior recommendations.
In this paper, we conduct a comprehensive empirical study demonstrating that state-of-the-art negative sampling strategies provide more accurate recommendations for active users than for inactive users.
We also find that increasing the number of negative samples for each positive item improves the average performance, but the benefit is distributed unequally across user groups, with active users experiencing performance gain while inactive users suffering performance degradation.
To address this, we propose a group-specific negative sampling strategy that assigns smaller negative ratios to inactive user groups and larger ratios to active groups.
Experiments on eight negative samplers show that our approach improves user-side fairness and performance when compared to a uniform global ratio.
@inproceedings{xuan2023evaluating,
title={Evaluating and addressing fairness across user groups in
negative sampling for recommender systems},
author={Xuan, Yueqing and Sokol, Kacper and Sanderson, Mark and
Chan, Jeffrey},
booktitle={Proceedings of the 34th ACM International Conference on Information and Knowledge Management},
publisher={ACM},
address={New York, NY, USA},
pages={3720--3729},
year={2025},
doi={10.1145/3746252.3761263}
}From research to certification with data-driven medical decision support systems (Dagstuhl Seminar 25052)
This report outlines the programme and outcomes of Dagstuhl Seminar 25052 “From Research to Certification with Data-Driven Medical Decision Support Systems”.
Our seminar addressed the complex challenges of transferring artificial intelligence systems from research labs into real-world clinical practice.
Bringing together clinicians, researchers and industry stakeholders, it explored the potential and pitfalls of deploying data-driven models in healthcare, highlighting the need for rigorous evaluation, human-centred design and responsible innovation.
Key discussions included regulatory hurdles, reproducibility issues, interpretability and human–machine collaboration.
Group sessions focused on evaluation frameworks and human factors in medical artificial intelligence system design.
The seminar laid the foundation for a collaborative research agenda aimed at safe, effective and ethical integration of data-driven predictive models into real-life clinical workflows.
@article{santos2025from,
title={From research to certification with data-driven medical
decision support systems ({Dagstuhl} {Seminar} 25052)},
author={Santos-Rodriguez, Raul and Sokol, Kacper and
Vogt, Julia E and Wellmann, Sven},
journal={Dagstuhl Reports},
volume={15},
number={1},
pages={201--220},
year={2025},
publisher={Schloss Dagstuhl--Leibniz-Zentrum f{\"u}r Informatik},
doi={10.4230/DagRep.15.1.201}
}External validation of predictive models for diagnosis, management and severity of pediatric appendicitis
Background: Appendicitis is a common condition among children and adolescents. Machine learning models can offer much-needed tools for improved diagnosis, severity assessment and management guidance for pediatric appendicitis. However, to be adopted in practice, such systems must be reliable, safe and robust across various medical contexts, e.g., hospitals with distinct clinical practices and patient populations.
Methods: We performed external validation of models predicting the diagnosis, management and severity of pediatric appendicitis. Trained on a cohort of 430 patients admitted to the Children’s Hospital St. Hedwig (Regensburg, Germany), the models were validated on an independent cohort of 301 patients from the Florence-Nightingale-Hospital (Düsseldorf, Germany). The data included demographic, clinical, scoring, laboratory and ultrasound parameters. In addition, we explored the benefits of model retraining and inspected variable importance.
Results: The distributions of most parameters differed between the datasets. Consequently, we saw a decrease in predictive performance for diagnosis, management and severity across most metrics. After retraining with a portion of external data, we observed gains in performance, which, nonetheless, remained lower than in the original study. Notably, the most important variables were consistent across the datasets.
Conclusions: While the performance of transferred models was satisfactory, it remained lower than on the original data. This study demonstrates challenges in transferring models between hospitals, especially when clinical practice and demographics differ or in the presence of externalities such as pandemics. We also highlight the limitations of retraining as a potential remedy since it could not restore predictive performance to the initial level.
@article{marcinkevics2024external,
title={External validation of predictive models for diagnosis,
management and severity of pediatric appendicitis},
author={Marcinkevi{\v{c}}s, Ri{\v{c}}ards and Sokol, Kacper and
Paulraj, Akhil and Hilbert, Melinda A and Rimili, Vivien and
Wellmann, Sven and Knorr, Christian and
Reingruber, Bertram and Vogt, Julia E and
Reis Wolfertstetter, Patricia},
journal={Frontiers in Pediatrics},
volume={13},
pages={1587488},
year={2025},
publisher={Frontiers Media},
doi={10.3389/fped.2025.1587488}
}Perfect counterfactuals in imperfect worlds: Modelling noisy implementation of actions in sequential algorithmic recourse
Algorithmic recourse suggests actions to individuals who have been adversely affected by automated decision-making, helping them to achieve the desired outcome.
Knowing the recourse, however, does not guarantee that users can implement it perfectly, either due to environmental variability or personal choices.
Recourse generation should thus anticipate its sub-optimal or noisy implementation.
While several approaches construct recourse that is robust to small perturbations – e.g., arising due to its noisy implementation – they assume that the entire recourse is implemented in a single step, thus model the noise as one-off and uniform.
But these assumptions are unrealistic since recourse often entails multiple sequential steps, which makes it harder to implement and subject to increasing noise.
In this work, we consider recourse under plausible noise that adheres to the local data geometry and accumulates at every step of the way.
We frame this problem as a Markov Decision Process and demonstrate that such a distribution of plausible noise satisfies the Markov property.
We then propose the RObust SEquential (ROSE) recourse generator for tabular data;
our method produces a series of steps leading to the desired outcome even when they are implemented imperfectly.
Given plausible modelling of sub-optimal human actions and greater recourse robustness to accumulated uncertainty, ROSE provides users with a high chance of success while maintaining low recourse cost.
Empirical evaluation shows that our algorithm effectively navigates the inherent trade-off between recourse robustness and cost while ensuring its sparsity and computational efficiency.
@article{xuan2024perfect,
title={Perfect counterfactuals in imperfect worlds:
{Modelling} noisy implementation of actions in sequential
algorithmic recourse},
author={Xuan, Yueqing and Sokol, Kacper and Sanderson, Mark and
Chan, Jeffrey},
journal={Machine Learning},
publisher={Springer},
volume={114},
number={8},
pages={187},
year={2025},
doi={10.1007/s10994-025-06821-1}
}Leveraging complementary AI explanations to mitigate misunderstanding in XAI
Artificial intelligence explanations can make complex predictive models more comprehensible.
To be effective, however, they should anticipate and mitigate possible misinterpretations, e.g., arising when users infer incorrect information that is not explicitly conveyed.
To this end, we propose complementary explanations – a novel method that pairs explanations to compensate for their respective limitations.
A complementary explanation adds insights that clarify potential misconceptions stemming from the primary explanation while ensuring their coherency and avoiding redundancy.
We introduce a framework for designing and evaluating complementary explanation pairs based on pertinent qualitative properties and quantitative metrics.
Our approach allows to construct complementary explanations that minimise the chance of their misinterpretation.
@inproceedings{xuan2025leveraging,
title={Leveraging complementary {AI} explanations to mitigate
misunderstanding in {XAI}},
author={Xuan, Yueqing and Sokol, Kacper and Sanderson, Mark and
Chan, Jeffrey},
booktitle={Swiss Conference on Data Science},
publisher={IEEE}
volume={2025},
pages={174--177},
year={2025},
doi={10.1109/SDS66131.2025.00034}
}How robust is your fair model? Exploring the robustness of prominent fairness strategies
With the introduction of machine learning in high stakes decision-making, ensuring algorithmic fairness has become an increasingly important task.
To this end, many mathematical definitions of fairness have been proposed, and a variety of optimisation techniques have been developed, all designed to maximise a given notion of fairness.
Fair solutions, however, tend to rely on the quality of training data, and can be highly sensitive to noise.
Recent studies have shown that robustness of many such fairness strategies – i.e., their ability to perform well on unseen data – is not a given and requires careful consideration.
To address this challenge, we propose robustness ratio, which is a novel criterion to measure the robustness of diverse fairness optimisation strategies.
We support our analysis with multiple extensive experiments on five benchmark fairness data sets, using three prominent fairness strategies, in view of four of the most popular definitions of fairness.
Our experiments show that while fairness methods that rely on threshold optimisation (post-processing) mostly outperform other techniques, they are acutely sensitive to noise.
This is in contrast to two other methods—correlation remover (pre-processing) and exponentiated gradient descent (in-processing) – which become increasingly fairer as the random noise injected into the data becomes larger.
Our findings offer a comprehensive overview of fairness strategies that proves invaluable when tasked with choosing the most suitable method for the task at hand.
To the best of our knowledge, we are the first to quantitatively evaluate the robustness of fairness optimisation strategies.
@article{small2022robust,
title={How robust is your fair model? {Exploring} the robustness of
prominent fairness strategies},
author={Small, Edward A and Shao, Wei and Zhang, Zeliang and
Liu, Peihan and Chan, Jeffrey and Sokol, Kacper and
Salim, Flora D},
journal={Data Mining and Knowledge Discovery},
volume={39},
number={4},
pages={41},
publisher={Springer},
year={2025},
doi={10.1007/s10618-025-01112-8}
}Artificial intelligence should genuinely support clinical reasoning and decision making to bridge the translational gap
Artificial intelligence promises to revolutionise medicine, yet its impact remains limited because of the pervasive translational gap.
We posit that the prevailing technology-centric approaches underpin this challenge, rendering such systems fundamentally incompatible with clinical practice, specifically diagnostic reasoning and decision making.
Instead, we propose a novel sociotechnical conceptualisation of data-driven support tools designed to complement doctors’ cognitive and epistemic activities.
Crucially, it prioritises real-world impact over superhuman performance on inconsequential benchmarks.
@article{sokol2024machine,
title={Artificial intelligence should genuinely support clinical
reasoning and decision making to bridge the translational
gap},
author={Sokol, Kacper and Fackler, James and Vogt, Julia E},
journal={npj Digital Medicine},
volume={8},
number={1},
pages={345},
year={2025},
publisher={Nature},
address={London, UK},
doi={10.1038/s41746-025-01725-9}
}Navigating explanatory multiverse through counterfactual path geometry
LIMEtree: Consistent and faithful surrogate explanations of multiple classes
Explainable artificial intelligence provides tools to better understand predictive models and their decisions, but many such methods are limited to producing insights with respect to a single class.
When generating explanations for several classes, reasoning over them to obtain a comprehensive view may be difficult since they can present competing or contradictory evidence.
To address this challenge we introduce the novel paradigm of multi-class explanations.
We outline the theory behind such techniques and propose a local surrogate model based on multi-output regression trees – called LIMEtree – that offers faithful and consistent explanations of multiple classes for individual predictions while being post-hoc, model-agnostic and data-universal.
On top of strong fidelity guarantees, our implementation delivers a range of diverse explanation types, including counterfactual statements favoured in the literature.
We evaluate our algorithm with respect to explainability desiderata, through quantitative experiments and via a pilot user study, on image and tabular data classification tasks, comparing it to LIME, which is a state-of-the-art surrogate explainer.
Our contributions demonstrate the benefits of multi-class explanations and wide-ranging advantages of our method across a diverse set of scenarios.
@article{sokol2020limetree,
title={{LIMEtree}: {Consistent} and faithful
surrogate explanations of multiple classes},
author={Sokol, Kacper and Flach, Peter},
journal={Electronics},
volume={14},
number={5},
pages={929},
year={2025},
publisher={MDPI},
DOI={10.3390/electronics14050929}
}Uncertainty quantification as a principled foundation for explainable artificial intelligence: A case study of counterfactual explanations
In this paper we argue that, to its detriment, transparency research overlooks many foundational concepts of artificial intelligence.
As an illustrating example we focus on uncertainty quantification in the context of counterfactual explainability, demonstrating that its broader adoption could address key challenges in the field.
To this end, we show how uncertainty can provide a principled unifying framework for counterfactual explainability by expressing the core counterfactual properties in terms of uncertainty, allowing us to build two variants of an explainer upon them – one based solely on uncertainty estimates and another pairing them with distance measured in the feature space.
Our comprehensive experiments illustrate highly competitive performance of our framework when compared to many state-of-the-art methods despite its radically simple design.
More broadly, the paper demonstrates that integrating artificial intelligence fundamentals into transparency research promises to yield more reliable, robust and understandable predictive models.
We posit that making artificial intelligence explainability truly uncertainty-aware is the first step towards this goal.
@article{sokol2025uncertainty,
title={Uncertainty quantification as a principled foundation for
explainable artificial intelligence: {A} case study of
counterfactual explanations},
author={Sokol, Kacper and Thies, Santo M A R and
H{\"u}llermeier, Eyke},
journal={arXiv Preprint},
pages={1--32},
eprint={2502.17007},
archivePrefix={arXiv},
year={2025},
doi={10.48550/arXiv.2502.17007}
}Comprehension is a double-edged sword: Over-interpreting unspecified information in intelligible machine learning explanations
Automated decision-making systems are becoming increasingly ubiquitous, which creates an immediate need for their interpretability and explainability.
However, it remains unclear whether users know what insights an explanation offers and, more importantly, what information it lacks.
To answer this question we conducted an online study with 200 participants, which allowed us to assess explainees’ ability to realise explicated information – i.e., factual insights conveyed by an explanation – and unspecified information – i.e, insights that are not communicated by an explanation – across four representative explanation types:
model architecture, decision surface visualisation, counterfactual explainability and feature importance.
Our findings uncover that highly comprehensible explanations, e.g., feature importance and decision surface visualisation, are exceptionally susceptible to misinterpretation since users tend to infer spurious information that is outside of the scope of these explanations.
Additionally, while the users gauge their confidence accurately with respect to the information explicated by these explanations, they tend to be overconfident when misinterpreting the explanations.
Our work demonstrates that human comprehension can be a double-edged sword since highly accessible explanations may convince users of their truthfulness while possibly leading to various misinterpretations at the same time.
Machine learning explanations should therefore carefully navigate the complex relation between their full scope and limitations to maximise understanding and curb misinterpretation.
@article{xuan2023can,
title={Comprehension is a double-edged sword: {Over}-interpreting
unspecified information in intelligible machine learning
explanations},
author={Xuan, Yueqing and Small, Edward and Sokol, Kacper and
Hettiachchi, Danula and Sanderson, Mark},
journal={International Journal of Human--Computer Studies},
volume={193},
pages={103376},
year={2025},
publisher={Elsevier},
doi={10.1016/j.ijhcs.2024.103376},
}2024
Preface of the Proceedings of the MAI-XAI Workshop
This paper describes the motivation and themes for the workshop on Multimodal, Affective and Interactive eXplainable Artificial Intelligence (MAI-XAI), which was collocated with the 2024 European Conference on Artificial Intelligence (ECAI).
In the era of Artificial Intelligence (AI), data scientists aim to enhance decision-making and automate processes, but many AI systems are challenging to interpret due to their opaque, black-box nature.
To address this, eXplainable AI (XAI) seeks to develop intelligent agents that provide understandable decisions and explanations, fostering better human-machine interaction while ensuring ethical principles like fairness and transparency are upheld.
The workshop focuses on improving XAI effectiveness through three key areas:
Multimodal XAI, Affective XAI, and Interactive XAI, emphasizing the need for responsible and trustworthy AI systems.
The workshop received 18 submissions covering all the three main topics mentioned in the call for papers.
All the submissions received three reviews.
Out of these papers, 9 were selected for presentation at the workshop, yielding an acceptance rate of 50%.
@inproceedings{alonso2024preface,
title={Preface},
author={Alonso-Moral, Jose M and Anthis, Zach and
Berlanga, Rafael and Catal{\'a}, Alejandro and
Cimiano, Philipp and Flach, Peter and
H{\"u}llermeier, Eyke and Miller, Tim and
Mindlin, Dimitry and Mitruț, Oana and Moise, Gabriela and
Moldoveanu, Alin and Moldoveanu, Florica and
Sokol, Kacper and Soroa, Aitor},
booktitle={Proceedings of the Workshop on Multimodal, Affective and
Interactive eXplainable Artificial Intelligence (MAI-XAI),
collocated with the European Conference on Artificial
Intelligence (ECAI)},
pages={i--v},
volume={3803},
publisher={CEUR Workshop Proceedings},
year={2024}
}Counterfactual explanations for clustering models
Clustering algorithms rely on complex optimisation processes that may be difficult to comprehend, especially for individuals who lack technical expertise.
While many explainable artificial intelligence techniques exist for supervised machine learning, unsupervised learning – and clustering in particular – has been largely neglected.
To complicate matters further, the notion of a “true” cluster is inherently challenging to define.
These facets of unsupervised learning and its explainability make it difficult to foster trust in such methods and curtail their adoption.
To address these challenges, we propose a new, model-agnostic technique for explaining clustering algorithms with counterfactual statements.
Our approach relies on a novel soft-scoring method that captures the spatial information utilised by clustering models.
It builds upon a state-of-the-art Bayesian counterfactual generator for supervised learning to deliver high-quality explanations.
We evaluate its performance on five datasets and two clustering algorithms, and demonstrate that introducing soft scores to guide counterfactual search significantly improves the results.
@article{spagnol2024counterfactual,
title={Counterfactual explanations for clustering models},
author={Spagnol, Aurora and Sokol, Kacper and Barbiero, Pietro and
Langheinrich, Marc and Gjoreski, Martin},
journal={arXiv Preprint},
pages={1--8},
eprint={2409.12632},
archivePrefix={arXiv},
year={2024},
doi={10.48550/arXiv.2409.12632}
}Cross-model fairness: Empirical study of fairness and ethics under model multiplicity
While data-driven predictive models are a strictly technological construct, they may operate within a social context in which benign engineering choices entail implicit, indirect and unexpected real-life consequences.
Fairness of such systems – pertaining both to individuals and groups – is one relevant consideration in this space;
algorithms can discriminate people across various protected characteristics regardless of whether these properties are included in the data or discernible through proxy variables.
To date, this notion has predominantly been studied for a fixed model, often under different classification thresholds, striving to identify and eradicate undesirable, discriminative and possibly unlawful aspects of its operation.
Here, we backtrack on this fixed model assumption to propose and explore a novel definition of cross-model fairness where individuals can be harmed when one predictor is chosen ad hoc from a group of equally well performing models, i.e., in view of utility-based model multiplicity.
Since a person may be classified differently across models that are otherwise considered equivalent, this individual could argue for a predictor granting them the most favourable outcome, employing which may have adverse effects on other people.
We introduce this scenario with a two-dimensional example and linear classification;
then, we present a comprehensive empirical study based on real-life predictive models and data sets that are popular with the algorithmic fairness community;
finally, we investigate analytical properties of cross-model fairness and its ramifications in a broader context.
Our findings suggest that such unfairness can be readily found in real life and it may be difficult to mitigate by technical means alone as doing so is likely to degrade predictive performance.
@article{sokol2024ethical,
title={Cross-model fairness: {Empirical} study of fairness and
ethics under model multiplicity},
author={Sokol, Kacper and Kull, Meelis and Chan, Jeffrey and
Salim, Flora},
journal={ACM Journal on Responsible Computing},
pages={1--27},
volume={1},
number={3},
year={2024},
publisher={ACM},
address={New York, NY, USA},
doi={10.1145/3677173}
}Equalised odds is not equal individual odds: Post-processing for group and individual fairness
Group fairness is achieved by equalising prediction distributions between protected sub-populations;
individual fairness requires treating similar individuals alike.
These two objectives, however, are incompatible when a scoring model is calibrated through discontinuous probability functions, where individuals can be randomly assigned an outcome determined by a fixed probability.
This procedure may provide two similar individuals from the same protected group with classification odds that are disparately different – a clear violation of individual fairness.
Assigning unique odds to each protected sub-population may also prevent members of one sub-population from ever receiving equal chances of a positive outcome to another, which we argue is another type of unfairness called individual odds.
We reconcile all this by constructing continuous probability functions between group thresholds that are constrained by their Lipschitz constant.
Our solution preserves the model’s predictive power, individual fairness and robustness while ensuring group fairness.
@inproceedings{small2023equalised,
title={Equalised odds is not equal individual odds:
{Post-processing} for group and individual fairness},
author={Small, Edward A and Sokol, Kacper and Manning, Daniel and
Salim, Flora D and Chan, Jeffrey},
booktitle={Proceedings of the 2024 ACM Conference on Fairness,
Accountability, and Transparency},
pages={1559--1578},
year={2024},
publisher={ACM},
address={New York, NY, USA},
doi={10.1145/3630106.3658989}
}Leveraging simulation data to understand bias in predictive models of infectious disease spread
The spread of infectious diseases is a highly complex spatiotemporal process, difficult to understand, predict, and effectively respond to.
Machine learning and artificial intelligence (AI) have achieved impressive results in other learning and prediction tasks;
however, while many AI solutions are developed for disease prediction, only a few of them are adopted by decision-makers to support policy interventions.
Among several issues preventing their uptake, AI methods are known to amplify the bias in the data they are trained on.
This is especially problematic for infectious disease models that typically leverage large, open, and inherently biased spatiotemporal data.
These biases may propagate through the modeling pipeline to decision-making, resulting in inequitable policy interventions.
Therefore, there is a need to gain an understanding of how the AI disease modeling pipeline can mitigate biased input data, in-processing models, and biased outputs.
Specifically, our vision is to develop a large-scale micro-simulation of individuals from which human mobility, population, and disease ground-truth data can be obtained.
From this complete dataset – which may not reflect the real world – we can sample and inject different types of bias.
By using the sampled data in which bias is known (as it is given as the simulation parameter), we can explore how existing solutions for fairness in AI can mitigate and correct these biases and investigate novel AI fairness solutions.
Achieving this vision would result in improved trust in such models for informing fair and equitable policy interventions.
@article{zufle2024leveraging,
title={Leveraging simulation data to understand bias in predictive
models of infectious disease spread},
author={Z{\"u}fle, Andreas and Salim, Flora and Anderson, Taylor and
Scotch, Matthew and Xiong, Li and Sokol, Kacper and
Xue, Hao and Kong, Ruochen and Heslop, David and
Paik, Hye-Young and MacIntyre, C Raina},
journal={ACM Transactions on Spatial Algorithms and Systems},
publisher={ACM},
address={New York, NY, USA},
volume={10},
number={2},
pages={1--22},
year={2024},
doi={10.1145/3660631}
}What does evaluation of explainable artificial intelligence actually tell us? A case for compositional and contextual validation of XAI building blocks
Despite significant progress, evaluation of explainable artificial intelligence remains elusive and challenging.
In this paper we propose a fine-grained validation framework that is not overly reliant on any one facet of these sociotechnical systems, and that recognises their inherent modular structure:
technical building blocks, user-facing explanatory artefacts and social communication protocols.
While we concur that user studies are invaluable in assessing the quality and effectiveness of explanation presentation and delivery strategies from the explainees’ perspective in a particular deployment context, the underlying explanation generation mechanisms require a separate, predominantly algorithmic validation strategy that accounts for the technical and human-centred desiderata of their (numerical) outputs.
Such a comprehensive sociotechnical utility-based evaluation framework could allow to systematically reason about the properties and downstream influence of different building blocks from which explainable artificial intelligence systems are composed – accounting for a diverse range of their engineering and social aspects – in view of the anticipated use case.
@inproceedings{sokol2024what,
title={What does evaluation of explainable artificial intelligence
actually tell us? {A} case for compositional and contextual
validation of {XAI} building blocks},
author={Sokol, Kacper and Vogt, Julia E},
booktitle={Extended Abstracts of the 2024 ACM CHI Conference on Human Factors in Computing Systems},
pages={1--8},
year={2024},
publisher={ACM},
address={New York, NY, USA},
doi={10.1145/3613905.3651047}
}Interpretable representations in explainable AI: From theory to practice
Interpretable representations are the backbone of many explainers that target black-box predictive systems based on artificial intelligence and machine learning algorithms.
They translate the low-level data representation necessary for good predictive performance into high-level human-intelligible concepts used to convey the explanatory insights.
Notably, the explanation type and its cognitive complexity are directly controlled by the interpretable representation, tweaking which allows to target a particular audience and use case.
However, many explainers built upon interpretable representations overlook their merit and fall back on default solutions that often carry implicit assumptions, thereby degrading the explanatory power and reliability of such techniques.
To address this problem, we study properties of interpretable representations that encode presence and absence of human-comprehensible concepts.
We demonstrate how they are operationalised for tabular, image and text data;
discuss their assumptions, strengths and weaknesses;
identify their core building blocks; and
scrutinise their configuration and parameterisation.
In particular, this in-depth analysis allows us to pinpoint their explanatory properties, desiderata and scope for (malicious) manipulation in the context of tabular data where a linear model is used to quantify the influence of interpretable concepts on a black-box prediction.
Our findings lead to a range of recommendations for designing trustworthy interpretable representations;
specifically, the benefits of class-aware (supervised) discretisation of tabular data, e.g., with decision trees, and sensitivity of image interpretable representations to segmentation granularity and occlusion colour.
@article{sokol2024interpretable,
title={Interpretable representations in explainable {AI}:
{From} theory to practice},
author={Sokol, Kacper and Flach, Peter},
journal={Data Mining and Knowledge Discovery},
volume={38},
number={5},
pages={3102--3140},
publisher={Springer},
year={2024},
doi={10.1007/s10618-024-01010-5}
}2023
(Un)reasonable allure of ante-hoc interpretability for high-stakes domains: Transparency is necessary but insufficient for comprehensibility
Ante-hoc interpretability has become the holy grail of explainable artificial intelligence for high-stakes domains such as healthcare;
however, this notion is elusive, lacks a widely-accepted definition and depends on the operational context.
It can refer to predictive models whose structure adheres to domain-specific constraints, or ones that are inherently transparent.
The latter conceptualisation assumes observers who judge this quality, whereas the former presupposes them to have technical and domain expertise (thus alienating other groups of explainees).
Additionally, the distinction between ante-hoc interpretability and the less desirable post-hoc explainability, which refers to methods that construct a separate explanatory model, is vague given that transparent predictive models may still require (post-)processing to yield suitable explanatory insights.
Ante-hoc interpretability is thus an overloaded concept that comprises a range of implicit properties, which we unpack in this paper to better understand what is needed for its safe adoption across high-stakes domains.
To this end, we outline modelling and explaining desiderata that allow us to navigate its distinct realisations in view of the envisaged application and audience.
@journal{sokol2023unreasonable,
title={{(Un)reasonable} allure of ante-hoc interpretability for
high-stakes domains: {Transparency} is necessary but
insufficient for comprehensibility},
author={Sokol, Kacper and Vogt, Julia E},
journal={ICML Workshop on Interpretable Machine Learning in
Healthcare},
pages={1--6},
year={2023},
doi={10.48550/arXiv.2306.02312}
}Simply Logical – The first three decades
This paper charts the evolution of the Prolog textbook Simply Logical – Intelligent Reasoning by Example from print with runnable programmes on a 3.5-inch diskette published in 1994, via various intermediate online versions, to the fully interactive online edition available today.
Our aim is to investigate – from both the writer’s and the reader’s perspectives – the potential impact of technology on educational materials.
The three authors of this paper present three distinct and complementary points of view, which come together to shape an interactive online book that offers an immersive learning experience.
Peter describes the philosophy behind the original book and experiences teaching from it.
Jan demonstrates how contemporary web programming has enabled a fully interactive realisation of the book’s philosophy.
Kacper reports how different technologies can be adopted to build a versatile authoring toolkit that underpins the interactive online edition of the book.
Collating these distinct yet coherent perspectives allows us to reflect on future opportunities for interactive Prolog teaching materials arising from the continuous development of web technologies.
The insights we offer should be of interest to teachers of Prolog as well as those wanting to find out about the latest educational technologies.
@incollection{flach2023simply,
title={Simply Logical -- {The} First Three Decades},
author={Flach, Peter and Sokol, Kacper and Wielemaker, Jan},
booktitle={Prolog: The Next 50 Years},
pages={184--193},
year={2023},
publisher={Springer},
address={Cham, Switzerland},
doi={10.1007/978-3-031-35254-6_15}
}Helpful, misleading or confusing: How humans perceive fundamental building blocks of artificial intelligence explanations
Explainable artificial intelligence techniques are developed at breakneck speed, but suitable evaluation approaches lag behind.
With explainers becoming increasingly complex and a lack of consensus on how to assess their utility, it is challenging to judge the benefit and effectiveness of different explanations.
To address this gap, we take a step back from sophisticated predictive algorithms and instead look into explainability of simple decision-making models.
In this setting, we aim to assess how people perceive comprehensibility of their different representations such as mathematical formulation, graphical representation and textual summarisation (of varying complexity and scope).
This allows us to capture how diverse stakeholders – engineers, researchers, consumers, regulators and the like – judge intelligibility of fundamental concepts that more elaborate artificial intelligence explanations are built from.
This position paper charts our approach to establishing appropriate evaluation methodology as well as a conceptual and practical framework to facilitate setting up and executing relevant user studies.
@article{small2023helpful,
title={Helpful, misleading or confusing: {How} humans perceive
fundamental building blocks of artificial intelligence
explanations},
author={Small, Edward and Xuan, Yueqing and
Hettiachchi, Danula and Sokol, Kacper},
journal={ACM CHI Workshop on Human-Centered Explainable AI},
pages={1--8},
year={2023},
doi={10.48550/arXiv.2303.00934}
}Mind the gap! Bridging explainable artificial intelligence and human understanding with Luhmann's functional theory of communication
Over the past decade explainable artificial intelligence has evolved from a predominantly technical discipline into a field that is deeply intertwined with social sciences.
Insights such as human preference for contrastive – more precisely, counterfactual – explanations have played a major role in this transition, inspiring and guiding the research in computer science.
Other observations, while equally important, have nevertheless received much less consideration.
The desire of human explainees to communicate with artificial intelligence explainers through a dialogue-like interaction has been mostly neglected by the community.
This poses many challenges for the effectiveness and widespread adoption of such technologies as delivering a single explanation optimised according to some predefined objectives may fail to engender understanding in its recipients and satisfy their unique needs given the diversity of human knowledge and intention.
Using insights elaborated by Niklas Luhmann and, more recently, Elena Esposito we apply social systems theory to highlight challenges in explainable artificial intelligence and offer a path forward, striving to reinvigorate the technical research in the direction of interactive and iterative explainers.
Specifically, this paper demonstrates the potential of systems theoretical approaches to communication in elucidating and addressing the problems and limitations of human-centred explainable artificial intelligence.
@article{keenan2023mind,
title={Mind the gap! {Bridging} explainable artificial intelligence
and human understanding with {Luhmann}'s
functional theory of communication},
author={Keenan, Bernard and Sokol, Kacper},
journal={arXiv Preprint},
pages={1--21},
eprint={2302.03460},
archivePrefix={arXiv},
year={2023},
doi={10.48550/arXiv.2302.03460}
}eXplainable Machine Learning – USI Course
Machine learning models require care, attention and a fair amount of tuning to offer accurate, consistent and robust predictive modelling of data. Why should their transparency and explainability be any different? While it is possible to easily generate explanatory insights with methods that are post-hoc and model-agnostic – LIME and SHAP, for example – these can be misleading when output by generic tools and viewed out of (technical or domain) context. Explanations should not be taken at their face value; instead their understanding ought to come from interpreting explanatory insights in view of the implicit caveats and limitations under which they were generated. After all, explainability algorithms are complex entities often built from multiple components that are subject to parameterisation choices and operational assumptions, all of which must be accounted for and configured to yield a truthful and useful explainer. Additionally, since any particular method may only provide partial information about the functioning of a predictive model, embracing diverse insights and appreciating their complementarity – as well as disagreements – can further enhance understanding of an algorithmic decision-making process.
This course takes an adversarial perspective on artificial intelligence explainability and machine learning interpretability. Instead of reviewing popular approaches used to these ends, it breaks them up into core functional blocks, studies the role and configuration thereof, and reassembles them to create bespoke, well-understood explainers suitable for the problem at hand. The course focuses predominantly on tabular data, with some excursions into image and text explainability whenever a method is agnostic of the data type. The tuition is complemented by a series of hands-on materials for self-study, which allow you to experiment with these techniques and appreciate their inherent complexity, capabilities and limitations. The assignment, on the other hand, requires you to develop a tailor-made explainability suite for a data set and predictive model of your choice, or alternatively analyse an explainability algorithm to identify its core algorithmic building blocks and explore how they affect the resulting explanation. (Note that there is a scope for a bespoke project if you have a well-defined idea in mind.)
@misc{sokol2023explainable,
title={{eXplainable} Machine Learning -- {USI} Course},
author={Sokol, Kacper},
howpublished={\url{https://xmlx-dev.github.io/usi-slides-mirror}},
year={2023},
doi={10.5281/zenodo.7646970},
}2022
What and how of machine learning transparency: Building bespoke explainability tools with interoperable algorithmic components
Explainability techniques for data-driven predictive models based on artificial intelligence and machine learning algorithms allow us to better understand the operation of such systems and help to hold them accountable.
New transparency approaches are developed at breakneck speed, enabling us to peek inside these black boxes and interpret their decisions.
Many of these techniques are introduced as monolithic tools, giving the impression of one-size-fits-all and end-to-end algorithms with limited customisability.
Nevertheless, such approaches are often composed of multiple interchangeable modules that need to be tuned to the problem at hand to produce meaningful explanations.
This paper introduces a collection of hands-on training materials – slides, video recordings and Jupyter Notebooks – that provide guidance through the process of building and evaluating bespoke modular surrogate explainers for tabular data.
These resources cover the three core building blocks of this technique:
interpretable representation composition, data sampling and explanation generation.
@article{sokol2022and,
title={What and How of Machine Learning Transparency: {Building} Bespoke Explainability Tools with Interoperable Algorithmic Components},
author={Sokol, Kacper and Hepburn, Alexander and Santos-Rodriguez, Raul and
Flach, Peter},
journal={Journal of Open Source Education},
publisher={The Open Journal},
volume={5},
number={58},
pages={175},
year={2022},
doi={10.21105/jose.00175}
}Analysing donors' behaviour in non-profit organisations for disaster resilience
With the advancement and proliferation of technology, non-profit organisations have embraced social media platforms to improve their operational capabilities through brand advocacy, among many other strategies.
The effect of such social media campaigns on these institutions, however, remains largely underexplored, especially during disaster periods.
This work introduces and applies a quantitative investigative framework to understand how social media influence the behaviour of donors and their usage of these platforms throughout (natural) disasters.
More specifically, we explore how on-line engagement – as captured by Facebook interactions and Google search trends – corresponds to the donors’ behaviour during the catastrophic 2019–2020 Australian bushfire season.
To discover this relationship, we analyse the record of donations made to the Australian Red Cross throughout this period.
Our exploratory study reveals that social media campaigns are effective in encouraging on-line donations made via a dedicated website.
We also compare this mode of giving to more regular, direct deposit gifting.
@inproceedings{rajapaksha2022analysing,
title={Analysing donors' behaviour in non-profit organisations for
disaster resilience},
author={Rajapaksha, Dilini and Sokol, Kacper and Chan, Jeffrey and
Salim, Flora and Prasad, Mukesh and Samarawickrama, Mahendra},
booktitle={Proceedings of the 2022 ISCRAM Asia Pacific Conference:
Social Media for Disaster Response},
publisher={Massey Unversity},
address={Palmerston North, New Zealand},
pages={258--267},
year={2022}
}FAT Forensics: A Python toolbox for algorithmic fairness, accountability and transparency
Today, artificial intelligence systems driven by machine learning algorithms can be in a position to take important, and sometimes legally binding, decisions about our everyday lives.
In many cases, however, these systems and their actions are neither regulated nor certified.
To help counter the potential harm that such algorithms can cause we developed an open source toolbox that can analyse selected fairness, accountability and transparency aspects of the machine learning process: data (and their features), models and predictions, allowing to automatically and objectively report them to relevant stakeholders.
In this paper we describe the design, scope, usage and impact of this Python package, which is published under the 3-Clause BSD open source licence.
@article{sokol2022fatf,
title={{FAT} {Forensics}: {A} {Python} toolbox for algorithmic fairness,
accountability and transparency},
author={Sokol, Kacper and Santos-Rodriguez, Raul and Flach, Peter},
journal={Software Impacts},
volume={14},
pages={100406},
year={2022},
publisher={Elsevier},
doi={10.1016/j.simpa.2022.100406}
}Simply Logical – Intelligent Reasoning by Example (Fully Interactive Online Edition)
“Simply Logical – Intelligent Reasoning by Example” by Peter Flach was first published by John Wiley in 1994. It could be purchased as book-only or with a 3.5 inch diskette containing the Prolog programs printed in the book (for various operating systems).
In 2007 the copyright reverted back to the author and book and programs have been freely available online ever since. The print version is no longer available through John Wiley publishers.
In 2015, as a pilot, we ported most of the original book into an online, interactive website using SWI-Prolog’s SWISH platform. Since then, we created the Simply Logical open source organisation committed to maintaining a suite of freely available interactive online educational resources about Artificial Intelligence and Logic Programming with Prolog. Visit our organisation’s website and GitHub profile for more details.
@book{flach2022simply,
title={Simply Logical -- Intelligent Reasoning by Example
(Fully Interactive Online Edition)},
author={Flach, Peter and Sokol, Kacper},
publisher={\url{https://book.simply-logical.space/}},
year={2022},
doi={10.5281/zenodo.7051040}
}BayCon: Model-agnostic Bayesian counterfactual generator
Generating counterfactuals to discover hypothetical predictive scenarios is the de facto standard for explaining machine learning models and their predictions.
However, building a counterfactual explainer that is time-efficient, scalable, and model-agnostic, in addition to being compatible with continuous and categorical attributes, remains an open challenge.
To complicate matters even more, ensuring that the contrastive instances are optimised for feature sparsity, remain close to the explained instance, and are not drawn from outside of the data manifold, is far from trivial.
To address this gap we propose BayCon: a novel counterfactual generator based on probabilistic feature sampling and Bayesian optimisation.
Such an approach can combine multiple objectives by employing a surrogate model to guide the counterfactual search.
We demonstrate the advantages of our method through a collection of experiments based on six real-life datasets representing three regression tasks and three classification tasks.
@inproceedings{romashov2022baycon,
title={{BayCon}: {Model}-agnostic {Bayesian} counterfactual generator},
author={Romashov, Piotr and Gjoreski, Martin and Sokol, Kacper and
Martinez, Vanina and Langheinrich, Marc},
booktitle={Proceedings of the 31st International Joint Conference on
Artificial Intelligence},
publisher={IJCAI Organization},
pages={740--746},
year={2022},
doi={10.24963/ijcai.2022/104}
}2021
Explainability is in the mind of the beholder: Establishing the foundations of explainable artificial intelligence
Explainable artificial intelligence and interpretable machine learning are research domains growing in importance.
Yet, the underlying concepts remain somewhat elusive and lack generally agreed definitions.
While recent inspiration from social sciences has refocused the work on needs and expectations of human recipients, the field still misses a concrete conceptualisation.
We take steps towards addressing this challenge by reviewing the philosophical and social foundations of human explainability, which we then translate into the technological realm.
In particular, we scrutinise the notion of algorithmic black boxes and the spectrum of understanding determined by explanatory processes and explainees’ background knowledge.
This approach allows us to define explainability as (logical) reasoning applied to transparent insights (into, possibly black-box, predictive systems) interpreted under background knowledge and placed within a specific context – a process that engenders understanding in a selected group of explainees.
We then employ this conceptualisation to revisit strategies for evaluating explainability as well as the much disputed trade-off between transparency and predictive power, including its implications for ante-hoc and post-hoc techniques along with fairness and accountability established by explainability.
We furthermore discuss components of the machine learning workflow that may be in need of interpretability, building on a range of ideas from human-centred explainability, with a particular focus on explainees, contrastive statements and explanatory processes.
Our discussion reconciles and complements current research to help better navigate open questions – rather than attempting to address any individual issue – thus laying a solid foundation for a grounded discussion and future progress of explainable artificial intelligence and interpretable machine learning.
@article{sokol2021explainability,
title={Explainability is in the mind of the beholder: {Establishing}
the foundations of explainable artificial intelligence},
author={Sokol, Kacper and Flach, Peter},
journal={arXiv Preprint},
pages={1--26},
eprint={2112.14466},
archivePrefix={arXiv},
year={2021},
doi={10.48550/arXiv.2112.14466}
}You only write thrice: Creating documents, computational notebooks and presentations from a single source
Academic trade requires juggling multiple variants of the same content published in different formats:
manuscripts, presentations, posters and computational notebooks.
The need to track versions to accommodate for the write–review–rebut–revise life-cycle adds another layer of complexity.
We propose to significantly reduce this burden by maintaining a single source document in a version-controlled environment (such as
git), adding functionality to generate a collection of output formats popular in academia.
To this end, we utilise various open-source tools from the Jupyter scientific computing ecosystem and operationalise selected software engineering concepts.
We offer a proof-of-concept workflow that composes Jupyter Book (an online document), Jupyter Notebook (a computational narrative) and reveal.js slides from a single markdown source file.
Hosted on GitHub, our approach supports change tracking and versioning, as well as a transparent review process based on the underlying code issue management infrastructure.
An exhibit of our workflow can be previewed at https://so-cool.github.io/you-only-write-thrice/.@article{sokol2021you,
title={You only write thrice: {Creating} documents, computational
notebooks and presentations from a single source},
author={Sokol, Kacper and Flach, Peter},
journal={ICLR Workshop on Beyond Static Papers: Rethinking How We Share
Scientific Understanding in Machine Learning},
pages={1--7},
year={2021},
doi={10.48550/arXiv.2107.06639}
}Towards intelligible and robust surrogate explainers: A decision tree perspective
Artificial intelligence explainability and machine learning interpretability are relatively young and fast-growing research fields that may seem chaotic and difficult to navigate at times.
Despite these immense endeavours, a universally agreed terminology and evaluation criteria are still elusive, with many methods introduced to solve a commonly acknowledged yet undefined problem, and their success judged based on ad hoc measures.
To address this challenge and lay foundation for our research, we formalise explainability (our preferred term) as a technology providing insights that lead to understanding, which both defines such techniques and fixes their evaluation criterion.
While the premise is clear, understanding largely depends upon the explanation recipients, who come with a diverse range of background knowledge, mental models and expectations.
Therefore, in addition to technical requirements, explainability tools should also embody various social traits as their output is predominantly aimed at humans.
To tackle this duality and organise a comprehensive collection of relevant properties, we introduce a unified explainable artificial intelligence taxonomy, which is a principled framework for reasoning about explainers.
While most of our contributions are strictly technical, this formalisation allows us to develop them with a human component in mind, which leads us to consider explainability as a social, bi-directional process based on contrastive statements.
Stemming from this research direction is Glass-Box – a conversational explainer that empowers its users to customise and personalise explanations in a natural language dialogue.
@phdthesis{sokol2021towards,
title={Towards intelligible and robust surrogate explainers:
{A} decision tree perspective},
author={Sokol, Kacper},
year={2021},
school={University of Bristol},
type={Doctoral Thesis}
}2020
One explanation does not fit all: The promise of interactive explanations for machine learning transparency
The need for transparency of predictive systems based on Machine Learning algorithms arises as a consequence of their ever-increasing proliferation in the industry.
Whenever black-box algorithmic predictions influence human affairs, the inner workings of these algorithms should be scrutinised and their decisions explained to the relevant stakeholders, including the system engineers, the system’s operators and the individuals whose case is being decided.
While a variety of interpretability and explainability methods is available, none of them is a panacea that can satisfy all diverse expectations and competing objectives that might be required by the parties involved.
We address this challenge in this paper by discussing the promises of Interactive Machine Learning for improved transparency of black-box systems using the example of contrastive explanations – a state-of-the-art approach to Interpretable Machine Learning.
Specifically, we show how to personalise counterfactual explanations by interactively adjusting their conditional statements and extract additional explanations by asking follow-up “What if?” questions.
Our experience in building, deploying and presenting this type of system allowed us to list desired properties as well as potential limitations, which can be used to guide the development of interactive explainers.
While customising the medium of interaction, i.e., the user interface comprising of various communication channels, may give an impression of personalisation, we argue that adjusting the explanation itself and its content is more important.
To this end, properties such as breadth, scope, context, purpose and target of the explanation have to be considered, in addition to explicitly informing the explainee about its limitations and caveats.
Furthermore, we discuss the challenges of mirroring the explainee’s mental model, which is the main building block of intelligible human–machine interactions.
We also deliberate on the risks of allowing the explainee to freely manipulate the explanations and thereby extracting information about the underlying predictive model, which might be leveraged by malicious actors to steal or game the model.
Finally, building an end-to-end interactive explainability system is a challenging engineering task;
unless the main goal is its deployment, we recommend “Wizard of Oz” studies as a proxy for testing and evaluating standalone interactive explainability algorithms.
@article{sokol2020one,
title={One explanation does not fit all: {The} promise of interactive
explanations for machine learning transparency},
author={Sokol, Kacper and Flach, Peter},
journal={KI -- K{\"u}nstliche Intelligenz},
volume={34},
number={2},
pages={235--250},
year={2020},
publisher={Springer},
doi={10.1007/s13218-020-00637-y}
}FAT Forensics: A Python toolbox for implementing and deploying fairness, accountability and transparency algorithms in predictive systems
Predictive systems, in particular machine learning algorithms, can take important, and sometimes legally binding, decisions about our everyday life.
In most cases, however, these systems and decisions are neither regulated nor certified.
Given the potential harm that these algorithms can cause, their qualities such as fairness, accountability and transparency (FAT) are of paramount importance.
To ensure high-quality, fair, transparent and reliable predictive systems, we developed an open source Python package called FAT Forensics.
It can inspect important fairness, accountability and transparency aspects of predictive algorithms to automatically and objectively report them back to engineers and users of such systems.
Our toolbox can evaluate all elements of a predictive pipeline: data (and their features), models and predictions.
@article{sokol2020fat,
title={{FAT} {Forensics}: {A} {Python} toolbox for implementing and
deploying fairness, accountability and transparency algorithms
in predictive systems},
author={Sokol, Kacper and Hepburn, Alexander and Poyiadzi, Rafael and
Clifford, Matthew and Santos-Rodriguez, Raul and Flach, Peter},
journal={Journal of Open Source Software},
publisher={The Open Journal},
volume={5},
number={49},
pages={1904},
year={2020},
doi={10.21105/joss.01904}
}FACE: Feasible and actionable counterfactual explanations
Work in Counterfactual Explanations tends to focus on the principle of “the closest possible world” that identifies small changes leading to the desired outcome.
In this paper we argue that while this approach might initially seem intuitively appealing it exhibits shortcomings not addressed in the current literature.
First, a counterfactual example generated by the state-of-the-art systems is not necessarily representative of the underlying data distribution, and may therefore prescribe unachievable goals (e.g., an unsuccessful life insurance applicant with severe disability may be advised to do more sports).
Secondly, the counterfactuals may not be based on a “feasible path” between the current state of the subject and the suggested one, making actionable recourse infeasible (e.g., low-skilled unsuccessful mortgage applicants may be told to double their salary, which may be hard without first increasing their skill level).
These two shortcomings may render counterfactual explanations impractical and sometimes outright offensive.
To address these two major flaws, first of all, we propose a new line of Counterfactual Explanations research aimed at providing actionable and feasible paths to transform a selected instance into one that meets a certain goal.
Secondly, we propose FACE:
an algorithmically sound way of uncovering these “feasible paths” based on the shortest path distances defined via density-weighted metrics.
Our approach generates counterfactuals that are coherent with the underlying data distribution and supported by the “feasible paths” of change, which are achievable and can be tailored to the problem at hand.
@inproceedings{poyiadzi2020face,
title={{FACE}: {Feasible} and actionable counterfactual explanations},
author={Poyiadzi, Rafael and Sokol, Kacper and Santos-Rodriguez, Raul and
De Bie, Tijl and Flach, Peter},
booktitle={Proceedings of the 2020 AAAI/ACM Conference on AI, Ethics, and
Society},
publisher={ACM},
address={New York, NY, USA},
pages={344--350},
year={2020},
doi={10.1145/3375627.3375850}
}Explainability fact sheets: A framework for systematic assessment of explainable approaches
Explanations in Machine Learning come in many forms, but a consensus regarding their desired properties is yet to emerge.
In this paper we introduce a taxonomy and a set of descriptors that can be used to characterise and systematically assess explainable systems along five key dimensions:
functional, operational, usability, safety and validation.
In order to design a comprehensive and representative taxonomy and associated descriptors we surveyed the eXplainable Artificial Intelligence literature, extracting the criteria and desiderata that other authors have proposed or implicitly used in their research.
The survey includes papers introducing new explainability algorithms to see what criteria are used to guide their development and how these algorithms are evaluated, as well as papers proposing such criteria from both computer science and social science perspectives.
This novel framework allows to systematically compare and contrast explainability approaches, not just to better understand their capabilities but also to identify discrepancies between their theoretical qualities and properties of their implementations.
We developed an operationalisation of the framework in the form of Explainability Fact Sheets, which enable researchers and practitioners alike to quickly grasp capabilities and limitations of a particular explainable method.
When used as a Work Sheet, our taxonomy can guide the development of new explainability approaches by aiding in their critical evaluation along the five proposed dimensions.
@inproceedings{sokol2020explainability,
title={Explainability fact sheets: {A} framework for
systematic assessment of explainable approaches},
author={Sokol, Kacper and Flach, Peter},
booktitle={Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency},
publisher={ACM},
address={New York, NY, USA},
pages={56--67},
year={2020},
doi={10.1145/3351095.3372870}
}2019
bLIMEy: Surrogate prediction explanations beyond LIME
Surrogate explainers of black-box machine learning predictions are of paramount importance in the field of eXplainable Artificial Intelligence since they can be applied to any type of data (images, text and tabular), are model-agnostic and are post-hoc (i.e., can be retrofitted).
The Local Interpretable Model-agnostic Explanations (LIME) algorithm is often mistakenly unified with a more general framework of surrogate explainers, which may lead to a belief that it is the solution to surrogate explainability.
In this paper we empower the community to “build LIME yourself” (bLIMEy) by proposing a principled algorithmic framework for building custom local surrogate explainers of black-box model predictions, including LIME itself.
To this end, we demonstrate how to decompose the surrogate explainers family into algorithmically independent and interoperable modules and discuss the influence of these component choices on the functional capabilities of the resulting explainer, using the example of LIME.
@article{sokol2019blimey,
title={{bLIMEy}: {Surrogate} prediction explanations beyond {LIME}},
author={Sokol, Kacper and Hepburn, Alexander and
Santos-Rodriguez, Raul and Flach, Peter},
journal={NeurIPS Workshop on Human-Centric Machine Learning},
pages={1--10},
year={2019},
doi={10.48550/arXiv.1910.13016}
}HyperStream: A workflow engine for streaming data
This paper describes HyperStream, a large-scale, flexible and robust software package, written in the Python language, for processing streaming data with workflow creation capabilities.
HyperStream overcomes the limitations of other computational engines and provides high-level interfaces to execute complex nesting, fusion, and prediction both in online and offline forms in streaming environments.
HyperStream is a general purpose tool that is well-suited for the design, development, and deployment of Machine Learning algorithms and predictive models in a wide space of sequential predictive problems.
@article{diethe2019hyperstream,
title={{HyperStream}: {A} workflow engine for streaming data},
author={Diethe, Tom and Kull, Meelis and Twomey, Niall and
Sokol, Kacper and Song, Hao and Perello-Nieto, Miquel and
Tonkin, Emma and Flach, Peter},
journal={arXiv Preprint},
pages={1--7},
eprint={1908.02858},
archivePrefix={arXiv},
year={2019},
doi={10.48550/arXiv.1908.02858}
}Desiderata for interpretability: Explaining decision tree predictions with counterfactuals
Explanations in machine learning come in many forms, but a consensus regarding their desired properties is still emerging.
In our work we collect and organise these explainability desiderata and discuss how they can be used to systematically evaluate properties and quality of an explainable system using the case of class-contrastive counterfactual statements.
This leads us to propose a novel method for explaining predictions of a decision tree with counterfactuals.
We show that our model-specific approach exploits all the theoretical advantages of counterfactual explanations, hence improves decision tree interpretability by decoupling the quality of the interpretation from the depth and width of the tree.
@inproceedings{sokol2019desiderata,
title={Desiderata for interpretability: {Explaining} decision tree
predictions with counterfactuals},
author={Sokol, Kacper and Flach, Peter},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence:
Student Abstract Track},
publisher={AAAI Press},
address={Washington, DC, USA},
volume={33},
number={01},
pages={10035--10036},
year={2019},
doi={10.1609/aaai.v33i01.330110035}
}Counterfactual explanations of machine learning predictions: Opportunities and challenges for AI safety
One necessary condition for creating a safe AI system is making it transparent to uncover any unintended or harmful behaviour.
Transparency can be achieved by explaining predictions of an AI system with counterfactual statements, which are becoming a de facto standard in explaining algorithmic decisions.
The popularity of counterfactuals is mainly attributed to their compliance with the “right to explanation” introduced by the European Union’s General Data Protection Regulation and them being understandable by a lay audience as well as domain experts.
In this paper we describe our experience and the lessons learnt from explaining decision tree models trained on UCI German Credit and FICO Explainable Machine Learning Challenge data sets with class-contrastive counterfactual statements.
We review how counterfactual explanations can affect an artificial intelligence system and its safety by investigating their risks and benefits.
We show example explanations, discuss their strengths and weaknesses, show how they can be used to debug the underlying model, inspect its fairness and unveil security and privacy challenges that they pose.
@inproceedings{sokol2019counterfactual,
title={Counterfactual explanations of machine learning predictions:
{Opportunities} and challenges for {AI} safety},
author={Sokol, Kacper and Flach, Peter},
booktitle={Proceedings of the AAAI Workshop on Artificial Intelligence
Safety 2019 co-located with the 33rd AAAI Conference on
Artificial Intelligence 2019 (AAAI 2019)},
publisher={CEUR Workshop Proceedings},
pages={1--4},
volume={2301},
year={2019}
}Fairness, accountability and transparency in artificial intelligence: A case study of logical predictive models
Machine learning – the part of artificial intelligence aimed at eliciting knowledge from data and automated decision making without explicit instructions – is making great strides, with new algorithms being invented every day.
These algorithms find myriads of applications, but their ubiquity often comes at the expense of limited interpretability, hidden biases and unexpected vulnerabilities.
Whenever one of these factors is a priority, the learning algorithm of choice is often a method considered to be inherently interpretable, e.g. logical models such as decision trees.
In my research I challenge this assumption and highlight (quite common) cases when the assumed interpretability fails to deliver.
To restore interpretability of logical machine learning models (decision trees and their ensembles in particular) I propose to explain them with class-contrastive counterfactual statements, which are a very common type of explanation in human interactions, well-grounded in social science research.
To evaluate transparency of such models I collate explainability desiderata that can be used to systematically assess and compare such methods as an addition to user studies.
Given contrastive explanations, I investigate their influence on the model’s security, in particular gaming and stealing the model.
Finally, I evaluate model fairness, where I am interested in choosing the most fair model among all the models with equal performance.
@inproceedings{sokol2019fairness,
title={Fairness, accountability and transparency in artificial intelligence:
{A} case study of logical predictive models},
author={Sokol, Kacper},
booktitle={Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and
Society: Doctoral Track},
publisher={Association for Computing Machinery},
address={New York, NY, USA},
pages={541--542},
year={2019},
doi={10.1145/3306618.3314316}
}2018
Releasing eHealth analytics into the wild: Lessons learnt from the SPHERE project
The SPHERE project is devoted to advancing eHealth in a smart-home context, and supports full-scale sensing and data analysis to enable a generic healthcare service.
We describe, from a data-science perspective, our experience of taking the system out of the laboratory into more than thirty homes in Bristol, UK.
We describe the infrastructure and processes that had to be developed along the way, describe how we train and deploy Machine Learning systems in this context, and give a realistic appraisal of the state of the deployed systems.
@inproceedings{diethe2018releasing,
title={Releasing {eHealth} analytics into the wild: {Lessons} learnt from
the {SPHERE} project},
author={Diethe, Tom and Holmes, Mike and Kull, Meelis and
Perello-Nieto, Miquel and Sokol, Kacper and Song, Hao and
Tonkin, Emma and Twomey, Niall and Flach, Peter},
booktitle={Proceedings of the 24th ACM SIGKDD International Conference on
Knowledge Discovery \& Data Mining},
pages={243--252},
year={2018},
publisher={ACM},
address={New York, NY, USA},
doi={10.1145/3219819.3219883}
}Conversational explanations of machine learning predictions through class-contrastive counterfactual statements
Machine learning models have become pervasive in our everyday life;
they decide on important matters influencing our education, employment and judicial system.
Many of these predictive systems are commercial products protected by trade secrets, hence their decision-making is opaque.
Therefore, in our research we address interpretability and explainability of predictions made by machine learning models.
Our work draws heavily on human explanation research in social sciences:
contrastive and exemplar explanations provided through a dialogue.
This user-centric design, focusing on a lay audience rather than domain experts, applied to machine learning allows explainees to drive the explanation to suit their needs instead of being served a precooked template.
@inproceedings{sokol2018conversational,
title={Conversational explanations of machine learning predictions
through class-contrastive counterfactual statements},
author={Sokol, Kacper and Flach, Peter},
booktitle={Proceedings of the 27th International Joint Conference on
Artificial Intelligence: Doctoral Consortium},
publisher={IJCAI Organization},
pages={5785--5786},
year={2018},
doi={10.24963/ijcai.2018/836}
}Glass-Box: Explaining AI decisions with counterfactual statements through conversation with a voice-enabled virtual assistant
The prevalence of automated decision making, influencing important aspects of our lives – e.g., school admission, job market, insurance and banking – has resulted in increasing pressure from society and regulators to make this process more transparent and ensure its explainability, accountability and fairness.
We demonstrate a prototype voice-enabled device, called Glass-Box, which users can question to understand automated decisions and identify the underlying model’s biases and errors.
Our system explains algorithmic predictions with class-contrastive counterfactual statements (e.g., “Had a number of conditions been different: …the prediction would change…”), which show a difference in a particular scenario that causes an algorithm to “change its mind”.
Such explanations do not require any prior technical knowledge to understand, hence are suitable for a lay audience, who interact with the system in a natural way – through an interactive dialogue.
We demonstrate the capabilities of the device by allowing users to impersonate a loan applicant who can question the system to understand the automated decision that he received.
@inproceedings{sokol2018glass,
title={{Glass-Box}: {Explaining} {AI} decisions with counterfactual
statements through conversation with a voice-enabled virtual
assistant},
author={Sokol, Kacper and Flach, Peter},
booktitle={Proceedings of the 27th International Joint Conference on
Artificial Intelligence: Demos},
publisher={IJCAI Organization},
pages={5868--5870},
year={2018},
doi={10.24963/ijcai.2018/865}
}2017
Conference report: International Joint Conference on Artificial Intelligence (IJCAI) 2017
@article{sokol2017conference,
author={Sokol, Kacper},
title={Conference report: {International} {Joint} {Conference} on
{Artificial} {Intelligence} ({IJCAI}) 2017},
journal={AISB Quarterly},
year={2017},
month={November},
volume={148},
pages={9--11}
}The role of textualisation and argumentation in understanding the machine learning process
Understanding data, models and predictions is important for machine learning applications.
Due to the limitations of our spatial perception and intuition, analysing high-dimensional data is inherently difficult.
Furthermore, black-box models achieving high predictive accuracy are widely used, yet the logic behind their predictions is often opaque.
Use of textualisation – a natural language narrative of selected phenomena – can tackle these shortcomings.
When extended with argumentation theory we could envisage machine learning models and predictions arguing persuasively for their choices.
@inproceedings{sokol2017role,
title={The role of textualisation and argumentation in understanding
the machine learning process},
author={Sokol, Kacper and Flach, Peter},
booktitle={Proceedings of the 26th International Joint Conference on
Artificial Intelligence: Doctoral Consortium},
publisher={IJCAI Organization},
pages={5211--5212},
year={2017},
doi={10.24963/ijcai.2017/765}
}The role of textualisation and argumentation in understanding the machine learning process: A position paper
Understanding data, models and predictions is important for any machine learning application.
Due to the limitations of our spatial perception and intuition, analysing high-dimensional data is inherently difficult.
Furthermore, black-box models achieving high predictive accuracy are widely used, yet the logic behind their predictions is often opaque.
Use of textualisation – a natural language narrative of selected phenomenon – can tackle these shortcomings.
When extended with argumentation theory we could envisage machine learning models and predictions arguing persuasively for their choices.
@inproceedings{sokol2017roleARW,
author={Sokol, Kacper and Flach, Peter},
title={The role of textualisation and argumentation in understanding
the machine learning process: {A} position paper},
booktitle={Automated Reasoning Workshop},
publisher={Department of Computer Science, University of Bristol},
pages={11--12},
year={2017}
}2016
Activity recognition in multiple contexts for smart-house data
Predicting human behaviour from smart-house data is an active and challenging area of research.
Applications include improving quality of life and personal healthcare, e.g. smart thermostats and detecting falls.
Smart-house technology is becoming increasingly popular and in many cases making a house smarter is as simple as installing off-the-shelf devices
In this paper we focus on predicting the future location of a person in a simulated smart-house environment.
We consider three different occupant types who exhibit various working patterns.
We develop a versatile model capable of adjusting to significant changes in a person’s behaviour without the need of model retraining.
To this end, we build a simple event calculus framework based on the Aleph Inductive Logic Programming system.
Event calculus helps to handle time and persisting sensor states.
Background knowledge encodes important information about the smart-house that is otherwise difficult to learn;
it also facilitates transferability of the model to different house layouts.
Moreover, rule models are white-box, hence human-readable.
Finally, we show that a versatile model performs significantly better than other models that do not explicitly account for the context.
@inproceedings{sokol2016activity,
title={Activity recognition in multiple contexts for smart-house data},
author={Sokol, Kacper and Flach, Peter},
booktitle={Proceedings of the 26th International Conference on
Inductive Logic Programming (Short Papers)},
pages={66--72},
publisher={CEUR Workshop Proceedings},
volume={1865},
year={2016}
}2014
Making sense of data streams: Complex event processing for controls applications
This project aims at building a tool to process live streams of data produced by various sensors and artificial generators.
To this end, a Java code is written, which uses Esper Complex Event Processing package to: receive data feeds, apply user defined rules and filters, and pass the resulting information to a clustering framework.
The last step employs Affinity Propagation based clustering algorithm, which choice is motivated by its dynamic adaptation to number of clusters in the data.
This is key feature in data streaming scenario as the number of clusters can evolve with time.
Furthermore, [Zhang et al., 2013] have documented overall good performance of Affinity Propagation in cases of live data analysis.
Finally, presented here approach is compared and contrasted against static clustering algorithms applied to data gathered from streams incoming over one run of the program, followed by in-depth results analysis.
@techreport{sokol2014making,
author={Sokol, Kacper and Tilaro, Filippo and Voitier, Axel},
title={Making sense of data streams: {Complex} event processing for
controls applications},
institution={CERN openlab Report},
doi={10.5281/zenodo.12628},
year={2014}
}