Banking on Language: A New Approach to Financial Data

Author: Denis Avetisyan

Researchers have developed a novel model that understands financial transactions by treating them as natural language, unlocking insights from limited data.

The distribution of downstream task performance across all metrics demonstrates the inherent variability in model outcomes, highlighting that even seemingly successful systems exhibit a range of capabilities and limitations.

This work introduces a multimodal foundational model for learning robust transaction embeddings through self-supervised learning on a large private banking dataset.

Despite the increasing volume of financial data, effectively integrating structured transactions with rich, yet often untapped, textual descriptions remains a significant challenge. This is addressed in ‘Open Banking Foundational Model: Learning Language Representations from Few Financial Transactions’, which introduces a novel multimodal approach to representing financial transactions as sentences, combining tabular data and descriptive text. Our results demonstrate that this model, trained via self-supervised learning, outperforms traditional methods-particularly in data-scarce open banking environments-and generalizes across diverse financial institutions. Could this represent a crucial step towards unlocking more comprehensive and insightful financial applications, from fraud detection to personalized customer experiences?

The Illusion of Insight: Data’s Broken Promises

Financial institutions are undergoing a significant transformation, increasingly dependent on the detailed analysis of individual transaction data to gain a comprehensive understanding of customer behavior and proactively manage financial risk. This reliance stems from the recognition that patterns hidden within spending habits, payment frequencies, and transaction types can reveal crucial insights – from identifying potential fraud and money laundering to predicting customer churn and personalizing financial products. The shift represents a move beyond traditional demographic-based risk assessments, allowing for a more nuanced and dynamic evaluation of each account. Consequently, the ability to effectively process and interpret this granular data is no longer simply advantageous, but rather a critical component of maintaining a competitive edge and safeguarding financial stability within the modern banking landscape.

Financial institutions aiming to decipher customer behavior through transaction data frequently encounter significant hurdles beyond simple data collection. Access to complete transaction histories is often restricted, and even when available, the sheer volume of useful data can be surprisingly low-many accounts contain fewer than 120 transactions, creating statistical challenges. Further complicating matters is the issue of imbalanced events; fraudulent transactions, for example, represent a tiny fraction of overall activity, meaning standard analytical techniques can struggle to accurately identify them. These combined limitations-scarcity, restricted access, and imbalance-demand innovative approaches to data handling and modeling if institutions hope to effectively leverage this valuable, yet challenging, resource.

Conventional feature engineering, while effective for static datasets, often falls short when applied to the dynamic nature of financial transactions. These methods typically extract isolated characteristics – such as transaction frequency or average amount – failing to account for the temporal dependencies inherent in sequential data. A purchase made on a Friday, for example, carries different implications than one made on a Monday, a nuance lost in many traditional approaches. This limitation hinders the ability to accurately model customer behavior and detect anomalies, as the predictive power relies heavily on understanding the order and context of each transaction. Consequently, institutions face challenges in fraud detection, risk assessment, and personalized financial services, necessitating more sophisticated techniques capable of capturing these complex, time-dependent relationships within the data.

Vectorizing Transactions: A New Hope (Probably)

Transaction embeddings transform individual financial transactions into numerical vectors, typically with dimensions ranging from 32 to 128, allowing computational analysis of transactional data. These vectors are not simply encodings of transaction amounts; they capture nuanced characteristics such as merchant category, time of day, and sequential relationships with prior transactions. This vector representation facilitates the application of machine learning algorithms, enabling tasks like fraud detection, customer segmentation, and anomaly detection that are difficult or impossible with traditional tabular data. The density of these vectors – meaning most values are non-zero – allows for efficient similarity comparisons and pattern recognition, effectively capturing the complex interplay of factors defining each transaction’s characteristics and its relationship to other transactions within a sequence.

Transaction embeddings are created by utilizing models designed to process sequential data, specifically adapting techniques from Natural Language Processing. Models like BERT, initially developed for text, are applied to the ordered series of financial transactions associated with a user or entity. CoLES (Contrastive Learning for Event Sequences) is another technique that explicitly focuses on learning representations from event sequences; it learns to distinguish relevant patterns within transaction histories by contrasting similar and dissimilar sequences. Both approaches capitalize on the temporal relationships inherent in transaction data, representing each transaction, or sequence of transactions, as a dense vector that captures its characteristics and relationships to others.

Contrastive learning, as implemented in the CoLES model, improves transaction embedding quality by training the model to differentiate between positive and negative pairs of transaction sequences. This is achieved by minimizing the distance between embeddings of similar, or “positive,” sequences – representing legitimate user behavior – and maximizing the distance between embeddings of dissimilar, or “negative,” sequences – often indicative of fraudulent activity or anomalous behavior. The process relies on defining a contrastive loss function that penalizes embeddings that fail to reflect these distinctions, thereby forcing the model to learn robust representations that capture meaningful patterns within transaction data. This approach is particularly effective because it does not require labeled data, instead learning from the inherent structure of transaction sequences to identify and emphasize differentiating features.

Downstream Tasks: Where the Rubber Meets the Algorithm

Transaction embeddings facilitate a variety of analytical applications within financial services and beyond. In fraud detection, these embeddings represent transactions as vectors, allowing algorithms to identify anomalous patterns indicative of fraudulent activity. For credit risk assessment, embeddings capture transactional behavior to provide a more granular view of an applicant’s creditworthiness beyond traditional credit scores. Customer segmentation benefits from embeddings by grouping customers based on their spending habits and transactional patterns, enabling targeted marketing and personalized services. These applications leverage the ability of embeddings to distill complex transactional data into a quantifiable format suitable for machine learning models.

Transaction embeddings facilitate improved performance when used as input features in a variety of machine learning models. Testing across 19 distinct downstream tasks – encompassing areas like risk assessment and customer profiling – demonstrated consistent gains in predictive accuracy when compared to models trained without these embeddings. Specifically, linear regression, alongside more complex algorithms, benefitted from the dense vector representations, suggesting the embeddings effectively capture relevant transactional information for enhanced model generalization and discrimination. This compatibility extends to numerous model types, simplifying integration into existing machine learning pipelines.

Integrating geolocation data with transaction embeddings allows for more detailed analytical capabilities. Specifically, the inclusion of geographic coordinates as external data points provides contextual information regarding transaction origins and potential anomalies. This enrichment facilitates the identification of patterns related to location-based fraud, regional risk assessments, and more precise customer segmentation based on purchasing behavior tied to specific geographic areas. The addition of geolocation data consistently improves the performance of downstream machine learning tasks by providing a more complete feature set for analysis.

The Illusion of Progress: Chasing Signals in a Noisy World

Financial institutions are increasingly leveraging transaction embeddings – numerical representations of financial transactions – to build more effective and scalable intelligence systems. Recent advancements, particularly the application of models like DistilBERT, have unlocked new possibilities in analyzing these complex data streams. This approach was validated through training on a substantial dataset encompassing the transaction histories of 10 million individual bank accounts, allowing for the identification of subtle patterns and anomalies previously obscured by data limitations. The result is a system capable of not only processing vast quantities of financial data with greater efficiency but also delivering more nuanced insights, ultimately supporting improved risk management and informed decision-making within the financial sector.

Financial institutions are increasingly leveraging advanced natural language processing to enhance their analytical capabilities, particularly in the face of historically challenging data limitations. Traditional methods often struggle with the scarcity of labeled financial data and the prevalence of imbalanced events-where fraudulent or unusual transactions represent a tiny fraction of overall activity. Recent studies demonstrate that models like DistilBERT and BERT consistently surpass the performance of conventional techniques, such as CoLES and manual feature engineering, across a diverse set of nineteen downstream tasks including fraud detection, anti-money laundering, and risk assessment. This improved performance isn’t merely incremental; the consistent outperformance suggests these models are capable of extracting more nuanced patterns and signals from transaction data, ultimately empowering institutions to make more informed decisions and proactively mitigate financial risks.

The rise of open banking is fundamentally reshaping the landscape of financial services by granting consumers greater control over their data and enabling secure data sharing with authorized third-party providers. This access to detailed account histories – traditionally siloed within individual institutions – unlocks a wealth of opportunities for innovation, moving beyond basic transactional data to offer truly personalized financial insights. Applications range from automated budgeting tools and proactive fraud detection to customized loan offerings and investment strategies tailored to individual spending habits and financial goals. By fostering a more competitive and collaborative ecosystem, open banking not only empowers consumers but also drives the development of increasingly sophisticated and responsive financial technologies, ultimately leading to more efficient and inclusive financial systems.

The pursuit of elegant models, as evidenced by this work on transaction embeddings, feels… familiar. It’s a predictable cycle. Researchers build these sophisticated frameworks – multimodal foundational models, in this case – hoping to capture the nuances of financial transactions. They tout performance gains, especially in low-data regimes. And production, inevitably, will unearth the edge cases. It always does. As Henri Poincaré observed, “Mathematics is the art of giving reasons, even when one has no right to do so.” The same could be said for machine learning; building a compelling narrative around a model doesn’t guarantee it will hold up when faced with the relentless chaos of real-world financial data. This paper’s focus on representing transactions as ‘sentences’ is a clever abstraction, but it’s only a matter of time before production finds a way to break the grammar.

What’s Next?

The enthusiasm for ‘foundational models’ in finance will, predictably, outpace their actual foundation. This work, translating transaction data into something resembling natural language, is a logical step-and a perfect candidate for eventual performance degradation as data distributions shift. The authors demonstrate gains in low-data regimes, which is encouraging, but production systems rarely stay low-data. The real test will be observing how quickly these learned representations decay under the weight of real-world noise and adversarial attacks – because someone will try to game the system.

The paper correctly identifies the multimodal aspect as crucial. However, simply combining tabular data and text is a starting point, not a solution. The true complexity lies in integrating all the modalities of banking-customer interactions, regulatory filings, macroeconomic indicators, the whims of the market. Each new input will require another layer of ‘representation learning’, another potential point of failure.

Better one well-understood heuristic than a black box claiming to ‘understand’ finance. The field seems determined to prove this wrong, but the logs will, eventually, tell the tale. The pursuit of truly generalizable financial models will continue, of course. It always does. But let’s not pretend that ‘scalability’ isn’t just a polite way of saying ‘untested’.

Original article: https://arxiv.org/pdf/2511.12154.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

The Illusion of Insight: Data’s Broken Promises

Vectorizing Transactions: A New Hope (Probably)

Downstream Tasks: Where the Rubber Meets the Algorithm

The Illusion of Progress: Chasing Signals in a Noisy World

What’s Next?

See also: