Who’s the Fairest of Them All? A Comparison of Methods for Classifying Tone and Causal Reasoning in Earnings-related Management Discourse

62 Pages Posted: 2 Dec 2020 Last revised: 8 Mar 2021

See all articles by Martin Walker

Martin Walker

University of Manchester - Manchester Business School

Steven Young

Lancaster University - Department of Accounting and Finance

Andrew Moore

Lancaster University - School of Computing and Communciations

Vasiliki E. Athanasakou

Saint Mary's University, Canada - Sobey School of Business

Paul Rayson

Lancaster University

Thomas Schleicher

University of Manchester - Manchester Business School

Date Written: October 30, 2020

Abstract

We compare the performance of machine learning algorithms and wordlists at replicating manual coding of sentence-level tone and attribution in earnings press releases. We train learning algorithms on a sample of manually annotated performance sentences and assess accuracy using a separate manually annotated holdout sample. Key findings are as follows. All methods detect negative sentences with lower accuracy than positive sentences. None of the approaches detect the presence of causal reasoning with high accuracy. Conditional on identifying a causal reasoning sentence manually, learning algorithms (but not wordlists) are able to distinguish between internal and external attributions. Absolute measurement errors exceed 20% for even the most reliable classification tasks such as tone and attribution type. No approach displays superior performance across all classification tasks and Naïve Bayes consistently underperforms other algorithms. Finally, even the best performing combination of classifiers struggles to detect self-attribution bias that is clearly evident with manual coding. We conclude that big data methods are not necessarily best for analyzing financial discourse, and that the value of manual coding should not be underestimated.

Keywords: Machine learning, text classification, manual scoring

JEL Classification: M40

Suggested Citation

Walker, Martin and Young, Steven and Moore, Andrew and Athanasakou, Vasiliki E. and Rayson, Paul and Schleicher, Thomas, Who’s the Fairest of Them All? A Comparison of Methods for Classifying Tone and Causal Reasoning in Earnings-related Management Discourse (October 30, 2020). Available at SSRN: https://ssrn.com/abstract=3721976 or http://dx.doi.org/10.2139/ssrn.3721976

Martin Walker

University of Manchester - Manchester Business School ( email )

Booth Street West
Manchester, M15 6PB
United Kingdom

Steven Young (Contact Author)

Lancaster University - Department of Accounting and Finance ( email )

The Management School
Lancaster LA1 4YX
United Kingdom
+441 5245-94242 (Phone)
+441 5248-47321 (Fax)

Andrew Moore

Lancaster University - School of Computing and Communciations

Bailrigg
Lancaster, LA1 4YX
United Kingdom

Vasiliki E. Athanasakou

Saint Mary's University, Canada - Sobey School of Business ( email )

Sobey Building 311
923 Robie Street
Halifax, Nova Scotia B3H 3C3
Canada

Paul Rayson

Lancaster University ( email )

School of Computing and Communications
Lancaster LA1 4YX
United Kingdom

Thomas Schleicher

University of Manchester - Manchester Business School ( email )

Booth Street West
Manchester, M15 6PB
United Kingdom

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
119
Abstract Views
595
rank
281,505
PlumX Metrics