Making ODR Human

• 1 Introduction/Overview

  This article discusses what we as human-centred design practitioners have learnt from researching and designing online dispute resolution (ODR) products both for clients and as part of our internal research and development initiatives.

  1.1 What Is Human-Centred Design?

  If design is the process of creatively and strategically solving problems, human-centred design is an approach that puts the people affected by the problems at the centre of the solution.¹x See https://www.designorate.com/characteristics-of-human-centered-design/. It begins with a focus on the people who would use a service, system, product or technology to understand the best way to meet their needs and then iteratively experiment to develop a solution.²x See T. Brown, Design Thinking, Harvard Business Review, June 2008. The approach has been applied successfully to complex problems in a range of fields, from providing interventions to make courts more user friendly³xM. Hagan, ‘A Human-Centered Design Approach to Access to Justice: Generating New Prototypes and Hypotheses for Intervention to Make Courts User-Friendly’, Indiana Journal of Law and Social Equality, Vol. 6, No 2, 2018, Article 2. to informing solutions to help prevent chronic diseases.⁴xMatheson et al., ‘Leveraging Human-Centered Design in Chronic Disease Prevention’, American Journal of Preventive Medicine, Vol. 48, No. 4, 2015, pp. 472-479.

  1.2 Our Approach to Human-Centred Design

  At Portable, we implemented human-centred design in our ODR product development using a variety of methods in our toolkit, including user interviews, co-design workshops and user testing. In our human-centred design process, we empathize with the user to understand them and the problem, generate ideas to solve the problem, prototype solutions to test and then learn and iterate according to how effectively the solution addresses user needs.
  We do these things to speak to, and learn from, users. Our ultimate aim is to understand them and their problems better. We may use this research to create user personas, journey maps and empathy maps, or we may simply learn that we were not speaking to the right people and that we need to revise our approach.
  This is because human-centred design is not just about tools. Our process requires openness and curiosity to make it possible. We believe designers need to be:

  • Empathetic: to connect and resonate with the real people

  • Integrative: so we can embrace complexity

  • Optimistic: in order to see possibility and opportunity

  • Collaborative: to bring others on the journey

  • Experimental: so that we can design minimally and fail joyfully

  Portable has applied human-centred design practices in many fields, including, but not limited to:

  • improving the response to family violence initiatives for the Queensland Government’s Keeping Women Safe in Their Home initiative

  • ideating a concept for a new business service for the Foundation for Young Australians initiative

  • enhancing the before-court and day-of-court experience for family violence victim-survivors for the Victorian Government Department of Premier and Cabinet and the Australian Government Department of Social Services

  • improving the process for politicians to manage their expenses for the Australian Government Department of Finance, and

  • improve training experiences for small businesses in Victoria for the Department of Economic Development, Jobs, Transport and Resources.

• 2 Applying Human-Centred Design to ODR

  2.1 How We Got Here

  We have been involved in justice sector reform since 2012. It forms a cornerstone of how we, as designers and developers, can respond to social need and policy failure. Our initial work was to build the first online Family Violence Intervention Order in Victoria, which was cited in the Royal Commission into Family Violence and emboldened us to actively work to put users at the centre of what we do. Since then, we have looked at how technology can be used to improve the before and after, as well as the day-of-court experience for citizens, co-designing initiatives for both as we are currently rolling out new products in Australia.
  We then looked at ODR as a logical next step. After researching the successes and challenges of platforms like MyLaw BC and Rechtwijzer, we saw an opportunity to build accessible, cost-effective and user-centred methods for resolving disputes with technology. A combination of research and informal chats with government clients led us to embark on our own internal R&D on what we could do to improve this experience for users. This led to a generic ODR workflow that formed the beginning of our internal R&D process. Mapping this dispute resolution workflow opened up the potential for opportunities to better understand the way people communicate in order to put their experience at the centre.

  2.2 Sort’d

  Our first ODR prototype, Sort’d, was built with user-centred design principles in mind. We wanted to make something that combined low-fi visual design and plain language communication with innovative technology (like tone analysis) to aid users in resolving their own problems. We were also able to test the concept of a guided assistant within the platform to offer contextual help and tips, as well as gradual disclosure of evidence to support claims based on the seriousness of the issue.
  Sort’d works in three rounds to help resolve personal disputes between two parties. Users progressively disclose information in each round in order to clarify the matter in dispute and create summaries of the information by cooperating with the other party.
  We wanted to incorporate what we liked from more traditional dispute resolution processes (like blind bidding) with a holistic, values-based approach by asking users to consider what they have to lose if the matter is not resolved. Each round included a natural language processing tool that identified rude, aggressive or accusatory speech patterns and checked for spelling and grammar. If a claim was made, users were prompted to support their claim through uploads or text explanations. At the end of the second stage, users are invited to sense check their claims with a third party. If users have not reached agreement by the third round, they are invited to ask themselves, ‘What’s important to you in the big picture? What might you lose by not reaching agreement?’ In informal user testing, we found this to be an invaluable way to help disputants to reconsider their entrenched positions.

• 3 The Victorian Civil and Administrative Tribunal (VCAT) Small Civil Claims

  3.1 Background

  In 2016, the Access to Justice report identified the potential to modernize and simplify user services at the Victorian Civil and Administrative Tribunal (VCAT), and recommendation 5.2 specifically included the need to “develop an online system for the resolution of small civil claims”.⁵xGovernment of Victoria, Access to Justice Review: Report and Recommendations, Volume 1, 2016, p. 281. Small civil claims are uniquely suited to ODR in that they are generally straightforward fact scenarios, have limited monetary values, and are usually only between two parties.⁶x Ibid., p. 277. This recommendation informed the 2018-2022 VCAT strategy, and VCAT organized a pilot for 2018 to test their hypothesis that ODR will improve access to justice for Victorians using small civil claims as a test case.
  In early 2018, VCAT contracted Portable to research the desirability of ODR by conducting user interviews and developing personas and journey maps. These visual research pieces were designed to help uncover user needs and limitations and prioritize user target groups.

  3.2 VCAT Desirability Study

  Before conducting user research, we identified existing barriers to accessing VCAT through desktop research, including identifying unique challenges experienced by people living with disabilities, cultural and linguistic barriers, access to services for people living remotely, and people who struggle with competing responsibilities due to work or carer status. Where necessary, we used customer satisfaction surveys, annual reports and ABS census reports to draw insights.
  In total, we conducted 34 interviews with different sections of the customer base, staff, members, and other stakeholders. On the basis of our interviews, we developed empathy maps of a registrar and member and high-fidelity personas of a small business owner applicant, a culturally and linguistically diverse (CALD) applicant, an Aboriginal customer, a potential applicant and a respondent. We also completed a high-fidelity journey map of the small business owner applicant, Jack Birch, which was used as the persona for the pilot.

  3.3 What We Learnt

  Not all problems can be fixed with technology. Some problems that users of a service like VCAT experience are much deeper and more complex. Using a human-centred design approach also showed us that technology is not the answer to every problem experienced by users of a service. In this case, there were wider issues that would have required more extensive service design engagements to resolve instead of imagining that ODR would automatically be adopted by this user group. The promise of a faster resolution does not resolve distrust of the justice systems by marginalized communities or resolve cultural and linguistic barriers. The needs we wanted to address through ODR would help some of these users but were not the best solution for others.
  This meant we had to prioritize our user base for the pilot through our personas and journey map. After conducting the interviews, we determined that small business owners who repeatedly use VCAT were the best user group to conduct the pilot with, given that:

  • they have the best understanding of small civil claims process

  • they represent a significant number of applicants

  • the majority of small business applicants attend a non-contested hearing

  • they generally make multiple small claims for unpaid debts at once

  • they are expected to lose up to two-thirds of their disputed amount in recovery costs during the dispute resolution process, either through filing feeds or lost time and resources

  What we learnt from our interviews with VCAT customers is that almost everyone would resolve their disputes online if it helped them reduce the time, cost and complexity of a legal issue. Specifically, interview participants would use ODR if it helped them to:

  • avoid travelling to a court or tribunal

  • receive language and educational support

  • ensure the other party responds

  • integrate their supporting documents

  • facilitate a timely resolution of their matter

  Staff generally liked the idea of implementing an ODR system in their service provision as long as it did not create additional or duplicating processes that would add to their already backlogged caseload. There were concerns about integration with a front-end system as well, since the case management system used by the civil list was outdated and struggled to integrate new services like the online application portal. The members we interviewed shared concerns about procedural fairness, reliability of evidence and testimony, and technological barriers preventing fair hearings. Overall, the benefits of ODR uptake for VCAT small civil claims list were perceived to vastly outweigh the challenges of taking on a new service offering.

• 4 Legal Services Commission South Australia

  4.1 Background

  More than 46,000 divorces were granted in Australia⁷x Marriages and Divorces, Australia, 3310.0, 2016, available at: www.abs.gov.au/ausstats/abs@.nsf/mf/3310.0. in 2016 and more than 14,000 consent order applications filed in the Family Court.⁸xFamily Court of Australia, Annual Report 2016-2017, available at: www.familycourt.gov.au/wps/wcm/connect/7456589e-fc98-409b-8d1c-7b909bd30dcf/2743-Family_Court_of_Australia_AnnualReport_2016_17_WEB.pdf?MOD=AJPERES&CVID=. These large caseloads have increased the length, cost and complexity of resolving disputes through formal legal processes, as noted in the Australian Law Reform Commission’s Review of the Family Law System.⁹xAustralian Law Reform Commission, Review of the Family Law System (Discussion Paper 86), October 2018. Disputes arising from the breakdown of a relationship can be particularly expensive for people with relatively few resources. Often, formalities like parenting plans or property settlement arrangements are ignored until a serious problem arises owing to cost and complexity.
  The Australian Government provided seed funding in early 2018 to investigate the introduction of ODR for couples separating or divorcing in Australia. The Legal Services Commission of South Australia (LSC) managed the project on behalf of National Legal Aid. Portable successfully responded to the opportunity to work with LSC to build the prototype ODRS tool in March 2018. Over the past year, Portable has been working to design an industry-first online property settlement tool by conducting eight 1-week-long design sprints with a dedicated internal team.

  4.2 Our Design Process

  After reading the initial interviews conducted on behalf of LSC by a market provider and following a review of existing research on other ODR applications for family disputes,¹⁰x See J. Zeleznikow, ‘The Split-up Project: Induction, Context and Knowledge Discovery in Law’, Law, Probability and Risk, Vol. 3, 2004, pp. 147-168; Bennett et al., Current State of Automated Legal Advice Tools, Networked Society Institute Discussion Paper 1, April 2018; J. Barnett & P. Treleaven, ‘Algorithmic Dispute Resolution—The Automation of Professional Dispute Resolution Using AI and Blockchain Technologies’, The Computer Journal, Vol. 61, No. 3, 2018. we determined that the two most significant barriers to people using ODR for their property settlement were:

  • Complexity: couples with significant assets or complex issues (such as child custody issues)

  • Cooperation/amicability between the separating partners: the ODR workflow requires cooperation between the parties to reach agreement on the total asset pool and then have a facilitated negotiation over the division of assets.

  This informed our definition of a minimum viable product that we used to guide product development, as follows:

  A tool to prove that ODR can help couples with a ‘simple’ (constrained) separation generate a fair and satisfactory resolution using AI.

  From here, our designers researched existing user stories, relevant law, jurisdictional constraints and the required functionality for a property settlement ODR tool. We worked over a series of eight-week sprints to design an alpha prototype in discrete blocks and then iterated our prototypes based on user testing.
  The ODR user stories that guided our design decisions were led by user needs identified through research, as well as from learnings we gained by developing the Sort’d prototype and conducting research for VCAT. The user needs were as follows:

  • Convenience and efficiency: Users do not want to take time off work, arrange childcare or travel for meetings

  • Flexibility: Users want a system that allows them to come in and out of the tool, save progress and upload documents at their own pace

  • Time: Users want some sort of time restriction or ‘move along’ process to ensure the process does not stall

  • Information: Users want clarity about key rights and obligations

  • Affordability: Users want an affordable way to divide their assets

  This led to the design principles that would guide our design process for the ODR tool. These principles underpinned the planning of the user flow as well as our interface designs and are as follows:

  • Conversational user interfaces (UI) to empower the user

  • Context-based education throughout the process

  • Asynchronous input to allow users to work in their own time

  • Design for mobile first to maximize usability

  • Opportunities to save progress and export current progress to ensure that the user receives a ‘takeaway,’ whether they complete the process online or not

  • Sensible guidance with plain language

  4.3 What We Learnt

  We are now developing a responsive Web application that allows users to agree upon their shared financial circumstances and facilitate a resolution of their property settlement. We have conducted four rounds of user testing with 23 people on our Alpha and Beta prototypes.
  This testing with real users of the system informed our design iterations by keeping us focused on designing systems people would want to use. Helpfully, we were able to directly test our Beta prototype with some of the same people who participated in the initial interviews. This allowed us to confirm our assumptions and design principles and modify the tone of voice, functionality and content as needed. We learnt that:

  • People want clarity and transparency

  • People need their online experience to integrate with offline services (such as legal and financial advice)

  • People want context and education throughout the process

  • People sometimes need a friendly nudge

  • People may need an authoritative decision maker to provide an objective benchmark

  • People want the tone to be authoritative, but not stuffy or legalistic

  Much of the feedback we got from users was about how helpful it was to view the process step by step and with plain language text with a conversational user interface.
  We also worked alongside LSC’s legal research team to develop early iterations of a machine learning algorithm that will automate existing court processes and introduced sentiment analysis software into the negotiation process. At first, we were concerned that people may not trust an automated decision-making feature within the app¹¹x See A. Sela, ‘Can Computers Be Fair? How Automated and Human-Powered Online Dispute Resolution Affect Procedural Justice in Mediation and Arbitration’, Ohio State Journal on Dispute Resolution, Vol. 33, No. 1, 2018, p. 91. or that the sentiment analysis software built into the communication tool may seem overbearing and paternalistic.
  User testing was again useful in dispelling our fears that people would not trust the system, and we were surprised to learn that many people would be more trusting of an authoritative automated system than of human intervention. As the following quotes demonstrate, Beta testing participants who had previously been through a property settlement process were largely receptive to the idea of an objective, automated decision-making tool.

• 5 Conclusion

  Overall, our design process has been a successful example of designing for users, but it has not been without its challenges. Writing legally relevant content that is thorough but also in plain English was a huge learning curve for our designers. Working with lawyers was hugely informative but also required us to constantly remind ourselves that we were designing for the minimum viable product, and for our target user who is amicable and not in an especially complex separation. Lawyers are trained to think of worst-case scenarios and extreme use cases, which would have significantly changed the scope of our project.
  Another significant challenge has been to test and prototype with users currently experiencing issues with property settlement, since it requires testing with users who are possibly emotionally activated. The separation process covers such a long period that it was easiest to test with users who had completed the whole process, but there was more we could have learnt from users who were currently in the process.
  We will continue to iterate on this prototype as we develop our product and add more features. We are also using what we learn from Sort’d, VCAT and LSC to inform the development of our next ODR project, Settle, which will combine user-centred front-end technology with efficient case management, in an effort to make the ODR process more usable for everyone involved.

Noten

• 1 See https://www.designorate.com/characteristics-of-human-centered-design/.

• 2 See T. Brown, Design Thinking, Harvard Business Review, June 2008.

• 3 M. Hagan, ‘A Human-Centered Design Approach to Access to Justice: Generating New Prototypes and Hypotheses for Intervention to Make Courts User-Friendly’, Indiana Journal of Law and Social Equality, Vol. 6, No 2, 2018, Article 2.

• 4 Matheson et al., ‘Leveraging Human-Centered Design in Chronic Disease Prevention’, American Journal of Preventive Medicine, Vol. 48, No. 4, 2015, pp. 472-479.

• 5 Government of Victoria, Access to Justice Review: Report and Recommendations, Volume 1, 2016, p. 281.

• 6 Ibid., p. 277.

• 7 Marriages and Divorces, Australia, 3310.0, 2016, available at: www.abs.gov.au/ausstats/abs@.nsf/mf/3310.0.

• 8 Family Court of Australia, Annual Report 2016-2017, available at: www.familycourt.gov.au/wps/wcm/connect/7456589e-fc98-409b-8d1c-7b909bd30dcf/2743-Family_Court_of_Australia_AnnualReport_2016_17_WEB.pdf?MOD=AJPERES&CVID=.

• 9 Australian Law Reform Commission, Review of the Family Law System (Discussion Paper 86), October 2018.

• 10 See J. Zeleznikow, ‘The Split-up Project: Induction, Context and Knowledge Discovery in Law’, Law, Probability and Risk, Vol. 3, 2004, pp. 147-168; Bennett et al., Current State of Automated Legal Advice Tools, Networked Society Institute Discussion Paper 1, April 2018; J. Barnett & P. Treleaven, ‘Algorithmic Dispute Resolution—The Automation of Professional Dispute Resolution Using AI and Blockchain Technologies’, The Computer Journal, Vol. 61, No. 3, 2018.

• 11 See A. Sela, ‘Can Computers Be Fair? How Automated and Human-Powered Online Dispute Resolution Affect Procedural Justice in Mediation and Arbitration’, Ohio State Journal on Dispute Resolution, Vol. 33, No. 1, 2018, p. 91.