• 1 Leah Wing

  I’m Leah Wing, the Director of the National Center for Technology and Dispute Resolution and had the joy and honour of working with Ethan in our academic department and in ODR for the past almost two decades now; and Brian, it’s a tremendous effort and so much work that you put in to making sure that we all can be here over the next 3 days. Thank you so much for all your work and thank you to UCD’s Sutherland School for Law for hosting. Chris and I are excited to be launching here and now with this panel, a framework for thinking about the parameters for what ODR is. Our presentation is based on a new publication, Framing the Parameters for Online Dispute Resolution¹x Framing the Parameters for Online Dispute Resolution. National Center for Technology and Dispute Resolution 2022, available at: https://odr.info/publications/. See also, Wing, L. ‘Mapping the Parameters of Online Dispute Resolution.’ International Journal of Online Dispute Resolution Vol. 9, No. 1, 2022, 3-16. issued today by the National Center for Technology and Dispute Resolution. I’m going to begin by talking about why we at the Center thought that now would be a good time to explore what the state of play is around what constitutes ODR.
  With the growing interest in applying AI to dispute resolution, there’s been an explosion in the growth of ODR platforms and obviously with a pandemic we have all seen tens of thousands, likely hundreds of thousands of people beginning to use video whether they’re third parties or whether they’re disputing parties and they’re engaging for the first time in what some people are considering ODR. This is taking place in alternative dispute resolution settings as well as in the courts and includes using AI even during in-person hearings such as for setting bail. Does such use of AI qualify as ODR? Does merely using a video conferencing platform mean you are engaged in ODR? The tremendous growth in the use of technologies in dispute handling, including the use of AI, has stimulated a lot of debate and discussion among people who are already engaged in ODR and it’s also expanded the number of stakeholders who are impacted by the decisions that are made inside the court and within alternative methods of dispute resolution. After more than 20 years of expanding use of online dispute resolution worldwide, we are now in a place where we have over a billion disputes that are generated online each year and those disputes have gone from being just interpersonal to now crossing all these different sectors from e-commerce to family to labour to environmental and other multi-party public disputes; additionally, it is employed for efforts as varied as peacebuilding to handling cyberbullying. But, as you well know, we’re not only managing disputes that start and remain online, we’re also using technology for disputes that start offline and many of our processes are also hybrid. As Sir Colin [Birss] and Mireze [Philippe] both have mentioned during this Forum, we’re harnessing technology developed for other purposes as well as building on innovations from the last 20 years, especially in the last 5 years, for developing ODR platforms that are particularly designed for handling disputes. The use of technology in alternative dispute resolution and courts had become ubiquitous in many ways even prior to the pandemic although many would not have framed their usage as ODR. Yet, the unfortunate entry of the pandemic in all of our lives has really impacted how we have been forced, some with enthusiasm and some begrudgingly, to have to think consciously about the increasing use and role of technology in handling disputes.
  Scholarship and twenty years of practice have shown us that there are significant benefits, well proven, for applying technology to dispute management: it increases efficiency and it really can increase access to justice. Companies are finding that when they handle disputes effectively and quickly that people become more loyal to their company and this increases profits. Research also shows that when aspects of technology are used creatively, for example when anonymity is offered and handled in a positive way, it can lead to more creative outcomes.²x Unpublished collaborative research by one of the authors, Leah Wing with Daniel Rainey, Ethan Katsh, Lee Osterweil, and Lori Clarke. Unfortunately, we have also seen that historic problems with courts and ADR such as repeat player bias or ways in which power imbalances get replicated inside dispute resolution processes can be exacerbated in the online setting. And new dilemmas have arisen, such as the inclusion of biased AI and a lack of transparency and accountability. Given the millions of new stakeholders due to the explosion of usage since the pandemic, we have many more people involved in and invested in the conversation about what ODR is and what it ought to be. Another shift occurring that has impacted how we can define ODR is one that Mireze noted earlier: silos are being broken down when we’re engaging with people from different disciplines – software engineers and program managers for tech firms are working together with lawyers and mediators and arbitrators. This is an incubator for transformation and is one which is sorely needed but it also creates significant dilemmas.
  We have insufficient infrastructure for ODR. While courts may be ready and willing to incorporate more technology, where’s the funding and where’s the consistency of funding so that they can stay up-to-date with new technology, just as an example? And, arguably, we have a lack of training. As we’re producing new arbitrators, mediators and lawyers, are they scaled up with thinking about ethics related to how they’re going to employ technology? Our standards in the ADR field, in particular, don’t include technology. We clearly do have, and I would say the EU has been one of the leaders in this for quite a number of years now, we do have some legislation and some regulation in some jurisdictions, but we’re also lacking that in many others. Lastly, as we are thinking about the context for the tremendous expansion of who’s engaging in discussions regarding what constitutes ODR, I would argue we can see this as a reflection of a maturation of the field. This reminds me of the important work of Professor Carrie Menkel-Meadow who in the early 1990s was commenting on how the attempts to regulate ADR and the increasing institutionalization of entities and practices were signs of the maturing nature of the field of ADR. She noted that the diversity of attempts at regulation reflected the reality on the ground that there was diversity of practice and perspective but also that there could be overlapping and competing forms of accountability. She argued that these were important discussions and debates to be having. The responses of many in the ODR field to the surge in use of technology in the context of the pandemic has been, I would argue, a mix of excitement and also concern in the face of the tremendous potential seen by bursts of promising innovation, the magnified risks, and the lack of sufficient training and infrastructure. This has bred rich engagement about what constitutes ODR in this new context.
  Here are just some of the typical questions that are now routinely being discussed in the field:

  • Does a process have to be held entirely online to be considered ODR?

  • Does merely employing video conferencing constitute ODR?

  • What if I just use e-case management but then we meet face-to-face, have I used ODR?

  • When I use blockchain as part of the development of a contract is this ODR since it can help prevent disputes over the originality of the contract and its date and timestamp?

  All of this leads us to consider what the implications are for how we define the parameters of ODR? One implication is that it can help us with our visioning. If we know where the boundaries are, it can help us see where we want to put our energy, both in solving problems and anticipating possibilities. It also, as Sir Colin pointed out, is urgent that we think about what constitutes ODR as we’re thinking about software development. Whether you’re choosing an off-the-shelf software package or you’re selecting a company to design an ODR system for your court or business, what the parameters of what you’re considering ODR will impact your choices. Clarity on the parameters of ODR is necessary for legislation and other instruments of regulation. It will also have an impact on expectations regarding the training of lawyers, ADR practitioners, human relations personnel and other professionals who handle conflict, particularly regarding ODR ethics and their implications for services. And I’ll get to this later, particularly in the next panel, but it also has implications for liability. It can drive innovation for transforming access to justice and service delivery and push us to go beyond merely trying to reform our processes with technology.
  Next, we are going to present about an ODR Framework that is being published today as Framing the Parameters for Online Dispute Resolution³x Framing the Parameters for Online Dispute Resolution. National Center for Technology and Dispute Resolution 2022, available at: https://odr.info/publications/. See also, Wing, L. ‘Mapping the Parameters of Online Dispute Resolution.’ International Journal of Online Dispute Resolution Vol. 9, No. 1, 2022, 3-16. by the National Center for Technology and Dispute Resolution. It is an attempt to map the discussion that is underway in the field now. It is not an attempt to provide a prescription; rather it is to provide a description of that discussion and we hope it will prove useful in the very important planning for the future.

• 2 Chris Draper

  So two of the biggest challenges we’ve heard in previous talks are, first, Sir Colin identifying the biggest challenge within our current legal system as being change management and Mireze [Philippe] pretty accurately identifying our community’s problem of tech people being unable to translate their view to us, where many challenges may also arise from us being unable to translate our views to them. So, as we are attempting to explain the objective of this framework in the most relatable way, we are needing a simple analogy; like how everyone in tech tells you they are doing ‘the Uber of’ whatever they are making.
  To help explain our framework, we first propose taking a moment to view ODR as transportation. For this analogy, the most important aspect of both ODR and transportation is that both are taking something from a start to a finish. Now there are many things that could be transported. It could be people, it could be things, it could be boxes, it doesn’t really matter. As long as it gets from one place to the other, I consider that transportation.
  Similarly, there are nearly unlimited ways to transport that something. There’s all kinds of fun methods for transporting these things, from horses to the Da Vinci helicopter. Obviously the Da Vinci one didn’t fly, so please don’t take that one, since I don’t think that’s even U.S. Federal Aviation Administration registered at all. However, from cars, trucks, rail, to whatever the vehicle is, our framework says transportation is any method of taking anything from a start to a finish.
  If we think of transportation in this manner, where the objective is the moving of anything from where it started to a reasonable finish using any type of manual to automated method, this is going to look a lot like what we’re seeing in the ODR space. Each different ODR element is attempting to do a slightly different thing. Yet, at the end of the day, all of them are transporting the human from one state of being to another. Hopefully, whether it’s resolving disputes or merely identifying the dispute, with some components that may be helping to automatically do anything from defining terms to generating an agreement, we are outlining a framework for describing the relationships between each system in an ODR process. My metaphor for illustrating this in a few moments will be using a transportation model.

• 3 Leah Wing

  Let’s think about the ODR Framework as a descriptive continuum where what is central is technology’s role in dispute handling. So we divided the Framework up into five levels with the first level not involving technology at all and Level 5 being exclusively reliant on technology with no reliance on humans. Therefore, it is really the amount of reliance on humans and technology that defines the levels, as described in this particular Framework. It’s not, just to emphasize, it is not differentiated by the type of tech because you could end up using some of the same pieces of technology or systems of technology in different levels and we’ll be talking a little bit more about that to illuminate that in a moment.
  Across the five levels, technology’s impact can include who can or who must rely on it and what its roles are. For example, is technology creating a space or a channel through which humans can generate options, humans can generate brainstorms, humans can do the diagnosis? Or is technology actually generating all the brainstorms and generating the decisions? What role does technology or do humans play in security and in administration? And across the levels, again, I would just point out that tech could be employed in very different phases and it’s important to note here, for example, you could use technology during the prevention stage on a particular level and then during the dispute resolution stage you could be using technology at another level with more reliance on AI, and then we can go back down to using less technology in another level as we move to the follow up stage in a particular dispute resolution case.
  Making the determination in the design of the ODR system or in the way you will handle a dispute when it will be considered ODR is going to potentially differ across the dispute type, the sector of society, the legal jurisdiction and cultures. In real life we know that ODR actually is more fluid and it is multidirectional as I’ve explained. You could start using a lot of technology where there’s a lot of automation and then you can move to a phase where humans are more in control. So, we can move up and down the levels. But we decided to frame it in a way that is more theoretical than reflecting the reality of what happens in an actual case in order to bring the focus on identifying the opportunities, risks and, very importantly, the responsibilities that come with increasing reliance on tech and decreasing reliance on humans. See Figure 1 for the ODR Framework.⁴x Framing the Parameters for Online Dispute Resolution. National Center for Technology and Dispute Resolution 2022, p. 7, available at: https://odr.info/publications/.

• 4 Chris Draper

  Let’s start looking at the issue of transportation, and most importantly looking at it as a metaphor useful to understand what Leah is talking about here: if we can use level of automation as a differentiator within an all-inclusive ODR proposal, what would that look like? Since ODR is focusing on transporting a human from a starting to a finishing state, we will walk through this framework from the perspective of a car transporting a human.

  An ODR Framework Used with permission from Chris Draper and Leah Wing
  

  
  The most interesting challenge in thinking about both modern and future cars is: who’s driving it? Nearly anyone with even a passing interest in technology has likely heard of Tesla autopilot, and most often assume ‘autopilot’ means the car is able to drive itself. However, this characterization is both inaccurate and misrepresents commonly accepted scientific approaches to automation.

  

  Tesla’s ‘autopilot’ design is not self-driving, which would be in the transportation world parallel to what is Level 5 in this ODR Framework, (and it is likely not even the ‘2.5’ they sometimes claim when trying to be even partially honest). In the world of technology design, these designations have meaningful design impacts. The way automation is defined in car systems, what we are calling Level 2 in the ODR Framework, is pretty much a driver assist compared to the Level 5 of full automation. To start ‘speaking tech’, we need to be able to translate ODR technologies into these kinds of numerical classifications. So, let’s start looking at our field, which is infusing more technology yet built around human processes, and let’s start trying to apply technology design concepts to how we are defining our systems.
  Using the prior transportation metaphor to illustrate how levels of reliance on technology are defined in other sectors will allow us to see how we could apply similar definitions in our field. That is the intent with the ODR Framework, as described in the associated paper,⁵x Ibid. and hopefully it will contribute to finding an alignment that we can communicate effectively across all interested parties about what the parameters of ODR are. If successful, we will hopefully use this as a way to ensure when there’s a new RFP in a court, when there’s a new proposal that says ‘we need ODR’, we can quickly turn to a more robust conversation and reach clarity around ‘what does that mean?’. This Framework is designed to be as expansive as the problems that we need solved when defining ODR because it is built around how engaged is the human at the centre of the system. Again, no technology will solve human problems. Human problems have human solutions. The technology only accelerates and amplifies human actions. So, the most important question we must constantly ask is: how does technology help, or how can it harm, in its process of accelerating and amplifying that human effort?
  Stepping through the Framework (see Figure 1) we see there is a top half which is going to be our less technology-intensive systems. The bottom half of the Framework outlines the more technology-dependent systems. Examining level by level, we’ll start with Level 0 (Zero). Level 0 is a ‘no technology’ ADR process, often thought of as traditional ADR, where we’re all sitting around the same table using paper and pencil. If you are at Level 0, there’s nothing that we consider a technology enhancement to the process. We are all around the table, we’re all just chatting. This is what we would consider a Level 0 in the ODR Framework, which we argue is not ODR.
  The first level that we likely all fully agree as the start of what ODR is Level 1. Level 1, as we’re proposing it, would be where you have a process in which there is no integration between the uses of different forms of technology. Level 1 would be an ODR system using things like video conferencing where the third party would typically be managing any video-based meeting. Or in the case of a negotiation, the two parties would enter the same video conferencing system, but neither would dominate that system. Each would actually be coming equally into a system that they do not control, a system they don’t have any connections to. Each technology in a Level 1 scenario is just an independent box. All parties join the box. They leave the box. So, from the metaphor of driving, for example, Level 1 is when the car alerts you that it’s on the edge of the road, but it won’t do anything about it. Level 1 ODR will still let you run off the road. On Level 1 you have a tech tool and you’re just all using it in an equal manner with no integrated procedural guardrails. For example, a Level 1 videoconferencing solution would allow either party to invite anyone else into the mediation or negotiation without the invite coming from a tech-driven preapproved pool of participants. In this level, appropriate use of the tool is the responsibility of each participant alone.
  Going to Level 2, this is when we have a centralized, partial automated dispute handling system. When looking at the transportation model, this is one where it not only beeps to say you’re going off the road, it will actually slow your car down and move you back a little bit. For ODR, Level 2 is defined here as being a system where each side may have its own technology, that technology may do something that is customized for that party, and it may communicate with each other but it may not. For example, Next Level Mediation is a tool that one side may choose to have for identifying illogical inconsistencies when interests are entered into the tool. In this case, both parties don’t have access to that full tool, but it is something where one party is informed through its use of that tool. That party then has the opportunity to influence their decisions or choices as a result of that technology, but the outputs of the tool don’t necessarily have to be shared or be directly communicated to the other party nor be used by the other party.
  Levels 3 to 5 are where we start getting into fully integrated, much more tech-dependent systems. Level 3 is what we would consider human-managed technology dispute systems. Using the AI terminology of the National Centre for State Courts,⁶x JTC Resource Bulletin, ‘Introduction to AI for Courts.’ Version 1.0, Adopted 27 March 2020, www.ncsc.org/__data/assets/pdf_file/0013/20830/2020-04-02-intro-to-ai-for-courts_final.pdf. where a human controls a system in a manner that is in the loop, on loop or out of the loop, Level 3 is a fully integrated system where the human is in the loop. The system may be able to make suggestions to one or both parties, yet those suggestions cannot affect the ODR process without the human taking an action. For example, Smart Settle will have some Level 3 elements where the system may be able to suggest the value of a counteroffer, yet that offer will not be made unless the human chooses to share it. Level 3 systems will help the parties to actually facilitate their engagement, but the human is a necessary component of the engagement occurring.
  Level 4 is where the system may be capable of actually making decisions on behalf of the parties. Level 4 systems are the first to start employing decision logic that is not entirely implemented by the human. In Level 4 systems, event trees or other scenario-defining tools will allow the ODR system to assist in arriving at agreeable terms, yet they are human on the loop systems requiring any party to jump in if needed to correct issues. Most e-commerce tools or other filtering-based tools, where the system moves the parties automatically towards what it considers the most appropriate outcome for them to accept, will typically be Level 4.
  Please keep in mind that these levels only refer to the level of automation in the system, and do not necessarily mean an ODR system is more advanced at a Level 4 than a Level 2. There are a large number of Level 4 systems that are highly problematic, especially given their use in the American justice system. For example, most facial recognition systems are Level 4 systems that cannot seem to avoid bad press. Many are not designed well; yet even with the best designed Level 4 systems, there remains a fundamental danger to people, and some more than others. When you have a human on the loop system where you need that party to step in, it means that system is pretty much going to run on its own more often than not. Sort of like a Tesla until it parks itself into a truck. When the system normally runs fine until it doesn’t, and it is the responsibility of the person to spot a problem they are not used to seeing, Level 4 systems do risk lulling parties into a state of system acceptance – especially for those who are not familiar with the wider system they are using.
  Level 5 is truly autonomous. These are human out-of-the-loop systems where the users do not have the opportunity to influence the outcomes because the system runs on its own. This is what we keep thinking will maybe someday exist, yet none do. No Level 5 system currently exists right now, nor do any of the unmanaged AI systems that would be needed for a Level 5 ODR system currently exist, either. At best, most AI systems are just human on the loop machine learning (ML) systems that are quasi-controlled by their design logic. There’s no court system using anything more than a managed ML system to actually run any of their ODR offerings. There’s nothing being used in the marketplaces more than a Level 4 right now. But if we’re getting to full automation, this framework would put those types of systems at Level 5.
  Recapping from Level 0 to 5, the intent is not to say 4 is better than 1 or 5 is better than 2. Depending on your situation, a Level 5 system could be just as problematic as our worst Level 4 systems currently in place today. For example, family disputes come in all shapes and sizes, and if an informed person wants to get paid in 500 rolls of toilet paper, why should we care? Let them do it. But repurposing a filtering-based system appropriate for fixing an Amazon shipment dispute probably won’t let you do that.
  This ODR Framework does not remove questions as to whether any specific system has appropriate levels of human or tech control, auditability or appropriate opportunities for appeal. This Framework helps the dispute resolution and legal world and the tech world include all manner of ODR systems in a way that define them as a function of automation so everyone can better understand where to start evaluating a system in its totality. In this Framework, we are outlining parameters about system automation to assist with considering its appropriateness for the problems you are trying to solve and processes you’re trying to manage and regulate.

• 5 Leah

  Thank you so much Chris. Now I shall emphasize again what Chris has just pointed out. I’m just thinking about when someone files a complaint and there’s an automated response that says, ‘Here’s your claim number, a [human] mediator will be in touch’. That may be, just for example, Level 4 and then you meet with a mediator whether it’s in person or over video conferencing inside an ODR platform and that might be an earlier level, so to speak. So, what level of ODR you are using doesn’t have to do with the sequencing; it can be multidirectional; rather what level someone is using at a particular time really has to do with the level of reliance on technology at that stage of the process.
  Referring back to what Chris said earlier, if we look at the broadest definition of what ODR can encompass, this Framework really shows the beginning and the end of that for the parameters for ODR. And we’re not making an argument that more restrictive definitions are either inaccurate or not useful. Instead, we hope that the distinguishment between the different levels is useful for narrowing down and thinking about – given the system that we need, given the legislation or the regulation or the software development that we need to handle the types of disputes we’re going to get, to handle the particular dispute that I’m going to be responsible for helping to resolve – what is the definition of ODR? What’s the parameter of ODR that fits? And this is particularly important when we think about regulation, legislation, and certification of practitioners and processes. Therefore, we hope that the ODR Framework will stimulate discussions like the one that we’re in the midst of now. And really for us, it places ethics at the centre and those ethical questions are not just theoretical they’re also directly related to practice. We encourage all those who are in courts or in other forms of dispute handling to consider the entire continuum in your discussions even if you’re going to narrow it to where you narrow it. We hope that ethics will be central to thinking about training and practice; and keep in mind the actual fluidity and multidirectionality of the processes. We quite frankly, assume that ODR’s parameters are going to evolve just as technology does. Technology is going to continue to disrupt boundaries, disrupt what we take for granted and also both raise risks and raise new possibilities. So, feel free to be in touch with Chris and I and as we mentioned before, later today you’ll be able to get a copy of this and the entire more descriptive paper that the National Center for Technology and Dispute Resolution is issuing today.

Noten

• 1 Framing the Parameters for Online Dispute Resolution. National Center for Technology and Dispute Resolution 2022, available at: https://odr.info/publications/. See also, Wing, L. ‘Mapping the Parameters of Online Dispute Resolution.’ International Journal of Online Dispute Resolution Vol. 9, No. 1, 2022, 3-16.

• 2 Unpublished collaborative research by one of the authors, Leah Wing with Daniel Rainey, Ethan Katsh, Lee Osterweil, and Lori Clarke.

• 3 Framing the Parameters for Online Dispute Resolution. National Center for Technology and Dispute Resolution 2022, available at: https://odr.info/publications/. See also, Wing, L. ‘Mapping the Parameters of Online Dispute Resolution.’ International Journal of Online Dispute Resolution Vol. 9, No. 1, 2022, 3-16.

• 4 Framing the Parameters for Online Dispute Resolution. National Center for Technology and Dispute Resolution 2022, p. 7, available at: https://odr.info/publications/.

• 5 Ibid.

• 6 JTC Resource Bulletin, ‘Introduction to AI for Courts.’ Version 1.0, Adopted 27 March 2020, www.ncsc.org/__data/assets/pdf_file/0013/20830/2020-04-02-intro-to-ai-for-courts_final.pdf.