Responsible Systems

Firms should anticipate regulatory attentions in 2024 to focus on:

• Risk Management: Regulators will examine risk management and governance of the design, development,  deployment, and ongoing monitoring of “automated systems” from the perspective of:
  • Safety and effectiveness (e.g., protections  against unintended or inappropriate  access or use). 
  • Anti-bias and anti-discrimination (i.e., protections  against, and ongoing testing).
  • Data governance and data privacy.
  • Transparency (e.g., what/how information  is used; potential impacts to businesses/  consumers).
  • Accountability and oversight.
• Fairness/Consumer Protection: Following the direction of the Administration, regulators will take a  “whole of government” approach to supervision and enforcement of fairness and consumer protections  in applications of “automated systems” under existing (and evolving) laws and regulations.
  • Key areas  of concern include data and data sets, model opacity and access, and design and use—which drive the decision-making enabled by the automated system or related tools. Regardless of the technology used,  supervisors will evaluate:
  • Fairness (e.g., UDAAP/UDAP, fair lending, “fair  and balanced” marketing, conflicts of interest).
  • Consumer and other legal protections (e.g., civil  rights, nondiscrimination, equal employment  opportunity).
• Purpose Limitation and Data Minimization: With the proliferation of consumer data collection alongside  the increasing application of automated systems, regulatory scrutiny and enforcement will focus on:
  • Limitations around collection, access, use,  retention, and disposal of consumer data for specific and/or explicit purposes, subject to  permission, consent, opt in/out, and authorization, as appropriate (e.g., only what is needed).
  • Limitations on data retention (e.g., only for the statedpurpose).
  • Safeguards on access and use.

• Regulatory Divergence: Diverging regulatory approaches or areas of supervisory  focus on automated systems would greatly expand the complexity of both risks and  compliance and necessitate reassessment of current and target state compliance  functions and approaches to compliance risk assessments. Impact assessments,  jurisdictional risks, regulatory awareness, and timing would also need to be considered.
• Jurisdictional Challenge: Legal challenges to regulatory approaches, such as  application of existing consumer protection regulations to examine firms’ systems,  technologies, data, and algorithms, could similarly present added complexity if  uncertainties around regulatory jurisdiction and authorities persist.

• “Trustworthiness”: Focus on the trustworthiness of automated systems and new  technologies, particularly around safety, efficacy, fairness, privacy, “explainability,” and  accountability. This will necessitate a holistic reassessment of the purpose and application of  automated systems throughout the firm, including data collection, inputs and outputs, use,  privacy, and security.
• Model Risk Management: Expectation that the design, development, deployment, and  monitoring of automated systems will be incorporated into the firms’ MRM framework (and  legacy risk frameworks will be adapted, as appropriate), including:
  • Areas such as approved use, ongoing monitoring, and risk ratings.
  • Protocols for modeling usage that align to the MRM standards.
  • Monitoring of legislative/regulatory actions necessitating potential changes to practices or  business models.
• Misuse or Inaccuracy: Risks associated with misuse or inaccuracy will be assessed through  the AI risk management framework; regulators will look for:
  • Robust development, implementation, and use (e.g., clear statement of purpose, sound  design, theory, or logic).
  • Effective validation conducted independently of design and development.
  • Sound governance, policies, and controls (e.g., access controls, training).