AI IN HEALTHCARE

• Summary: The US Department of Health and Human Services (HHS) Office of the Chief Artificial Intelligence Officer (OCAIO) aims to facilitate effective collaboration on AI efforts across HHS agencies and offices. The site outlines HHS’s AI strategy, highlights AI accomplishments and priorities at HHS, and provides an inventory of HHS AI use cases. The site also provides a collection of AI-focused laws, regulations, executive orders and memoranda driving HHS’ AI efforts.

• Summary: Published by the HHS OCAIO in September 2021, the Trustworthy AI Playbook includes HHS-specific guidance on major trustworthy AI concepts and how HHS leaders can confidently develop, use and deploy AI solutions. The playbook also lists the current statutory authorities that HHS believes it can use to regulate AI in healthcare (see Appendix III in the linked playbook). OCAIO also released an executive summary of the playbook. Of particular interest is the below graphic showing HHS’s directive to staff on how to regulate AI.
  

• Summary: FDA, Health Canada, and the UK’s Medicines and Healthcare products Regulatory Agency (MHRA) announced the five jointly identified guiding principles for predetermined change control plans (PCCPs) for machine-learning enabled devices. The term PCCP describes a plan, proposed by a manufacturer, that specifies certain planned modifications to a device, the protocol for implementing and controlling such modifications, and the assessment of impacts from such modifications. The document is intended to outline foundational considerations that highlight the characteristics of robust PCCPs and encourage international harmonization. The five guiding principles are: (1) focused and bounded; (2) risk-based; (3) evidence-based; (4) transparent; and (5) total product lifecycle perspective. The five guiding principles for PCCPs draw upon the 10 ten Good Machine Learning Practice (GMLP) guiding principles developed by FDA, Health Canada, and MHRA in 2021. Stakeholders can provide feedback through the FDA public docket (FDA-2019-N-1185).
• Digital Health Center of Excellence
  • Summary: The Digital Health Center of Excellence (DHCoE) was created by FDA to empower stakeholders to advance healthcare by fostering responsible and high-quality digital health innovation. DHCoE is responsible for aligning and coordinating digital health work across the FDA. This site provides links to a variety of resources, draft guidance and events related to AI/ML.

• Summary: This site offers a starting point for stakeholders interested in any aspect of AI at CMS. It provides links to foundational governance documents on AI, noting that those who wish to engage in AI-related activities, whether as a CMS employee, partner or vendor, should be aware of federal policies regarding the application of AI. The site also provides details on AI programs and initiatives at CMS.

• Summary: This blog series explores current and potential uses of AI, predictive models and machine learning algorithms in healthcare, and the role that ONC can play in shaping their development and use. Topics covered include increasing transparency and trustworthiness of AI in healthcare, minimizing risks and rewards of machine learning, and risks posed to patients by these technologies.

• Summary: The FTC Office of Technology (OT) hosted a Tech Summit to bring together a diverse set of perspectives across academia, industry, civil society organizations and government agencies for a series of conversations on AI. The summit consisted of several remarks from FTC regulators and panels on chips and cloud infrastructure, data and models, and consumer applications. Key takeaways include:
• • FTC announced that it is issuing compulsory orders to five major technology companies requiring them to provide information about their AI partnerships.
  • Speakers were concerned about consolidation in the chips and cloud infrastructure markets.
  • Data privacy and model transparency were emphasized as important anti-trust enforcement areas.
  • Regulators tied market consolidation to lower quality and higher cost products, privacy violations and discrimination.

Contact our team for a complete summary of the virtual summit.

• Summary: FTC discusses “model-as-a-service” companies, which develop and host AI models to make such models available to third parties via an end-user interface or an application programming interface. FTC stated in the blog post that, like most AI companies, model-as-a-service companies have a continuous desire for data to develop new or customer specific models or refine existing ones. FTC warns that the desire for continuous data can be at odds with a company’s obligation to protect user’s data, undermining peoples’ privacy or resulting in the appropriation of a firm’s competitively significant data. The blog warns that model-as-a-service companies that fail to abide by their privacy commitments to their users and customers (regardless of how or where such commitment was made), or engage in misrepresentations, material omissions, or misuse of data, may be liable under the laws enforced by FTC. The blog provides several examples of FTC enforcement actions stemming from these kinds of conduct.

• Summary: FTC provides tips for mitigating risk of FTC Act violations arising from unfairness in the development or use of AI tools.

• Summary: FTC discusses deception in generative AI tools and key questions developers should ask.

• Summary: FTC offers insight into its thought process when evaluating whether AI marketing claims are deceptive.

• Summary: On June 16, 2022, the FTC submitted its report to Congress required by the 2021 Appropriations Act discussing whether and how AI could be used to identify, remove or take other appropriate action to address online harms. The report details seven areas within the FTC’s jurisdiction in which AI could be useful in combatting online harms and provides recommendations on reasonable policies, practices and procedures for such uses and for any legislation that may advance them.

• Summary: FTC highlights key pillars in managing consumer protection risks associated with AI and algorithms, including transparency, explainability, fairness and accountability.

• Summary: FTC offers tips for the truthful, fair and equitable development and use of AI and identifies Section 5 of the FTC Act, the Fair Credit Reporting Act and the Equal Credit Opportunity Act as statutory authorities for FTC enforcement activity.

• Summary: This site serves as a starting point for stakeholders interested in NTIA policymaking activities, which are principally focused on helping to develop policies necessary to verify that AI systems work as they claim.

• Summary: This site serves as a landing page for stakeholders interested in NIST’s research and standards development activities regarding AI.

• Summary: On December 19, 2023, NIST issued a Request for Information (RFI) to assist in its efforts to carry out several of its responsibilities under the Executive Order on Safe, Secure and Trustworthy Development and Use of Artificial Intelligence. According to the release, the responses to the RFI will support NIST’s efforts to evaluate capabilities relating to AI technologies and develop a variety of guidelines called for in the Executive Order. The RFI specifically calls for information related to generative AI risk management, AI evaluation, red-teaming, reducing the risk of synthetic content, and the development and implementation of AI-related consensus standards, cooperation and coordination. Comments may be submitted via the Federal e-Rulemaking Portal, email or mail. Responses are due February 2, 2024.

• Summary: This site provides an overview of NIST’s responsibilities under President Biden’s Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. The Executive Order directs NIST to develop guidelines and best practices to establish consensus-based industry standards that promote the development and deployment of safe, secure, and trustworthy AI systems, which NIST will carry out through several different activities as detailed on the site. NIST will release more information about its role under the Executive Order on November 9, 2023.
• In addition to working with government agencies, NIST intends to engage with the private sector, academia, and civil society as it produces the guidance called for by the Executive Order. These activities will build and expand on existing NIST efforts in several of the areas covered in the Executive Order, such as generative AI and AI risk management. As part of this engagement, NIST is calling for participants in a new consortium supporting AI safety. The Artificial Intelligence Safety Institute Consortium will help develop tools to measure and improve AI safety and trustworthiness. Interested organizations with relevant technical capabilities should submit a letter of interest by December 2, 2023. More details on NIST’s request for collaborators are available in the Federal Register. NIST plans to host a workshop on November 17, 2023, for those interested in learning more about the consortium and engaging in the conversation about AI safety.

• Summary: U.S. Secretary of Commerce Gina Raimondo announced that the National Institute of Standards and Technology (NIST) is launching a new public working group on AI to help NIST develop key guidance to support organizations in addressing the special risks associated with generative AI and the opportunities and challenges associated with generative AI technologies. The public working group will comprise volunteers from the private and public sectors. Those interested in joining the NIST Generative AI Public Working Group should complete this form. Registration closes July 9, 2023.

• Summary: This platform supports and operationalizes the NIST AI Risk Management Framework (AI RMF) and accompanying playbook. It provides a repository of foundational content, technical documents, and AI toolkits, including standards, measurement methods and metrics, and data sets. Over time, it will provide an interactive space that enables stakeholders to share AI RMF case studies and profiles, educational materials and technical guidance related to AI risk management.

• Summary: On January 26, 2023, NIST released its “Artificial Intelligence Risk Management Framework (AI RMF)”. The framework is designed to equip stakeholders with approaches that increase the trustworthiness of AI systems, and to help foster the responsible design, development, deployment and use of AI systems over time. The AI RMF is intended to be a living document that is to be reviewed frequently and updated as necessary. The framework has an accompanying playbook that provides suggested actions for achieving the outcomes laid out in the framework.

• Summary: This discussion paper was released to help inform potential future rulemaking on the use of AI/ML in drug development. The paper describes current and potential uses of AI/ML in drug discovery, clinical and non-clinical research, postmarket surveillance and advanced pharmaceutical manufacturing, and raises several questions for stakeholder input. The Federal Register notice can be found here. Comments were due August 9, 2023.

• Summary: This draft guidance includes recommendations on the information to be included in a Predetermined Change Control Plan to be submitted in marketing submissions for AI and ML-enabled devices. The purpose of a Predetermined Change Control Plan is to account for certain planned or expected device modifications that would otherwise normally require a premarket approval supplement, new de novo submission or new 510(k) under applicable regulations. Comments were due July 3, 2023. See our summary of the draft guidance here.

• Summary: On March 1, 2023, CDER issued a discussion paper soliciting public comments on areas for consideration and policy development associated with application of AI to pharmaceutical manufacturing. The discussion paper includes a series of questions to stimulate feedback from the public, including CDER and CBER stakeholder. The comment period closed on May 1, 2023.

• Summary: This final guidance provides clarity on FDA’s oversight and regulatory approach regarding clinical decision support (CDS) software intended for healthcare professionals and the types of CDS functions that do not meet the definition of a device as amended by the 21st Century Cures Act. See our summary of the draft guidance here.

• Summary: This proposed rule includes proposals to “promote greater trust in the predictive decision support interventions (DSIs) used in healthcare to…enable users to determine whether predictive DSI is fair, appropriate, valid, effective and safe.” The proposed transparency, documentation and risk management requirements impact developers that participate in the ONC Health IT Certification Program and those that create predictive DSIs that are enabled by or interface with certified Health IT Modules. The proposed rule was published in the Federal Register on April 18, 2023. Comments were due on June 20, 2023.

• Summary: This showcase spotlights how federal agencies and industry partners are championing the design, development and deployment of responsible, trustworthy AI in health IT. Speakers included representatives from numerous federal agencies, Congress, the American Medical Association, academic health and research centers, and industry. The agenda, presentation slides and event recording are available to the public.

• Summary: WHO released new guidance on the ethics and governance of LLMs used in healthcare. As LLMs are starting to be used for more specific health-related purposes, WHO outlined a variety of risks, such as inaccurate or biased information, models trained on poor quality or biased data, privacy risks, inaccessibility and unaffordability, automation bias, and cybersecurity risks, among others. Accordingly, the guidance outlines more than 40 recommendations for consideration by governments, technology companies and healthcare providers to promote the appropriate use of LLMs to protect population health.

• Summary: WHO released a new publication listing key regulatory considerations for the use of AI for health in response to the growing need to responsibly manage the proliferating AI technologies in healthcare. It highlights 18 considerations across the following six categories: (1) documentation and transparency; (2) risk management and AI systems development lifecycle approach; (3) intended use and analytical and clinical validation; 4) data quality; (5) privacy and data protection; and (6) engagement and collaboration. The publication emphasizes the importance of establishing AI systems’ safety and effectiveness, rapidly making appropriate systems available to those who need them and fostering dialogue among stakeholders. The publication is intended to be a listing of key regulatory considerations and a resource that can be considered by all relevant stakeholders in medical devices ecosystems, including developers who are exploring and developing AI systems, regulators who might be in the process of identifying approaches to manage and facilitate AI systems, manufacturers who design and develop AI-embedded medical devices, health practitioners who deploy and use such medical devices and AI systems, and others working in these areas. The full publication can be accessed here.

• Summary: The WHO released a statement calling for caution to be exercised in using AI-generated large language model tools such as ChatGPT for health-related purposes. The WHO expressed concern that the caution that would normally be exercised for any new technology is not being exercised consistently with large language model tools, including adherence to key values of transparency, inclusion, public engagement, expert supervision and rigorous evaluation, and enumerated several areas of concern with the use of such technology. The WHO proposed that these concerns be addressed, and clear evidence of benefit be measured before their widespread use in routine health care and medicine, whether by individuals, care providers or health system administrators and policymakers.

• Summary: The report identifies the ethical challenges and risks with the use of AI in healthcare, as well as six consensus principles to ensure AI works to the public benefit of all countries. The report is the product of input from experts in ethics, digital technology, law, human rights and health ministries. The WHO report includes a set of recommendations to governments and developers for oversight of AI in the delivery of healthcare, seeking to hold all stakeholders – in the public and private sectors – accountable and responsive to the healthcare workers who will rely on AI and the communities and individuals whose health will be affected by its use.