Law resources

Bias testing under NYC LL 144

New York City's Local Law 144 sets rules for employers that use automated tools in hiring. This page sets out what the law asks of a firm and how Suited's testing supports it.

Local Law 144 of 2021 governs how employers and employment agencies use automated employment decision tools in hiring and promotion in New York City. The Department of Consumer and Worker Protection began enforcing it on July 5, 2023. The department's page on the law links the rules and an FAQ, and it is the place to check the current text.

What the law requires

A bias audit within the past year

An employer may not use a covered tool unless it has had a bias audit within one year before use. The audit is run by an independent auditor, meaning someone who was not involved in using, developing or distributing the tool and who has no employment relationship or financial interest that the rules exclude.

Specific calculations

The audit calculates selection rates or scoring rates and impact ratios across sex categories, race and ethnicity categories and their intersections.

A published summary

Before use, the employer publishes a summary of the most recent audit on the employment section of its website. The summary gives the date of the audit, the source of the data and an explanation of it, the number of people assessed in each category, including an unknown category, and the selection or scoring rates and impact ratios the audit calculated. The rules add further disclosures in some cases, such as an audit that used test data or one that left a very small category out of the impact ratios, so check the full list in the rules. It also gives the distribution date, which is the date the employer began using the tool. A clearly identified link meets the requirement, and the summary stays up for at least 6 months after the tool's latest use.

Notice to candidates

Candidates and employees who live in New York City receive notice at least 10 business days before the tool is used. For candidates the notice can go on the careers page, in the job posting or by mail or email. For employees being considered for promotion it goes in a written policy or procedure, in the job posting or by mail or email. It says that the tool will be used, which job qualifications and characteristics it assesses and how to request an alternative selection process or a reasonable accommodation under other laws, where one is available.

The employer also posts the type and source of the data the tool collects and its retention policy on the employment section of its website, with instructions for requesting that information in writing. It answers a written request within 30 days.

Whether it applies to your firm

The law covers a tool that substantially assists or replaces discretionary decision making. The rules define that as relying on the tool's output alone, weighting it more than any other criterion or using it to overrule conclusions drawn from other factors, including human decision making.

Whether the law applies to a firm depends on how that firm uses the tool in its own process, including how much weight a score carries against interviews, grades and the rest of the file. That analysis belongs to the firm and its counsel.

How Suited tests its models

Every firm-specific model is tested for adverse impact across race and gender before it is deployed. The testing follows the framework the EEOC Uniform Guidelines define: 4/5ths-rule selection-rate ratios, backed by standardized effect sizes. Models that produce unjustified disparate impact are not deployed.

Every model also has an independent bias audit before deployment and annually after that, whether or not the firm's use of Suited falls under Local Law 144. Suited's models are audited by Holistic AI and have been found to show no evidence of disparate impact, on a standalone or an intersectional basis.

Demographic information is never an input to scoring. Candidates can choose to provide it, and Suited collects it so that every model can be tested.

What stays with the firm

The law's obligations sit with the employer. Suited supplies the scores and the audit work behind them, and it does not accept or reject candidates. The firm decides how scores enter its process, publishes the audit summary where the law requires one and gives candidates the required notice.

A checklist for the next recruiting cycle

Each item has an owner at most firms: counsel for the first, recruiting for the rest. Ask your Suited team for the most recent audit summary for your firm's model when you start.

  • Counsel has reviewed how recruiting uses each tool against the rules' definition of substantial assistance.
  • Recruiting holds the most recent audit for each tool and knows the date the next one is due.
  • The audit summary, with every figure and disclosure the rules require and the distribution date, is on the careers site behind a clearly labeled link, and stays up for 6 months after the tool's last use.
  • The notice reaches candidates and promotion candidates at least 10 business days before use, by the method the rules allow for each.
  • The notice explains how to request an alternative selection process or an accommodation.
  • The careers site explains the data type, source and retention policy and how to request them, and someone owns written requests and answers within 30 days.

Know more about your next class before you commit to it.

Tell us your timeline. Suited shows you what your team can know about every candidate before offers go out.

See how your firm hires