core(robots-txt): add `Content-Signal` directive by Lulucmy · Pull Request #16767 · GoogleChrome/lighthouse

Lulucmy · 2025-10-19T13:12:32Z

Summary
Cloudflare recently introduced a Content-Signal robots.txt directive so site owners can control which AI crawlers may use their content and how. It already appears on 3.8 millions websites Cloudflare-managed sites, but Lighthouse currently flags it as an “Unknown directive.”
This PR updates the robots.txt safelist and its corresponding tests so the directive is recognized.

Resources:

Related Issues/PRs
None

tectrixdev · 2026-01-05T17:07:03Z

+1, this would be very useful.

soywig · 2026-01-08T12:03:02Z

+1 too

connorjclark · 2026-01-12T22:47:51Z

Thanks!

Puneethgr · 2026-02-18T17:46:13Z

The issue is still present. Is the Content-Signal PR fix deployed to prod yet?
Another person mentioned the same issue here: #16776 (comment)

The following line in robots.txt is auto-managed by Cloudflare:

Content-Signal: search=yes,ai-train=no

The PageSpeed test shows this error for me:

Unknown Directive

Screenshot below:

karenalmeida18 · 2026-02-18T18:22:12Z

The issue is still present

KasimAhmic · 2026-02-26T10:56:40Z

The issue is still present

The issue is still present. Is the Content-Signal PR fix deployed to prod yet? Another person mentioned the same issue here: #16776 (comment)

Looks like it's scheduled to release with Chrome 146 on March 10, 2026:

Puneethgr · 2026-03-14T10:37:41Z

Looks like it's scheduled to release with Chrome 146 on March 10, 2026:

https://github.com/GoogleChrome/lighthouse/releases/tag/v13.0.2#:~:text=robots%2Dtxt%3A%20add%20Content%2DSignal%20directive

https://chromiumdash.appspot.com/schedule#:~:text=Feb%2025%2C%202026-,Stable%20Release,-Tue%2C%20Mar%2010

Chrome 146 version is the Stable release deployed on March 10, but the robot.txt issue (Error: Unknown directive) for Content-Signal is still present in PageSpeed test as of March 14.

PageSpeed test URL tested on 14 March 2026: https://pagespeed.web.dev/analysis/https-puneethgr-com/65o2ft4vey?form_factor=desktop

robots.txt file section:

# BEGIN Cloudflare Managed content

User-agent: *
Content-Signal: search=yes,ai-train=no
Allow: /

Puneethgr · 2026-03-24T09:30:57Z

Just an update: when running the test locally using node cli with my website, the robots.txt shows valid (SEO score 100) and parsing of Content-Signal is successful (attached screenshot below)

Command: node cli https://puneethgr.com --view

  LH:status Auditing: Page isn’t blocked from indexing +1ms
  LH:status Auditing: robots.txt is valid +0ms

Local test result:

But the official PageSpeed test still shows a SEO score of 92 (Unknown directive error for Content-Signal: search=yes,ai-train=no): https://pagespeed.web.dev/analysis/https-puneethgr-com/8bgijy0wrk?form_factor=desktop
Not sure why this is happening.

soywig · 2026-03-24T15:46:46Z

the issue was solved on Chromium 146. PageSpeed still runs Chromium 145. Google just has to update their Chromium version. i guess we just gotta wait till then.

solonic22-cpu · 2026-03-24T20:39:39Z

Summary
Cloudflare recently introduced a Content-Signal robots.txt directive so site owners can control which AI crawlers may use their content and how. It already appears on 3.8 millions websites Cloudflare-managed sites, but Lighthouse currently flags it as an “Unknown directive.”
This PR updates the robots.txt safelist and its corresponding tests so the directive is recognized.

Resources:

https://contentsignals.org

Cloudflare Gives Creators New Tool to Control Use of Their Content

Related Issues/PRs
None

solonic22-cpu · 2026-03-24T21:06:31Z

78d357da8b54aec2672b76bee522974d8d42db5c

solonic22-cpu · 2026-03-24T21:08:02Z

9806bc4

solonic22-cpu · 2026-03-24T21:32:09Z

https://github.com/nixsticks/todos/blob/# Finding public code that matches GitHub Copilot suggestions

Learn how to view code references when GitHub Copilot makes suggestions that matches publicly available code.

This version of this article is for Copilot in JetBrains IDEs. For Copilot on other platforms, click the appropriate tab above.

This version of this article is for Copilot in Visual Studio Code. For Copilot on other platforms, click the appropriate tab above.

This version of this article is for Copilot on the GitHub website. For Copilot on other platforms, click the appropriate tab above.

This version of this article is for Copilot in Visual Studio. For Copilot on other platforms, click the appropriate tab above.

Introduction

If you allow GitHub Copilot to make suggestions that match publicly available code or use a product that does not support "Block" mode, Copilot will display references to any similar code that is found. See GitHub Copilot code referencing.

Prerequisites

References to matching code are only generated if you use a product that does not support "Block" mode, or if Copilot is configured to allow suggestions that match publicly available code. This is configured in either your personal or organization settings.

For more information, see Managing GitHub Copilot policies as an individual subscriber or Managing policies and features for GitHub Copilot in your organization.

View code references for inline suggestions

You can view code references in the log file for your JetBrains IDE.

In your JetBrains IDE, select Help > Show Log in Finder/Explorer.

The log file is displayed in your file manager. For example, for IntelliJ IDEA the log file is called idea.log.
Open the log file in your JetBrains IDE.
Search for "[Public Code References]."

Example log entry

2025-02-26 09:22:12,045 [5581906] INFO - #copilot - [Public Code References] Text found matching public code in file:///Users/mona-lisa/git-repos/test-repo/fizzbuzz.js [Ln 1, Col 10] near fizzBuzz() ...:
  1) [NOASSERTION] https://github.com/nixsticks/todos/blob/ae427a721c7784da64a619ba17f60637fe1cc819/Loops/fizzbuzz/fizzbuzz.js
  2) [GPL-3.0] https://github.com/voloslg/algocasts/blob/34b423517486f908ca167b390d3b8bd05653829f/exercises/fizzbuzz/index.js

The log entry includes the following details:

The date and time you accepted the suggestion.
A "Public Code References" message telling you that similar code was found.
The path to the file in which the suggestion was added.
The line and column number where the suggestion was added.
A list of matches, including:
- The license type for the matching code—or NOASSERTION if no license was found.
- The URL of the file on GitHub.com where the matching code was found.

Verifying the code referencing functionality

You can verify that code referencing is working by prompting Copilot to add some commonly used code and checking the output in the log.

Create a file called fizz-buzz.js and open it in the editor.
Display the log as described in the previous section.
In the editor, type:
```
function fizzBuzz()
```
With a space after the closing parenthesis.

GitHub Copilot should suggest code to complete the function. Typically the suggestion will be a common implementation of the fizz buzz algorithm that will match publicly available code on the GitHub website.
To accept the suggestion, press Tab.
Check whether any entries for similar code have been added to the log.

View code references for inline suggestions

You can find code references in one of the GitHub Copilot logs in Visual Studio Code.

In Visual Studio Code, open the Output window by selecting View > Output from the menu bar.
In the dropdown menu at the right of the Output window, select GitHub Copilot Log (Code References).
Leave the GitHub Copilot Log (Code References) view displayed while you use GitHub Copilot in Visual Studio Code.

When you accept an inline suggestion that matches code in a public GitHub repository, an entry is added to the log.

The log entry includes the following details:
- The date and time you accepted the suggestion.
- The name of the file in which the suggestion was added.
- "Similar code at" followed by the location in the file where the suggestion was added.
- An extract of the code that was added by the inline suggestion.
- The license type for the matching code, if found, otherwise unknown.
- The URL of the file on GitHub.com where the similar code was found.

Example log entry

2025-03-27 12:17:54.759 [info] file:///Users/monalisa/fizzbuzz.js Similar code at  [Ln 2, Col 8] let i = 1; i <= 100; i++) {  let output = '';  if (i % 3 === 0) {  output += 'Fizz';...
2025-03-27 12:17:54.759 [info] License: unknown, URL: https://github.com/octo-org/octo-repo/blob/8563f3b1d4f33952b22212b86e745539d1567ed1/examples/fizzBuzz.js
2025-03-27 12:17:54.759 [info] License: MIT, URL: https://github.com/octo-org/monalisa/blob/7e974691f4c8e6bc55f9b50688f05d746d1bc52b/exercises/2/fizz-buzz.js

Verifying the code referencing functionality

You can verify that code referencing is working by prompting Copilot to add some commonly used code and checking the output in the log.

Create a file called fizz-buzz.js and open it in the editor.
Display the log as described in the previous section.
In the editor, type:
```
function fizzBuzz()
```
With a space after the closing parenthesis.

GitHub Copilot should suggest code to complete the function. Typically the suggestion will be a common implementation of the fizz buzz algorithm that will match publicly available code on the GitHub website.
To accept the suggestion, press Tab.
Check whether any entries for similar code have been added to the log.

View code references for inline suggestions

You can find code references in the GitHub Copilot log in Visual Studio.

In the menu bar, click View.
In the dropdown menu, click Output.
In Output view, click the box to the right of "Show output from" and select GitHub Copilot.
Leave the log displayed while you use GitHub Copilot in Visual Studio Code.

When you accept an inline suggestion that matches code in a public GitHub repository, an entry is added to the log.

The log entry includes the following details:
- The time you accepted the suggestion. Click the "Show Timestamp" clock icon if the time is not displayed.
- The description [Completions Public Code Match Information].
- The license type for the matching code, if found, otherwise NOASSERTION.
- The URL of the file on GitHub.com where the similar code was found.

Example log entry

09:39:16:203	[Completions Public Code Match Information] Similar code with license type [MIT] https://github.com/octo-org/octo-repo/blob/34deb75eb6a2e22483ed465a6aec38c02eb2536e/routines/quicksort.js

Verifying the code referencing functionality

You can verify that code referencing is working by prompting Copilot to add some commonly used code and checking the output in the log.

Create a file called fizz-buzz.js and open it in the editor.
Display the log as described in the previous section.
In the editor, type:
```
function fizzBuzz()
```
With a space after the closing parenthesis.

GitHub Copilot should suggest code to complete the function. Typically the suggestion will be a common implementation of the fizz buzz algorithm that will match publicly available code on the GitHub website.
To accept the suggestion, press Tab.
Check whether any entries for similar code have been added to the log.

View code references for Copilot Chat

If a response in Copilot Chat includes matching code, this is indicated at the end of the response by the following text:

Similar code found with n license types - View matches

Click View matches to display details of the matched code in a new editor tab.

For each example of matching code, the editor displays:
- The license type for the matching code, if known.
- The URL of the file on GitHub.com where the matching code was found.
- A code snippet showing the matching code.
In the editor, Ctrl+click (Windows/Linux) or Command+click (Mac) a URL to view the full file on GitHub.com.

If a response in Copilot Chat includes matching code, this is indicated at the end of the response by the following text:

Similar code found with n license types - View matches

Click View matches to display details of the matched code in a new editor tab.

For each example of matching code, the editor displays:
- The license type for the matching code, if known.
- The URL of the file on GitHub.com where the matching code was found.
- A code snippet showing the matching code.
In the editor, Ctrl+click (Windows/Linux) or Command+click (Mac) a URL to view the full file on GitHub.com.

If a response in Copilot Chat includes matching code, this is below the suggested code by the following text:

Found similar code in public repos. View matches

Click View matches to open the GitHub Copilot log, if it is not already open, and add details of the matched code.

The details include:

The time you added the details to the log. Click the "Show Timestamp" clock icon if the time is not displayed.
The description [Code Match] as the first log entry before the list of matching code.
The license type—if found—for each instance of similar code.
The URL of the file on GitHub.com where the matching code was found.
A code snippet showing the matching code.

Logging example

09:24:10:525	[Code Match] Similar code with 2 license type(s) [MIT, NOASSERTION]
09:24:10:525	## License: MIT
09:24:10:525	https://github.com/octo-org/octo-repo/tree/127aac4ab27a42706af01be80f7aae3b83f44fbc/buzzfizz.py
09:24:10:525	```
09:24:10:525	for i in range(1, n + 1):
09:24:10:525	        if i % 3 == 0 and i % 5 == 0:
09:24:10:525	            print('FizzBuzz')
09:24:10:525	        elif i % 3 == 0:
09:24:10:525	            print('Fizz')
09:24:10:525	        elif i % 5 == 0:
09:24:10:525	```
09:24:10:525	## License: NOASSERTION
09:24:10:525	https://github.com/octo-org/monalisa/tree/011308746e53b26b128fa53c044a2527c39231f0/fizz-buzz.py
09:24:10:525	```
09:24:10:525	i % 3 == 0 and i % 5 == 0:
09:24:10:525	            print('FizzBuzz')
09:24:10:525	        elif i % 3 == 0:
09:24:10:525	            print('Fizz')
09:24:10:525	        elif i % 5 == 0:
09:24:10:525	            print('Buzz')
09:24:10:525	        else:
09:24:10:525	            print(i)
09:24:10:525	```

View code references for Copilot Chat

When Copilot Chat provides a response that includes code that matches code in a public GitHub repository, this is indicated beneath the code suggestion:

< > Public code references from n repositories

To see details of the matching code:

Click the "Public code references..." text, under the code suggestion.

A list of GitHub repositories containing matching code is displayed in a dropdown, together with licensing information, if found.
Click the name of a repository to display that repository on GitHub.com.

View code references for Copilot coding agent

When Copilot provides a response that includes code that matches code in a public GitHub repository, this is indicated in the agent session logs with a link to display details of the matched code. For more information, see Tracking GitHub Copilot's sessions.

Conversation

Lulucmy commented Oct 19, 2025

Uh oh!

tectrixdev commented Jan 5, 2026

Uh oh!

soywig commented Jan 8, 2026

Uh oh!

Uh oh!

connorjclark commented Jan 12, 2026

Uh oh!

Puneethgr commented Feb 18, 2026

Uh oh!

karenalmeida18 commented Feb 18, 2026

Uh oh!

KasimAhmic commented Feb 26, 2026

Uh oh!

Puneethgr commented Mar 14, 2026

Uh oh!

Puneethgr commented Mar 24, 2026

Uh oh!

soywig commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

solonic22-cpu commented Mar 24, 2026

Uh oh!

solonic22-cpu commented Mar 24, 2026

Uh oh!

solonic22-cpu commented Mar 24, 2026

Uh oh!

solonic22-cpu commented Mar 24, 2026

Introduction

Prerequisites

View code references for inline suggestions

Example log entry

Verifying the code referencing functionality

View code references for inline suggestions

Example log entry

Verifying the code referencing functionality

View code references for inline suggestions

Example log entry

Verifying the code referencing functionality

View code references for Copilot Chat

Logging example

View code references for Copilot Chat

View code references for Copilot coding agent

Further reading

Uh oh!

Puneethgr commented Apr 12, 2026

Uh oh!

soywig commented Apr 12, 2026

Uh oh!

HichemTab-tech commented Apr 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

soywig commented Mar 24, 2026 •

edited

Loading