This issue is to keep track of all the tasks required to remove the Tok dependency from the downstream tools (linter and formatter).

Notes:

• Certain tasks might need to be clubbed with the parser changes
• The implementation logic for certain usages isn't finalized yet

Linter

• Move these rules to the AST checker:
  • PLE1300, PLE1307 (Avoid lexer usage in PLE1300 and PLE1307 #11406)
  • UP031
  • W605 (Move W605 to the AST checker #11402)
• doc_lines_from_tokens (Use TokenKind in doc_lines_from_tokens #11418)
• Following rules can be updated to directly use TokenKind (via Add Tokens newtype wrapper, TokenKind iterator #11361):
  • COM812, COM818, COM819 (Move most of token-based rules to use TokenKind #11420)
  • E301 .. E306 (blank line rules) (Use TokenKind in blank lines checker #11419)
  • E701 .. E703 (Move most of token-based rules to use TokenKind #11420)
  • ISC001, ISC002 (Move most of token-based rules to use TokenKind #11420)
  • PLE2510, PLE2512 .. PLE2515 (Move most of token-based rules to use TokenKind #11420)
  • UP034 (Move UP034 to use TokenKind instead of Tok #11424)
  • W391 (Move most of token-based rules to use TokenKind #11420)
• remove_import_members uses lex method (Replace most usages of lex_starts_at with Tokens #11511)
• extract_noqa_line_for uses the kind information from the string token
• locate_cmp_ops uses the lex method (Replace lex usages #11562)
• tokens_and_ranges extracts the comment ranges from the token stream
• unsorted_dunder_all and unsorted_dunder_slots extracts the value from the string token (Replace most usages of lex_starts_at with Tokens #11511)
• UP032 uses the lex_starts_at method (Replace most usages of lex_starts_at with Tokens #11511)
• I001 (trailing_comma) uses the lex_starts_at method (Replace most usages of lex_starts_at with Tokens #11511)
• Stylist extracts the indentation and quote information from the token (Update Stylist, Indexer to use tokens from parsed output #11592)
• Indexer extracts trivia and string ranges, line continuation start value (Update Stylist, Indexer to use tokens from parsed output #11592)

Formatter

• Move away from lex_starts_at in write_suppressed_statements_starting_with_trailing_comment (Replace lex_starts_at with Tokens in the formatter #11515)

Internal document: https://www.notion.so/astral-sh/Downstream-work-items-551b86e104a34054b7192675550a6c25?pvs=4