Clojure translation part II by dimitris-m · Pull Request #517 · opengrep/opengrep

dimitris-m · 2026-01-02T17:02:27Z

Changes

Removes synthetic variables from the intermediate variables, since the tokens used are real tokens with different content in the target file. This improves dataflow traces in text / json / sarif.
Improves printing of matches which in some cases failed with macroexpanded code.
Encodes the special ops as is done in other languages.
Adds loop and recur (this can be used with loop or functions, in tail position). Note: I did not add a test for defn + recur because it made me realise we have other issues that are related and need to be fixed, and these are orthogonal to the translation.
Improves patterns:
- (...) matches a block and not a Call(..., []);
- ..., when function parameters are expected, becomes [...];
- ..., when a single name is expected, becomes $_.
Fixes 1.14.0 - Rule parse error in rule clojure.lang.security.documentbuilderfactory-xxe.documentbuilderfactory-xxe #518.
Removes the IL macroexpansion mechanics; this approach ended up being too problematic compared to macroexpansion at AST generic translation time.
Adds comments and TODOs: some will be dealt with in this PR.
There will be Part III.

In macroexpansion for `cond->` and `cond->>`, we introduce fake variables in synthetic let bindings, and we don't want these to pollute the `intermediate_variables` produced in outputs. This is because these fake variables reuse real tokens that have no relevance in terms of content, but are the closest and most reasonable choices for such variables. Showing them is bound to confuse tooling downstream.

Without this edit, adding `--dataflow-traces` will sometimes fail to print traces where a location spans more than 1 line. This happens with clojure macroexpanded code.

This fixes an small issue where intermediate variable tokens derived from qualified x/y in associative destructuring included the slash, and printed in json and sarif as /y.

Note that `($F ...)` will no longer match such expressions; but `(:K ...)` will, and similarly for `(::kwd exp)` and `(::$K ...)`.

If we are to do this, we need to cover more cases as detailed in the comment.

corneliuhoffman · 2026-01-06T11:11:27Z

languages/lisp/tree-sitter/Macro_expand_clojure.ml

 let is_macroexpandable (todo_kind : string) =
  match todo_kind with
-  | "->" | "->>" | "cond->" | "cond->>" | "as->" | "ShortLambda"
+  | "as->" | "ShortLambda"


wait so the -> liek things are no longer macroexpandable?

They are, but at translation to generic.
There were 2 methods: during IL translation and during to-generic.

Now what is kept is to-generic. That means we shuffle CST stuff not generic AST.

Until before this commit I had both methods there, hoping to resurrect the IL method but I decided to abandon it.

corneliuhoffman · 2026-01-06T11:14:57Z

languages/lisp/tree-sitter/Parse_clojure_tree_sitter.ml


 let qualified_name_regex_str = "^\\(.+\\)/\\(.+\\)$"

+let fake_variable_ident = "G__1111"


oh common, call it G_666

corneliuhoffman

I really liked the details in the recur/loop stuff ... this is better and better, soon will be a flag language for us

We now relax tha parsing of clojure functions, to ensure that semgrep-rules don't fail to get loaded because of parsing error. The change is moderate: - Translate `...` to `[...]` when expecting function arguments. This enables the pattern `(defn $F ... ...)` which otherwise fails to parse. - Convert `...` to `$_` in contexts where a single name is expected, for example in function name position. - We interpret `( ... )` as a block, not as function call Call(..., []). Similarly for ( ... e1 e2) etc.

Decided to abandon this avenue. Too many issues.

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [opengrep/opengrep](https://github.com/opengrep/opengrep) | minor | `v1.13.2` → `v1.14.1` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>opengrep/opengrep (opengrep/opengrep)</summary> ### [`v1.14.1`](https://github.com/opengrep/opengrep/releases/tag/v1.14.1): Opengrep 1.14.1 [Compare Source](opengrep/opengrep@v1.14.0...v1.14.1) #### Improvements - Clojure translation part II by [@dimitris-m](https://github.com/dimitris-m) in [#517](opengrep/opengrep#517) - C#: Allow implicit variables in properties to be taint sources by [@maciejpirog](https://github.com/maciejpirog) in [#516](opengrep/opengrep#516) - Add core flags `dump_rule` and `dump_patterns_of_rule` as options in the show command by [@maciejpirog](https://github.com/maciejpirog) in [#519](opengrep/opengrep#519) #### Bug fixes - Fix: pass signature databaseb to lambda analysis, handle method mutation tainting by [@corneliuhoffman](https://github.com/corneliuhoffman) in [#520](opengrep/opengrep#520) #### Tech debt - Fix CHANGELOG.md, OPENGREP.md, remove unused files by [@dimitris-m](https://github.com/dimitris-m) in [#523](opengrep/opengrep#523) **Full Changelog**: <opengrep/opengrep@v1.14.0...v1.14.1> ### [`v1.14.0`](https://github.com/opengrep/opengrep/releases/tag/v1.14.0): Opengrep 1.14.0 [Compare Source](opengrep/opengrep@v1.13.2...v1.14.0) #### Improvements - Support for higher-order functions in intrafile taint analysis by [@corneliuhoffman](https://github.com/corneliuhoffman) in [#469](opengrep/opengrep#469) and [#513](opengrep/opengrep#513) - Clojure: Improved support for Clojure (incl. tainting) by [@dimitris-m](https://github.com/dimitris-m) in [#501](opengrep/opengrep#501) - Dart: Improved support for Dart by [@maciejpirog](https://github.com/maciejpirog) in [#508](opengrep/opengrep#508) - C#: Better handing of extension methods and extension blocks by [@maciejpirog](https://github.com/maciejpirog) in [#514](opengrep/opengrep#514) #### Fixes - Bump cygwin install action by [@dimitris-m](https://github.com/dimitris-m) in [#503](opengrep/opengrep#503) and [#509](opengrep/opengrep#509) **Full Changelog**: <opengrep/opengrep@v1.13.2...v1.14.0> </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

dimitris-m requested review from corneliuhoffman, maciejpirog and willem-delbare as code owners January 2, 2026 17:02

dimitris-m force-pushed the dm/clojure-part-ii branch from 4972a8d to c1fa853 Compare January 2, 2026 17:03

dimitris-m removed the request for review from willem-delbare January 2, 2026 17:04

dimitris-m added taint lang Add or improve language support labels Jan 2, 2026

dimitris-m force-pushed the dm/clojure-part-ii branch from 03ee23e to a7afa9b Compare January 3, 2026 13:37

dimitris-m added 10 commits January 5, 2026 22:50

fix output: Semgrep OSS => Opengrep

ef7dd60

fix dataflow traces output for macroexpanded code

20e69b5

Without this edit, adding `--dataflow-traces` will sometimes fail to print traces where a location spans more than 1 line. This happens with clojure macroexpanded code.

fix: from x/y extract token with y and not /y

177a6d2

This fixes an small issue where intermediate variable tokens derived from qualified x/y in associative destructuring included the slash, and printed in json and sarif as /y.

improve comments and add some more todos

e194e81

encode special ops

afad689

translate (:kwd exp) to G.DotAccess (exp, _, :kwd)

a7991d4

Note that `($F ...)` will no longer match such expressions; but `(:K ...)` will, and similarly for `(::kwd exp)` and `(::$K ...)`.

revert: (:kwd e) to DotAccess (exp, _, :kwd)

96254d9

If we are to do this, we need to cover more cases as detailed in the comment.

add loop and recur

69f0684

make recur work with functions too

35229be

dimitris-m force-pushed the dm/clojure-part-ii branch 6 times, most recently from 3721920 to 2968876 Compare January 6, 2026 00:36

dimitris-m changed the title ~~WIP: Clojure translation part ii~~ Clojure translation part II Jan 6, 2026

dimitris-m force-pushed the dm/clojure-part-ii branch 2 times, most recently from 6e9a2aa to a65c5b2 Compare January 6, 2026 02:09

corneliuhoffman reviewed Jan 6, 2026

View reviewed changes

corneliuhoffman approved these changes Jan 6, 2026

View reviewed changes

maciejpirog approved these changes Jan 6, 2026

View reviewed changes

dimitris-m force-pushed the dm/clojure-part-ii branch 2 times, most recently from c1319bf to c48b272 Compare January 6, 2026 13:11

dimitris-m added 2 commits January 6, 2026 13:16

remove macroexpansion at IL gen time

20e46fa

Decided to abandon this avenue. Too many issues.

dimitris-m force-pushed the dm/clojure-part-ii branch from c48b272 to 20e46fa Compare January 6, 2026 13:17

dimitris-m merged commit 5dd8f3e into main Jan 6, 2026
3 checks passed

dimitris-m deleted the dm/clojure-part-ii branch January 6, 2026 13:24

maciejpirog mentioned this pull request Jan 6, 2026

1.14.0 - Failed to scan BenchmarkJava/scorecard/content/js/bootstrap.js #521

Closed

3 tasks

dimitris-m mentioned this pull request Jan 6, 2026

Release v1.14.1 #524

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clojure translation part II#517

Clojure translation part II#517
dimitris-m merged 12 commits intomainfrom
dm/clojure-part-ii

dimitris-m commented Jan 2, 2026 •

edited

Loading

Uh oh!

corneliuhoffman Jan 6, 2026

Uh oh!

dimitris-m Jan 6, 2026

Uh oh!

dimitris-m Jan 6, 2026

Uh oh!

corneliuhoffman Jan 6, 2026

Uh oh!

corneliuhoffman left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		let qualified_name_regex_str = "^\\(.+\\)/\\(.+\\)$"

		let fake_variable_ident = "G__1111"

Conversation

dimitris-m commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

corneliuhoffman Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

dimitris-m Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

dimitris-m Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

corneliuhoffman Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

corneliuhoffman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dimitris-m commented Jan 2, 2026 •

edited

Loading