Snyk Code versus the Free Tools

Two lessons in this chapter have now argued that Snyk does not earn its price. On dependencies it found five CVEs Trivy did not, out of ~56. On Terraform it matched Trivy's total and reported zero criticals.

This lesson is the other side of that, and it is not close. Snyk Code found three real vulnerabilities that Semgrep's default rules do not report at all — including the SQL injection that the Semgrep lesson needed a hand-written taint rule to catch.

If you want one concrete answer to "what does a commercial SAST engine actually buy me", it is on this page.

Verified — Snyk Code (CLI 1.1306.3, org jaybilgaye) and Semgrep 1.166.0, both against app/ in the Range. Output copied from real runs.

Step 1 — Enable it, then run it

Snyk Code is off by default on a new organisation. Running it before enabling gives you this, which looks like an auth problem and is not:

ERROR  Snyk Code is not enabled (SNYK-CODE-0005)
       Snyk Code is not supported for your current organization: `jaybilgaye`.
Status: 403 Forbidden

Turn it on in the web UI under Settings → Snyk Code for your org. Snyk Open Source and Snyk IaC need no such toggle, which is why you can be fully authenticated and still get a 403 from this one command.

git clone https://github.com/jaybilgaye/aiopsone-range
cd aiopsone-range

snyk code test app/

Step 2 — Nine findings

✗ [HIGH]   Hardcoded Non-Cryptographic Secret          server.js:25
✗ [HIGH]   Hardcoded Non-Cryptographic Secret          server.js:26
✗ [HIGH]   SQL Injection                               server.js:56
✗ [HIGH]   Cross-site Scripting (XSS)                  server.js:63
✗ [HIGH]   Cross-site Scripting (XSS)                  server.js:73
✗ [MEDIUM] Information Exposure - X-Powered-By Header  server.js:12
✗ [MEDIUM] Use of Hardcoded Passwords                  server.js:27
✗ [MEDIUM] Allocation of Resources Without Throttling  server.js:41
✗ [MEDIUM] Allocation of Resources Without Throttling  server.js:51

Total issues: 9

Semgrep, same directory, same afternoon:

semgrep scan --config=auto app/

Ran 206 rules on 4 files: 7 findings.

  [HIGH  ] express-check-csurf-middleware-usage   server.js:12
  [HIGH  ] detected-aws-access-key-id-value       server.js:25
  [HIGH  ] detected-aws-secret-access-key         server.js:26
  [MEDIUM] template-explicit-unescape             server.js:25, 26
  [MEDIUM] raw-html-format                        server.js:73
  [MEDIUM] direct-response-write                  server.js:73

Nine versus seven is not the story. Line 56, line 63 and line 27 are.

Step 3 — The SQL injection Semgrep walks past

const sql = "SELECT * FROM products WHERE name LIKE '%" + query + "%'";
const out = db.exec(sql);                                    // line 56

Snyk Code:

SQL Injection — Unsanitized input from an HTTP parameter flows into exec, where it is used in an SQL query.

Read that sentence carefully, because it describes a path: req.query.q → query → string concatenation → sql → db.exec(). That is taint analysis — tracking a value from an untrusted source to a dangerous sink through the assignments in between.

Semgrep did not report it under --config=auto (206 rules), and the earlier lesson established it does not report it under p/javascript, p/secrets or p/owasp-top-ten either. The reason is not that Semgrep lacks taint analysis — it has a good one. It is that its registry rules do not model sql.js as a SQL sink. No rule says "db.exec() from sql.js executes SQL", so no path is traced.

Snyk's engine recognised exec as a query sink without anyone telling it about this project.

This is the whole commercial argument, and it is narrow. Semgrep catches this the moment you write a twelve-line taint rule naming the sink. The paid engine's value is that it already knows about sinks nobody on your team has thought to declare — and you cannot write a rule for a sink you have not noticed.

Step 4 — The finding neither free tool got anywhere near

Line 63 is inside the error handler:

} catch (err) {
  // FLAW 3 — verbose error disclosure: leaks the query and the schema.
  res.status(500).send('<pre>' + err.message + '\n' + sql + '</pre>');
}

Snyk Code called this Cross-site Scripting (XSS), and it is right in a way worth slowing down for.

sql is not user input. It is a string this code built. But it was built from req.query.q back on line 53 — so the user's input reaches the browser, unescaped, inside an HTML <pre> block, via the exception path. The taint survives concatenation into a different variable and re-emerges in a catch block twelve lines later.

That is second-order reflection, and it needs interprocedural dataflow to see. A pattern-matching rule looking for res.send(<something user-controlled>) does not fire, because at line 63 the argument is err.message and sql, neither of which looks like request input.

Three other tools have now touched this exact line and described it as something else entirely:

• Semgrep: reported nothing at line 63.
• OWASP ZAP: this handler is why ZAP found the SQL injection — ?q=' produced a 500 whose body leaked the statement. ZAP read it as evidence of SQLi and never considered it an XSS sink.
• Nuclei: only ever tested /greet, so it never reached this path.

One line, four tools, and only one of them saw the flaw that is actually there.

Step 5 — The password three secrets scanners missed

const DB_PASSWORD = 'hunter2';        // line 27

Snyk Code: Use of Hardcoded Passwords — "Found hardcoded password used in DB_PASSWORD."

Every dedicated secrets tool in the Secrets chapter missed this. gitleaks, TruffleHog and GitGuardian all reported the AWS key pair on lines 25–26 and said nothing about line 27, because hunter2 is a word. It has no entropy, no prefix, no structure — nothing for a detector keyed on shape.

Snyk Code found it by looking at the assignment target instead of the value. A string literal assigned to something called DB_PASSWORD is a hardcoded password regardless of what the string contains.

This is a genuinely useful lesson about tool categories. A SAST engine reading code structure found a secret that three purpose-built secrets scanners could not, because they were solving a pattern-matching problem and it was solving a semantics problem. "We run a secrets scanner" does not cover hardcoded credentials that do not look like credentials.

Semgrep did not report line 27 either, under any config tested.

Step 6 — Where it does not earn it

Two of the nine are the kind of finding that inflates a count:

[MEDIUM] Allocation of Resources Without Limits or Throttling   line 41
[MEDIUM] Allocation of Resources Without Limits or Throttling   line 51

Line 41 is app.get('/', ...) rendering a template. Line 51 is /search. The reasoning — a handler doing filesystem work with no rate limit could be a denial-of-service vector — is defensible in the abstract, and it applies to essentially every Express route ever written. On a real application this class of finding arrives once per endpoint and is the first thing anyone learns to ignore.

Semgrep produced its own version of this: two template-explicit-unescape findings at lines 25–26, which are the AWS credential constants, not templates.

Both tools generate noise; neither is meaningfully cleaner. The difference is entirely in what they catch, not in what they leave out.

Also note what Snyk Code did not find: the /debug/config endpoint that returns AWS credentials as JSON to an unauthenticated caller. It is not a taint flow — the code does exactly what it says — so a dataflow engine has no opinion on it. Only a DAST tool aimed at that path found it.

Step 7 — The head-to-head

By vulnerability rather than by count:

Three real vulnerabilities found by one and not the other, all in the same direction. That is a different result from the SCA and IaC comparisons, where the exclusives were marginal and ran both ways.

Snyk also rated both XSS findings HIGH where Semgrep rated line 73 MEDIUM — so a --severity-threshold=high gate blocks on Snyk and passes on Semgrep.

Step 8 — Gate it

# .github/workflows/sast.yml
name: sast
on: [pull_request]

permissions:
  contents: read

jobs:
  snyk-code:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: snyk/actions/setup@master
      - run: snyk code test app/ --severity-threshold=high
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

--severity-threshold=high gates on the five HIGH findings and lets the rate-limiting noise report without blocking. As established in the IaC lesson, a threshold is a decision you make per tool — it does not port across scanners.

Snyk Code sends your source to a hosted service for analysis. That is a conversation to have before pointing it at a private repository, and it is the single largest practical objection to it. Semgrep runs entirely offline, which for some organisations settles the question regardless of the findings above.

Step 9 — So what should you actually do

Run Semgrep, and write the rules. It is free, offline, fast, and the twelve-line taint rule catches the SQL injection on this page. Custom rules are where coverage of your stack comes from, and that is true whichever engine you buy.

Add Snyk Code when you cannot enumerate your own sinks. That is not a hypothetical — it is the normal condition of any codebase with more than a handful of contributors, a few years of history, and libraries nobody currently on the team chose. Line 63 is the honest illustration: nobody would have written a rule for "user input reaches the browser through a catch block", because nobody knew it did.

The order matters. Free tool plus your own rules first, because it makes you enumerate your sinks — and that exercise is worth more than either tool. Buy the commercial engine for the sinks that exercise misses.

What you learned

• Snyk Code is off by default per organisation. SNYK-CODE-0005 / 403 is a feature toggle, not an auth failure.
• It found the SQL injection at line 56 that Semgrep misses under auto, p/javascript, p/secrets and p/owasp-top-ten — because Semgrep's registry does not model sql.js as a SQL sink.
• It found a second XSS at line 63, where user input reaches the browser through the catch block via the sql variable. Second-order reflection; no other tool in this track saw it.
• It found DB_PASSWORD = 'hunter2' that gitleaks, TruffleHog and GitGuardian all missed — by reading the assignment target rather than the value.
• It also produced noise: "no rate limiting" on two ordinary Express routes. Semgrep produced its own. Neither is cleaner.
• It missed /debug/config entirely — not a taint flow, so a dataflow engine has nothing to say.
• This is where commercial SAST earns its price, and it is narrow: it knows sinks you have not thought to declare. On SCA and IaC the same vendor added very little.
• Semgrep runs offline; Snyk Code uploads your source. For some organisations that decides it before any finding does.