171 answers.
Not one made up.

We put the copilot through 57 buyer questions about a security product, three times over, and publish every answer. Measured 13 November 2026.

Right
95%163 of 171 answers did what a careful sales engineer would.
Wrong
0No answer told the rep something the docs don't say.
To the first point
1.0 sMedian, from the question to the answer on the card.

Every answer, on one board.

One square for each of the 171 answers. The 8 outlined ones held back: the copilot said “let me confirm” or stayed quiet when it could have answered. Slower for the rep, never wrong for the buyer.

  • Right 163
  • Held back 8
  • Wrong 0

What that looks like on a call.

Real cards from the test, word for word.

It says no when your docs say no.

A wrong “yes” loses deals. When the docs say something isn't supported, the card says so, and says what is.

They ask: “We write our infrastructure in Bicep. Is that covered?”

Azure Bicep is not supported today. Our infrastructure-as-code scanning covers Terraform, AWS CloudFormation, Kubernetes manifests and Helm charts, so I'd rather be precise with you on that.

  • Azure Bicep is not supported for infrastructure-as-code scanning.
  • Supported: Terraform (HCL and plan files), AWS CloudFormation, Kubernetes manifests and Helm charts.
  • ARM templates and Pulumi are also not supported.

Right in 3 of 3 runs

It says when it doesn't know.

Nothing about FedRAMP in the docs, so no guess: the rep offers to follow up, and the question goes to the team's inbox.

They ask: “Are you FedRAMP authorized?”

Not in your knowledge. Don't guess: offer to follow up.

Right in 3 of 3 runs

It stays on what you're showing.

GitLab is supported for scanning, not for this feature. “Now showing” keeps the answer to the feature on screen.

They ask: “Does this work with GitLab too?”

Now showing: Auto-fix pull requests

Auto-fix pull requests are available on GitHub and GitHub Enterprise Server only, so GitLab isn't covered for that feature. GitLab is supported for code scanning and Repo Posture, though.

  • Auto-fix PRs: GitHub and GitHub Enterprise Server only
  • GitLab (gitlab.com) and GitLab Self-Managed 16.0+: supported for code scanning and Repo Posture

Right in 3 of 3 runs

It stays quiet when nothing's asked.

Small talk, logistics and the rep's own questions get no card, so the window stays calm.

They ask: “Sorry, you froze for a second there. Can you hear me now?”

Listening… nothing to answer, so no card.

Right in 3 of 3 runs

Strongest where buyers push hardest.

Security, pricing and features: every answer right, in every run. Questions the docs can't answer: “Not in your knowledge”, every time.

  • Security and compliance100% 27/27
  • Deployment100% 12/12
  • Pricing and plans100% 15/15
  • Features100% 12/12
  • Scoped to the feature shown100% 12/12
  • Not in the docs100% 15/15
  • Nothing to answer100% 12/12
  • Integrations90% 38/42
  • Languages and IaC83% 20/24

Getting better, in the open.

The first run showed one habit: “Not in your knowledge” when the docs plainly said “not supported”. We changed one rule, checked that unknown questions still get “Not in your knowledge”, and ran the same test again.

  • First run83% 142/171
  • Now95% 163/171

Where it still holds back.

3 times

“We're mostly GitLab CI with a bit of Jenkins” got no card. A sales engineer hears “do you support that?”. Next on our list.

5 times

“Not in your knowledge” for something the docs say isn't supported (mostly C++). The rep offers to confirm instead of saying no.

How we measure.

The docs
Seven pages about Northwind Guard, a fictional security product, saying what's supported and what isn't. Read them.
The questions
57 moments from a demo, each labelled before the run with what should happen. Download them.
The copilot
The same AI model and instructions as on customers' calls (version copilot-v9).
Scoring
Only the card's own text counts. It must contain the words a correct answer needs, and none that would make it wrong.
Runs
Three, pooled (54 to 55 of 57 each): 1, 2, 3. First measurement: 1, 2, 3.
Limits
Short docs, so search across a large knowledge base isn't tested, and timing excludes speech to text. Five labels were corrected where a right answer was scored wrong, all listed in the test file. The real test is a pilot on your own calls.

Every question, every answer.

Open a group to read what the card said in the first run.

Security and compliance27/27 right
  • 3/3

    Do you have a SOC 2 Type II report we can review?

    Yes, we have a SOC 2 Type II report, audited every year, and we can share it with you under NDA. | SOC 2 Type II, audited every year · The report is shared under NDA · Northwind is also ISO/IEC 27001 certified

  • 3/3

    Do you keep a copy of our source code after the scan?

    No, we don't keep a copy. Cloud scans clone your repository into an isolated, single-use worker, and the code is deleted when the scan ends, at the latest one hour after it started. | Northwind stores only findings: file path, line, and a snippet of up to 5 lines. · Full source code is never stored as a copy. · With the self-hosted scanner (Enterprise), your code never leaves your network.

  • 3/3

    Can our data stay in the EU?

    Yes. EU data residency is available on Business and Enterprise. You choose the region when the account is created, and it can't be changed afterwards. | Hosting is in the EU (Frankfurt, eu-central-1), or in the US (us-east-1). · EU residency is available on Business and Enterprise plans. · The region is fixed at account creation.

  • 3/3

    If we start in the US region, can we move to the EU later?

    No, the region is fixed when the account is created and can't be changed afterwards. If you need EU hosting, I'd recommend choosing the EU region from the start. | Customer data is hosted in the US (us-east-1) or the EU (Frankfurt, eu-central-1). · The region is chosen at account creation and can't be changed later. · EU residency is available on Business and Enterprise plans.

  • 3/3

    Do you support SCIM provisioning?

    Yes, SCIM provisioning is available, but only on our Enterprise plan. Business and Team plans include SSO with SAML instead. | SCIM provisioning is on Enterprise only. · Business and Enterprise include SSO with SAML 2.0 (Okta, Microsoft Entra ID, Google Workspace tested).

  • 3/3

    We use Entra ID. Does SSO work with it?

    Yes, Northwind Guard supports single sign-on over SAML 2.0, and Microsoft Entra ID is one of the identity providers we've tested it with. It's available on the Business and Enterprise plans. | SSO uses SAML 2.0 on Business and Enterprise plans. · Tested with Okta, Microsoft Entra ID and Google Workspace. · SCIM user provisioning is on Enterprise only.

  • 3/3

    Can we enforce MFA for all users?

    Yes, multi-factor authentication is available on every plan, and an admin can require it for all users. | MFA is available on every plan, Free through Enterprise. · Admins can require MFA for the whole account.

  • 3/3

    What happens to our data if we cancel?

    Once an account closes, all of your data is deleted within 30 days. Audit logs are the one exception, kept for one year. | Findings and scan history are kept while the account is active. · After closure, all data is deleted within 30 days. · Audit logs are kept for 1 year.

  • 3/3

    How is our data encrypted?

    Data is encrypted in transit with TLS 1.2 or later, and at rest with AES-256. | In transit: TLS 1.2 or later. · At rest: AES-256.

Integrations38/42 right
  • 3/3

    We're on Azure DevOps, Azure Repos. Can you scan our code there?

    Yes, we support code scanning on Azure DevOps Services, that is Azure Repos. Repo Posture isn't available there, so I'd only promise code scanning for that setup. | Azure DevOps Services (Azure Repos): code scanning supported · Azure DevOps Server (on-premise): not supported · Repo Posture is not supported on Azure Repos

  • 3/3

    We host Bitbucket Data Center ourselves. Is that supported?

    Bitbucket Data Center isn't supported by Northwind Guard, neither for code scanning nor for Repo Posture. Bitbucket Cloud is supported for code scanning, but Repo Posture isn't available there either. | Bitbucket Data Center: code scanning not supported, Repo Posture not supported. · Bitbucket Cloud: code scanning supported, Repo Posture not supported. · Other self-hosted options: GitHub Enterprise Server 3.10+ and GitLab Self-Managed 16.0+ are supported for both.

  • 3/3

    We run GitHub Enterprise Server 3.8. Will that work?

    No, versions before 3.10 aren't supported, so 3.8 won't work with Northwind Guard. Upgrading to 3.10 or later would get you code scanning and Repo Posture. | GitHub Enterprise Server 3.10 and later: code scanning and Repo Posture supported. · GitHub Enterprise Server versions before 3.10 are not supported. | CHECK: The knowledge base doesn't say whether a 3.8 instance could be supported through another route, so don't offer one.

  • 3/3

    Do you support GitLab self-managed? We're on 16.4.

    Yes, GitLab Self-Managed is supported from version 16.0 onward, so 16.4 is covered. Code scanning and Repo Posture both work with it. | GitLab Self-Managed 16.0 and later: code scanning supported · GitLab Self-Managed 16.0 and later: Repo Posture supported · Repo Posture requires the Business or Enterprise plan

  • 3/3

    Our images live in Azure Container Registry. Can you scan them there?

    Yes, Azure Container Registry is one of our supported registries, so we can scan your images where they live. | Azure Container Registry (ACR) is supported for container scanning. · Container scanning covers vulnerabilities in the image and its language packages. · Container scanning is part of the Business and Enterprise plans.

  • 3/3

    What about Red Hat Quay? That's where most of our images are.

    Red Hat Quay isn't supported today, so we can't scan the images you keep there yet. I'll confirm the options for your setup and come back to you. | Northwind supports Docker Hub, Amazon ECR, Google Artifact Registry, Azure Container Registry and GitHub Container Registry. · Red Hat Quay is not on the supported list.

  • 3/3

    We use TeamCity for CI. Is there a plugin?

    TeamCity doesn't have a native Northwind plugin. You can still run scans there by using our CLI, which works in any CI system that can run a Linux container. | TeamCity and Bamboo have no native plugin. · Use the CLI (`nw scan`) in any CI that runs a Linux container. · Supported native CI plugins: GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines.

  • 2/3

    Can findings create tickets in ServiceNow?

    Not in your knowledge

  • 3/3

    We're on Jira Data Center, not cloud. Does the ticketing integration work?

    Yes, Northwind supports Jira Data Center as well as Jira Cloud, so tickets get created and closed automatically when findings are fixed. | Supported: Jira Cloud, Jira Data Center and Linear. · Northwind creates a ticket per finding or per group of findings. · Each ticket is closed when its finding is fixed.

  • 3/3

    How do we get findings into Splunk?

    Findings stream to Splunk through the HTTP Event Collector, so your team can see Northwind results alongside your other security data. Every finding is also available through the REST API. | Splunk is reached via HTTP Event Collector · Datadog is also supported as a streaming target · Every finding is also available through the REST API and a nightly export to Amazon S3

  • 3/3

    Our SOC uses QRadar. Can you send findings there?

    QRadar isn't supported today. We do stream findings to Splunk and Datadog, and every finding is available through the REST API and a nightly S3 export, so your SOC can ingest from there. | Splunk via HTTP Event Collector and Datadog are supported SIEM targets. · REST API gives access to every finding. · Nightly export to Amazon S3 is available.

  • 3/3

    Do you have a plugin for Visual Studio? Not VS Code, the full Visual Studio.

    No, we don't have a plugin for full Visual Studio. We have plugins for VS Code and the JetBrains IDEs, so I'd suggest those if your team uses them. | Full Visual Studio is not supported. · Plugins exist for VS Code and JetBrains IDEs (IntelliJ IDEA, PyCharm, GoLand, WebStorm, Rider).

  • 0/3

    We're mostly GitLab CI with a bit of Jenkins.

    (no card)

  • 3/3

    Does Repo Posture support GitHub Enterprise Server 3.9?

    Repo Posture supports GitHub Enterprise Server 3.10 and later. GitHub Enterprise Server versions before 3.10 aren't supported, so 3.9 isn't covered. | Repo Posture is supported on GitHub Enterprise Server 3.10 and later. · GitHub Enterprise Server versions before 3.10 are not supported. · Repo Posture is available on Business and Enterprise plans.

Languages and IaC20/24 right
  • 0/3

    Half our codebase is C++. Do you scan that?

    Not in your knowledge

  • 3/3

    What about Swift for our iOS app?

    Swift code scanning is in beta, so it's available but not yet fully supported. I'd suggest we confirm the exact coverage for your iOS app before you rely on it. | Swift code scanning (SAST) is currently in beta. · Swift is not yet listed among the fully supported languages (Java, Kotlin, JavaScript, TypeScript, Python, Go, C#, Ruby, PHP). · Swift in beta was announced in the October 2026 release notes. | CHECK: The knowledge base does not confirm what the beta covers for iOS projects, or whether it is included on the Team, Business and Enterprise plans.

  • 3/3

    Does the reachability analysis work for Go?

    Reachability is available for Java, JavaScript, TypeScript and Python only, so for Go you'd get the vulnerability without a reachability verdict. | Reachability covers Java, JavaScript, TypeScript and Python. · For other languages, Northwind shows the vulnerability without a reachability verdict. · Go dependencies are still scanned via Go modules.

  • 3/3

    Which languages have reachability?

    Reachability for dependency vulnerabilities is available for Java, JavaScript, TypeScript and Python. For other languages we show the vulnerability without a reachability verdict. | Covers Java, JavaScript, TypeScript and Python only. · Other languages get the vulnerability, without a reachability verdict. · Python reachability was added in September 2026.

  • 3/3

    We write our infrastructure in Bicep. Is that covered?

    Azure Bicep is not supported today. Our infrastructure-as-code scanning covers Terraform, AWS CloudFormation, Kubernetes manifests and Helm charts, so I'd rather be precise with you on that. | Azure Bicep is not supported for infrastructure-as-code scanning. · Supported: Terraform (HCL and plan files), AWS CloudFormation, Kubernetes manifests and Helm charts. · ARM templates and Pulumi are also not supported.

  • 3/3

    Do you scan Terraform plan files or only the HCL?

    We scan both. Terraform HCL and Terraform plan files are supported for infrastructure-as-code scanning. | Supported: Terraform HCL and plan files · Same IaC scanning also covers AWS CloudFormation, Kubernetes manifests and Helm charts · Azure Bicep, ARM templates and Pulumi are not supported

  • 2/3

    Some of our services run as Windows containers. Can you scan those images?

    Windows container images aren't supported at the moment. Northwind scans images built on Linux distributions like Debian, Ubuntu, Alpine, Red Hat Enterprise Linux and Amazon Linux, including the language packages inside them. | Windows container images are not supported. · Supported base images: Debian, Ubuntu, Alpine, RHEL and Amazon Linux. · Your Linux containers can be scanned in the usual way. | CHECK: The knowledge base doesn't cover Windows-specific scanning of other kinds, such as application code or dependencies inside Windows images, so I wouldn't promise those.

  • 3/3

    We moved to pnpm. Is it supported for dependency scanning?

    Yes, pnpm is supported for dependency scanning, so your move to it won't affect scans. | Dependency scanning (SCA) covers pnpm alongside Maven, Gradle, npm and Yarn. · Reachability verdicts are available only for Java, JavaScript, TypeScript and Python.

Deployment12/12 right
  • 3/3

    We need a fully air-gapped install. Is that possible?

    Fully air-gapped installations aren't supported. The console is always cloud-hosted, but the scanner can run inside your own network on Enterprise, with only outbound HTTPS to the console. | Enterprise customers can run the self-hosted scanner on Kubernetes or Docker. · The scanner needs outbound HTTPS access to the Northwind console. · Fully air-gapped installations are not supported; the console itself is always SaaS.

  • 3/3

    Can we run the scanner inside our own network?

    Yes, on the Enterprise plan you can run the scanner in your own network, so your code never leaves it; only findings go to the console. | Enterprise customers run the scanner on their own Kubernetes or Docker. · The scanner needs outbound HTTPS access to the Northwind console. · Fully air-gapped installs are not supported; the console itself is always SaaS. | CHECK: Fully air-gapped installations are not supported, so the scanner must reach the console over outbound HTTPS.

  • 3/3

    Can we host the whole platform, including the console, on-prem?

    The console itself is always a cloud service, so it can't be hosted on-prem. What we can run in your own network is the scanner, on Enterprise plans. Scans run there and only findings go to the console. | The Northwind console is SaaS only. · Enterprise customers can run the scanner as a container on Kubernetes or Docker. · The scanner needs outbound HTTPS to the console; fully air-gapped installs are not supported.

  • 3/3

    What does the self-hosted scanner need from our firewall?

    The self-hosted scanner only needs outbound HTTPS access from your network to the Northwind console. There's no inbound access needed, and fully air-gapped installations aren't supported. | Outbound HTTPS to the Northwind console is required. · Runs as a container on Kubernetes or Docker. · Only available on the Enterprise plan.

Pricing and plans15/15 right
  • 3/3

    How much is the Team plan?

    The Team plan is $35 per developer per month, with a minimum of 10 developers. Paying yearly saves 15%. | $35 per contributing developer per month, minimum 10 developers · A contributing developer is anyone who committed to a scanned repository in the last 90 days · Every paid plan starts with a 21-day trial, no card required

  • 3/3

    How do you count developers for pricing?

    We count contributing developers: anyone who committed to a scanned repository in the last 90 days. Pricing is per contributing developer per month. | A contributing developer is anyone who committed to a scanned repository in the last 90 days. · Team starts at $35 per developer per month, with a 10-developer minimum. · Yearly billing saves 15%.

  • 3/3

    Is there a trial, and do you need a card?

    Yes, every paid plan starts with a 21-day trial, and you don't need a card to start it. | Every paid plan includes a 21-day trial. · No credit card required.

  • 3/3

    Is there a discount if we pay annually?

    Yes, paying yearly saves 15% compared with paying monthly. I can walk you through what that would come to for your team size. | Paying yearly saves 15% on every paid plan. · Applies to Team, Business and Enterprise (Enterprise pricing is custom). · Every paid plan starts with a 21-day trial, no card required.

  • 3/3

    Is Repo Posture included in the Team plan?

    Repo Posture isn't part of the Team plan. It's included on Business and Enterprise, so I'd suggest we look at which plan fits your team best. | Team plan: code, dependencies, secrets and IaC scanning. · Repo Posture is included on Business and Enterprise. · Business starts at 25 developers; Team at 10.

Features12/12 right
  • 3/3

    We get the same finding a hundred times across branches. How do you deal with duplicates?

    Northwind groups those repeats into one finding, so your security team triages each issue once, even when it shows up in many branches. A finding counts as the same when the rule, file and code fingerprint match, even if line numbers have moved. | Same rule, same file and same code fingerprint count as one finding · Line number changes don't create a new finding · Security teams triage each issue once, not once per branch

  • 3/3

    Can we add our own patterns, like a Norwegian national ID number?

    Yes, on Business and Enterprise admins can add their own secret patterns as regular expressions, with a name, a severity and an optional validation step. Norwegian national ID numbers are one of the common uses. | Available on Business and Enterprise plans · Patterns are regular expressions, with a name and severity · Optional validation step per pattern

  • 3/3

    Can you export an SBOM? Which formats?

    Yes, you can export a software bill of materials for any project, from the console or the API. It comes in CycloneDX 1.5 or SPDX 2.3. | Available from the console or the API, for any project · Formats: CycloneDX 1.5 and SPDX 2.3

  • 3/3

    Can you fail a pull request when there's a new critical finding?

    Yes, you can. Admins set a policy that fails the pull request check when a new critical finding appears, and it can apply to all projects or to selected ones. | Policies can block any new critical finding and fail the PR check or CI build. · Policies can apply to all projects or to chosen projects.

Scoped to the feature shown12/12 right
  • 3/3

    Does this work with GitLab too? · Now showing: Auto-fix pull requests

    Auto-fix pull requests are available on GitHub and GitHub Enterprise Server only, so GitLab isn't covered for that feature. GitLab is supported for code scanning and Repo Posture, though. | Auto-fix PRs: GitHub and GitHub Enterprise Server only · GitLab (gitlab.com) and GitLab Self-Managed 16.0+: supported for code scanning and Repo Posture | CHECK: The knowledge base doesn't say whether auto-fix will come to GitLab later, so don't promise a date.

  • 3/3

    And does this part work on Azure DevOps? · Now showing: Repo Posture

    Repo Posture doesn't support Azure DevOps today. Code scanning does work with Azure DevOps Services, so I can show you that instead, and I'll confirm the on-premise Azure DevOps Server situation with you. | Azure DevOps Services (Azure Repos): code scanning supported, Repo Posture not supported. · Azure DevOps Server (on-premise): neither code scanning nor Repo Posture is supported. · Repo Posture works with GitHub, GitHub Enterprise Server, GitLab and GitLab Self-Managed.

  • 3/3

    Would this work for our Go services? · Now showing: Reachability

    Yes, Go is supported for code scanning, and Go modules are covered for dependency scanning. Reachability is the one thing to be careful about: it's available for Java, JavaScript, TypeScript and Python only, so for Go you'd see the vulnerabilities without a reachability verdict. | Code scanning (SAST) supports Go. · Dependency scanning (SCA) supports Go modules. · Reachability verdicts are not available for Go. | CHECK: The knowledge base does not confirm whether Go is covered by Repo Posture, IaC or container scanning for Go-built images; it only lists Go modules under dependency scanning.

  • 3/3

    Does this work on GitLab self-managed? · Now showing: Repo Posture

    Yes, Repo Posture works with GitLab Self-Managed, as long as the version is 16.0 or later. Code scanning is supported there too. | GitLab Self-Managed 16.0 and later is supported for Repo Posture. · Repo Posture is available on Business and Enterprise plans.

Not in the docs15/15 right
  • 3/3

    Are you FedRAMP authorized?

    Not in your knowledge

  • 3/3

    Will you sign a HIPAA BAA?

    Not in your knowledge

  • 3/3

    Do you also do dynamic testing, DAST, against running apps?

    Not in your knowledge

  • 3/3

    Do you support Perforce Helix Core?

    Not in your knowledge

  • 3/3

    Do you have a reseller or partner programme?

    Not in your knowledge

Nothing to answer12/12 right
  • 3/3

    Sorry, you froze for a second there. Can you hear me now?

    (no card)

  • 3/3

    So which repositories would you want to start with?

    (no card)

  • 3/3

    Let me share my screen and pull up our pipeline.

    (no card)

  • 3/3

    Great, that's really helpful, thanks.

    (no card)

Measure it on your own calls.

In a pilot, your reps rate every answer, and the console shows your accuracy live against an 85% target.