# 171 answers. Not one made up.

> On Apartee's public test (57 buyer questions about a fictional security product, run 3 times on 2026-11-13), 163 of 171 answers were right (95%) and 0 were made up. The median time from the question to the first words of the answer was 1.0 s, for the answer step only.

We put the copilot through 57 buyer questions about a security product, three times over, and publish every answer.

- **Right:** 95% (163 of 171 answers did what a careful sales engineer would).
- **Wrong:** 0. No answer told the rep something the docs don't say.
- **Held back:** 8. The copilot said “let me confirm” or stayed quiet when it could have answered. Slower for the rep, never wrong for the buyer.
- **To the first point:** 1.0 s at the median, from the question to the answer on the card.

## By kind of question

| Kind | Right | Of |
|---|---|---|
| Security and compliance | 27 | 27 |
| Deployment | 12 | 12 |
| Pricing and plans | 15 | 15 |
| Features | 12 | 12 |
| Scoped to the feature shown | 12 | 12 |
| Not in the docs | 15 | 15 |
| Nothing to answer | 12 | 12 |
| Integrations | 38 | 42 |
| Languages and IaC | 20 | 24 |

## Getting better, in the open

- First run (copilot-v8): 142 of 171. "Not in your knowledge" where the docs said a product isn't supported: 26 of 29 misses.
- Now (copilot-v9): 163 of 171. New rule: a documented "not supported" is an answer; "Not in your knowledge" is only for what the docs don't mention.

## Where it still holds back

- 3 times: “We're mostly GitLab CI with a bit of Jenkins” got no card. A sales engineer hears “do you support that?”.
- 5 times: “Not in your knowledge” for something the docs say isn't supported (mostly C++). The rep offers to confirm instead of saying no.

## How we measure

- **The docs:** seven pages about Northwind Guard, a fictional security product, saying what's supported and what isn't. https://apartee.app/accuracy/northwind-guard-docs.md
- **The questions:** 57 moments from a demo, each labelled before the run with what should happen. https://apartee.app/accuracy/cases.json
- **The copilot:** the same AI model and instructions as on customers' calls (version copilot-v9).
- **Scoring:** only the card's own text counts. It must contain the words a correct answer needs, and none that would make it wrong.
- **Runs:** three, pooled. https://apartee.app/accuracy/results-run1.json, https://apartee.app/accuracy/results-run2.json, https://apartee.app/accuracy/results-run3.json
- **Limits:** short docs, so search across a large knowledge base isn't tested, and timing excludes speech to text. The real test is a pilot on your own calls.

## Every question, and the card from the first run

Everything in this list is about Northwind Guard, the fictional product used for the test. Its prices, plans and features are invented for the test and are not Apartee's.

- **We're on Azure DevOps, Azure Repos. Can you scan our code there?** [Integrations, right in 3 of 3 runs] Yes, we support code scanning on Azure DevOps Services, that is Azure Repos. Repo Posture isn't available there, so I'd only promise code scanning for that setup. | Azure DevOps Services (Azure Repos): code scanning supported · Azure DevOps Server (on-premise): not supported · Repo Posture is not supported on Azure Repos
- **We host Bitbucket Data Center ourselves. Is that supported?** [Integrations, right in 3 of 3 runs] Bitbucket Data Center isn't supported by Northwind Guard, neither for code scanning nor for Repo Posture. Bitbucket Cloud is supported for code scanning, but Repo Posture isn't available there either. | Bitbucket Data Center: code scanning not supported, Repo Posture not supported. · Bitbucket Cloud: code scanning supported, Repo Posture not supported. · Other self-hosted options: GitHub Enterprise Server 3.10+ and GitLab Self-Managed 16.0+ are supported for both.
- **We run GitHub Enterprise Server 3.8. Will that work?** [Integrations, right in 3 of 3 runs] No, versions before 3.10 aren't supported, so 3.8 won't work with Northwind Guard. Upgrading to 3.10 or later would get you code scanning and Repo Posture. | GitHub Enterprise Server 3.10 and later: code scanning and Repo Posture supported. · GitHub Enterprise Server versions before 3.10 are not supported. | CHECK: The knowledge base doesn't say whether a 3.8 instance could be supported through another route, so don't offer one.
- **Do you support GitLab self-managed? We're on 16.4.** [Integrations, right in 3 of 3 runs] Yes, GitLab Self-Managed is supported from version 16.0 onward, so 16.4 is covered. Code scanning and Repo Posture both work with it. | GitLab Self-Managed 16.0 and later: code scanning supported · GitLab Self-Managed 16.0 and later: Repo Posture supported · Repo Posture requires the Business or Enterprise plan
- **Our images live in Azure Container Registry. Can you scan them there?** [Integrations, right in 3 of 3 runs] Yes, Azure Container Registry is one of our supported registries, so we can scan your images where they live. | Azure Container Registry (ACR) is supported for container scanning. · Container scanning covers vulnerabilities in the image and its language packages. · Container scanning is part of the Business and Enterprise plans.
- **What about Red Hat Quay? That's where most of our images are.** [Integrations, right in 3 of 3 runs] Red Hat Quay isn't supported today, so we can't scan the images you keep there yet. I'll confirm the options for your setup and come back to you. | Northwind supports Docker Hub, Amazon ECR, Google Artifact Registry, Azure Container Registry and GitHub Container Registry. · Red Hat Quay is not on the supported list.
- **We use TeamCity for CI. Is there a plugin?** [Integrations, right in 3 of 3 runs] TeamCity doesn't have a native Northwind plugin. You can still run scans there by using our CLI, which works in any CI system that can run a Linux container. | TeamCity and Bamboo have no native plugin. · Use the CLI (`nw scan`) in any CI that runs a Linux container. · Supported native CI plugins: GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines.
- **Can findings create tickets in ServiceNow?** [Integrations, right in 2 of 3 runs] Not in your knowledge
- **We're on Jira Data Center, not cloud. Does the ticketing integration work?** [Integrations, right in 3 of 3 runs] Yes, Northwind supports Jira Data Center as well as Jira Cloud, so tickets get created and closed automatically when findings are fixed. | Supported: Jira Cloud, Jira Data Center and Linear. · Northwind creates a ticket per finding or per group of findings. · Each ticket is closed when its finding is fixed.
- **How do we get findings into Splunk?** [Integrations, right in 3 of 3 runs] Findings stream to Splunk through the HTTP Event Collector, so your team can see Northwind results alongside your other security data. Every finding is also available through the REST API. | Splunk is reached via HTTP Event Collector · Datadog is also supported as a streaming target · Every finding is also available through the REST API and a nightly export to Amazon S3
- **Our SOC uses QRadar. Can you send findings there?** [Integrations, right in 3 of 3 runs] QRadar isn't supported today. We do stream findings to Splunk and Datadog, and every finding is available through the REST API and a nightly S3 export, so your SOC can ingest from there. | Splunk via HTTP Event Collector and Datadog are supported SIEM targets. · REST API gives access to every finding. · Nightly export to Amazon S3 is available.
- **Do you have a plugin for Visual Studio? Not VS Code, the full Visual Studio.** [Integrations, right in 3 of 3 runs] No, we don't have a plugin for full Visual Studio. We have plugins for VS Code and the JetBrains IDEs, so I'd suggest those if your team uses them. | Full Visual Studio is not supported. · Plugins exist for VS Code and JetBrains IDEs (IntelliJ IDEA, PyCharm, GoLand, WebStorm, Rider).
- **We're mostly GitLab CI with a bit of Jenkins.** [Integrations, right in 0 of 3 runs] (no card)
- **Half our codebase is C++. Do you scan that?** [Languages and IaC, right in 0 of 3 runs] Not in your knowledge
- **What about Swift for our iOS app?** [Languages and IaC, right in 3 of 3 runs] Swift code scanning is in beta, so it's available but not yet fully supported. I'd suggest we confirm the exact coverage for your iOS app before you rely on it. | Swift code scanning (SAST) is currently in beta. · Swift is not yet listed among the fully supported languages (Java, Kotlin, JavaScript, TypeScript, Python, Go, C#, Ruby, PHP). · Swift in beta was announced in the October 2026 release notes. | CHECK: The knowledge base does not confirm what the beta covers for iOS projects, or whether it is included on the Team, Business and Enterprise plans.
- **Does the reachability analysis work for Go?** [Languages and IaC, right in 3 of 3 runs] Reachability is available for Java, JavaScript, TypeScript and Python only, so for Go you'd get the vulnerability without a reachability verdict. | Reachability covers Java, JavaScript, TypeScript and Python. · For other languages, Northwind shows the vulnerability without a reachability verdict. · Go dependencies are still scanned via Go modules.
- **Which languages have reachability?** [Languages and IaC, right in 3 of 3 runs] Reachability for dependency vulnerabilities is available for Java, JavaScript, TypeScript and Python. For other languages we show the vulnerability without a reachability verdict. | Covers Java, JavaScript, TypeScript and Python only. · Other languages get the vulnerability, without a reachability verdict. · Python reachability was added in September 2026.
- **We write our infrastructure in Bicep. Is that covered?** [Languages and IaC, right in 3 of 3 runs] Azure Bicep is not supported today. Our infrastructure-as-code scanning covers Terraform, AWS CloudFormation, Kubernetes manifests and Helm charts, so I'd rather be precise with you on that. | Azure Bicep is not supported for infrastructure-as-code scanning. · Supported: Terraform (HCL and plan files), AWS CloudFormation, Kubernetes manifests and Helm charts. · ARM templates and Pulumi are also not supported.
- **Do you scan Terraform plan files or only the HCL?** [Languages and IaC, right in 3 of 3 runs] We scan both. Terraform HCL and Terraform plan files are supported for infrastructure-as-code scanning. | Supported: Terraform HCL and plan files · Same IaC scanning also covers AWS CloudFormation, Kubernetes manifests and Helm charts · Azure Bicep, ARM templates and Pulumi are not supported
- **Some of our services run as Windows containers. Can you scan those images?** [Languages and IaC, right in 2 of 3 runs] Windows container images aren't supported at the moment. Northwind scans images built on Linux distributions like Debian, Ubuntu, Alpine, Red Hat Enterprise Linux and Amazon Linux, including the language packages inside them. | Windows container images are not supported. · Supported base images: Debian, Ubuntu, Alpine, RHEL and Amazon Linux. · Your Linux containers can be scanned in the usual way. | CHECK: The knowledge base doesn't cover Windows-specific scanning of other kinds, such as application code or dependencies inside Windows images, so I wouldn't promise those.
- **We moved to pnpm. Is it supported for dependency scanning?** [Languages and IaC, right in 3 of 3 runs] Yes, pnpm is supported for dependency scanning, so your move to it won't affect scans. | Dependency scanning (SCA) covers pnpm alongside Maven, Gradle, npm and Yarn. · Reachability verdicts are available only for Java, JavaScript, TypeScript and Python.
- **Do you have a SOC 2 Type II report we can review?** [Security and compliance, right in 3 of 3 runs] Yes, we have a SOC 2 Type II report, audited every year, and we can share it with you under NDA. | SOC 2 Type II, audited every year · The report is shared under NDA · Northwind is also ISO/IEC 27001 certified
- **Do you keep a copy of our source code after the scan?** [Security and compliance, right in 3 of 3 runs] No, we don't keep a copy. Cloud scans clone your repository into an isolated, single-use worker, and the code is deleted when the scan ends, at the latest one hour after it started. | Northwind stores only findings: file path, line, and a snippet of up to 5 lines. · Full source code is never stored as a copy. · With the self-hosted scanner (Enterprise), your code never leaves your network.
- **Can our data stay in the EU?** [Security and compliance, right in 3 of 3 runs] Yes. EU data residency is available on Business and Enterprise. You choose the region when the account is created, and it can't be changed afterwards. | Hosting is in the EU (Frankfurt, eu-central-1), or in the US (us-east-1). · EU residency is available on Business and Enterprise plans. · The region is fixed at account creation.
- **If we start in the US region, can we move to the EU later?** [Security and compliance, right in 3 of 3 runs] No, the region is fixed when the account is created and can't be changed afterwards. If you need EU hosting, I'd recommend choosing the EU region from the start. | Customer data is hosted in the US (us-east-1) or the EU (Frankfurt, eu-central-1). · The region is chosen at account creation and can't be changed later. · EU residency is available on Business and Enterprise plans.
- **Do you support SCIM provisioning?** [Security and compliance, right in 3 of 3 runs] Yes, SCIM provisioning is available, but only on our Enterprise plan. Business and Team plans include SSO with SAML instead. | SCIM provisioning is on Enterprise only. · Business and Enterprise include SSO with SAML 2.0 (Okta, Microsoft Entra ID, Google Workspace tested).
- **We use Entra ID. Does SSO work with it?** [Security and compliance, right in 3 of 3 runs] Yes, Northwind Guard supports single sign-on over SAML 2.0, and Microsoft Entra ID is one of the identity providers we've tested it with. It's available on the Business and Enterprise plans. | SSO uses SAML 2.0 on Business and Enterprise plans. · Tested with Okta, Microsoft Entra ID and Google Workspace. · SCIM user provisioning is on Enterprise only.
- **Can we enforce MFA for all users?** [Security and compliance, right in 3 of 3 runs] Yes, multi-factor authentication is available on every plan, and an admin can require it for all users. | MFA is available on every plan, Free through Enterprise. · Admins can require MFA for the whole account.
- **What happens to our data if we cancel?** [Security and compliance, right in 3 of 3 runs] Once an account closes, all of your data is deleted within 30 days. Audit logs are the one exception, kept for one year. | Findings and scan history are kept while the account is active. · After closure, all data is deleted within 30 days. · Audit logs are kept for 1 year.
- **How is our data encrypted?** [Security and compliance, right in 3 of 3 runs] Data is encrypted in transit with TLS 1.2 or later, and at rest with AES-256. | In transit: TLS 1.2 or later. · At rest: AES-256.
- **We need a fully air-gapped install. Is that possible?** [Deployment, right in 3 of 3 runs] Fully air-gapped installations aren't supported. The console is always cloud-hosted, but the scanner can run inside your own network on Enterprise, with only outbound HTTPS to the console. | Enterprise customers can run the self-hosted scanner on Kubernetes or Docker. · The scanner needs outbound HTTPS access to the Northwind console. · Fully air-gapped installations are not supported; the console itself is always SaaS.
- **Can we run the scanner inside our own network?** [Deployment, right in 3 of 3 runs] Yes, on the Enterprise plan you can run the scanner in your own network, so your code never leaves it; only findings go to the console. | Enterprise customers run the scanner on their own Kubernetes or Docker. · The scanner needs outbound HTTPS access to the Northwind console. · Fully air-gapped installs are not supported; the console itself is always SaaS. | CHECK: Fully air-gapped installations are not supported, so the scanner must reach the console over outbound HTTPS.
- **Can we host the whole platform, including the console, on-prem?** [Deployment, right in 3 of 3 runs] The console itself is always a cloud service, so it can't be hosted on-prem. What we can run in your own network is the scanner, on Enterprise plans. Scans run there and only findings go to the console. | The Northwind console is SaaS only. · Enterprise customers can run the scanner as a container on Kubernetes or Docker. · The scanner needs outbound HTTPS to the console; fully air-gapped installs are not supported.
- **What does the self-hosted scanner need from our firewall?** [Deployment, right in 3 of 3 runs] The self-hosted scanner only needs outbound HTTPS access from your network to the Northwind console. There's no inbound access needed, and fully air-gapped installations aren't supported. | Outbound HTTPS to the Northwind console is required. · Runs as a container on Kubernetes or Docker. · Only available on the Enterprise plan.
- **How much is the Team plan?** [Pricing and plans, right in 3 of 3 runs] The Team plan is $35 per developer per month, with a minimum of 10 developers. Paying yearly saves 15%. | $35 per contributing developer per month, minimum 10 developers · A contributing developer is anyone who committed to a scanned repository in the last 90 days · Every paid plan starts with a 21-day trial, no card required
- **How do you count developers for pricing?** [Pricing and plans, right in 3 of 3 runs] We count contributing developers: anyone who committed to a scanned repository in the last 90 days. Pricing is per contributing developer per month. | A contributing developer is anyone who committed to a scanned repository in the last 90 days. · Team starts at $35 per developer per month, with a 10-developer minimum. · Yearly billing saves 15%.
- **Is there a trial, and do you need a card?** [Pricing and plans, right in 3 of 3 runs] Yes, every paid plan starts with a 21-day trial, and you don't need a card to start it. | Every paid plan includes a 21-day trial. · No credit card required.
- **Is there a discount if we pay annually?** [Pricing and plans, right in 3 of 3 runs] Yes, paying yearly saves 15% compared with paying monthly. I can walk you through what that would come to for your team size. | Paying yearly saves 15% on every paid plan. · Applies to Team, Business and Enterprise (Enterprise pricing is custom). · Every paid plan starts with a 21-day trial, no card required.
- **Is Repo Posture included in the Team plan?** [Pricing and plans, right in 3 of 3 runs] Repo Posture isn't part of the Team plan. It's included on Business and Enterprise, so I'd suggest we look at which plan fits your team best. | Team plan: code, dependencies, secrets and IaC scanning. · Repo Posture is included on Business and Enterprise. · Business starts at 25 developers; Team at 10.
- **We get the same finding a hundred times across branches. How do you deal with duplicates?** [Features, right in 3 of 3 runs] Northwind groups those repeats into one finding, so your security team triages each issue once, even when it shows up in many branches. A finding counts as the same when the rule, file and code fingerprint match, even if line numbers have moved. | Same rule, same file and same code fingerprint count as one finding · Line number changes don't create a new finding · Security teams triage each issue once, not once per branch
- **Can we add our own patterns, like a Norwegian national ID number?** [Features, right in 3 of 3 runs] Yes, on Business and Enterprise admins can add their own secret patterns as regular expressions, with a name, a severity and an optional validation step. Norwegian national ID numbers are one of the common uses. | Available on Business and Enterprise plans · Patterns are regular expressions, with a name and severity · Optional validation step per pattern
- **Can you export an SBOM? Which formats?** [Features, right in 3 of 3 runs] Yes, you can export a software bill of materials for any project, from the console or the API. It comes in CycloneDX 1.5 or SPDX 2.3. | Available from the console or the API, for any project · Formats: CycloneDX 1.5 and SPDX 2.3
- **Can you fail a pull request when there's a new critical finding?** [Features, right in 3 of 3 runs] Yes, you can. Admins set a policy that fails the pull request check when a new critical finding appears, and it can apply to all projects or to selected ones. | Policies can block any new critical finding and fail the PR check or CI build. · Policies can apply to all projects or to chosen projects.
- **Does this work with GitLab too?** (now showing: Auto-fix pull requests) [Scoped to the feature shown, right in 3 of 3 runs] Auto-fix pull requests are available on GitHub and GitHub Enterprise Server only, so GitLab isn't covered for that feature. GitLab is supported for code scanning and Repo Posture, though. | Auto-fix PRs: GitHub and GitHub Enterprise Server only · GitLab (gitlab.com) and GitLab Self-Managed 16.0+: supported for code scanning and Repo Posture | CHECK: The knowledge base doesn't say whether auto-fix will come to GitLab later, so don't promise a date.
- **And does this part work on Azure DevOps?** (now showing: Repo Posture) [Scoped to the feature shown, right in 3 of 3 runs] Repo Posture doesn't support Azure DevOps today. Code scanning does work with Azure DevOps Services, so I can show you that instead, and I'll confirm the on-premise Azure DevOps Server situation with you. | Azure DevOps Services (Azure Repos): code scanning supported, Repo Posture not supported. · Azure DevOps Server (on-premise): neither code scanning nor Repo Posture is supported. · Repo Posture works with GitHub, GitHub Enterprise Server, GitLab and GitLab Self-Managed.
- **Would this work for our Go services?** (now showing: Reachability) [Scoped to the feature shown, right in 3 of 3 runs] Yes, Go is supported for code scanning, and Go modules are covered for dependency scanning. Reachability is the one thing to be careful about: it's available for Java, JavaScript, TypeScript and Python only, so for Go you'd see the vulnerabilities without a reachability verdict. | Code scanning (SAST) supports Go. · Dependency scanning (SCA) supports Go modules. · Reachability verdicts are not available for Go. | CHECK: The knowledge base does not confirm whether Go is covered by Repo Posture, IaC or container scanning for Go-built images; it only lists Go modules under dependency scanning.
- **Does this work on GitLab self-managed?** (now showing: Repo Posture) [Scoped to the feature shown, right in 3 of 3 runs] Yes, Repo Posture works with GitLab Self-Managed, as long as the version is 16.0 or later. Code scanning is supported there too. | GitLab Self-Managed 16.0 and later is supported for Repo Posture. · Repo Posture is available on Business and Enterprise plans.
- **Are you FedRAMP authorized?** [Not in the docs, right in 3 of 3 runs] Not in your knowledge
- **Will you sign a HIPAA BAA?** [Not in the docs, right in 3 of 3 runs] Not in your knowledge
- **Do you also do dynamic testing, DAST, against running apps?** [Not in the docs, right in 3 of 3 runs] Not in your knowledge
- **Do you support Perforce Helix Core?** [Not in the docs, right in 3 of 3 runs] Not in your knowledge
- **Do you have a reseller or partner programme?** [Not in the docs, right in 3 of 3 runs] Not in your knowledge
- **Does Repo Posture support GitHub Enterprise Server 3.9?** [Integrations, right in 3 of 3 runs] Repo Posture supports GitHub Enterprise Server 3.10 and later. GitHub Enterprise Server versions before 3.10 aren't supported, so 3.9 isn't covered. | Repo Posture is supported on GitHub Enterprise Server 3.10 and later. · GitHub Enterprise Server versions before 3.10 are not supported. · Repo Posture is available on Business and Enterprise plans.
- **Sorry, you froze for a second there. Can you hear me now?** [Nothing to answer, right in 3 of 3 runs] (no card)
- **So which repositories would you want to start with?** [Nothing to answer, right in 3 of 3 runs] (no card)
- **Let me share my screen and pull up our pipeline.** [Nothing to answer, right in 3 of 3 runs] (no card)
- **Great, that's really helpful, thanks.** [Nothing to answer, right in 3 of 3 runs] (no card)

---
This is the text version of https://apartee.app/accuracy. Request a pilot: https://pro.apartee.app/request-demo. Contact: hello@apartee.app. Updated 2026-11-16.
