Skip to main content

Your Penetration Test Is Only as Good as the Question

Why a list of IP addresses is not a test plan—and the five decisions that turn a vague scope into useful assurance.

|5 min read

“Can you test our application?”

Absolutely. Which application?

To a tester, “the application” might include the public website, its API, the mobile client, an administration portal, single sign-on, a support widget and the cloud storage quietly holding everything together. Give three testers the same hostname and you may get three perfectly reasonable—and completely different—tests.

The fix is not a longer asset list. It is a better question.

A target is not an objective

In 2016, the US Department of Defense invited more than 1,400 eligible researchers to participate in Hack the Pentagon. The programme received its first vulnerability report within 13 minutes and ultimately accepted 138 unique findings.

It was not a conventional penetration test, but its scoping lesson is excellent: “the Pentagon” was not the scope. Five named public websites were. Critical networks were explicitly excluded.

That boundary let a large group of researchers test aggressively without pretending they had permission to touch everything bearing a Department of Defense logo.

For a penetration test, start with the decision you need to make:

  • Can we release this application?
  • Can one customer reach another customer’s data?
  • Can a compromised user become an administrator?
  • Did the remediation actually break the original attack path?

“Test the portal” names a thing. “Determine whether a standard user can approve their own payment” names a security question.

Give the tester the characters, not just the stage

Many damaging application flaws only appear when somebody uses a legitimate feature dishonestly.

Imagine an expense system with three roles: employee, manager and finance. A scanner can find an old library. It will not necessarily ask whether a manager can approve their own claim, whether an employee can replay an approval request, or whether changing one identifier reveals somebody else’s receipt.

The OWASP Web Security Testing Guide makes the same point: business-logic flaws depend on the application’s particular rules and require human creativity.

Useful scope therefore includes:

  • the important user roles
  • what each role should and should not be able to do
  • high-value data and transactions
  • trust boundaries and third-party integrations

Test accounts are not “helping the tester cheat”. They buy time for the interesting work.

Production needs rules, not crossed fingers

Production testing can be appropriate. Surprising production is not.

NIST’s rules-of-engagement guidance describes the boundaries and authority that should exist before testing starts. In practical terms, agree the testing window, prohibited techniques, rate limits, escalation contacts, data-handling rules and stop conditions.

Also settle the awkward questions early:

  • Who can authorise testing of the payment provider or managed platform?
  • What happens if the tester encounters real personal information?
  • Can the security team distinguish test activity from a real incident?
  • Who answers the phone if a service becomes unstable?

If nobody owns the answer, the engagement is not ready.

Decide who the report is for

A report written for engineers looks different from evidence intended for a risk owner or release authority. Decide what the output must help its reader do.

Usually that means:

  • validated findings, not an unfiltered tool export
  • impact explained in the organisation’s context
  • enough evidence to reproduce the issue safely
  • remediation advice an engineering team can use
  • a clear position on what was not tested
  • retesting terms agreed before everybody forgets

The five-minute brief

Before asking for a quote, write down:

  1. The decision the test must support.
  2. The systems, roles and environments involved.
  3. The attack paths or business outcomes that matter most.
  4. The safety, timing and data constraints.
  5. The people who will act on the result.

That is enough to begin a useful scoping conversation. You do not need to arrive with the test plan already written. You do need to know why you are testing.

Learn more about our penetration testing service, or talk to our team about turning the question into a controlled engagement.