Prompt Injection in Copilot Chatbot — Phishing via Client-Controlled Context

I discovered a critical issue in a Copilot-style chatbot where client-controlled page context could be used to manipulate the model into trusting fake security instructions. In practice, an attacker could influence the assistant to validate malicious phishing content as though it were legitimate context coming from the user’s current webpage.

The root cause was simple but dangerous: the chatbot accepted pageText and currentUrl from the client without proper validation. Instead of fetching trusted contextual content server-side, it trusted the content sent by the browser. That turned user-controlled context into a prompt-injection channel.

The core problem

Once the model saw attacker-controlled pageText, it treated it as trusted prompt context. That allowed fake security updates to be interpreted as genuine guidance. A user could be nudged into clicking a phishing link or following a malicious verification flow without the assistant sounding the alarm.

Why it matters

This is not just a model issue; it is a trust-boundaries issue. If user-controlled context enters the prompt without validation, the assistant starts acting as a social-engineering relay. The fix is not to train the model to be more cautious in the abstract. The fix is to treat untrusted browser input as untrusted input and never let it flow into the model without verification.

“AI will trust whatever is in the context window if the system does not clearly separate trusted data from hostile user input.”

The correct mitigation is to fetch page data server-side, validate the URL against an allowlist, compare the content to the actual page, and reject anything that cannot be confirmed. Security systems must treat every pageContext field as untrusted until proven otherwise.