Putting ChatGPT to Work on Technical Writing

Mike Mason, Global Head of Technology at Thoughtworks, recently spent several weeks testing ChatGPT as a writing assistant for his technical and consulting work. His brief: drafting “blips” for the Thoughtworks Technology Radar — short assessments of individual technologies published every few months. To avoid confidentiality issues, he cleared the experiment with his employer and legal team, and restricted use to material intended for public distribution.

Mason had three writing jobs in mind for the AI: drafting Radar blips, generating side articles on broader industry trends, and rewriting Radar themes for non-technical business readers. The results were mixed — useful for first drafts and prompting follow-up material, but deficient in accuracy and depth, and prone to what he calls “bubbly optimism.” His overall conclusion: ChatGPT works best as an iterative assistant where you request small chunks of content with well-crafted prompts, then review carefully.

A Concrete Example: “Casual Management of Webhooks”

Mason’s first real task was drafting a blip about the dangers of casually managing webhooks in chat platforms such as Slack or Microsoft Teams. The substance: with remote work, teams automate messages via webhooks into internal channels, but too often treat them as configuration rather than credentials. Webhooks can contain API keys, grant privileged access, and if committed to repos in plain text, can be extracted for phishing.

Mason asked ChatGPT for a blip “in the style of the Thoughtworks technology radar,” feeding in detailed notes and discussion. The model returned coherent text with the right general tone — and a good opening that frames the trend. One issue: the final paragraph was a call to action (“Let’s not wait for a major incident…”) that Radar blips don’t include. That was easy enough to cut, and the essentials — treating webhooks as configuration, storing them in git, exposure through platform design — all came through.

Notably, with more than a decade of published Radar content available, Mason didn’t need to provide elaborate style samples; asking for the style by name sufficed. The edited result was published after a company feedback round.

A Second Test: “Giskard” and the Optimism Problem

The second example involved a blip for Giskard, a quality-assurance tool for AI models that identifies biases and enables collaboration between data scientists and business stakeholders. Mason supplied brief notes — the team had no production experience yet, only user conversations and marketing material.

ChatGPT’s draft praised Giskard as a “novel” solution that “distinguishes itself” and has “unique value proposition.” Mason immediately noticed the problem: the tone does not suit an “Assess” blip, which should be measured. ChatGPT trained on marketing-heavy content and human preference data will happily produce hype no one in Mason’s team would write. He would normally edit out words like “novel” and “unique” entirely.

A more telling response came from a colleague specialising in machine learning in Finland. She remarked that the draft sounded like a junior person with little understanding of typical ML workflows and revised significant portions before it was publication-ready. Mason’s takeaway: ChatGPT output generally needs skilled expert review, and this test proved that a human expert in the target domain has to check — and often substantially rewrite — AI first drafts.

Beyond individual blips, Mason also pushed ChatGPT toward larger pieces. For the “Macro Trends” overview articles he contributes to the Radar companion publishing pipeline, he tried using the LLM to identify common threads across many blips — a type of cross-cutting higher-level insight generation.

The result was more disappointing. The model’s tendency to “hallucinate” became a blocker. When asked to create a high-level narrative about the key themes behind four years of Technology Radar blips, ChatGPT fabricated connections between blips that never existed. To salvage a useful outcome, Mason found he had to narrow the scope — asking for mini-reflections on specific themes — while weeding out invented material carefully. Full independence from human judgement is not realistic with these models.**

Practical Takeaways
  • Draft oriented: ChatGPT is best for starting a piece — creating baseline text to shape and edit from.
  • Density matters: Overly long sections can be shaved, but identifying and cutting empty “call to action” or hype phrasing remains human work.
  • Iterative approach: Rather than asking for a complete article, prompt smaller chunks like “draft a section covering X.” Each subset then needs a focused review.
  • Domain experts needed: On specialised topics even credible-sounding output requires an experience check — and, as in the Giskard case, often major revisions.

Mason’s overall score for ChatGPT as a writing assistant? Useful for concrete, well-scoped drafting tasks, but not reliable for independent analysis or broad narrative construction. If you write iteratively — feeding in small pieces and expecting generated prose to start your thinking, not finish it — the technology can genuinely save time. For high-stakes publications, expert judgement will be essential for the foreseeable future.**

ChatGPT is good at translating material from one format into another, as the author discovered when turning Radar blips into a section of a Thoughtworks “macro trends” article. The task was to take notes on next-generation knowledge management tools — specifically Logseq and Obsidian — and expand them into a three-paragraph trend analysis.

The resulting draft was competent but needed editing. The tension was the same the author had seen in the draft Radar blips: the AI’s default register is relentlessly upbeat. Phrases like “more crucial than ever” and “transforming the way teams manage” read as hyperbolic, and the overall effect skews bland. A manual edit pass that swapped in more measured language fixed the tone problem fairly quickly.

The second structural issue was more persistent. Regardless of whether the author asked a direct question or requested a specific article section, ChatGPT defaulted to an introduction/body/summary arrangement. That meant repeatedly discarding or heavily condensing the opening and closing paragraphs to make the output fit the article’s actual shape. The finished piece is available on the Thoughtworks website.

Repackaging Radar content for a business audience

A different exercise asked ChatGPT to adapt a Radar theme on accessibility for a less technical audience. The author supplied a short technical paragraph and requested a split into “what is it” and “business context” sections.

The model responded with a clean one-or-two-sentence definition and a paragraph on the business implications. The author then asked a follow-up question about the meaning of “accessibility” and another about advice for businesses; both answers were solid. The author judged this output quite favorably — it was a useful, time-saving translation of technical content into a form suitable for business stakeholders, and the generated advice was reasonable.

Where ChatGPT helps in technical writing

Based on these exercises, ChatGPT has real value in the writing workflow, but the boundaries of that value are clear. Its main strengths:

  • Knowledge: It gives instant recall of how others have framed similar topics, which often surfaces relevant points the author can choose to incorporate.
  • Time-savings: Its output makes a serviceable first draft that reduces the time to a usable text.
  • Suggested additions: When fed an outline, it will propose extra sections — the model suggested “goals and expectations” and “strategies for improving ChatGPT’s output,” both of which the author adopted.

Where ChatGPT falls short

The weaknesses are equally instructive:

  • Accuracy and depth: Outputs need manual review, and the model’s hard information cutoff means recent developments are simply absent.
  • Overly positive tone: The default optimism is unsuitable when a piece calls for critical analysis, as with Technology Radar blips.
  • Context limitations: Both GPT-3.5 and GPT-4 can lose earlier context, though GPT-4 has a larger window.
  • Anchoring: Asking the model to expand its own output can produce near-identical text rather than added detail. The fix was to start fresh, using the best output so far as the new prompt.

Improving the model’s output

ChatGPT’s usefulness in technical writing depends heavily on how it is directed. Several practices help:

  • Use a context-setting prompt. Define the writing style, context and audience up front; reuse the same prompt to keep consistency across sessions.
  • Work iteratively. Treat the first response as an initial pass and steer the model toward the right shape, folding successful instructions back into the context prompt.
  • Focus on smaller chunks. GPT-3.5 handles individual sections better than whole articles. Even with GPT-4, the approach of outline-first, then section-by-section prompts is more reliable.
  • Treat it as a peer, not a subordinate. The model tends to mirror the register of the prompt, so a professional prompt improves the chance of professional output.

Keep a Critical Eye on AI Output

During my experiments, I also used ChatGPT to help learn new tech stacks. At times, GPT-4 went offline mid-session, and I found myself unexpectedly stuck without my AI assistant. Though GPT-3.5 remained available, it wasn't as capable, and I strongly preferred GPT-4. The experience made clear how quickly one can come to depend on these tools. To avoid over-reliance and preserve quality, it helps to follow a few practical guidelines.

  • Scrutinize everything: Check the AI’s output for factual errors, relevance, and whether you actually agree with its claims or implications.
  • Look for gaps: Once you have a polished draft in front of you, it’s easy to miss subtle points you would have made yourself. Actively analyze the text for missing nuance before you accept it.
  • Budget for editing: Critically reviewing and refining AI-generated content takes real time. The promised productivity gain may be smaller than expected, and if you can't invest that effort, the final quality might suffer.
  • Remember who owns it: The byline is yours. Treat ChatGPT as a helper, but hold yourself accountable for the final published work.

Ultimately, my experiments with ChatGPT as a writing assistant proved successful, though not in the way I initially expected. I didn't save time overall; instead, the AI gave me a steady starting point that helped maintain momentum. Using LLMs to generate first drafts and suggest additions seems broadly applicable across software work—from writing stories and acceptance tests to crafting documentation. I don't see these tools replacing human experts; you still need someone skilled to steer and verify the output. But I do expect such tools to help knowledge workers produce better results, and learning to work effectively with them has already become a crucial skill.