When to Shortlist a Software Development Agency
This guide is for CTOs, founders, and product leaders who already have a shortlist of software development agencies. You need a rigorous framework to make the final call. This guide skips the basics and focuses on the evaluation criteria, red flags, and questions that separate agencies who deliver from those who disappoint.
What Really Matters at the Shortlisting Stage
Most shortlisting mistakes happen before the first formal evaluation meeting. These four principles keep your process grounded in evidence, not proposal polish.
- Portfolio relevance beats volume. Ten case studies in unrelated industries tell you less than two highly relevant projects with clear outcome data. Always ask for comparable projects: same sector, similar complexity, comparable team size. Outcome data means metrics, not testimonials.
- Proposals show what they built; references show how it went. Ask references specifically about timeline adherence, communication under pressure, and how the team handled problems, not just whether the project was broadly “successful.” The difficult moments reveal the real delivery culture.
- Enterprise and healthcare specialisations need verifiable proof. Look for named client work, verified credentials, and demonstrated regulatory knowledge. Marketing claims without specifics are not evidence of specialisation, they are marketing.
- Architecture conversations must come before scope definition. Agencies that skip discovery and jump straight from brief to wireframes are optimising for billing speed, not your outcome. Requirements that are not properly explored become expensive changes mid-project. Getting this right early is what separates scalable product architecture from a codebase that needs rebuilding in eighteen months.
An Evaluation Framework That Works
Many evaluations focus on day rates and proposal polish. But the agencies that look best on paper do not always deliver. Use this framework to evaluate every agency on your shortlist against the same criteria. Every dimension in this table is also a negotiation point. The same principles behind vendor negotiation for startups apply just as directly to a software agency contract as to any other vendor relationship.
| Evaluation Dimension | What to Look For | What to Watch Out For |
|---|---|---|
| Technical Depth | Architectural decisions explained with project-specific reasoning and examples | Generic tech stack lists without context or justification |
| Domain Experience | Named clients in your industry with proven, measurable results | Broad claims with no specifics, no named references, and no data |
| Delivery Process | Clear sprint cadence, structured demos, and defined escalation paths | Agile mentioned without real operational detail or examples |
| Team Structure | Named senior engineers assigned before contract signature | Generic team descriptions with no named contacts until after signing |
| Post-Project Continuity | Defined maintenance terms with SLA included before contract | Support described as “available on request” with no contractual structure |
| Commercial Accountability | IP ownership terms and change order process documented in the proposal | Ownership and change management treated as afterthoughts or left vague |
Enterprise Software Development: What to Check
Enterprise projects carry higher complexity and stricter governance requirements. The questions below are designed to verify real capability. Agencies without enterprise experience will struggle to answer them with specifics.
- Program governance: Does the agency have dedicated PMO or programme managers with proven enterprise experience? Ask for org charts showing who manages stakeholder reporting and escalations, not just who writes code.
- Security and compliance: Can they provide verified ISO 27001 or SOC 2 Type II certifications with audit dates? General security claims without dated, independent verification are not acceptable for enterprise engagements.
- System integration experience: Ask for specific stories about real ERP, CRM, and identity provider integrations, including the challenges they encountered and how they resolved them. Smooth success stories only tell half the picture.
- Stakeholder management: How do they structure steering committees and handle scope change formally? Ask how a specific scope conflict was managed under time or budget pressure. The answer reveals more than any process document.
- Scalability evidence: Can they point to infrastructure decisions they made specifically to support enterprise load and multi-region deployments? Ask for the architecture decisions, not just the outcomes.
If an agency struggles to answer these questions with specifics — named clients, audited certifications, and real integration stories — they are not ready for enterprise-scale work. This is true regardless of how their proposal reads. Probe these areas before you invite them to a final shortlist presentation. These questions map directly to what it actually takes to build enterprise-grade systems. Such systems support compliance, security, and scale from day one, not as features bolted on after the fact.
Healthcare Software Development: The Non-Negotiables
Healthcare software has zero margin for compliance gaps. These are non-negotiable requirements, not optional quality indicators. They are the baseline for any agency working in this space.
- HIPAA compliance documentation: The agency must have documented HIPAA compliance processes, not just general awareness. Ask to see their data handling policies and how they apply in practice during development and testing.
- Real EHR integration experience: Demonstrated integrations with platforms like Epic or athenahealth, with specific stories, not just platform logos. Ask what broke and how they resolved it.
- HL7 and FHIR data exchange: Practical experience building or consuming HL7 messages and FHIR APIs. Generic claims about “interoperability” without specific protocol experience are not sufficient.
- Medical device software regulations: Understanding of IEC 62304 for medical device software and FDA SaMD guidance where relevant. If your software touches clinical workflows, this is non-negotiable.
- Business Associate Agreements: BAAs must be a standard part of the engagement, not negotiated as an add-on. Any agency treating BAAs as optional does not understand the regulatory baseline for healthcare work.
Discovery Phase: The Ultimate Diligence Signal
If a core technical assumption in your project is still unproven, ask whether the agency's discovery process includes a structured proof of concept. This step should happen before full development begins. This is often the fastest way to de-risk the riskiest part of the build. How an agency runs discovery tells you more about their delivery quality than their portfolio or sales pitch. Use the table below to interpret what you observe.
| Discovery Behaviour | What It Signals | What You Should Do |
|---|---|---|
| Challenges your brief | They protect your outcome, not just their billing | Engage deeply, and test the quality of their thinking and how well they understand your business context |
| Skips discovery, jumps to wireframes | Prioritises billing speed over delivery quality | Red flag, ask explicitly what happens when requirements change mid-project |
| Produces written, signed discovery deliverables | Structured, formal change management process | Ask to see examples of discovery documents from past projects |
| Charges for the discovery phase | They treat discovery as professional work with real value | This is the right approach, discovery that costs nothing is usually worth nothing |
| Focuses on business outcomes, not just features | Understands the commercial value of the build | Ask what they would improve in your current brief based on what they’ve heard |
What Separates a Genuine Custom Software Agency
Custom software development requires fundamentally more rigour than platform configuration or template builds. The table below shows what separates agencies with genuine custom capability from generalists who adapt existing platforms and call it custom work. You might still be deciding whether your project genuinely needs custom development, or would be better served by a platform build. Our broader guide to web development services breaks down when each approach actually makes sense.
| Capability | Genuine Custom Agency | Generalist Agency |
|---|---|---|
| Architecture Decisions | Defends stack and structure choices with project-specific reasoning | Recommends the same stack irrespective of project context |
| Requirements Management | Produces formal SRS documents with sign-off; changes go through a defined process | Requirements evolve informally via email threads |
| IP and Code Ownership | Standard contracts assign all work product to the client from day one | Ownership discussed informally or left vague until contract signing |
| Testing and QA | Automated test coverage defined upfront with continuous QA throughout | Testing often last-minute or cut under time pressure |
| Documentation | Technical documentation delivered as a standard part of the project scope | Documentation promised in proposals but rarely complete at handover |
These standards are not always included in proposals without a direct request. Before shortlisting further, ask each agency to confirm their position on SRS documentation, IP assignment, automated test coverage targets, and technical documentation delivery. Their answers, and how quickly they can answer, will tell you a great deal about their process maturity. Getting this documentation and handoff wrong is one of the most common reasons development handoffs fail. Clients are left to sort out gaps in support and knowledge transfer after the agency has moved on.
Ready to Decide?
Your shortlist probably includes at least one agency that meets these rigorous standards. Use this framework and these questions to identify who it is before signing.If you want expert help reviewing your shortlist and brief, book a free 30-minute evaluation session, no sales pitch, just clear advice.Once you’ve narrowed your shortlist using this framework, the next step is making sure your project with a software development agency starts smoothly. That covers contracts, IP ownership, and the first 30 days.
Frequently Asked Questions
Two to four is the right number: enough to compare meaningfully without creating an evaluation process so large it consumes weeks of stakeholder time. Use directories like Clutch or DesignRush to filter for relevance to your sector and project type. Then hold brief chemistry calls of 20-30 minutes with each one. Use these calls to assess communication quality and whether they ask good questions, before you invest time in detailed proposals.
A genuine custom software agency tailors architecture specifically to your business requirements. It runs formal discovery with signed deliverables and produces SRS documentation. It also assigns IP to the client in standard contracts and delivers technical documentation as a project output. A generalist agency typically reuses proven stacks and adapts existing platforms. This is faster and cheaper for standard requirements. But it produces shortcuts and costly rework when your needs deviate from their defaults. The key test: ask them to defend their proposed stack choice for your specific project. Vague or generic answers indicate a generalist approach.
Enterprise engagements require formal programme governance: dedicated PMO resource, structured steering committees, and defined escalation paths. They require compliance certifications that can be independently verified (ISO 27001, SOC 2 Type II) with recent audit dates. System integration experience must be proven with specific stories, not portfolio logos. Stakeholder management must be structured and formal, with a defined change management process for scope changes. The critical rule: verify all of these claims through references, not portfolios. References reveal how the agency performed under real conditions. Portfolios show the outcome they want you to remember.
Ask each vendor to map their deliverables against a shared checklist. Several scope items commonly differ and affect price comparability. Testing and QA: is automated test coverage included, or is testing manual only? SEO and performance optimisation: in scope, or out? Technical documentation: delivered at handover, or available only on request? Post-launch support: a SLA-backed retainer, or ad hoc? You can only compare prices fairly once you have mapped inclusions and exclusions against the same reference list. A lower headline price that excludes documentation, testing, and structured support often costs more in total.
References are most valuable when the questions focus on behaviour under pressure, not just project outcomes. The most revealing questions: How did the agency communicate when a milestone slipped? Can you describe a specific time requirements changed: what was the process and the cost? Were the senior engineers named in the proposal actually on your project throughout? Was post-launch support structured with SLA terms, or was it ad hoc? Would you hire them again for a project of similar complexity? Listen for specifics. Vague praise ('they were great to work with') is not the same as evidence of delivery quality.
Discovery quality is the single best predictor of delivery quality. An agency that runs thorough discovery, challenges your brief, produces signed written deliverables, and charges appropriately for the phase is showing you something important. They treat requirements as the foundation of the project, not an administrative step before billing starts. Agencies that skip discovery, or offer it for free as a courtesy call, are showing you something too. They will handle ambiguity mid-project informally, with the risk sitting on your side. If an agency cannot explain their discovery process with specifics and show you example outputs, this should be a firm red flag.
Four terms must be in the contract before signing. They should never be negotiated as add-ons. (1) IP and code ownership: all work product, including source code, design assets, and documentation, must be assigned to you at project end as standard. (2) Post-launch SLA: support must be contractually defined with response time commitments, not available informally. (3) Change order process: scope changes must go through a defined, documented process with agreed pricing methodology. (4) Payment tied to milestones with defined sign-off criteria: payments triggered by calendar date rather than delivery milestones transfer risk to you. Any agency that treats these as negotiation points rather than standard terms is telling you something important about their client relationships.
Any project that handles protected health information (PHI), integrates with EHR platforms, or involves clinical workflows requires a specialist agency. The compliance and regulatory baseline includes documented HIPAA processes, FHIR and HL7 experience, IEC 62304 knowledge for medical device software, and Business Associate Agreements as a standard contract element. A general agency cannot credibly claim this baseline without a significant track record in the sector. A useful test: ask the agency to describe how they managed HIPAA requirements on a recent engagement. Have them include the data handling policies they used during development and testing. Agencies without genuine healthcare experience will answer in general terms. Specialists will answer with specifics.