In response to: https://dfwstartupcommunity.slack.com/archives/testing-feedback/p1468805613000002
It always is, a difficult problem to approach for sure.
Some of the complexities are in the pretext and fluidity of the relationships. Person A who has previous technology product experience or product management experience is going to have a much different experience with Provider A than Person B who is without that experience has.
A bit of set up for some (most likely contrived) context:
Person A - Business Founder who has previous technology product, product management or engineering (of any kind) experience — engineers tend innately understand complexity and dependencies. Person B - Business Founder who has no previous technology development experience Person C - Business Founder who wants to be wholly focused on operating and not highly involved in building or maintaining product. This person is likely to want to hire an intermediary project manager to handle the firm for them.
Provider A - Smaller group/team, doesn’t focus much on brand/PR and foregoes a lot of project management work in an effort to stay lean and keep costs low for their clients. Provider B - Medium sized group, 2-5 teams in the company. More emphasis is placed on process at Provider B than A because processes are how you ensure quality as your teams grow and become more autonomous. Provider C - Big shop, great brand name, works with tons of big guys with contingency budgets and more defined projects.
Challenges surrounding a client’s previous exposure and empathy
Provider A may be a better fit for Person A than for Person B, and Provider C a better fit for Person C. Simultaneously, the cost of subsidizing and bridging the gap between Person A and Person B’s skill-sets is a cost that must be accounted for.
Whether they hire all roles through an outside provider or hire some project manager internally and then development externally, Person A and Person B can’t be expected to pay the same amount. In many cases Person B, who has the least exposure and empathy to the process needs much more structured check-ins, explanations and reporting than someone who is closer to skill-set of Person A above. That comes at a cost of time and it’s hard for anyone to work for free.
Provider C is pure business and doesn’t care, they just wants you to tell him dates that you will make. They will have lined up demos for the dates that you agreed on, and if you didn’t understand what they wanted then they will be really angry and feel betrayed. You have now made them look like a fool and potentially damaged their credibility and perception with a number of important people.
Challenges surrounding needs and requirements
Unless Persons A, B or C have existing businesses where they are making money and you are simply automating that, your requirements are going to be brittle and mostly wrong. Even in the case of translating existing businesses you can get a lot wrong.
Estimates can’t be bench-marked when you start something new and you can’t trust estimates outside of benchmarked ones. A provider's up front estimates will be wrong and this causes 2 potential issues:
- The provider estimates far too low. Incorrectly set expectations and it turns out to be a complete nightmare.
- The provider (3/4)x their estimate to give themselves huge margins against the unknown.
- Many clients believe that you are expensive and just trying to take advantage of them.
- If the project is fixed bid and the provider "does well" (is lucky), then you gain massive margins.
- If the project is fixed bid and they do badly, then it eats their company’s lunch.
Providers can misunderstand or even not care about your potential future talent pool. Person B and C might have little to no idea what “language” they should go for. Most of the time the language doesn’t matter, but the size of the talent pool and their support community that they will be hiring from after they become independent does.
Challenges surrounding politics and social issues
- Provider C may give Person B a special discounted rate because they are friends.
- Provider A may give Person C different levels of attention or pro bono work while reaching for brand value - “we can put HP’s logo on our site!”
- Provider B may give Person A a unique deal because Person A feels like they have just been “screwed” by Provider A. Provider B’s services may not be that much better, but Person A now has more empathy and believes that it is way better since the shock factor is now lower.
Furthermore, Provider A might have had Employee B who was the absolute glue and success driver during Person A’s project, and now Employee B is gone for Person B’s project with Provider A. That adds a whole new level of granularity to the situation.
Challenges surrounding how we begin to address this problem
Everything is very situational and that makes comparisons very, very difficult. Business schools even encounter this with their case studies for MBAs. There is always missing context and the story that is retold after a success is rarely a complete picture of who did what and why things worked — direct correlation is just hard.
One big step forward that I believe there is for developing a sort of “provider review” service is to develop the metrics that we believe clearly stratify the highest probability of success in the context of your personal skill-set.
Clients, their needs, Providers, and people at Providers all change regularly; so where do we decide to generalize and draw the line. After we draw that line, how long is the feedback cycle for us to realize whether our metrics are meaningful or not? How can we more directly correlate good experiences to any of the many possible new inputs?