lean-startup, customer-developmentCustomer interviews are expensive to conduct in terms of time and money. Say you want to know what is the most popular method for web-scraping or data-retreival in artificial intelligence. You could ask an expert in this matter, but getting to that expert would be a lot of work. There are currently a lot of well-developed tools for web-scraping and automated textual analysis such as Scrapy, BeautifulSoup, NLTK etc.
If you are a programmer and can relatively easily build automated scripts to build programs which can give you a quantitative answer to such questions, (which cant always be found by Googling), why not build such automated scripts instead of doing customer interviews?
How do you guys decide when to do customer interviews to answer questions that may arise when you are building products vs doing some automated data mining on the web?
If you are a programmer and can relatively easily build automated scripts to build programs which can give you a quantitative answer to such questions, (which cant always be found by Googling), why not build such automated scripts instead of doing customer interviews?
I’d put three reasons forward (by no means an exhaustive list):
Automated data collection sometimes is much more complex or expensive than it looks at first glance. Particularly web scraping, owing to the scores of bot countermeasures in the wild.
Sticking to quantitative data only can lead you to miss the forest behind the tree.
If you measure a large number of things, you’re almost guaranteed to get a “statistically significant” finding regardless of reality, as illustrated by the chocolate weight loss hoax.
Product management offers an example of the dangers of using quantitative findings without doing qualitative interviews. If you stick to using stats and analytics on how users use your product, you may end up concluding that you’re meeting what you thought your users’ requirements were, when in reality your users are all bitching about the convoluted workflow you’re imposing them - and the missing functionality that you’re not aware of - when they’re solving their actual problem. For instance, regular usage of log viewers that are scattered all over the place might be hiding the fact that your product is doing a pathetic job at helping your users troubleshoot whatever your product allows them to do.
How do you guys decide when to do customer interviews to answer questions that may arise when you are building products vs doing some automated data mining on the web?
You’ll get different opinions depending on who you ask. Personally, I view quantitative data as mostly worthless without qualitative context:
As an aside, qualitative studies need not be large scale. Key findings will usually surface after a half dozen well conducted interviews - they’ll be the ones most or all of your interviewees will have complained about. Focus on those, solve them, then rinse and repeat with a new batch of users.
All content is licensed under CC BY-SA 3.0.