Back in 2017, Kristel Kruustük told us her inspiring story about starting her software testing company, Testlio.
Kruustük and her husband Marko Kruustük started Testlio in 2012, and within five years they had built it up into a global testing company with more than a thousand freelance testers. But in the past year the company went through some big changes. Kruustük decided to refocus her efforts on her main passion, testing software, and she became chief testing officer. She also had her first child.
And the company decided to hire Steve Semelsberger, one of its early investors, as CEO. Now the San Francisco-based team is expanding its base and providing testing services to customers whose apps reach more than 1.5 billion monthly testers, compared to 650 million monthly users in September 2017. The company is also profitable.
Testlio has 65 employees and access to 4,000 validated freelancer testers from more than 100 countries. Kruustük said the company pays better than market rates, and the main hubs for workers are San Francisco, Los Angeles, Austin, and Estonia. Customers include Microsoft, Hotels.com, CBS, the NBA, and American Express.
I spoke with Kruustük and Semelsberger recently. She said 40% of her employees are women, and 50% of the company’s leadership team is female.
Here’s an edited transcript of our interview.
Above: Kristel Kruustük is cofounder and chief testing officer at Testlio.
Image Credit: Testlio
GamesBeat: How are things different today, with the change in titles? Has the company changed under your new CEO?
Kristel Kruustük: I can give you a bit of the background story. In the beginning of Testlio, we went through the TechStars accelerator. That’s where we first got introduced to Steve Semelsberger, as one of the mentors in the TechStars program. After graduating TechStars, we stayed in very close contact.
Last summer, when my co-founder Marko — we had built the company to a good position, and we started thinking about our strengths and our weaknesses and the places where we could do more. Eventually we started talking to Steve, who had just made a switch in his career. He’s been an investor, an advisor, and a long-time friend to Testlio. We felt that if there was anyone who could take up the CEO position, it would be Steve.
This gave me the opportunity to focus on the things that I’m passionate about. I started the company as a tester, and now I’m really getting back to the roots, our testing network and our testers. It’s been awesome. The company is doing well. We’re now 65 people. We have thousands of testers in our network. When we last spoke, the products we test touched about 650 million people. Now it’s 1.5 billion. We’ve grown the company and it’s going well. I’m excited for what the future brings.
Steve Semelsberger: I’ve loved Kristel and her co-founder Marko since I first met them six years ago. I was honored and excited to participate as a seed round investor, and then become a formal advisor to the company. I’ve been working behind the scenes with Testlio for many years now. I was thrilled to come in at the moment I did last fall. Kristel and Marko and the team had done so many things well. We were in a period of launching a new platform to grow.
Our theme for this year has been mindful growth. We’re growing nicely. One other data point to add around what’s changed is we’ve become profitable. In Q1 and Q2 of this year we were profitable. We were both net income profitable as well as cash flow positive in Q2, while retaining a strong cash position. I look at my job as one of building on all the job that Kristel and Marko and the team have done, equipping the entire team to do the things that they most love to do.
Above: Testlio’s offices
Image Credit: Testlio
GamesBeat: How much bigger is the company now than it was in 2017 or so?
Semelsberger: I’ll address that around three dimensions. The first dimension, we look at the users who use our clients’ apps. The business is fundamentally an app testing business, a managed service. We provide our people, a network of contract and freelance testers, and then proprietary software. We measure how our customers apps’ are used in the wild.
As Kristel mentioned we’ve gone from about 650 million users of our clients’ apps to more than 1.5 billion users. We’ve also put a lot of energy and effort into having testers available wherever our clients need them. When we spoke to you last, we had about 1,000 validated testers. We’ve increased standards for becoming validated with Testlio. At the same time we’ve grown to about 4,000 validated testers, and about 15,000 total members of our global testing network. We have testers in about 100 countries all over the world, which equips us to do localization and location-centric testing for our clients.
For example, we work with a very large social media company, and we have testers in 40 countries. They do a combination of functional and linguistics testing. The team of testers we’ve built — there are about 80 testers for this one client — are also product users and product evangelists. They know and love the product of our client. We’ve really aspired, under Kristel’s leadership, to be able to test in any location, on any device, in any language.
The other dimension–while getting profitable, we’ve significantly increased our revenue per head. We’ve been growing revenue. If you look at us over a two-year period, we’ve been about flat on headcount. We were about 65 people when you spoke with us last. We actually made some hard decisions and got smaller for a while, and then over the last year we’ve grown back up to about 65 employees.
GamesBeat: When you’re testing things in different languages, are you also finding the kinds of things that are culturally inaccurate? That’s the work of people who call themselves geographers, consultants making sure that something in one language is properly translated on the cultural front as well as making it technically accurate.
Kruustük: Yes, we do. When we work with our testers, they’re located in those countries. They live their entire lives there, and they’re testing products that want to be localized.
Semelsberger: One example, we work with a large home rental marketplace provider. We helped them enter into Japan. They had a major push, a large marketing campaign, and an entirely new product experience for Japan. Our testers on the ground in Japan worked for us doing a combination of usability, language, and functional testing.
They discovered that when they went to use different versions of different terms for bathrooms–there’s how you refer to as a bathroom as a place with a shower versus a place with a WC, and that’s important in Japanese. Those are the kinds of things that having people who understand great product experiences and who are culturally tuned in to different home rental experiences could find on the ground.
Kruustük: There are all kinds of small things that make a huge impact. The way you format dates in an app, for example. It all matters. When we’re working with a big social media company in a place like Saudi Arabia, for example, we had to make sure the camera was not enabled in the app. There are restrictions to keep in mind.
Above: Testlio’s pledge
Image Credit: Testlio
GamesBeat: When you’re doing different kinds of support, social media applications are going to be a lot different from games and other kinds of software.
Kruustük: Exactly. Two years ago, when we last talked, we discussed testing in the games industry. Gamers are the testers in the game industry, so to speak. But when we talk about big media corporations, it’s a totally different taste. They need to make sure that everything works for people in New York or in Chicago or in Edmonton or anywhere else. There are big differences when it comes to games testing.
GamesBeat: If companies want to do this on their own, what is your pitch now as far as how Testlio can do it better?
Semelsberger: I’m in Austin, Texas, which has been my home for the last 20 years or so. I work closely with our sales and services teams, which are primarily here in the U.S. A large percentage of our customer base is in the U.S., and then a large percentage of our services, engineering, product, and back office functions are in Europe.
A lot of what we do is we augment great in-house work. Most of our clients do automated testing at a unit test level in house. Automated testing is critical. It’s fundamentally something that engineering teams manage as part of the client’s processes. What we bring in is highly burstable and flexible human-centered testing. We can deploy large teams of testers, typically working in pretty short micro-assignments, one- to three-hour test windows each. We can bring, 20, 30, 50, 100 testers to a specific candidate release, often in key locations and working in key languages, in a very short period of time, typically a 24-hour window. We can provide lots of feedback. Companies can decide what they might want to change and improve before they go into production.
This works really well for native apps specifically. A lot of our clients are releasing native apps about twice a month. They have periods where they take all their web services and everything that’s been done for the front-end native app and package it all up. That’s the moment where they hand off for testing and we spring into action with our global team. With that rhythmic, burstable approach to testing, it’s very efficient and economically valuable for our clients. It’s the evolution of outsourcing, and a really nice to complement to what publishers might do in house.
GamesBeat: As far as the kinds of data that you can collect across the whole client base, what are some insights you get as far as the problems a lot of companies share in common? What can you learn through your analytics?
Semelsberger: Our customers look to us for information about what we’re discovering in testing. That includes how certain devices and operating systems are performing. They also look to us for information around how different engineering and product teams are releasing quality build candidates. Often we’re working with many teams within a client, and we can do cross-team comparisons and comparison evaluations.
If we look at overall speed and pace, the time it takes teams to go from an initial sprint kickoff to the production release, and how efficient testing is within that window, it’s important to us that we report quality information to our customers. We look closely at things like report to resolve rates. We do a lot of work to categorize information we give our clients, so that they can then make quick decisions on what to take action against immediately, and what they might put into a backlog. We’re always working closely with our clients around process improvements and providing data.
We deliver that data through things like run reports, which we’ll email to our clients, and then we have a collection of dashboards within our software that our clients can go into and look at. Then we do things like quarterly business reviews, where we come to our clients proactively and make recommendations based on some of the things we’re seeing. Ideas about what we can change, ideas about what they can change, and ideas about things we can change together.
Kruustük: Our product has evolved a lot over the past two years. I talked before about how scalability is going to be one of our challenges. How do we gather important information for our customers so they can release quicker? The mobile industry is brutal. There are hundreds of competitors for gaming and mobile app makers. It’s really easy to switch. We try to gather information as much as we can. Our platform today enables that, which is awesome.