crowdsourcing and quality control

CrowdFlower offers a number of different options for quality control. Variables include setting contributors – and you can choose internal, external or both. You can set the contributors by geography or language, which would for example allow you greater granularity when translating into languages with a lot of regional variants, such as French or Spanish, and this can also be set by capability level.

A company can use test questions to gauge the accuracy of the platform – and to help identify contributors who are best suited for the role. A company with a particular vertical could use test questions to help find contributors that are particularly strong in translating for that vertical.
There are also controls on time, pay, and those sorts of variables. A company might decide that it doesn't want to pay much, but would do so knowing that this might reduce the overall quality of the human part of the translation. Paying more might get better talent.

The different options can influence the results in a lot of different ways. The accuracy of the crowd workers is influenced in a few ways. First, with respect to things like pay and time, the quality should be directly related to how much time someone has to get it right, and how much you are willing to pay for talent and/or time, assuming that time factors into the rate of pay at all. Workload might factor into the rate of pay though – translating an entire website in a day will get a lower level of accuracy than translating a press release in a week. These variables are passive selection, where the choice of variables will narrow the pool of potential contributors from the crowd, in a way that will influence the quality.

The test questions and contributor selection variables will help a company to get the results that it wants. When the customer has more control over the contributor, then the customer has greater ability to get the person or people that they want. So the different contributor variables are a form of active selection, where the customer is directly aiming for a particular type of contributor and when they find that person they select for them. A company wanted to translate into Spanish for the Mexican market would set a geography at the minimum, lest the get a Spaniard or Argentinian who would be less accurate in their translation work into Spanish. As noted above, there are use cases where a company in a niche market would seek out any contributor who understands that market's jargon, and how that jargon translates into the target language.

The variables will also influence on completion time. This can be done directly based on the choice of deadline and what the company is willing to pay, of course. But completion time will also be influenced by some of the geographic and competency criteria. A person from Canada might be generally familiar with British English, but will take longer than a native Briton, for whom the language comes naturally. But this nature level can also be selected by the looking for people with a high level of translating experience, as they are more likely to understand the specific issues that can arise with that language pairing. A less experienced member of the crowd is likely to take longer, even if they deliver the same accuracy level as the more experienced translator –the correlation between productivity and tenure is fairly well-established.

Payment can be selected – the customer decides what they are willing to pay and then the crowd members self-select for what they are willing to pay. But pay can also be influenced by things like the volume of work, the time frame, and the desired quality level. The latter is probably the biggest influencer of payment per page, the reason being that the machine is going to translate x % of the work, and then the amount of attention that the crowd member needs to pay is going to be determined in part by the quality level desired by the customer. If the customer only wants the translation to be proofread for syntax and grammar, that will cost less than if the crowd is also checking for optimal wording, and more nuanced elements of translation.
Payment will also in part be determined by selecting for geography. Translators living in low-cost countries will often be willing to work for less, simply because their money goes further. People living in higher cost areas will be less likely to work for anything but the highest rates, which means that selecting for geography could impact on the price as well.