From what population was this data compiled?
The data included in the B-KNorms was collected from 498 City University of New York (CUNY) students. These students provided over 85,000 responses to the general knowledge questions included in the database. The majority of students were male (46% female) with an average age of 20.8 (SD = 2.82). Self-reported ethnicity was divided in the following way: 37.1% Asian, 24.7% white, 9.8% more than one race, 7.2% Black/African American, and 1.4% Native Hawaiian/Pacific Islander, with 29.7% of students identifying as Hispanic or Latinx.
How did the participants complete the questions?
While the research designs varied somewhat over five different studies, all participants were prompted to give a free recall response to each question they saw. Most participants saw a sample of questions that was adapted based on how accurate their answers were. If their accuracy rates exceeded a target level, they were given more difficult questions to bring their accuracy level down to target. If their accuracy rates dipped too low beneath target, they were given easier questions to try to bring their accuracy level up to target. We determined the average accuracy of each question by dividing the raw frequency with which the correct answer was given by the number of times the question was sampled. Thus, a question with a higher accuracy score is relatively easy because the correct answer was given more frequently.
How were the questions obtained?
The questions were derived from a variety of sources. Some of them came from the Nelson and Narens (Journal of Verbal Learning and Verbal Behavior 19:338
Have these questions been validated?
We have extensively validated these questions with our sample from CUNY, predominately Baruch College. Baruch College is a selective senior college in the CUNY system (for more information about Baruch’s student population, please see this page). This sample may differ in a few significant ways from the sample you are interested in using. Our sample identified as largely Asian (when they identified their ethnicity; 37.1%) and 24.7% white. Many Baruch students are also international students, and may have different familiarity with certain questions compared to students who completed all their education in the US, since some of the questions refer specifically to American history that is generally taught in lower grades in US schools. Baruch students also performed very well on New York-specific questions (e.g., “Over which river is the George Washington Bridge?”) that may be harder for students from other regions. Indeed, the difficulty of geographical questions may be particularly influenced by the geographical location of the population answering these questions. Nonetheless, we found a strong correlation (r = 0.82) for a subsample of B-KNorms items that were also tested in Tauber et al. (2013). Tauber et al. (2013) did not explicitly report the ethnicities of the students in their sample, but their sample was primarily from a Midwest university (Kent State University, which is 75% white), suggesting that the overall patterns of our data will be robust to some variation in population characteristics. Notably, Tauber et al. (2013) also used a somewhat different method for eliciting answers, suggesting that the database metrics will also be fairly robust to differences in general knowledge recall paradigms. If you are looking for more precise metrics of question difficulty and/or answer frequency, however, it is always a good idea to validate measures within the specific population you wish to use. Generalizability across samples is a particular issue if you with to use geographically specific questions.
How should we reference the B-KNorms?
The B-KNorms should be acknowledged in any manuscripts originating from projects using the B-KNorms in the following manner:
“Development of the B-KNorms was overseen by Jennifer Mangels and supported by the Network Science Collaborative Technology Alliance of the Army Research Laboratory. Please contact Jennifer Mangels at email@example.com for more information concerning the B-KNorms.”
A digital copy of manuscripts resulting from use of the B-KNorms should be sent to Jennifer Mangels <firstname.lastname@example.org>.