Books about Research Methods (5)

I have been thinking about developing a research methods course focusing on the survey method. Survey is a very old research technique. According to Earl Babbie’s “The Practice of Social Research”, the use of survey could be traced to the Old Testament – “After the plague the Lord said to Moses and to Eleazar the son of Aaron, the priest, ‘Take a census of all the congregation of the people of Israel, from twenty old and upward’” (Numbers 26: 1-2). Surveys are mainly used in studies that have individual people as the units of analysis, and to collect original data for describing a population too large to observe directly. In LIS research, survey is one of the most frequently used research methods. For example, survey research can be conducted to study user satisfaction with reference service, to assess student learning in information literacy instruction, to examine user awareness of mobile services provided by the library, etc.

To develop a course on survey research, the first thing is to find a textbook. I looked around and decided on “Internet, mail, and mixed-method surveys – the tailored design method”, by Don Dillman, Jolene Smyth and Leah Christian. It’s a comprehensive guide to designing and administering surveys. I particularly like how they talk about constructing survey questions – this is the most critical element in questionnaire design. I like their writing style too – straightforward and easy to understand. The only regret is that they did not include much discussion of analyzing survey responses. It’s understandable though – data analysis in survey research is a huge topic that warrants a book of its own. But I guess this means I will have to find other readings for this topic then. Time to dive back into the literature!

Use of incentives in social research

Recently I have had some interesting discussions with my students in the Research Methods class about the use of incentives in research studies. We all know that incentives help recruit and retain study participants, but some people consider them a form of undue influence or corruption of judgment, which may lead to biased data and damage the validity of the study. This makes me wonder, in what circumstances incentives are just innocuous incentives, and in what circumstances they morph into a temptation that attracts people to participate in a study against their better judgment?

I heard a program on the radio the other day, and it was about Amazon.com’s exclusive invitation-only club called Vine. Members of Vine are the top-reviewers of Amazon, and Amazon sends them freebies ranging from earbuds to color printers. A professor from NYU believes that such a program would lead to biased product reviews, because “as humans we are hard-wired to give in to this sort of, you know, enticement where if you continuously get things for free, then you’re more likely to be biased positively than biased negatively”. However, a spokesperson from Amazon claimed that Vine reviewers are not positively biased – they actually give lower star ratings that the average reviewers on the site. Her theory is that “it’s because they take that role so seriously to give as much sort of unbiased perspective on reviewing that product”.

This was quite an interesting story. So I dug around a bit more to find some scholarly literature about using incentives in research studies, and stumbled upon this article “Ethics in Human Subjects Research: Do Incentives Matter?” The authors stated “incentives become problematic when conjoined with the following factors, singly or in combination with one another: where the subject is in a dependency relationship with the researcher, where the risks are particularly high, where the research is degrading, where the participant will only consent if the incentive is relatively large because the participant’s aversion to the study is strong, and where the aversion is a principled one”.

This article is quite helpful to inform decisions concerning incentives. But we first have to be able to offer incentives before we consider how to offer them. I guess this means I have to get back to grant-writing now. 🙂

Talking about publishing (2)

2. Developing a strategic publishing and presentation plan

To me, being strategic about publishing means being prepared in every step of the process.

Identify the potential publication venue.
As mentioned earlier, it is important to develop a list of key journals in one’s field of research. To consider what journals to include, here are a few ideas discussed in the Webinar titled “SIG III: Getting Published in Reputable International Journals and Other Publications”: which journals do you consider prestigious; which are highly cited in your specialization; where do your senior colleagues present and publish; and what is the journal’s impact factor. Once you narrow down a few choices, for each journal: read the table of contents of the journal where you want to submit – at least a year’s worth (who and what they’re publishing, there may be a special issues); check to see if your key terms are indexed by the journal and check to see where the journal is indexed; and check the background of the editor and the editorial board. When the list is complete, you may choose one journal from it as a target venue for each manuscript you start working on.
Prepare the manuscript.
A research project, especially a complex one, may produce more than one publication. As we complete the project, we need to think about how to carve out the different pieces and turn them into multiple publications. Then, we can set a timeline for writing the manuscripts one by one. Writing is often the most difficult part in the process. It takes time, discipline and persistence. The aforementioned Webinar had a few suggestions on honing one’s writing skills: avoid jargon, define major terms and concepts, use terms consistently, ask trusted colleagues for feedback, consider hiring a proof reader, read about writing, e.g., Virginia Tufte (Artful Sentences), Gopen & Swan (The science of scientific writing), practice writing, keep a journal (or a blog), and take advantage of the writing support provided by one’s institution.
The publication decision.
There are usually four decisions – accept as is, minor revision, major revision, and rejection. The first two are practically good news, so we will just talk about how to deal with the latter two. Major revision can be a mixed bag. Sometimes the editor can decide whether to accept your revision, and sometimes the revised manuscript has to go out for another round of peer review. As for the reviewers’ comments, we may not agree with all of them, or be able to address all of them. When deciding whether to continue with the revision or withdraw the submission, we need to consider how well we can address the reviewers’ comments in the revision, and whether it’s worth the time and effort. If we do decide on revising, we need to document how each point in the reviewers’ comments was addressed – if we couldn’t fully address a particular point, it’s necessary to provide a detailed explanation. Such a document is often required when resubmitting the revised manuscript. If the publication decision is rejection, we can still recycle the manuscript and find another venue to submit it. Handling major revision and rejection can be frustrating – the key is not to take it personally and develop a thick skin for criticism.

Talking about publishing (1)

I will be joining our Gateway PhD Program’s residency in the upcoming ASIS&T annual conference in Montreal, and I have been asked to talk to the doctoral students about publishing. I’d like to use this blog post to clear my thoughts and figure out what I’m going to say. I was given a couple of specific topics that need to be covered, which is helpful and I can organize my thoughts around them.

1. Selecting publication and presentation venues

There are usually two kinds of research journals – scholarly and professional. Both publish peer reviewed articles, written by and for people who have experience or expertise in a particular field, and both are respected for the research and information they provide about the topics they cover. The difference lies in the community of author and readers – for scholarly journals, author and readers are scholars and academic researchers, and their articles mostly serve the purpose of theoretical, methodological and knowledge advancement in a specific domain. Therefore, scholarly journals can be quite demanding when it comes to the rigor of a study’s research design. Professional journals focus more on the practical implications of research – how the published research can inform actual practice. Authors of professional journals are usually practitioners, who conduct research to address actual problems they encounter at work and publish about it.

Whether choosing a scholarly journal or a professional journal as the potential venue for publication, it’s important to have a clear idea of what our research is about and who we want to share this research with. It would be helpful to establish a list of four or five core journals that often cover our research areas and become familiar with their style and requirements (reviewing past issues would help). Before we start writing up our research, we can scan the list quickly and decide which journal we’d like to submit it to, and then complete the manuscript according that journal’s guidelines.

Conference is another venue where we can disseminate our research – there are also scholarly conferences and professional conferences, and the difference is similar to what’s discussed above. For conference presentations, some conferences require submissions of full papers that go through the peer review process, and some only require a proposal or an abstract to be viewed by a committee instead of peer reviewers. In some fields, conference proceedings have the same status as journal publications, while in some others, presenting at conferences is not valued as much as publishing in journals. Despite all the differences, conference is a great venue to have interactions with people who share our research interests and can offer us feedback on expanding/refining/improving our own research. Again, it’s important to identify one or two key conferences that welcome the types of research we do, and become familiar with their submission deadlines and requirements.

(to be continued)

Content analysis of LIS journal articles: What kinds of research are LIS practitioners engaged in

Over the years of teaching LIBR 285, Research Methods in Library and Information Science, I have always wanted to do one thing – to look at the journal publications of practitioner researchers and see what kinds of research they are producing, and how their research is informing the decisions they make in their practice. This will help my students better understand and appreciate the value of the research methods course, and therefore embrace it more willingly and enthusiastically. So I developed a grant proposal based on this idea and submitted it to the SJSU Research Scholarship and Creative Activity (RSCA) Program, and luckily it got funded, which means I will spend my next summer working on this project.

I plan to conduct a critical content analysis of the articles published in the core practitioner-oriented LIS research journals in the past decade, identifying the topics, trends, methods, strengths, and weaknesses of practitioner research. The content analysis will (1) determine the topics studied by practitioners; (2) provide a historical view of LIS research; (3) identify what methodological designs have been used and whether they have been used appropriately; and (4) ascertain exemplary areas of practitioner research as well as areas in need of improvement. Hopefully, findings from this study will yield insights for enhancing both research methods education for LIS students and research training for practitioners.

Speaking of content analysis, a book that has helped me a lot with this methodology is “Content Analysis: An Introduction to Its Methodology”, by Klaus Krippendorff. Especially the chapters on evaluative techniques offer some good ideas of addressing issues related to reliability and validity. Maybe it’s time to revisit the book before I start this project of analyzing journal articles.

(reposted from SLIS CIRI Blog)

Books about Research Methods (4)

If you are a LIS practitioner interested in learning more about qualitative research methods, G.E. Gorman and Peter Clayton’s book, “Qualitative Research for the Information Professional: A Practical Handbook” might be a good choice. It is a comprehensive manual of how to conduct qualitative research, and covers a variety of methods for qualitative data collection, analysis and reporting.

For researchers who know which specific qualitative research method to use, but aren’t sure about how to exactly go about it, this book would come in very handy. For example, we want to conduct a case study, but we don’t know the detailed procedures of using this method, reading Gorman and Clayton’s book will be quite helpful.

However, I find the book a little week on the research conception aspect. It talks little about how to identify research problems that could benefit from qualitative research. In order for people to truly understand when to use qualitative research to address a research problem, it would be helpful to provide more examples. The authors did provide a table outlining the distinctive features of qualitative research (e.g. the purpose is to contextualize, interpret and understand participant perspectives), but for each feature, it would’ve been helpful to use an existing study or two from the literature to illustrate it. Maybe for their next edition, they may consider incorporating more examples, so readers will have a more concrete idea as to how to determine whether qualitative research is appropriate for a particular research inquiry.

Overall, the book is written in a clear and concise fashion, and it’s a fast read too. I’d recommend it to people who want to know more about the technical details of using a specific qualitative method.

Determining the proper research methods (3)

In the previous two posts, we looked at how we could choose a proper research method for our study needs, specifically, the qualitative or quantitative instruments we can use to gather data. But a research study’s methodological design is more than determining the specific data collection instruments. There is a broader consideration – the overall framework of the design. This framework could be a particular type of research design or a research paradigm that provides guidance for the data collection process.

For example, we are conducting a study to evaluate the impact of a new pedagogical approach on learning outcomes, and we may consider the experimental/semi-experimental design. In our study, we have an independent variable – new pedagogical approach, and a dependent variable – learning outcomes. We can treat the independent variable as the experimental stimulus, and see what happens when it’s present/absent. Given our resources, we may further consider whether a classical experiment is possible, or we should use semi-experimental designs like the time-series design or using non-equivalent control groups. Once we are clear about the design, we can further figure out the specific data gathering methods – methods we can use to collect data that measures the two variables.

Another example – let’s say we are interested in finding out what competencies are essential for reference librarians. Survey could be an appropriate method to use – we can send out surveys to reference librarians and ask them to self-report the competencies they consider essential to their work. But what if we are hoping to generate a consensus from the study, where the list of competencies is agreed upon by the study participants? In that case, we may use the Delphi study design as the design framework to guide the data collection process – the first step is to select a panel of experts (e.g. established authors in journals like RUSQ, leaders of the profession such as the head of RUSA), and then, we administer rounds of surveys among them to reach a consensus on what competencies are essential for reference librarians.

In qualitative research, there are numerous research paradigms that we can use as the framework of methodology –case study, grounded theory, participatory action research, institutional ethnography, ethnomethodology, and naturalism. These different paradigms represent different approaches to qualitative research. There are no specific methods attached to these paradigms. The distinctions are epistemological, having to do with what data mean, regardless of how they are collected.

Now it’s time to conclude the series of posts on determining the proper research methods – from the overall design to specific data collection methods, we need to understand both our own study needs/constraints and the characteristics and affordance of each method/design in order to make the best decision.

Determining the proper research methods (2)

Once we know whether we know what type of research we will be focusing one (qualitative vs. quantitative), the next step is to figure out the specific method for data collection. For quantitative research, the frequently used methods include survey, quantitative observations, and quantitative content analysis. There are different factors to consider, but the characteristics of each method carry the most weight. For example, we are conducting a study to find out if there’s any relationship between library users’ socio-economic status and their reading behavior. The socio-economic status is defined by annual income, and the reading behavior is indicated by the number of books they check out, the genres of the books, and check-out frequency, etc. We may distribute survey to all library users and ask them to provide information on these variables, but mail survey could be costly, and online survey often gets a very low response rate. Besides, it’s always imposing when we send out survey questionnaires. So, is there any other method that could overcome these limitations and get us the same data? One alternative is quantitative content analysis. If we have access to all library users’ library records, we can anonymize them, and then look at each user’s address (where people live could be an indicator of their level of income, and therefore socio-economic status) and their borrowing record. This method is completely unobtrusive, and will give us all the information we need on the two variables in the study, and maybe more accurately than a survey would because self-reported data is not always 100% precise (there might be memory lapses, incorrect interpretation of the question, etc.)

When it comes to qualitative research, the popular methods include field observations, focus group interview, and qualitative interview. Again, it’s important that we understand the affordance of each method in order to determine which one suits our needs best. For example, a library has a new teen space and we would like to find out how teenagers are using this space. We could go to the teen space and observe what teen library users are doing there (reading books, resting on the comfy couch, meeting in groups, etc.). Being a complete observer helps us maintain our objectivity, but the detachment from the phenomenon we are studying makes it difficult to get an in-depth understanding. So, we could invite teenagers to participate in focus group interviews and talk about how they are using the space and what they think about it. Such interview will surely provide us with a more in-depth view of the use of the teen space, but guiding and moderating a focused conversation is not easy and takes a lot practice and if not done properly, it would result in biased data.

Now we know that understanding each data collection method’s characteristics (advantages, disadvantages, etc.) is the most important factor in determining which method to use in our own study. There are also other factors, such as our access to the study population (how can we sample), the demographic of our study population (if we are studying children, we need to be super careful about surveying/interviewing them), and constraints like budget (do we have the money to provide incentives) and time frame (how much time do we have to complete the study).

(to be continued)

(picture source http://alephunky.deviantart.com/art/Choose-your-heart-190249846)

Determining the proper research methods (1)

In early August, our Gateway PhD students will come to San Jose for a one-week residency, and I was asked to give a talk to them about choosing proper research methods for their studies. To prepare for this talk, I will be sorting out my thoughts on this blog, and hopefully I will find the best way to cover this complex topic in the thirty minutes I’m given.

We all know that there are two types of research – quantitative and qualitative. Quantitative research seeks to describe observations of a phenomenon in quantitative measures, and results of quantitative research are usually numerical represented. Qualitative research, on the other hand, defies quantification and it captures the nuanced details of a phenomenon that cannot be observed by quantitative methods. Which type of research to pursue has everything to do with the nature of one’s research topic and research problem.

For example, we are studying people’s attitudes toward a new library policy, and we may approach it quantitatively. We may administer a survey among library patrons. On the survey, there are five statements representing different attitudes toward the policy, and patrons are asked to select the one they most agree with. Findings of the study can be described via measures like frequency distribution, mode, or even correlational measures (e.g. the relationship between demographic variables and the statement choice). For the same topic, we may also approach it qualitatively. Instead of using the survey instrument, we gather patrons in the library conference room to conduct focus group interviews. This means of inquiry will give us an in-depth view of their attitudes toward the policy, which will be a much fuller view than what the five statements can cover.

Now we are at a dilemma – which type of research should we engage in? Well, we need to go back to our original problem – people’s attitude toward the new library policy. How do we operationally define the variable “attitude toward the policy”? We may ask questions like – has there been any research about the new policy? Do we know enough about this policy to generate an exhaustive list of attributes for the variable (e.g. a list of statements to describe every possible attitude)? Is it our goal to find out how many people have what attitudes, or do we just want to understand how exactly patrons respond to this policy?

As you can see, these questions are helping us decide whether we want to pursue this topic deductively or inductively. The deductive approach allows us to go from general to specific – that is, we have a general theory, and we want to test it out in specific cases. The inductive approach is the other way around – we go from specific to general, and we make observations of specific cases and draw conclusions from that. So, if our answers to the above questions are – yes we do know enough to general an exhaustive list of attributes for the variable “attitude” and we do want to find out how many library patrons have what kinds of attitudes, it means we are approaching the topic deductively and should engage in quantitative research. On the other hand, we may approach the topic inductively and pursue qualitative research to find out what exactly are people’s attitudes toward the library policy.

(to be continued)

Books about Research Methods (3)

This book I’m introducing here, Edward Tufte’s “Visual Explanations: Images and Quantities, Evidence and Narrative”, is actually not about research methods. However, it is still relevant as it talks about how to visually display data and information, which can be results of research studies. Reporting and presenting research is as important as conducting research. After all, one of the goals of doing research is to inform practice via the dissemination of the findings. So, the topic of reporting and presenting of research definitely deserves a place in any research methods class.

I first came to know about Tufte’s book when I was still a doctoral student. My officemate Ron strongly recommended it to me. It was indeed a good and informative read – I enjoyed the thorough discussion of how to use charts and graphs to effectively display numerical information. The most impressive part of this book was the chapter on the Challenger disaster in 1986. The engineers had concerns about the launch, but failed to communicate their worries to NASA due to ill-designed graphics. Tufte reconstructed the data and produced convincing visual display suggesting the launch should have been postponed.

This book is very helpful for people who do a lot of quantitative research. It helps us understand there are creative and yet effective ways to present the seemingly boring numerical data. I have two presentations coming up in August, and both of which are about some quantitative studies I did. So I guess it’s time for a refresher read of this book.