Research methods in library assessment

I gave a workshop about research
methods in library assessment at Santa Clara University Library earlier this
month. It was a wonderful opportunity to connect with SCU librarians. The
workshop focused on two things: 1) how to develop the research question in a
library assessment project; and 2) how to determine the proper research methods
for conducting the assessment. Research question develop is the critical first
step in any research project – Albert Einstein once said “If I had an hour to
solve a problem I’d spend 55 minutes thinking about the problem and 5 minutes
thinking about solutions.” In the workshop, we first talked about the progressive
focusing process of formulating the research question (overall goal –>
objectives –> research questions), and discussed the criteria we can use to
evaluate the quality of a research question. ACRL’s 2015 January Assessment in
Action Report titled “ACADEMIC LIBRARY CONTRIBUTIONS TO STUDENT SUCCESS:
Documented Practices from the Field” provided some examples of research
questions in library assessment, and we spent some time examining each one of
them (see the picture below) – it was a great way to look closely at the process of developing research
questions. Then we reviewed the major research designs (based on the analytic
objective, time dimension, and degree of variable manipulation) and the frequently
used qualitative and quantitative methods in library assessment. Quantitative
methods seem to be more popular in assessment projects.

I’m glad to have this opportunity
to chat with SCU librarians about research methods – I admire their initiative
to engage in library assessment using valid and reliable research methods. This
workshop made me think about, as LIS research methods instructors, what we can
do to work more closely with libraries and librarians that conduct assessment.
Looks like I need a lunch date with some of my fellow methods instructors and
my assessment librarian friends. 🙂

A recent international research collaboration

International collaboration is always a refreshing and even
enlightening experience to me. I have worked with two librarians from Tsinghua
University Library in China on a couple of projects and absolutely enjoyed it. I
have known them for more than 10 years so we have a very efficient and pleasant
relationship. Last year when I was at IFLA,
I met a librarian from Ghana and we had good conversations about library
research. This spring we worked on a project together to evaluate the reference
services at University of Education, Winneba (UEW) in Ghana. We decided to
approach the evaluation from the user perspective, and identified the Key
Performance Indicators (KPIs) based on the RUSA Guidelines
for Behavioral Performance. A user survey was conducted to measure the KPIs
and to examine how users use and perceive library reference services. The
findings were illuminating. For example, regarding how the reference service
should be improved, some users recommended that librarians be better trained.
It’s interesting that these users realized that the service inadequacies were a
result of insufficient personnel training. It is also interesting to note that
personal interests is the second most popular motivation for users to use the
reference service, which is auspicious and shows that students and faculty
trust reference librarians with their personal information needs. And yet, some
of the things we take for granted such as electricity and computers/copiers,
were still a concern for some library users at UEW in Ghana – they felt there
weren’t enough of them to meet their needs.

Overall the study was a great learning experience for me. I’m
glad that our findings will help UEW library determine the service areas where
improvements are most needed and develop necessary training programs to address
them. On a personal note, the study opened my eyes to how reference services
work at an academic library in Ghana, and how library users are experiencing the
services. It will be published in Library Review soon, so keep an eye on it if
you are interested.

2015 Institute of Research Design for Librarianship

Last week we concluded the second
year of the Institute of Research Design for
Librarianship (IRDL), an IMLS-funded project that seeks to create a
learning opportunity and a support system for academic librarians who want to
improve their research skills and increase their research output. It’s another
great 9-day institute with librarian scholars from around the country. This
year, the project directors made some changes based on the feedback from last
year’s cohort so that the scholars could have more time to write and consult
with us instructors. I truly enjoyed talking to each scholar about their
research projects. I was impressed by the innovation and initiative
demonstrated in their projects – for example, one scholar was interested in
implementing a texting-based outreach and reference service and see if that
impacts students’ library anxiety, and another scholar was thinking about
creating library tutorials using Vine and see if that would be an effective way
to impart information literacy skills. I can’t wait for them to complete their
study and get published.

Among this year’s cohort, three
were graduates from my home institution, SJSU School of Information, and two of
them took classes with me – one was in my Research Methods class, and another
was in my Reference and Information Services class. It’s great to see them
become active researchers in this profession and engage in evidence based
practices to improve their work. I’m very proud of them. This is exactly what I’m
hoping to see through research methods education both in LIS degree programs
and beyond (like IRDL).

The campus of Loyola Marymount
University was as beautiful as ever. I feel lucky that IRDL is held at LMU library, in a spacious room with a
gorgeous view. Many thanks to Marie and Kris, the two wonderful IRDL project
directors, who took good care of us while we were there. Looking forward to
next year already!

Research topics from my research methods class in fall 2014

My research methods class in fall 2015 focused on research. Students in the class were expected to develop a survey research proposal. Here’s a collection of their survey study topics. Many of them are needs assessment and program/service evaluation studies.

Does attending the “Sopa de letras” workshop series for parents cause an increase in behaviors that are considered beneficial for literacy and learning of children?
What programs would college bound teens like to see the library incorporate into its services in order to help them with college preparation?
Do series-based library workshops improve the job skill sets of unemployed or low-income adults?
Why are patrons using or not using the library databases?
What kinds of professional development opportunities should the library offer to help teachers better understand the services the library could offer in relation to the Common Core State Standards?
What services (staff preparedness, physical layout), collections and programs do public libraries in the state of California offer their visually impaired patrons?
What is the effectiveness of the Mill Valley Public Library’s current marketing tactics in their efforts to market library programs and events?
At what rates do elementary school librarians and teachers use digital audio players and media for audio reinforcement over physical audio players and media, or vice versa, and why?
What about the existing library teen services/collection/spaces/programs do KDBS students think is adequate, inadequate, and needs improvement?
Does the pedagogical approach of Reading Workshop impact the range of genre selection for independent reading for upper elementary students (3rd-5th grade) from their school library?
How do musicians perceive the usability of the intrinsic qualities of digital scores?
What are the reasons for immigrant populations’ non-use of the library?
Do adults have interest in attending arts classes at the Cambridge Public Library?
What genre/types of books are young adults most interested in?
Does creating a young adult space in a library enhance the overall library experience for people in this age group?
Do Spanish-speaking patrons feel their public libraries provide materials and programming that reflect their needs and interest of their communities?
What specialized accommodations for existing services do higher education academic libraries provide for students with auditory and visual disabilities?
Is there a relationship between academic librarians’ instructional practices and their background in and knowledge of instructional pedagogies, design, and standards?
What are the reasons of teen use or non-use of the teen section of the Monmouth Public Library website?
Does Facebook marketing contribute to library program attendance?
Why are the communities of Latino/Hispanic parents not aware that public libraries offer bilingual children’s books?
What is the awareness of grandparents raising grandchildren regarding public services in the Boise, Idaho area?
What services, collections, and programs are libraries offering to better serve patrons with autism spectrum disorder (ASD)? Are library staff members receiving ASD awareness training?
What marketing practices are currently employed by public libraries to promote their digital information service?
What factors do college students consider when choosing to read a book for pleasure?
How has training in assistive resources, or lack thereof, of Maricopa County Librarians impacted their ability to assist patrons with disabilities?
Do information literacy (IL) courses provided by the public library during school visits improve teenage patrons’ ability to assess and utilize the information from the public library’s digital resources?
What information resources do teenagers use when visiting the public library? How satisfied are teens with the results they received while interacting with those information sources?
What factors influence the participation of ethnic communities during public library story times?
How satisfied are San Diego Public Library patrons with the library’s job-seeking resources?
How efficiently are the services and information resources provided by the Chicago Public Library system satisfying the needs and concerns of the lesbian, gay, bisexual, transgender and queer/or questioning community in Chicago, Illinois?
What diabetes resources are a necessary part of a health library collection in order to best serve the Hispanic population to manage diabetes?
What are public library staff member’s opinions and levels of satisfaction with the following aspects of the Symphony software interface: aesthetics and functionality?

Setting the research agenda

A few weeks ago I gave a talk at SJSU Gateway PhD students’ virtual residency about how to set the research agenda. To prepare for that talk, I looked back in the past 11 years and thought about how I have been planning, conducting and disseminating my research since I was a doctoral student. I was able to summarize a few useful (hopefully) tips from my experience and share them with our PhD students. I’m posting them here too.

1. Set aside blocks of time designated for “research thinking”. I often do my “thinking” while I’m cleaning the house or cooking – such thinking could be about anything related to research (e.g. ideas for the next research project, how to interpret the data in a completed project and what arguments to make, etc.)

2. Reading favorite journals/blogs/Websites to keep up to date with research developments in one’s field.

3. Record/organize all research-related thoughts/ideas for later consideration – I use Evernote, but a traditional notebook will do too.

4. Join/form a journal club with colleagues/students – exchanging critical evaluation of the published literature with my peers is intellectually stimulating.

5. Serve on the editorial board or as a peer reviewer for journals/grants/conferences – this is a great opportunity to see the most recent research in one’s field.

6. When attending conferences/workshops, it is worth paying attention to work-in-progress presentations.

7. If interested in a research topic, it would be helpful to teach a graduate seminar course on that. Teaching and research are mutually beneficial.

8. Be flexible – set long term goals and short term objectives, and conduct elf-examination at regular intervals and adjust accordingly.

9. Develop a writing routine – this is hard and takes much self-discipline.

10. Quantify the publication targets – for example, every year aim for X number of journal articles, X number of conference presentations, etc.

11. Identify 3-5 journals or conferences for research dissemination – be familiar with their topical coverage, editorial style, submission guidelines and everything else that needs to be noted.

12. Collaborate – potential collaborators may be colleagues, students, researchers connected at conferences or even from other disciplines.

13. Enhance research profile via various online venues – social media (e.g. blog/tweet about your research, create a presence on ResearchGate, academia.edu or Google Scholar), institutional repository, publisher’s marketing tools

14. Be active and be part of the ongoing scholarly conversation/debate.

Formulating the research question

Last week, students in our SJSU Gateway PhD program gathered in San Jose for a one-week residency. This year we have 9 incoming students, and it’s great to chat with them about their research interests and share their passion and excitement of embarking on a new journey. I have two new students this year – one of them is interested in study the relevance of information literacy skills, and another one seeks to better understand the role of medical librarians in improving the public’s health literacy. We spent quite a bit of time talking about developing their research questions in the past week.

The formulation of research question, is probably the utmost important step in a research process. It anchors the entire research study. Yet I have seen too many studies that proceeded without a clearly defined research question. I always ask students to think about what they want to achieve with their study , what is the research problem they hope to resolve, and what their research objectives are. Answers to these questions are helpful for us to understand what exactly it is we try to study, and hence formulate a research question to properly reflect that. Once we have the research question, we can then move on to operationally define each concept and variables in the question, and then start thinking about research design and methodology.

In the many research methods textbooks I have read, research question formulation and development is a topic that’s often lacking. One book that addresses it relatively well is Babbie’s “The Practice of Social Research” (I wrote a blog post about it). And that’s one of the reasons I choose it to be the textbook for my research methods class. The new semester is starting in just a couple of weeks, and in the fall, I will be teaching a new research methods course that focuses on survey research. One of the course assignments is to complete a proposal for a research study that’s appropriate for survey research. Thus, students will need to come up with a research question that has to be answered by a survey study. I wonder how that will impact the question development process. Will make it harder or easier? I think it’s the latter. We’ll see.

2014 Institute for Research Design in Librarianship

In the last two weeks of June, the first IRDL cohort gathered in the beautiful library of the beautiful Loyola Marymount University and went through an intensive professional development program focusing on research design and methodology (see my previous post about this program). As a research methods geek, I was thrilled and honored to be one of the instructors there. I was impressed with the IRDL scholars’ motivation, interest and enthusiasm in conducting quality research and enhancing the rigor of LIS research and publications. Their research topics covered a variety of LIS practices, including the quickly emerging area of data librarianship. During my first two days there, I got to talk with all four members of the cohort whose topics were related to data management, data literacy and data service. Our conversations were illuminating and gave me a lot to think about in terms of preparing out students for this area. Among the twenty-five scholars, four were SJSU SLIS alum – one of them took the Reference and Information Services course with me, and another one was my advisee. It’s great to see them at IRDL and chat with them about their experience at SLIS. I’m so proud of what they have accomplished and what they will be accomplishing after IRDL.

There so many memorable moments throughout the program. The highlight was the scholars’ presentations on the last day, where they talked about their research idea and how it had taken shape during the two weeks. It’s very rewarding to see that they are leaving IRDL with a fairly concrete research plan and ready to dive in and churn out peer-reviewed articles one after another. Hopefully by the end of the three-year IRDL grant, we will be able to compile a list of publications resulting from the scholars’ research projects at IRDL.

In addition to the great discussions and interactions with the IRDL scholars and my co-instructors Greg and Michael, I really appreciated how well LMU library had taken care of us – the classroom was spacious with a great view, the catering service was wonderful, and our host Kris, Marie and Carol were so thoughtful and warm. I can’t thank them enough for making this experience a great memory of summer 2014. I’m already looking forward to 2015!

Oh, before I forget, I tweeted more in the past two weeks than I did in the previous five years, and I think I will keep tweeting at that frequency. Social media, I’m serious about you now.

Student research ideas from fall 2013 Research Methods class

I have finally finished all my grading. It’s been a joy reading all the research proposals my students developed in the Research Methods class. Here are their research questions:

What are the evidence gathering activities and types of research used in special libraries for decision-making?
What is the need among librarians, and library workers for assessment tools to identify the underlying values of technology tools and cost of digital participation for patrons?
Is user satisfaction effected by the accuracy of the content provided by the Digital Humanities Project?
Are the Chinese senior patrons who were foreign born and immigrated to the U.S. as an adult or senior, satisfied with the Chinatown branch’s bilingual information services and Chinese seniors’ lifelong learning programs?
Is social media an effective marketing tool for small branch libraries?
What is the impact of makerspaces in libraries on library traffic and the circulation of materials?
How do project resources and participant community affect citizen science activity?
How does the inclusion of tablets in storytime affect children’s library usage?
How do video game programs impact teenagers’ actual use of library services in San Francisco Public Library?
How do individuals newly diagnosed with multiple sclerosis perceive the usefulness of the information resources available to them in choosing a disease-modifying medication?
Does the availability of research guides in a non-academic health science library have a positive relationship with user satisfaction with search results on the library’s website?
To what extent are the branches of the San Diego Public Library collecting racially diverse young adult novels?
What are the elements in library design and services that lead to improved use for library users and non-users?
Does the addition of comic books and graphic novels to a library’s collection lead to increased patron usage of other library materials and library services in public libraries?
What are the specific challenges archivists encounter saving digital audio materials?
How do users’ usage rates of virtual reference services compare with usage rates of traditional reference services at public libraries, and what are the reasons behind the use/non-use of such services?
Are Web 2.0 tools, used in academic libraries’ information literacy instruction, effective in aiding the students to improve their information literacy skills?
How effective and accurate are librarians using the chat reference service at the San Francisco Public Library in providing information to patrons on sexual health questions?
What is the extent of Aboriginal groups involved in digital preservation of indigenous knowledge activities in preparing, adopting, and presenting ‘protocols’, or ‘codes of ethics’?
How effective is the San Francisco Public Library main location’s Teen Center at bringing teens into the library to participate in events and use the libraries resources?
What interactive applications do users prefer in a women’s ministry website?

Quite a variety, aren’t they? I find this topic particularly interesting – “What is the need among librarians, and library workers for assessment tools to identify the underlying values of technology tools and cost of digital participation for patrons”. The student writes in her proposal:

“Emerging among the different types of literacies now vital for navigating through our information environment is a type of literacy that concerns technology and information ethics, where a user of technology, rather than just obtaining skills to use technology, can identify the underlying values that have shaped its design and determine whether becoming a participant in those values would therefore be desirable. This form of literacy assumes that technology does not represent a moral neutral and that, very often in current digital culture, using technology requires an exchange of information for a service or tool. Instead, technology reflects values, whether or not it has been intentionally designed with those values, and there are often costs that must be weighed before participating. The need to address this is growing in importance as we use an increasing amount of technology tools daily with little to no assessment. A lack of assessment leaves very important questions unasked: what values are supported in the use of a technology tool? How is a participant’s use of a technology being monitored and why? What is the participant’s information used for? Who owns this information when the participant that created it does not?”

Technology and information ethics is indeed an increasingly important topic, and I wonder what role our library and information professionals can play in enhancing people’s awareness of this issue. Looking forward to more research about it.

Books about Research Methods (5)

I have been thinking about developing a research methods course focusing on the survey method. Survey is a very old research technique. According to Earl Babbie’s “The Practice of Social Research”, the use of survey could be traced to the Old Testament – “After the plague the Lord said to Moses and to Eleazar the son of Aaron, the priest, ‘Take a census of all the congregation of the people of Israel, from twenty old and upward’” (Numbers 26: 1-2). Surveys are mainly used in studies that have individual people as the units of analysis, and to collect original data for describing a population too large to observe directly. In LIS research, survey is one of the most frequently used research methods. For example, survey research can be conducted to study user satisfaction with reference service, to assess student learning in information literacy instruction, to examine user awareness of mobile services provided by the library, etc.

To develop a course on survey research, the first thing is to find a textbook. I looked around and decided on “Internet, mail, and mixed-method surveys – the tailored design method”, by Don Dillman, Jolene Smyth and Leah Christian. It’s a comprehensive guide to designing and administering surveys. I particularly like how they talk about constructing survey questions – this is the most critical element in questionnaire design. I like their writing style too – straightforward and easy to understand. The only regret is that they did not include much discussion of analyzing survey responses. It’s understandable though – data analysis in survey research is a huge topic that warrants a book of its own. But I guess this means I will have to find other readings for this topic then. Time to dive back into the literature!

Use of incentives in social research

Recently I have had some interesting discussions with my students in the Research Methods class about the use of incentives in research studies. We all know that incentives help recruit and retain study participants, but some people consider them a form of undue influence or corruption of judgment, which may lead to biased data and damage the validity of the study. This makes me wonder, in what circumstances incentives are just innocuous incentives, and in what circumstances they morph into a temptation that attracts people to participate in a study against their better judgment?

I heard a program on the radio the other day, and it was about Amazon.com’s exclusive invitation-only club called Vine. Members of Vine are the top-reviewers of Amazon, and Amazon sends them freebies ranging from earbuds to color printers. A professor from NYU believes that such a program would lead to biased product reviews, because “as humans we are hard-wired to give in to this sort of, you know, enticement where if you continuously get things for free, then you’re more likely to be biased positively than biased negatively”. However, a spokesperson from Amazon claimed that Vine reviewers are not positively biased – they actually give lower star ratings that the average reviewers on the site. Her theory is that “it’s because they take that role so seriously to give as much sort of unbiased perspective on reviewing that product”.

This was quite an interesting story. So I dug around a bit more to find some scholarly literature about using incentives in research studies, and stumbled upon this article “Ethics in Human Subjects Research: Do Incentives Matter?” The authors stated “incentives become problematic when conjoined with the following factors, singly or in combination with one another: where the subject is in a dependency relationship with the researcher, where the risks are particularly high, where the research is degrading, where the participant will only consent if the incentive is relatively large because the participant’s aversion to the study is strong, and where the aversion is a principled one”.

This article is quite helpful to inform decisions concerning incentives. But we first have to be able to offer incentives before we consider how to offer them. I guess this means I have to get back to grant-writing now. 🙂