Using ChatGPT in Research Design

In the past two weeks, we completed the research methods training portion of 2023 IRDL Online. IRDL is a continuing education program for academic and research librarians designed to create a growing community of confident librarian-researchers. The research methods training portion was provided in the form of an online summer research workshop, where I was an instructor. It’s incredibly energizing and exciting to talk to a group of research-minded librarians about conducting research to improve professional practice. One theme that arose from our discussions was the use of AI in research design. One of the librarian scholars gave it a try and asked ChatGPT to write survey questions for a research scenario.

Here’s her prompt: I need to write a survey to find out how academic librarians engage in professional development. ACRL will use the survey findings to develop the Guidelines for Professional Development for Academic Librarians, which they believe will provide guidance for academic librarians as they explore professional development opportunities and make professional development plans. Can you write 4 questions for this survey?

ChatGPT’s response:
Certainly! Here are four sample questions for your survey on how academic librarians engage in professional development:

1. How frequently do you engage in professional development activities related to your role as an academic librarian?
• Rarely or never
• Occasionally (1-2 times per year)
• Regularly (3-5 times per year)
• Frequently (more than 5 times per year)

2. What types of professional development activities do you find most beneficial in enhancing your skills and knowledge as an academic librarian? (Select all that apply)
• Workshops and conferences
• Webinars and online courses
• Professional association memberships and events
• Formal education (e.g., degree programs)
• Informal learning (e.g., self-study, reading journals)
• Mentoring and coaching programs
• Other (please specify)

3. How do you typically seek out information about professional development opportunities? (Select all that apply)
• Internal communication channels (e.g., email, intranet)
• External websites and online resources
• Professional networks and colleagues
• Professional association publications and newsletters
• Supervisors and managers
• Other (please specify)

4. What factors influence your decision to participate in a professional development activity? (Select all that apply)
• Relevance to current job responsibilities and career goals
• Availability of funding or financial support
• Recommendations from colleagues or supervisors
• Reputation and credibility of the provider or organization
• Flexibility in terms of time and scheduling
• Recognition or credentialing opportunities
• Other (please specify)

Looking at these survey questions, I think ChatGPT did a nice job in operationalizing the key concepts in the research question based on the research goal. There’s room for improvement in wording survey questions that uphold the basic rules for survey design such as being concise and unambiguous, avoiding leading questions or double barreled questions, and making sure response categories are exhaustive and mutually exclusive. Still, this can be a good start if we are get stuck in our instrument design and don’t know where to begin.

There’s been discussion about using AI in manuscript writing such as this Science article “As scientists explore AI-written text, journals hammer out policies“. Looks like we will need more discussion about using AI in assisting the research process such as designing survey questionnaire, interview guide or a content analysis codebook.

My new book – “Enhancing Library and Information Research Skills: A Guide for Academic Librarians”

My two wonderful colleagues, Kris
Brancolini and Marie Kennedy, and I wrote a book together based on our
experience with the Institute for Research
Design in Librarianship (IRDL), an IMLS-funded program that provides
professional development opportunities for academic librarians to improve their
research skills. The book is titled “Enhancing Library and Information Research
Skills: A Guide for Academic Librarians”, in which we covered the whole
spectrum of being a practitioner-researcher in the academic library setting.

It was a great experience working with
Kris and Marie on this book. They both are strong advocates for academic
librarians’ engagement in research and use of research evidence to inform
decision making. We truly hope that this book will help academic librarians
around the nation to become more aware of the value of research to academic
librarianship, develop a solid understanding of the research process, and ultimately
improve their confidence and competency in conducting and applying research in
their professional practice.

IRDL 2017 and Interview as a research method

Last week I went to the beautiful campus
of Loyola Marymount University for IRDL 2017. I’m glad that this wonderful
research methods training program received funding to continue for another
three years – more academic librarians will benefit from it and gain important
skills to become more competent and confident practitioner researchers. I had
great conversations with the participants this year. They were all working on
interesting projects – e.g. how ARL libraries design their fundraising page on
the library website, how students understand or misunderstand library jargon,
how students’ interpretation of research differs from faculty’s expectation and
observation of their research behavior, just to name a few. I can’t wait to
read more about these great projects in journal publications.

In the past week, I taught during the
day, and worked on other projects at night (yes, it’s a productive week for
me!). Particularly, I took full advantage of LMU library’s subscription to SAGE
Research Methods, and read a number of articles about the research method –
interview. There are different ways to use interview in gathering qualitative
data. Here’s a summary of them from the SAGE Encyclopedia of Qualitative
Research:

Convergent
interviewing – a technique that aims to collect, analyze, and interpret
people’s experiences, opinions, attitudes, beliefs, and knowledge that converge
around a set of interviews. It was created primarily to address issues in under-researched
areas. It permits in-depth interviewing by promoting a cyclical research
process that requires ongoing analysis as part of the overall strategy. Interviewers
engage in a constant comparative reflexive process that permits detailed rich
content and theoretical sampling as researchers seek to continuously test
emerging interpretations from early interviews in subsequent interviews.
Cognitive
interviewing – it encompasses a variety of approaches for eliciting qualitative
data on how participants interpret and respond to a wide variety of situations.
Cognitive interviewing increasingly is used in the evaluation of technology
interfaces such as websites and tools for informatics. It is used in education
to understand how students think about content and respond to test items and in
marketing to understand how to evaluate products better.
Conversational
interviewing – an approach used by research interviewers to generate verbal data
through talking about specified topics with research participants in an
informal and conversational way. Interviewers and interviewees rely on
taken-for-granted assumptions about how everyday talk occurs and how speakers
make meaning of one another’s utterances. In emphasizing features of mundane conversation,
conversational interviewers strive to facilitate a research environment in
which participants feel free to participate in extended discussions of research
topics in a less hierarchical environment than that convened in structured
interview settings.
Narrative interview
– an interview that is organized to facilitate the development of a text that can
be interpreted through narrative analysis. Narrative analysis is guided by a
theory of narrative, and these theories of narrative vary in the influence of
the reader, the text, and the intent of the author on interpretation. For this
reason, the content and structure of a narrative interview will depend both on
the theory of narrative being used in the analysis and on the research
question.
Interactive interviewing
– an interpretive practice for getting
an in-depth and intimate understanding of people’s experiences with emotionally
charged and sensitive topics such as childbirth, illness, loss, and eating
disorders. Emphasizing the communicative and joint sense-making that occurs in
interviewing, this approach involves the sharing of personal and social
experiences of both respondents and researchers, who tell (and sometimes write)
their stories in the context of a developing relationship.
The critical
incident technique (CIT) – originally, it refers to a set of procedures to
collect direct observations of human behaviors in a way that facilitates their
use in solving practical problems and developing broad psychological
principles. Over the years, it has been increasingly applied to studying psychological
states or experiences, and emphasis has shifted from direct observation by
experts to retrospective self-report in interviews.
Co-constructed
narratives – this refers to stories jointly constructed by relational partners
about epiphanies in their lives. This approach offers a way for participants to
actively construct a version of a relational event that provides insight,
understanding, and an in-depth and complex reflection on what occurred. As
such, this mode of doing research provides an alternative to traditional interviewing,
especially when the topic under consideration is emotionally charged, personal,
and sensitive.

Creative Research Methods for LIS Research II.

In library research, an important source
of data is input from the community of library users. How are users using
library collections/services/facilities? How satisfied are they with their
library experience? What are the inadequacies in meeting their needs? User
input is critical in helping library professionals understand how successful
the library is in serving the user community. Traditionally, librarians gather
user input through survey, in-depth interview and focus group. To make the
research process more engaging to both the researcher and library users, they
may consider more creative alternatives, such as the photovoice method. Through
the utilization of photographs and stories, this method allows community
members to identify and represent issues of importance to them, enabling policy
makers to develop a greater understanding of the issues, and formulate
effective and comprehensive strategies to address them in a way that is also
meaningful for the community.

Photovoice is a qualitative method often
used in community-based
participatory research (CBPR), where community members take photos related
to a particular issue and tell their stories behind the photos in a facilitated
discussion. Community members are believed to be more imaginative and observant
of community issues than the most experienced photographers and photo
journalists. In library and information science (LIS), photovoice is still
relatively new and not widely used. Recently I wrote an article to review this
method and discuss its implications for LIS research. The article is published
in Library Hi Tech, titled “Photovoice:
A Creative Method to Engage Library User Community”.

Since Photovoice is a CBPR method, it is
often applied in action-oriented projects, where a partnership among the key
stakeholders is established and all partners are equitably involved in the
process with the aim of combining knowledge and action for community
improvement . The key stakeholders include researchers, community members and
decision makers, and they participate in the photovoice process collaboratively
to explore community issues and produce an action plan to address them. However,
photovoice projects are known for being time consuming as they usually extend
over several weeks, which can be burdensome for some community participants and
lead to high attrition rates. The cost related to photography and exhibition
can be a concern too. Without community members’ sustained commitment and
adequate funding, it is unlikely for the photovoice project to succeed.

Photovoice provides an engaging way for
libraries to examine user needs, perceptions or behavior in order to provide
better services to them. Combining visuals and narratives, this method may help
capture a nuanced view of topics that are sensitive, vague or difficult to
articulate verbally, and thus may generate data unexpected when using
interviews or focus groups. The collaborative nature of the photovoice method
allows community members take on the role of co-researchers as they participate
in photovoice projects. It provides an empowering mechanism for library users
to become actively engaged in issues pertinent to the user community and
advocate for their concerns, needs and desires. With active user involvement,
library outreach efforts are more likely to be successful.

Methods in Information Literacy Research – ECIL 2016 Reflections

A few weeks ago I attended the European Conference on Information Literacy (ECIL)
in Prague. It was my first time attending this conference and it was a great
experience. Six people (students, alum and faculty) from our Gateway PhD
program were there and it was fabulous to see our program so well represented
at this conference.

The conference covered a wide
variety of information literacy related topics, and what interested me most
were the presentations about methods in information literacy research. Here’s a
summary of them

Autoethnography: Research as
Reflection, Inclusion and Empowerment (by Deitering, Anne-Marie; Schroeder,
Robert; Stoddart, Richard)

The three presenters were involved
in a project where a learning community of librarians developed their
autoethnographies. The provided an overview of autoethnography and discussed
their experience and reflections on this project. It was quite refreshing to me
as because in authoethnography the
researcher is the subject of research. I found an article that provided a detailed
explanation of this method – here’s the link http://nsuworks.nova.edu/tqr/vol19/iss37/3/.

Using Phenomenographic Methods to
Support Political Information Use (by Smith, Lauren)

There are quite a few
phenomenographers in information literacy research, including some of my
colleagues in our Gateway PhD program. So I know a little bit about this
method. The presenter talked about her study using this method to examine how
high school students are aware of, acquire, engage with and apply political
information. It’s a comprehensive study and well designed. Her presentation
slides can be located here.

Critical Incident Technique in
Information Literacy Research in the XXI Century (by Cisek, Sabina Barbara)

The presenter shared her review
and analysis of studies that used Critical Incident technique in information
literacy research. This method has not been widely applied to studying
information literacy related topics. It’s more popular in examining information
seeking behavior. Here’s a brief
definition of this method “The critical incident technique consists of a set of
procedures for collecting direct observations of human behavior in such a way
as to facilitate their potential usefulness in solving practical problems and
developing broad psychological principles. The critical incident technique
outlines procedures for collecting observed incidents having special significance
and meeting systematically defined criteria” (Flanagan,
1954). The presentation slides can be located here.

Creative Research Methods for LIS Research I.

Recently I have been reading “Creative Research Methods in the Social Sciences: A Practical Guide” by Helen Kara. It’s refreshing to read about the various approaches in arts-based research, research using technology, mixed-methods research and trans formative research. It made me think how these methods can be applied in LIS research.

One of the methods is vignettes, which I happened to co-write a paper about with two wonderful librarians, Allison Benedetti and John Jackson. Here’s an excerpt from our paper:

“Vignettes are short
stories about hypothetical characters in hypothetical circumstances, to whose
situation the interviewee is invited to respond. As a
methodological tool, vignettes can be used in focus group interviews, in-depth
interviews, or survey interviews, where the interviewee is invited to draw upon
their own experience, and provide perceptions, opinions, beliefs, attitudes,
and diagnostic predictions about how the fictional character in the vignette
will behave. When
observing or placing an individual in a particular context would not be
possible for logistical or ethical reasons, vignettes are often considered. Approximating a real-world situation, vignettes
allow for features of the context to be specified so that the interviewee can
make normative statements about a set of social circumstances rather than
provide their responses in a vacuum.

Vignettes
are often presented as a written narrative that the interviewee can read. Vignettes
must carry sufficient detail to allow the interviewee to visualize the
hypothetical circumstances as an actual situation. Particularly, the
situational elements of a vignette need to be carefully specified and the main
characters in the vignette are usually given names. Following each vignette,
the researcher may ask an open-ended question or a closed question with a set
of response categories from which the interviewee can choose. Probes are used
when necessary for the interviewee to elaborate on their responses.

In LIS research, vignettes have not been widely used. Given that vignettes
are helpful to depersonalize sensitive topics and encourage respondents to talk
more openly, they could be potentially useful for researching the attitudes and
behaviors of scholars related to publishing, copyright, open access practices,
and practically any area in which libraries are curious about the behaviors and
habits of users. Vignettes could also be used with students to teach about or
demonstrate concepts related to plagiarism and academic integrity. User
experience and web design practices have long been utilizing personas or
archetypes to evaluate designs and functionality of interfaces; there may be a
place for vignettes to augment these practices, perhaps with remote or online
usability studies.”

I would like to continue explore the use of creative methods in LIS research. Writing a blog series about this would help me document my exploration. So this post would be the first in this series. The next method I hope to explore is photovoice. A member of the 2016 IRDL cohort is considering using it. How great is that!

Research-based Library Practices Summit in Doha, Qatar

Last week I had the pleasure of
going to Doha, Qatar to give a workshop at the Research-based Library Practices
Summit. The Summit was organized by librarians from University College London
and Carnegie Mellon University in Qatar and attended by librarians from
multiple countries in the Gulf region. My workshop focused on providing an
overview of the frequently used quantitative and qualitative research methods
in library and information science such as survey, focus group interview,
in-depth interview and content analysis, and explaining the process of making a
methodological decision based on the research question. The objective of the
workshop is for participants to gain knowledge and skills to determine which
qualitative and quantitative methods are most appropriate to address their
research questions. I used the following questions to help them analyze their research
question and make the methodological decision:

·
Are qualitative research methods appropriate for
your research question?

a.
How much previous research exists on your topic?

b.
Do you intend to capture a nuanced, in-depth
view of your topic?

c.
Do you intend to describe your observations via
the identification of themes/patterns?

d.
Is there existing qualitative content you can
analyze, or do you have to collect original qualitative data?

e.
Which qualitative data collection method do you
plan to use – focus group interview, in-depth interview or field observations?

·
Are quantitative research methods appropriate
for your research question?

a.
Is your research question a relationship
question or a causality question?

b.
Has there been extensive research conducted on
your topic that the range of Reponses or variations of behavior are already
known?

c.
Do you intend to quantify your observations and produce
generalizable conclusions?

d.
Is there existing statistics you can analyze, or
do you have to collect original quantitative data?

e.
Which quantitative data collection method do you
plan to use – quantitative content analysis, quantitative observations, or
survey?

Overall the workshop went pretty
well. We had a lot of good discussions. I was impressed with the participants’
eagerness to conduct research to improve their practice. This kind of “research-mindedness”
is exactly what we need in the profession.

During my stay in Doha, I had the
opportunity to visit the state-of-the-art Qatar National Library. It’s still
under construction, so we had to wear safety hat and vest on the tour. It’s an
amazing building. I loved the open design and the high level of automation.
There is even a restaurant in the library – a real one with a kitchen, not just
a café where you can only get sandwiches. Unfortunately we weren’t allowed to
take pictures inside. But here’s a picture of us in front of the building.

Here are a couple of pictures I
took of a gorgeous corridor in the Education City, where the Summit was taking
place. Gorgeous view, isn’t it?

Quantitative research workshop

Quantitative research has always
been a challenging topic in my teaching of research methods. Quite a number of
my students are innately disinclined to deal with numbers and find statistical
analysis uninteresting and even unnerving. So every time I cover quantitative research,
I always start with this quote from Earl Babbie’s The Practice of Social
Research:

““Empirical research is first and
foremost a logical rather than a mathematical operation. Mathematics is merely
a convenient and efficient language for accomplishing the logical operations inherent
in quantitative data analysis. This textbook is not intended to teach you
statistics or torture you with them. Rather, I want to sketch out a logical
context within which you might learn and understand statistics.”

It is important that students understand
statistics are just a means to an end – a tool that we can use to help us accomplish
our research objective by making sense of quantitative data. In my teaching, I
focus primarily on the conceptual understanding of statistics. Students are
expected to master what the frequently used statistical measures are, when to
use them, for what types of variables and what kinds of analytic objectives, and
don’t have to worry about the computational process. They can explore that on
their own – there are many tutorials on YouTube.

At this year’s SCELC Research Day, I gave a
workshop on quantitative data analysis, and this “conceptual understanding”
strategy seemed to have worked well. We had interesting discussions around the
tables and it’s great to see how the librarians planned to use the statistics
covered in the workshop in the analysis of their existing data sets. I just
wish there were more time for us to do some hands on exercises. Nonetheless, it
was a great experience talking about research methods with librarians, as always.

Research methods in library assessment

I gave a workshop about research
methods in library assessment at Santa Clara University Library earlier this
month. It was a wonderful opportunity to connect with SCU librarians. The
workshop focused on two things: 1) how to develop the research question in a
library assessment project; and 2) how to determine the proper research methods
for conducting the assessment. Research question develop is the critical first
step in any research project – Albert Einstein once said “If I had an hour to
solve a problem I’d spend 55 minutes thinking about the problem and 5 minutes
thinking about solutions.” In the workshop, we first talked about the progressive
focusing process of formulating the research question (overall goal –>
objectives –> research questions), and discussed the criteria we can use to
evaluate the quality of a research question. ACRL’s 2015 January Assessment in
Action Report titled “ACADEMIC LIBRARY CONTRIBUTIONS TO STUDENT SUCCESS:
Documented Practices from the Field” provided some examples of research
questions in library assessment, and we spent some time examining each one of
them (see the picture below) – it was a great way to look closely at the process of developing research
questions. Then we reviewed the major research designs (based on the analytic
objective, time dimension, and degree of variable manipulation) and the frequently
used qualitative and quantitative methods in library assessment. Quantitative
methods seem to be more popular in assessment projects.

I’m glad to have this opportunity
to chat with SCU librarians about research methods – I admire their initiative
to engage in library assessment using valid and reliable research methods. This
workshop made me think about, as LIS research methods instructors, what we can
do to work more closely with libraries and librarians that conduct assessment.
Looks like I need a lunch date with some of my fellow methods instructors and
my assessment librarian friends. 🙂

Research topics from my research methods class in fall 2014

My research methods class in fall 2015 focused on research. Students in the class were expected to develop a survey research proposal. Here’s a collection of their survey study topics. Many of them are needs assessment and program/service evaluation studies.

Does attending the “Sopa de letras” workshop series for parents cause an increase in behaviors that are considered beneficial for literacy and learning of children?
What programs would college bound teens like to see the library incorporate into its services in order to help them with college preparation?
Do series-based library workshops improve the job skill sets of unemployed or low-income adults?
Why are patrons using or not using the library databases?
What kinds of professional development opportunities should the library offer to help teachers better understand the services the library could offer in relation to the Common Core State Standards?
What services (staff preparedness, physical layout), collections and programs do public libraries in the state of California offer their visually impaired patrons?
What is the effectiveness of the Mill Valley Public Library’s current marketing tactics in their efforts to market library programs and events?
At what rates do elementary school librarians and teachers use digital audio players and media for audio reinforcement over physical audio players and media, or vice versa, and why?
What about the existing library teen services/collection/spaces/programs do KDBS students think is adequate, inadequate, and needs improvement?
Does the pedagogical approach of Reading Workshop impact the range of genre selection for independent reading for upper elementary students (3rd-5th grade) from their school library?
How do musicians perceive the usability of the intrinsic qualities of digital scores?
What are the reasons for immigrant populations’ non-use of the library?
Do adults have interest in attending arts classes at the Cambridge Public Library?
What genre/types of books are young adults most interested in?
Does creating a young adult space in a library enhance the overall library experience for people in this age group?
Do Spanish-speaking patrons feel their public libraries provide materials and programming that reflect their needs and interest of their communities?
What specialized accommodations for existing services do higher education academic libraries provide for students with auditory and visual disabilities?
Is there a relationship between academic librarians’ instructional practices and their background in and knowledge of instructional pedagogies, design, and standards?
What are the reasons of teen use or non-use of the teen section of the Monmouth Public Library website?
Does Facebook marketing contribute to library program attendance?
Why are the communities of Latino/Hispanic parents not aware that public libraries offer bilingual children’s books?
What is the awareness of grandparents raising grandchildren regarding public services in the Boise, Idaho area?
What services, collections, and programs are libraries offering to better serve patrons with autism spectrum disorder (ASD)? Are library staff members receiving ASD awareness training?
What marketing practices are currently employed by public libraries to promote their digital information service?
What factors do college students consider when choosing to read a book for pleasure?
How has training in assistive resources, or lack thereof, of Maricopa County Librarians impacted their ability to assist patrons with disabilities?
Do information literacy (IL) courses provided by the public library during school visits improve teenage patrons’ ability to assess and utilize the information from the public library’s digital resources?
What information resources do teenagers use when visiting the public library? How satisfied are teens with the results they received while interacting with those information sources?
What factors influence the participation of ethnic communities during public library story times?
How satisfied are San Diego Public Library patrons with the library’s job-seeking resources?
How efficiently are the services and information resources provided by the Chicago Public Library system satisfying the needs and concerns of the lesbian, gay, bisexual, transgender and queer/or questioning community in Chicago, Illinois?
What diabetes resources are a necessary part of a health library collection in order to best serve the Hispanic population to manage diabetes?
What are public library staff member’s opinions and levels of satisfaction with the following aspects of the Symphony software interface: aesthetics and functionality?