Assessment
The MIT Office of Graduate Education (OGE) offers a range of programs, services, and funding to make graduate education at MIT empowering, exciting, holistic, and transformative. In support of this mission, OGE has developed and experimented with diverse professional development (PD) offerings to support communication skills for graduate students – a high priority need identified by the Task Force on Graduate Student Professional Development in 2013.
MIT’s Teaching & Learning Lab (TLL) and educational research company Applied Curiosity Research (ACR) conducted formative research to support this effort, utilizing mixed-methods to identify and understand promising strategies and potential challenges to providing relevant and engaging content to students. This memo* outlines the research findings obtained during successive studies that evaluated the various products (outlined above) offered by Gradcommx.
*Memo available in the series of documents linked below.
Relevant Documents The following documents were key during the development of Gradcommx
TFPRO
Gradcommx Research Memo
MOOC Beta Test Report
Technically Speaking Comic Series Educational Assessment
Overview of Findings
The comics were highly appealing and engaging as a way to practice and review communication skills and may be a useful way to differentiate GradCommX (see Section IV for more details). Despite this, careful distribution and outreach will be required to reach students. Consider existing, relevant events and workshops at which to distribute the comics, including: student orientation, departmental retreats, TA trainings, and student or departmental research presentations. To the extent possible, call attention to the comics at these events to differentiate from other materials.
Overall, evidence suggests the comics are a clear, accessible, and engaging means of supporting students’ communication skills. Students consistently reported the comics provided an easy-to-read review or reminder of critical information that can support their professional development. While the comics were perceived as less useful for Sloan students that receive abundant departmental support and training, students from other departments believed they filled a need for promoting effective communication skills.
Responses suggest that distribution should be aligned to events or times of year when communication is most relevant for students to maximize reach and effectiveness, such as TA trainings, orientation, and departmental events, or dissertation defense and presentation season. Students did not believe passive distribution strategies would find an audience given the volume of available resources at MIT.
Recomendations
Consider the following recommendations to increase the usability of the comics and expand the reach to grad students.
Testimonials
[themify_icon icon=fa-quote-right icon_bg=”#111″ icon_color=”#fff” style=”large”]
Comic Group 1:
“I was actually surprised by how captivating it was.”
[themify_icon icon=fa-quote-right icon_bg=”#111″ icon_color=”#fff” style=”large”]
Comic Group 2:
“[The comics] induced this fun element that is usually sucked out of all the academic activities you have to do on a daily basis.”
Gradcommx Live Educational Assessment
Overview Of Findings
The course was highly appealing and engaging and there was evidence of impact on student’s knowledge and attitudes in both the professor- and TA-led courses (see Gradcommx Live 1.0 & 2.o sections for details). A few findings are worth highlighting in this overview:
- The 2-week, 12-hour course format was perceived as an effective mix of convenient and focused;
- Students perceived face-to-face practice, rapid feedback and iteration, and access to the professor as the most salient and effective course elements;
- When TAs lead the course, more transparent communication about their roles and expectations should be provided to students;
- Despite the appeal and evidence of impact, recruitment was challenging and may require a more sustained and strategic effort.
Gradcommx Live 1.0
The live GradcommX class is an adaptation of the undergraduate class 6.UAT Oral Communication offered the in Electrical Engineering and Computer Science (EECS) department. The GradcommX version was modified to target the types of oral presentations most often given by graduate students and early career scientists and condensed into six two-hour sessions scheduled over a two-week period. Each session began with a group improvisation activity designed to: a) help students loosen up; b) gain confidence performing in front of their classmates; and c) practice relevant public speaking techniques. Sessions also included brief lectures and videos as well as reading comics that featured key concepts of giving a research or technical presentation. Much of the classroom time focused on tailored activities to practice specific presentation skills, practicing presentations of their research, and giving and receiving feedback. Students also had brief homework assignments to prepare for the following day.
Overall, evidence overwhelmingly supports that GradcommX Live was impactful, engaging, and appealing for participating students. Students demonstrated gains across all measured outcomes, including: 1] general perception of their communication skills; 2] self-reported comfort with communication across select settings; 3] self-reported confidence communicating about their research; and 4] course-specific content knowledge, despite having high scores at pre-test (See Tables 1.1 – 1.3). While this small sample size does not provide generalizable results of efficacy, these findings provide a clear trend that is corroborated by focus group findings.
Students attending the focus group (n=5) compared the course favorably to other MIT professional development experiences, reported high-levels of engagement and enjoyment with the course, reported an intention to recommend the course to other students and use the skills in their lab groups, and provided anecdotes of immediate impact on their professional and academic pursuits.
[themify_icon icon=fa-quote-right icon_bg=”#111″ icon_color=”#fff” style=”large”]
Focus Group Participant
“I’m taking a writing course and the professor actually said, ‘you know what?’ Your writing has improved a lot in the last two weeks. What happened?”’
Further
Considerations:
Gradcommx Live 2.0
GradcommX Live – IAP was a professional development course led by Tony Eng over two weeks during the January 2018 IAP period. The course was similar to GradCommX Live 1.0, but professor Eng supported three teaching assistants (TAs) who played a significant role in the implementation of the IAP class, leading some activities during the first week and serving as primary instructors for much of the second week. Again, participating students (n=7) completed pre-/post-course surveys and four students attended a 60-minute focus group to discuss the salient features of the course.
Overall, feedback suggests the course was highly valued by participants. Although student confidence in communicating their research did not increase significantly, student knowledge of the communication skills taught in the class did show a small but statistically significant increase. The small sample size (n=7) may partially explain the statistically insignificant improvement in confidence.
Students found the TAs to be effective, rating the usefulness of their activities and feedback 4.7 out of 5. The only negative aspect of having TAs take the lead cited by the students was the lack of any introduction or explanation of their role in the class. Both on surveys and in the focus group, students said they wished TAs had been formally introduced to the class. Because they were not introduced and their role was never explained, students were left with the impression that they were just covering for Tony during the second week.
[themify_icon icon=fa-quote-right icon_bg=”#111″ icon_color=”#fff” style=”large”]
Focus Group Participant
“I don’t know what actually happened. But the way it felt to me is that Tony started leading the class. And then all of a sudden, this week he had a commitment from 10 to 12. And that’s why the TAs all of a sudden took over. That’s how it felt for me.”
Further
Considerations:
Gradcommx The Game: Inspace Attack! Educational Assessment
Overview Of Findings
The students who reviewed the online game appreciated the concept and content of the emails and the digital comics. The small number who looked at the additional content in the “Hard Mode” of the game (i.e. suggested assignments with scoring rubrics) indicated it was relevant and useful. However, the email format of the course was not generally appealing for students. It’s difficult to disentangle the general lack of interest in the online game from the timing of the pilot test, as students were asked to play the game and provide feedback during busy end-of-term period nearing finals and thesis submission deadlines. Additional pilot testing, debugging, and revisions will be required to effectively engage students with this format.
In April 2018 students were invited via OGE email to pilot GradcommX The Game: INSPACE ATTACK!, an online professional development program. The game presented the GradcommX Live communication skills content through an email-based, gamified narrative that leads participants through assignments, videos, comics, and other course content. The following is a brief description of the game concept and mechanics:
Game Concept
Using retro/pixelated graphics emulating 1980’s computer games, players are tasked with protecting INSPACE, an area of consciousness dedicated to working out problems, from the attacking Nega-U. Players learn a communications skill in each level needed to defeat one of the monsters from Nega-U’s army: Specifikos, Myopical, Borwal, Cloc, Foci, and finally Nega-U.
Game Mechanics
The game was distributed through a series of automated emails. Once students signed up via the GradX website, MailChimp sent students an email introducing the game. Each subsequent level consisted of an email containing game story text, a comic, and a link to “Hard Mode” – a website with additional assignments, videos, and content. At the end of each email, students clicked a button to indicate they had completed the level, signaling the defeat of the level boss and triggering the next email.
Overall, reactions to the game were mixed. When asked to rate on a scale of one to ten how likely they are to recommend the game to colleagues, the fifteen survey respondents rated a mean of 5.20. Among the 6 respondents who completed more than half of the game levels, the mean rating was 8.33. This may indicate that students who valued the game were more likely to persist or that greater exposure to the game content increased students’ appreciation of it. Additional data from students who complete the game is needed to better understand this finding.
[themify_icon icon=fa-quote-right icon_bg=”#111″ icon_color=”#fff” style=”large”]
Survey Respondent
“The game was great because it made me realize all of the ways I could improve my presentations – not just through the exercises, but the key concepts themselves.”
The greatest barrier to student participation in the pilot was timing. Because the game was launched near the end of the semester, most students were too busy to complete the game. Thus, feedback on the game is limited to a small sample size. However, the data provided by participants, along with data collected from students who expressed an interest in the game but did not participate, offer some useful insights.
Further
Considerations:
The Great Communicators Podcast Educational Assessment
Overview Of Findings
There was little evidence of appeal for the Great Communicators podcasts. The concept of using podcasts to meet OGE’s goals were viewed favorably by many students, and podcasts have potential for reaching students at scale – podcasts continue to grow in popularity, offer flexible consumption to listeners, and are easily disseminated and shared. However, the preferences of podcast consumers (i.e. discerning, motivated more by entertainment than learning) and crowded ecosystem of high-quality, informative podcasts would require OGE to carefully curate the topics, guests, and host to be relevant and useful for students. If resources are available and these assets are framed and aligned to the broader OGE offerings, consider the following for episode creation:
- Interview credible sources who provide insights from personal experience that promote the importance of communication skills;
- Ensure the host, format, and guests are engaging and entertaining, as students are still generally listening to podcasts for entertainment;
- Provide usable, tangible strategies that are aligned or related to the with the main points of the GradCommX class and comics;
- Create an ongoing podcast series to cover a variety of professional development topics such as non-academic career options and entrepreneurial skills.
In general, students participating in two focus groups (n=12) reported the Great Communicators podcasts did not provide enough new information and failed to meet their expectations based on the titles and the speakers. However, many students across the study perceived podcasts as an appealing medium for learning about communication skills. Podcasts provide a compelling opportunity to reach students at scale, but realizing this goal will require careful framing and curation.
Salient Findings
In general, the distribution of podcast listeners in this sample matches available market data, which found 26% of US adults consume podcasts monthly and 16% listen weekly.
Evidence from these groups suggests they listen while performing activities that don’t require 100% focus, like rote household activities (e.g. cooking, cleaning), travel (e.g. commuting, walking), or going to the gym. This again is supported by an annual market analysis conducted by Edison Research which showed 51% of listeners listen outside of their home (www.edisonresearch.com/podcast-research/). In this context, the goal of podcasts needs to be first to entertain, with learning coming as an added bonus.
Listening tastes varied, but focused primarily on a blend of entertainment, distraction, or an excuse to feel a little productive during rote activities.
[themify_icon icon=fa-quote-right icon_bg=”#111″ icon_color=”#fff” style=”large”]
Pod Group 1:
“So I tend to gravitate towards stuff like probably, “Pod Save America” or “United States of Anxiety,” that you learn something, but it’s not that serious. I can tune out and do what else I need to do.”
[themify_icon icon=fa-quote-right icon_bg=”#111″ icon_color=”#fff” style=”large”]
Pod Group 2:
“I feel everyone’s using podcasts as their peripheral vision around things they want to be interested in, but not dive too deeply into.”
Discussion & Recommendations
The current format was not engaging or effective for students, but podcasts could be a useful asset for GradX as part of a PD series. The development of the podcasts need to be carefully considered to meet: 1) discerning tastes of regular listeners; 2) specific perceptions of what podcast are; and 3) common listening patterns. It may be that podcasts are better suited to build a case for communication as a skill that warrants practice, to be redesigned to align to specific GradCommX lessons from the course or the comics, or broadened in an ongoing series to address professional development topics beyond communications.
As an example, the novel concepts presented by Tony during the live course would make an entertaining and useful podcasts. Tony could discuss Why improv games make you a better presenter – you won’t have time to practice each talk, so you’d be better served preparing how to think on your feet so you prepare less later.
Further
Considerations:
More Details
The information listed above is a part of a larger assessment of elements found within Gradcommx.
To access the complete assessment of all elements found within Gradcommx go to: https://oge.mit.edu/about/committees/tfpro/
