header_animate_D3

Assessment

assessment2

The MIT Office of Graduate Education (OGE) offers a range of programs, services, and funding to make graduate education at MIT empowering, exciting, holistic, and transformative. In support of this mission, OGE has developed and experimented with diverse professional development (PD) offerings to support communication skills for graduate students – a high priority need identified by the Task Force on Graduate Student Professional Development in 2013.

MIT’s Teaching & Learning Lab (TLL) and educational research company Applied Curiosity Research (ACR) conducted formative research to support this effort, utilizing mixed-methods to identify and understand promising strategies and potential challenges to providing relevant and engaging content to students. This memo* outlines the research findings obtained during successive studies that evaluated the various products (outlined above) offered by Gradcommx.

*Memo available in the series of documents linked below.

Relevant Documents The following documents were key during the development of Gradcommx

TFPRO

2013 Report from the Task Force on Graduate Student Professional Development

Gradcommx Research Memo

An assessment of the educational efficacy of all Gradcommx elements delivered in the summer of 2018 by the MIT Teaching & Learning Lab and Applied Curiosity Research.
Available upon request.

MOOC Beta Test Report

Available upon request.

Technically Speaking Comic Series Educational Assessment

Overview of Findings

The comics were highly appealing and engaging as a way to practice and review communication skills and may be a useful way to differentiate GradCommX (see Section IV for more details). Despite this, careful distribution and outreach will be required to reach students. Consider existing, relevant events and workshops at which to distribute the comics, including: student orientation, departmental retreats, TA trainings, and student or departmental research presentations. To the extent possible, call attention to the comics at these events to differentiate from other materials.

Overall, evidence suggests the comics are a clear, accessible, and engaging means of supporting students’ communication skills. Students consistently reported the comics provided an easy-to-read review or reminder of critical information that can support their professional development. While the comics were perceived as less useful for Sloan students that receive abundant departmental support and training, students from other departments believed they filled a need for promoting effective communication skills.

Responses suggest that distribution should be aligned to events or times of year when communication is most relevant for students to maximize reach and effectiveness, such as TA trainings, orientation, and departmental events, or dissertation defense and presentation season. Students did not believe passive distribution strategies would find an audience given the volume of available resources at MIT.

comics3

Keeping Audience In Mind

Choosing The Right Words

Giving Context

Highlighting
Differences

Controlling Focus

Explaining While
Exploring

Giving Context

Using Narrative

Explaining The Technical

Buying The Audience Time

Being Concise

Summarizing
The Summary

Recomendations

Consider the following recommendations to increase the usability of the comics and expand the reach to grad students.

Distribute broadly through relevant events

Despite evidence that comics are engaging and useful, there are overwhelming resources available at MIT. Relevant distribution events could include: student orientation, departmental retreats, TA trainings, and student or departmental research presentations. To the extent possible, call attention to the comics at these events to differentiate from other materials.

Align outreach to relevant times

Focus outreach and distribution during times in the year when presentation skills and communications are salient for students – dissertation defense and job search season.

Partnerships with schools and departments

To better align with relevant distribution channels, OGE may need to partner with academic departments, schools, labs, and professors. Consider strategic partnerships to access large groups of students during relevant times and events, as well as departmental emails and newsletters. Sloan should be considered a low priority, given their increased access to communication PD.

Traffic to OGE

If partnerships are developed with MIT departments and schools, consider using their mass emails and newsletters to drive traffic to OGE to pick up comics and use the opportunity to connect students to other services.

Summary page

If time and budget allow before printing the comics for the 2018-19 school year, add brief, bulleted summaries at the end of each comic. This will help students review the material after their first read and encourage them to keep the comics longer.

Optimize digital comics

If distribution or student engagement through social media is part of the GradX marketing/awareness strategy, consider optimizing comics for online reading or creating excerpt panels that are easily shared.

Testimonials

[themify_icon icon=fa-quote-right icon_bg=”#111″ icon_color=”#fff” style=”large”]

Comic Group 1:

“I was actually surprised by how captivating it was.”

 

[themify_icon icon=fa-quote-right icon_bg=”#111″ icon_color=”#fff” style=”large”]

Comic Group 2:

“[The comics] induced this fun element that is usually sucked out of all the academic activities you have to do on a daily basis.”

 

Gradcommx Live Educational Assessment

Overview Of Findings

The course was highly appealing and engaging and there was evidence of impact on student’s knowledge and attitudes in both the professor- and TA-led courses (see Gradcommx Live 1.0 & 2.o sections for details). A few findings are worth highlighting in this overview:

  • The 2-week, 12-hour course format was perceived as an effective mix of convenient and focused;
  • Students perceived face-to-face practice, rapid feedback and iteration, and access to the professor as the most salient and effective course elements;
  • When TAs lead the course, more transparent communication about their roles and expectations should be provided to students;
  • Despite the appeal and evidence of impact, recruitment was challenging and may require a more sustained and strategic effort.

Gradcommx Live 1.0

The live GradcommX class is an adaptation of the undergraduate class 6.UAT Oral Communication offered the in Electrical Engineering and Computer Science (EECS) department. The GradcommX version was modified to target the types of oral presentations most often given by graduate students and early career scientists and condensed into six two-hour sessions scheduled over a two-week period. Each session began with a group improvisation activity designed to: a) help students loosen up; b) gain confidence performing in front of their classmates; and c) practice relevant public speaking techniques. Sessions also included brief lectures and videos as well as reading comics that featured key concepts of giving a research or technical presentation. Much of the classroom time focused on tailored activities to practice specific presentation skills, practicing presentations of their research, and giving and receiving feedback. Students also had brief homework assignments to prepare for the following day.

Gradcommx Live 1.0 Students

Overall, evidence overwhelmingly supports that GradcommX Live was impactful, engaging, and appealing for participating students. Students demonstrated gains across all measured outcomes, including: 1] general perception of their communication skills; 2] self-reported comfort with communication across select settings; 3] self-reported confidence communicating about their research; and 4] course-specific content knowledge, despite having high scores at pre-test (See Tables 1.1 – 1.3). While this small sample size does not provide generalizable results of efficacy, these findings provide a clear trend that is corroborated by focus group findings.

Students attending the focus group (n=5) compared the course favorably to other MIT professional development experiences, reported high-levels of engagement and enjoyment with the course, reported an intention to recommend the course to other students and use the skills in their lab groups, and provided anecdotes of immediate impact on their professional and academic pursuits.

[themify_icon icon=fa-quote-right icon_bg=”#111″ icon_color=”#fff” style=”large”]

Focus Group Participant

“I’m taking a writing course and the professor actually said, ‘you know what?’ Your writing has improved a lot in the last two weeks. What happened?”’

 

Further

Considerations:

Further Testing Suggested

The course demonstrated impact and was highly engaging for students. However, a 12-hour live course with dedicated attention from an MIT professor is difficult to reach students at scale. The interpersonal experiences students enjoyed are incredibly difficult to recreate in a completely online course. Further there was no evidence from these focus group, student survey responses, or the professor’s description of the course goals and objectives, that a fully digital course would have a similar impact on student outcomes or match the level of engagement reported here (this was partially confirmed in Section V, but the results may be skewed by the timing of the research). Based on these findings, a TA-led course was tested in which the professor led half the course hours and led the rest of the course (see Section II).

An earlier version of this report suggested the development of a hybrid course that combined in-person practice session with online activities and assignments, further reducing the amount of classroom time for students and professors. This format may warrant pilot testing in the future, but considerable planning would be required to find the balance of online content/assignments (e.g. readings, videos, assignments) with interactive courses, workshops, or practice groups. If this or a similar format is pursued in the future, it would be important to consider the salient course elements described above: face-to-face access, condensed timeframe to allow sustained focus over short period, small groups of students, improvisation games, and opportunities for rapid iteration and feedback.

Another approach to scale live learning opportunities include assisting smaller communities of MIT students to organize and practice the skills. The role of GradX could be to provide materials and resources to develop and reinforce communication skills and to facilitate both formal (e.g. courses, workshops) and informal (e.g. student working groups, Q&As) practice.

Gradcommx Live 2.0

GradcommX Live – IAP was a professional development course led by Tony Eng over two weeks during the January 2018 IAP period. The course was similar to GradCommX Live 1.0, but professor Eng supported three teaching assistants (TAs) who played a significant role in the implementation of the IAP class, leading some activities during the first week and serving as primary instructors for much of the second week.  Again, participating students (n=7) completed pre-/post-course surveys and four students attended a 60-minute focus group to discuss the salient features of the course.

Overall, feedback suggests the course was highly valued by participants. Although student confidence in communicating their research did not increase significantly, student knowledge of the communication skills taught in the class did show a small but statistically significant increase. The small sample size (n=7) may partially explain the statistically insignificant improvement in confidence.

Gradcommx Live 2.0 Students

Students found the TAs to be effective, rating the usefulness of their activities and feedback 4.7 out of 5.  The only negative aspect of having TAs take the lead cited by the students was the lack of any introduction or explanation of their role in the class.  Both on surveys and in the focus group, students said they wished TAs had been formally introduced to the class. Because they were not introduced and their role was never explained, students were left with the impression that they were just covering for Tony during the second week.

[themify_icon icon=fa-quote-right icon_bg=”#111″ icon_color=”#fff” style=”large”]

Focus Group Participant

“I don’t know what actually happened. But the way it felt to me is that Tony started leading the class. And then all of a sudden, this week he had a commitment from 10 to 12. And that’s why the TAs all of a sudden took over. That’s how it felt for me.”

 

Further

Considerations:

Consider moving comics and videos from coursework to homework

Both TAs and students suggested that having students watch videos and read the comics during the class was not a good use of time. One of the TAs pointed out that the students could have used a break during the sessions but that they were expected to read the comics during their 15 minute break.

Continue the 2-week course format

As with the fall run of the class, students appreciated the two-week format. Because of competing demands for grad student’s time, attending a live class for more two weeks would be not be feasible.

Increase transparency about the role of the professor & the TAs

Both the TAs and the students reported there was uncertainty and confusion about the role of the TAs and the professor’s availability for the course that interfered with the course. In the future, it may benefit all stakeholders to clearly indicate the role and level of involvement of the TAs and the professor.

Consider testing the TA model again before expansion

Based on the IAP session, TAs may be a viable means of expanding the reach of live-class professional development. However, the challenge of getting more students to sign up or commit to professional development offerings must be addressed before this can be fully tested.

Gradcommx The Game: Inspace Attack! Educational Assessment

Overview Of Findings

The students who reviewed the online game appreciated the concept and content of the emails and the digital comics. The small number who looked at the additional content in the “Hard Mode” of the game (i.e. suggested assignments with scoring rubrics) indicated it was relevant and useful. However, the email format of the course was not generally appealing for students. It’s difficult to disentangle the general lack of interest in the online game from the timing of the pilot test, as students were asked to play the game and provide feedback during busy end-of-term period nearing finals and thesis submission deadlines. Additional pilot testing, debugging, and revisions will be required to effectively engage students with this format.

In April 2018 students were invited via OGE email to pilot GradcommX The Game: INSPACE ATTACK!, an online professional development program. The game presented the GradcommX Live communication skills content through an email-based, gamified narrative that leads participants through assignments, videos, comics, and other course content. The following is a brief description of the game concept and mechanics:

Game Concept

Using retro/pixelated graphics emulating 1980’s computer games, players are tasked with protecting INSPACE, an area of consciousness dedicated to working out problems, from the attacking Nega-U. Players learn a communications skill in each level needed to defeat one of the monsters from Nega-U’s army: Specifikos, Myopical, Borwal, Cloc, Foci, and finally Nega-U.

Game Mechanics

The game was distributed through a series of automated emails. Once students signed up via the GradX website, MailChimp sent students an email introducing the game. Each subsequent level consisted of an email containing game story text, a comic, and a link to “Hard Mode” – a website with additional assignments, videos, and content. At the end of each email, students clicked a button to indicate they had completed the level, signaling the defeat of the level boss and triggering the next email.

Gradcommx The Game: Inspace Attack!

Overall, reactions to the game were mixed. When asked to rate on a scale of one to ten how likely they are to recommend the game to colleagues, the fifteen survey respondents rated a mean of 5.20. Among the 6 respondents who completed more than half of the game levels, the mean rating was 8.33. This may indicate that students who valued the game were more likely to persist or that greater exposure to the game content increased students’ appreciation of it. Additional data from students who complete the game is needed to better understand this finding.

[themify_icon icon=fa-quote-right icon_bg=”#111″ icon_color=”#fff” style=”large”]

Survey Respondent

“The game was great because it made me realize all of the ways I could improve my presentations – not just through the exercises, but the key concepts themselves.”

 

The greatest barrier to student participation in the pilot was timing. Because the game was launched near the end of the semester, most students were too busy to complete the game. Thus, feedback on the game is limited to a small sample size. However, the data provided by participants, along with data collected from students who expressed an interest in the game but did not participate, offer some useful insights.

Further

Considerations:

Further Testing Suggested

The challenges associated with the game’s technical issues and the timing of this study makes it difficult to draw conclusions about the game and create recommendations. Additional beta- and pilot-testing is certainly needed to uncover effective ways of leading and distributing the game.

It’s important to acknowledge that the actual content on the GradCommX website was perceived as useful and engaging and many students reported they would use it again. It should therefore be a priority to populate this website and continue to experiment with ways to draw students to the site.

The Great Communicators Podcast Educational Assessment

Overview Of Findings

There was little evidence of appeal for the Great Communicators podcasts. The concept of using podcasts to meet OGE’s goals were viewed favorably by many students, and podcasts have potential for reaching students at scale – podcasts continue to grow in popularity, offer flexible consumption to listeners, and are easily disseminated and shared. However, the preferences of podcast consumers (i.e. discerning, motivated more by entertainment than learning) and crowded ecosystem of high-quality, informative podcasts would require OGE to carefully curate the topics, guests, and host to be relevant and useful for students. If resources are available and these assets are framed and aligned to the broader OGE offerings, consider the following for episode creation:

  • Interview credible sources who provide insights from personal experience that promote the importance of communication skills;
  • Ensure the host, format, and guests are engaging and entertaining, as students are still generally listening to podcasts for entertainment;
  • Provide usable, tangible strategies that are aligned or related to the with the main points of the GradCommX class and comics;
  • Create an ongoing podcast series to cover a variety of professional development topics such as non-academic career options and entrepreneurial skills.

In general, students participating in two focus groups (n=12) reported the Great Communicators podcasts did not provide enough new information and failed to meet their expectations based on the titles and the speakers. However, many students across the study perceived podcasts as an appealing medium for learning about communication skills. Podcasts provide a compelling opportunity to reach students at scale, but realizing this goal will require careful framing and curation.

Salient Findings

In general, the distribution of podcast listeners in this sample matches available market data, which found 26% of US adults consume podcasts monthly and 16% listen weekly. 

Evidence from these groups suggests they listen while performing activities that don’t require 100% focus, like rote household activities (e.g. cooking, cleaning), travel (e.g. commuting, walking), or going to the gym. This again is supported by an annual market analysis conducted by Edison Research which showed 51% of listeners listen outside of their home (www.edisonresearch.com/podcast-research/). In this context, the goal of podcasts needs to be first to entertain, with learning coming as an added bonus.

firstimage2 1

Listening tastes varied, but focused primarily on a blend of entertainment, distraction, or an excuse to feel a little productive during rote activities.

[themify_icon icon=fa-quote-right icon_bg=”#111″ icon_color=”#fff” style=”large”]

Pod Group 1:

“So I tend to gravitate towards stuff like probably, “Pod Save America” or “United States of Anxiety,” that you learn something, but it’s not that serious. I can tune out and do what else I need to do.”

 

[themify_icon icon=fa-quote-right icon_bg=”#111″ icon_color=”#fff” style=”large”]

Pod Group 2:

“I feel everyone’s using podcasts as their peripheral vision around things they want to be interested in, but not dive too deeply into.”

 

Discussion & Recommendations

The current format was not engaging or effective for students, but podcasts could be a useful asset for GradX as part of a PD series. The development of the podcasts need to be carefully considered to meet: 1) discerning tastes of regular listeners; 2) specific perceptions of what podcast are; and 3) common listening patterns. It may be that podcasts are better suited to build a case for communication as a skill that warrants practice, to be redesigned to align to specific GradCommX lessons from the course or the comics, or broadened in an ongoing series to address professional development topics beyond communications.

As an example, the novel concepts presented by Tony during the live course would make an entertaining and useful podcasts. Tony could discuss Why improv games make you a better presenter – you won’t have time to practice each talk, so you’d be better served preparing how to think on your feet so you prepare less later.

Further

Considerations:

Framing

What is this and why should I listen? There was a mismatch between The Great Communicators title and the content being delivered.

Credibility

Why is this person being interviewed and why can I learn from them? Also, who is this person doing the interview and why do I care? Scholarship on podcasting suggests listeners look for a connection with the host as well as the content.

More narrative

Behind the scenes stories from MIT professors – give interesting context for the lessons, where the main point of each podcast is supported by real-life experience, lessons, and things they can’t get from other places.

Short and More instructional

Dense with information, professors use audio examples to demonstrate technique and mistakes. This suggestion doesn’t actually seem to mesh with the participant feedback and stated preferences, but make appeal non-podcast users. This type of a podcast feels more like the structure of a video.

More Details

The information listed above is a part of a larger assessment of elements found within Gradcommx. 

To access the complete assessment of all elements found within Gradcommx go to: https://oge.mit.edu/about/committees/tfpro/

Assessment Prepared By: