Building Flashcards for ASL

Motivation

Studying vocabulary in a new language is normally a simple, if time-consuming process: get a vocab list, transform it into a flashcard deck, then drill the deck to learn the words (preferably with a spaced repetition system (SRS) software like Anki).

 

For most language classes, the vocabulary is written words, which can easily be typed or written into front/back flashcards. However, American Sign Language (ASL) is a visual language, and signs need to be seen -- via a video -- to be learned. This means a simple typed vocabulary list isn't enough. 

Instead, ASL classes typically provide vocab lists as videos showing an instructor signing a set of words, and each video is accompanied by a list of words in it, written out as ASL "gloss" (a basic transcription of the sign's meaning). Given these materials, one way to study is to watch the video through, pausing after each sign and self-testing the sign's meaning (to learn ASL → target language), and then to sign each word on the vocab list, checking against the video after each word (to learn target language → ASL).

 

However, this approach has a few drawbacks:

1. It's hard to focus study time on signs you know poorly (videos are sequential and don't have sign markers within the video)
2. Reviewing on the go can be challenging (need both video & vocab list side-by-side)
3. Brain gets used to recalling words in a defined, sequential order (which may not be as effective at training out-of-order (free) recall, which tests and real life both require)
4. Finding the sign for any given word can be a very slow process (need to review scores of videos over a semester, dozens of vocab lists -- though a site with a dictionary like this can be helpful) 

 

Building a flashcard deck with modern software effectively handles all these problems. Modern flashcard apps with SRS algorithms are available on all devices, present cards in a random order with cards you get wrong presented more frequently, and are easily searchable. However, actually making these flashcards for ASL isn't particularly easy because they are videos, not text, and for ease of creation and distribution, the videos provided by instructors typically contain multiple signs.

 

So, the overall workflow for making ASL flashcards is: Split vocab videos into clips containing one sign each, line up the videos and the corresponding ASL gloss, then import the collection to a flashcard software.

 

When I started taking ASL in university, I thought I could just find a video editing software, split up the videos, and export all the clips. I couldn't find software that fit the bill within a student budget then, but the "export as individual clips" option in Davinci Resolve may be an option now. Since I didn't find that at the time, I moved on to spinning a lower-level solution using the very powerful video processing library, ffmpeg. I later wrapped up the video splitting tool with some other scripting to further automate the process of taking vocab list + video → Anki flashcards.

Design

The final app has three main components: the video splitting pipeline, the post-processing steps, and the UI. The first two components are depicted in the architecture diagram.

 

The video splitting pipeline takes two inputs: a CSV file and a vocabulary video:

1. CSV File: Contains two columns, word and start_time. The word column is a list of the ASL gloss representation of each sign in the video, in the order that they are signed. The start_time column contains the timestamp that each sign starts in the video.
2. Vocabulary Video: This is the teacher signing each sign to be learned.

 

The pipeline’s operation is then as follows:

1. Begins with ffmpeg splitting the video into clips along the boundaries marked by the start_time column, which produces (number of words + 1) video clips (the first video is any empty space before the first sign and can be discarded).
2. The next step is to take the word column and rename each clip with the sign it corresponds to.
3. With the clips named by the vocab word, the pipeline can create either an Anki .apkg file (Anki's deck format, suitable for direct import), or a CSV file accompanied by a ZIP file of the video clips, intended for manual Anki import (or for other programs).
1. There's also an option to add Anki tags to the generated Anki deck, e.g., asl101-week1, for better deck organization.

 

The UI wraps up the pipeline into a graphical interface that's easy to use, and also adds instructions on how to use the application and prepare the start_time column.

Usage, Link, and Code

The app is live at https://aslapp.spyn.us. 

 

Instructions to use the app are available within the live app or here: https://github.com/spolcyn/asl-flashcards/blob/master/docs/docs.md, and the source code is available here: https://github.com/spolcyn/asl-flashcards. If you have Docker installed, the app can be built and run locally with a quick docker compose up app. Otherwise, follow the installation instructions in the Github repo.

Conclusion

This pipeline minimizes time spent on non-productive flashcard creation overhead, and maximizes time spent actually learning the content for ASL. As a method of study, flashcard decks also have the advantage that long-term retention is much easier, as they remain easy to use even after the course is over. They can also be easily shared with future students. 

 

For me, building the pipeline was an interesting exercise in various technologies, from its basic phase as a copy-and-pasted ffmpeg CLI command, through being a shell script and then a Streamlit app during its UX development phase, and now finally into its containerized, deployed phase, accessible by everyone. It was also quite useful in making the most of the ASL course I took!