Enabling the visually impaired to access charts on the web using deep learning
Original Story by Niklas Elmqvist
Sparks of Innovation: Stories from the HCIL
The fact that visualization leverages the human visual system to convey data is not only inherent in the name “visualization” itself, it is also endemic to the discipline. A huge part of the visualization field concerns itself with visual design guidelines, graphical perception studies, and color theory. Furthermore, much of human thinking is couched in visual terms, in that to understand something is called “seeing it,” in that we want to bring our thoughts into “focus,” and in that we strive to make our ideas “clear.” But what if all you have are words and no pictures; that is, what if you are visually impaired? Is the power of visualization forever closed to you, or worse, are you actively barred from accessing important data about our world?
This is particularly true on the internet. The web has had a revolutionary impact on improving information access for the visually impaired, who use so-called screen readers to transform a visual display to text, sound, or Braille. However, besides textual content, the web also holds hundreds and thousands of charts stored as images. While a photograph can manually or automatically labeled, screen readers do not work well for this kind of data-rich images. Since images are just collections of colored pixels, encoding data in maps, line graphs, and barcharts essentially means locking away the data for all but sighted users. Furthermore, most websites do not contain the raw data that generated these charts. While it is true that accessibility standards are on the rise, there is a vast collection of legacy charts on the web where no such data will ever be made available.
We tackle this problem in our recent work, “Visualizing for the Non-Visual: Enabling the Visually Impaired to Use Visualization”, where the idea is to have the user’s web browser automatically detect charts encoded as images. For any such image, the browser will send the image to a server which interprets the chart encoded in the image and returns the raw data. The browser then replaces the chart image in the webpage with a table, which can be directly navigated using a screen reader, indexed by a search engine, or rendered anew with an interactive visualization. The tool is currently available as a prototype extension to Google Chrome.