Research Guides: Text and Data Mining: TDM: Tools & Technology

Considerations for Mining

Contact your Subject Specialist librarian for data source options, or try the Finding Data library guide. Contact dataservices@vt.edu for additional recommendations on processes and tools for manipulating, analyzing, and visualizing text or data. Some general considerations for pulling large amounts of data:

Use an official API or bulk download service if provided.
Web scraping is an option, but can have drawbacks. Consider the amount and speed with which you are pulling data, and put forth a good faith effort to identify terms of use from your sources.
If accessing subscription or licensed content, be sure to check the policies or terms.

Tool Lists:

General: Tutorials, Toolsets, and Resource Lists
Data Processing: Getting, Scraping, Extracting, Transcribing, and Cleaning Data from websites, documents
Analyzing and Visualizing Data
Coding language applications for TDM
- Python
- R

Tutorials, Toolsets, and Resource Lists

ContentMine
The usage of ContentMine tools can be learned step-by-step with the help of these tutorials.
Data Visualization and Analysis Tools
Computerworld chart - 30+ free tools for data visualization and analysis.
DHbox
A digital humanities laboratory with essential applications through simple sign-in via a web browser.
DiRT Directory
A registry of digital research tools for scholarly use.
Introduction to Text Analysis
This workbook provides a brief introduction to digital text analysis through a series of three-part units.
Programming Historian
Novice-friendly, peer-reviewed tutorials that help humanists learn a wide range of digital tools, techniques, and workflows.
TAPoR
Text Analysis Portal for Research - Discover research tools for studying texts.

Data Processing: Getting, Scraping, Extracting, Transcribing, and Cleaning Data from websites, documents

ChemDataExtractor
Automatically extract chemical information from scientific documents
ContentMine: getpapers
Get metadata, fulltexts or fulltext URLs of papers matching a search query.
DHbox
A digital humanities laboratory with essential applications through simple sign-in via a web browser.
DiRT Directory
A registry of digital research tools for scholarly use.
From the Page
Open-source software for collaborative project support, from simple, plain-text transcription through bilingual digital editions. Indexing, discussion, security options, and version control features.
Google Fusion Tables
An experimental data visualization web application to gather, visualize, and share data tables.
OpenRefine
A powerful tool for working with messy data: cleaning it; transforming it from one format into another; and extending it with web services and external data.
Scrapy
An open source and collaborative framework for extracting the data you need from websites.
Scribe
A framework for crowdsourcing the transcription of text-based documents, particularly documents that are not well suited for Optical Character Recognition
Scripto
A free, open source tool enabling community transcriptions of document and multimedia files.
Tabula
Extract data tables from PDFs
TAPoR
Text Analysis Portal for Research - Discover research tools for studying texts.
TypeWright
A tool for correcting the text-version of a document made up of page images.
Ubiqu+Ity
Generates statistics and web-based tagged text views for your text/s, using the DocuScope dictionary or your own rules.
WordSeer
A text analysis environment for Humanities scholars that combines visualization, information retrieval, sensemaking and natural language processing to make the contents of text navigable, accessible, and useful.

Analyzing and Visualizing Data

Bookworm
Bookworm is a simple and powerful way to visualize trends in repositories of digitized texts.
Data Analysis and Visualization Tools
Computerworld chart - 30+ free tools for data visualization and analysis.
DiRT Directory
A registry of digital research tools for scholarly use.
Gephi
The leading visualization and exploration software for all kinds of graphs and networks. Gephi is open-source and free.
Google Books NGram Viewer
Graphical displays of how words or phrases have occurred in the Google Books corpus of books.
Google Charts
Interactive charts for browsers and mobile devices.
Google Fusion Tables
An experimental data visualization web application to gather, visualize, and share data tables.
Tableau Public
Visualize and Share Your Data in Minutes—For Free.
TAPoR
Text Analysis Portal for Research - Discover research tools for studying texts.
VOSViewer
A software tool for constructing and visualizing bibliometric networks, for co-citation, bibliographic coupling, co-authorship relations, or co-occurrence networks of important terms.
Voyant
A web-based reading and analysis environment for digital texts.
WordSeer
A text analysis environment for Humanities scholars that combines visualization, information retrieval, sensemaking and natural language processing to make the contents of text navigable, accessible, and useful.

Python Resources

Python
A programming language that lets you work quickly
and integrate systems more effectively.
Cleaning Data in Python
A tutorial on Cleaning Data in Python from the University of Toronto Map & Data Library. *You may need to navigate to the tutorial link via a list on the left side of the landing page.
Good Tables
Validate tabular data with Python
Pandas
Data analysis and modeling for Python
PDFMEF
A multi-entity extraction framework for academic documents
SciPy
A Python-based ecosystem of open-source software for mathematics, science, and engineering
SMaPP Toolkit
Python library for interacting with smapp collections.
txtorg
A Python-based utility that leverages Whoosh and Apache Lucene to facilitate text preprocessing and management.

R Resources

R
A free software environment for statistical computing and graphics.
TidyText
Text Mining with R book companion website and resource.
R and Data Mining Courses
Directory of free online courses related to R and Data Mining.
tabulizer
Provides R bindings to the Tabula java library, which can be used to computationally extract tables from PDF documents.
Quanteda
An R package for the Quantitative Analysis of Textual Data.
Austin
Austin is an R package for doing things with words. Right now that means scaling them in the style of Wordscores and Wordfish.
stm: Structural Topic Model
A general framework for topic modeling with document-level covariate information. The covariates can improve inference and qualitative interpretability and are allowed to affect topical prevalence, topical content or both.
tesseract
Extract text from an image. Requires that you have training data for the language you are reading. Works best for images with high contrast, little noise and horizontal text.

Need Assistance?

Assistance:

Content sources: Subject Specialists

Analysis and Tools: dataservices@vt.edu