Skip to content

Instantly share code, notes, and snippets.

@Ssenseii
Last active September 29, 2024 09:18
Show Gist options
  • Select an option

  • Save Ssenseii/cb9b1542bdd60ed040319d5c6b3fa2bb to your computer and use it in GitHub Desktop.

Select an option

Save Ssenseii/cb9b1542bdd60ed040319d5c6b3fa2bb to your computer and use it in GitHub Desktop.
Extract Wikipedia Lists Into a Javascript Dictionary. I needed this for a text pre-processing project.
/*
Test Case: https://en.wikipedia.org/wiki/List_of_English_abbreviations_made_by_shortening_words
Creates a js dictionary from a list in wikipedia.
modules requires! Cheerio & fs
The result might need a little cleaning
*/
const fs = require("fs");
const cheerio = require("cheerio");
// Step 1: Read the HTML file content
fs.readFile("abbreviations.html", "utf8", (err, html) => {
if (err) {
console.error("Error reading file:", err);
return;
}
/* Load the HTML content into Cheerio */
const $ = cheerio.load(html);
/* Initialize an empty object to store abbreviations */
const abbreviations = {};
/* Step Loop through each abbreviation <dt> and its associated definitions <dd> */
$("dl").each((i, dl) => {
let currentAbbreviation = "";
$(dl)
.children()
.each((j, el) => {
if ($(el).is("dt")) {
// Get abbreviation key (text content of <dt>)
currentAbbreviation = $(el).text().trim();
abbreviations[currentAbbreviation] = [];
} else if ($(el).is("dd") && currentAbbreviation) {
// Get the meaning of abbreviation (text content of <dd>)
const meaning = $(el)
.text()
.trim()
.replace(/\[\d+\]/g, ""); // Remove references like [1]
abbreviations[currentAbbreviation].push(meaning);
}
});
});
/* Convert the meanings array to a string and output the final result */
for (let abbr in abbreviations) {
abbreviations[abbr] = abbreviations[abbr].join(" / ");
}
console.log(JSON.stringify(abbreviations, null, 2));
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment