Press "Enter" to skip to content

Technical Development As A Region: Part 3

Be sure to read Part 1 and Part 2 of this series first.


Data storage and retrieval – these are four words that encompass a wide range of items. For the layperson, it is often a worthless pursuit to understand; for the technically inclined, it is typically either outside their scope or smack dab in the middle. 

In my previous article, I briefly covered how to parse XML from the NationStates API. I will now discuss how to store that data and how you can use it to automate some regional tasks. 

Typically, when one makes a request to an API, the data is returned as an object of the requests type (assuming you are using the requests library for web requests). However, this object is not persistent, meaning that it does not exist across console sessions. You cannot ‘copy and paste’ the requests object either – instead, you have to store it in some other manner. But what if you parse it?

Now you have more granular data (e.g. nation counts, World Assembly endorsements) without the other metadata (e.g. HTTP status codes, headers) associated with the information you want. However, it is still not formatted in a way that you can save without using other libraries.

Fortunately, there are options available. JSON files, text files, and databases (relational or non-relational) are but a few of the options that one can select from in order to store information. For starting out, we will use text file storage – while it is not the best option, it is one of the most simple.

Let’s make a brief request to print the population of a nation:

import requests
from bs4 import BeautifulSoup

headers = {
   "User-Agent": "Nation: United Calanworie, Demonstration Query for NST"
}

with requests.get("https://nationstates.net/cgi-bin/api.cgi?nation=united_calanworie&q=population", headers=headers) as req:
   soup = BeautifulSoup(req.text, "lxml")
   pop = soup.population.string
   print(pop)

As you can see here, this script hits my nation and asks for the population, then parses the XML result and prints it to the console. However, that is not super helpful if, for example, I want that logged to a file. Fortunately, Python has excellent file managers and context managers

We are going to add a few more lines to this code now:

import requests
from bs4 import BeautifulSoup

headers = {
   "User-Agent": "Nation: United Calanworie, Demonstration Query for NST"
}

with requests.get("https://nationstates.net/cgi-bin/api.cgi?nation=united_calanworie&q=population", headers=headers) as req:
   soup = BeautifulSoup(req.text, "lxml")
   pop = soup.population.string
   with open("pop.txt", "w") as tf:
       tf.write(pop)
       print(f"Written to {tf.name}!")

Okay, so this is a little complicated. What is with the tf, and with open() commands?

tf is a variable that I have created to name pop.txt in order to use it in the program.

with, on the other hand, is a special piece of code called a ‘context manager’. A context manager helps manage your sessions and have everything stay neat and clean across your code. It is considered good practice to close file IO (Input/Output) sessions after you are done using them, and with allows for that to be handled with minimal effort. Of course, this only saves a number to a text file, which for many is all that they need. However, if you are interested in being a power user and learning databases, then continue on.

A Brief Introduction to Databases

What is a database? For all intents and purposes, a database is just one big file that stores information in a specific way that enables access later on. There are multiple database softwares out there, but my personal favorite is MongoDB, so that is what I will use here in this article. 

You will need to install MongoDB in order for this to work, but it is not as simple as installing python – unless you’re running a *nix based system, in which case it will be.

As an extra note, I highly recommend running your database on a dedicated server. This means you should not run your database on the same machine that you plan to develop your code on. If you do not believe this is possible for you, look into cloud hosting. This is important so that you have maximum system resources dedicated to running your code rather than other processes. 

Installing MongoDB is a fairly simple process if you are on Linux. Simply run the following commands:

Add the mongo public key to your keychain
wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
Update your apt list
Change this command based on your linux distribution, the official docs have more to say on that matter
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
Now that you've done this, update your system
sudo apt-get update
Install Mongo DB
sudo apt-get install -y mongodb-org
Stop the packages from getting updated and stay on your mongo version
echo "mongodb-org hold" | sudo dpkg --set-selections
echo "mongodb-org-server hold" | sudo dpkg --set-selections
echo "mongodb-org-shell hold" | sudo dpkg --set-selections
echo "mongodb-org-mongos hold" | sudo dpkg --set-selections
echo "mongodb-org-tools hold" | sudo dpkg --set-selections

If you are on Windows, it gets slightly more complex. Follow these instructions to install it. 

With MongoDB, you can collate all forms of information, and store them in a non-relational database. Python interfaces with it incredibly well and you can do some really fun stuff.

With a running Mongo server (follow the docs to get one started), you can get started on collecting data and storing it in a database. 

You should use the same scripts that you have been using to fetch the data so far, but add these lines to it:

import requests
from bs4 import BeautifulSoup
import pymongo

client = pymongo.MongoClient(YOUR CONNECTION URL HERE
db = client.demonstration_db
coll = db.demonstration_collection

headers = {
   "User-Agent": "Nation: United Calanworie, Demonstration Query for NST"
}

with
requests.get("https://nationstates.net/cgi-bin/api.cgi?nation=united_calanworie&q=population", headers=headers) as req:
   soup = BeautifulSoup(req.text, "lxml")
   pop = soup.population.string
   to_insert = {"Population": pop}
   coll.insert_one(to_insert)

If you look at your database with something like MongoDB Compass, you will see a record with the population in it.

Some ideas for what you can do with this include Discord bots, tracking trends over time and graphing them with something like matplotlib, or just having a reference database so you can look up information over time.

All-in-all, there are many tasks that can be automated that I encourage you to explore. This includes but is not limited to tracking World Assembly endorsements, population growth, API recruitment, and dispatches. Most importantly: have fun with it!

We accept guest contributions!

If you have a knack for writing, have a strong opinion about something on NationStates, or would like to cover something often overlooked, we would love to hear from you! If you are interested, join our Discord server and message Chief Content Officer Llo (@Llo#1475).