with to ensure the file is closed properly.\n- Use the CSV module to parse comma and tab separated datasets.\n\n**Time Estimation: 1H30M**\nHere we’ll give a quick tutorial on how to read and write files within Python.
\n\n\nAgenda\nIn this tutorial, we will cover:
\n\n
\n- Setup
\n
For this tutorial, we assume you’re working in a notebook (Jupyter, CoCalc, etc) so we’ll run a quick “setup” step to download some CSV data:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-1", "source": [ "import urllib.request\n", "# Download a copy of Hamlet.\n", "urllib.request.urlretrieve(\"https://gutenberg.org/cache/epub/1524/pg1524.txt\", \"hamlet.txt\")\n", "# Download some COVID data for Europe\n", "urllib.request.urlretrieve(\"https://opendata.ecdc.europa.eu/covid19/vaccine_tracker/csv/data.csv\", \"vaccinations.csv\")\n", "# And a fastq file\n", "urllib.request.urlretrieve(\"https://gist.github.com/hexylena/7d249607f8f763301f06c78a48c3bf6f/raw/a100e278cee1c94035a3a644b16863deee0ba2c0/example.fastq\", \"example.fastq\")" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-2", "source": "And now we’re ready to get started learning about files!
\nReading from and writing to files in Python is very straightforward, we use open() to open a file to read from it. Files are accessed through something called a file handle, you’re not accessing the file itself, you’re opening a connection to the file, and then you can read lines from through that file handle. When you open a file handle, you must specify it’s mode:
| Mode | \nPurpose | \n
|---|---|
r | \nWe will read from this file handle | \n
w | \nWe will write to this file handle | \n
a | \nWe will append to this file handle (we cannot access earlier contents!) | \n
You will mostly use r and w, a is especially useful for writing to program logs where you don’t really care what was written before, you just want to add your new logs to the end of the file.
Here we introduce a new bit of syntax, the with block. Technically with begins a “context manager” which allows python to setup some things before the block, run some contents in the block, and automatically handle cleanup of this block. When you open a file, you must close it when you’re done with it (otherwise bad things can happen!) and with prevents most of those issues.
In the above code snippet after the second line, the file (referred to by handle) is automatically closed.
\n\n\n\nInput: Using\nwith\nwith open('file.txt', 'r') as handle:\n print(handle.readlines())\n\n\nOutput: Not using\nwith\nhandle = open('file.txt', 'r')\nprint(handle.readlines())\nhandle.close() # Important!\n
Let’s see what’s in our file. `
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-5", "source": [ "print(lines[0])\n", "print(lines[1])\n", "print(lines[2])" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-6", "source": "Notice how it prints out a blank line afterwards! This is due to a \\n, a newline. A newline just tells the computer “please put content on the next line”. We can see it by using the repr() function:
Every line that’s read in ends in a newline currently. This is done because if we wanted to write it back out, we would need to preserve those newlines, or all of the content would be on one giant line. Let’s try writing out a file, it’s just like reading in a file!
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-9", "source": [ "with open('hamlet-copy.txt', 'w') as handle:\n", " # readlines reads every line of a file into a giant list!\n", " for line in lines:\n", " handle.write(line)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-10", "source": "Check this file in your folder, does it look right? Is it identical in size to the original?
\nLet’s use the file’s contents for something useful. This file specifically is the play Hamlet, by Shakespeare. The contents are formatted with a speaker indicated with all capital letters, followed by their lines (potentially spread over multiple lines of the file.)
\nHAMLET. Madam, how like you this play?\n\nQUEEN. The lady protests too much, methinks.\n\nHAMLET. O, but she’ll keep her word.\nSo let’s count up how many times each character speaks! (Roughly)
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-11", "source": [ "with open('hamlet.txt', 'r') as handle:\n", " # readlines reads every line of a file into a giant list!\n", " lines = handle.readlines()\n", "\n", "speakers = {}" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-12", "source": "Here we’ve initialised the lines variable with the contents of the text, and setup speakers as a dictionary that will let us track how many times each character speaks. Next let’s define a function to check if a character is speaking on that line. It is if it meets two conditions: the first word is all caps, and it ends with a ..
We can use that function later to check if a line starts with a speaker
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-15", "source": [ "# Loop over every line we read in\n", "for line in lines:\n", " # Split by default splits on whitespace.\n", " words = line.split()\n", "\n", " # Are there words on this line? We can't access the first word if we\n", " # haven't any words.\n", " if len(words) == 0:\n", " continue\n", "\n", " # Check if the first word is uppercase, and the last character is a `.`,\n", " # then it's a character speaking.\n", " if is_speaker(words[0]):\n", " # Give this an easier to remember and understand name.\n", " speaker = words[0]\n", "\n", " # Have we seen this speaker before? If not, we should add them to the\n", " # speakers dictionary. Hint: Try removing this to see why we do this.\n", " if speaker not in speakers:\n", " speakers[speaker] = 0\n", "\n", " # Increment the number of times we've seen them speak.\n", " speakers[speaker] = speakers[speaker] + 1" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-16", "source": "Ok! We’ve done a couple things here that all fall into the category of defensive programming. As programmers, we often accept input from users or from unknown sources. That input may be wrong, it may have bad data, it may be trying to attack us. So we respond by checking very carefully if things match our expectations, and rejecting the input otherwise. We did a couple things here for that:
\n.)continue to skip it if it wasLet’s see who was the most chatty:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-17", "source": [ "for key, value in speakers.items():\n", " print(key, value)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-18", "source": "We’ve clearly caught a number of values that aren’t expected, some section headers (the numeric values, and some rare values we don’t expect.)
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-19", "source": [ "for key, value in speakers.items():\n", " if value > 1:\n", " print(key, value)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-20", "source": "Hamlet, the titular character, has the vast majority of turns speaking throughout the play:
\n| Character | \nTurns speaking | \n
|---|---|
| HAMLET | \n358 | \n
| HORATIO | \n107 | \n
| KING | \n102 | \n
| POLONIUS | \n86 | \n
| QUEEN | \n69 | \n
Writing a file out is exactly like reading a file, we just use the different file mode w to indicate we wish to write to a file:
Check the file’s contents in your folder, does it look like you expect? Remember your \\ns!
A common use case is transforming one file’s contents into another file format or file type. Let’s do a very simple example of that, taking a FASTQ file and transforming it into a FASTA file. Remember first that a FASTQ file looks like:
\n@M00970:337:000000000-BR5KF:1:1102:17745:1557 1:N:0:CGCAGAAC+ACAGAGTT\nGTGCCAGCCGCCGCGGTAGTCCGACGTGGCTGTCTCTTATACACATCTCCGAGCCCACGAGACCGAAGAACATCTCGTATGCCGTCTTCTGCTTGAAAAAAAAAAAAAAAAAAAACAAAAAAAAAAAAAGAAGCAAATGACGATTCAAGAAAGAAAAAAACACAGAATACTAACAATAAGTCATAAACATCATCAACATAAAAAAGGAAATACACTTACAACACATATCAATATCTAAAATAAATGATCAGCACACAACATGACGATTACCACACATGTGTACTACAAGTCAACTA\n+\nGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGFGGGFGGGGGGAFFGGFGGGGGGGGFGGGGGGGGGGGGGGFGGG+38+35*311*6,,31=******441+++0+0++0+*1*2++2++0*+*2*02*/***1*+++0+0++38++00++++++++++0+0+2++*+*+*+*+*****+0**+0**+***+)*.***1**//*)***)/)*)))*)))*),)0(((-((((-.(4(,,))).,(())))))).)))))))-))-(\n| Line | \nContents | \n
|---|---|
| 1 | \nIdentifier, prefixed with @ | \n
| 2 | \nSequence | \n
| 3 | \n+ | \n
| 4 | \nQuality scores | \n
And a fasta file looks like this, where > indicates a sequence identifier, and it is followed by one or more lines of ACTGs.
>M00970:337:000000000-BR5KF:1:1102:17745:1557 1:N:0:CGCAGAAC+ACAGAGTT\nGTGCCAGCCGCCGCGGTAGTCCGACGTGGCTGTCTCTTATACACATCTCCGAGCCCACGAGACCGAAGAACATCTCGTATGCCGTCTTCTGCTTGAAAAAAAAAAAAAAAAAAAACAAAAAAAAAAAAAGAAGCAAATGACGATTCAAGAAAGAAAAAAACACAGAATACTAACAATAAGTCATAAACATCATCAACATAAAAAAGGAAATACACTTACAACACATATCAATATCTAAAATAAATGATCAGCACACAACATGACGATTACCACACATGTGTACTACAAGTCAACTA\nIn the setup portion we downloaded a FASTQ file, now let’s extract all of the sequences from this file, and write them out as a FASTA file. Why would you want to do this? Sometimes after sequencing a sample (especially metagenomics), you want to blast the sequences to figure out which organisms they belong to. A common way to do that is BLAST which accepts fasta formatted sequences. So we’ll write something to convert these formats, removing the + and quality score lines.
That’s it! Check if your fasta file looks correct.
\n\n\nIf you want to know which item number you’re on while you’re looping over a list, you can use the function
\nenumerate()\nTry out this code to see how it works:\nfor index, item in enumerate(['a', 'b', 'c']):\n print(index, item)\nTry using it to clean up the above code.
\n
If you’re reading data from a comma separated value (CSV) or tab separated value (TSV) file, you should use the built in csv module to do this. You might ask yourself “Why, csv parsing is easy” and that is a common thought! It would be so simple to do something like
But you would be wrong! This code has a subtle bug that you might not see until someone generates data that specifically affects it, with “quoted” columns. If you have a table like
\n| Patient | \nLocation | \nDisease Indications | \n
|---|---|---|
| Helena | \nDen Haag, the Netherlands | \nZ87.890 | \n
| Bob | \nLittle Rock, Arkansas, USA | \nZ72.53 | \n
| Jane | \nLondon, UK | \nZ86.16 | \n
This would probably be exported as a CSV file from Excel that looks like:
\nPatient,Location,Disease Indications\nHelena,\"Den Haag, the Netherlands\",Z87.890\nBob,\"Little Rock, Arkansas, USA\",Z72.53\nJane,\"London, UK\",Z86.16\nNote that some columns are quoted. What do you think will happen with the following code?
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-27", "source": [ "csv_data = \"\"\"\n", "Patient,Location,Disease Indications\n", "Helena,\"Den Haag, the Netherlands\",Z87.890\n", "Bob,\"Little Rock, Arkansas, USA\",Z72.53\n", "Jane,\"London, UK\",Z86.16\n", "\"\"\".strip().split('\\n')\n", "\n", "# Please don't do this :)\n", "for line in csv_data:\n", " print(line.split(','))" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-28", "source": "Does that look right? Maybe not. Instead we can use the csv module to work around this and properly process CSV files:
That looks a lot better! Now we’ve properly handled the quoted columns that contain one or more , in the middle of our file. This is actually one of the motivating factors in using the TSV format, the tab character is much more rare in data than ,. There is less chance for confusion with poorly written software.
Let’s read in some statistics about vaccinations:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-31", "source": [ "import csv\n", "\n", "vax = []\n", "with open('vaccinations.csv', 'r') as handle:\n", " csv_reader = csv.reader(handle, delimiter=\",\", quotechar='\"')\n", " for row in csv_reader:\n", " # Skip our header row\n", " if row[0] == 'YearWeekISO':\n", " continue\n", " # Otherwise load in the data\n", " vax.append(row)\n", "\n", "print(vax[0:10])" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-32", "source": "Here we have a 2 dimensional array, a list of lists. Each row is an entry in the main list, and each column is an entry in each of those children.
\nOur columns are:
\n| Column | \nValue | \n
|---|---|
| 0 | \nYearWeekISO | \n
| 1 | \nReportingCountry | \n
| 2 | \nDenominator | \n
| 3 | \nNumberDosesReceived | \n
| 4 | \nNumberDosesExported | \n
| 5 | \nFirstDose | \n
| 6 | \nFirstDoseRefused | \n
| 7 | \nSecondDose | \n
| 8 | \nDoseAdditional1 | \n
| 9 | \nUnknownDose | \n
| 10 | \nRegion | \n
| 11 | \nTargetGroup | \n
| 12 | \nVaccine | \n
| 13 | \nPopulation | \n
Let’s subset the data to make it a bit easier to work with, maybe we’ll just use the Dutch data (please feel free to choose another column though!)
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-33", "source": [ "country = 'NL'\n", "\n", "subset = []\n", "for row in vax:\n", " # Here we select for a country\n", " if row[1] == country:\n", " subset.append(row)\n", "print(subset[0:10])" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-34", "source": "That should be easier to work with, now we only have one country’s data. Let’s do some exercises with this data:
\n\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-35", "source": [ "# Try things out here!" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-36", "source": "Question: Which vaccines were given?\nWhich vaccines were given? Use the
\nsubsetto examine which vaccines were given in the Netherlands. Tip: ifxis a list,set(x)will return the unique values in that list.\n👁 View solution
\n\nTo figure out which vaccines were given, we can look at column 12:
\n\nvaccines = []\nfor row in subset:\n vaccines.append(row[12])\nprint(set(vaccines))\nWe can use the
\nsetfunction to convert the list to a set, and show only the unique values.
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-37", "source": [ "# Try things out here!" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-38", "source": "Question: How many of each vaccine were given?\nHow many of each were given?
\nTip: use the accumulator pattern.\nTip: Columns 5, 7, and 8 have doses being given out to patients.
\n\n👁 View solution
\n\n\ndoses = {}\nfor row in subset:\n brand = row[12]\n if brand not in doses:\n doses[brand] = 0\n doses[brand] = doses[brand] + int(row[5]) + int(row[7]) + int(row[8])\nprint(doses)\n
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-39", "source": [ "# Try things out here!" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-40", "source": "Question: How many of each vaccine were exported? received?\nHow many of each were exported? received?
\nTip: you only need to loop once.\nTip: you will need to handle an edge case here. Try and find it out!
\n\n👁 View solution
\n\n\nexport = {}\nreceived = {}\nfor row in subset:\n brand = row[12]\n if brand not in export:\n export[brand] = 0\n if brand not in received:\n received[brand] = 0\n if row[4]:\n export[brand] = export[brand] + int(row[4])\n if row[3]:\n received[brand] = received[brand] + int(row[3])\nprint(export)\nprint(received)\n
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-41", "source": [ "# Try things out here!" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-42", "source": "Question: When was the first dose received?\nTip: use
\nbreak, and check how many doses were given!\n👁 View solution
\n\n\nfor row in subset:\n if row[5] and int(row[5]) > 0:\n print(f\"On {row[0]}, {row[5]} doses of {row[12]} were given\")\n break\n
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-43", "source": [ "# Try things out here!\n", "percent_vaccinated_per_week = []\n", "# Write code here!\n", "\n", "\n", "\n", "\n", "# When you're done, you should have a 'results' variable\n", "# You may need to `pip install matplotlib`\n", "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "plt.plot(percent_vaccinated_per_week)\n", "plt.xlabel('Week')\n", "plt.ylabel('Percent Vaccinated')" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-44", "source": "Question: Transform the data for plotting\nLet’s say you want to plot the fraction of the population that has been vaccinated by the various points in time.
\n\n
\n- Subset the data further for TargetGroup (column 11) set to ‘ALL’
\n- Create an accumulator, to count how many doses have been given their FirstDose at each week
\n- Use column 13 (population) to calculate the fraction of the population that has been given one of those doses at each week
\nThe output should be a list of percentages ranging from [0.0 to 1.0].
\n\n👁 View solution
\n\n\ntotal_doses = 0\npercent_vaccinated_per_week = []\nfor row in subset:\n if row[11] != 'ALL':\n continue\n total_doses = total_doses + int(row[5])\n percent_vaccinated_per_week.append(total_doses / int(row[13]))\n
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-45", "source": [ "# Try things out here!" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-46", "source": "Question: Which vaccines were given?\nWrite this out to a new file, with two columns. The index and the Percent Vaccinated. It should be a comma separated file, and should have a header. Save this as a csv file named
\nweekly-percent-vax.csvTip: use a
\ncsvwriter, it works exactly like acsvreader. You can use thewriterow()function to write out a row.\nTip: Useenumerate()to get a list of items with indexes.\n👁 View solution
\n\n\nwith open('weekly-percent-vax.csv', 'w') as handle:\n writer = csv.writer(csv_data, delimiter=\",\", quotechar='\"')\n for row in enumerate(percent_vaccinated_per_week):\n writer.writerow(row)\n
Congratulations on getting this far! Hopefully you feel more comfortable working with files.
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "cell_type": "markdown", "id": "final-ending-cell", "metadata": { "editable": false, "collapsed": false }, "source": [ "# Key Points\n\n", "- File reading requires a mode: read, write, and append\n", "- Use the CSV module to parse CSV files.\n", "- do NOT attempt to do it yourself, it will encounter edge cases that the CSV module handles for you\n", "- Use a `with` block to open a file.\n", "\n# Congratulations on successfully completing this tutorial!\n\n", "Please [fill out the feedback on the GTN website](https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-files/tutorial.html#feedback) and check there for further resources!\n" ] } ] }