Today, I applied my plan from yesterday.
To format the items in each cell, I learned about
an elegant and efficient way to create lists in Python.
For each cell,
I used a list comprehension
to create a list of all the diagnoses within the year interval.
Then, I merged the elements together with
During my research on how to combine the diagnoses together,
I came across
a very informative write-up
that warned against iteration. The author writes,
“[i]teration in Pandas is an anti-pattern
and is something you should only do when you have exhausted every other option”.
That shocked me!
I remember that I and fellow Energize Andover members had often used
iterrows() in our code.
Reading on, I learned that iterating through
DataFrames is much slower than
the other options that Pandas provides for multi-element operations.
One option is using vectorized functions.
In contrast to iteration, where single elements are processed repeatedly,
vectorization is where multiple elements are processed
with a single operation simultaneously.
This leads to great improvements in performance for large datasets.
Here is the post’s benchmark of different methods that add two columns together.
As you can see, the vectorized implementation
vec really outperforms
the other implementations at a large number of elements
I feel enlightened with this new understanding of Pandas.
I hope to gain some experience working with vectorized functions
and hopefully rewrite my old Energize Andover code to use them instead of