Mailman is perhaps one of the most popular mailing lists in the world, yet searching and dealing with archives is absolutely hideous. Mailman stores every email sent and received in a .mbox file internally, but for most mailing lists, this is not accessible, and users must access archives either by vieiwing individual emails in HTML, or else downloading the tar-zipped archives for each month, which are stored in a weird format which is not documented.

For a recent project, I wanted to make a Mailman list’s archive searchable, as it contained a great knowledgebase. While many lists are public, these contained potentially private, sensitive information about my university, thus it is not possible to simply use Google to index and search for us.

After a weekend of reverse engineering a number of different email formats that Mailman seems to store archives as, we now have a workin Mailman archive parser, which can be used easily, creatively named mailman-downloader. The script downloads and decodes the archives, searching and replacing email mimetypes which Mailman scrubs, inserting mimetype boundaries and truncating these boundaries as they overflow. Author’s email addresses are reformed from the obfuscated “foo at bar dot com” form.

The heart of the decoder:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

        with open(mboxFileName,'w') as outMBox:
            resp = br.open(url)  # get response object
            lines = resp.read().splitlines()  # we actually need the full file, must buffer into memory. 
            lineNum = 0
            while lineNum < len(lines):
                line = lines[lineNum]
                if line.find("From ") == 0: #There are multiple From fields
                    line = line.replace(" at ", "@")
                elif line.find("From: ") == 0: #There are multiple From fields
                    line = line.replace(" at ", "@")
                elif line.find("Message-ID: ") == 0:
                    messageid_stripped = line[line.find('<')+1:line.rfind('>')]
                    messageid_stripped = messageid_stripped.replace('@','')
                    messageid_stripped = messageid_stripped.replace('.','')
                    messageid_stripped = messageid_stripped[0:55]
                    if lineNum +2 < len(lines) and lines[lineNum+2].find('--') ==0:
                        boundary = lines[lineNum+2][2:] #readahead and find the
                        line = line +"\n" + 'Content-Type: multipart/mixed;boundary="%s"' % boundary
                    else:
                        pass
                elif line.find("-------------- next part --------------") == 0:  # some messages have this crap
                    line = None
                elif line.find("Skipped content of type") == 0:  # some messages have this crap
                    line = None
                if line != None:
                    outMBox.write(line + "\n")
                lineNum +=1
        return overWrote, False

In the repository, a simple script to upload the downloaded archives to gmail is included, fully automating the process of creating a gmail-based full-text archive searcher, more to come on that soon.

Any questions, comments? Please file an issue and or submit a pull request.

I have recently been working heavily with MATLAB for an introduction to Microeletronic Circuits course at Olin College, where we explore the properties and behavior of transistors. The class has a significant lab component, which means I am preparing a large number of different figures using MATLAB. In the process, I have found a number of pain points: The MATLAB default fonts are far too small to be legible when embedded in a LaTeX document. When calling saveas on a figure, the image produced is the same size as whatever the frame of the current figure is.

1
2
3
4
5

function doplot(path_to_mfile,figure_folder_path,figure_name)
    newfigure();
    run(path_to_mfile);
    saveformatfig(figure_folder_path,figure_name);
end

There are gender related issues in the software industry, in startup communities, and in many different places. Just looking at HackerNews and the twittersphere you can see how popular the “Gangbang Interviews” and “Bikini Shots”: Silicon Valley’s Brogrammer Problem article is.

This topic is well written about and I have no new value to add to the wealth of information and brilliant analysis. These issues run deep, they are deeply ingrained in the social context in which we are educated and live in, sometimes they are very blatant (center image), sometimes they are less obvious (but no less significant (left and right)).

What I want to talk about is how we describe these issues, and how we frame these issues.

Currently the way that the problem is phrased, men are immediately made defensive. And you can’t blame them (even if you can, you must will yourself to understand why they feel this way). Men feel attacked, because in essence, their wrongs are being pointed out– men are being scolded, publicly and with great humiliation. Sponsorship is removed and public twitter spates occur, where men are (rightly so) told how wrong they are, and often times in ways that seem like personal attacks. Often times actions are pointed out as being sexist or derisive, and the protagonist of these actions obviously feels as though those same characteristics are being bestowed upon him. Hell, sometimes they are intentionally bestowed upon the protoganist:

@csanz @rekatz and even by allowing it to be posted without making it clear it isn’t your brand or your values- you can do better.
— Shanley Kane (@shanley) March 22, 2012

Our current approach accomplishes nothing

If men get defensive, they will band together and get more defensive, and this solves nothing (actually, this is largely what caused the issue in the beginning). What we need to do is reframe the problem, we need to make it easy for men to agree with the issues being pointed out. We need to work to understand how we can discuss these issues in a way such that men do not feel attacked. Not because we want to be ‘nice’ or ‘gentle’ towards men, but because we want to see change. We want these gender issues to be addressed, and in order for that to happen we must have men acknowledge it and think about it, and change it. (And women too!) I don’t quite know how yet, but I know that there are many smart people out there that can invent great ways to communicate this issue such that we boys/men don’t feel attacked.

*Note that in these musings, I refer to the group which must be driven to action as men, while there is of course a variety of different genders that actually need to be driven to action on the issue.

Oh how different the story could have been:

The DOD was blind while hostile forces attacked our country. Silently and invisibly, foreign forces moved troops and supplies during dusk, coordinating a devastating attack. Lacking funding to upgrade and retrofit their satellites’ telescopes, the DOD was unable to see and predict the attack, leaving us blind. Meanwhile, NASA stared deep into the galaxy, drawing the lines of constellations and guessing about the origins of the universe.

I just read a very interesting article by Alexis Madrigal describing how “our military had two, unflown, better-than-Hubble space telescopes just sitting around.”

Reading the article and commentary from a wide variety of sources, there was a discernible trend: many individuals (the author included) felt that the distribution of funding indicated by the DOD’s surplus was troubling. At first, I agreed entirely. It seemed strange that our economic environment allowed NASA to be struggling for funding whilst another agency seemed to be wasting budget. The distribution seemed to imply that we did not wish to focus on research and rather upon war. Further, it begged to question just how powerful the DOD’s other telescopes could be, if these held such small value that they could simply be gifted.

But then, I stopped. And after a moment, I realized that a very different story could have been told. Imagine the opposite was true (as many people implied). Imagine that as NASA was stargazing (of course, this is not entirely what they do), our nation was attacked because the DOD did not have access to the equipment it needed. What would HackerNew’s and the public’s reaction be in that case? It is hard to imagine anything but outrage.

While it may be the case that we are dealing with two economic and thus moral extremes, I think it quite simply illustrates that fundamentally we are conflicted in the distribution of resources, but that it is a very difficult task. On the one hand there are things both tangible and intangible to be discovered in the cosmos (planetary bodies, or deeper understanding of our physical universe), whilst on the other there is a looming unknown, one which can cause great harm.

If it is life or stargazing, I think I may just have to choose life. This one time.

By now, I assume many of your have heard about Kony 2012 and have likely seen the exceptionally well made video, on YouTube.

After watching the viral video, I found myself motivated to “make a difference,” but I paused and asked myself what kind of difference, and how? Why is it that this video was so appealing to me, why was it so profoundly motivating and so quickly gaining momentum? And finally, what was this organization actually striving to do?

The first question is on the surface, very easily answered: the movie appealed to my sentiments. But many things appeal to my sentiments, many movements are aligned with my concept of morality, why this one? The answer, I quickly found, is that this campaign is exceptionally well marketed. This video has a number of high profile musical tracks, has been well promoted and is simply put, a work of art. While such works of art are incredible at raising awareness, I began to question what it is that this group actually does.

After some quick research, I found a number of sites that have done real research into the group, and have found some things that I believe should be significant in influencing the public’s opinion of this emotional, well intended campaign.

I first found a picture of the founders of Kony 2012 posing with weapons, with what appear to be members of the Sudan People’s Liberation Army:

I then decided to investigate how this organization used their funds; finding an audit in 2011 by Considine Considine. Last year, the organization spent $8,676,614. Only some 30% went to their charity, ‘Direct Services’. More than $1 million was spent on salaries and compensation, with ‘Production Costs’ and ‘Travel’ constituting almost $2 million (see page 7). These figures are, to say the least, less than spectacular, with Charity Navigator awarding them two out of four stars for transparency( Charity Navigator).

Am I saying that this organization is bad or wrong? No, but what I am saying is simply this: think twice before buying posters, bracelets and cross posting material from Kony 2012, make sure that not only does their cause interest you, but also ensure that what they do aligns with what you believe.

Read more: http://visiblechildren.tumblr.com.nyud.net/