On the rise of happiness

Happy people are more successful: they live longer lives, lose weight faster, have steadier relationships and on and on and on… oh and they have prettier pictures on Instagram. Sigh.

Are you happy? Just the thought of that question could be the beginning of a downward spiral: “Wait, why am I not happy? Why am I not happy that I am not happy? Why do I catch myself every time in this unhappy state? Maybe I’m depressed. I just want to be happy”.

In fact, what exactly is happiness? No, not the definition of happiness by others but happiness to you. Who gets to define it and why. Why do almost all self-help books and productivity blogs, and countless tips on Quora end with “be happy”?

The challenge with this happiness story is the propagation of the idea of a never-broken happiness streak — you must always be happy no matter what, no matter when. Truth is there’s nothing like continuous happiness. In fact, it is against the core of being human. If you are always happy then it means no room for unhappiness which invariably means we cannot empathize with others in sorrow. Yet, empathy is what makes us human.

Experiencing different emotions–both positive or negative in high or low magnitude–is what defines life. Otherwise, how can you enjoy the fullness of a happy state if you’ve never experienced any unhappy moment. The experience of an unhappy state is what makes one appreciate the delta of any happiness when it occurs.

So yes, be happy AND don’t obsess when you are not because there’s no such thing as perpetual happiness. If somehow you catch yourself thinking “why am I unhappy”, “why am I unhappy that I’m unhappy” then immediately add “screw it, I’m good” and move on with your life. There’s no best happiness state. There’s just you, your struggles and your attempt at understanding the world. Move on.

Who gets to label the problems?

Richard Heeks in his seminal paper emphasizes the evolution of ICTD and the major differences between ICT4D 1.0 and 2.0. ICT4D stands for information and communication technologies for international development.

In differentiating both versions, the former, ICT4D 1.0, focused on providing solutions that ICTD researchers felt were sufficient for the people — simply ‘copy’ and ‘paste’ solutions from parts of the world where it worked and it’ll certainly work in your target population, while the latter concentrated on involving the people for whom the technological intervention is designed. In retrospect, individuals who pushed 1.0 should not be heavily criticized for they had to start from somewhere, and they laid the foundation for 2.0 researchers to step back and objectively observe their actions. This back and forth is what indeed helps formulate and develop new research agenda. However, there are still a few salient factors that are yet to be addressed in both ICTD versions.

Who gets to choose what ICTD problems should be solved? In fact, who gets to label diverse phenomena as ICTD-worthy issues? Is it the government or the media, top researchers, big corporations or rising entrepreneurs? History has showed on many occasions that humans are cognitively biased and ultimately driven by selfish measures — it is arguable that altruism truly exists in any human interventions. But without delving into philosophical arguments on terminologies, we, as ICTD researchers, need to step further back to the why. It is imperative that we observe the lens with which world phenomena are viewed as ICTD problems because any interventions made will leave an indelible mark on the target communities, good or bad. If anyone can label any circumstance as a problem that needs to be solved and jump into communities to run experiments or build technologies, then who holds the innovator responsible for effects meted on the lives of inhabitants of the communities?

Predicting power dynamics with Machine Learning on emails (PART 2)


PART 2: predicting power dynamics with Machine Learning


machine_learning

Cont’d from PART 1.

Now we know what major words were used in the Enron dataset. So if you saw a random email and tried to guess if it was upspeak or downspeak, a simple rule is check if “Shirley” or “Enron” appears. If there, then it’s downspeak, otherwise it’s upspeak. That simple.

But what about downspeak emails without “Shirley” or “Enron”?
For instance, one downspeak email talks about interviewing someone but “Shirley” or “Enron” didn’t appear. We definitely need something better. This is where Machine Learning algorithms come in.
Process Summary
  • Given a dataset of sent emails with labels “upspeak/downspeak”, let’s use the words in each email as features, “upspeak/downspeak” as binary labels, and trained with 3 different ML classifiers that appeal to me: naive bayes, SVM, and decision tree. There was no intuition involved in selecting the 3 classifiers used; they were chosen in order to get a feel of machine learning predictions.

Process Details & Python Programming

  • Enron email containing labels for upspeak/downspeak. Upspeak means the email was from someone of lower hierarchy in the company to someone of higher rank; downspeak means email was sent from a higher ranked person to a lower ranked person. There is no detail about the level of rankings between both parties, except that one is higher and the other is lower.
  • Reading the file involved stripping “\n” from every line. While reading every line, I aggregated all upspeak messages to one python list and all downspeak messages to another list.
  • Using bag of words approach from this online tutorial, I created features from the upspeak and downspeak python lists. Bag of words simply creates a dictionary presence of a word. For instance, the default case for “You are awesome” becomes {“You”: True, “are”: True, “Awesome”:True}. Beyond this, I created a variation of bag of words using same approach but eliminating stop words. So using the same example, result is: {“Awesome”: True}. Then a few more variations described below were made.
  • From the labelled data, training set (90%) and test set (10%) were made. Using “all words” as features, I got 82% accuracy with NaiveBayes classifier. Inspired by Danco’s iterative ML process in the last class, I decided to tweak for possible improved accuracy.

Let’s improve accuracy by removing stopwords

  • Digging more into NLTK classification tutorials, I imported this file to create better features. First I started with “bag_of_non_stopwords”, which eliminated all English stopwords i.e. high frequency words such as the, I, to, etc. The performance improved by about 4%. Neat!
  • Next, I ran the algorithm 5 times to see the overall performance but the result was disappointing: the performance seemed to fluctuate — sometimes better, other times worse than “bag_of_words”. Looking deeper into my code and it was evident that the performance would fluctuate since features are shuffled every time a classifier is trained.
  • One quick fix is to average out accuracy after n number of runs. I chose 10.

Let’s tweak the features and add other classifiers

  • I added more variations of features such as “bag_of_bigrams”, “bag_of_trigrams” while including more classifiers such as DecisionTree, and SVM. All of these were tested over 10 iterations.
  • Somehow, the performance for Naive Bayes appeared to be the same for different n-grams with NaiveBayes bigrams having the best output. I expected non stopwords to be the best because stopwords seemed like ‘text noise’ but I’m still not sure why Bigram outperformed others; perhaps, overfitting is at work in the other algorithms. 
  • Investigating other classifiers was rather disappointing. The first shocking result was that my program was stuck — the accuracy results wouldn’t print upon using SVM or Decision Tree.

Decision Tree/SVM: takes longer to compute and gives poor accuracy

  • Naive Bayes ran in 30 seconds but the other classifiers — DecisionTree and SVM — were still ongoing after 5 minutes even though the training set had been reduced from 90% to 10% of the featureset. Looking more at the theory of these classifiers, it makes sense that DecisionTree takes a long time since “best features” have to be selected after computing information gain. But each computation takes a long time as one goes down the tree. One solution is to reduce the training set size but this might lead to underfitting as the classifier wouldn’t have been trained enough.
  • SVM finds a hyperplane and calculates the maximum margin between support vectors (closest points to the hyperplane). However, I’m still missing the connection about why it takes so long.

Results

Bag_of_words/Classifier Naive Bayes SVM* Decision Tree*
all_words 0.8021 0.5121 0.6477
non_stopwords 0.8181 0.5225 0.6405
bigram_words 0.8307 0.5092 0.6375**
trigram_words 0.8213 0.5026 0.6412**

*training set was reduced from 90% to 10%

**still running even after 15 minutes so had to be terminated

From the data above, bigram_words for Naive Bayes classifier performs best while trigram_words for SVM performs the least. The result also shows that for text classification Naive Bayes is fast and performs reasonably well.

Challenges

  • This NLTK tutorial was helpful until I got to the part of making my own corpus with two labels: pos and neg. After some good time spent, I eventually figured how to make a corpus but not how to assign “categories”. Googling around didn’t seem too helpful either.
  • This tutorial conflicted with the initial tutorial in the creation of feature set that a classifier expects. Even this bitbucket code where I got the bag_of_words functions, made things worse when trying to use their labelled feature set function. The different sources seemed to contradict each other so I burned several hours trying to figure out why I had bugs.
Ethical concerns
  • The findings in this study will be retracted if the original email senders feel that their privacy has been bridged. Although this is highly unlikely as the Enron email dataset has undergone multiple privacy protection process.
  • Extending the methods applied here to personal emails should be done with care. Just because one can analyze their email does not mean your findings can be published to the public. For instance, analysis of incoming emails could show choice of words that a frequent sender emailing you would rather keep these words between you and them.
  • One simple rule of thumb is to fill an Institution Review Board (IRB) form. If not applicable to you then discuss with the parties involved what data you want to collect, the forms of analysis involved, and what they allow you to share, if any.
Lessons
This is a combination of lessons learned from implementing code, utilizing libraries and reporting results:
  • When available, use qualitative methods to better understand results of quantitative analysis. This is useful for sanity checks and it is necessary regardless of your results. Qualitative methods could be asking your participants what they think about your research procedure or the results obtained, whether it agrees with your intuition and why.
  • Machine Learning is many times common sense even though the buzz can many times make it seem complicated. Okay, not entirely true, the mathematical theory still seems complicated for now.
  • Implementing algorithms helps to better understand the theory behind the work.
  • In making a classifer for NLTK, your training data should be a python list of tuples i.e. [ (feature, label), (feature, label), (feature, label)]. Then each tuple should be of the format (feature, label) — plus your feature as a python dictionary: feature_name: feature_value}.
  • Using python ML classifiers is easy once you get past formatting your dataset into features.
Limitations & Future Work
  • Extend the definition of “power dynamics” beyond organizational hierarchy to employee requests especially in decentralized work places. For instance, if in your email you ask a favor from a fellow employee at a startup then that email is upbeat. It doesn’t matter if hierarchically, you’re above the person, on the same level, or below.
  • Use minor words as features for training Machine Learning classifiers.
  • The distribution of the data clearly affects the Machine Learning performance. In this work, the dataset collected was sampled from a bigger email collection. However, there are no details about the sampling process. As such, the machine learning classifiers could be prone to overfitting and consequently perform poorly in prediction when new data is encountered.
  • Performing k-cross validation should be used for perhaps increased accuracy.
Okay, I’ve read/skimmed all these, so what? How does this apply to me
  • From a particular timeline (maybe 6 months), copy a set of your outgoing emails into a file and as you deem fit, label them upspeak or downspeak (same format as training.txt).
  • Save your file as “training.txt” in order to replace the current file.
  • python enron_analysis.py
  • If you are a little ambitious, tweak the code so that you have a different test file (instead of the current way of splitting training data into training and test).
  • If you did the previous step, then your prediction could give you simple insights about how much upspeak/downspeak you perform. You may start linking this to actual events in your life i.e. how you felt during the period of sending those labelled emails.
  • Hooray, you have built a hypothesis function that can classify any new outgoing email as upspeak or downspeak. Go forth and predict all over the world!

Tools used

  • R
  • Python
  • NLTK
  • Wordle

Files

References

Predicting power dynamics with Machine Learning on emails (PART 1)

Target Audience

PART 1 targets general public while PART 2 is for a more technical audience who have at least heard of “Machine Learning classifiers”.

Background

We either affect or are affected by people. You influence your family at home and/or your colleagues at work. At home, it’s perhaps the way you roll your eyes when your sibling says something you absolutely dislike; stare your kids down when they perform an action you disprove; or scream in excitement when your team snatches a touchdown in Super Bowl. At work, it’s probably the way you pack your bags at the end of a workday; attach sticky notes to your monitor as task reminders; or even slowly exit a boring conversation to return to your desk. In all of these, your actions either affect people or you are affected by their actions. Both could concurrently happen. There are countless fascinating examples that capture daily influence but let’s focus on one: written words.

Consider for a moment all your written conversations — this could be text messaging, Facebook chat, WhatsApp, Viber, Email, you-get-the-point. What kind of words appear the most? If you selected your top 3 words across all messages, what will they be? Perhaps, your word choice changes with different platforms and even people on the other communication end. Clearly, the way you write to your boss is different from how you write to your friend in a distant country. Let’s focus on your writing at work since for the average person, more time is spent at work than at home. One easy way to capture your work writing is through email. Does your email at work differ across employees?

By analyzing your email, you could discover the emotions expressed through your words and see how they change over time; you may discover consistent boot-licking every time you email your boss; or you find out that you always use “please” to forcibly sound polite. In this work, we investigate what kinds of words are used in a set of emails at a company, Enron. Then we predict if an email was sent by an employee of higher or lower organization hierarchy.

Big Question:

Given a work environment, what can we infer about power dynamics in written conversations based on usage of minor and major words?

In Natural Language Processing (NLP), minor words are stop words; these words have no meaning when considered alone and they are usually prepositions and conjunctions. On the other hand, major words are words with standalone meanings and they include nouns, adjectives, and verbs.

For instance:
Jane is awesome and loves to go shopping.
Minor words: “is, and, to”;
Major words: “Jane, awesome, loves, go, shopping”.

Small Questions:

  • How do major words frequency compare between upspeak and downspeak language corpus.
  • Using major words, how accurately can we predict power dynamics in an outgoing email i.e. is the email from higher employee to lower employee or vice-versa?

Upspeak email: from employee of lower ranking to one of higher ranking.
Downspeak email: from employee of higher ranking to one of lower ranking.

Power dynamics can be defined in multiple ways but our focus is on organizational hierarchy. So if a Vice President sends an email to a secretary then it is downspeak; when the secretary emails the Vice President then it is upspeak. Even though in reality, you have more than binary email labels, for simplicity, every email in the dataset have been labelled as either downspeak or upspeak.

Where is your dataset coming from?

  • Enron email dataset
 While there are multiple data streams to answer the small questions, this dataset was chosen simply for convenience and availability. The data has already been cleaned and properly labelled with the format as:

<label>
**START**
<email words>
**EOM**

where <label> is either upspeak or downspeak and <email content> is the actual email sent. **START** and **EOM** (end of message) represent the beginning and end of the email words. This pattern is repeated for every line throughout the document.


PART 1: major words comparison for upspeak and downspeak


Visualization

upspeak
upspeak

downspeak
downspeak

From the words above, the size grows bigger as the frequency of the word increases. That means for upspeak, “Please” is used more than “new” while for downspeak, “meeting” is used more than “see”.

But why do majority of the emails sent as upspeak or downspeak have the word “please” or “Please”. Since wordle is case-sensitive, we can infer that “Please” starts most sentences while “please” is used in between words.

What do you think is happening?

It appears Enron employees try to be polite in their emails whether upspeak or downspeak. Interestingly, we see that upspeak have more “call”, “information” and “questions” while downspeak has “Enron”. We may conclude that when people send emails to others of higher status, they are always seeking to have a meeting or they have questions that should be answered so maybe a “call” would help. Perhaps, managers emailing their department always stress the goals of Enron so that the words appear in only downspeak. Or maybe when people of higher hierarchy send their emails, their signature has their title and the company name “Enron”. But who/what is “Shirley”? Maybe it’s just “noise” in the text?

Let’s jump into the actual emails.

What is actually happening

In upspeak, “please” shows up in “please let me know if you have any questions”, “if you have any questions give me a call” written in multiple variations. This is not surprising as most companies want employees to be as polite as possible especially when communicating with clients. But we’re not entirely off.

In downspeak, “please” appears in “please let …. know about this information”, “please be at this meeting… I have not received many responses, if you have any questions let me know”. Same word, slightly different usage.

Turns out Shirley is the point of contact –most probably HR Head– when a new employee has been interviewed and ready to join “Enron”. Same contact is used when compiling event expenses, inviting people to Enron, passing information across teams, etc. Since a lot of these emails happen, what you have is a bolded “Shirley” and the word “Enron”.

Now we know what major words were used in the Enron dataset. So if you saw a random email and tried to guess if it was upspeak or downspeak, a simple rule is check if “Shirley” or “Enron” appears. If there, then it’s downspeak, otherwise it’s upspeak. That simple.
But what if it is a downspeak email and “Shirley” or “Enron” isn’t present? If there are many emails like this then we’d have too many errors. Can we predict better? Yes, and this is where Machine Learning takes the dance floor. The next section has more technical details in order to aid replication or extension to other applications. If not interested in the technical explanation, skim through to get overall goals. See you in PART 2.
References

Marriage conflict and face-to-face synchronicity

Introduction to terminologies

marriage-conflictFace-to-Face (F2F): people facing each other
Synchronicity: (informal definition) moving in tandem; you just ‘click’.
Synchronicity: (webster definition) “the coincidental occurrence of events and especially psychic events (as similar thoughts in widely separated persons or a mental image of an unexpected event before it happens) that seem related but are not explained by conventional mechanisms of causality —used especially in the psychology of C. G. Jung” [1]

F2F synchronicity: this is how people take talk turn-by-turn when involved in a face-to-face conversation. This simple art can provide insightful feedbacks about people’s feelings during conversations especially conflict conversations.

Imagine for a minute that you are involved in an argument with someone — perhaps a close friend, your best friend, boss or significant other. Do you think your F2F synchronicity and facial expressions significantly change compared to when you’re catching up over a delicious-free-getaway dinner with red wine? (It’s okay if you still can’t imagine anything provided you get the point; you might want to re-read if you don’t.)

Background for question of interest:

Your best friend, a marriage counsellor, has just texted you saying that you should catch up at their place. You get there and after walking you to the living room, you’re left to a laptop playing a short movie while your friend fetches you lemon drink. The movie has a couple talking with each other but you can’t hear the sound because of bad audio. Your friend comes back saying “oh, I’m trying to infer the probability of a divorce based on their facial expressions and body movements.”  “Oh wow, how?”, you ask. You definitely want in on this now because by now you already know about F2F synchronicity and facial expressions. What signs do you look for?

Question of interest:

If given a collection of videos showing couples engaged in a marriage conflict session, can you infer their emotional states by observing their F2F synchronicity and facial expressions?

Possible signs:

You decide to start with yourself as an example. Hmmm, how do your F2F synchronicity and facial expressions change when you’re arguing (read: in conflict) compared to when you are collaborating. Perhaps, you may:

  • talk while the other person talks,
  • interrupt more often
  • make more negative facial expressions — disgust, fear, anger, etc
  • make increasing negative comments with time [2]
  • make less positive comments
  • have less eye contact by completely looking away etc

What other unlisted signs can you think of?

Disclaimer: I am not a marriage expert, neither do I in any way give marital counseling —  in fact I am not married! I just find some of the literature about conflicts interesting and since marriage conflicts have been long studied, insights from this area could be used in other non-related aspects of life.

References

  1. http://www.merriam-webster.com/dictionary/synchronicity
  2. Gottman, John Mordechai, and Robert Wayne Levenson. “The timing of divorce: predicting when a couple will divorce over a 14‐year period.” Journal of Marriage and Family 62.3 (2000): 737-745.
  3. Image source: http://www.feelmorebetter.com/marital-conflict/

‘Hello world’ with mathematical proofs – why care?

“Prove that root 2 is irrational.”
If you read the sentence above and thought “Why in the world do mathematical proofs exist?” then welcome to my world! I’ve always believed that mathematical proofs exist so professors can torment students. If you are in Engineering or precisely Computer Science or Mathematics, then chances are you have an idea of what I mean. Mindless proofs that you can’t seem to make a connection with the physical world. For instance, it is easy to see that knowing how to proving root 2 is irrational will not suddenly make you see the light in life. Or maybe it will?

Do you think proofs matter?

It is always easy to make up reasons about why proofs are pointless especially if because many proofs are hard to connect with the physical world. If you are on the extreme side of disliking proofs, then before you think up your endless reasons to avoid them, consider the few counter-points for why they matter:

Yes, proofs matter because:

  • With a proof, you are certain a claim always works.
  • Without proofs, it is sometimes difficult to find counter-examples to claims. For example, the equation n^2 + n + 41 = prime number (for all positive integers). This seems true but it is false. There’s an example that contradicts the claim. Hint: the number is less than 45.
  • Not all test-cases for a claim can be checked with a computer. For instance, 313(x^3 + y^3) = z^3 (for all positive integers), has a contradiction. This is a 1000-digit number!
  • Intuition gained from proofs can help you debug your programs if you are a programmer.

There are many more reasons. If you are interested in beginner proof cases then take a look at the book in reference section.

References

  1. http://www.cs.princeton.edu/courses/archive/spr10/cos433/mathcs.pdf
  2. http://www.faradayschools.com/favourite-bits/quick-reads/what-is-proof/
  3. http://faradayschools.com/wp-content/uploads/proof.jpg