Intro to Machine Learning

for the Arts

Blog Post: 10/06

Using Interactive Data


Response to Fiebrink’s talk can be found here:


In my Designing For Live Performances Class I have used PoseNet so that a dancer could generate music based off her movements.

Through this project, I became really interested in the importance of bodily movement. Specifically, I started to think a lot about embodied cognition, or this idea that our actions could control/influence  our experiences.

Dreaming Up:

Based off those thoughts, I thought it would be really interesting to use a user's movement for a performance art installation. (I understand that the assignment was to dream up a possibility for a audio/visual performance, however, I find myself disinterested in creating a light show or a concert. 

In this performance art installation, the user's movement would be able to control the progression of a mini movie/video playing on a TV screen. Based of their position, they would be able to rewind, fast forward, or pause the movie. The movie would be the story of a couple. In the beginning, there would be shots of them falling in love, this would continue into the development of their relationship, and then ultimately their break up. The user could choose based off their movements whether to start the story for the beginning, keep it paused on the happy moments the couple had, or leave it at their break-up. It would be an exploration of control over time and the human experience.

Production Assignment:

For this week's assignment, I wanted to break this idea down a bit and experiment with the elements. Specifically, I was curious if the user should be able to slow down or speed up the narrative. Or if it should be the user's facial expressions and not their bodily movement controlling the narrative. Since I did not have a narrative down yet, I decided to a pick a song. I found myself inspired by this Youtube video:

I found it fascinating how much the song, in particular it's tone and meaning, changed once slowed down. The concept related back to this idea of physical control of abstract experience that I was discussing earlier. So, I decided to make my assignment for this week exactly that!

In this, based off a user's facial expression they can slow "Baby" by Justin Bieber down and have it at a normal speed.  In this cased, sad faces slow it down and normal faces keep it the same.


Here is me in the process of making my code. As you can see, I first used an if else statement rather than just inserting the facial interactive data into the speed function. However, this was just to test to make sure the numbers were coming in correctly. I later changed to have the facial interactive data go directly into the speed function influencing the speed of the song.

Here's it in action!

Blog Post: 9/25

Data Set

Here is my data set

For this assignment, I choose to download my social media data (a very scary thought since I have had all my social media accounts since I was about 13)


For this assignment, I will be discussing my Twitter data.


The data is different pieces of information about my presence on the app. This ranges from my profile to more specific pieces of content such as the ads that I engage with, the ads I look at, my likes, my flowers the amount of times I have changed my email address.

Here are the names of all the types of data.


Since there are so many files, I will be only discussing the “ad impressions” data file. The data is in a JSON format. I am not really sure how to determine the dimensions of a data set that is in JSON. But the data does seem to be organized  based off “device information”,  “promoted tweet info”, “display location”, advertiser info”, “impression time" and “matched targeting criteria”.





I think the data set is pretty healthy and clean; however, at some points that the matched Targeting criteria becomes creepily ambiguous. For instance for a Samsung ad, the targeting type  just says “tailored audiences” and the value just says “holdout”. In other ads, the type will say “keywords” and the value will be like “technology” which is far more transparent.


The data was collected for two reasons. It was formatted this way for me, so that Twitter could be more transparent with their users ( in response to the horrible event that happened with Facebook and Cambridge Analytical).


However, originally, Twitter probably was collecting this data to give to companies to show how successful their ads on this site are (specifically how many people they were able to make an impression on). From then, I am sure the company took the data to analyze and improve their marketing tactics for ads on Twitter.


While the data has an id number for each ad, the id number is meaningless since it does not actually link to the ad. So I have no idea what the ad looked like. I think it would be helpful to see the ad so that I can analyze any other trends.


It could be interesting to take all the “targeting criteria” and train a model to make an inference on who I am and my other interests. Based off what it knows about selective key words, could it guess what else I could be interested in? I know there are models that already do that but it still would be cool to see!

Blog Post: 9/10

Image Classification

Blog Post: 9/10

Image Classification

ImageNet is layered with biases. For example, when I searched “man”, I was recommended sharks rather than a human man. When I opened up the “shark” page, the description of shark states “Large aggressive shark widespread in warm seas; know to attack humans”. I was surprised to see description that was so opinion heavy rather than based on scientific definitions. 


The Wikipedia page on sharks, on the other hand, does not even mention attacks until the ninth section and until then refrains to purely scientific descriptions of the animals.Since the shark data is so heavily coated with biases it possible other data sets in this database are as well, raising possible ethical concern. 


Also, the word choice for voting if a photo was an accurate match or not was also really interesting (or perhaps a better word would be strange). ImageNet chose the word “Typical” for accurate matches and “Wrong” for incorrect matches. Typical and wrong feel like such weighted words, and I am curious why they did not just opt for “Correct” versus “Incorrect”.


I am not sure how I feel about the fact that ImageNet does not own any of the images. I understand that it is similar to any search engine, however, I did not know ImageNet existed until now. While I know things can end up on Google, it is uncomfortable to think about the various other sites that are using my images without my direct knowledge.



When playing around with the ml5 examples, someone next to me pointed out that I was interacting with the image classification example in a way that was quite similar to how I would interact with a baby. I would show it a water bottle and ask “do you know what this is” and get way too excited when it got it right. I started to wonder why I thought of it like a child and realized that it was because it got a lot of the objects wrong. Since it’s knowledge was limited and made a lot of errors, I was not scared of it and I guess unknowingly connected to something child-like. I wonder what that says about “error” with machine learning and if errors can be leveraged. Of course, outside of this tangent, the errors themselves were interesting. The image classification really relied on the object being isolated, without nothing in the background. This made me realize that classification is a really holistic recognition process.


In another tangent, I thought it was super interesting that it recognized Apple products. While the classification is outdated and thought all Apple products were iPods, I thought it was super interesting that it called my iPhone and even my AirPods an “iPod”. I guess this points to how great the system design of apple products are.  

Blog Post: 9/15

Teaching Machine

I found the Teaching Machine to be a great tool for understanding the relationship between data and a machine learning model. In class, we discussed how the models are based on pixel organization. I did not really understand what that meant until I was playing around with my p5 sketch. In my sketch, when I "drink water from a bottle" I should receive feedback. In my mind, that meant that whenever the camera "sees" the bottle on my mouth it would determine that I was "drinking water". However, as I played around, I realized that just moving my arm towards my face, even with no bottle in hand, triggered the model to think I was drinking water. So the model was not focusing on the bottle, but instead pixels around my shoulders and face-- determining if my arm/hand was in that area. This realization provided me a great deal of insight on how pixels are read by the model.

Here is what I made and here is the link to the p5 sketch:

I did face a weird issue with one of my models. I wanted to train a model to recognize signs in American Sign Language. However, some reason when I would bring that model link into p5 it simply did not work. I would get nothing in the console.

Blog Post: 9/6

Shower Temperature AI

Page 23:





Page 24:

In addition to suggesting how I should reply to an email, Gmail will now auto finish my sentences for me.


Page 25:

Email Inbox: Predictive Text

Check depositing: The app will take the photo of the check for me, I do not even need to click “capture”

Texting: Predictive Text

Netflix: Recommendations and ordering of content

Google: Recommendations and ordering of content

Social Media: Recommendations and ordering of content

Automated message systems: I am not sure what this is referencing? Like spam calls?



Page 26:

What do we gain from AI: Technology becomes a lot easier to use and understand

What do we lose: Our privacy and agency


Page 27:

Task At Hand: Getting the Shower Temperature Just right for both my desires and my health (sometimes I take showers that are too warm for my sensitive skin— especially in the winter)


How Can AI Help: By “keeping track of my habits and health” to determine what kind of temperature I would like in the shower/what is also best for my health


What Role do Humans have: Providing health and habit data


What Data do I need: Daily habit data, weather data, home temperature data, health issue data


How will responsibly gather the data: The weather and home temperature data are not as invasive as the daily habit and health issue data. I could responsibly collect the health issue data by only relying on what the users provide me and consent to giving. I believe I can implement a similar plan of action for the daily habit data (by only taking what is given and not collecting other pieces)



Page 28:



  1. Based on data collected during user testing, the machine is already set to meet “average” human temperature needs (controlled to be without health issues/sensitivity)

  2. Users can input data about specific health issues and sensitivity, adjusting the AI accordingly

  3. User can input temperature preferences (ex: I prefer it to be warm in the morning and cold at night)

  4. As users use the shower, and adjust the shower temperature by themselves, the system collects this data

  5. AI starts regulating the temperature based off additionally changes it is seeing

  6. User continues to provide inputs on comfort

  7. AI continues to iterate and adjust



Page 36:

  1. a. Describe a personal algorithm you use in your life:

Autocomplete emails with predictive text

b. Describe another algorithm you could use to do the same task:

An algorithm that provides your template responses once you input what kind of email you are writing

c. Which of these two is most efficient? 

The first one because it is more specific towards how I would complete the email and my current needs


  1. a. Describe a personal algorithm you use in your life:

Netflix suggesting content

b. Describe another algorithm you could use to do the same task: 

An algorithm that shows you what people around you are watching the most

c. Which of these two is most efficient? 

The first one because once again it prioritizes my current needs


  1. a. Describe a personal algorithm you use in your life:

The auto-select feature on Photoshop

b. Describe another algorithm you could use to do the same task:

I could use the more basic tool and outline the object I want to select by hand

c. Which of these two is most efficient? 

The first one because it reduces the amount of time it will take me to do one small task



Page 38:


Data to figure out if someone is fun: how positively their close friends feel about the, how positively their colleagues think about the,  how positively strangers would rate them, how long they are able to hold a conversation on average, amount of friends they have, their serotonin levels, the serotonin levels of people around them when in their company, 


Data to figure out if someone is interesting: the number of hobbies they have, the number of times they leave their home, the amount of schooling they have done, the number of jobs they have held, the number of places they have visited if they participate in extracurriculars,



Page 39:

Fun people data set:

  1. Friend groups — positive feelings

  2. In public social settings — positive feelings

  3. Serotonin levels — higher levels 

  4. Workplace — positive feelings


Interesting people data set:

  1. Hobbies — the higher number indicates more depth

  2. Travel destination — the higher number indicates more depth

  3. Clubs — higher number indicates more depth

  4. Outside of residence — a higher number of times leaving indicates higher levels of interesting

Page 40:

  1. Assigns weights to each data point

  2. Assign weighted data to each person based on how many data points they meet

  3. Those with the highest scores are the most interesting


Page 41:

I am not sure, but it feels like fun and interesting are so culturally subjective. So perhaps I forgot to take in different cultural perspectives. It just feels like an algorithm should be more complicated!