{"id":3349,"date":"2019-05-23T19:23:31","date_gmt":"2019-05-23T17:23:31","guid":{"rendered":"https:\/\/geekosas.com\/?p=3349"},"modified":"2026-05-23T19:24:00","modified_gmt":"2026-05-23T17:24:00","slug":"uncovering-averages-part-2","status":"publish","type":"post","link":"https:\/\/geekosas.com\/index.php\/2019\/05\/23\/uncovering-averages-part-2\/","title":{"rendered":"Uncovering averages Part 2"},"content":{"rendered":"

A few days ago I wrote the article Uncovering Averages<\/a> which basically did was to open an average value into factors using trees. Please read the article before continuing.<\/p>\n

In that example analysis I created the dataset and therefore knew exactly where the change was, which was in the cause column, only for the connectivity case, but since the tree did not split on that dimension, it only left a "clue about where to look".<\/p>\n

\"\"<\/p>\n

Since this Data Science world is a mix of obsession and creativity, I kept thinking and thinking about how to make all the factors appear until I hit the nail using linear regression.<\/p>\n

The methodology is the same; what changes is that the beta corresponds to the average rating and the average of each predictor variable corresponds to the rate.<\/p>\n

First we load the datasets, which look like this:<\/p>\n

library(tidyverse)\nlibrary(broom)\nlibrary(caret)\ndata1 = readRDS('dataset1.rds')\ndata2 = readRDS('dataset2.rds')\n> head(data1)\n  id        causa genero region nota\n1  1       equipo hombre  norte    6\n2  2        saldo  mujer  norte    8\n3  3  facturacion hombre  norte    2\n4  4        saldo  mujer centro    6\n5  5 conectividad  mujer centro    9\n6  6 conectividad hombre centro    4<\/code><\/pre>\n
Then we dummy\u2011code the variables so that the categories are redundant:<\/p>\n
encoder <- dummyVars( ~ nota + causa + genero + region, data1)\ndataset1 <- predict(encoder, data1)\ndataset2 <- predict(encoder, data2)<\/code><\/pre>\n\n
\"\"<\/figure>\n\n
And we run a model for each period and use tidy<\/code> from broom<\/code> to format the regression parameters as a data frame (note that tidy<\/code> removes the parameters that could not be estimated due to linear dependence; essentially they are contrasts and the estimate is 0 \u2013 we will need that later).<\/p>\n
fit1 = lm(nota ~ .,data.frame(dataset1))\nfit2 = lm(nota ~ .,data.frame(dataset2))\nparams1 = tidy(fit1) %>% select(term,estimate)\nparams2 = tidy(fit2) %>% select(term,estimate)\nparams1\n> params1\n  term              estimate\n1 (Intercept)         7.87  \n2 causaconectividad  -1.98  \n3 causaequipo        -0.844\n4 causafacturacion   -2.88  \n5 generohombre       -1.02  \n6 regioncentro       -0.925\n7 regionnorte         0.0486<\/code><\/pre>\n
Now we calculate the proportion (average value) of each column in the dummy\u2011coded dataset:<\/p>\n
porciones1 = dataset1 %>% data.frame() %>% summarise_all(mean) %>% gather(term,porcion)\nporciones2 = dataset2 %>% data.frame() %>% summarise_all(mean) %>% gather(term,porcion)\nporciones1\n                term porcion\n1               nota   5.577\n2  causaconectividad   0.293\n3        causaequipo   0.216\n4   causafacturacion   0.101\n5         causasaldo   0.390\n6       generohombre   0.704\n7        generomujer   0.296\n8       regioncentro   0.576\n9        regionnorte   0.213\n10         regionsur   0.211<\/code><\/pre>\n
And we cross the above datasets, filling the contrast betas with 0 and for the intercept the proportion will be 1 because it affects all elements:<\/p>\n
dataset = params1 %>%\n  left_join(params2, by = "term",suffix = c("_1","_2")) %>%\n  full_join(porciones1,by = "term") %>%\n  left_join(porciones2,by = "term",suffix = c("_1","_2")) %>%\n  filter(term!="nota") %>%\n  mutate_at(vars(starts_with("e")),~replace_na(.,0)) %>%\n  mutate_at(vars(starts_with("p")),~replace_na(.,1))\n> dataset\n   term              estimate_1 estimate_2 porcion_1 porcion_2\n1 (Intercept)           7.87        7.80      1        1     \n2 causaconectividad    -1.98       -2.87      0.293    0.304\n3 causaequipo          -0.844      -0.966     0.216    0.212\n4 causafacturacion     -2.88       -2.98      0.101    0.0985\n5 generohombre         -1.02       -1.00      0.704    0.290\n6 regioncentro         -0.925      -0.864     0.576    0.586\n7 regionnorte           0.0486      0.174     0.213    0.201\n8 causasaldo            0           0         0.39     0.386\n9 generomujer           0           0         0.296    0.71  \n10 regionsur             0           0         0.211    0.213<\/code><\/pre>\n
Note that the product of the estimate column and the proportion column gives the average, so we can say we have decomposed the average into factors:<\/p>\n
sum(dataset$estimate_1 * dataset$porcion_1)\n[1] 5.577\n> mean(data1$nota)\n[1] 5.577\n> sum(dataset$estimate_2 * dataset$porcion_2)\n[1] 5.6705\n> mean(data2$nota)\n[1] 5.6705<\/code><\/pre>\n
The above holds because the sum of squared residuals in linear regression is zero.<\/p>\n
Now we use the same trick as in the previous article, calculating the change in beta and the change in the sample, and with that we determine the contributions:<\/p>\n
dataset = dataset %>%\n  mutate(\n    delta_freq = porcion_2 - porcion_1,\n    delta_nota = estimate_2 - estimate_1\n  )\ndataset = dataset %>%\n  mutate(aporte_dfreq = estimate_1 * delta_freq,\n         aporte_dnota =  porcion_2 * delta_nota\n  )<\/code><\/pre>\n\n
\"\"<\/figure>\n\n
What is interesting is that the sum of the contributions equals the change in rating:<\/p>\n
sum(dataset$aporte_dnota) + sum(dataset$aporte_dfreq)\n[1] 0.0935\n> mean(data2$nota) - mean(data1$nota)\n[1] 0.0935<\/code><\/pre>\n
Therefore we can say that we have decomposed the average, and just like in the previous result, if we look at how much the average changed due to rating and due to sample, we get the following:<\/p>\n
sum(dataset$aporte_dfreq)\n[1] 0.4012351\n> sum(dataset$aporte_dnota)\n[1] -0.3077351<\/code><\/pre>\n
Result similar to the previous article.<\/p>\n
On the other hand, going back to the contribution table, we can see that a 41% decrease in the number of men surveyed caused an increase of 0.42 in the rating, which cancels out the 0.27\u2011point decrease caused by the worse evaluation of the connectivity cases.<\/p>\n
Finally, this model seems superior to the tree one, but to perform linear regressions, certain assumptions are needed that are not required for trees; therefore, we must be a little more careful.<\/p>\n
I hope this is useful, and I\u2019ll be checking the comments to see if there\u2019s anything you want me to address.<\/p>\n
Cheers!<\/p>","protected":false},"excerpt":{"rendered":"
A few days ago I wrote the article Uncovering Averages which basically did was to open an average value into factors using trees. Please read […]<\/a><\/p>\n<\/div>","protected":false},"author":1,"featured_media":2762,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-3349","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-sin-categoria"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2019\/08\/averages.jpeg?fit=500%2C345&ssl=1","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8vjqF-S1","jetpack-related-posts":[{"id":3347,"url":"https:\/\/geekosas.com\/index.php\/2019\/05\/23\/uncovering-averages\/","url_meta":{"origin":3349,"position":0},"title":"Uncovering averages","author":"Daniel Fischer","date":"2019-05-23","format":false,"excerpt":"As they say, averages hide many things. In the article gender-pay-gap-en-tecnologia we saw an analysis that showed how, for that data, the salary difference between men and women can be explained by factors other than gender. Now we are going to look at a technique, based on machine learning, that\u2026","rel":"","context":"In "Sin categor\u00eda"","block_context":{"text":"Sin categor\u00eda","link":"https:\/\/geekosas.com\/index.php\/category\/sin-categoria\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2019\/08\/distribution.png?fit=1200%2C767&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2019\/08\/distribution.png?fit=1200%2C767&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2019\/08\/distribution.png?fit=1200%2C767&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2019\/08\/distribution.png?fit=1200%2C767&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2019\/08\/distribution.png?fit=1200%2C767&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":3339,"url":"https:\/\/geekosas.com\/index.php\/2019\/05\/23\/gender-pay-gap-in-technology\/","url_meta":{"origin":3349,"position":1},"title":"Gender Pay Gap in Technology","author":"Daniel Fischer","date":"2019-05-23","format":false,"excerpt":"The Gender Pay Gap is the difference that exists on average in the salaries of Men vs. Women. Today there are people who attribute this to discrimination, while others say it is due to the decisions that men on average make versus those of women. Since both opinions have merit,\u2026","rel":"","context":"In "Sin categor\u00eda"","block_context":{"text":"Sin categor\u00eda","link":"https:\/\/geekosas.com\/index.php\/category\/sin-categoria\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2019\/02\/GenderPayGap-201803070107196681-20180404082357920.jpg?fit=619%2C413&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2019\/02\/GenderPayGap-201803070107196681-20180404082357920.jpg?fit=619%2C413&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2019\/02\/GenderPayGap-201803070107196681-20180404082357920.jpg?fit=619%2C413&ssl=1&resize=525%2C300 1.5x"},"classes":[]},{"id":3321,"url":"https:\/\/geekosas.com\/index.php\/2018\/05\/23\/separate-effects-and-cohort-analysis\/","url_meta":{"origin":3349,"position":2},"title":"Separate Effects and Cohort Analysis","author":"Daniel Fischer","date":"2018-05-23","format":false,"excerpt":"In subscription businesses (Newspapers, Cell Phones, Insurance, etc...), the business is always the same: acquire a customer and then receive cash flows associated with a service provided by the company. The day the customer cancels the service is called CHURN, and the customer becomes inactive, suspending both revenue and service.\u2026","rel":"","context":"In "Sin categor\u00eda"","block_context":{"text":"Sin categor\u00eda","link":"https:\/\/geekosas.com\/index.php\/category\/sin-categoria\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.geekosas.com\/wp-content\/uploads\/2018\/06\/pressent-value.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.geekosas.com\/wp-content\/uploads\/2018\/06\/pressent-value.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/www.geekosas.com\/wp-content\/uploads\/2018\/06\/pressent-value.png?resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.geekosas.com\/wp-content\/uploads\/2018\/06\/pressent-value.png?resize=700%2C400 2x"},"classes":[]},{"id":3319,"url":"https:\/\/geekosas.com\/index.php\/2018\/05\/23\/have-video-games-gotten-worse\/","url_meta":{"origin":3349,"position":3},"title":"Have video games gotten worse?","author":"Daniel Fischer","date":"2018-05-23","format":false,"excerpt":"Introduction \/ Abstract A data scientist is one who manages to make data speak to them; it is basically a conversation, where you ask questions and the data answers. In this notebook I want to share my latest conversation with this dataset of scores assigned to different video games. The\u2026","rel":"","context":"In "Sin categor\u00eda"","block_context":{"text":"Sin categor\u00eda","link":"https:\/\/geekosas.com\/index.php\/category\/sin-categoria\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2018\/04\/consoles-800x491.jpg?fit=800%2C491&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2018\/04\/consoles-800x491.jpg?fit=800%2C491&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2018\/04\/consoles-800x491.jpg?fit=800%2C491&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2018\/04\/consoles-800x491.jpg?fit=800%2C491&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":3291,"url":"https:\/\/geekosas.com\/index.php\/2017\/05\/23\/movies-2016\/","url_meta":{"origin":3349,"position":4},"title":"Movies 2016","author":"Daniel Fischer","date":"2017-05-23","format":false,"excerpt":"Movies make us laugh, cry, and some... sleep, so I decided to do a small analysis on 2016 movies. As with Video Games and Data Science, we did web scraping from www.metacritic.com to generate a database, in which, for each movie we obtained the following information: Country of Origin Genres\u2026","rel":"","context":"In "Sin categor\u00eda"","block_context":{"text":"Sin categor\u00eda","link":"https:\/\/geekosas.com\/index.php\/category\/sin-categoria\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.geekosas.com\/wp-content\/uploads\/2017\/03\/histogramas-300x120.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.geekosas.com\/wp-content\/uploads\/2017\/03\/histogramas-300x120.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/www.geekosas.com\/wp-content\/uploads\/2017\/03\/histogramas-300x120.png?resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.geekosas.com\/wp-content\/uploads\/2017\/03\/histogramas-300x120.png?resize=700%2C400 2x"},"classes":[]},{"id":3288,"url":"https:\/\/geekosas.com\/index.php\/2017\/05\/23\/parking-law\/","url_meta":{"origin":3349,"position":5},"title":"Parking Law","author":"Daniel Fischer","date":"2017-05-23","format":false,"excerpt":"In Chile, on February 15 (correct me if I'm wrong), the parking law was enacted, which among other things requires parking companies to: Compensate customers who are victims of theft. Prohibit fines for lost tickets. Choose one of the following two pricing methods: Charge per minute actually used. Charge per\u2026","rel":"","context":"In "Sin categor\u00eda"","block_context":{"text":"Sin categor\u00eda","link":"https:\/\/geekosas.com\/index.php\/category\/sin-categoria\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2017\/02\/1487175020-auno760729.jpg?fit=799%2C533&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2017\/02\/1487175020-auno760729.jpg?fit=799%2C533&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2017\/02\/1487175020-auno760729.jpg?fit=799%2C533&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/geekosas.com\/wp-content\/uploads\/2017\/02\/1487175020-auno760729.jpg?fit=799%2C533&ssl=1&resize=700%2C400 2x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/geekosas.com\/index.php\/wp-json\/wp\/v2\/posts\/3349","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/geekosas.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/geekosas.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/geekosas.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/geekosas.com\/index.php\/wp-json\/wp\/v2\/comments?post=3349"}],"version-history":[{"count":1,"href":"https:\/\/geekosas.com\/index.php\/wp-json\/wp\/v2\/posts\/3349\/revisions"}],"predecessor-version":[{"id":3350,"href":"https:\/\/geekosas.com\/index.php\/wp-json\/wp\/v2\/posts\/3349\/revisions\/3350"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/geekosas.com\/index.php\/wp-json\/wp\/v2\/media\/2762"}],"wp:attachment":[{"href":"https:\/\/geekosas.com\/index.php\/wp-json\/wp\/v2\/media?parent=3349"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/geekosas.com\/index.php\/wp-json\/wp\/v2\/categories?post=3349"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/geekosas.com\/index.php\/wp-json\/wp\/v2\/tags?post=3349"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}