forked from UrbanInstitute/education-data-package-r
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME.Rmd
310 lines (249 loc) · 10.5 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
---
output: github_document
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r, echo = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = '#>',
fig.path = 'README-'
)
```
# educationdata
[](https://travis-ci.org/UrbanInstitute/education-data-package-r)
Retrieve data from the Urban Institute's [Education Data API](https://ed-data-portal.urban.org/) as a `data.frame` for easy analysis.
**NOTE**: By downloading and using this programming package, you agree to abide
by the Data Policy and Terms of Use of the Education Data Portal. For more
information, see [https://educationdata.urban.org/documentation/#terms](https://educationdata.urban.org/documentation/#terms).
## Installation
To install `educationdata`:
* Install the `devtools` package if you don't already have it, and run:
```{r gh-installation, eval=FALSE}
# install.packages('devtools') # if necessary
devtools::install_github('UrbanInstitute/education-data-package-r')
```
## Usage
```{r quickstart-01, message=FALSE}
library(educationdata)
df <- get_education_data(level = 'schools',
source = 'ccd',
topic = 'enrollment',
subtopic = list('race', 'sex'),
filters = list(year = 2008,
grade = 9:12,
ncessch = '340606000122'),
add_labels = TRUE)
str(df)
```
The `get_education_data()` function will return a `data.frame` from a call to
the Education Data API.
```{r quickstart-02, eval=FALSE}
get_education_data(level, source, topic, subtopic, filters, add_labels)
```
where:
* level (required) - API data level to query.
* source (required) - API data source to query.
* topic (required) - API data topic to query.
* subtopic (optional) - Optional `list` of grouping parameters for an API call.
* filters (optional) - Optional `list` query to filter the results from an API
call.
* add_labels - Add variable labels (when applicable)? Defaults to `FALSE`.
* csv - Download the full csv file? Defaults to `FALSE`.
## Available Endpoints
```{r endpoints, echo=FALSE}
source('R/get-endpoint-info.R')
df <- get_endpoint_info("https://educationdata.urban.org")
df$years_available <- gsub('and' ,'', df$years_available)
df$years_available <- gsub('\u20AC' ,'-', df$years_available)
df$years_available <- gsub('\u00E2' ,'', df$years_available)
df$years_available <- gsub('\u201C' ,'', df$years_available)
df$optional_vars <- lapply(df$optional_vars,
function(x) paste(x, collapse = ', '))
df$required_vars <- lapply(df$required_vars,
function(x) paste(x, collapse = ', '))
df <- df[order(df$endpoint_url), ]
vars <- c('section',
'class_name',
'topic',
'optional_vars',
'required_vars',
'years_available')
knitr::kable(df[vars],
col.names = c('Level',
'Source',
'Topic',
'Subtopic',
'Main Filters',
'Years Available'),
row.names = FALSE)
```
## Main Filters
Due to the way the API is set-up, the variables listed within 'main filters'
are the fastest way to subset an API call.
In addition to `year`, the other main filters for certain endpoints
accept the following values:
### Grade
| Filter Argument | Grade |
|-------------------|-------|
| `grade = 'grade-pk'` | Pre-K |
| `grade = 'grade-k'` | Kindergarten |
| `grade = 'grade-1'` | Grade 1 |
| `grade = 'grade-2'` | Grade 2 |
| `grade = 'grade-3'` | Grade 3 |
| `grade = 'grade-4'` | Grade 4 |
| `grade = 'grade-5'` | Grade 5 |
| `grade = 'grade-6'` | Grade 6 |
| `grade = 'grade-7'` | Grade 7 |
| `grade = 'grade-8'` | Grade 8 |
| `grade = 'grade-9'` | Grade 9 |
| `grade = 'grade-10'` | Grade 10 |
| `grade = 'grade-11'` | Grade 11 |
| `grade = 'grade-12'` | Grade 12 |
| `grade = 'grade-13'` | Grade 13 |
| `grade = 'grade-14'` | Adult Education |
| `grade = 'grade-15'` | Ungraded |
| `grade = 'grade-16'` | K-12 |
| `grade = 'grade-20'` | Grades 7 and 8 |
| `grade = 'grade-21'` | Grade 9 and 10 |
| `grade = 'grade-22'` | Grades 11 and 12 |
| `grade = 'grade-99'` | Total |
### Level of Study
| Filter Argument | Level of Study |
|-------------------|----------------|
| `level_of_study = 'undergraduate'` | Undergraduate |
| `level_of_study = 'graduate'` | Graduate |
| `level_of_study = 'first-professional'` | First Professional |
| `level_of_study = 'post-baccalaureate'` | Post-baccalaureate |
| `level_of_study = '99'` | Total |
## Examples
Let's build up some examples, from the following set of endpoints.
```{r example-endpoints, echo = FALSE}
df <- df[df$section == 'schools' & df$topic == 'enrollment', ]
knitr::kable(df[vars],
col.names = c('Level',
'Source',
'Topic',
'Subtopic',
'Main Filters',
'Years Available'),
row.names = FALSE)
```
The following will return a `data.frame` across all years and grades:
```{r example-01, eval=FALSE}
library(educationdata)
df <- get_education_data(level = 'schools',
source = 'ccd',
topic = 'enrollment')
```
Note that this endpoint is also callable by certain `subtopic` variables:
* race
* sex
* race, sex
These variables can be added to the `subtopic` argument:
```{r example-02, eval=FALSE}
df <- get_education_data(level = 'schools',
source = 'ccd',
topic = 'enrollment',
subtopic = list('race', 'sex'))
```
You may also filter the results of an API call. In this case `year` and
`grade` will provide the most time-efficient subsets, and can be vectorized:
```{r example-03, eval=FALSE}
df <- get_education_data(level = 'schools',
source = 'ccd',
topic = 'enrollment',
subtopic = list('race', 'sex'),
filters = list(year = 2008,
grade = 9:12))
```
Additional variables can also be passed to `filters` to subset further:
```{r example-04, eval=FALSE}
df <- get_education_data(level = 'schools',
source = 'ccd',
topic = 'enrollment',
subtopic = list('race', 'sex'),
filters = list(year = 2008,
grade = 9:12,
ncessch = '3406060001227'))
```
The `add_labels` flag will map variables to a `factor` from their
labels in the API.
```{r example-05, eval=FALSE}
df <- get_education_data(level = 'schools',
source = 'ccd',
topic = 'enrollment',
subtopic = list('race', 'sex'),
filters = list(year = 2008,
grade = 9:12,
ncessch = '340606000122'),
add_labels = TRUE)
```
Finally, the `csv` flag can be set to download the full `.csv` data frame. In
general, the `csv` functionality is much faster when retrieving the full data
frame (or a large subset) and much slower when retrieving a small subset of a
data frame (especially ones with a lot of `filters` added). In this example,
the full `csv` for 2008 must be downloaded and then subset to the 96
observations.
```{r example-06, eval=FALSE}
df <- get_education_data(level = 'schools',
source = 'ccd',
topic = 'enrollment',
subtopic = list('race', 'sex'),
filters = list(year = 2008,
grade = 9:12,
ncessch = '340606000122'),
add_labels = TRUE,
csv = TRUE)
```
## Summary Endpoints
You can now access the new summary endpoint functionality using the
`get_education_data_summary()` function.
```{r example-07, eval=FALSE}
df <- get_education_data_summary(
level = "schools",
source = "ccd",
topic = "enrollment",
stat = "sum",
var = "enrollment",
by = "fips",
filters = list(fips = 6:8, year = 2004:2005)
)
```
In this example, we take the `schools/ccd/enrollment` endpoint and retrieve the
`sum` of `enrollment` by `fips` code, filtered to `fips` codes 6, 7, 8 for the
`year`s 2004 and 2005.
The syntax largely follows the original syntax of `get_education_data()`: with
three new arguments:
* `stat` is the summary statistic to be retrieved. Valid statistics include:
`avg`, `sum`, `count`, `median`, `min`, `max`, `stddev`, and `variance`.
* `var` is the variable to run the summary statistic on.
* `by` is the grouping variable(s) to use. This can be a single string,
or a vector of multiple variables, i.e., `by = c("fips", "race")`.
Some endpoints are further broken out by subtopic. These can be specified using
the `subtopic` option.
```{r example-08, eval=FALSE}
df <- get_education_data_summary(
level = "schools",
source = "crdc",
topic = "harassment-or-bullying",
subtopic = "allegations",
stat = "sum",
var = "allegations_harass_sex",
by = "fips"
)
```
Note that only some endpoints have an applicable `subtopic`, and this list is
slightly different from the syntax of the full data API. Endpoints with
`subtopics` for the summary endpoint functionality include:
* schools/crdc/harassment-or-bullying/allegations
* schools/crdc/harassment-or-bullying/students
* schools/crdc/restraint-and-seclusion/instances
* schools/crdc/restraint-and-seclusion/students
* college-university/ipeds/enrollment-full-time-equivalent/summaries
* college-university/ipeds/fall-enrollment/age/summaries
* college-university/ipeds/fall-enrollment/race/summaries
* college-university/ipeds/fall-enrollment/residence/summaries
* college-university/scorecard/student-characteristics/aid-applicants/summaries
* college-university/scorecard/student-characteristics/home-neighborhood/summaries
For more information on the summary endpoint functionality, see the
[full API documentation](https://educationdata.urban.org/documentation/index.html#summary_endpoints).