Platform Engineering Team/Data Value Stream/Data Gateway: Difference between revisions
EEvans (WMF) (talk | contribs) →Category Metrics: add 400 status error objects Tag: 2017 source edit |
EEvans (WMF) (talk | contribs) document rfc3339 params |
||
Line 265: | Line 265: | ||
===== Category Metrics ===== |
===== Category Metrics ===== |
||
{{Note |
|||
| text = start and end parameters are of form [https://www.ietf.org/rfc/rfc3339.txt RFC3339] |
|||
}} |
|||
{| class="wikitable" |
{| class="wikitable" |
||
|+ |
|+ |
||
Line 329: | Line 332: | ||
|} |
|} |
||
===== Media File Metrics ===== |
===== Media File Metrics ===== |
||
{{Note |
|||
| text = start and end parameters are of form [https://www.ietf.org/rfc/rfc3339.txt RFC3339] |
|||
}} |
|||
{| class="wikitable" |
{| class="wikitable" |
||
|+ |
|+ |
||
Line 395: | Line 400: | ||
===== Pageviews Per Category Monthly ===== |
===== Pageviews Per Category Monthly ===== |
||
{{Note |
|||
| text = start and end parameters are of form [https://www.ietf.org/rfc/rfc3339.txt RFC3339] |
|||
}} |
|||
{| class="wikitable" |
{| class="wikitable" |
||
|+ |
|+ |
||
Line 461: | Line 468: | ||
===== Pageviews Per Media File Monthly ===== |
===== Pageviews Per Media File Monthly ===== |
||
{{Note |
|||
| text = start and end parameters are of form [https://www.ietf.org/rfc/rfc3339.txt RFC3339] |
|||
}} |
|||
{| class="wikitable" |
{| class="wikitable" |
||
|+ |
|+ |
||
Line 527: | Line 536: | ||
===== Edits Per Category Monthly ===== |
===== Edits Per Category Monthly ===== |
||
{{Note |
|||
| text = start and end parameters are of form [https://www.ietf.org/rfc/rfc3339.txt RFC3339] |
|||
}} |
|||
{| class="wikitable" |
{| class="wikitable" |
||
|+ |
|+ |
||
Line 593: | Line 604: | ||
===== Edits Per User Monthly ===== |
===== Edits Per User Monthly ===== |
||
{{Note |
|||
| text = start and end parameters are of form [https://www.ietf.org/rfc/rfc3339.txt RFC3339] |
|||
}} |
|||
{| class="wikitable" |
{| class="wikitable" |
||
|+ |
|+ |
Revision as of 00:32, 13 June 2024
Data Gateway Service
Background
The Analytics Query Service (aka AQS) exploits a pattern whereby periodic batch analytics jobs are used to maintain materialized datasets, durably persisted to facilitate low-latency access via HTTP APIs. Other use-cases of this pattern have been identified, and there is interest in constructing a platform to lower the barrier to entry, making it easier, faster, and more conducive to experimenting with new datasets. It has been suggested that on top of the (6) existing, such a platform could quickly grow by 10s of datasets in the near-term. At this scale, managing bespoke access to database(s) for an arbitrary number of internal teams would likely become problematic. Even something as mundane as a fleet-wide upgrade of native drivers could be prohibitively expensive when code owners are many, varied in resources and priorities, or worse when code becomes orphaned entirely.
One way of overcoming these concerns is to de-couple client applications from the underlying database. There are many potential benefits from this -for example- preserving the future ability to transparently migrate or redistribute datasets among clusters, or to better manage resource utilization, caching, etc. The Data Gateway service is an experimental approach to decouple clients from the underlying database, by providing a generic HTTP interface to published datasets.
The premise is simple: Candidate datasets are purpose-built, and expect results that are verbatim (or nearly so) to what is stored (including attribute naming). The Data Gateway is nothing more than thin layer wiring HTTP semantics to a database table, and returning JSON-serialized results (an array of rows containing one or more JSON objects).
Published Datasets
Image Suggestions
API
Suggestions
URL | /public/image_suggestions/suggestions/{wiki}/{page_id}
| ||||||
---|---|---|---|---|---|---|---|
Method | GET
| ||||||
Params | None | ||||||
Data | None | ||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{
"rows": [
{
"wiki": "anwiki",
"page_id": 3326,
"id": "644c90bc-ba40-11ec-ba4c-f0d4e2e69820",
"image": "14_Agosto_2016_(1).jpg",
"confidence": 80.0,
"found_on": null,
"kind": ["istype-commons-category"],
"origin_wiki": "commonswiki",
"page_qid": "Q123",
"page_rev": 1797958,
"section_heading": "section title (null if this is an article-level suggestion)",
"section_index": 1
},
...
}
| ||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| ||||||
Example | $ curl http://api.example.org/public/image_suggestions/suggestions/anwiki/3326
| ||||||
Notes |
Feedback
URL | /private/image_suggestions/feedback/{wiki}/{page_id}
| ||||||
---|---|---|---|---|---|---|---|
Method | GET
| ||||||
Params | None | ||||||
Data | None | ||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 100
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...]}
| ||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| ||||||
Example | $ curl http://api.example.org/private/image_suggestions/feedback/enwiki/53848
| ||||||
Notes |
Instanceof (cache)
URL | /private/image_suggestions/instanceof_cache/{wiki}/{page_id}
| ||||||
---|---|---|---|---|---|---|---|
Method | GET
| ||||||
Params | None | ||||||
Data | None | ||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 100
Date: Mon, 11 Apr 2022 22:07:59 GMT
{
"rows": [
{
"wiki": "anwiki",
"page_id": 3326,
"instance_of": ["Q112099", "Q3624078", "Q6256"],
"page_rev": 1797958
}
]
}
| ||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| ||||||
Example | $ curl http://api.example.org/private/image_suggestions/instanceof_cache/anwiki/3326
| ||||||
Notes |
Title (cache)
URL | /private/image_suggestions/title_cache/{wiki}/{title}
| ||||||
---|---|---|---|---|---|---|---|
Method | GET
| ||||||
Params | None | ||||||
Data | None | ||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 100
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...]}
| ||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| ||||||
Example | $ curl http://api.example.org/private/image_suggestions/title_cache/enwiki/Banana
| ||||||
Notes |
Commons Impact Metrics
API
Category Metrics
URL | /public/commons/category_metrics_snapshot/{category}/{start}/{end}
| |||||||||
---|---|---|---|---|---|---|---|---|---|---|
Method | GET
| |||||||||
Params | None | |||||||||
Data | None | |||||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...] }
| |||||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| |||||||||
Example | $ curl /public/commons/category_metrics_snapshot/:category/:start/:end
| |||||||||
Notes |
Media File Metrics
URL | /public/commons/media_file_metrics_snapshot/{media_file}/{start}/{end}
| |||||||||
---|---|---|---|---|---|---|---|---|---|---|
Method | GET
| |||||||||
Params | None | |||||||||
Data | None | |||||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...] }
| |||||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| |||||||||
Example | $ curl /public/commons/media_file_metrics_snapshot/:media_file/:start/:end
| |||||||||
Notes |
Pageviews Per Category Monthly
URL | /public/commons/pageviews_per_category_monthly/{category}/{category_scope}/{wiki}/{start}/{end}
| |||||||||
---|---|---|---|---|---|---|---|---|---|---|
Method | GET
| |||||||||
Params | None | |||||||||
Data | None | |||||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...] }
| |||||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| |||||||||
Example | $ curl /public/commons/pageviews_per_category_monthly/:category/:category_scope/:wiki/:start/:end
| |||||||||
Notes |
Pageviews Per Media File Monthly
URL | /public/commons/pageviews_per_media_file_monthly/{media_file}/{wiki}/{start}/{end}
| |||||||||
---|---|---|---|---|---|---|---|---|---|---|
Method | GET
| |||||||||
Params | None | |||||||||
Data | None | |||||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...] }
| |||||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| |||||||||
Example | $ curl /public/commons/pageviews_per_media_file_monthly/:media_file/:wiki/:start/:end
| |||||||||
Notes |
Edits Per Category Monthly
URL | /public/commons/edits_per_category_monthly/{category}/{category_scope}/{edit_type}/{start}/{end}
| |||||||||
---|---|---|---|---|---|---|---|---|---|---|
Method | GET
| |||||||||
Params | None | |||||||||
Data | None | |||||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...] }
| |||||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| |||||||||
Example | $ curl /public/commons/edits_per_category_monthly/:category/:category_scope/:edit_type/:start/:end
| |||||||||
Notes |
Edits Per User Monthly
URL | /public/commons/edits_per_user_monthly/{user_name}/{edit_type}/{start}/{end}
| |||||||||
---|---|---|---|---|---|---|---|---|---|---|
Method | GET
| |||||||||
Params | None | |||||||||
Data | None | |||||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...] }
| |||||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| |||||||||
Example | $ curl /public/commons/edits_per_user_monthly/:user_name/:edit_type/:start/:end
| |||||||||
Notes |
Top Pages Per Category Monthly
URL | /public/commons/top_pages_per_category_monthly/{category}/{category_scope}/{wiki}/{year}/{month}
| ||||||
---|---|---|---|---|---|---|---|
Method | GET
| ||||||
Params | None | ||||||
Data | None | ||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...] }
| ||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| ||||||
Example | $ curl /public/commons/top_pages_per_category_monthly/:category/:category_scope/:wiki/:year/:month
| ||||||
Notes |
Top Wikis Per Category Monthly
URL | /public/commons/top_wikis_per_category_monthly/{category}/{category_scope}/{year}/{month}
| ||||||
---|---|---|---|---|---|---|---|
Method | GET
| ||||||
Params | None | ||||||
Data | None | ||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...] }
| ||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| ||||||
Example | $ curl /public/commons/top_wikis_per_category_monthly/:category/:category_scope/:year/:month
| ||||||
Notes |
Top Viewed Categories Monthly
URL | /public/commons/top_viewed_categories_monthly/{category}/{category_scope}/{wiki}/{year}/{month}
| ||||||
---|---|---|---|---|---|---|---|
Method | GET
| ||||||
Params | None | ||||||
Data | None | ||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...] }
| ||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| ||||||
Example | $ curl /public/commons/top_viewed_categories_monthly/:category/:category_scope/:wiki/:year/:month
| ||||||
Notes |
Top Pages Per Media File Monthly
URL | /public/commons/top_pages_per_media_file_monthly/{media_file}/{wiki}/{year}/{month}
| ||||||
---|---|---|---|---|---|---|---|
Method | GET
| ||||||
Params | None | ||||||
Data | None | ||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...] }
| ||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| ||||||
Example | $ curl /public/commons/top_pages_per_media_file_monthly/:media_file/:wiki/:year/:month
| ||||||
Notes |
Top Wikis Per Media File Monthly
URL | /public/commons/top_wikis_per_media_file_monthly/{media_file}/{year}/{month}
| ||||||
---|---|---|---|---|---|---|---|
Method | GET
| ||||||
Params | None | ||||||
Data | None | ||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...] }
| ||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| ||||||
Example | $ curl /public/commons/top_wikis_per_media_file_monthly/:media_file/:year/:month
| ||||||
Notes |
Top Viewed Media Files Monthly
URL | /public/commons/top_viewed_media_files_monthly/{category}/{category_scope}/{wiki}/{year}/{month}
| ||||||
---|---|---|---|---|---|---|---|
Method | GET
| ||||||
Params | None | ||||||
Data | None | ||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...] }
| ||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| ||||||
Example | $ curl /public/commons/top_viewed_media_files_monthly/:category/:category_scope/:wiki/:year/:month
| ||||||
Notes |
Top Edited Categories Monthly
URL | /public/commons/top_edited_categories_monthly/{category}/{category_scope}/{edit_type}/{year}/{month}
| ||||||
---|---|---|---|---|---|---|---|
Method | GET
| ||||||
Params | None | ||||||
Data | None | ||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...] }
| ||||||
Error | Errors are JSON objects conforming to RFC7807 (Problem Details for HTTP APIs) with a content-type of application/problem+json .
| ||||||
Example | $ curl /public/commons/top_edited_categories_monthly/:category/:category_scope/:edit_type/:year/:month
| ||||||
Notes |
Top Editors Monthly
URL | /public/commons/top_editors_monthly/{category}/{category_scope}/{edit_type}/{year}/{month}
| ||||||
---|---|---|---|---|---|---|---|
Method | GET
| ||||||
Params | None | ||||||
Data | None | ||||||
Success | Example:HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 5000
Date: Mon, 11 Apr 2022 22:07:59 GMT
{"rows": [...] }
| ||||||
Error | Errors are JSON objects conforming to [rfc:7807 RFC7807 (Problem Details for HTTP APIs)] with a content-type of application/problem+json .
| ||||||
Example | $ curl /public/commons/top_editors_monthly/:category/:category_scope/:edit_type/:year/:month
| ||||||
Notes |